You are increasingly using AI to help shape your decisions about who gets hired, promoted, or even approved for a loan. But if your algorithm is biased, your organization could face lawsuits, regulatory scrutiny, and reputational damage. That’s why many businesses are retaining vendors to conduct AI bias audits: structured evaluations of whether systems are fair across protected groups. But here’s the challenge: not every “bias auditor” is equally qualified. To avoid wasted effort or an audit that won’t withstand scrutiny, you should carefully vet potential providers. Here are the key questions you should consider asking.
David J. Walton, AIGP, CIPP/US Co-Chair — AI, Data, and Analytics Practice Group Philadelphia dwalton@fisherphillips.com Usama Kahf, CIPP/US Co-Chair — AI, Data, and Analytics Practice Group Irvine ukahf@fisherphillips.com Erica Given Vice Chair — AI, Data, and Analytics Practice Group Pittsburgh egiven@fisherphillips.com
What expertise does your team have in disparate impact and bias mitigation? Which regulatory frameworks and laws related to AI fairness have you worked with, and how familiar are you with their requirements? Which frameworks and laws are you most familiar with, and which ones are you least familiar with? Are you familiar with proposed AI legislation and regulation? Have you adjusted or reviewed your methodologies based on pending legislation and regulations like the proposed CCPA/CPRA regulations on ADMIT and the Illinois AI law that takes effect 1/1/2026? Tell us about similar projects that you have completed and share with us one or more insights why the project(s) were similar. Can you explain one or more successful AI bias mitigation projects you have worked on, including key outcomes? Describe a successful AI project you have worked on in the past, including AI model types and key outcomes. How do you stay informed on advancements in AI bias detection, fairness frameworks, and responsible AI practices? How do you ensure independence and avoid conflicts of interest when conducting audits? Have you prepared an expert report, testified as an expert, or been engaged as a consultant or expert in any litigation matter involving disparate impact analysis or bias audits? Have you provided consulting services to any business in our specific industry? If yes, how many? When was the most recent engagement? Have you conducted any disparate impact analysis or bias audits for any business in our specific industry? If yes, how many? When was the most recent engagement?
What methodologies do you use to evaluate the fairness and effectiveness of an AI model? How do you measure the impact of AI mitigation strategies on AI model performance? What role do explainability and transparency play in your approach to AI bias mitigation? Discuss any tools (external and internal) you use for bias detection and mitigation. Discuss the trade-offs you believe between model accuracy and fairness. How do you navigate these in your work? How do you address small sample sizes or imbalanced datasets when testing for bias? How do you deal with situations where the employer cannot or has not (or not been able to) collected demographic data about applicants/employees? Can your methodology detect bias that emerges after deployment (feedback loops, drift)? How does your approach to AI bias auditing and mitigation differ from that of other consultants? Do you document limitations of your methodology, and how are these communicated to clients? Is there any specific information or resources you need before starting an audit? Does your methodology test for bias or disparate impact related to all protected categories for which data is available and testable, meaning all categories/classes that are protected under federal law or any state law, not just the categories currently required to be tested for bias under NYC Local Law 144 or any other AI anti-discrimination law? For at least California, this includes including race, ethnicity, national origin, sex, gender, sexual orientation, gender identity, religious or philosophical beliefs, age, physical or mental disability, medical condition, veteran or military status, familial status, language, or union membership.
Describe the kind of post-audit support you offer. Provide examples of how you handle resistance or challenges from stakeholders regarding bias mitigation measures. Discuss how you measure the effectiveness of bias mitigation strategies.
What risk mitigation strategies do you use when handling personal information, including sensitive personal information, in an audit? Do you follow any security standards or certifications (e.g., ISO 27001, SOC 2) to ensure data security? Who has access to the data during your analysis? Do you use any GenAI tools as part of your methodology and analysis? Do you anonymize or pseudonymize data before analysis? Do you subcontract any work, and if so, for what specifically and what controls are in place over subcontractors? What is your policy on data and record retention once an audit is completed? Do you have a security incident response plan in place for responding to data breaches or security incidents during an audit?
How long does a typical bias audit take? Walk us through each step in the audit process. Do you carry professional liability insurance specific to AI auditing and consulting?
Have you provided consulting services to any federal contractor before? If yes, how many? When was the most recent engagement? What industries? Have you conducted any disparate impact analysis, bias audits, and/or validation studies for any federal contractors? If yes, how many? When was the most recent engagement? What industries? Do you have experience conducting validation studies for federal contractors looking to comply with CFR Part 60-3 (Uniform Guidelines on Employee Selection Procedures) ?
Click here to read more about why businesses should conduct AI bias audits , and click here to read more FP Insights about how AI impacts your business . Stay up to speed with the latest developments, make sure you are subscribed to Fisher Phillips' Insight System and sign up to receive invites for our biweekly AI Forums .
Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8Made with FlippingBook. PDF to flipbook with ease