2,512 Loan Officers Skip AI Bias Checks When Bonuses at Stake

A new study of 2,512 people acting as loan officers on real $10,000 requests found that most avoided reviewing explanations of how the AI assessed risk. Fewer than half chose to view explanations about how risk determinations were made. The pattern intensified when participants' own bonuses depended on repayment rates, and when they were told the explanations might flag potential race- or gender-based bias.

The researchers offered each participant AI-generated risk scores on loan applications. Most borrowers took the AI's recommendation. Click-through to understand the AI's reasoning dropped below 50%. When participants did view the explanation, they overrode the AI more often – the tool's recommendations only got challenged when users saw the logic behind them.

The finding carries a concrete implication for fintech lenders like Upstart (UPST) and traditional banks rolling out AI credit models. These institutions have poured billions into explainable AI, the idea that transparency lets humans catch bias and error. The study suggests transparency only works if the human in the loop has a reason to use it. When incentives pull the other way – a bonus tied to repayment, or anxiety about discovering bias – the explainability feature becomes a dead link.

A model that nobody challenges is a model that reinforces its own blind spots. If loan officers skip bias audits, the portfolio could accumulate hidden risk: higher defaults in protected groups, or regulatory exposure from fair-lending enforcement. The Consumer Financial Protection Bureau has signaled interest in AI explainability. New rules requiring companies to prove that employees actually review explanations – not just that explanations exist – would narrow the cost advantage of AI-driven lending.

The study's authors argued that companies must do more than add a "why" button. Compensation structures, audit requirements, and organizational norms must reward the act of questioning the AI, not just using it. A loan officer who catches a model error should earn a bonus. Lenders that already bake human override incentives into their operations are better positioned. Those that rely on transparency alone have work to do.