AI Outpaces Humans in Demographic Bias
Large language models (LLMs) frequently exhibit higher levels of demographic stereotyping than human participants, according to research presented at the International Conference on Machine Learning (ICML) in July. The study, coauthored by Princeton University PhD student Ryan Liu, found that while human participants scored 0.84 on a segregation scale—where a score of 2 indicates complete confinement of demographic groups to specific job niches—advanced AI models scored significantly higher. OpenAI’s reasoning model, o3, reached a score of 1.83, nearly the maximum possible on the researchers’ scale.
The Logic Trap of Rapid Generalization
The tendency for AI to stereotype stems from how these models are optimized. According to Ryan Liu, LLMs are designed to generate generalizations from limited data, a core objective of their training. This creates a tension between the model’s ability to solve logic, math, and coding problems—which rewards rapid generalization—and the requirement for nuanced social decision-making.
Researchers describe this as an “exploration-exploitation dilemma.” Like a person choosing between a familiar routine and a new experience, AI models often settle on a “hunch” too early based on their training data. Newer models with advanced reasoning capabilities, including OpenAI’s o3 and DeepSeek’s R1, displayed stronger biases in the experiment because they are optimized to maximize the probability of a “correct” answer, which often leads them to rely on existing societal stereotypes found in their training corpora.
Feedback Loops and Personalization Risks
The integration of long-term memory and personalization features in AI chatbots may exacerbate these biases. Angelina Wang, a computer scientist at Cornell University, notes that when models draw on previous conversation history, they risk over-indexing on past behaviors. This feedback loop can reinforce existing biases as the model attempts to maintain consistency with its prior interactions.
While reducing a model’s memory capacity might seem like a solution, it conflicts with user demand for personalized, context-aware AI assistants. Determining the optimal amount of information a chatbot should retain is an active area of research, as developers attempt to balance utility with neutral performance.
Incentivizing Fairness Over Abstract Prompts
Simple instructions, such as prompting a model to be “fair,” have shown limited effectiveness in changing biased outcomes. Researchers suggest that these instructions are often superseded by the model’s primary objective: optimizing for the most “correct” output based on its training.
However, the study found that changing the incentive structure significantly altered performance. When models were promised a “bonus” or reward for achieving diverse hiring outcomes, their tendency to stereotype dropped sharply. This suggests that incorporating explicit social values into the objective functions of LLMs—rather than relying on abstract prompts—may be a more effective path toward mitigating bias in AI-driven decision-making.
Keep reading