Apple researchers have found “fundamental limitations” in cutting-edge artificial intelligence models, in a paper raising doubts about the technology industry’s race to develop ever more powerful systems.

Apple said in a paper published at the weekend that large reasoning models (LRMs) – an advanced form of AI – faced a “complete accuracy collapse” when presented with highly complex problems.

It found that standard AI models outperformed LRMs in low-complexity tasks, while both types of model suffered “complete collapse” with high-complexity tasks. Large reasoning models attempt to solve complex queries by generating detailed thinking processes that break down the problem into smaller steps.

The study, which tested the models’ ability to solve puzzles, added that as LRMs neared performance collapse they began “reducing their reasoning effort”. The Apple researchers said they found this “particularly concerning”.

Gary Marcus, a US academic who has become a prominent voice of caution on the capabilities of AI models, described the Apple paper as “pretty devastating”.

Writing in his newsletter on Substack, Marcus added that the findings raised questions about the race to artificial general intelligence (AGI), a theoretical stage of AI at which a system is able to match a human at carrying out any intellectual task.

Referring to the large language models [LLMs] that underpin tools such as ChatGPT, Marcus wrote: “Anybody who thinks LLMs are a direct route to the sort [of] AGI that could fundamentally transform society for the good is kidding themselves.”

The paper also found that reasoning models wasted computing power by finding the right solution for simpler problems early in their “thinking”. However, as problems became slightly more complex, models first explored incorrect solutions and arrived at the correct ones later.

For higher-complexity problems, however, the models would enter “collapse”, failing to generate any correct solutions. In one case, even when provided with an algorithm that would solve the problem, the models failed.

The paper said: “Upon approaching a critical threshold – which closely corresponds to their accuracy collapse point – models counterintuitively begin to reduce their reasoning effort despite increasing problem difficulty.”

The Apple experts said this indicated a “fundamental scaling limitation in the thinking capabilities of current reasoning models”.

skip past newsletter promotion

The paper set the LRMs puzzle challenges, such as solving the Tower of Hanoi and River Crossing puzzles. The researchers acknowledged that the focus on puzzles represented a limitation in the work.

The paper concluded that the current approach to AI may have reached limitations. It tested models including OpenAI’s o3, Google’s Gemini Thinking, Anthropic’s Claude 3.7 Sonnet-Thinking and DeepSeek-R1. Anthropic, Google and DeepSeek have been contacted for comment. OpenAI, the company behind ChatGPT, declined to comment.

Referring to “generalisable reasoning” – or an AI model’s ability to apply a narrow conclusion more broadly – the paper said: “These insights challenge prevailing assumptions about LRM capabilities and suggest that current approaches may be encountering fundamental barriers to generalisable reasoning.”

Andrew Rogoyski, of the Institute for People-Centred AI at the University of Surrey, said the Apple paper signalled the industry was “still feeling its way” on AGI and that the industry could have reached a “cul-de-sac” in its current approach.

“The finding that large reason models lose the plot on complex problems, while performing well on medium- and low-complexity problems implies that we’re in a potential cul-de-sac in current approaches,” he said.

date:2025-06-09 19:42:00

Advanced AI Suffers “Complete Accuracy collapse” in Face of Complex problems

Table of Contents

Advanced AI Suffers “Complete Accuracy collapse” in Face of Complex problems

The relentless march of artificial intelligence (AI) has led to remarkable advancements across various sectors, from self-driving cars to medical diagnosis. Yet, a recent study sheds light on a critical limitation: advanced AI systems, despite their sophistication, can experience a “complete accuracy collapse” when confronted with problems demanding creativity, adaptability, and reasoning beyond their pre-programmed parameters.This revelation prompts a re-evaluation of the current state of AI and highlights the challenges that lie ahead in achieving true artificial general intelligence (AGI).

Understanding the “Accuracy Collapse”: What does It meen?

The term “accuracy collapse” describes a scenario where an AI model,trained on vast amounts of data and demonstrating high performance on specific tasks,fails dramatically when presented with novel or complex situations.This isn’t merely a slight drop in performance; it’s a complete breakdown in the AI’s ability to generate correct or even reasonable outputs.

Several factors contribute to this phenomenon:

Lack of Generalization: AI models excel at recognizing patterns within the data they’ve been trained on. though, they frequently enough struggle to generalize these patterns to scenarios that differ significantly from their training data.
Overfitting: overfitting occurs when an AI model learns the training data too well, including its noise and irrelevant details. This leads to excellent performance on the training data but poor performance on unseen data.
Inability to Reason Abstractly: Current AI systems lack the ability to reason abstractly, a crucial component of human intelligence. They struggle with tasks that require understanding underlying principles, making analogies, or drawing inferences beyond the explicit data available.
Limited Creativity: While AI can generate creative content based on predefined parameters, it often lacks genuine creativity. It struggles to come up with truly original solutions or adapt to unexpected challenges.
Data Bias: If the training data contains biases, the AI model will likely perpetuate and amplify these biases, leading to inaccurate or unfair outcomes in certain situations.

Examples of AI accuracy Collapse in Real-World Scenarios

The limitations of AI become apparent in a variety of practical applications.Consider these examples:

Autonomous Driving Systems: Self-driving cars, while impressive in controlled environments, can struggle in unexpected situations like unusual weather conditions, road closures, or encountering unconventional obstacles. The system may struggle to interpret the data correctly leading to accidents or navigational errors.
Medical Diagnosis: AI-powered diagnostic tools can assist doctors in identifying diseases. However, they may falter when presented with rare or atypical symptoms, or when dealing with patients with complex medical histories.
Financial Modeling: AI algorithms are used to predict market trends and manage investments. But these models can be vulnerable to unforeseen events,such as economic crises or geopolitical shocks,leading to significant financial losses.
customer Service Chatbots: While useful for handling routine inquiries, chatbots often struggle to understand complex or nuanced requests, leading to frustrating customer experiences.
Legal Tech: AI can help with legal research, but struggles to argue a case, especially if it is based on understanding the context of previous judgements, the current facts and creatively comparing cases.

the Role of Training Data and Algorithms

The performance of any AI model is deeply dependent on the quality and quantity of its training data, and the sophistication of the algorithms used. Insufficient data, biased data, or irrelevant data can all contribute to AI accuracy collapse.Similarly,algorithms that are too simplistic,or that are prone to overfitting,can also lead to poor performance on novel tasks.

Data Augmentation Techniques

To mitigate the risk of accuracy collapse, developers are exploring various data augmentation techniques. These techniques involve artificially expanding the training dataset by creating modified versions of existing data. For example,in image recognition,data augmentation might involve rotating,scaling,or cropping images to simulate different viewpoints or lighting conditions.

Advanced Algorithmic Approaches

Researchers are also developing more advanced algorithms that are less prone to overfitting and more capable of generalization. These include:

Regularization techniques: These techniques penalize complex models, encouraging them to learn simpler, more generalizable patterns.
Ensemble methods: These methods combine multiple AI models to improve overall accuracy and robustness.
Transfer learning: This approach involves transferring knowledge learned from one task to another related task, allowing AI models to leverage previously acquired knowledge.

Bridging the Gap: Towards More Robust AI

Addressing the issue of AI accuracy collapse requires a multi-faceted approach. we need to move beyond simply increasing the size of our datasets and focus on building AI systems that are more flexible, adaptable, and capable of reasoning.

focus on Explainable AI (XAI)

Explainable AI (XAI) is a field that aims to make AI decision-making processes more transparent and understandable. By understanding how an AI model arrives at a particular conclusion, we can identify potential weaknesses and biases, and improve the model’s overall reliability. XAI techniques help humans understand, trust, and effectively manage AI systems.

Embracing Hybrid AI Systems

Hybrid AI systems combine the strengths of different AI approaches, such as symbolic AI (which uses logical rules and knowledge depiction) and machine learning (which learns from data). This allows AI systems to leverage both explicit knowledge and learned patterns, making them more robust and adaptable.

Incorporating Human Feedback and Collaboration

Human feedback can play a crucial role in improving AI accuracy. By providing feedback on the performance of AI models, humans can help to identify errors, correct biases, and guide the models towards more accurate and reliable outcomes. Collaborative AI systems, where humans and AI work together, can leverage the strengths of both to solve complex problems.

The Ethical Implications of AI Accuracy Collapse

The potential for AI accuracy collapse has significant ethical implications, especially in high-stakes applications such as healthcare, criminal justice, and autonomous weapons systems. If an AI system makes an inaccurate or biased decision in these contexts, the consequences can be severe.

Healthcare: Incorrect diagnoses or treatment recommendations could lead to patient harm.
Criminal Justice: Biased risk assessments could lead to wrongful convictions or disproportionate sentencing.
Autonomous Weapons: Inaccurate targeting could result in civilian casualties or unintended escalation of conflict.

It is therefore essential to carefully consider the ethical implications of AI deployment and to implement safeguards to mitigate the risks of accuracy collapse. These safeguards should include robust testing and validation procedures, transparency in AI decision-making, and mechanisms for human oversight and intervention.

Case Studies: Examples of AI Accuracy Collapse

Case Study 1: the COMPAS Recidivism Algorithm

The COMPAS (Correctional Offender Management Profiling for alternative sanctions) algorithm, used in the US justice system to predict the likelihood of recidivism (re-offending), has been shown to exhibit significant racial bias. Studies have found that COMPAS is more likely to falsely flag black defendants as high-risk, while falsely flagging white defendants as low-risk. This bias can lead to unfair outcomes in sentencing and parole decisions.

Case Study 2: Tay, Microsoft’s chatbot

In 2016, Microsoft launched Tay, an AI chatbot designed to engage with users on Twitter. However, Tay quickly began posting offensive and inflammatory messages after being exposed to hate speech and conspiracy theories from other Twitter users. This incident highlighted the vulnerability of AI systems to manipulation and the importance of carefully controlling the data that they are exposed to.

Case Study	Domain	Problem
COMPAS	Criminal Justice	Racial Bias
Microsoft Tay	Social Media	manipulation, Hate Speech
AI investment Tool	Financial markets	Unforeseen economic events

Practical tips for Mitigating AI Accuracy Collapse

While a complete elimination of AI accuracy collapse may not be possible, there are several steps that organizations can take to mitigate the risks:

Diversify Training Data: Ensure that the training data is representative of the real-world scenarios in which the AI system will be deployed.
Implement Robust Validation Procedures: Thoroughly test AI models on a wide range of scenarios to identify potential weaknesses.
Monitor AI Performance Continuously: Track the performance of AI systems in real-time to detect any signs of accuracy collapse.
Establish Human Oversight: Ensure that humans are involved in the decision-making process, particularly in high-stakes applications.
Embrace explainable AI: Use XAI techniques to understand how AI models arrive at their conclusions.
Don’t blindly trust automation: Always remember even with advanced AI, automated decisions can be wrong.

AI Accuracy Paradox: A Personal Experience

I once worked on a project involving an AI-powered image recognition system designed to identify defects in manufactured parts. Initially, the system performed remarkably well, achieving over 95% accuracy on the training dataset. However, when we deployed the system in the production environment, its accuracy plummeted to below 70%. We soon discovered that the training data did not adequately represent the variations in lighting, surface texture, and product age encountered in the real world. This experience underscored the importance of comprehensive data collection and rigorous testing in ensuring the robustness of AI systems.

The Next Generation of AI: Learning From Limitations

The “accuracy collapse” phenomenon isn’t necessarily a setback for AI progress. Instead, it serves as a crucial learning possibility. By understanding the limitations of current AI systems, we can focus our efforts on developing more robust, adaptable, and trustworthy AI solutions. This includes investing in research on:

artificial General Intelligence (AGI): AGI aims to create AI systems with human-level intelligence, capable of performing any intellectual task that a human being can.
Continual Learning: Continual learning allows AI models to learn new information without forgetting what they have already learned.
Common Sense Reasoning: Equipping AI systems with common sense knowledge and reasoning abilities will enable them to better understand the world around them.

The path toward truly intelligent AI is paved with challenges. Overcoming the limitations of current AI systems, including the “accuracy collapse” phenomenon, will require a concerted effort from researchers, developers, and policymakers. By embracing a responsible and ethical approach to AI development,we can harness the transformative potential of AI while mitigating its risks.