The AI Implementation Cliff: Why 95% of GenAI Pilots Fail to Deliver
Despite billions of dollars invested in generative AI (GenAI) projects, a staggering 95% of companies are failing to achieve measurable returns, according to a recent report from MIT’s NANDA project published in August 2025. This finding, initially released in July 2025, highlights a critical gap between AI investment and tangible business outcomes. While the technology holds immense promise, scaling AI initiatives remains a significant challenge for organizations across industries.
The Core Obstacles to AI Scaling
Several factors contribute to the high failure rate of GenAI pilots. These include vendor underperformance, the high cost of implementation, and, crucially, poor data quality. Yet, a central issue is the lack of seamless integration of AI tools into existing workflows. Simply put, exciting AI employ cases often fail to gain traction when they don’t address real-world operational needs or are too difficult for end-users to adopt.
Moving Beyond Experimentation: The Demand for ROI
The era of unfettered AI experimentation is drawing to a close. Enterprise leaders and investors are now demanding that Chief Information Officers (CIOs) demonstrate measurable returns on AI investments. This shift in expectations is forcing organizations to move beyond proof-of-concept projects and focus on scaling successful AI applications.
CIO Insights: Lessons from Successful AI Implementations
To understand what’s working – and what isn’t – InformationWeek spoke with three CIOs about their experiences scaling AI initiatives within their organizations.
- Sean McCormack, CIO at First Student: Deployed Halo, an integrated AI platform for vehicle tracking, safety, communication, and payroll.
- Brian Schaeffer, CIO at OceanFirst Bank: Implemented Microsoft Copilot for 150 employees to enhance Bank Secrecy Act (BSA) and Anti-Money Laundering (AML) due diligence work.
- Padma Sastry, CIO at Lowell Community Health Center: Rolled out an AI-powered voice system to support its patient call center, handling thousands of daily inquiries.
Five Practices for Scaling AI
Despite operating in different industries and pursuing diverse AI use cases, McCormack, Schaeffer, and Sastry share common approaches to scaling AI. These five practices have been instrumental in moving their efforts beyond the pilot phase.
1. Identify a Workable Use Case
The foundation of any successful AI implementation is a clearly defined and valuable use case. First Student’s McCormack emphasized the importance of understanding existing processes and pain points. He conducted a thorough technology walkthrough with each team to identify areas where AI could deliver the most impact.
OceanFirst Bank focused on the time-consuming and complex process of BSA/AML compliance. Schaeffer recognized that AI could significantly accelerate this process, reducing the time required to analyze business relationships from days to minutes.
Lowell Community Health Center identified the need to improve patient access and support, particularly for patients with limited English proficiency. This led to the implementation of an AI-powered voice system for call triage and language support.
2. Compact and Steady Wins the Race
All three CIOs adopted a measured approach to launching and scaling their AI initiatives. Sastry stressed the importance of starting small and demonstrating value before expanding. Schaeffer described the process as building a “layer cake,” acknowledging the foundational work required before realizing significant gains.
First Student’s McCormack followed a similar iterative approach, beginning with ideation and A/B testing, followed by development, piloting, and enterprise rollout. This phased approach allowed for continuous feedback and refinement.
3. Picking the Right Vendor
Selecting the right vendor is crucial for successful AI implementation. Sastry emphasized the need to find a vendor that understood the specific needs and challenges of a federally qualified health center. She prioritized vendors willing to collaborate and adapt to the health center’s unique requirements.
4. Tracking Success and Failure
Establishing clear metrics to track the performance of AI initiatives is essential. McCormack’s team at First Student tracks metrics related to driver safety, such as instances of distracted driving and seatbelt usage. Schaeffer’s team at OceanFirst Bank uses a Power BI dashboard to monitor AI utilization and accuracy. Sastry’s team at Lowell Community Health Center tracks cost savings and patient experience metrics, such as call abandonment rates.
5. Fail Prompt
Not every AI project will succeed. McCormack advocates for a “fail fast” approach, encouraging organizations to quickly identify and abandon unsuccessful initiatives. He emphasized the value of learning from mistakes and focusing resources on promising projects. Schaeffer highlighted the importance of peer networking to learn from the experiences of other organizations.
What Doesn’t Work: Three Common Pitfalls
Alongside successful practices, several common mistakes can derail AI initiatives.
Enthusiasm Without Purpose
Driven by hype, organizations sometimes pursue AI projects without a clear understanding of their business value. Sastry cautioned against scaling AI simply because it’s exciting, emphasizing the need to focus on solving defined operational problems.
Forgetting the End User
AI tools must be user-friendly and seamlessly integrated into existing workflows to ensure adoption. McCormack’s experience with providing tablets to drivers highlighted the importance of considering the practical needs of end-users.
Waiting to Tackle Data and AI Governance
Establishing robust data governance and AI ethics frameworks is critical before scaling AI initiatives. Schaeffer emphasized the importance of prioritizing data quality and security. Sastry stressed the need to embed governance into workflows from the outset.
The Ongoing Battle of Scale
Scaling AI is an ongoing process that requires continuous iteration, learning, and adaptation. CIOs must remain vigilant about evolving regulatory guidelines, security threats, and technological advancements. By focusing on clear use cases, prioritizing data quality, and fostering collaboration between business and IT teams, organizations can increase their chances of realizing the full potential of AI.