The Governance Gap: Why AI Agents Are Running Wild in Enterprise Systems

In an era where artificial intelligence (AI) is reshaping enterprise operations, a critical oversight is emerging: the lack of governance for autonomous AI agents. According to a recent analysis, 79% of organizations now have some form of AI agent in production, with 96% planning to expand their use. However, the rapid adoption of these agents has outpaced the development of frameworks to manage their impact on system stability. This gap is creating a new class of production incidents that enterprises are ill-equipped to handle.

The Hidden Risks of Autonomous AI Agents

AI agents, designed to autonomously manage tasks like service restarts, traffic rerouting, and resource scaling, are increasingly integral to enterprise infrastructure. However, their actions often trigger cascading failures that are difficult to trace. For example, an agent might restart a service cluster to address latency, unaware that three other services are handling peak traffic, a shared connection pool is at 87% utilization, or a dependent database is undergoing a background index rebuild. The result is a “thundering herd” effect, where the agent’s intended fix exacerbates the problem.

These incidents are often misclassified. The AI Incidents Database reports a 21% rise in AI-related incidents from 2024 to 2025, but this count likely underrepresents the true scale. Many organizations lack incident classification systems that recognize autonomous agent actions as root causes, leading to postmortems that fail to address the underlying governance issues.

The Resilience Budget Model: A New Approach

To mitigate these risks, experts like Sayali Patil—a leader in enterprise AI reliability at Cisco and Splunk—advocate for a “resilience budget” model. This approach treats system capacity to absorb stress as a consumable resource, rather than a static threshold. Key signals in this model include:

• SLO burn rate: Measures how close a system is to breaching service-level objectives.
• Dependency saturation: Tracks the utilization of shared resources like connection pools.
• Application behavioral signals: Monitors user-facing metrics such as session completion rates and API patterns.

By integrating these signals, enterprises can dynamically assess whether an agent’s action aligns with the system’s current capacity. If the resilience budget is depleted, agents must wait or escalate decisions to human operators.

Challenges in AI-Driven Chaos Engineering

While large language models (LLMs) are being used to generate chaos hypotheses from dependency graphs and incident reports, their effectiveness is limited by outdated data. A model trained on stale dependency graphs may propose experiments with incorrect blast radius assumptions, risking unplanned outages. Postmortems, by contrast, offer validated insights into real-world failures, making them a more reliable source for hypothesis generation.

Crucially, AI should not replace human judgment in ambiguous scenarios. Decisions about whether to act must account for factors beyond technical metrics, such as pending deployments, on-call staffing levels, or customer commitments. As Patil emphasizes, “A circuit breaker that hands ambiguous cases to a human is not a weakness—it’s a necessity.”

The Path Forward for Enterprise Governance

Enterprises must audit their existing AI agents to map their actions against live system signals. This process will reveal agents operating outside resilience accounting frameworks, a common issue among large-scale adopters. Key steps include:

• Defining explicit floor conditions for agent actions.
• Modeling agent decisions as experiments, not isolated events.
• Implementing intent-based verification to ensure compliance with predefined boundaries.

As Gartner predicts that 33% of enterprise software will include agentic AI by 2028, the urgency to address these governance gaps is clear. Organizations that fail to act risk not only system instability but also reputational damage and financial losses.

Key Takeaways

• 79% of enterprises use AI agents, but many lack frameworks to manage their impact.
• Cascading failures from agent actions are often misclassified in postmortems.
• The resilience budget model treats system capacity as a dynamic, consumable resource.
• LLMs can aid chaos engineering but are limited by stale data; postmortems offer more reliable insights.
• Human oversight remains critical in ambiguous scenarios to prevent consequential errors.

As AI agents become more prevalent, the need for robust governance has never been more urgent. By bridging the gap between autonomy and accountability, enterprises can harness the benefits