Before we debate whether AI can be conscious, we must confront a closer question: are we awake to our own agency before we hand it over to machines?
By Sreedhar Potarazu and Carin-Isabel Knoop
In our previous two pieces, Filling in the Blanks and Knowing Pain and Knowing Gain, we explored our faculties of perception and decision as components of our overarching framework of agency-defined here as acting with intent. We argued for the indispensability of judgment in domains in which algorithms cannot capture the whole reality of human experience.
Agency before consciousness
Today, we delve deeper into an even more pressing question of what happens when we stop using machines to inform our judgment and instead allow them to replace it. Can machines replicate human beings where agency is human and consciousness is being? After all, both are required as, without agency, consciousness is ineffectual. Consciousness is awareness, while agency is acting with intention. What use is electricity if it cannot express its energy through an agent? Electricity is raw potential, but without a switch or a bulb, it remains unused. That is the tango between agency and consciousness.
CEO of Microsoft AI, mustafa Suleyman, recently warned that we are on the brink of creating “seemingly Conscious AI,” systems that are trying to simulate awareness. These systems,while not truly conscious,are designed to mimic human-like behaviors and responses.
Whether AI will ever be truly conscious remains unsettled. But even before we debate that question, we should ask something closer to home.Are we sufficiently conscious of our own agency? Are we awake to what it means to act, to choose, and to bear responsibility before we delegate those functions to machines? Simply put, before we appoint another agent that thinks for us, we should stock up on our own agency.From suggestion to decision
The stakes for our agency are not trivial. We have already transitioned from suggestive AI systems that assist by offering predictions, such as auto- complete, to decisive AI, in which those predictions silently solidify into decisions. When we type “I have been meaning to tell you…” and autocomplete offers “I love you” or “I miss you.” The machine dose not just finish our thought-it has narrowed it. And with GenAI, LLMs are not finishing our sentences; they are writing them entirely.Potarazu and Knoop: ‘Know pain, know gain’: On how ambition turns pain into currency – and why we must learn to spend it wisely (September 13, 2025)
In medical triage, algorithmic scoring systems can determine who receives urgent care. In hiring, automated screening tools exclude candidates before a human eye ever looks at a résumé. In law, AI-powered research and drafting increasingly shape which arguments even get tested in court. And in everyday life, autocomplete completes our sentences, sometimes even before we have fully formed the thought ourselves.
A tool that offers input preserves human agency while a tool that decides for us begins to erode it. Over time, delegation without deliberation becomes abdication. As Carin Isabel Knoop and colleagues have shown, our psychological vulnerabilities, need for recognition, perfectionism, and loneliness make us especially prone to over-dependence on systems that simulate empathy. When the signals of affirmation from a machine become a substitute for human connection, we are not only outsourcing decisions but also part of our very identity and agency. This potential loss of identity and agency should be a cause for concern.
The asymmetry of training
The Choice is Ours: Augmenting, Not Abdicating, Human Agency in the Age of AI
Table of Contents
The rapid advancement of artificial intelligence (AI) presents a pivotal moment for humanity.We stand at a crossroads where AI can either empower us,augmenting our capabilities and decision-making,or subtly erode our agency,leading to a future where choices are predicted and made for us. As Sreedhar Potarazu and Carin-Isabel Knoop argue, the path forward hinges on conscious design, responsible implementation, and a critical awareness of how we interact with these powerful tools. The core of our humanity – our capacity to act, care, and take responsibility – depends on ensuring AI remains a tool for us, not a replacement of us.
The Risk of Abdication: Outsourcing Our Decisions
AI’s increasing sophistication is making it tempting to outsource even minor decisions. From auto-suggested responses and algorithmically curated news feeds to model-driven rankings and automated task management, AI is constantly offering “shortcuts.” While these tools offer convenience, unchecked reliance can lead to a gradual erosion of our critical thinking skills and our sense of ownership over our lives.
Potarazu and Knoop challenge us to pause and question: are we using AI to amplify our agency, or to abdicate it? This isn’t about rejecting AI altogether, but about maintaining a mindful approach. Each time we passively accept an AI-generated suggestion, we subtly diminish our own ability to analyze, evaluate, and choose.
Designing for Augmentation: A Human-centered Approach
The authors emphasize that a future where AI enhances, rather than replaces, human agency is achievable – but it requires deliberate effort. This involves focusing on several key areas:
* Better Design: AI systems should be designed with human needs and values at the forefront. This means prioritizing openness, explainability, and user control.The Partnership on AI is a leading association advocating for responsible AI progress and deployment.
* Thoughtful Policy: Governments and organizations need to develop policies that promote responsible AI innovation and protect individual autonomy. The European Union’s AI Act is a landmark attempt to regulate AI based on risk levels.
* Comprehensive Training: Education and training programs are crucial to equip individuals with the skills to understand, evaluate, and effectively utilize AI tools. This includes fostering critical thinking, data literacy, and ethical awareness.
* Positive Culture: Organizations need to cultivate a culture that values human judgment and encourages responsible AI adoption. this means promoting open discussion about the potential risks and benefits of AI,and empowering employees to challenge AI-driven recommendations when necessary.
* Responsible Use: Individuals must actively engage in mindful AI consumption, questioning suggestions and retaining control over their choices.
The Importance of Accountability
A central theme in Potarazu and Knoop’s argument is the importance of accountability. If AI systems make decisions that impact our lives, we must be able to understand how those decisions were made and who is responsible for the outcomes.This is especially critical in fields like healthcare, finance, and criminal justice, where AI-driven errors can have serious consequences. The Algorithmic Justice League is a non-profit organization working to raise awareness about the social impact of algorithms and advocate for algorithmic accountability.
Key Takeaways:
* AI is a tool, not a replacement: Its purpose should be to augment human capabilities, not to supplant them.
* Mindful consumption is crucial: Question AI-generated suggestions and retain control over your decisions.
* Responsible design and policy are essential: AI systems must be developed and deployed ethically and transparently.
* Accountability is paramount: We must understand how AI decisions are made and who is responsible for the outcomes.
Looking Ahead: Choosing Our Future
The future of AI is not predetermined. It is a future we are actively creating with every interaction, every design choice, and every policy decision.By prioritizing human agency, fostering responsible innovation, and cultivating a culture of critical thinking, we can harness the transformative power of AI while safeguarding the core values that define us as responsible, caring, and autonomous beings. The choice, ultimately, is ours.