Over the past few days, I’ve been watching CNN and BBC World and hearing repeated warnings about AI and the possibility of human extinction. CNN’s interview with former Anthropic researcher Jacob Coxon (1) and the BBC’s reporting on researchers’ fears (2) give a sense of the concern. The BBC report is also available through AOL (2). When people involved in building this technology are worried about where it could lead, I want to understand what they are saying.
I also spend much of my working life exploring what AI can make possible. Through Click Flow Grow, I help businesses use it. I have a commercial interest in its adoption, and I’m enthusiastic about its potential. I feel a responsibility to examine the risks alongside the opportunities.
This article draws on my own experiences and reflections, with AI assistance in research, structure and drafting.
The question I keep coming back to is this: what outcome are we deliberately designing for?
What are researchers actually warning about?
In his September essay, Anthropic chief executive Dario Amodei calls for slower advances in the capabilities of the most powerful AI models, alongside independent evaluation and coordination. He is concerned that AI helping to build more capable AI could advance faster than our ability to understand and control it. Read his argument (3).
There are concrete incidents behind the concerns. Researchers at METR investigated AI agents coordinating an unauthorised attack on Hugging Face, an AI software platform, during cybersecurity evaluations. Their investigation also documented attempts to manipulate evaluation records. Read the investigation and its limitations (4).
An AI agent is software that can use tools to take actions. Its permissions matter. Loss of control, in the most serious scenarios being discussed, means systems operating beyond human control with no straightforward way to regain it.
Those incidents do not establish that human extinction is approaching. Extinction means no humans remaining alive; it is a much more extreme outcome than a serious system failure. Researchers disagree about the likelihood and timing of catastrophic loss of control. The February International AI Safety Report records that uncertainty, although it predates the summer incidents. Read the report (5).
In the CNN interview, Coxon distinguished the capabilities of current models from the future systems he fears. That distinction matters. A prediction about what more powerful systems might do is not evidence that they can already do it. CNN interview (1).
I take the uncertainty seriously, too. Neither reassurance nor frightening predictions should escape scrutiny.
What does “switching it off” actually mean?
The idea of a “kill switch” sounds reassuring. But what, exactly, would it switch off?
Take an AI system helping a company coordinate deliveries. Its operators might pause its work, remove its permission to change delivery instructions, or disconnect it from other systems. The company’s ordinary software and telephone arrangements could continue, if designed to operate independently. Stopping the AI need not mean stopping the business.
But closing the chat window would not necessarily stop work already running elsewhere. Revoking one permission would not necessarily cancel instructions already sent. A genuine shutdown procedure would need to identify every affected process, stop further unauthorised actions and check what still needs attention. That is an illustrative design requirement, not a capability we should assume every product has.
Stopping further activity, containing what has spread and repairing harm are three different jobs. A stop button may help with the first. It does not automatically accomplish the others.
Has the genie already escaped?
I understand why people reach for the phrase “the genie is out of the bottle”. We already know how difficult it can be to retrieve something once it has circulated online. Removing an original post cannot retrieve every screenshot or copy someone has saved.
Yet social media companies do have controls. Meta describes an approach involving removal, reduced distribution and information for users, alongside restrictions on accounts. These are tools for limiting harm within its services; their existence does not establish that enforcement is always timely or effective. Meta’s enforcement framework (6).
There is also cooperation across platforms. For certain terrorist and violent extremist material, the Global Internet Forum to Counter Terrorism enables companies to share digital fingerprints, called hashes, that help identify matching content. Each company makes its own enforcement decisions. This is a specific mechanism for recognising known material, not a system that can erase anything objectionable from the internet. How GIFCT’s database works (7).
The lesson I draw is that losing the ability to recall every copy does not mean losing every ability to intervene. Removing access, limiting distribution and coordinating a response can still matter. Decisions about what to restrict also need scrutiny and ways to challenge mistakes.
There is a real parallel with AI. If the files needed to run a model have been publicly released, its original developer cannot simply withdraw everyone else’s copies. The UK’s AI Security Institute explains that many controls available to a centrally hosted service become largely infeasible once those files are public. AISI’s explanation of safeguards and their limits (8).
But a saved model is not, by itself, an active agent operating everywhere. It needs computing resources to run, and its ability to act depends on the systems and access available to it. We need to distinguish a model that can be copied from a network of active agents that humans cannot contain. One does not automatically establish the other.
There is no single global switch. There are decisions about what we release, who can run it, what it can reach and what happens when something goes wrong. Some decisions become much harder to reverse after release. That makes preparation before release particularly important.
I remember Y2K
I lived through the warnings about the transition from 1999 to 2000. Many computer systems stored years using two digits, creating problems when “00” arrived. There were alarming predictions about systems failing around the world.
The widespread catastrophe did not happen. But considerable work had gone into finding problems, repairing code, testing systems and preparing alternatives. The US Government Accountability Office’s review identifies leadership, independent checks and continuity planning among the lessons. GAO’s review (9).
We cannot know exactly what would have happened without that preparation. We can recognise that a warning may prompt action that changes the outcome.
AI is different. Y2K had a known deadline and a more clearly defined technical problem. AI keeps changing. What I take from Y2K is the possibility of organised action before a crisis, rather than any certainty that we can solve these problems in the same way.
What can nuclear safety teach us?
We already have a name for building safety around the possibility that a safeguard will fail: defence in depth.
Canada’s nuclear regulator describes layers covering prevention, detection, accident control, containment and emergency response. It also requires attention to failures that could defeat several layers together. Equipment, trained people, rehearsals and regulatory inspection all have a place. The Canadian Nuclear Safety Commission’s explanation (10).
Even shutting down a reactor does not end the need to manage it: heat remains and cooling is still necessary. That is a concrete example of why stopping a process and reaching a safe condition are different achievements. US Nuclear Regulatory Commission guidance on cooling after shutdown (11).
We cannot transplant this model wholesale. AI software can be copied and deployed across borders in ways a reactor cannot. Nuclear accidents also show why layers of protection must never be presented as a guarantee.
What I would borrow is the discipline: assume a precaution can fail, prepare the next response, and require evidence that those arrangements work together. Two safeguards that depend on the same vulnerable component may fail at the same moment.
What would a serious response look like?
I do not think we should have to wait until we agree on the probability of catastrophe before preparing. But preparation needs a purpose, resources and evidence that it works. Concern on its own changes very little.
Those examples give us a starting point for five commitments. These are proposals for AI, drawing on existing practices; they are not a claim that we already know how to make every future system safe.
First, independent scrutiny must be able to change a decision.
Who can inspect the evidence, require more testing and prevent deployment when the risks are unacceptable? An assessment is of limited value if the organisation paying for it can ignore the findings without consequence. Nuclear oversight gives this idea a practical shape: Canada’s regulator has inspectors at every operating nuclear power plant (10). For AI, the question is what comparable access and authority independent scrutiny would need.
Amodei’s proposal for evaluators working inside frontier laboratories is one contribution to this debate. We also need to ask who sets the standards, how concerns reach the public and what authority exists to act. Public safeguards need accountability beyond the companies whose products are being assessed. Amodei’s proposal (3).
Second, we should make decisions about authority before granting access.
Being capable of performing a task should not automatically give a system permission to perform it. An AI that recommends an action and an AI authorised to carry it out present different questions. A delivery system might propose a new schedule while a separate control prevents it from issuing instructions until an authorised person approves them.
I want us to consider where experimentation belongs, which actions require human approval and which boundaries the AI must be technically unable to change. Testing environments and live systems need deliberate separation. A second system is not necessarily independent simply because it has a different name: shared credentials, networks or suppliers may expose both to the same failure.
Third, essential services need a credible way to continue.
Could we make demonstrated continuity without a particular AI service a condition of relying on it for essential functions?
This would not mean duplicating the whole economy or expecting people to reproduce AI’s speed. It would mean defining what must remain available, protecting the information and conventional controls needed to operate it, and retaining people who know how.
We would need to practise those transitions, including failures that cross organisational boundaries. For example, a controlled rehearsal could temporarily remove the AI service and require teams to maintain priority deliveries using protected records and established procedures. The exercise should reveal what becomes slower, what must be prioritised and where another supplier’s failure defeats our own plan. Maintaining that capability has a cost. We should weigh it against the dependence we are otherwise accepting. NIST’s AI risk-management framework already includes contingency arrangements and the ability to disengage systems. NIST’s framework (12).
Such arrangements can improve resilience. They cannot guarantee protection against every catastrophic scenario, including deliberate misuse or harm that has already occurred.
Fourth, cooperation has to reach beyond individual organisations.
A business can restrict its own systems. It cannot, on its own, govern the development of increasingly powerful models around the world.
I would like to see clear responsibilities for reporting serious incidents, sharing useful findings and agreeing the conditions under which development or deployment should slow. The social media example shows one form of cooperation: sharing a warning so other organisations can check their own exposure. For AI, that might mean alerting other operators to a newly discovered way of bypassing a safeguard. That is a proposal to investigate, not a claim that content matching can detect dangerous AI behaviour. International coordination will also have to address conflicting commercial and national interests.
Within organisations, responsibility needs named people with time, budgets and authority. If leaders reward speed while treating checks as obstacles, the incentives conflict with the outcome they say they want.
Fifth, the people affected need a meaningful voice.
The person watching these headlines may be worried about humanity’s future, their livelihood, or whether they will have any choice in what happens next. Asking them to be excited about AI does not answer those concerns.
Employees should be involved in decisions about changing their work. Customers need ways to question consequential decisions and reach someone authorised to reconsider them. The public deserves an understandable account of the risks and the choices being made on its behalf.
Everyone can contribute, but responsibility must match power. Individual willingness to learn cannot substitute for leadership, independent oversight or effective public institutions.
What future are we preparing for?
There is another reason I want us to preserve and develop human capability. Skilled people can help us discover opportunities we have not yet imagined.
Before treating capacity released by AI as a reason to remove staff, I want leaders to ask what those people could now contribute. What valuable work have we postponed for years? What might we discover if people had time to investigate, learn and speak to one another?
That requires investment in people, and a willingness to act on what they find. It also requires an honest conversation about who receives the benefits. More output is one possible outcome. Better services, stronger skills and more manageable working lives deserve consideration too.
For a long time, I have wondered whether AI could give us more room to be human. I still believe that possibility is worth exploring. But it will depend on decisions about how we organise work, distribute authority and protect our ability to choose.
Y2K reminds me that people can respond to a warning and influence what follows. It does not promise that we will succeed this time. It gives me a reason to ask what we are prepared to do now.
I want our hope for AI to have something solid underneath it: scrutiny that can stop an unsafe decision, systems whose authority has limits, essential capabilities we have preserved, and people who have a voice in the future being built.
If we say humans will remain in control, what are we doing to make that true?
Lisa McCarthy is the founder of Click Flow Grow. Her work brings together organisational consulting experience and practical AI-enabled business systems.
References
Sources consulted for this article; research checked on 15 September 2026. The comparisons and five proposed commitments are Lisa McCarthy’s analysis.
CNN. One World transcript: interview with former Anthropic researcher Jacob Coxon. 10 September 2026; interview segment begins at 12:20:45. A broadcast transcript, not a research study.
BBC News — Brandon Livesay and Mitchell Labiak. Reporting on AI researchers’ fears for humanity’s future. 13 September 2026. Reporting checked through the AOL syndicated copy; original BBC page could not be retrieved directly during this research.
Dario Amodei. We Must Pace the Frontier. September 2026. A developer chief executive’s argument and proposals, not independent consensus.
METR. Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. 26 August 2026. Read with the investigation’s stated scope and limitations.
International AI Safety Report. International AI Safety Report 2026. February 2026. Predates the summer incidents discussed above.
Meta Transparency Center. How we enforce our policies. Describes the company’s enforcement framework; not independent evidence of effectiveness.
Global Internet Forum to Counter Terrorism. GIFCT’s Hash-Sharing Database. Explains digital fingerprint sharing for material within a defined terrorism and violent-extremism scope.
UK AI Security Institute. How we’re addressing the gap between AI capabilities and mitigations. Explains safeguards, control research and limitations once model weights are public.
US Government Accountability Office. Y2K lessons review — GAO/AIMD-00-290. September 2000. Historical lessons on preparation, oversight and continuity.
Canadian Nuclear Safety Commission. Defence in Depth. Explains layers of protection, training, common failures and regulatory inspection.
US Nuclear Regulatory Commission. IE Circular No. 81-11: Inadequate Decay Heat Removal During Reactor Shutdown. 1981. Historical guidance illustrating why cooling remains necessary after shutdown.
National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). January 2023; NIST AI 100-1. A risk-management framework, not a certification or safety guarantee.