AI risk is often framed as a story about a future machine becoming hostile. But the more immediate danger may be less cinematic: systems that are deployed faster than society can understand, govern, or absorb them. The modern version of Roko's “basilisk” is not a single model waiting to punish us, but a race of incentives where companies, states, workers, and users all feel forced to accelerate. This essay argues that the real problem is not intelligence alone, but the architecture around it: who controls AI systems, what permissions they receive, how they are verified, and how their power reshapes human agency.
Artificial intelligence has been treated as the defining technology of this era for years now. The standard advice has been simple: get onboard, upskill, adapt. “AI won’t replace humans,” we are told, “humans who use AI will replace humans who don’t.”The comparison is usually the internet. Yes, it changed everything, but it did not erase work entirely; it rearranged it. Bank clerks became computer-assisted bank employees. Office workers moved from paper to software. Entire industries adapted, but the human stayed in the loop.
Step back for a second and look at the world with context. Does today feel like the world we inhabited five years ago? Even two years ago? Even six months ago? We've been accelerating off a cliff in an arms race against the human race, by the human race. To recap, in the last few years we've gone from AI that could barely hold a coherent conversation without collapsing into hallucinated nonsense, to AI that can reason through tasks, write and debug code, use tools, browse files, operate agents, generate research plans, and sit inside actual company workflows. In the last year alone, we've seen memory systems, agent frameworks, multimodal models, local models, open-weight models, and cheap deployment stacks move from "cool demo" territory into "this can replace parts of someone's job" territory. We've released models small enough to run locally while still being useful, models good enough to automate knowledge work, and models powerful enough that labs openly debate what is and isn't safe to release. This isn't a "companies are laying off workers because they think AI will replace everything, watch them come crawling back when they realize they can't" situation. The next steps in AI have very little to do with some magical new machine learning breakthrough and way more to do with the architecture built around the models. Architecture that makes them stable. Architecture that gives them memory, tools, permissions, verification, orchestration, and guardrails. Architecture that lets them enter real-world workflows reliably enough to cut the need for large parts of the workforce in the first place.
And AI is still nowhere near its peak in terms of actual utility and depth. While model strength itself might hit a peak soon, the age of optimization and architecture is yet to come. Models will only become cheaper, workflows using models will only get more reliable, implementation of AI into companies will only get easier. We've been on a scorched trajectory over the last few years, on a narrow unwavering path towards a peak on what the model can actually do. When we do reach that peak, all attention is going to be diverted into architecture and methods to safely and cheaply implement this into the real world. Whatever limitations of LLMs exist today -- hallucinations, cost -- are not "un-fixable" and will soon be approached by the very frontier that has developed a model to the state it's in now.
Look at Gemma 4: an open model from Google designed for local agentic intelligence, with on-device Android support and major efficiency gains. This wasn't driven by a single new scientific breakthrough. It's more like a set of architectural and systems-level improvements: identifying inefficiencies in how transformer-based models allocate computation, restructuring the model around those bottlenecks. This, in my opinion, is only just the beginning for architectural development of AI. Even if AI were to magically stop becoming "better" today, the world of application of Artificial Intelligence is not closed. There exist a near infinite number of optimizations to make, architecture to create around AI and depth to be built. The point isn't just that models are getting smarter; it's that the surrounding architecture is making them cheaper, more private, and deployable closer to the user.
Let me be direct: I believe the answer isn't stopping AI development. It's building architecture around it. Better harnesses. Governance layers. Deterministic control systems. Reliable infrastructure. Impenetrable sandboxes. Protocol. Guardrails. That's where safety actually lives. The models themselves may or may not plateau soon, but we've only just begun to scratch the surface on actual architectural growth.
To those unfamiliar with Roko's Basilisk, it is a thought experiment from the early 2010s which suggests that a future omnipotent superintelligence might punish those who did not contribute to its growth despite knowing of its existence. It suggests an obligation to further growth of AI so as to protect oneself from the Basilisk's wrath, which directly leads to the creation of the very Basilisk that enacts revenge on defaulters. The Basilisk manifests itself to AI labs in the form of other AI labs, an entity that if it sees its potential through, will enact revenge, irrelevance, upon those who chose not to contribute to the growth. Labs and the people working at such institutions have reached a kill or be killed state when it comes to AI, either support the growth of the very being that threatens your existence, or sit still and watch while the rest of the world gaps you anyways in building AI.
This is where the Basilisk framing matters beyond the thought experiment. When people treat AI as a single monster being built by a few identifiable people, they start looking for a few identifiable people to blame. The fear becomes personalized. The system-level problem gets reduced to CEOs, labs, and villains. But that completely misunderstands the nature of the race. The danger is not one person pressing a button. The danger is an incentive structure that makes everyone feel like they have to keep pressing forward.
This is why I write this after the attacks on Sam Altman's house. The first time, a Molotov. The second time, someone shot directly at it. I heavily condemn actions like this, not because I think Altman is the center of AI, but because attacking one person proves the exact misunderstanding I'm arguing against. AI is not confined to one CEO, one company, or even one country. Anyone participating in violence like this is not stopping AI. They are just attacking a human being while the actual race continues everywhere else. There is no real "safety" from AI by removing CEOs of big tech companies or imposing restrictions on growth of LLMs. We opened Pandora's box to development of AI the second we touched GPT-2.0 and people got a real taste for what Artificial Intelligence could be. There is no stopping researchers or passionate, curious people from developing and developing their own artificial intelligence models.
Reductions in funding or law-imposed restrictions can shape where development happens, who has access to frontier-scale compute, and how models are deployed, but they cannot erase the underlying race. At best, regulation changes the terrain; it does not remove the incentive to build. A revolution overthrowing big tech isn't going to fix anything. Any smart individual is now equipped with the knowledge and resources to run LLMs locally. Reductions in funding would only lead to revision and development of cheaper models.
The entire AI industry is built on assumptions and certain brittle, excessive architecture, each of which will be scrutinised and broken to produce cheaper, better models if and when the copious amounts of money being thrown at this runs dry. If Artificial intelligence truly is the path to superintelligence, no revolution, of whatever magnitude, could fix it. In trying to destroy AI development outright, we risk building the very thing we claim to fear: a world where access to the most significant scientific tools of the century is controlled by another nation's elite. This is not nihilism or a "we're all doomed" manifesto, I do not believe so. I do believe attacking another human being because you disagree with him or are scared of what he's building, is inhuman. The real path to safety against AI is thorough implementation of protocol, guardrails and architecture around the AI that does end up being developed. This could be in many forms, better harnesses, governance architecture, better training data, even better infra like impenetrable sandboxes.
The problem with "PauseAI" is not that safety advocates are stupid. It is that a pause aimed only at frontier labs does not automatically pause the incentive structure, the global competition, or the open-source diffusion already underway. A pause can buy time, but only if that time is used to build enforceable safety architecture: compute governance, deployment rules, evals, liability, sandboxing, and control systems.
People like to bring up nuclear weapons, stating MAD -- Mutually Assured Destruction -- as a pro AI safety argument. I personally disagree with this sentiment, because MAD has one key part that makes it function as a treaty. "Assured" destruction is very different from the possibility of destruction. MAD also only applies when the risk is mutual. If one country restricts itself while another keeps building, the result is not mutual restraint; it is an asymmetry where the side that does not pause accumulates leverage. That is the risk of a PauseAI approach that is not globally enforceable: it may move development elsewhere rather than stopping it. We as a species do not tend to step back in the face of a very real possibility of extinction, unless we know for sure it will be the end.
Think about the first ships launched into space. There was obvious controversy: extra-terrestrial viruses or entities could possibly wreak havoc on Earth. Did this possibility stop NASA? We set up safeguards we deemed sufficient and launched toward the moon. There's nuance in that decision -- the lack of atmosphere in outer space made it lower-stakes -- but the core truth remains, we reach into darkness willingly, and we build safeguards to manage what we find.
That is the actual choice in front of us. Not acceleration versus safety, but reckless acceleration versus governed acceleration. The Basilisk is not some future model waiting to punish us. The Basilisk is the race itself: the incentive structure that makes stopping extraordinarily difficult and makes architecture, governance, and control the only realistic frontier.