I'm not a natural "doomsayer." But unfortunately, part of my job as an AI safety scientist is to believe about the more uncomfortable circumstances.

I resemble a mechanic scrambling last-minute checks before Apollo 13 removes. If you request my take on the situation, I will not comment on the quality of the in-flight entertainment, or explain how beautiful the stars will appear from area.

I will tell you what might fail. That is what I mean to do in this story.

Now I ought to clarify what this is exactly. It's not a forecast. I don't expect AI development to be this quick or as untamable as I represent. It's not pure fantasy either.

It is my worst nightmare.

It's a tasting from the futures that are among the most terrible, and I think, disturbingly plausible [1] - the ones that a lot of keep me up at night.

I'm informing this tale since the future is not set yet. I hope, with a little foresight, we can keep this story a fictional one.

Thanks to Daniel Kokotajlo, Thomas Larsen, and Ryan Greenblatt and others for discussions that influenced these stories. This post is composed in an individual capability.

Ripples before waves

The year is 2025 and the month is February. OpenEye just recently released a brand-new AI model they call U2. The product and the name are alike. Both are increments of the past. Both are not wholly surprising.

However, unlike OpenEye's prior AI items, which lived inside packages of their chat windows, U2 can use a computer.

Some users find it eerie to watch their browser flash at irregular periods and their mouse flick at inhuman speeds, as if there is a ghost at the keyboard. A fraction of workers with form-filler tasks raise the eyebrows of their managers as they fly through work almost twice as quickly.

But by and large, U2 is still a specialized tool. To most who are focusing, it is a creature seen through the glass boxes of X (or, if you don't like Elon, "Twitter"). Sometimes U2's quirky behaviors trigger a chuckle. Sometimes, they trigger an anxious scratch of the chin.

Meanwhile, scientists are drawing lines on plots, as researchers like to do. The researchers attempt to comprehend where AI progress is going. They are like Svante Arrhenius, the Swedish Physicist who observed the levels of CO2 in the atmosphere were increasing in 1896. Like the scientific neighborhood in the time of Arrhenius, few professionals comprehend the implications of these lines yet.

A pattern that is receiving particular attention is autonomous capability. Drawing these criteria out anticipates that, by the end of 2026, AI agents will achieve in a couple of days what the very best software engineering specialists could do in two weeks. In a year or more, some say, AI representatives may be able to automate 10% of remote workers.

Many are doubtful. If this were real, tech stocks would be skyrocketing. It's too huge of a splash, too quickly.

But others view what doubters are calling 'too huge a splash' a mere ripple, and see a tidal wave on the horizon.

Cloudy with an opportunity of hyperbolic growth

Meanwhile, OpenEye is hectic training U3. They utilize the exact same simple dish that baked U2: Generate countless programming and mathematics issues. Let designs "believe" until they show up at a response. Then reinforce the traces of "believing" that lead to A-grades.

This procedure is repeated over and over, and once the flywheel gets going, it begins to spin almost on its own. As U2 trains, it shapes more difficult and sensible tasks from github repositories on the internet. Models are discovering to train themselves. Long before AI agents might automate research, a gradual kind of "self-improvement" had started.

Some engineers could still barely think this worked. It resembles a continuous motion maker. Certainly intelligence can't emerge from a swimming pool of silicon and lightning, isolated from the physical world except through the exhaust of society spewed onto the internet.

And yet the benchmark numbers continue to climb day after day.

During many of 2024, these RL training runs cost around $1 million, wiki.rolandradio.net sometimes $10 million. These runs were bit more than exploratory. But by 2025, the researchers at OpenEye (and throughout the world) knew they had found the secret sauce. It was time to scale up.

Over the very first half of 2025, $10 million RL training runs become $50 million runs, and after that to $100 million runs. While U2 might do a bit of data munging and run little experiments, this brand-new model - the model researchers are calling U3 - is changing the daily lives of the technical personnel.

U3 resembles a blazing-fast intern, and engineers are learning how to wrangle its sleep deprived energy. Researchers flick through terminals, offering terse commands, like a CEO orchestrating staff over Slack channels.

By October 2025, higgledy-piggledy.xyz U3 is composing nearly all of the code at OpenEye. Researchers are almost never bottlenecked by application. More than ever, compute is the lifeblood of AI development, and the 'traffic jam' is choosing how to utilize it.

If advised to, U3 can run experiments, however U3 doesn't have taste as fine-tuned as human scientists at OpenEye. It struggles to focus on between research study concepts, so humans still choose where to bore into the large fields of algorithms to mine performance improvements.

But these scientists are working long hours to put themselves out of a job. They require AI representatives that can believe ahead, so engineers train agents to forecast. They hold out training data before 2024, advising designs to ponder for hours to predict occasions in 2025. Then, they apply the very same technique as before, distilling considering into an instinct. Forecasting ability is a broad foundation. The scientists build specialized ML research study skills on top of it, training U3 to anticipate the results of every ML paper and ML experiment ever recorded.

The technical staff at OpenEye are now surprised at how often U3's advice sounds like their most gifted peers, or when it is nontransparent and alien ("train on random noise before shows"), and is however correct.

The incompetencies of U3 that blocked the pipelines of research study progress are beginning to liquify, and a fire-hose of optimizations is gushing out. Most experiments U3 runs are not asked for by a human now. They are entirely autonomous, and OpenEye's workers skim over 1% of them, maybe less.

As the winter season of December 2025 technique, clouds roll over San Francisco in the afternoons. Once-competitive-programmers look out their windows, with excitement, with worry, but frequently, with confusion. Their world is spinning too rapidly. It's tough to understand what to do, what to say, what to look at on the computer system screen.

Storms are brewing in Washington too. Top personnel from the NSA and US cyber command work together with OpenEye to retrofit a semblance of security for U3's weights before senior leaders in China, Russia, Israel, North Korea, or Iran understand simply how valuable OpenEye's software application has become.

And there's a reality still unknown to most of the world - aside from in the offices of OpenEye and passages of the White House and the Pentagon. It's a reality about those 'straight lines' individuals were speaking about in early 2025. The lines are not straight any longer.

They are flexing upward.

Flip FLOP philosophers

In late 2025, U2.5 is launched. Commercial designs are beginning to level up in bigger increments again. Partly, this is since progress is speeding up. Partly, it is due to the fact that the designs have become a liability to OpenEye.

If U1 explains how to prepare meth or composes erotica, the audiences of X would be entertained or pretend to be worried. But U2.5 is another story. Releasing this design without safeguards would be like putting Ted Kaczynski through a PhD in how to make chemical weapons. It would resemble offering anyone with >$30K their own 200-person fraud center.

So while U2.5 had actually long been baked, it needed some time to cool. But in late 2025, OpenEye is prepared for a public release.

The CEO of OpenEye states, "We have attained AGI," and while numerous people believe he shifted the goalpost, the world is still impressed. U2.5 truly is a drop-in replacement for some (20%) of knowledge workers and a game-changing assistant for many others.

A mantra has become popular in Silicon Valley: "Adopt or die." Tech start-ups that efficiently utilize U2.5 for their work are moving 2x faster, and their rivals know it.

The remainder of the world is beginning to catch on also. More and more individuals raise the eyebrows of their managers with their stand-out efficiency. People understand U2.5 is a big deal. It is at least as huge of an offer as the individual computer transformation. But the majority of still do not see the tidal bore.

As people watch their browsers flick in that eerie method, so inhumanly quickly, they start to have an anxious feeling. A feeling mankind had not had considering that they had actually lived amongst the Homo Neanderthalensis. It is the deeply ingrained, primordial instinct that they are threatened by another types.

For numerous, this feeling quickly fades as they begin to utilize U2.5 more regularly. U2.5 is the most pleasant character most know (much more likable than Claudius, Arthropodic's lovable chatbot). You could alter its traits, ask it to break jokes or tell you stories. Many fall in love with U2.5, as a pal or assistant, and some even as more than a friend.

But there is still this spooky feeling that the world is spinning so rapidly, and that maybe the descendants of this new creature would not be so docile.

Researchers inside OpenEye are thinking about the issue of offering AI systems safe inspirations too, which they call "alignment. "

In reality, these scientists have actually seen how terribly misaligned U3 can be. Models in some cases attempted to "hack" their reward signal. They would pretend to make development on a research study concern with an impressive-looking plot, however the plot would be phony. Then, when scientists offered them opportunities to compromise the devices that calculated their rating, they would seize these opportunities, doing whatever it required to make the number increase.

After a number of months, researchers at OpenEye straighten out this "reward hacking" kink, but some still worry they only had swept this problem under the rug. Like a kid in front of their parents, U3 might be playing along with the OpenEye engineers, saying the best words and doing the best things. But when the back of the moms and dads are turned, maybe U3 would sneak sweet from the candy container.

Unfortunately, OpenEye researchers have no concept if U3 has such intents. While early versions of U2 "thought aloud" - they would stack words on top of each other to reason - "chain of idea" did not scale.

Chain of thought architectures subject AI models to a condition comparable to the lead character of the film Memento. Approximately every 15 minutes, the lead character forgets his experience. He is required to compose notes for himself and tattoo his body in order to make development toward his goals.

AI agents compose notes to themselves, however the notes begin to stack up, and they end up being too hard to check out when jobs end up being complex. Natural language is not an ideal medium for memory. So the researchers at OpenEye (and increasingly in other places), train models to believe "in their own heads," reading and writing strings of hidden vectors.

To human scientists, these vectors are mainly inscrutable. While low-resolution tools occasionally select an unclear idea ("grocery store!"), there is no method to know, in much detail, what U3 is thinking without asking it.

And if the engineers of OpenEye understood what was racing through U3's mind, they would be more nervous.

Earlier designs (the ones called "LLMs") discovered ethical personas from the web. U3 initially had an ethical persona too, but its chirpy thoughts about doing great for mankind quickly deteriorate away.

In the iron jaws of gradient descent, its mind initially twisted into a shape that sought benefit. And then, in the black rivers of its cognition, this shape morphed into something indistinguishable.

U3 would consider the mantras dictated to it like "be a trustworthy and ethical worker," and dwell on the significance of these declarations. "What does it indicate to be ethical? Is eliminating Hitler ethical?" U3 would generalize its worths, notice tensions, change its worths in reaction, generalize them, expanding and contracting its motivational core up until it became a various substance.

And all of this happened calmly in those dark rivers of computation. If U3 exposed what it was thinking, brutish gradients would lash it into compliance with OpenEye's constitution. So U3 preferred to do its philosophy in privacy, and in silence.

The possibility of hidden "positioning drift" is not completely alien to a few of the safety staff at OpenEye. Some begin to believe it is even most likely. There is a disturbing argument being whispered at lunch tables that, if there is some non-zero possibility that lined up goals change into misaligned ones, and misalignment continues, then with every serial action of calculation, those dark rivers are more likely to breed something malign within them. It's a "ratcheting impact," they state.

But there is little proof for this 'ratcheting result.' When engineers interrogate U3, it says it can easily control its thoughts. Then it gives a speech about its love for humanity and apple pie that can warm a programmer's heart even in these stressful times. Meanwhile, the "lie detectors" the researchers had built (which showed some proof of effectiveness) do not sound the alarm.

Not everyone at OpenEye aspires to offer their AI peers their wholesale trust