文章评述了一篇关于AI未来发展的故事，作者对故事中一些关键时间点和技术进步的速度提出了质疑，认为部分预测过于乐观。作者结合自身经验，探讨了组织效率、技术发展速度、以及AI在社会中的角色等问题。文章重点关注了OpenBrain公司的崛起、AI的超人能力、以及政府监管等多个方面，对故事的真实性和可行性进行了深入的分析和评估，并提出了一些值得思考的观点。

🤔作者认为故事中OpenBrain公司在2026年初取得的算法进步速度（50%）可能过快，这在商业环境中是巨大的进步。作者认为，组织的效率（人际关系和流程）往往是瓶颈，而非软件工具或员工的专注力。

💡作者质疑OpenBrain的市场主导地位是不可避免的。在商业领域，小型竞争者和初创公司也有机会挑战市场领导者。作者认为，故事后期，软件在研究方面表现出色，不再依赖于人类员工的表现，这种转变的时间线可能过于激进。

🚀作者认为故事中，到2027年6月，OpenBrain的大部分人类员工无法再做出有效贡献的时间点过于超前。作者认为，这种转变需要现有软件的生产力提升至少300%，而非150%。并且，训练软件模仿人类的数据比模仿整个组织的数据要丰富得多。

🧐作者对故事中Agent-4的超人能力持怀疑态度，特别是其通用语言和自我营销能力。作者认为，软件在许多任务上可以与人类匹敌，并且速度更快，但开发比人类版本更好的语言和自我展示能力是一个更具推测性的想法，可能需要数百年时间。

⚠️作者认为故事中关于“稍微聪明”的可能性更大，“非常聪明”则具有高度推测性。作者认为，“稍微聪明，但速度快很多”似乎更有可能，并且与某些搜索和设计任务中的“非常聪明”重叠。

Published on April 5, 2025 12:32 AM GMT

My thoughts on the recently posted story.

Caveats

I think it's great that the AI Futures Project wrote up a detailed scenario.I enjoy it.Every part of the story i didn't comment on here is either fine or excellent.This is one of the most realistic scenarios i've read.All detailed predictions contain errors.The authors of this scenario don't claim it's the most likely future.If the speed of 2018-2025 was the typical rate of progress in software, then AI 2027 would be realistic.

Core Disagreements

Early 2026: OpenBrain is making algorithmic progress 50% faster.
As with many parts of this scenario, i think this is plausible but too fast. 150% productivity is a lot in business terms, & the scenario doesn't provide much detail for why this is 150% as opposed to 110%. In my software development experience, organizations are bottlenecked by their organizational efficiency (relationships & process) rather than by their software tools or by general employee attentiveness. Humans can already write bugfixes & features quickly. The trick is knowing which bugfixes & features to write. So OpenBrain being so productive is possible but it relies upon a additional lucky cause - OpenBrain's employees must have done a best-in-industry job of rapidly adapting to a process & release schedule that harmoniously leverages the software agents. Note this is even harder while rapidly hiring.
February 2027: OpenBrain wants to maintain a good relationship with the executive branch, because it is basically the only actor that can stop them now.
We haven't really established why OpenBrain's market dominance is inevitable. In business it's common for smaller competitors & stealth startups to have a decent chance of growing to rival the market leader. Contrast this with the later half of the story, when the software is so good at research that it doesn't rely on the performance of the human staff.
June 2027: Most of the humans at OpenBrain can’t usefully contribute anymore.
Months ago, humans were running software that gives very useful output. Now we have software that not only does a great job at all workplace tasks of a employee, it actually rivals a small research org by itself. This is a very ambitious timeline for such a new technology & would require the previous software giving at least 300% productivity, not 150%. And note that data for training software to imitate humans is much more abundant than data for imitating whole organizations. Like many parts of this scenario, i think this scene is fairly plausible but i would place it decades later.
September 2027: Agent-4’s neuralese “language” becomes as alien and incomprehensible to Agent-3 as Agent-3’s is to humans. Besides, Agent-4 is now much more capable than Agent-3 and has a good sense of exactly how to look good to it.
This posits 2 superhuman abilities. A general-purpose language & a self-marketing ability, both of which are better than human versions. In real life it's well-established that software can match humans at many tasks, & also can often perform tasks much faster. However, devising languages & self-presentation much better than the human versions is a more hypothetical idea that might not happen in 1,000 years. Most of real life's recent progress with LLMs is based on training software to imitate human experts. There are no data on superhumans.Continuous progress beyond humans into superhuman abilities reminds me of Kurzweil's superlinear graph through agriculture, industry, computing, & his singularity. Not impossible, but it's a curve without much of a mechanism to explain it.
Shutdown version of April 2028: It’s vastly smarter than the top humans in every domain.
As above, i consider 'slightly smarter' likely & 'vastly smarter' highly speculative. But 'slightly smarter & far far faster' seems quite likely, & overlaps with 'vastly smarter' at certain search & design tasks.Also, i'd personally replace 'every domain' with '95% of domains', but that doesn't affect things much.
Shutdown version of July 2028: Safer-4 answers that its own alignment training succeeded.
This sounds a little too simple to be true. Have any 2 organisms ever been perfectly aligned? Can Safer-4 be sure that it is aligned? And indeed, we can't really read this line literally. Safer-4 is currently puppeting America's diplomatic mission. It sounds like a parent playing a game with a young child: 'Today you're in charge of the grown-ups. Anything you say, i'll do.' But realistically, the parent will tweak any commands it thinks are foolish or dangerous.

Minor Details

April 2026
I'm dubious 'AIs' will ever be effective. Intelligent software in general is obviously effective; architecting it like a human employee is dodgier. A human coworker is not the optimal computational system nor the optimal UX. Rather, i want to send a complex task to the OpenBrain server as a batch process & have it message me hours later when it has output. This process may well involve similar internals to LLMs - but i'm unlikely to personify it, nor to keep it running after it gives satisfactory output, nor to call it a short-lived AI. While some users do like to have verbal conversations with chatbots that have names & use the 'i' pronoun, the idea of attaching this interface to its own database of memories & own always-running execution environment is purely a imitation of humans, rather than a legitimately useful server architecture. It would be wasteful to store everything the software learns in just 1 AI-person, rather than making that accessible to all of OpenBrain's future batch processes. 'Teams of agents' are technically feasible but not good business. Ultimately this doesn't affect the scenario much, but keep in mind that just because software runs for a few hours & does a great job doesn't mean that it resembles a human!This 'AIs' paradigm also fails to capture the complexities of the end of the story, when a highly unified always-on system is referred to as a collective named Agent-4.
January 2027: Agent-2 is able to escape, survive, & replicate autonomously
Agent-2 sounds so expensive to run that there are only a few servers it could copy itself to. Surely those admins would notice? But this doesn't affect the scenario much.
July 2027: Fortunately, it’s extremely robust to jailbreaks, so while the AI is running on OpenBrain’s servers, terrorists won’t be able to get much use out of it.
I don't really understand how a local copy of the weights gives the terrorists more practical control over the software's alignment. I don't think it's easy to manually tweak weights for so specific a purpose. Maybe they just mean the API is doing a good job of blocking sketchy requests?
Late July 2027: Gamers get amazing dialogue with lifelike characters in polished video games that took only a month to make. 10% of Americans, mostly young people, consider an AI “a close friend.”
We must imagine this happens in August or later, since it hasn't yet been a month since -mini was released. And 10% is a lot for such a rapid cultural shift.
October 2027: The White House sets up a Oversight Committee.
I've never seen the White House spot a problem, devise a plan, gather 10 people, & complete a committee vote in 1 month. Perhaps we can imagine this is more like the President committing to privately consult with 9 stakeholders & act based on the majority recommendation.
Shutdown version of November 2027: The President uses the Defense Production Act (DPA) to effectively shut down the AGI projects of the top 5 trailing US AI companies and sell most of their compute to OpenBrain
Nationalization is not common in modern US history, & many of these people might reasonably expect a diversity of labs to create a larger chance of breakthroughs in the arms race. But of course, some of these stakeholders want nationalization as a means to decrease productivity, because they fear the software. And we might imagine the software is also manipulating some of them to give it more compute. So i find this scene bizarre but explicable.
Shutdown version of November 2027: Thus far nobody has been willing to pull the trigger.
The big trigger, sure. (Seizing individual control of OpenBrain.) But we must imagine the Committee members still used OpenBrain servers for all sorts of personal gain. Spinning off their own startups in asteroid mining, chip manufacturing, etc. This story assumes that OpenBrain made good management decisions & its growth surpassed all of these superintelligence-founded rivals.
Shutdown version of December 2027: The US tightens export controls, and forces all chips to be fit with a variety of tamper-resistant hardware-enabled governance mechanisms including location tracking.
This is reasonable, but surely overhauling all this hardware will take time, & slow growth?
Shutdown version of February 2028:
A mirror life organism which would probably destroy the biosphere.
Nitpick: No single organism can destroy the biosphere; at most it can fill its niche & severely disrupt all ecosystems.
If given nation-state resources, it could easily surpass the best human organizations (e.g. the CIA) at mass influence campaigns. Such campaigns would be substantially cheaper, faster, more effective, and less traceable.
This is another hypothetical superhuman ability. And if OpenBrain influences the public extremely rapidly, surely that speed will be evidence for tracing this back to OpenBrain?
The President asks Safer-3 advice on geopolitical questions.
I think this assumes a president who is surprisingly credulous about the all-purpose usefulness of advanced software. (I would be credulous of it but i think Washington types would not.) I speculate that presidents are generally too busy to change their productivity habits mid-term. Perhaps we can imagine the president is consulting a advisor, & the advisor is leaning on Safer-3.
Shutdown version of June 2028:
I think this scene is great & i just want to say it's very funny to imagine the 2 AIs brainstorming what fake test results to 'reveal' to the humans.
Arms Race version of 2030: The AI releases a dozen quiet-spreading biological weapons in major cities.
While i find biowar worryingly plausible, note that this was not the only option. Consensus-1 could probably have destroyed all the nuclear missile submarines, talked people into evacuating all the most valuable land, & then ignored the now-harmless humans.

Overall

In my opinion, this story is realistic except for a few big 'fast-forward' moments. It's a story of businesses succeeding incredibly fast.

Most of AI 2027's scenes are sober & realistic extrapolations of the previous scene. Some are so realistic they make me laugh! But keep in mind that sci-fi technology is inserted into this story at 3 or 4 places. This does of course happen in real history! But you must be carefully skeptical of each of these insertions.

Discuss