I'm Deeply Worried About LLM Progress

Starburst has shown concerning progress. We'll have a meeting to discuss.

"I therefore suspect, with low confidence, that o3 and its contemporaries will fail to break this barrier. If a model passes this barrier, that would be a massive cause for concern."

I wrote those words just over two months ago, referring to the failure of o3-mini-high, at the time the most capable publicly available model in that regard, to solve a simple puzzle game I made at a particular stage. The model performed significantly worse than I expect the average American would, and massively worse than a very smart friend of mine who first playtested it. Since the release of ChatGPT and subsequent popularity of LLMs, I've consistently viewed these models as knowledgeable idiots. Despite being impressively capable (and therefore potentially dangerous) in certain regards, peeking beneath the hood has revealed massive deficits in basic reasoning.

My prediction was wrong, and this is changing. Gemini 2.5-pro, recently released, was the first LLM to cross the above-mentioned barrier, and the imminent o3 and o4-mini will likely cross it too. The newest LLMs have something resembling a genuine ability to reason at a basic level. To be clear, I'm absolutely not claiming that these models are anywhere close to fully eclipsing all human ability. On Starburst, the aforementioned puzzle game, and many other tasks, smart people can still vastly outperform any publicly available AI, and nearly certainly any AI currently in existence. But the reasoning tasks where a dumb human can outperform state-of-the-art models are, optimistically, rare, and those where a typical human can do so are likely to rapidly diminish in the near future. The water is still shallow, and many of us are tall, but it's fast rising, and the storm shows no signs that it'll abate.

I don't know exactly where things are going. I've seen plausible predictions of singularity in two or three years. Even if those don't materialize, and I'm far from confident that they will, many more mundane threats are now anywhere from very possible to nearly assured: more addictive social media, better political propaganda, increasing consolidation of corporate and government power, mass unemployment, general unrest, and much more.

We may not have very long until things get bad. This is looking like it might be part of a Fermi Paradox solution. I deeply hope that I'm overreacting, but every sign I see suggests I'm not.

For those of you in or near the Philadelphia area, we'll hold an in-person meeting on Sunday, April 20th at 2pm to discuss our benchmark, why the particular threshold crossed is significant, and the risks rapidly descending upon us. If you have the ability, I highly encourage you to attend, and to encourage others to join you. Text me at 484-800-2275 for the address. Even if you can't make it, if you want to hear more or have anything useful to offer on the topic, please reach out.

Further updates as we get them. Godspeed.