You can do (fictional) theoretical physics today
An unusual puzzle game shows tentative promise as an intelligence test.
Note: Nothing in this article will spoil the game or prevent you from getting a fair score.
Starburst is a simple, text-based puzzle game where one tries to figure out the laws of physics of an imaginary universe. It requires almost no math or physics background beyond arithmetic. In this fictional universe, you get to do theoretical physics, to use intellect to wrest grand understanding from sparse knowledge--all without actually needing any physics or much math background, and in a way that's meaningful for everyone from a supergenius to someone on the verge of needing institutionalization. If you're interested in trying Starburst, email me at chapinalc@gmail.com or text me at 484-800-2275.
In Starburst, players get observations of "celestial" objects in the fictional universe. Initially, observations are sparse and primitive. Gradually, players gain access to increasingly powerful observational technologies, eventually including a full map and catalog of the universe. The goal is to provide a complete description of how the universe works with as few turns as possible. The game is designed so players who need to can skip ahead to whatever technological level they're able to start making headway at.
Very early signs suggest that Starburst is a promising intelligence test candidate. Solving it involves using a broad range of problem-solving faculties in concert. Unlike typical puzzle games, which are purely deductive, solving Starburst also involves making judgment calls in the face of open-ended uncertainty and using informal reasoning. No advanced background is required to understand the task or solution. Given the broad range in observational technology, the game gradually progresses from impossible to trivial-for-most, presumably giving it discernment from ~1.5 to >5.
The very beginning of the sparse Starburst early-game.
As mentioned, Starburst does not require a math/physics background. It's not yet clear, however, whether a background in math or physics provides an advantage (or rather, provides an advantage not commensurate with a genuine intelligence increase from such a background.) Very limited observations strongly suggest that background doesn't provide a massive advantage, but we haven't yet eliminated the possibility of it offering a moderate one. This question will be properly answered as we observe players with diverse levels of math/physics background and intelligence, especially intelligent ones without such backgrounds.
Given the need for deep, understanding-based reasoning and genuine novel problem-solving to solve Starburst, I've given various LLMs (including o1 and Gemini 1206) Starburst. All have failed abysmally. At a rough guess, that puts the intelligence of these models around 2.3-2.6. Assuming Starburst does well enough measure a genuine ability to reason (which I suspect to be roughly true, but am less confident about in the lower end), this means two things:
1. Current LLMs' ability to think lags far behind their ability to regurgitate solutions to already-solved problems. They're much better at faking intelligence than having it.
2. Despite that, these LLMs are likely genuinely smarter than a significant minority of people.