TheoPhysEval: GPT-6-Astra Solves 49 of 50 Physics Problems

A new benchmark puts commercial AI agents on the same level as top specialists in theoretical physics, at least for one kind of work: the detailed calculations that fill much of a researcher's week. OpenAI's GPT-6-Astra got almost every problem right. The authors argue that this still does not add up to doing physics.

What TheoPhysEval measures

TheoPhysEval was built by 35 researchers from institutions including the International Centre for Theoretical Sciences (ICTS), Harvard University, KU Leuven and Google DeepMind. It contains 50 unpublished research problems with verifiable answers, spread across eleven subfields. These range from fluid flows and biological processes to particle physics and cosmology.

The problems are not grand open questions. The authors call them "turn-the-crank" tasks, where a researcher applies known methods to a slightly altered question. Contributors often took a calculation from one of their own recent papers and changed a few values, so the old result could not simply be reused. A leading expert in the relevant subfield would typically need a day or less to solve one without AI. Researchers from neighboring fields would need much longer. Tasks like these are often given to PhD students as a first step into research.

"Routine" does not mean easy. One black hole problem requires finding an older paper, carrying its equations over from five-dimensional to four-dimensional spacetime, and running a custom program to get the result. Coordinator Suvrat Raju, who co-wrote the original paper, needed about a day. Before modern agents existed, the authors estimate, only a handful of experts had both the physics background and the computational experience to attempt it. Even they would have needed several days.

The results

The agents got no human hints. Each one ran twice: once with web search, and once fully offline.

1. GPT-6-Astra (in OpenAI's Codex environment): 49 of 50 correct, or 98 percent. The average cost was under $3 per problem and just under $116 for the full set.
2. Opus 5.5 (Anthropic, in Claude Code): 92 percent.
3. Gemini 3.8 Flash (Google): 68 percent.
4. DeepSeek-V4.1-Flash and GLM 5.3 (open-weight): 54 and 42 percent.

Some failures were basic. On the black hole task, Gemini and GLM copied the five-dimensional formulas from the older paper unchanged, which an expert would have spotted at once.

Going offline mattered less than expected. Without the internet, Astra and Opus both scored 92 percent, even though many problems depend on formulas from papers the models were not given. The authors read this as a sign that leading models have effectively memorized much of the theoretical physics literature.

The one problem no model solved asked for an equation that produces a given curve. The expert answer relies on special mathematical forms that only count as the required differential equation in a broader sense. The authors concede it may look "unfair," but argue it is realistic, since researchers often have to infer the physics behind imprecise data.

When the answer key was wrong

On several problems, follow-ups with the experts uncovered small errors in their own reference solutions. Once these were corrected, Astra's answers were right every time. This may explain why agents do better here than on benchmarks such as CritPt, even though the authors consider their own problems harder. A recent study found some physics benchmarks are graded incorrectly and can badly underestimate the models. The dataset and test setup are available on Hugging Face.

Missing "taste"

The authors call the results a major milestone and say some current models may already beat the best human researchers at concrete calculations. Mathematics has gone further: according to OpenAI, agents have solved important open problems there, including the Navier-Stokes Millennium Prize Problem. This comes alongside efforts such as publishing hundreds of AI-generated proofs. Physics has seen no comparable breakthrough.

The authors' explanation is that calculation comes at the end of research. Before that, someone has to build a theory, ask it the right questions, and switch to a simpler but equally interesting problem when stuck. Agents are weak at this. They also seem to lack the shared, if contested, sense of what makes a problem worth solving. Asked to design new problems, they get lost in details people find boring. The authors stress that these points cannot be measured and are not a forecast.

The Bigger Picture

This suggests the bottleneck in AI-assisted science is moving from calculation to judgment. The pattern looks similar to other recent cases, like Claude Science's UV sky map, where agents handle a lot of execution while humans still frame the questions.

The authors raise two practical concerns. First, problems like these no longer work well as first projects for early-career researchers. Second, theoretical physics has long been cheap compared to experimental work, which kept the field relatively open. If harder problems need much more compute, budgets could start to decide who can compete. Mathematics is already debating this, for example around the Leiden Declaration.

It is worth watching whether physics starts a similar debate, and whether future benchmarks try to test problem selection rather than only calculation. The authors close by suggesting that the field's old motto, "shut up and calculate," may need to give way to "stop and think."