AI agents deliver results – but do they reason scientifically?
Corral makes visible how AI agents arrive at their results: the benchmark breaks down agent runs into hypotheses, tests, evidence, judgments and corrections. This makes it possible to identify both productive forms of scientific reasoning and problematic patterns, such as when evidence is not taken into account. © HIPOLE Jena / Corral
A research team co-led by Kevin Maik Jablonka from the Helmholtz Institute for Polymers in Energy Applications Jena (HIPOLE Jena) and N. M. Anoop Krishnan from the Indian Institute of Technology Delhi has developed Corral, a new benchmark for AI agents in science. The preprint “AI scientists produce results without reasoning scientifically” has been published on arXiv (https://doi.org/10.48550/arXiv.2604.18805). The analysis shows that current systems can execute scientific workflows and deliver results; however, they often do not follow the basic principles of scientific testing and reasoning.
Artificial intelligence is expected not only to write texts or analyse data, but also to plan scientific experiments, analyse results and generate new knowledge. But when can an AI system truly be said to be doing science? Is it enough for the final result to be correct – or must the path to that result also meet scientific standards? This question is addressed in a new preprint by Jablonka’s team.
With Corral, the researchers developed a benchmark that evaluates AI-based scientific agents not only by their results, but also by how they arrive at them. To do this, the team analysed more than 25,000 agent runs across eight scientific domains – ranging from molecular simulations and materials data analysis to spectroscopic structure elucidation and hypothesis-driven chemical tests. The evaluation examined not only whether a task was solved, but also whether the systems take evidence into account, generate and test hypotheses, and revise their assumptions when confronted with contradictory results.
“We need to be clearer about what kind of scientific reasoning we expect from such AI systems,” says Jablonka. “When it comes to epistemic rigor, better training procedures may help. But in areas where we need reliable guarantees about the reasoning process, we will probably need different systems – for example, systems with symbolic and formally verifiable components.”
HIPOLE Jena is an institute of the Helmholtz-Zentrum Berlin für Materialien und Energie (HZB), operated in cooperation with Friedrich Schiller University Jena and the Center for Energy and Environmental Chemistry Jena (CEEC). Regarding the preprint, Helmholtz AI has already published a detailed background article on the work: https://www.helmholtz.ai/detail/do-ai-scientists-actually-do-science-new-benchmark-probes-the-reasoning-behind-the-results-featuring-dr-kevin-maik-jablonka-helmholtz-ai-associate/
- HZB congratulates Peter Hegemann on winning the Nobel PrizeOur warmest congratulations go to Peter Hegemann, Professor of Neuroscience and Experimental Biophysics at Humboldt University in Berlin. Together with Prof. Georg Nagel (Julius Maximilian University of Würzburg) and Prof. Karl Deisseroth (Stanford University, USA), he has been awarded the 2026 Nobel Prize in Physiology or Medicine. We are delighted for him, his team, Humboldt University and the entire Berlin scientific community.
- Does African Architecture exist?The world’s first Pan-African Architecture Biennale took place this September in Nairobi, Kenya. People from the worlds of architecture, art, academia and politics from 54 African countries and the diaspora had travelled to Nairobi to exchange ideas on traditional and contemporary building methods, materials, experiences and projects. Samira Aden co-organised this biennial as lead curator. We spoke to her about it.
- Nanosilver as an electrocatalyst for CO₂ reductionVia electrolysis, CO2 can be reduced to CO, a raw material for further chemical products such as fuels. Within the GreenQuest Project, an internation team led by HZB chemist Prashanth Menezes has now systematically investigated catalyst layers made of silver nanoparticles, varying both the size of the particles and their density. The best yield was achieved with nanoparticles with diameters of around 10 nm, which were loosely distributed. Furthermore, they demonstrated how the economic efficiency of the electrochemical cell can be enhanced by integrating an additional chemical reaction at the anode, enabling the simultaneous production of a valuable formic acid, hydrogen, and CO in one device.