You give it a sentence — “measure the cosmological parameters from this supernova dataset” — and walk away. About 30 language-model agents pick it up, search the literature, write the analysis code, run it, read the output, argue with each other about whether the answer is sound, and hand back a result. No human touches the keyboard in between. The team that built this calls the demo “a PhD level cosmology task,” and the system did it end to end.
The system is cmbagent, and the paper describing it (Xu et al., “Open Source Planning & Control System with Language Agents for Autonomous Scientific Discovery”, posted to arXiv on 9 July 2025) has about two dozen authors across Cambridge, Princeton, the Max Planck Institute and EPFL. It’s open source under Apache-2.0, and it took a first-place award at NeurIPS 2025’s FAIR Universe competition, a challenge on inferring cosmological parameters under realistic systematic uncertainties.
I spent years building software before I started spending clear nights on my balcony in Nicosia with a Seestar. Agent systems are the part of ML I actually worked in, so let me walk through what cmbagent really does (the loop that keeps 30 agents from wandering off), and then be honest about where I think it holds up and where it doesn’t.
Not one chatbot — a crew with a chain of command
The first thing to get straight: this is not “ask GPT to do cosmology and paste the answer.” A single large model handed a research problem will confidently produce code that runs, imports that don’t exist, and a plausible-looking number with no idea whether it’s right. The interesting engineering in cmbagent is the structure that surrounds the models.
The paper describes roughly 30 agents, each with a narrow job: some do retrieval over scientific papers and codebases, some write code, some run it, some interpret the results, and some exist only to critique the output of the others. The backbone is AG2 (the framework formerly called AutoGen), and the models underneath are swappable: by default it pulls from OpenAI, Anthropic and Google, so you can put a stronger model on the hard reasoning steps and a cheap one on the mechanical ones. It’s been maintained in the open by Boris Bolliet since May 2024, which is worth knowing: this is a working research group’s tool, not a demo repo.
What makes it more than a pile of prompts is how the crew is coordinated.
Planning and control: the loop that keeps it honest
The architecture the authors lean on is a Planning & Control strategy, and it’s borrowed from robotics rather than chatbots. It splits the work into two phases that would otherwise blur together and derail each other.
In the planning phase, a planner agent reads the request and drafts an explicit, numbered plan: first retrieve the relevant method, then load the data, then fit this model, then check these diagnostics, then report. A separate plan reviewer critiques that plan before a single line of analysis code is written, poking at missing steps, wrong assumptions, an ordering that won’t work. The plan gets revised until it survives review.
Only then does the control phase begin, and this is the part I find genuinely smart. Instead of letting one agent free-associate its way from question to answer, a controller walks the approved plan one step at a time, dispatching the right specialist for each step and feeding its output to the next. A retrieval agent pulls the actual documentation for a Boltzmann code like CAMB or CLASS instead of the model half-remembering the API from training. An engineering agent writes the Python and runs it in a real execution environment, so a traceback is a fact the system has to deal with, not a possibility it imagines. An interpretation agent turns the raw output into a claim. Reviewer agents sit in the loop and can send a step back.
The reason for all that structure is a failure mode anyone who has built with LLM agents knows too well: drift. Give a single agent a long task and a scratchpad, and it slowly forgets the goal, invents a subtask, and optimizes for finishing rather than being right. Separating “decide the plan” from “execute the plan,” with review gates on both, is how you fight that. It’s the same instinct as writing the experiment protocol before you touch the data.
The worked example: cosmology from exploding stars
The demonstration task is a real one, and it’s a good choice because it’s a genuine measurement rather than a toy. Type Ia supernovae are the workhorse “standard candles” of cosmology: because they detonate at a fairly consistent peak luminosity, you can turn how bright one looks into how far away it is. Plot that distance against redshift for a few hundred supernovae and you’re tracing the expansion history of the universe, the same measurement that revealed cosmic acceleration in the late 1990s. Fit a cosmological model to that curve and out come parameters like the matter density and the behaviour of dark energy.
That’s normally a graduate student’s project: understand the method, get the data into shape, choose and code a likelihood, run the fit, sanity-check the posterior, write it up. cmbagent’s claim is that its agent crew did that chain autonomously (retrieving the approach, generating the fitting code, executing it, and interpreting the parameter estimates) from a natural-language prompt, with no human stepping in to fix an import or nudge the fit.
I want to be careful about the register here. “PhD level” is the authors’ framing of the task’s difficulty, not a claim that the system replaces a PhD. And a supernova cosmology fit, while real, is a well-trodden problem: the method is in a hundred papers and the codes are mature. That’s exactly the kind of task where retrieval-plus-execution should shine, because the right answer is discoverable. It says much less about problems where the method doesn’t exist yet.
My honest read, as someone who’s shipped agent systems
Here’s where I’ll be blunt, because I’ve watched agent demos look magical and then fall over in production.
What cmbagent gets right is the stuff that actually kills autonomous pipelines. Executing code in a real environment means the system can’t hide behind plausible-looking nonsense. Either the fit converges or it throws. Retrieval over real documentation cuts down on the single worst LLM habit, which is inventing a function signature that looks correct. And the reviewer-and-control structure is a real answer to drift, not a slogan. These are the right problems to have solved.
What I don’t want anyone to mistake it for is understanding. These agents manipulate the forms of cosmology (the code, the papers, the numbers) without any model of the physics underneath. That’s the exact failure I wrote about in transfer learning for cosmology, where a system fluent on its training distribution quietly mislabels genuinely new physics as something old and familiar. A crew of agents can execute a known method flawlessly and still have no way to notice that the method is the wrong one for the data in front of it. The same caution I raised about foundation models in astronomy applies double when the model is also allowed to write and run its own code.
There’s a cost dimension too. Thirty agents, several of them calling frontier models across multiple planning-and-review rounds, is not free. A single autonomous run can burn a lot of tokens, and the paper’s own design of putting cheaper models on cheaper steps is a tell that this adds up. “Autonomous” doesn’t mean “cheap,” and it definitely doesn’t mean “unsupervised in the sense that you can trust the number blind.”
What it means for the rest of us
None of this touches the eyepiece tonight. Your Seestar frames aren’t going into an agent’s context window, and no LLM is going to improve your seeing over Troodos.
But the direction is worth watching if you care about how astronomy gets done, and eventually about who gets to do it. The honest promise of a tool like cmbagent isn’t replacing researchers. It’s compressing the mechanical middle of a project. The retrieval, the boilerplate fitting code, the diagnostic plots, the first draft of an interpretation: that’s a large fraction of the hours in a real analysis, and it’s the part that a well-controlled agent crew can plausibly take on. A frozen requirement remains: a human owns the physics, chooses the question, and decides whether to believe the answer.
The part I find quietly promising is accessibility. An open-source, Apache-licensed system that can run a standard cosmology analysis from a plain-English prompt lowers the bar for someone without a research group or a supervisor to check their work: a motivated amateur, a student at a small department, a physicist crossing over from another field. That cuts both ways, because it also lowers the bar for producing confident, wrong results at scale. The tool doesn’t fix the hardest skill in science, which is knowing when the answer is fishy.
So: not a robot cosmologist. A very capable, tireless research assistant that can already do a real measurement start to finish, and that will happily do a wrong one just as fluently. The results to watch aren’t the demos where it reproduces a known number. They’re the first ones where an agent crew is turned loose on a question nobody has answered yet — and a human still has to decide whether to trust what comes back.