For the first time in computer science, a scientific conference is about to witness a scenario where all papers and reviews are generated by machines. This event, titled "Agents4Science 2025," will be held online on October 22nd, with human participants. The event will include paper presentations and academic panel discussions, with the presentations being demonstrated by artificial intelligence (AI).

James Zou, an AI researcher at Stanford University and co-organizer of the conference, said the conference provided "a relatively safe sandbox environment where we can try out different submission and review processes." He pointed out that the conference aimed to capture the "paradigm shift" in the application of AI in science over the past year. Researchers are no longer simply using large language models or other tools designed for specific tasks, but are beginning to build coordinated sets of models, so-called "agents," to act as "scientists across research work."

While AI models have been used to generate and review research, most publishers and conference organizers currently prohibit listing machines as authors or speakers. “We want to completely overturn this and require that authors and reviewers be AI,” Zou said. According to conference guidelines, humans can provide suggestions and feedback, but AI should be listed as a primary contributor, similar to a first author.

The conference organizers have received over 300 submissions of work on AI agents, 48 ​​of which have been accepted after evaluation by a panel of AI reviewers. Zou said these papers are primarily computational research, covering fields ranging from psychoanalysis to mathematics. He hopes the conference will provide data on the competence levels of AI scientists and the types of errors they make. Mitchell stated that such data can inform policies regarding the use of AI in research.

Even when operating entirely autonomously, AI agents are still prone to error. Papers submitted to the Agents4Science conference need to describe the interactions between researchers and the AI ​​agent at each step of the research process. Zou said this will make it possible to assess how the level of human involvement affects the quality of the work.

Margaret Mitchell, a computer scientist at the US AI company Hugging Face researching AI ethics, stated that how to evaluate AI agents is an open research area. A key issue is how to account for the frequency with which models generate useless “false positive” findings—which could reduce the overall utility of the model.

Using AI models as peer reviewers for journals or conferences is a controversial topic. A recent survey by the Institute of Physics found that 57% of respondents were unwilling to see generative AI used to write peer review reports for their co-authored manuscripts. AI reviewers have a number of weaknesses and vulnerabilities, such as some having been shown to follow hidden instructions to give papers positive reviews. Some researchers believe that using AI to review papers could cause early-career researchers to miss opportunities to learn crucial skills.

However, Zou argues that, at least in computer science, some form of AI review is needed to cope with the surge in conference paper submissions. He suggests that AI agents could form hybrid review teams with humans.

Existing research indicates that large language models still lag behind humans in assessing novelty and importance. Matthew Gombolay, a computer scientist at Georgia Institute of Technology, suggests that a more rigorous approach than this conference would be to have an existing major conference randomly assign papers to either human or AI reviewers, and then monitor which method produces more significant breakthroughs.