This will be a first in computer science: all papers and comments at a scientific conference will be generated by machines. The event, titled "Agents4Science 2025," will be held online on October 22nd, with human participants. The event will include paper presentations and panel discussions by academics.

James Zou, an AI researcher at Stanford University and co-organizer of the conference, said the conference provided "a relatively safe sandbox environment where we can try out different types of submission and review processes." He noted that the conference aimed to capture the "paradigm shift" that has emerged in the application of AI in science over the past year. Researchers are now building collaborative teams of models, or so-called "agents," in addition to using large language models (LLMs) or other tasks-specific tools, acting as "scientists across research domains."

While AI models have been used to generate and review research, most publishers and conference organizers currently prohibit listing machines as authors or speakers. “We want to completely overturn this and require that authors and reviewers be AI,” Zou said. According to conference guidelines, while humans can provide suggestions and feedback, AI should be listed as a primary contributor, similar to a first author.

The conference organizers have received over 300 submissions from AI agents, 48 ​​of which have been accepted after evaluation by an AI review panel. Zou said these papers are primarily computational research, not involving physical experiments, and cover multiple fields ranging from psychoanalysis to mathematics. He hopes the conference will provide data on the competence levels of AI scientists and the types of errors they make. Mitchell stated that such data could inform policies regarding the use of AI in research.

Margaret Mitchell, a computer scientist at the US AI company Hugging Face researching AI ethics, stated that how to evaluate AI agents is an open research area. A key issue is how to account for situations where models frequently generate useless “false positive” findings—which could reduce the overall utility of the model.

If AI agents are left to operate autonomously, they are still prone to errors. Therefore, papers submitted to Agents4Science 2025 need to describe the interactions between researchers and AI agents at each step of the research process. Zou says this will make it possible to assess how the level of human involvement affects the quality of the work.

Furthermore, the use of AI models as peer reviewers for journals or conferences is a highly controversial topic. A recent survey by the British Institute of Physics found that 57% of respondents were unwilling to see generative AI used to draft peer review reports for papers they co-authored. AI reviewers have a number of weaknesses and vulnerabilities; for example, some AI may follow hidden instructions to give a paper a positive review. Some researchers believe that using AI to review papers may mean that researchers early in their careers miss the opportunity to learn crucial skills.

However, Zou argues that, at least in computer science, some form of AI peer review is needed to cope with the surge in conference paper volume. He suggests that AI agents could form hybrid review teams with humans.

Existing research indicates that LLMs are still inferior to humans in assessing novelty and importance. Matthew Gombolay, a computer scientist at Georgia Institute of Technology, suggests a more rigorous approach than this conference: randomly assigning papers from an existing major conference to either human or AI reviewers and then monitoring which method yields more significant breakthroughs.