Even as of August 2026, a common way to evaluate AI (in the public.. and even "scientific" discourse) is to give a model a prompt, inspect the answer, and declare that “AI can” or “AI cannot” do the task. Even reputed scientists do this.
Attached is a slide from recent talk to add some distinction.
It would be strange to remove all of these supports from a scientist, ask one question, and treat the first response as a measure of scientific capability. It is also strange to do this with AI, especially with all the infrastructure that surrounds models (e.g. Codex). A couple of years ago, we had to wrangle with langchain, langgraph and clunky things... not anymore. Come on!
Make no mistake, one-shot prompting can still tell us something about the raw behavior of a model. To make meaningful claims, we need to specify what system was tested. Did it have access to the relevant papers and data? Did you allow it to write and execute code? Could it use a symbolic algebra system**? Did it retain intermediate results? Was it allowed to test competing hypotheses? What evidence was it required to produce? What happened when a check failed?
I also had another slide to distinguish scientific ML and AI, because calling everything AI doesn't help either. SciML (for the most part) predicts, approximates, provides decisions within human generated workflows, whereas AI performs cognitive work (emulation of human cognition.. e.g. things that I mentioned above that a scientist does). Of course, this is not a strict definition (and something that the broader ML community might or might not agree), but it is definitely something relevant to the science community.
** Actually this is to gently poke fun at those who dismiss recent successes of AI in math to 'neurosymbolic systems'. Frontier models have had access to symbolic algebra manipulation for two years. Being skeptical is a sign of a good scientist (especially in times of great hype), but being aware of moving goalposts is.. as well.. :)