Terminal-Bench-Science: Evaluating AI agents on scientific research workflows

 (terminal-bench-science.ai)

18 points | by matt_d 1 hour ago

3 comments

  • akshay_akula 24 minutes ago
    Evals on actual research workflows is the right direction, most agent benches are toy tasks.
    [-]
    • delichon 0 minutes ago
      Like

        * Optimize the design of the tokamak to maximize joules output without setting the atmosphere on fire.
  * Mutate this virus for maximal gain of function.
  * Revise this algorithm for optimal engagement by users age 5 to 12.
      I kind of prefer toy benchmarks.
  • vatsachak 20 minutes ago
    Damn. These things aren't AGI... but I don't care.

    Luna is good enough for me to give a parser spec and have it write one.

  • rubslopes 1 hour ago
    I'm glad to see that GPT Sol beats Opus at least in Mathematical Sciences, because that's my need right now, and I much prefer GPT's prose style.