Full Archive · Page 23

Research archive, page 23

Browse entries 529–552 of 1576. Return to the first page to search and filter the complete collection.

METR March 15, 2024 analysis

Measure post-training and scaffold changes as separate evaluation variables

METR’s March 2024 study compares model versions and agent scaffolds on 195 tasks across twelve families. Later post-training versions of the same base models improved measured autonomy, while the additional scaffold gains studied were small and not statistically significant. The task selection, limited enhancement methods and lack of model-weight access constrain generalization. The study demonstrates why an evaluation should specify the exact model version, tools, prompts and selection procedure; it does not establish a ceiling on gains from future elicitation.