AI Red Team Engineering notes, research references, and field notes on AI systems, responsible AI, governance, and compliance.
Global search

Search AI Red Team

Search across the complete knowledge base.

Enter at least two characters to begin.
↑↓ select Enter open Browse the full research archive
Full Archive · Page 21

Research archive, page 21

Browse entries 481–504 of 1576. Return to the first page to search and filter the complete collection.

Search the full archive
METR February 17, 2026 analysis

Transcript-based time savings are an upper bound, not a productivity experiment

A METR exploratory analysis uses 5,305 coding-agent transcripts from seven staff to estimate time saved on AI-assisted tasks. A model estimates counterfactual human effort, while message activity approximates actual human time. The judge was checked against only 34 human estimates, and transcript selection excludes work done without AI. Task substitution, specialization and uncertain success judgments further limit interpretation. The resulting ratios are proposed soft upper bounds, not causal measurements of overall productivity.

Model EvaluationAI Engineering
Read summary Source link
METR February 10, 2026 analysis

A computational AI-timeline model makes automation assumptions inspectable

METR’s February 2026 modeling note builds a compact forecast of AI research automation from research labor, compute and software efficiency. Its equations connect capability progress to the share of tasks automated while representing remaining human work as a bottleneck. The accompanying model is useful for examining sensitivity to automation speed and research returns. The headline date follows selected parameter assumptions, many based on judgment, rather than an observed capability threshold. The model omits important effects, including parts of research judgment and the wider economy, and does not establish when real research organizations will become autonomous.

Model EvaluationAI Engineering
Read summary Source link
METR July 14, 2025 analysis

Human-time horizons need validation within each evaluation domain

METR’s July 2025 analysis asks whether task-completion horizons transfer beyond its original software evaluations. It reanalyzes nine existing benchmarks, fitting success against human completion time where suitable measurements are available. The relationship is useful in several settings but differs across domains and weakens when proxies such as video length or task fees stand in for human effort. Some fits require constrained assumptions, and agents, datasets and scaffolds are not matched across benchmarks. The study supports checking the metric’s validity locally rather than treating a single horizon as a universal measure of autonomy.

Model Evaluation
Read summary Source link
METR June 27, 2025 analysis

METR proposes evidence questions for frontier-model risk disclosure

METR’s June 2025 proposal asks what AI developers should disclose about serious model risks, including internal capabilities, monitoring, safety investigations and incident history. It organizes questions around information that outsiders cannot reliably infer from public model behavior, and discusses disclosure incentives and the limits of developer self-reporting. Sensitive evidence could be assessed by trusted reviewers rather than exposed publicly. This is a proposed transparency framework, not a tested assurance standard or legal compliance checklist. Its practical value is turning broad safety statements into specific requests for evidence and accountable review.

AI ComplianceModel Evaluation
Read summary Source link
METR August 6, 2024 guide

Agent benchmarks: inspect task weighting and budget-dependent failure

METR’s August 2024 evaluation update describes expanding its task set, using human completion time to organize difficulty and weighting task families to limit domination by families with many variants. It examines how success changes with additional agent time and where progress levels off under a simple harness. Sparse human baselines and the mixture of easy and hard tasks limit interpretation. The reusable lesson is that an aggregate score depends on task composition, weighting, stopping behavior and scaffolding, not just the model.

Model EvaluationAI Engineering
Read summary Source link
Prepare retrieval data with explicit inclusion rules and deletion checks video thumbnail Play video
AI Engineer October 5, 2026 video

Prepare retrieval data with explicit inclusion rules and deletion checks

Leo Platzer and Jeff Koss demonstrate a document-preparation workflow using a fictional legal-operations collection. Classification produces metadata that drives inclusion rules, followed by source inspection and human corrections before retrieval. Sensitivity, duplicate and superseded documents are selection concerns rather than problems that embeddings alone solve. Scheduled synchronization must also account for deleted source files. The demonstration provides a reusable curation process, but does not establish calibrated classification confidence or a measured retrieval improvement that transfers to another organization.

AI EngineeringModel Evaluation
Open notes Watch on YouTube
Open-model serving: evaluate cache locality, decoding and quantization together video thumbnail Play video
AI Engineer October 3, 2026 video

Open-model serving: evaluate cache locality, decoding and quantization together

Sujee Maniyam and Dylan Bristot explain why identical open-model weights can behave differently across serving stacks. Their overview covers routing requests toward reusable KV state, offloading that state from GPU memory, separating prefill from decode, and using a small draft model whose tokens a larger model verifies. Quantization and application-specific draft training introduce additional quality and workload dependencies. The presentation supplies mechanisms but not sufficient benchmark conditions to generalize its reported speedups.

AI EngineeringModel Evaluation
Open notes Watch on YouTube