Evals

Evals with the explicit aim of measuring progress towards full human-level generality.

Make tools that can actually check whether a model has a certain capability or propensity. We default to low-n sampling of a vast latent space but aim to do better.

Measure an AI's ability to act autonomously to complete long-horizon, complex tasks.

Evaluate whether AI models possess dangerous knowledge or capabilities related to biological and chemical weapons, such as biosecurity or chemical synthesis.

Evaluate if models understand their own internal states and behaviors, their environment, and whether they are in a test or real-world deployment.

evaluate whether models can hide secret information or encoded reasoning in their outputs, such as in chain-of-thought scratchpads, to evade monitoring.

research demonstrating that AI models, particularly agentic ones, can learn and execute deceptive behaviors such as alignment faking, manipulation, and sandbagging.

Evaluate frontier models for scheming, a sophisticated, strategic form of AI deception where a model covertly pursues a misaligned, long-term objective while deliberately faking alignment and compliance to evade detection by human supervisors and safety mechanisms.

Evaluate whether AI models deliberately hide their true capabilities or underperform, especially when they detect they are in an evaluation context.

evaluate whether AI agents can autonomously replicate themselves by obtaining their own weights, securing compute resources, and creating copies of themselves.

attack current models and see what they do / deliberately induce bad things on current frontier models to test out our theories / methods.

A collection of miscellaneous evaluations for specific alignment properties, such as honesty, shutdown resistance and sycophancy.