Skip to content
by InfinyAI ES

IA en un minutoNewsModels

Models

OpenAI's chief scientist proposes perplexity as a shared metric for measuring AI model progress

IA en un minuto newsroom · Editor: Jon Elgezabal

In 30 seconds

Pachocki, chief scientist at OpenAI, wants perplexity to serve as a common yardstick across labs and as a basis for agreeing safety rules. OpenAI is showing its internal figures: GPT-3.5 scores about 0.99, GPT-4 about 0.91, and new abilities in cybersecurity and mathematics emerged under 0.8. Pachocki admits the metric's flaws and calls for limits nobody crosses without a prior oversight milestone.

OpenAI's chief scientist, Jakub Pachocki, wants AI labs to adopt perplexity as a common yardstick for comparing model progress and for setting shared safety standards. Perplexity gauges how accurately a model guesses the next token (a fragment of text); the lower the figure, the better the prediction. He makes the case in a post titled "Perplexity and Intelligence", published on OpenAI's website.

As a first step, OpenAI is publishing its internal pretraining measurements (pretraining is the initial training phase on a large corpus of data). Measured as log-perplexity on a dataset drawn from the company's internal codebase, GPT-3.5 sits at around 0.99 and GPT-4 at around 0.91. OpenAI says transformative capabilities in areas such as cybersecurity and mathematics appeared once values dropped below 0.8, and that a model at roughly 0.76 began producing what the post calls "groundbreaking" results on open mathematics problems the company has been putting to its models for over a year. Pachocki concedes that releasing the numbers shows competitors more clearly where OpenAI stands, and argues that the urgency of agreeing on shared safety measures matters more.

The post also looks ahead. Going by the recent pace, the author expects further transformative capabilities when models reach the band of roughly 0.68 to 0.70, and admits he does not know what they will be or whether anyone is testing for them today. He adds that a wave of fundamental pretraining advances could conceivably arrive as early as 2027 if AI itself takes part in that research, and considers it unlikely that work on alignment and monitoring will move quickly enough to permit that full acceleration.

Pachocki also proposes common limits for the whole industry, based on perplexity and other measurements, which developers would commit not to cross until a concrete milestone in understanding and monitoring AI behavior has been reached. As an example of such a milestone he cites a successor to chain-of-thought (CoT) monitors. He also asks frontier model developers to agree on norms for evaluating these models and stress-testing their safety cases before first use, along with shared transparency standards.

He acknowledges the metric has problems: each lab uses different data, it does not measure specific tasks, it misses advances made outside pretraining (chain-of-thought reasoning does not lower perplexity even though it sharply raises capabilities), and nobody knows where its floor lies, the lowest value that can be reached.

It is a proposal from OpenAI, not an industry agreement.

Why it matters · analysis and opinion

Until now, labs have compared their models using tests of specific tasks that, according to the post itself, fall short within a short time. Pachocki's proposal aims at something else: a figure tied to base training that would help decide when to slow down. OpenAI showing its numbers is a step the company itself admits was not easy, but the idea only works if the other labs publish theirs and agree to measure them in a comparable way, and none of that has been agreed. Nor does the post say who would police the limits or what would happen if someone crossed them. For anyone using these models at work, nothing changes today. What the text does leave is a warning from the maker itself: it expects jumps in capability that are hard to foresee and acknowledges that the tools for overseeing them are lagging behind.

Official source: OpenAI · Written with the help of AI: how we make the news

AI in your inbox, every day or every Friday

The stories that matter, each one in a minute. With the source for every one.

Choose one or both:

Sign up and you are in: the daily arrives every night and the weekly on Friday mornings. You can unsubscribe from any email.