IA en un minutoNewsBusiness and products
OpenAI explains how it grew the storage serving more than 1 billion ChatGPT users
IA en un minuto newsroom · Editor: Jon Elgezabal
In 30 seconds
Habitat, the system where OpenAI keeps the data for ChatGPT, the API and Codex, runs on Azure Cosmos DB and already tops 500 petabytes. The company describes two outages in late 2024 that led it to watch the platform together with Azure's engineers. With the service rewritten in Rust and a cache, a typical read now takes about 10 milliseconds.
OpenAI has published on its engineering blog the second part of a series about Habitat, the online storage platform that many of its products depend on, among them ChatGPT, the API and Codex, plus internal services. According to the company, Habitat has to answer in milliseconds to more than 1 billion users spread across 40 geographic regions, and it already serves more than 500 petabytes of data. The first part, published on September 11, told how it went from a Python library to a service.
This second instalment looks at the layer underneath: Azure Cosmos DB, Microsoft's cloud database. OpenAI's largest Cosmos DB account stores tens of petabytes and handles more than 20 million queries per second, and the company says that, as far as it knows, no other company has deployed Cosmos DB this large. The post describes a journey in three phases: first, a single shared container in each account; then, separate containers for each large service or feature; and finally, a return to shared containers by default, with the ability to move data between them. The Cosmos DB team built an improved serverless mode for OpenAI that let it stop watching and expanding containers by hand. Since then, according to OpenAI, it has had no provisioning incidents and no "noisy neighbour" incidents, workloads that hog shared resources.
The post also recounts two outages. In late 2024, a power cut in the main Azure region of one of its largest Cosmos DB accounts showed that moving the biggest accounts to a secondary region would take several hours, something neither OpenAI nor the Cosmos DB team had tried at that size. Around a month later, while moving them back to the original region, they caused another serious outage. Today the Habitat and Azure teams monitor every layer together, from OpenAI's services down to Cosmos DB's physical storage.
More than 90% of requests to Habitat are reads, and speeding them up has been a priority this year. To do so, they keep data close to users when capacity allows, use a cache and have rewritten the service in Rust, after it was first built in Python. Altogether, the median read time worldwide has dropped from roughly 120 milliseconds to roughly 10, a 12-fold improvement since they started focusing on performance at the start of the year. The slowest requests still have to fetch data that is not in the cache, often from distant regions.
For the next 1 billion users, OpenAI says it is developing its own exabyte-scale storage system, designed for Habitat's bulkiest data, which pairs Rockset's RocksDB Cloud core with replication based on Raft.
Why it matters · analysis and opinion
The lesson that applies outside OpenAI is that a managed service stops being fully managed once usage grows enough: at some point provider and customer have to share tests, dashboards and responsibility for failures. The two outages the post describes came from something nobody had rehearsed, moving huge accounts from one region to another, and that holds for any company with a recovery plan it has never put to the test. The in-house system OpenAI is already building also suggests that not even that partnership will cover the next leap on its own. For smaller teams, the practical step is to really rehearse a region switch, measure where each request spends its time and only then decide whether anything needs rewriting.
Official source: OpenAI · Written with the help of AI: how we make the news
