{"id":722089,"date":"2025-11-11T05:02:40","date_gmt":"2025-11-11T05:02:40","guid":{"rendered":"https:\/\/blog.roblox.com\/?p=209254"},"modified":"2025-11-11T05:02:40","modified_gmt":"2025-11-11T05:02:40","slug":"how-were-making-robloxs-infrastructure-more-efficient-and-resilient","status":"publish","type":"post","link":"https:\/\/arcader.org\/news\/how-were-making-robloxs-infrastructure-more-efficient-and-resilient\/","title":{"rendered":"How We\u2019re Making Roblox\u2019s Infrastructure More Efficient and Resilient"},"content":{"rendered":"

As Roblox has grown over the past 16+ years, so has the scale and complexity of the technical infrastructure that supports millions of immersive 3D co-experiences. The number of machines we support has more than tripled over the past two years, from approximately 36,000 as of June 30, 2021 to nearly 145,000 today. Supporting these always-on experiences for people all over the world requires more than 1,000 internal services. To help us control costs and network latency, we deploy and manage these machines as part of a custom-built and hybrid private cloud infrastructure that runs primarily on premises.\u00a0\u00a0<\/span><\/p>\n

Our infrastructure currently supports more than 70 million daily active users around the world, including the creators who rely on Roblox\u2019s <\/span>economy<\/span><\/a> for their businesses. All of these millions of people expect a very high level of reliability. Given the immersive nature of our experiences, there is an extremely low tolerance for lags or latency, let alone outages. Roblox is a platform for communication and connection, where people come together in immersive 3D experiences. When people are communicating as their avatars in an immersive space, even minor delays or glitches are more noticeable than they are on a text thread or a conference call.<\/span><\/p>\n

In October, 2021, we experienced a system-wide outage. It started small, with an issue in one component in one data center. But it spread quickly as we were investigating and ultimately resulted in a 73-hour outage. At the time, we shared both <\/span>details about what happened<\/span><\/a> and some of our early learnings from the issue. Since then, we\u2019ve been studying those learnings and working to increase the resilience of our infrastructure to the types of failures that occur in all large-scale systems due to factors like extreme traffic spikes, weather, hardware failure, software bugs, or just humans making mistakes. When these failures occur, how do we ensure that an issue in a single component, or group of components, does not spread to the full system? This question has been our focus for the past two years and while the work is ongoing, what we\u2019ve done so far is already paying off. For example, in the first half of 2023, we saved 125 million engagement hours per month compared to the first half of 2022. Today, we\u2019re sharing the work we\u2019ve already done, as well as our longer-term vision for building a more resilient infrastructure system.<\/span><\/p>\n

\"Roblox<\/h3>\n

Building a Backstop<\/span><\/h3>\n

Within large-scale infrastructure systems, small scale failures happen many times a day. If one machine has an issue and has to be taken out of service, that\u2019s manageable because most companies maintain multiple instances of their back-end services. So when a single instance fails, others pick up the workload. To address these frequent failures, requests are generally set to automatically retry if they get an error. <\/span>
\n<\/span><\/p>\n

This becomes challenging when a system or person retries too aggressively, which can become a way for those small-scale failures to propagate throughout the infrastructure to other services and systems. If the network or a user retries persistently enough, it will eventually overload every instance of that service, and potentially other systems, globally. Our 2021 outage was the result of something that\u2019s fairly common in large scale systems: A failure starts small then propagates through the system, getting big so quickly it\u2019s hard to resolve before everything goes down.\u00a0<\/span><\/p>\n

At the time of our outage, we had one active data center (with components within it acting as backup). We needed the ability to fail over manually to a new data center when an issue brought the existing one down. Our first priority was to ensure we had a backup deployment of Roblox, so we built that backup in a new data center, located in a different geographic region. That added protection for the worst-case scenario: an outage spreading to enough components within a data center that it becomes entirely inoperable. We now have one data center handling workloads (active) and one on standby, serving as backup (passive). Our long-term goal is to move from this active-passive configuration to an active-active configuration, in which both data centers handle workloads, with a load balancer distributing requests between them based on latency, capacity, and health. Once this is in place, we expect to have even higher reliability for all of Roblox and be able to fail over nearly instantaneously rather than over several hours.<\/span><\/p>\n


\n<\/span>\"Replicated<\/span><\/p>\n

Moving to a Cellular Infrastructure<\/span><\/h3>\n

Our next priority was to create strong blast walls inside each data center to reduce the possibility of an entire data center failing. Cells (some companies call them clusters) are essentially a set of machines and are how we\u2019re creating these walls. We replicate services both within and across cells for added redundancy. Ultimately, we want all services at Roblox to run in cells so they can benefit from both strong blast walls and redundancy. If a cell is no longer functional, it can safely be deactivated. Replication across cells enables the service to keep running while the cell is repaired. In some cases, cell repair might mean a complete reprovisioning of the cell. Across the industry, wiping and reprovisioning an individual machine, or a small set of machines, is fairly common, but doing this for an entire cell, which contains ~1,400 machines, is not.\u00a0<\/span><\/p>\n

For this to work, these cells need to be largely uniform, so we can quickly and efficiently move workloads from one cell to another. We have set certain requirements that services need to meet before they run in a cell. For example, services must be containerized, which makes them much more portable and prevents anyone from making configuration changes at the OS level. We\u2019ve adopted an infrastructure-as-code philosophy for cells: In our source code repository, we include the definition of everything that\u2019s in a cell so we can rebuild it quickly from scratch using automated tools.\u00a0<\/span><\/p>\n

Not all services currently meet these requirements, so we’ve worked to help service owners meet them where possible, and we’ve built new tools to make it easy to migrate services into cells when ready. For example, our new deployment tool automatically \u201cstripes\u201d a service deployment across cells, so service owners don\u2019t have to think about the replication strategy. This level of rigor makes the migration process much more challenging and time consuming, but the long-term payoff will be a system where:\u00a0<\/span><\/p>\n