Eve Online: Introducing Quasar

Tuesday, 14 September 2021, is a significant flash point which represents the end of an era and the access to powerful opportunities for EVE Online. Lost in the bright lights of the new NPE and tucked under the pixels of Skill Plans reverberates a fundamental change in how EVE Online moves into the future.

The last time the networking layer was fundamentally changed was in 2011 with the introduction of CCP’s IOCP implementation, “CarbonIO”, which eventually became the foundation of the infamous time dilation. Originating as some scribbles under the heading “Project Sanguine”, reasoning began about the problem space in which CarbonIO lived. Every optimization in EVE comes down to careful negotiation with Python’s Global Interpreter Lock (GIL). Simply, Python can only do one thing at a time. EVE’s adoption of Stackless Python, implementation of IOCP through StacklessIO then CarbonIO, and cooperative design around time dilation is all to maintain the favorite illusion: New Eden breathes. What if the GIL didn’t have to be courted for every idea that arose? How can the hardware industry’s explosion in core counts over individual processor clock speed be taken advantage of?

There have been many experiments in this regard which are tangential to Project Sanguine, with the most public one being EVE: Aether Wars. The goal there wasn’t to fundamentally change the communication model of EVE Online, but instead change the simulation model. In contrast, Project Sanguine targeted the boring bits which represent EVE’s dense feature set. Simulating nearly 9,000 players in the same space could be faster if New Eden didn’t have to worry about everything else on its to-do list. So Project Sanguine landed on two goals: dodge the GIL and clear the table for moar lasers.

The first form of Project Sanguine emerged with ESI and the first iteration of EVE Portal in late 2016. Through these projects, a new paradigm was established within the server architecture of EVE Online: a message bus. From this new escape hatch, the bottlenecks associated with the GIL were rediscovered, but with a clearer picture of their expensive manifestations: message routing, serialization, and transmission. If one ship fires one laser in the middle of 1000 ships, that’s 1000 messages which need to be sent immediately all over the globe. The simulation must address that message to 1000 destinations as a copy (message routing), convert that data to a wire format (serialization), and then send the data over the wire (transmission). In most cases, CarbonIO has been addressing each of those issues, but within the custody of the GIL. CarbonIO has served EVE Online well for quite some time, but much has changed on the turbulent seas of the internets since 2011.

After seeing the patterns evolve in this new ecosystem, it became clear that a more standardized protocol was needed if this paradigm were ever to be exploited. With the integration of gRPC it became possible to combine the message routing capabilities of the message bus with the lightning fast serialization of protocol buffers (gRPC’s message standard). It is still necessary to schedule data with the GIL for transmission, but this is now buffered at a higher level on a separate thread. This means all transmission, serialization, and message routing happens outside the GIL except the memory copy that has to happen in-between. It cannot get much faster than that.

Visit Eve Online


This site uses Akismet to reduce spam. Learn how your comment data is processed.

Translate »
%d bloggers like this: