Anyone ever try migrating a game server from one machine to another, to get more or less resources depending on load? AWS offers live migration for both Xen and KVM VMs. It's sometimes used when a multithreaded web server needs more threads.

Live migration uses a clever trick. First, all memory pages are marked as dirty (un-copied) by the hypervisor. Dirty pages then start to be copied, over the local network, from source to target machine. It's a lot like writing pages out to virtual memory on disk. Meanwhile, the program is still running, creating more dirty pages. There's thus a race between the copier, making copies of clean pages, and the program, making pages dirty. The endgame is freezing the program, copying the remaining pages, and starting up the version on the target machine. All networking and file connections are of course shifted over. The freeze period may be as little as a few hundred milliseconds.

This is a standard AWS offering, not some experimental thing. It's been in use for years.

The question is, has anyone tried doing this underneath a running game server? And, if they have, how long was the “freeze” period? Did it work out?

I'm thinking about this as a strategy for big-world systems which need to apply more resources to areas with more load.

What I'm looking at here is the Second Life/Open Simulator architecture. Open Simulator is open source, compatible at the message level. Big, seamless world, spread across processes on multiple servers. Each process runs one region, 256m x 256m. Currently, each process is mostly single-thread, but that could be fixed for at least two of the major internal bottlenecks (user scripts and physics).

Now you want to allocate more resources to regions with more players. Restarting the region server and starting it on a different machine allows this, but the region blanks out for about two minutes and all users are kicked out of that region. If that could be reduced to a second or two, without user kick-out, adjusting resources more often becomes possible.

So the idea is that when traffic builds up at a club, or there's a big meeting, or a city gets busy, restart the region on hardware with more CPUs. Conversely, idle regions get migrated to smaller machines and have multiple region processes running on one CPU.

This is sort of a low-rent way to do what Spatial OS does, without a rewrite.

Not AWS MGN, but AWS KVM or XEN migration. MGN is copying disks from non-AWS servers into AWS, trying to bring them into sync.

KVM or XEN migration requires that both source and target machines share networked disk storage. Since that's the normal case in AWS, that's not a big problem.

It's surprising how little is written about this on the Web, which suggests it's not used much.

hplus0603 said:
If you can manage to multi-thread physics and user scripts, which is a bit of an if …

Depends on the physics engine. SL uses Havok, which claims to be multi-threaded. The approach is to divide the world into sections which are not currently interacting physically, and solve each separately. That's a good fit to how SL works. Open Simulator seems to be using single-thread Bullet, and I'm not sure how well that works. On the Second Life side, physics tends to use about 5%-10% of frame time. Scrips use most of the time.

Script parallelism is not too hard, because of the way Linden Scripting Language works. There's not much state. Each script has 64KB of memory. There's no synchronization between scripts; they all free-run independently. It's quite possible, and normal, for some part of the system state, like an object location, to change while a script is running. So scripts can be run concurrently.

I'm not working on this directly; I'm tied up trying to write a new client in Rust. What I'm trying to do is to find potential solutions to big-world scaling problems. Second Life, old and clunky though the implementation is, comes closer to the “metaverse” than anything out there. There's no real limit to the size of the world except your server budget.

But don't bunch up. 20 to 40 avatars max per region before it gets sluggish. That's the biggest scaling problem now. If a solution to that problem is found, the other dimension of scaling has been cracked.

That's the hard problem technically. The migration stuff is to make it cost-effective. At any given time, more than half the regions in Second Life are empty of users. Those could be stacked up 4 to a CPU core, if you could give them more resources gracefully when someone showed up. That frees up AWS resources for those regions where someone has a busy club with 200 people, and you'd like to move that region to a 16 or 32 or 64 core AWS instance.

What you describe doesn't sound like its worth the effort.

There is already a bunch of well-established tech to let people co-exist on servers easily. It works in either direction, with letting players share data or not depending on whatever your game wants. It can work with players sharing a server collaboratively, or people coexisting in the same virtual world data but independent of each other with tech often called “phasing”. Passing players between servers is a solved problem.

In a game world where you do rolling updates (not common, but still happens) you migrate everyone from one server to the next and it all just works. Then when the player count is zero on that server, bring it down and do the work.

This tech already works well at controlling the costs of managed servers. When a server gets below a threshold migrate them off the box and shut off the instance. When a server gets above a threshold migrate them to other servers and potentially start another server if they're all getting busy.

So I don't see a benefit to migrating virtual machines in this case. There is no need to keep the virtual machine itself running and migrated between hardware. Just start up a server and shift players over to it when you want, or shift the players off the server and kill the server when you're done.

frob said:
letting players share data

The world itself, with its own objects and activity, needs to live someplace. I'm looking at this from the Second Life/Open Simulator perspective, where the land has a life of its own. Even when nobody is around, some things continue. Too much, by most game standards. There are animals grazing, NPCs working, vehicles running around, and even plants growing. This is all user-scripted; the game engine just sees this as non-player objects running. There's a slowdown mode when nobody is around, but all the state is maintained.

These systems are land-centric, not avatar-centric. That's common in building-type games, from Minecraft to Factorio. In a way, avatars are just visitors.

hplus0603 said:
I think not much is written about Xen/KVM migration in Amazon, because it's not a feature they support.

I suspect you're right. It's a listed feature, but all I see about it are how-to guides copy-pasted from the documentation.

Trying to pull scripting out of the simulator process is probably going to result in more overhead. Typical scripts don't do much work per event. RPC overhead would dwarf the actual work.

The Roblox people are going all-out on massive numbers of avatars in scene. There's a “Metaverse” discussion on the Roblox site where they talk about having 50,000 avatars in a stadium, you wave a flag to your friend on the other side of the stadium, and they can see it and wave back.