I did something similar to what's been discussed here a while ago as a hobby project (although not at MMO scale, intended to handle 200~ users max). After testing it with 80~ clients I found what I developed was an immensely over-engineered hackjob which actually needed all of those over-engineered hacks to function. And then I wasted a few months of my life fixing it up. At 130~ users I realized the hacks I made to get this far wouldn't cut it when scaled up. I'd imagine if you keep scaling up it just keeps getting harder and harder.

Also you can't stop cheating in this setup without SOME form of dedicated validator servers that manage conflicts. They'll just create headless clients that validate fake data while invalidating clients that invalidate the cheater.

Actually I think I have the source somewhere if you really want it, but it's probably not human readable and doesn't scale up as big as you seem to want it.

Also this whole peer-to-peer architecture would at most take me one day to write up, so it's really not that big of a deal.

Oh. Oooh. AAAAAAAAAH!


So, anyway, another option is to allow players to host servers, and use a central listing/discovery server.
You can document the protocol and let users type in the name of the discovery server, so if you ever take down your servers, fans can keep it going.
Servers that are hosting will post to your listing server once every 4 minutes.
Your listing server will remove servers from the list if they haven't posted in the last 5 minutes.
Clients can get a list of servers that match some criteria (game mode, host username, etc.)
That's really all you need. A bonus would be to also let the listing server support NAT punch-through, but that's not needed for a simple solution (tell users to port forward when hosting a server.)

:P

I'm planning on having clients directly enter the IP of the host (the same way that Minecraft does it), instead of querying a server list (which someone has to manage).

I initially did think of programming my own RESTful API for listing servers and querying, but then I decided not to since I plan on publishing the game to Steam in the future (which has their own matchmaking via the Steam API).

Also this whole peer-to-peer architecture would at most take me one day to write up, so it's really not that big of a deal.

MAYBE the initial structure you think will work will take a day. Then, when you start to use it, you'll run into an issue. then, when you get more users, you'll get more issues. Then, when you have a lot of players, you'll be running into different issues all the time, all related to trying to use peer-to-peer, and you're time will be dedicated 100% to trying to find a solution for all the problems. You'll fix one problem, and another will arise, all the while, you'll be hearing about cheaters doing this or that.

Obviously, with any networking solution you select you're going to have some amount of debugging you're going to run into, but, if you're trying to do peer-to-peer with any sort of game that requires sub-second communications amongst all the players, it's going to be 1000x worse.

With that said, if you're still going to write this peer-to-peer framework, I truly hope it does well, and you can come back with a synopsis of it and we can all learn from it.

Yeah, that's very true. I take back what I said about taking 1 day to program this, 1 week is a more realistic estimate if you consider the whole nine yards.

Thanks! I'm gunna be creating a prototype game this weekend that uses this architecture and will be handling roughly the same amount of network traffic as my main game. I'll be sure to post an update on how things turn out.

I did something similar to what's been discussed here a while ago as a hobby project (although not at MMO scale, intended to handle 200~ users max). After testing it with 80~ clients I found what I developed was an immensely over-engineered hackjob which actually needed all of those over-engineered hacks to function. And then I wasted a few months of my life fixing it up. At 130~ users I realized the hacks I made to get this far wouldn't cut it when scaled up. I'd imagine if you keep scaling up it just keeps getting harder and harder.

Also you can't stop cheating in this setup without SOME form of dedicated validator servers that manage conflicts. They'll just create headless clients that validate fake data while invalidating clients that invalidate the cheater.

Actually I think I have the source somewhere if you really want it, but it's probably not human readable and doesn't scale up as big as you seem to want it.

You can prevent fake data by having each user sign their commands with their own digital signature and a time-stamp that can't be reused. Only the primary host (other than the user themselves) would have access to the user's public key.

However every user would need to have a copy of the primary host's public key in order to validate the commands that are getting relayed from the host.

These public keys can be exchanged once a user joins; and we don't really need to worry about MITM attacks since this is just a game and we don't expect players to be eavesdropping on other players' networks.

Also could you elaborate on what factors prevented you from scaling up the number of players?

Thanks for the input! :)

Yeah, that's a good point actually.

I just realized that we don't even need proper authentication since we're assuming that MITM attacks are out of the picture.

A simple and effective way for the 1st host to validate commands from all the users is to have each user send a shared secret (i.e. a randomly generated 32 byte string) to the 1st host when they join the server. All future commands will then need to contain this shared secret in order to be valid.

In order for the 2nd host to validate commands from the 1st host, the 1st host would have to use the shared secret when sending commands to the 2nd host.

And since the IP of reg users are not visible to other reg users, we don't need to worry about fake commands from the 2nd host to the reg users.

This should be good enough I think? Thanks again, you saved me from typing out some unnecessary code :P

EDIT: Addressing your second point, that's not a problem since everyone's simulation is based on the host. In fact, if a user pokes around in their memory, all they're gunna do is mess up their own simulation.

that's not a problem since everyone's simulation is based on the host

So if I run a host, I can cheat by poking at the memory of the process.

This is exactly the issue. Consider the following: A hacker sets up a host where he modifies the memory, and also clients that are set up to confirm the host's actions. The host and the client both verify that the host's information is valid, and communicate that to other hosts.

It's not a secure structure unless you have a dedicated audit machine above each host. But that adds overhead and latency in most cases.

Also could you elaborate on what factors prevented you from scaling up the number of players?

Everything gets more complex. The issue I found was that players would eventually have too much load (processing or network), and have to resolve expensive things like collisions/bounds checking.

In order to help resolve it I essentially had each host act like a zone in a typical MMO, but this added an immense ammount of host migration work. With a system where each host only handles players around him, and communicates state changes to the nearest few hosts at a smaller frequency it was able to handle a good ammount of players.

But nowhere near MMO levels.

I figure'd the only way to reasonably scale it up to the levels you're talking about would be that you need to host dedicated application servers to extrude as much non-time critical code as possible, as well as some dedicated hosts to handle high volume areas. Basicallly what you're looking at is recreating something simimlar to Eve online's application sharding techniques leveraging P2P as well.

Not so say that you shouldn't try, though. I learned a ton about P2P networking while working on my project. I also learned it's just not for me, and haven't touched it since :P Especially with machines being so cheap now, you can easilly run servers on commodity hardware.

If you're actually aiming to do anything other than learning though, I'm working with a fantastic bare-bones library called "darkrift", and am able to get way more players on commodity hardware pretty trivially. A version 2's coming out soon based on this networking stack

https://github.com/DarkRiftNetworking/Hazel-Networking

If you've got experience with optimizing C#/.net GC I'd strongly recommend going this route instead. $150~ for an unlimited license/no monthly fee (or just building your own solution ontop of Hazel) vs working for probably over a year getting everything sorted out, if it's even possible entirely P2P (Which what I've seen indicates it's really not feasible)