Does the server have to be a c#/unity program? If server hardware resources are limited, I would set up a simple server program running the absolute minimal physics simulation you can get away with, on a physics engine such as Bullet. This would be authoritative over the player movement. You can then use unity, or whatever you are using to power your clients, to drive a local physics simulation on the clients for a kind of client-side prediction. These could also simulate extra visual objects that the server does not need to care about, for nicer visuals etc.

Using two physics engines like this obviously requires a bit of tinkering to get them to agree on everything, but for simple player locomotion, that should not be that much work (disclaimer: I don't know if your locomotion is indeed simple).

On 11/22/2018 at 5:43 AM, Hodgman said:

On a slower scale, e.g. characters who walk around a world, the exact same issues apply. The jankiness is just exaggerated and more obvious in the race car example

I think the other thing to consider is how important a collision is. I mean if just stops a character's movement for a moment it may mot be a big deal, even if it never happened on the other client. In a car game where it might cause a catastrophic crash on one client and not the other, it's a different story. I found it very annoying in Age of Conan when you were stopped by something you couldn't see because the server ruled over the client in this way.  I would much prefer to see someone else's characters pause for a second for seemingly no reason than have mine bump into something invisible. This brings us too response. If only your character is effected when you bump into someone as in some MMOs, it doesn't really matter to any other client what happened on yours.

12 hours ago, Septopus said:

I'm trying to avoid using 3D hardware on the server,

That is exactly my point. You can make the server a windowless program. It won't require any graphics hardware, it won't even require a window system.

12 hours ago, Septopus said:

Again though, the goal is to build the system in a distributed fashion, if I did add a server physics simulation, I would want to distribute that to multiple servers as well...

I don't see how this is incompatible with server authoritative physics. You can have a system where server instances are spun up depending on need and player count. (Obviously this all depends on the game type). If the server is only responsible for simulating the gameplay-relevant physics and propagating that to all clients you can even attempt to run multiple server instances on a single machine.

Anyway, you seem to have a direction you want to go, so that's probably what you should do.

Well, the trick is not to use an engine for those parts. Write a "text" program in C++ (or C# or whatever you are comfortable with) and link to the physics library. In the case of Bullet this is a collection of headers/classes. Everything runs completely on the CPU.

I do recognize this might require you to write parts of the simulation code twice, ie. once for the server and once for the client, but you can be smart about it. I have worked on a project in Unity where we separated the simulation code from the presentation code. The simulation code was not allowed to access any of the unity classes, only standard C# library code. This meant writing your own Vector3 classes and other helpers you needed to run the simulation. The presentation code was allowed to reference everything in the simulation code + all of unity, and it was responsible for rendering (or "presenting") the world.

The simulation code was then easily packaged into its own C# "console program" and run on a server without Unity.

As for simulating the server-side physics, there are several ways to do that. You can write all of your physics code yourself, which may or may not be feasible. That way the exact same code can run on the client and server. Another way is to use a physics engine. If you are using C# you may have to use a wrapper for this as most physics engines are native code. I have used a wrapper for the ODE physics engine in the past. AFAIK there are c# wrappers for Bullet as well.

Unity obviously uses PhysX under the hood so you may have to experiment a little to get your server-side physics to match the client(s). This is also true if you use PhysX directly on the server, as Unity has its own layers in between in the game code and the raw physics code. This is what I was referring to earlier (the "tinkering" part). However, with state replication, you'll be doing gradual corrections on the clients so it shouldn't be too difficult to get it working quite well.