Game Environment : Large, traversable vehicles similar to Space Engineers.
Networking Solution :
For Vehicles : Vehicles are under direct on-server computer control. Players can adjust settings on the vehicle but ultimately fine control is from the computer. Players can update vehicle state controls. On their client, instantly when a button is pressed, the state controls update to indicate the new state. A small window of time exists where a player can "undo" a command and the command will "undo" if the message reaches the server in time. As commands reach the server, the server incorporates these new command states into it's simulation of vehicle dynamics. Then, the server sends back new update messages - basically a big message using TCP/IP that contains all of the new states for the game world in the future. Server always runs about 0.5 seconds in the future.
Clients are always interpolating the game world knowing what the future state will be. It's interpolation from a known past state, to a known future state. The idea is to make all movements always correct, down to the millimeter - the client does physics integration per frame, while the server uses a different timestep period, and the client applies a correction factor per frame so that the end result always ends up in the given future state.
For direct client movement, P2P and PeertoServer UDP frames. Every time a client performs a movement or use or tool switch command, etc, it sends via UDP to both all peers who are close enough to see the client and to the server any actions taken. Clients then use future state prediction for player models and player shovable small objects only. (future state prediction being inaccurate) . (like many other games, every time the player performs an action, their client UDP spams both other clients and the game server with a UDP message every 20ms or so. The message spam continues until an ACK for that particular UDP message arrives. The server then UDP spams all clients who need to be "relayed" this message - so at all times the UDP link is being used between between both clients and between client and server so that there is the least possible latency between 1 client performing an action and another client knowing about it. If a client has other clients blocked via firewall, the 100% packet loss in the link eventually causes a client to stop wasting bandwidth sending to clients who are not receiving. Similarly, since firewalls can block UDP from the server, the TCP update frames also contain the same embedded data so that the data gets there eventually)
A consequence of this is that the player's model cannot ever block a moving object in game. If a player is caught between, say, an elevator descending and the floor, the player must be crushed even if they are wearing indestructible armor. All world objects are unstoppable as far as the player's model is concerned.
To use TCP and UDP simultaneously like this I'll need a more clever throttling mechanism so TCP send rate is governed by UDP packet loss.
For larger environments and for handling of clients with limited bandwidth, there will also need to be a subscription system, where clients can subscribe to network events, limiting their subscriptions to stay within their network bandwidth availability, but that isn't needed to get a basic version of the game working.
