🎉 Celebrating 25 Years of GameDev.net! 🎉

Not many can claim 25 years on the Internet! Join us in celebrating this milestone. Learn more about our history, and thank you for being a part of our community!

Back to Networking and Multiplayer

How can I optimize Linux server for lowest latency game server?

Gintas · 2016-06-17T22:15:20

Hey, I'm running a game server for my mobile game. It runs as a Java 8 application and uses Kryonet. It uses both UDP and TCP quite oftenly - UDP for spamming update packets and TCP for shooting and a few other events. TCP and UDP uses different ports, not the same one. I'm currently running 3 game server application instances, every on different ports of course. Linux server runs on VPS, stock Ubuntu 14 with a few packages installed. Some players are crying, that they have high ping(> 100), even though they can play other games online well. I am aware, that location plays a big role, this question is not really about it. What could I do? Is my OS choice bad?

Networking and Multiplayer Programming

Started by zgintasz May 07, 2016 07:23 PM

29 comments, last by hplus0603 8 years ago

JohnnyCode

1,084

May 13, 2016 02:56 PM

What can I do for my Linux server to minimize latency as much as possible? Are there any config params?

If your application correctly acquires a socket/tcp connection/udp endpoint, I don't see what problems you are trying to search for in the OS instead of the very application.

Java is memory managed language, a great deal for active documents in http protocol etc., but instead of adviceing you to code in c++ the server application, you can still do following:

-Make your server application allocate new objects as least as possible, to not have memory being freed, fragmented, moved, too often since you are using a memory managed language.

(you can design a server to allocate/free memory only for a per-client income/outcome, this would help latency, since I gess, your game protocol has more constantial clients, if you have maximum amount of clients known, preallocate them)

- What about your clients serving logic? How do you read their data? how do you serve them data back?

- How do you measure respond latency?

zgintasz

398

Author

May 13, 2016 10:49 PM

I have just pushed the update for my game, so I will take a look at how it will go. I actually am allocating quite a fewobjects every loop. But I'm making thread sleep, so that it runs only 20x a second, so I adjust the sleep time with consideration on how long it took to finish the last iteration. For measuring latency, I use Kryonet's updateReturnTripTime(). I just looked into it, and it turns out it is using TCP... But I'm calling it only 2x a second. I need to check ping repeatedly so that I can extrapolate positions from update packets.

I found these parameters on this ServerFault question. Do they actually do anything useful?


net.core.rmem_default = 10000000
net.core.wmem_default = 10000000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

Game I'm making - GANGFORT, Google Play, iTunes

hplus0603

11,920

May 13, 2016 11:42 PM

If you update only 20 times a second, you are introducing at least a 50 ms latency in the ping/round-trip time to all players.

enum Bool { True, False, FileNotFound };

zgintasz

398

Author

May 14, 2016 09:31 AM

But that it not the latency I am measuring, because Kryonet's networking runs on a separate thread, so main thread's sleep time does not affect it.. And sending update packet only 20x a second and interpolating everything looks and feels nice in reality, I don't think there is anything wrong with it.

Game I'm making - GANGFORT, Google Play, iTunes

hplus0603

11,920

May 14, 2016 06:52 PM

that it not the latency I am measuring

Then you're measuring a latency number that does not accurately reflect the game playing experience. That may lead you to optimize the wrong thing.

I don't think there is anything wrong with it.

Lots of games do fine at 20 Hz, or even less, so I agree. My main comment comes from the question of "what can I do to shave latency off the server?"
It seems to me that you have no data nor real reason to believe that that's the actual bottleneck for your experience, so the best answer is probably "nothing" until you receive data to tell you otherwise :-)

enum Bool { True, False, FileNotFound };

zgintasz

398

Author

May 22, 2016 10:46 AM

Hmm, I noticed there is still some lag and high ping. I noticed it sometimes lags even with low ping. Here are some stats:

Why is the CPU always rising? How can I debug what is wrong?

Game I'm making - GANGFORT, Google Play, iTunes

hplus0603

11,920

May 22, 2016 06:38 PM

These graphs do not show latency at all. Number of packets and amount of bytes just shows that you have more traffic some times than others, it doesn't show you anything about the quality of that traffic.

That being said, to figure out what a system is doing with the CPU, you can use "top" to first figure out which processes are doing the most work, and then, depending on what process it is, figure out what it's actually doing.
If it's your own program, you can turn on a profiler, and see where it's spending the most time.

You can also use tools like "oprofile" and "strace" to figure out what's going on.
Oprofile lets you sample where in programs you most often spend time, and this is helpful to figure out which process is taking the most time, and if you have debug symbols for that process, figure out what part of code. For Java, it can't get you to source line level accuracy though.
Strace prints out all the system calls the program is doing. From that, you can work your way backwards to figure out what part of code it's in.

enum Bool { True, False, FileNotFound };

zgintasz

398

Author

June 17, 2016 01:53 PM

I fixed the CPU rise, turns out I wasn't removing one bullet type from physics world.

However, ping problems are still there. How can I debug what is wrong? I'm using Google Compute Engine, maybe they limit network performance?

E.g. asia server is located in Taiwan and player in South Korea has ping ~140 on WiFi, so he cannot play. What should I do?

Game I'm making - GANGFORT, Google Play, iTunes

hplus0603

11,920

June 17, 2016 05:24 PM

How is the ping measured?
Do you measure it in the game, or using the command-line "ping" tool?
What does a "traceroute" to the server from the player say?

Another thing to watch out for: Google Compute Engine is not optimized for low-latency real-time applications. You may suffer from virtualization-induced scheduling jitter in the game process. That kind of jitter would be visible to all players, not just one player, so I don't know if it applies in this case.

Finally: 140 milliseconds isn't that uncommon over the greater internet. If your game is unplayable at 140 milliseconds ping, then your game is not designed to be playable in general over the internet.

enum Bool { True, False, FileNotFound };

zgintasz

398

Author

June 17, 2016 07:26 PM

The ping is measured in-game. Client sends a ping packet to the server, server sends it back and then client measures how long did it take.

Hmm, I don't know the traceroute, I don't live in there. How would it help?

How much ping can that jitter add up? The CPU is staying up to ~15% all the time now.

Do you mean I should somehow optimise my game to let players play with a bigger ping? It's a 2D platformer shooter, you can take a look at it by clicking the link in my signature.

I'm really not sure how should I be solving this ping problem...

Game I'm making - GANGFORT, Google Play, iTunes