🎉 Celebrating 25 Years of GameDev.net! 🎉

Not many can claim 25 years on the Internet! Join us in celebrating this milestone. Learn more about our history, and thank you for being a part of our community!

Back to Networking and Multiplayer

Serving large files over TCP

x · 2015-01-22T01:26:27

My server needs to send some large files to my clients over the TCP connections. These are personal files, so they cannot be hot linked through a CDN url. These files are also stored on S3. What I am thinking right now is to have my server downloads these files from S3, then send it over the tcp. However, now I am concerned with the memory usage because server needs to download the file first, which then will be kept in memory before sending it over to clients. If these files averaged to about 5MBs, and I have 1000 connected clients, and there is one file for each client, that means server must have at least 8-10GB of RAM. The 5MB/file average is a actually pretty conservative measure. Some of these files can be as big as 20MBs. Is there a better alternative?

Networking and Multiplayer Programming

Started by alnite January 13, 2015 07:06 PM

27 comments, last by hplus0603 9 years, 5 months ago

hplus0603

11,920

January 22, 2015 01:26 AM

I would assume that a "dumb" server like this could be load balanced behind any ol' load balancer. Anything from HAProxy to Amazon Elastic Load Blaancing. Each separate instance has no state, so you can spin them up as much as you want.

If you also do who-can-see-what permissions, then the database of that may need load balancing at some point (but that point is surprisingly far into the future.) Hash-based application level sharding will probably work just fine for this. Define 8192 logical databases, hash the user name, modulo by the number of databases, and talk to that particular logical database instance. As long as you don't database join across different customers, this will work just fine. (Or use Amazon SimpleDB :-)

enum Bool { True, False, FileNotFound };