Jonor said:
Thanks for the information, the biggest hurtle I’ve found so far is finding a example of TLS being used with c++ an TCP. Seems finding documentation is nonexistent or I am just looking in the wrong places.

It is nearly always implemented in a library rather than writing yourself. Just like you could write your own video codecs or your own zip decoder, it isn't normally done. A skilled team could implement it, certainly, but it is far easier to use an existing library that is already certified and validated to work correctly. Both Unity and Unreal support encrypted network connections if you enable it, both added it about two years ago but it requires some effort to configure and to provide a cypher suite. Steam's GameNetworkingSockets are encrypted using a suite based on QUIC and hardware accelerated AES, and Amazon's Lumberyard has optional support for TLS/DTLS.

As for performance, it depends on the cypher suite being used. An older AES256 algorithm is compute heavy but has hardware support on many chips. There are software-only cyphers that are much faster, and some cypher suites that use significantly smaller cyphers sacrificing security difficulty, but for game content where data is stale within seconds that often isn't a problem. (You wouldn't want it for voice chat, though.) No matter what you choose there will be a cost for the cypher system.

I had to think about that topic also several times in the past and will have to in the future as well. My thoughts on the topic are that first, you don't need to use AES256 while AES128 is almost as secure. The increased key size in AES Block Cipher Encryption doesn't really matter to the basic algorithm because it increases only the amount of cycles used to encrypt a block, not the security perse. From experience, even a software AES algorithm is already fast and small enougth to be implemented by your own at a good level of proof.

But the real problem is still the handshake. It turned out in the past from mistakes of several big companies like Sony for example, that reusing the same key over and over is not the best practise and putting a fixed key into the software is also the worst idea you can have. So the most power should be put into thinking about how to generate and provide a proper encryption key. My favorite solution was provided from Bitcoin. Some slight interest into financial stuff and the first Bitcoin hype brought me to study the technology and have a brief insight into crypto security related to it as well. Please note, I'm far away from being a crypto expert but had some success on implementing the algorithms on my own with proof to well established libraries.

Something I really like to use is AES in combination with ECDSA. ECC (elliptic curves) and the data signing based on them, are somehow fun to use and secure as well, to verify identities. So a solution we used in a commercial project in the past was to provide an ECDSA based handshake from totally random keys and establish an AES encrypted secure communication channel with diffie hellman key exchange over ECC. This means that both sides use some crypto math to compute the same public shared key from the ECDSA public key of the other end, which was used to sign the handshake message. You can then use that key to compute another, secret because not shared key-pair, which is different on each side, and feed that key-pair back to the ECDSA algorithm. So instead of the public key used to sign the handshake message, following messages have to be verified with the new not shared but computed key. Because it isn't shared, it is considered to be secure. We then used the public key of the end we want to send a message to AES128 encrypt the message because the sender computed the public key of the target and the target computed the private key to sign their messages with it, which leads to the same public key by design of ECC. We used smaller (128 bit) and so faster key-pairs for generic ECC messages because they are random anyways and compute in less time, while more relevant but less often messages, like login with username and password, are secured by 384 bit key-pairs.

As from what I read, this is somehow similar to how TLS works. However, the only SSL/TLS implementation that I found from googling was that from OpenSSL and to be honest, the library is a mess!

I didn't know about QUIC before but will definitely have a closer look into it as well. In the end it depends how much secure data you transmit between client/server or client/client in a p2p network. Updating position data for some players is for sure less significant if you perform proper sanity checks in your game as chat messages, account or login data. The most important thing to keep attention to and what makes the Bitcoin principle so successfull is to no trust anyone, especially not the croud ?

Handshaking only needs to occur once, and the initial connection cost is already covered by a ton of other tasks in games. Key exchange is rather complex and a common source of security bugs for young libraries.

As for the level of security, it helps to understand your threat models and know when your data becomes stale. Most game data is stale within seconds, if someone decrypts it and can use it twenty seconds later won't hurt the game, the big advantage is to prevent a range of immediate communication attacks, so even an easily-broken cypher suite like DES can work, although it doesn't have hardware support these days. Attackers may employ complex tools, but they're not spending millions of dollars to continuously decypher game data, instead those attackers would simply compromise the box. Using a proper encryption system can help solve problems like eavesdroppers and session splitting, but doesn't need a cypher that can stand up for eons.

Voice chat is a little trickier both for threat models and stale data. Some people discuss more private details on game chat, and probably deserves a higher grade of encryption and people would expect more than a few seconds of security. It can also be subject to wiretap laws, which vary around the world.

Jonor said:
thank you for all this valuable information, just out of curiosity have you checked out MBed before? Its fairly lightweight mBed.org I’ve also looked at QUIC

You have an option to wrap by CLI code some C# System class from library, that gives encrypted streams. I do not know how much up to date they are though.