source61 said:
If using TLS is too much of a hassle, consider using plain RSA.

This is not a good idea for most users.

The weaknesses of crypto implementations are generally not in the crypto primitives themselves, but in poor implementations or poor choices. Even alleged professionals get this wrong – WEP encryption for WiFi was quickly broken because of such a poor choice, for example.

RSA is also super slow in practice. It's great to verify signatures, and verify that the other side is who they say they are, it also provides a convenient channel over which to exchange encryption keys for a more efficient symmetric cypher, but it also fails hard if you don't have a very strong certificate/key management scheme. And, even if you can securely establish good keys, with just a single wrong choice in how you then use those keys to implement your packet stream, you may lose all of that goodness.

1024bit RSA costs about an average of 1ms of CPU time per encryption / decryption separately

1024 bits aren't considered particularly secure these days. And you don't want to implement it yourself if you can help it – even if you have a good infinite-precision math library to fall back on, specifics about seed primes and such end up being important. Libraries like Crypto++ and Botan are already tested and debugged; I would highly recommend using those if you need that particular feature.

you can use RSA → XTEA.

I used to be a fan of TEA/XTEA because of the low runtime overhead myself, but it's not actually a part of any real cryptographic standard, which means it doesn't get the level of scrutiny that a standard algorithm gets, and thus there may be weaknesses in the algorithm that you don't know about, but a potential attacker does. XTEA is also a 64 bit block cipher, which is pretty small by today's standards.

Modern CPUs all have AES instructions built in, and they will encrypt data faster than any software implementation of an encryption algorithm. This is one of many benefits of actually sticking to a standard algorithm!

@hplus0603
I never recommended implementing cryptography on your own.
As you said, it's generally not a good idea to implement cryptography on your own, however OP said he was already trying to implement cryptography on his own, given that I recommended RSA in case he decides to continue trying to implement cryptography on his own rather than trying to force silk gloves on him.
In other words I think your criticism of my post is a little misdirected, if you want to tell someone not to implement cryptography on their own you should direct that to OP/whomever asked for it in the first place, from my perspective I just gave him what he asked for since personally I don't like to go around telling everybody what they should or shouldn't do, imho usually such topics are filled with personal bias and answers that often ends up just confusing new members instead of giving a simple practical answer to the question at hand and insert caveats when necessary.

1024 bits is considered less secure nowadays like you said, but afaik it's not broken yet, so it's considered “secure enough” (for now), though I know it's recommended and common practice to use 2048 bits. I've never tried implementing higher than 1024 bits of RSA since it's sufficient for my use case for now, but I'm pretty sure it's trivial to do (doubling the size of the primes?).

RSA is slow compared to say XTEA, which is why I suggested complimentary XTEA to go, however I don't think suggesting AES on its own makes any sense here, AES isn't assymetric so you can't use it alone for encrypting network data without exposing your keys to everyone making the cryptography useless, and in my experience AES has a high overhead for encrypting small chunks of data unless the library I tested out a while back just happened to be unoptimized/poorly implemented.

Let me know if you still think I'm wrong, cryptography isn't my area of expertise.

Anyhow, for learning about cryptography I'll still recommend RSA and XTEA. It's the gold standard for network encryption, relatively easy to understand and implement initially for some level of protection, and can later be “upgraded” with OAEP for example while in beta or some amount of time before releasing the game.
It's also the choice of use by CipSoft's Tibia, a multi-million dollar 2D MMORPG that's been going strong for nearly 25 years now with millions of accounts and tens of thousands of live players at any time to this day, so apparently it was good enough for someone.

Edit: Changed my post here from trying to give fair and neutral criticisms to joining the whole “online upmanship” this guy seems to be participating in, why not ?
Hint: Lay your criticisms really thick on and try to avoid giving any real suggestions that could be picked apart. Lets go!

hplus0603 said:

source61 said:
If using TLS is too much of a hassle, consider using plain RSA.

This is not a good idea for most users.

What's good for any user is to follow whatever path they're on. If you want to lecture someone you should save it for OP for already having decided to try to implement cryptography on his own, I just gave him the tool he asked for and you're basically nose diving in just to say “you're wrong” in 30 lines of generalities and buzzwords. Nice.

RSA is also super slow in practice.

I already indicated exactly how much RSA costs (with the obvious caveat that it will vary depending on the hw used), so I'm not sure why you felt the need to say this (twice).

It's great to verify signatures, and verify that the other side is who they say they are, it also provides a convenient channel over which to exchange encryption keys for a more efficient symmetric cypher, but it also fails hard if you don't have a very strong certificate/key management scheme.

What a nice little ensemble of armchair buzzwords. May I ask if you have a youtube channel?

1024bit RSA costs about an average of 1ms of CPU time per encryption / decryption separately

1024 bits aren't considered particularly secure these days.

That's kind of a fair criticism (although afaik RSA 1024 still isn't cracked, it's just good practice to double it), although it should be completely trivial to upgrade to 2048 bits, so it sounds to me like you're honestly just nitpicking for whatever +1 internet points you can get, so lets play that game then.

And you don't want to implement it yourself if you can help it

Once again you already said this at the beginning.

even if you have a good infinite-precision math library to fall back on, specifics about seed primes and such end up being important. Libraries like Crypto++ and Botan are already tested and debugged; I would highly recommend using those if you need that particular feature.

Fair.

you can use RSA → XTEA.

I used to be a fan of TEA/XTEA because of the low runtime overhead myself, but it's not actually a part of any real cryptographic standard, which means it doesn't get the level of scrutiny that a standard algorithm gets, and thus there may be weaknesses in the algorithm that you don't know about, but a potential attacker does. XTEA is also a 64 bit block cipher, which is pretty small by today's standards.

Valid criticism for say a F500 company, not sure what you're on about preaching this stuff to indie game developers on an internet forum when fun fact: RSA → XTEA is used by a multi-million dollar game company CipSoft (Revenues in 2018: €14,026,000), a massive 2D MMORPG that's been going good for almost 25 years now that's still using RSA → XTEA to this day. Then again if you're trying to defend yourself against say the NSA or something you might want to drop the XTEA, sure.

Modern CPUs all have AES instructions built in, and they will encrypt data faster than any software implementation of an encryption algorithm. This is one of many benefits of actually sticking to a standard algorithm!

I'm not a cryptography expert (and I'm pretty sure you're not one either considering you're recommending using AES here, a symmetric cryptography suite, for online encryption purposes) so I'm unable to dispute or review these claims as a whole, all I know is when I experimenting with using AES with the most popular Python cryptography library out there a couple of years ago AES was really slow for small chunks data.
Either way AES can't afaik be properly used on its own for online encryption, so the whole criticism/suggestion kinda falls flat on its feet on those grounds.

All of that said, yes, if you have a TLS library and the technical resources to use it available then go for it, otherwise I personally at least would recommend giving RSA a try, it's been the gold standard for online encryption for a long time, and although plain RSA was dropped from the latest version of TLS 1.3 due to a lack of “forward secrecy”, an additional (optional) security layer for encryption in case of key compromise, RSA with a combination of other suites is still available (source).
Some more reading material about RSA in 2019 forward for anyone interested: https://crypto.stackexchange.com/questions/67145/is-rsa-in-decline-across-the-board?noredirect=1

Short addendum: Sorry about all the posts, wasn't aware that the edit functionality wasn't working properly on this website, or if it's just me it's not working on either Firefox or Chrome, so I don't know. Lesson learned… Sort of.. ?

@source61 I'm sorry you feel lectured to; my point was to suggest possible alternatives for the OP in the context of your post.

To address one of the statements that need clarification:

I suggested using AES instead of XTEA, not AES instead of RSA. As you say, the symmetric payload cipher plays a different role than the public-key handshake cipher. (That being said, most modern implementations use ECDSA instead of RSA these days. Properly implemented RSA is still a viable alternative.) The fact that other people use XTEA, especially in an older product, doesn't mean that it's the best choice for someone building a new system today. If you're going to pick some library and algorithm for a new system, it's always best to pick the “recommended” options rather than the “older but still not obsolete options.”

Anyway, my recommendation is: If you can use TLS, use TLS, ideally with a modern library at level 1.3. Ideally using ECDSA and AES. If you need UDP, and can use DTLS, use DTLS. If you want to experiment with the leading edge, give QUIC a look – it may end up being the simplest, safest, most up-to-date option. And prefer to use Botan or Crypto++ over trying to write your own, if you have to make a different trade-off.

In the end, chances are that the success of your game will be much more dependent on whether it's a fun game, than dependent on whether the Mossad can read the chat messages between your players.

hplus0603 said:

@source61 I'm sorry you feel lectured to; my point was to suggest possible alternatives for the OP in the context of your post.

I'm sorry you feel the need to forge a lawyer's apology “on behalf of my feelings” instead of actually sticking to the topic at hand, doesn't exactly take away anything from the experience I've had of you throughout as someone fixated on online upmanship / scoring cheap +1 points by saying literally anything that will make you “look good”.

I suggested using AES instead of XTEA, not AES instead of RSA.

That certainly wasn't clear (or really makes any sense) considering you were laying heavy criticism on RSA without mentioning any other cryptographic “alternative” than AES in your post, but ok.

The fact that other people use XTEA, especially in an older product, doesn't mean that it's the best choice for someone building a new system today. If you're going to pick some library and algorithm for a new system, it's always best to pick the “recommended” options rather than the “older but still not obsolete options.”

What are you on about? Are you just spewing generalities and buzzwords again to try to make yourself look good or is there any evidence to indicate that XTEA is obsolete or outdated that you could share with us?

Anyway, my recommendation is: If you can use TLS, use TLS, ideally with a modern library at level 1.3. Ideally using ECDSA and AES. If you need UDP, and can use DTLS, use DTLS. If you want to experiment with the leading edge, give QUIC a look – it may end up being the simplest, safest, most up-to-date option. And prefer to use Botan or Crypto++ over trying to write your own, if you have to make a different trade-off.

In the end, chances are that the success of your game will be much more dependent on whether it's a fun game, than dependent on whether the Mossad can read the chat messages between your players.

Sure.