Did you try other codecs, more suitable for lower bitrates?

OGG files was mentioned already, but OGG is just a container. The audio stream would be compressed with specific codec, and Vorbis is typically associated with OGG. It's much better than MP3, but probably not any better than AAC.

But check also if Opus codec is supported by your target platform (still within OGG container). They claim that Opus is much better than the other "general sound compression" competitors at lower bitrates: http://opus-codec.org/comparison/

Great comment thanks for your suggestion! The issue I've run into with OGG is with wide compatibility on mobile and desktop browsers, I was using this as reference https://developer.mozilla.org/en-US/docs/Web/HTML/Supported_media_formats#Browser_compatibility - The current sound implementation process is done using Mozilla webdev to use audio nodes and read from one large sound sheet, in which the code picks out the required sound at different intervals (I don't have much knowledge of this as you can tell)

As it's a html5 game, compatibility is a struggle- and aac in mp4, is the best option I can find for encoding. But I'm lost for a way to improve the quality of mp4s at such a low bit rate. They do sound a lot better than mp3s though.  

Found a great summary for browser specific sound design - 

https://www.gamasutra.com/view/feature/134761/aaalite_audio_design_challenges_.php?page=3

It's a little dated mentioning mp3s over mp4, but had no idea that it's best to start out in as higher sample rate as possible in the DAW, as the codec takes care of bit conversion.