the data in JSON object is too big for transmission

Is that actually a fact? Have you measured it? Have you made an estimate of how much you can save? Are those savings worthwhile?
It may be that paying attention to encodings (making key names short, not sending unnecessary fields, etc) will make it as small as it needs to be, and you can then spend your limited time on something else.

If you need to do binary encoding/decoding, then you should be using an ArrayBuffer with a Int8Array view.

I don't know how big it is. But it will be bigger than what I am using now, as the example shows on MessagePack's home page. I used JSON in my first version but as I have iterated several versions from then on all my data is transferred using MessagePack, which has a very easy to use API. Then after I have observed some of the most well known games of similar type I think I better do the binary conversion and do it early. I have used TypedArray in some of my code and the outcome is pretty good. I am just wondering if I need to keep the 3rd party serialization tool or build my own serialization protocol without it.

You need to do more research and get to the point where you understand why some formats are more efficient. Formats like protobuf not sending field names is an obvious one. Varint compression is huge once you understand how it works (you can send X-bit integers using fewer then X bits).

But the big picture is that it's a combination of a number of things that impact network usage and overall performance. Using techniques to simply not send data you don't need to send is just as if not more important then optimizing the data format. Structuring your code to take the best advantage of things like varint compression, makes a huge difference.

For example a trick I use is I never send floats for stuff like position updates. I send integers using varint compression and decide on the highest decimal precision I actually need. I multiply/divide to convert floats to ints and visa versa at that precision. That results in huge savings for the type of data that makes up most of my network traffic.

Currently the best general approach I know of is varient/MSB encoding combined with using integers to represent as much as you can. I've just found it to give the best results over the largest variety of use cases in multiplayer games.

And I also have to factor in integration with other frameworks I might be using. Like I might be using Akka or MS Orleans as my core server framework, and if they natively support protobuf, well that means I don't have to take the GC hit to deserilize my format and then serialize again into theirs. And on the server if you are working with message rates normal to say an mmo or fps game, it's object creation and GC that eventually becomes your bottleneck.

What I always tell people that are relatively new at this is no, don't even think about creating your own format until you first have a solid understanding of how existing formats work and you have gone through creating at least one working game of the specific genre you are tackling. That's the best overall advice I can give.

I ran into a similar problem when I was an indie game developer back in 2012. I found this awesome service called PubNub (disclaimer I now work there), and I created a multiplayer game by sending JSON packets back at forth. It was called Draw & Guess Online and it got tens of thousands of downloads. The amount of money I made from it totally outweighed the cost of PubNub since it's basically free if you are sending messages efficiently. Recently I've been working on a game that showcases how you can efficiently send JSON packets using PubNub that I think you could benefit from since I'm storing all the information you listed above: timestamp, player.x, player.y etc

https://github.com/pubnub/Ninja-Multiplayer-Platformer

Message me at schuetz@pubnub.com if you have any integration problems or want to learn more.

I am just wondering if I need to keep the 3rd party serialization tool or build my own serialization protocol without it.

There's no right or wrong answer. You need to work out what the actual problem is that you need to solve. And since you're not measuring anything yet, you have no data on which to base a choice. Some games work fine with just JSON, some use an off-the-shelf binary protocol, some will pack the bytes directly ourselves, but the choice depends on a lot of factors specific to that game. Just looking at existing games and trying to copy them is not a good approach unless you understand why they made the choices they did. (And throwing in TypedArray when you already have a system for binary serialisation seems like a recipe for bugs, but hey.)