the server would have to use ALOT of Pub Sub signals

Yes. When you have a system where ANYONE can talk to ANYONE, you naturally end up with an N-squared problem, and when N is big, the solution has to be big.

The best thing you can do is to horizontally shard everything, including your Redis instances (if you use Redis for your message bus -- it's fine for light-to-medium duty, but will fall down on heavy systems or when you need guaranteed low latency.)
Split your game into many instances (this often comes naturally.) Split your community into many instances (separate guilds, forums, worlds, topics, etc.)
Then use some kind of service discovery to figure out which Redis to talk to for which channel.

Thus, the group "friends of user X" lives on Redis 3, and the group "game instance 32" lives on Redis 5, or whatever.
You then have to come up with a method for when to create and tear down groups (and how to keep them alive) as well as for mapping group IDs to specific Redis instances.
"Consistent Hashing" is one common solution to the second problem. Timeout when groups are empty is a common solution to the first one.
If you have eight Redis instances, calculate crc32(name) and modulo 8 to get the Redis instance to talk to, for example.

One thing you can do is to only keep pub/sub channels per player, and identify the sender in the message.
Thus, when sending a message to "guild X," you actually iterate over all users that are members of guild X, and send a message on each of those users' queues.
This is great for cases where you may have many topics, but each topic has a finite number of users. (Can you cap the size of guilds to 20, for example?)

Also, there exists nothing pre-built that actually scales to "big" in this space. Google Hangouts does it. Facebook Chat does it. Skype does it. We do it. Each of those implementations is home-grown, and not available for purchase (unless you want to buy a company.)
It turns out that ACTUALLY deploying and working out all the bugs of such a system requires significant effort, at REAL scale (simulation doesn't cut it,) so it's basically impossible to do this as open source.
The closest you can come is ejabberd, which for a very long time actually lost scalability as you added nodes (it didn't federate well) but it might actually be able to scale out these days. Don't know of an instance with more than about 10,000 users, though.

(Perhaps someone will post a link to one!)