Right now I have a debug system that allows the player to change it's observed character from the one you're controlling to another player. This moves the camera to their perspective.

Now, if I want to make this a proper feature for players to use I probably want to make that system be server authority by having the server tell the player what character to observe. Seems straight forward.

But that doesn't fix the problem entirely. What's stopping a player from saying “ignore what the server says, spectate this enemy player" through memory manipulation? I guess if a malicious player is in deep enough to know how to force this setting in memory, they can probably do a bunch of nasty stuff like move the camera around the map anyway so maybe worrying about this is moot anyway?

NetworkDev19 said:
But that doesn't fix the problem entirely. What's stopping a player from saying “ignore what the server says, spectate this enemy player

Yeah, like you said, a player can already manipulate things like the view-matrices you upload to the GPU; or dump the GPUs data to view anything he wants anyway. You can make such things harder and try do make some automatic cheat-detection; but in the end all those tools can be circumvented. As long as the game is run on the players PC, the player in the end has full control. So you should rather have a good tool for players reporting cheaters and the capacity to ban malicious players (without just banning tons of innocent players due to unwarranted reports).

The simplest solution is: Do not send the whole map from server to client – only the surroundings of where the player is allowed to look.

If they have no data they can manipulate the local position or camera on client as they want, but will still not see much more.