 |
PRooFPS-dd Dev Doc 1.0
PRooFPS-dd Developer Documentation
|
|
PRooFPS-dd Dev Doc 1.0
PRooFPS-dd Developer Documentation
|
PRooFPS-dd uses the networking subsystem of PGE.
This page is basically the continuation of PGE documentation's Networking page.
From v0.2.7, packet rate calculations are in the PRooFPS-dd-Packet-Rates Excel workbook!
There are many ways to cheat in multiplayer games, and PGE doesn't provide protection against it.
However, a good implementation in application level can overcome some forms of cheating.
A common approach is to treat the server as the only authorative instance and make clients show only a replication of server state.
My game PRooFPS-dd has such client-server model implemented in it.
For example: player movement. When a player presses a button to move, it should send a request/command message to the server about the keypress, and the server calculates the actual movement of the player.
Then it replies back to the client(s) with the updated position of the player who requested the move, client(s) receive(s) the reply and move(s) the player to the position calculated by the server.
Since server takes care of the entire game state, simulate physics, calculate new positions, etc. and replicates game state to clients, it is the only authorative element of the multiplayer game session.
This way it is more difficult for clients to do anything from an illegal position, e.g. put themselves out of map bounds intentionally because always the server calculates their position based on client inputs that can be rejected as well.
Another example is how the weapons work: when a player pressen a button to shoot, an attack request is sent to the server, and server decides if the player can actually shoot, and if so, it will create a bullet.
Since the server keeps track of the available and current weapons for each player, there is no client-side cheat that will allow the player to use arbitrary weapon, also there is no use of modifying the weapon files on client-side since server is using only the server-side files.
PGE currently does not give explicit support on features like linearly interpolated player positions, it just gives a basic framework to establish connections between clients and the server as described above.
However, my game PRooFPS-dd implements some of these features so I'm giving some words about this topic here.
Until PRooFPS-dd v0.1.2 Private Beta my naive approach was to tie input sampling to rendering frame rate and send messages between server and clients as soon as input was detected.
As already explained above, in general it is good to select the server as the only authorative instance in the network to provide a basic implementation against cheating.
So in my naive approach, when client player pressed a button to move, it did not move the player object, just sent a message to server about the input.
The server processed the message and replied back as soon as possible.
The updated player positions were in server's response, so on client side I updated the player position upon receiving the response from the server.
This approach looks good if you have high frame rate (e.g. 60 FPS) because client request and server response messages happen very fast, so player movement looks smooth.
However, it has multiple downsides as well:
Although the first issue could be solved by calculating new player position based on measured delta time elapsed since last update instead of using constant values, we would still have the second issue.
Although physics is not part of this page, in general the variable delta time-based physics is not a good approach anyway because of multiple reasons (e.g. different machines with different delta will calculate different floating point results) so I've implemented fixed delta time approach, more you can read about it:
Implementing phyics update with fixed delta time approach helped a lot to introduce the tick-based implementation as described below.
Now back to the networking part.
From now on the techniques I'm describing are very common in multiplayer games and the terms I'm using are same or very similar as in Counter-Strike.
We want framerate-independent player movement, so we have our framerate-independent physics implemented as well, we should tie the input sampling, physics and messaging together, so they will be done with a different rate than the framerate.
These things (input sampling, game state update, simulation, physics, messaging) tied together into a single operation called tick.
The number of a tick is executed per second is called tickrate.
This is the theory you can read everywhere on the internet.
But actually my implementation still does the input sampling part per frame and I'm more carefully explaining this later.
So the essence of moving away from the naive approach is to understand that we execute different part of core game code at different rates:
Another rule is that framerate >= tickrate.
So it is totally ok to have framerate 60 while having tickrate 20. This means that we maximize the number of iterations of main game loop at 60, we target 60 rendered frames every second, and we do only 20 ticks every second.
This also eases the requirement of CPU processing power and network bandwidth.
However, it is not that trivial to say that: okay, from now on I'm just sampling and sending input from client to server at 20 Hz because then the player will really feel the delay in the movement.
So with my implementation the framerate -> tickrate transition mostly helped reducing network traffic in server -> client direction but not in the other direction.
So in the next sections I explain in more detail how I introduced rate-limiting on client- and server-side.
Since physics simulation is done on server-side, introducing tickrate mostly affected the server: using fixed delta approach lead to more reliable player position updates.
Sending new player states to clients is done now at tickrate, which in case of 20 Hz introduced a ~66% reduction in network traffic in server -> client direction compared to having the framerate for this rate as in v0.1.2.
Even though the bullet travel update traffic from server to client direction was also reduced due to the above, I decided to simply stop sending this kind of traffic.
The reason is that even though server simulates bullet travel, clients can also do it on their side. They just simulate the bullet travel, don't care about the hits.
So still server informs the clients about the born and removal of a bullet (e.g. if the bullet hit a player or wall), but between bullet born and removal the clients can move the bullet on their own.
This also greatly reduced server -> client direction traffic.
Some operations became continuous operations: when enabled, server executes the action in every tick until explicitly stopped.
An example for this is player strafe: once player starts strafing, server simulates it at tickrate until player stops strafing.
This way we managed to stop clients from storming the server with inputs such as strafe at their framerate.
It is important to understand that for such actions we should not only send out message when a button is pressed but also when it is released.
With the tickrate introduced, we successfully solved the problem of a slower machine not be able to keep up with processing messages when faster machines are also present in the network.
As I already explained above, with my implementation the framerate -> tickrate transition mostly helped reducing network traffic in server -> client direction but not in the other direction.
To reduce traffic in client -> server direction, one idea was the continuous operation that I already explained above, so I'm not explaining it here again.
This was used also for the attack (left mouse button) action, once the button is pressed, server simulates that, no need to continuously send it by client.
The basic rule doesn't change:
For other actions such as changing weapon with mouse scroll or keyboard, or reloading the weapon, I introduced rate-limiting with a simple delay:
a predefined amount of time MUST elapse before the client can send another same kind of message to the server.
Note that there is no use of storming the server with higher rate in tricky ways because the server also calculates with the minimum delays for rate-limiting thus there is no benefit for the player to storm the server.
For other actions like updating weapon angle (client is moving the crosshair by mouse movement), I had to introduce a more sophisticated rate-limiting method:
Details in pseudocode can be checked later on this page in function handleInputAndSendUserCmdMove().
With low tickrate the physics calculations might not be precise enough. Even with 20 Hz tickrate we saw we could not jump on some boxes or fall in between some boxes.
So I decided to introduce the cl_updaterate CVAR that controls how often server should send the updates to clients.
It is somehow dependent of the tickrate: in every tick we can either send out updates to clients or postpone to another tick.
So if we set high tickrate like 60 Hz, we can have the very precise physics calculations while having cl_updaterate as 20 Hz keeps the required bandwidth low.
Rules:
I also introduced another CVAR called physics_rate_min.
It allows running multiple physics iterations per tick, so if after all you still set a lower tickrate, you can still have more precise physics.
The question in this case: why do we even have tickrate then if physics and server -> client updates can have different rate?
The answer is that not only these are handled in a tick on server-side, but other stuff also like updating player- and map item-respawn timers, and in the future some more additions will be in place too.
So in general it is good if we have the flexibility of fine-tuning these values.
The following pseudocode shows what functions are invoked by runGame() that are relevant from networking perspective.
Server and client instances have the same runGame() code. Some parts are executed only by the server or the client, that is visible from the pseudocode anyway.
Some parts were changed between different versions, in those cases I specified the changes with version numbers.
In the comments I mention what kind of messages are generated with approximated rates total PKT rate and per-client PKT rate.
We are estimating with an intense situation when 8 players are playing the game, and everyone is moving, shooting, and picking up a weapon item at the same time.
I also mention AP (action point) wherever I think change should be introduced.
Rx Packet Rate shows the number of received packets processed per second by server or client.
I also calculate the estimated Rx Packet Data Rate based on the received packet rate and size of packets.
The improvements through versions are very decent and were really needed to solve the packet congestion issue.
In this section we talk about client -> server traffic.
Considering 8 players:
In this section we talk about server -> client traffic.
Considering 8 players, the results are to a single client from the server:
Considering 8 players, the results to ALL clients from the server (because above shows results to 1 client from the server):
just multiply above results by 7 (server sending to itself avoids GNS level thus we multiply by nClientsCount instead of nPlayersCount):
The detailed explanation of the packet rates of each function is below:
Currently server in every tick invokes serverSendUserUpdates() that sends out MsgUserUpdateFromServer to all players about the state of a player if any state is dirty.
Because this is happening frequently, we could use unreliable connection for this instead of reliable to reduce overhead of reliable connection, even though I'm not sure about the amount of the overhead.
Note that reliable and unreliable are 2 different ways of sending messages using GNS, and none of them is using TCP.
Both use UDP.
The difference is that reliable messages are automatically retransmitted in case of loss, can be expected to be received exactly once, and their order is guaranteed to be same on the receiver side as on the sender side.
Some info about the message segment differences here.
Because loss of such message would NOT be a big problem. But it should be done as QuakeWorld/Counter-Strike does: even if player data is NOT dirty, the state is sent out.
This can overcome the inconsistency issues caused by packet loss: if a packet is missing, no problem, the next packet will bring fresher data anyway.
However, at this point I'm still not convinced if I should start experimenting with this though.
At the same time I'm also thinking that there is no packet loss in small LAN environment.
How server code should work without client-side prediction:
Remember that with the naive approach, we immediately processed the messages received from clients.
We don't do this anymore, since server also has tickrate and we should stick to it: game simulation and input sampling is happening in each tick, not in each frame.
So whenever a client input message is received on server-side, instead of processing it immediately, we enqueue it.
Server dequeues all received messages at its next tick, and responses will be also sent out at this time in the separate serverSendUserUpdates().
Remember: the lower the value of cl_updaterate, the more we depend on client-side lerp and input-prediction to smooth out player movement experience.
Interesting fact: the original Doom used P2P lockstep mechanism multiplayer, which at that time was not good to be played over the Internet.
Then Quake introduced the client-server model with the client-side lerp, which was good for LAN, but less good on Internet with bigger distances between machines.
So they introduced client-side prediction in QuakeWorld.
On the internet you can read that: in every tick (instead of every frame), player input is sampled and sent as a message to the server.
I already described this earlier why I think this is not good as it introduces noticable delay:
with framerate 60 versus tickrate 20, a keypress might be sent to server (1000/20) - (1000/60) = ~33 milliseconds later.
And this latency would be on client-side, that would be added to latency between client-server. I think that would be NOT acceptable.
Maybe later I will change my mind.
Another improvement would be: we don't even need to send messages in every tick, we can just further enqueue messages over multiple ticks, and send them at lower rate than tickrate, to further reduce required bandwidth.
This lower rate is called command rate, rule is: tickrate >= command rate.
As optimization, we could send these client messages in 1 single packet to the server, since sending each message in different packet introduces too high overhead.
They say that the "maximum safe UDP payload is 508 bytes".
GNS also uses UDP under the hood.
As of June of 2023 (in PRooFPS-dd v0.1.2 Private Beta), size of PgePacket was 268 Bytes, room for application message (MsgApp struct) was 260 Bytes.
Size of a MsgUserCmdMove struct was 20 Bytes, which means that by implementing placing multiple messages into a single packet we could send more than 10 such messages in a single PgePacket.
Even though I'm not sure what is the size of the whole packet/message sent by GNS, I'm pretty sure it is still below this 508 Bytes when being added to the size of a PgePacket.
Due to the low tickrate (e.g. 50 ms, meaning 20 ticks per second), the player movement can appear choppy and delayed.
This is why we need some tricks here:
With client-side lerp, the last received player coordinate (from server) is cached, the player object is NOT YET positioned to that coordinate.
Instead, the player object is moved between its current position and the cached position using linear interpolation. The object position is updated in every rendered frame.
This way the movement of player object will be continuous even though we receive updated positions less frequently. This removes the choppiness but delay will remain.
Note that we apply this technique for all player objects at client-side, and also at server-side.
We have to be careful though, because this introduces a bit of lag, due to player object position will be always some frames behind the cached server position.
So it is better to do the interpolation fast to keep object position close to the cached position i.e. keep interpolation time cl_interp as a small value.
Note that it might be a good idea to cache not only the latest but the 2 latest positions received from server, and set the lerp time to be as long as twice the delay between 2 updates received from server.
For example, if tickrate is 20 i.e. delay between updates from server is 50 ms, we can set lerp time cl_interp to 2x50ms, so if 1 update is dropped for any reason, the lerp can still continue as it is not yet finished anyway.
TODO: add debug CVAR that can show the cached/server position on client side of objects so we can see the delay compared to server.
I'm expecting the player object to be delayed relative to the debug box with the lerp, but ahead with the client-side prediction!
With client-side prediction, we don't need to use lerp for the current player, because we don't wait for server response for the client's user input.
We keep lerp only for other players' objects.
This is a fundamentally different approach because we discussed earlier that clients always wait the response from the server.
With this approach we move our player object immediately based on local input, and send the usual message to server.
We need to introduce a unique message index in the sent message and the response message as well. This index to be used later when processing response from server. Client also saves the sent messages to a queue along with the calculated player object position, because it will need them later when it receives response from server.
Server will respond back as usual, and upon receiving the new coordinates, we check if the predicted values are correct: the truth is always what server responds.
We can dequeue the stored messages having message index less than or equal to the message index present in server response, and if there is difference in player position in server response compared to the player position in the enqueued message with same index, we align player object position to what server has just responded to us and we replay the remaining stored messages at client-side so that the player object will be correctly positioned based on server's latest confirmed state.
Note that obviously we don't need to send the replayed messages again to server, since those commands were already sent to server earlier, we will get response for them too a bit later from the server.
This way server remains the only authoritive instance in the network, but we let clients see themselves a bit ahead in time compared to the server, and hopefully there will be only rare occasions when we need to correct the predicted positions at client side.
"The client sees the game world in present time, but because of lag, the updates it gets from the server are actually the state of the game in the past. By the time the server sent the updated game state, it hadn’t processed all the commands sent by the client." Note that this approach also means that clients also have to simulate physics, otherwise they cannot properly predict new player positions, e.g. they need to do collision check against walls.
Following CVARs (config variables) are available for tweaking networking in CS 1.6:
Details:
1.12.0