- Messages
- 715
- Reaction score
- 5,443
- Points
- 750
- Staff
- #1
@Bolli told me to write an effort post about these updates, so in case you are really interested, here it is. This is somewhat (although not extremely) technical, so it might not be that interesting for normal players to read.
Introduction
FPS on the server has always been one of the major issues of the gamemode. Everyone just accepted that perp FPS is bad and never really looked at how to improve it. A lot of this was due to the fact that perp was not a well coded gamemode (and spoiler, most of the fps issues were in the very old parts of it) and people did not want to look at the utter mess some of the code is in. It is also always a lot more exciting to work on flashy new features than something that is not visible at first glance, especially considering it was never a certainty FPS could be improved in a meaningful capacity.However, recently I have had more time on my hands and FPS was one of the issues raised with us so many times, that we finally got around to looking at it.
Chapter 1: The Preparations
One major issue with improving FPS is to know where to even start. Exactly what part of the gamemode is impacting performance to a significant degree?A big mistake one can make is to go look for every small piece of code that could be optimized to be slightly faster. Most of the time, this is not going to actually help because it misses the bigger picture. If you make a function 2000 times faster when it only takes up 0.1% of the total time a frame takes to render, you will get at most an improvement in FPS of just 0.1%, which is insignificant.
Instead it is much better to improve things that take up a majority of the time rendering each frame, even if just by much smaller percentages, than wasting a lot of time micro-optimizing every possible thing. So how do we find these parts of the gamemode, that we should spend the time on trying to optimise it?
The tool of choice for finding performance bottlenecks in applications are called profilers. They usually hook into functions being called and measure how long each function invocation takes. Then they aggregate the results into some nice graph or table and you can very easily just read off the bottlenecks.
There already is a profiler for gmod that could be used called dbugr. I have evaluated this profiler several times in the past, but it has never given me accurate results. I do not know exactly why this is but it was no help. Another alternative is called FProfiler, but using it will degrade performance to such a point, where the results are simply not useful anymore.
So we need a working profiler that somehow finds bottlenecks without ruining performance by itself. Since this does not exist yet apparently, I made one myself. I could write quite a lot about all the difficulties writing a profiler for gmod, but I would rather focus on the FPS improvements.
Just to quickly summarise, my profiler profiles hooks and network functions and aggregates them nicely. Here is a picture of it in action after the bottlenecks have been fixed:
Chapter 2: FPS Improvements
The fixes for the bottlenecks we found were essentially in two areas. Caching slow functions and restricting entities to only cost performance when they are in your vicinity. The motivation for this approach was essentially that performance was a lot better whenever the demo was restarted.One big improvement that @TinySlayer spearheaded, was caching results ents.FindByClass(). This function is used in multiple rendering hooks to draw outlines or text over some entities. For example the text over money/items and the headlight sprites for vehicles used this.
The issue with the original implementation from valve, is that to find all the vehicles in the game, we do not just immediately get them, but rather it loops through every entity that is loaded and checks if the entity is a car. As we can already see, this means that when you already have a lot of entities loaded, this will have to loop through a lot more entities.
The solution is to simply cache the cars/money/etc. entities we already know in a table. Then finding all the cars that are currently spawned simply involves just looking up the entry in the table, which is blazingly fast.
You can improve this function even more by only returning entities that are actually nearby, because we do not care about entities that are on the other side of the map when we are drawing text over them.
The solution we chose to approach this problem was to simply only have entities in oru table that are within our PVS (essentially the area that we could possibly see from our current position). This means that when you are inside a room or another secluded area, we will only consider entities that are within this very same room. This is why you now have absolutely great performance inside most closed rooms in the game.
The other major issue was that any entity that a client has ever loaded wants to “Think” every frame. This means some function on the entity is called where it can do simulations, such as growing for planter boxes, adjusting card positions for the poker table or similar. This is not acceptable since we do not want to simulate a planter box growing at bazaar, when we are in the suburbs.
The solution here is very simple. As soon as an entity leaves our PVS, we just disable the Think function until it reappears. This is really that simple (although some small things have to be ensured, like stopping sounds so they do not play anymore for these entities). The hard part was actually figuring out how to tell the source engine to do that, which meant I had to dig through a bit of the source code where I found that setting the next “Think” time to -1 stops an entity from thinking and setting it to -1293 (Why????) makes it resume.
There are also a lot of other small things that I have taken care of but they are not nearly as interesting/big as the two examples above.
Chapter 3: Stutter Fixes
Another major annoyance when playing on our server were stutters that happen very frequently. They seemingly happened a lot when moving around the map and especially when no clipping, thus making being an admin a wholly annoying experience.Using the profiler was no help here, as these stutters do not occur every frame and are seemingly somewhat disconnected from hooks. However, I was already suspecting what the cause of the frequent stutters was: Vehicle/Player Materials.
The way we modify vehicles/players to be fully customized per player (especially color in conjunction with metal), rather than just having a preset options players can pick from, is quite involved. It works by essentially creating multiple Materials for every player that contain the base textures and color/metal modifications for example for the skin of a car.
The issue is that the source engine is quite buggy when it comes to setting sub materials and therefore you need to reapply the materials every time a car/player comes into view or the cars will appear white (a bug you have seen a lot in the past).
As I expected this was exactly the cause of the stutters, and reapplying the materials apparently took ~50ms per car. When you move from one area where you cannot see the cars in the parking lot at bazaar, into an area where you can, this could easily cause a stutter of half a second if 10 cars are parked in the area.
What was unexpected is that it is not actually applying the materials, or setting colors on the materials that is taking a lot of time. But rather it is simply getting the list of the original materials a vehicle has that takes a long time (we need this information to know which materials we need to replace).
Apparently whenever you get the list of materials of a model Garry’s Mod reads the entire model from disk then searches for all the materials on disk in all the different folders they could be. Depending on the complexity of the model and the speed of your drive, this can easily take up to 200ms in a bad case.
The fix I applied was to simply cache the materials for a model once they have been looked up, but this still means that the first time they are looked up you will experience some stutters.
I opened up a Garry’s Mod Issue to get this fixed, you can find it here.
Conclusion and Future Work
Performance seems to be a lot better now, but there are still a lot of open questions/issues that I want to address in the next few days:- Why is performance when looking at cars abysmal. Spawning a lot of cars in sandbox does not have nearly as much of a performance impact as it does seem to have on our server. If we manage to fix this, we would probably have decent performance for everyone in basically all situations
- In source, maps are divided into boxes called visleafs. They are used for the client to determine what is rendered (and for the server to know what the client can possibly see in any location). When in a visleaf, you only draw visleafs that are potentially visible from the one you are standing in. Paralake is currently extremely bad when it comes to visleafs. There are a lot of places where you will be in a visleaf that goes all the way up to the skybox (such as here). This means that esentially any visleaf is potentially visible from the one you are standing in and you are more or less rendering any entity within clipping distance. For example, standing in front of the office you will render all the poker tables in Hungriges Schwein, even though there is no real way of being able to see them. Fixing this requires updating the map, which is not something I can just do.
- Items at bazaar are another performance nightmare. People selling tens of boxes of rifle ammo or similar are always going to cost performance, maybe there is a better way to sell things? Also, the shop signs and items draw text, which for whatever reason costs a lot of performance, so maybe some improvements can be made in that regard too.
- Serverside performance has not been touched at all, but I guess at some point maybe we should investigate this too so we could have more players on the server while not having worse FPS for players!
Whenever we fix some of these issues, we will probably write a quick update in this thread.
