Okay, this is a pretty bad example if your trying to compare it with a live server...
Almost everything is a ent in some way or another in GMod, the player thats a ENT, every weapon you have on you, those are ents, every NPC, those are ents, every car, hecc that can be more then one ent!
Live servers can have thousands and thousands of entities, that can cause lag, however recreating said environment in a single player game is pretty hard. Point I am making is this, ents can cause lag in massive quantities, even without physics. Fixing physics won't fix ent lag, it will always been a issue on a server like PERP when it hits around 110+ players