You could try running your garry's mod in high priority mode and just don't tab out when you are connecting to the server which might help speed things up, out of curiosity too what hardware are you playing on?
You can change the priority of the gmod executable whilst it is open in your task manager which is temporary until the game is shut down, if you want to keep it on high priority every time then simply add "-high" to the launch options arguments in the game properties on Steam.