When you are using the most common cmdlets, you definitely do not need to load all PowerCLI modules.
Investing a bit of time to find out which modules actually need to be loaded, saves you a lot of time later.
Taking into account the number of modules, and the certificate checking for the signed modules, does require some time I'm afraid.
If you find it "ridiculous" I suggest opening an SR.
There are ways to limit the number of modules depending on the environment you're working in.
Perhaps the PowerCLI Dev Team might investigate and officialise those methods.