If it is because the people ghosting are server admins, say the helpdesk team that is incharge of deploying and so on and as a help desk person they don't get admin rights to the servers. Consider giving them there own server (doesn't have to be server grade hardware) that is only running ghost.
It isn't a good idea to set up Ghost and any other services any way so moving to a smaller dedicated helpdesk driven box may be a win/win for security and performance. One thing that happens when ghostcast sessions are active is that the network and disk utilization can reach maxed out limits and so any other services that are on that server can suffer and become unavailable or unresponsive until the Ghosting session is done.
I agree also with Randall with ghost you can wipe out a system so if you don't trust a user with admin rights to the ghost server do you trust then not to nuke your entire domain with a disk wipe task?
Cheers.