You might worry about malware getting into your virtual machines through other methods (e.g. if you drag-and-drop an infected file, or access a CD with an infected file, etc.), but apart from that, it should be fine. You might consider going into the virtual machine's Settings and removing the network adapter to make sure you don't accidentally enable it.