Great question! The scenario you're describing with NSX Manager and firewall rules is a common point of confusion. NSX Manager primarily uses segments to apply firewall rules, as these segments map directly to logical networks within NSX.
The issue you're facing, where firewall rules don't apply to VMs that aren't part of an NSX segment, is due to the NSX architecture that requires VMs to be on a segment in order for firewall rules to take effect. When a VM is not part of an NSX segment, it isn't part of the NSX logical network, so the rules can't be applied directly.
To answer your question more directly :-) - if the VMs are not part of an NSX segment, you will need to manually handle the IP or MAC addresses in the firewall rules. While the NSX Manager knows these details, the firewall rules can only be applied to segments, not directly to VMs unless they are part of that segment.