If that works via IP then there is no reason why everything else wouldn't.
I suspect it's a general communication error because those commands using fpauth always dumbly fail in the same way as if you entered wrong credentials, even if the problem is at communication layers 2-4.
So can you ping the host using it's (FQDN) hostname from the vMA or nslookup it?
What does this curl yield:
# curl -Ivk https://[FQDN]
What if you register the host using it's IP with vifp addserver?