Not sure whats going on but this is what is happening.
Using the client from server subnet it works fine.
Using the client from the "desktop" subnet, it does not work.
The weird thing is that it works from the "desktop" subnet for the other nodes in the cluster!
There is no firewall in between so that cannot be the problem either.
It can't be a host issue as it works from the same subnet.
When I check esxcfg-firewall -q output I see the following,
Chain INPUT (policy DROP 2588K packets, 323M bytes)
pkts bytes target prot opt in out source destination
2127K 739M ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
512K 199M valid-tcp-flags tcp -- * * 0.0.0.0/0 0.0.0.0/0
517K 199M valid-source-address !udp -- * * 0.0.0.0/0 0.0.0.0/0
5747K 1826M valid-source-address-udp udp -- * * 0.0.0.0/0 0.0.0.0/0
3550 164K valid-source-address tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x16/0x02
23 1806 icmp-in icmp -- * * 0.0.0.0/0 0.0.0.0/0
2520K 1415M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
129 6632 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:902 state NEW
618 29744 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 state NEW
153 7476 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 state NEW
323 106K ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spts:67:68 dpts:67:68
318 29574 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:427
21 1012 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:427 state NEW
95 4468 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:2050:2250 state NEW
55 48620 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpts:2050:2250 state NEW
46 2356 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:8042:8045 state NEW
1140K 283M ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpts:8042:8045 state NEW
92 4507 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5989 state NEW
2256 187K ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:161
31 1640 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 state NEW
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:6390
22 1136 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2301
75 3620 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:280
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:6391
24 1168 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:2381
23 1188 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:6389
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:6392
It does not refer to 903 on any of the hosts. I am no firewall expert so I might be overlooking something but as far as I know ESX always takes care of this and you only have to ensure ports are open in the firewall if there is one in between.
The suggestion you made does work but I am still interested in finding out why the problem occurs in first place.
Cheers