How is the network adapter connected to the switch? Is it via optical modules/cables, or DAC/TwinAx cables? Please remember that passive DAC/TwinAx cables are only supported up to a length of 3m.
How are the switch ports configured? Last time I did s.th. like this I had to set them to fixed speed/duplex (25gbps/full).
On the ESXi host, you may only be able to configure the speed for both ports on one of the ports. Please verify wheter the 25gbps option is available in the drop down menu for the other (likely the first) port.
André