hosts NTP configuration not working...

[sidewaysclever]

Hi - running essentials plus license with x3 hosts.

noticed the other day that our datarecovery job had not run, looking a bit closer revealed it was a time sync issue between the appliance and the host. I then realised that all three hosts are between 5 & 10 minutes adrift of each other despite all having the same NTP settings defines in the "time configuration"

the ntp settings point to our two normal LAN based NTP servers (these then in turn get their time from a few internet based servers). All vms & workstations are syncing to the same LAN NTP servers fine - but for some reason the hosts are not.

I've refreshed the properties (which includes restarting the NTP service option) - it all looks ok, no errors that I can see, but the times are not being updated. The hosts are losing about a minute every two/three days or so. For the time being I've just manually set the time on each, but I'd like to find out whats stopping them.

the hosts are three identical Dell Poweredge R610 servers.

Any ideas what may be stopping them updating?

cheers.

[marcelo_soares]

Check on the security profile (at configuration tab) if the NTP client service is not blocked on outgoing connections.

Marcelo Soares

VMWare Certified Professional 310/410

Virtualization Tech Master

Globant Argentina

Consider awarding points for "helpful" and/or "correct" answers.

[sidewaysclever]

Hi Marcelo - thanks, but the ntp service is checked & running.

regards.

[Chamon]

What do you see when you log into the ESX host ad do a service ntpd restart ? Can it sync then? Did you set this from the GUI or from the console? Are you using ip address or FQDN? Can the ESX host ping the time servers?

[sidewaysclever]

thanks chamon -

I've been doing all of this so far from via the vsphere client connected to our vcenter

you're reference to ping got me thinking...

so I ssh'd to the console on one of the hosts and tried to vmkping to one of the time servers - nothing.

in fact I can't vmkping anything on our LAN and I can't ping the other SC's on the other hosts. The only thing I can ping from the SC are our x2 ISCSI ports.

Wierd - the infrastructure is all up and running fine - 16 VMs, backups etc...

Looks like outbound traffic from the SC is not getting out - I can ping all three sc from our lan by name and IP, but can't ping anything from the SC. The error is

sendto() failed (No route to host)

[Chamon]

Can you do a normal ping and not vmkping and connect to the time servers? If your vmkernel portgroup is on a segregated LAN for iSCSI then it will not be able to connect to anything outside of there. What network do you manage the esx hosts from? Can you show us your vswitch config? Are you using IP address or FQDN for the time servers? If FQDN then you need either DNS (recomended) or each host needs these entries for each other and the time servers in their host files.

[sidewaysclever]

Hi - i've attached a vswitch config from one of the hosts.

the service console is configured with a LAN network IP - I manage the infrastructure from a workstation on the same network using vsphere client connecting to the vcenter server.

I have tried connecting to the three service consoles directly via web browser and can do so sucessfully

I have tried the ntp servers by FQDN name and ip - no sucess.

I can also migrate VM's between hosts (when VM's switched off as no vmotion license) - which to me, indicates that the hosts can talk to each other even though they can't VMKping each other.

[Chamon]

he Service console and VMKernel will share the same default gateway settings. If when you created the vmkernel vswitches and portgroups you added a default gateway on that network then that would be your problem. Go back to your settings under the configuration tab -> DNS and Routing ->proporties -> Routing and adjust the default gateway to reflect the default gateway of the service console network. Also make sure your DNS settings are correct for the service console network and then give it a try. This is assuming you are not running your iSCSI through a router :smileywink:

[sidewaysclever]

Hi - thanks for that, yes, I think this may be pointing to the problem.

I don't have a default gateway defined for the ISCSI vmkernel.

I did'nt give it one because I figured there was no routing required on a dedicated ISCSI network and I don't have a router on this vmkernel network anyway.

The idea for the seperate network is to isolate ISCSI from our LAN - if i give it the same gateway as the SC, won't the two networks start talking to each other and we'll have ISCSI traffic spilling over into our LAN ?

thanks

[nirvy]

The vmkping command sources ping packets from the local vmkernel port rather than the service console. You should not be testing connectivity to your NTP Servers using vmkping, you should use the normal ping utlility.

Here are a few methods to help troubleshoot NTP:

A poster above asked you to run the command service ntpd restart. This is a pretty good way to see immediately if NTP is working as you should see either:

 ntpd: Synchronizing with time server: [ OK ]

or

ntpd: Synchronizing with time server: [ FAILED ]

This does not however guaruntee your NTP setup is good though, so moving on....

Another way to verify is to use NTP Query. Run ntpq -p on the CLI. This will show you a list of peers known to the server as well as their states. Useful stats from this are the collumns: Remote (shows the addresses of the known NTP servers), poll (shows when the next update will be) and when (a counter that counts up to the value in poll - when they match the update occurs.). If you want to have ntpq refresh, run the command in watch: watch "ntpq -p"

Another thing you can do is use tcpdump to watch for NTP conversations on the wire. Use the syntax: tcpdump -i <vswif> port 123 where <vswif> = the vswif interface for your service console.

An example dump of a working NTP setup is:

# tcpdump -i vswif0 port 123

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on vswif0, link-type EN10MB (Ethernet), capture size 96 bytes

12:54:42.418881 IP myhost.domain.com.ntp > 192.168.100.254.ntp: NTPv4, Client, length 48

12:54:42.418955 IP 192.168.100.254.ntp > myhost.domain.com.ntp: NTPv4, Server, length 48

Hope this helps!