ESXi

Expand all | Collapse all

NTP broken after ESXi 7u3 upgrade

bondebond

bondebondNov 15, 2021 10:16 PM

shashikrishnak

shashikrishnakApr 27, 2022 11:39 PM

  • 1.  NTP broken after ESXi 7u3 upgrade

    Posted Oct 24, 2021 03:34 PM

    NTP time sync appears to have broken after the 7U3 upgrade.  Anyone else run into this and have suggestions on how I might fix?  I've deleted and re-created the service, checked FW policy, tried different servers...  Nothing helps.

    Original Build: VMware ESXi, 7.0.2, 17867351

    actyler1001_1-1635089620144.png

     

    New Build: VMware ESXi, 7.0.3, 18644231

    actyler1001_0-1635089532365.png

     



  • 2.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Oct 24, 2021 07:11 PM

    For us, the warning in the GUI appears to be a false positive and it clears after 10-15 minutes. I tested with changing the host's time to be off, and it adjusted itself to match the NTP source.

     

    Try checking your NTP via the command line. 

    watch "ntpq -p 127.0.0.1"

     

    There are some other options you can use as well.  I believe there is a KB for more troubleshooting.



  • 3.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Oct 24, 2021 10:09 PM

    as far as i have heard its a known issue in 7.0 U3 and a false positive

    you shall see some 2 warnings

    "Host has lost time synchronization" and "Time service is currently not synchronized"

    this should ideally be fixed in an upcoming patch

    P.S 7.0 U2 and earlier dont have any issues, changing the NTP servers wont help, i have tested in my lab

     

     



  • 4.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Oct 24, 2021 10:16 PM

    Great, at least this issue isn't such a big deal.  VMware just keeps churning out one garbage release after another with no regard for QA.  Not impressed.

    Regards,
    Adam Tyler



  • 5.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 08, 2022 08:13 PM

    I regret contradicting an earnest post, but this is not a benign problem. My 2012 Server Essentials (not installed by me, installed before I got here, just sayin.) DC reboots randomly from 6 hours to 6 days due to the NTP issue. I had to define the time locally on the server in powershell. NTD is basically worthless.

     

    On top of that, we got a new release of 7.0.3 that did not fix the issue.

     

    A false positive can be safely ignored. This is not that.



  • 6.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 14, 2022 04:25 PM

    Any updates from any other folks on this?  We receive alerts from time to time about the NTP service failing.  We see the service showing uptime within minutes or hours depending when we check.  Running 7u3c, on these hosts.  We have no reported issues from older 6.5 and 6.7 hosts.



  • 7.  RE: NTP broken after ESXi 7u3 upgrade

    Broadcom Employee
    Posted Feb 14, 2022 10:04 PM

    Hello,

     

    Please open an SR with VMware support with all ESXi host logs of 7.0 U3c where you are seeing NTP related alerts along with screenshots etc., to get it checked.



  • 8.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 20, 2022 08:31 PM

    Buried in the ESXi 7.0 Update-3c Release Notes, FINALLY an acknowledgement of the ongoing NTP issues:

    "NTP optional configurations do not persist on ESXi host reboot.  When you set up optional configurations for NTP by using ESXCLI commands, the settings might not persist after the ESXi host reboots.  This issue is resolved in this release. The fix makes sure that optional configurations are restored into the local cache from ConfigStore during ESXi host bootup."

    **HOWEVER**  After rebooting the Host its working, BUT if you DARE to go into the ESXi host on Web interface directly (not vCenter), and touch or even just verify the NTP settings, and click Save, then..... Kaboom! 

    Issue #1 of 2:

    It STOPS the NTP service, you CANNOT click on the Actions to Restart NTP, and basically... you must Reboot the Host to get NTP back again. 

    EVEN STRANGER, if you change the settings for NTP (again, on the ESXi web interface directly), it will REVERT to local host clock and LOOK like it FAILED to Save NTP settings, BUT... then if you reboot, the changes ARE indeed read upon Host Reboot and the NTP service WILL start using the settings that LOOKED like they failed to Save in the GUI (meaning the GUI will show it correctly), but cannot be read until Reboot is done for ESXi host.

    AND AGAIN, then, if you DARE touch the NTP settings, it will AGAIN revert to local host clock, STOP the NTP service, and stay that way in the GUI until you reboot the ESXi host. 

    Issue #2 of 2 (exasperates issue#1):

    On the ESXi web interface for Host, navigate to Manage > System > Time & date > Actions button.  It does NOTHING!  Tried in Chrome and Firefox (the "latest" versions as of 2/20/2022).

    This compounds the issue #1 above, because you cannot Stop or Start the NTP service from this Web GUI at all without being able to click that Actions button that USED to work great.

    To reiterate Summary, as I am in total disbelief:

    ESXi 7.03c Generic host from VMware direct ISO download only, Fully supported hardware auto-recognized all hardware in Host, there is no vCenter so thats eliminated as a reason, all to prove this is an Out-Of-The-Box ONGOING BUG with NTP on ESXi 7.03c...  No driver fiddling or other VIB updates etc. Just TRY to setup NTP with local ESXi host only (no vCenter), the only 1-change was the HostName in DCUI upon installed first-reboot, and it STILL has this same NTP problem!

    In short, I can't believe it....

    I've never seen such a critical service as NTP be so broken in vSphere for so long, even after VMware THOUGHT they "fixed" it in the release notes for 7.0 Update-3c... But Nope!

    I have a wild-guess theory: 

    Starting with ESXi 6.7 and additionally 7.0u3 adding more new features in general to 7.0u2, VMware is compressing and encrypting startup files based on TPM 2.0 (and if does not exist generates similar in a locally encrypted file).  I am taking a wild guess there are some low-level scripts core to the NTP service, and potentially other services, that have not yet been 100% patched or rewritten to support this heightened security with local esxi hosts using crypto plus the New Features added in specifically the ESXi 7.0 Update-3.....remaining unfixed for Update 3a, Update 3b, and tried to fix but still partially broken in 7.0 Update 3c....

    And remember: Clean ESXi 7.0u3c Install, No VIB updates, No vCenter complexities; Still NTP issues!

    Thoughts on this?



  • 9.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 20, 2022 09:21 PM

      What do you mean by 7.0 U3c NTP related alerts? NTP is simply just still broken as shown by trying to use the ESXi 7.0 u3c web GUI; please read my post on the Reproducible ESXi 7.0 u3c behavior of NTP that we are complaining about still being broken in-detail. Thank you!



  • 10.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 21, 2022 10:19 AM

    Hi,

     

    Just deployed an ESXi host versione 7.0U3c but I was able to configure NTP via WEB interface, changed IT multiple way and it persisted on reboot every time, but only when in the "service tab" ntpd was set to "start" (It take some time to "react"), by default is "stop".

    Regards.

       

     



  • 11.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 25, 2022 09:05 PM

    1. At the Manage > System > Time & date > Actions button, does that button work for you?

    It does nothing when I click it trying Google Chrome and Mozilla Firefox.

    2. If you change from System Clock to NTP, then from NTP to System Clock, and finally one last time System Clock back to NTP, does it Save and Work for you?

    It does not Save my changes to use NTP, it reverts every time back to System Clock for Host.

    Note:  I'm using a SuperMicro server on the VMware hardware compatibility list supported and listed by vSphere 7, no OEM VIB's required.

    Thoughts?



  • 12.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 25, 2022 11:33 PM

     

    Well, the "action buttons" in fact apparently does nothing with ESXi 7.0U3c and below, I don't remember exactly the release when that button "started" to be "useless" because I don't like to bother around something that does not work as intended / expected. 

     

    The annoying part is that when NTP had been configured and set to start / stop with the host but, if in the "service tab" the state of the NTP client remain as "stopped" (the default) that status persist to subsequent reboot (NTP service does not start with the host) till set to start (then I can change the NTP setting at runtime without issues).

     

    In my "lab" the main system are a pair of DELL R730 set up to be a "replica" of a small, but real, production environment.

     

     

     

     

     



  • 13.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 26, 2022 05:01 PM

     

    Thanks for the detailed clarification on the annoying NTP behavior you observed.

    Just like with other OS's, there are always multiple ways in different ESXi interfaces to use certain feature(s).

    So based on your reply, it sounds like you've validated the complaints being discussed in this form, and you simply use vSphere in a different manner/interface than I detailed - but have the same NTP issues/bug as those being discussed (in general).

    In other words, hopefully these posts of the different ways to observe and prove these bug(s) with NTP on ESXi 7.0u3c, so they can finally be fixed by VMware once and for all - Sooner vs later!

     



  • 14.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 26, 2022 06:44 PM

     

    Well,

     

    IMHO the NTP service itself is not really broken but the way the managment tools intreract with it is debatable.

     

    I was bitten by this (and other related issues in the middle):

    NTP service auto start is not working in ESXi 7.0 (80189) (vmware.com)

    The hostd service in ESXi 7.0U3 crashes due to memory corruption (86283) (vmware.com)

    About the behaviour of the "action button" an article found on the web (a blog) reported what you noted dated April 2020. As we well known all build of ESXi 7.0U3 before the current one was retired.

    Of course if an NTP time source is not as reliable as it should be, then is another kind of history.

    Regards,

    Ferdinando

     

     



  • 15.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 28, 2022 01:11 PM

    For what its worth...we worked with both VMWare and with Nutanix on the issue i mentioned earlier.  Both have known issues with NTP, they may or may not be related.  The main issue we were able to identify on the VMWare side was that most of the hosts are having issues with the ntp service restarting and we didn't find a solution or even really the cause, but they're aware of the issue.  On the Nutanix side they have a conf file which after 7.u3c has been made read only (for most nutanix users this won't make much sense, but this doesn't seem to be an issue in earlier esxi versions 6.x), and even if i chmod it to rw, it changes back at 3am on most* of the cvms.  Both companies say they're working on the issues and were aware of them before i contacted them.



  • 16.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 28, 2022 05:50 PM

    Unreal to me that vSphere 7 isn't a stable product yet. vSphere 6.x goes EOL by the end of the year too.  I've been waiting for the bugs to get worked out of 7 before upgrade, doesn't look like that is ever going to happen.



  • 17.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 28, 2022 06:37 PM

    Don't move until you have:

    a) a healthy lab environment in which you have proved everything out, including simulating your network, iscsi, autostart sequence, etc

    b) no other choice

     

    In my case, I built a new vmware environment.

    I used the trial license to build a simple 3 host vcenter essentials managed domain including a new 2019 DC, then purchased and applied a permanent VMWare Essentials License. I carefully created copies of 2 key production servers (one my PDC, a 2012 Essentials server, long story, don't ask) using vmware converter, and only put them in production after proper testing. I am not linking it here because it is dangerous, no longer supported, and I can't be held responsible for your results. It is also the only way I have ever successfully converted a live raid 5 array into a working VM. It can be done other ways, by others more skilled than I.

    With 3 VMs in production, all on ESXi 7.0.3u, two identical Dell Poweredge R340's.

    3 relatively small servers on 2 relatively beefy platforms. It ran fine for weeks of testing, and almost a month live in production.

    And then my pdc shut itself down without warning. Boot it back up, check logs, the time suddenly differed by 5 hours from the other 2 servers. Check it out, the pdc was on a host that lost time. I lost hours that day.. I thought I had fixed it. Frankly, it was a lot of starting and stopping the ntp service using different means. Inside the web client (which I hate and wish very much was replaced by a working, locally installed client like it should be, but whatever) using ssh, bashing the service in cli, I couldn't tell if it was actually running or not because vcenter presented an error indicating that the service was not running. I didn't take a screenshot at the time, but if you are reading this you probably don't need one. This started at 10am during production. All of my CAD Designers (and I am one of them when things are working properly) got separated from their data and had to work offline without the vault.

    There is more to the story, but this is already a wall of text. I will add that I did not pay for support, and as such am not allowed by the web portal to create a support ticket. Therefore it is not helpful for a vmware support technician, who means well, to suggest that I do that. Also, I have read thousands of words related to this issue, and have gone as far as to re-create my environment using the Dell specific image for my Servers. It is somehow related to MS implementation of TPM2, in a way no one is really talking about. I smell a zero-day here somewhere.

    It is still there. NTP will drop, your host will report UTC, and down goes your domain. Just set the NTP server on your PDC to point to your router, or something else that runs firmware inside your domain, and move on to another problem. I'm sure it will be fixed by the time 8 is released.



  • 18.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 28, 2022 07:37 PM

    I've done some more digging on the "how did this happen" question, and I have to wonder:

    It seems vSphere 7.0's Update-3 added many new features; Making it a LOT more like a Feature Upgrade than a Maintenance Update - and this entire thread is based on NTP breaking starting at U3 (along with other issues introduced).

    This explains to me and my own way of thinking as to why all of 7.0's U3, U3a, U3b, were all pulled upon discovery of its critical core flaw, and then made available again in 7.0 U3c...

    vSphere 7.0's initial release at least had the basics like NTP working - It's those new U3 features injecting critical unforeseen issues like fundamental NTP.

    So, I'll just speculate that an Update in general does not have the same level of regression testing that a full all-new release like 7.0's launch would be as heavily debugged, then beta tested, and scrutinized as a preview release prior to production recommendations. In addition, considering many customers were waiting for the first few Updates to arrive to trust 7.0 in production thinking it was only a Maintenance Update but in reality somewhat a Feature Upgrade this unfortunately has bitten everyone including VMware.

    Looking through the release notes of the latest prior vSphere 7.0 Update-2e (released Feb 15th 2022), it does not have any mention whatsoever about NTP.. because as mentioned at the very top of this thread - NTP was never broken until U3, U3a, U3b, and attempted fix but still quasi-broken in U3c's release!

    Thus IMHO it seems to maintain credibility, VMware has a few choices prior to October 15th 2022:

    1.) Fix all fundamental issues like NTP for vSphere 7.0 Update/Upgrade-3 as an expedited priority.

    2.) Extend vSphere 6.7's EOL/EOS date AGAIN  (like they already did on June 3rd 2020 for 11 months, now supported up to Oct 15th 2022).

    3.) Post an advisory that customers using vSphere for critical infrastructure may want to consider the latest build of 7.0 U2 until said U3 issues are all 100% confirmed resolved OR 7.1 is launched with serious assurances.

    Just my $0.02 speculation after all this wasted time on 7.0u3 reading KB's and Release Notes....



  • 19.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 28, 2022 08:17 PM

     

    I agree with your fundamental points.

    I would add, the user base of VMWare is constantly changing. A certain percentage of those users are power users that are lab testing every release to death. Another portion of those users are like me, with some level of mastery below expert, but aware enough of the fundamentals to make use of the free/nearly free platform. If this latter group does not do their research, they may assume that the latest release is the way to go, especially if the release ID is beyond x.0. I looked at 7.0.3 and thought, surely they have the major bugs worked out by now, and I was wrong.

    I don't want to sound like I am beating up on the developer of a, by and large, superb product that is made available essentially for free. At the same time, if you find out that the backend time-keeper of a hardware emulator intended to support the bulk of business operations for millions of users around the world isn't working, maybe don't bury the lead. Tell people at the point of download, before implementation, that they need an alternative ntp for the foreseeable future. Now we are in a situation where 6 is at EOL, we can't trust 7, and are looking sideways at 8. If there were a clear alternative (don't say Hypervisor) I would take it.

    Again, I am a grifter here. I don't manage 2000 hosts with HA and vSAN. I paid <$1k. But if this is happening to me, it must be affecting the big guys too.



  • 20.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 28, 2022 08:25 PM

    Well said, both  and  ..



  • 21.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Mar 01, 2022 12:09 AM

    I'm a product manager in the vSphere team and monitoring this thread for a while. Unfortunately, we are unable to reproduce some of the issues you have reported internally. Would you be willing to share some of these issues and support bundles from your environment so that we can investigate this further?



  • 22.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Mar 01, 2022 01:05 PM

    This is an excerpt from one of my emails with support:

    Although, this has been one of the known ones with 7.0 U3, you can definitely apply the workaround steps as shown in the article: https://kb.vmware.com/s/article/86255?lang=en_US

     

    Though i'm not certain my issue is the same as what has been described in some of the above posts.

     

    VMware ESXi, 7.0.3, 19193900



  • 23.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Oct 31, 2021 02:02 AM

    Just applied build VMware ESXi, 7.0.3, 18825058, NTP still broken.  Unbelievable.

    -Adam



  • 24.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 05, 2021 01:42 PM

    On my system, it appears to be an issue with using aggregated uplinks.  The diagnostic output from "Test Services":

    Service analysis started on host: esxi.domain.com
    Test started at: 2021-11-05T13:36:14Z
    Time Service is administratively enabled.
    Verifying NTP service.
    NTP server: 172.16.4.11 resolves IPv4: XXX.XXX.XXX.XXX
    Virtual NIC vmk0 : Admin: Up
    IP Interface: vmk0 IPv4 Address: STATIC XXX.XXX.XXX.XXX
    IP Interface: vmk0 connected to 43 on distributed vswitch
    Virtual Switch distributed vswitch needs at least one configured uplink
    IP Network Stack: defaultTcpipStack
    No physical NICs are connected to vmk interface
    Firewall Rule: ntpClient allows traffic on port: 123
    ntpd is running, PID: 2098587
    Kernel clock type: ntp
    NTP is in sync
    Peering with: XXX.XXX.XXX.XXX
    Statum: 2
    Accuracy to within: 17.196500 msecs
    Polling every: 8 secs
    Network delay round trip: 0.239000 msecs
    Difference from remote clock: -0.095681 msecs
    Service analysis completed.

     

    So even though the test has the alert of "Configuration is not working normally. " at the top, NTP still is sync'd to within 17 msec. Obviously it has an uplink somewhere - just using LACP uplinks to two different switches for redundancy....

    I hope they fix this soon...



  • 25.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 03, 2021 02:58 PM

    i REALLY get sick auf VMware!

    the ntp servers are two guests with vmware-tools sync disabled and all other guests sync with the host, restarted ntpd on both timeservers, forced ntpdate and restarted the ntp service on both hosts - guess which one is 7.0.3....... damned can VMware stop messing around with public cloud and what not instead doing their basics homework?

    [root@esx1:~] ntpq -p 127.0.0.1
        remote           refid      st t when poll reach   delay   offset  jitter
    ==============================================================================
    +buildserver.the 116.203.119.16   3 u   41   64  377    0.144   +3.152   0.737
    *ntp2.thelounge. 217.196.145.42   2 u   50   64  377    0.118   +4.310   0.870


    [root@esx2:~] ntpq -p 127.0.0.1
        remote           refid      st t when poll reach   delay   offset  jitter
    ==============================================================================
    buildserver.the 116.203.119.16   3 u    3   64  177    0.113  +1339.8   0.037
    ntp2.thelounge. 217.196.145.42   2 u   56   64  177    0.078  +1341.2   0.035



  • 26.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Apr 11, 2023 07:22 PM

    Give us bad updates to force us to the cloud where we have to trust them more with more to lose.  I totally agree.  If you can't get step 1 right, don't bother with step 2.  Greedy for the cloud money where we were promised that somehow introducing a middle man saves us money.  That having your data in someone else's hands is more secure.  That adding an internet connection to your possible points of failure makes it more robust.    



  • 27.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Apr 15, 2023 07:20 PM

    A few pages back, my NTP solution continues to work for those who've spent the time to analyze, read, and try it:

    https://communities.vmware.com/t5/ESXi-Discussions/NTP-broken-after-ESXi-7u3-upgrade/td-p/2874086/page/6

    For smaller deployments, NTP works fine in ESXi 7.0.3, you do not "need" external NTP within your local network.

    But the config within VCSA does seem to get corrupt in certain upgrade or patch scenarios. It seems all the latest ESXI/VCSA patches for 7.0.3 (aka Update3), help to resolve this issue but the commands I posted are still required to return to a normal non-error state (sometimes, not always, depends on deployment).

     

     



  • 28.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 05, 2021 08:16 AM

    Hi!

    Someone has any update in this issue? Is there a patch or workaround?

    Thank you!!



  • 29.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 05, 2021 09:57 AM

    after restart and resync all ntpd servers and a lot of time it seems at least to stop sending alerts - however, it#s a bad joke after 12 years that each and every update is a tradeoff between solving problems and bring a ton of new ones



  • 30.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 22, 2022 01:53 PM

    Still an issue as of 6/22/22!



  • 31.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 23, 2022 09:52 PM

    Yep, same here 06/23/22



  • 32.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 06, 2021 06:32 PM
    1. On vSphere Client, go to Configure -> System -> Time Configuration tab, select "Network Time Protocol" and click on EDIT button. 
       

       

    2. From the configuration box, uncheck "Enable monitoring events" and click on OK button.            
       

       

      this is the workaround i'm aware of and its supposed to be getting fixed in the upcoming release of vsphere
     

     



  • 33.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 07, 2021 08:22 AM

    They might fix it in the next release.  Wonder what new bugs they'll introduce?  Hey guys, its VMware here, we've fixed the NTP problem, but have to apologize that all hosts with USB ports PSOD randomly.  We'll fix that in a few months and introduce something else more awful.  At least you are enjoying the rollercoaster right?

    fed up.  Time to look at alternate products.



  • 34.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 14, 2022 04:58 PM

    This worked for me, in that the time has now synchronised  but the results of the test still say failed. At least the time is correct now and will allow for me to progress config!



  • 35.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 09, 2021 10:10 AM

    Same problem here, after some troubleshooting, we are facing same problem

    Hopefully, it will be fixed in the next release. 

     



  • 36.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 15, 2021 02:26 AM

    I too ran into this upon upgrading hosts to 7U3a. I wasn't able to immediately fix it and didn't bother with a reboot because the host clearly had perfect time and was indeed sync'd with the NTP server, as much as that warning wanted be to believe.

     

    That was a couple days ago. I went back just now and ran "test services" for NTP, and it resolved itself and showed NTP as synchronized. So I'm not sure how long one would have to wait after updating to this release to run the test services function, but nevertheless, the error is gone now after doing nothing special to the host except that test function.



  • 37.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 15, 2021 08:10 PM

    It is STILL broken in 7u3b build # 18905247



  • 38.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 15, 2021 10:16 PM

    Yes, it is still broken in 18905247



  • 39.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 15, 2021 10:39 PM

    Confirmed, I've also applied 18905247 and my hosts are reporting NTP is still broke.  But I am sure that some kubernetes feature that none of us use or care about got fixed.



  • 40.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 19, 2021 01:12 PM

    The same on 18644231. Does anyone know how to treat?



  • 41.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 19, 2021 04:39 PM

    Bro, at this point VMware doesn't know how to treat.  They're just churning out terrible software right now.  You have to choose between running stable, but vulnerable older builds or have stuff just not work properly.  It's unfortunate, VMware used to be a decent software company.



  • 42.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 19, 2021 04:47 PM

    "VMware used to be a decent software company" - Yeah, *before* Dell put their dirty fingers on it - for the sake of god now that VMware becomes a own company things will get better before it hurts too much any i take money and time in my hands migrationg to KVM/Proxmox

    normally you shouldn't recognize that there are hypervisors in the game



  • 43.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Nov 19, 2021 06:58 PM

    This just in, VMware has officially pulled the following builds.  Unreal!

    https://kb.vmware.com/s/article/86398

    • vSphere ESXi 7.0 Update 3    (build 18644231)
    • vSphere ESXi 7.0 Update 3a  (build 18825058)
    • vSphere ESXi 7.0 Update 3b  (build 18905247)
    • vSphere vCenter 7.0 Update 3b (build 18901211)


  • 44.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 03, 2021 02:15 PM

    I just finished converting 2 ancient whiteboxes to VMs and moved our whole domain onto VMWare. I "assumed" the best course of action would be to install the latest release of ESXi. It took me 2 weeks of banging my head against the wall trying to install vsphere, only to discover that NTP was the problem. Manually configure the time and volia, it works. How can I trust our entire infrastructure to a dev team that can't set the time on the d@#$ VCR?

     

    I would have been better off installing 6 and waiting for 7 to be stable. That would be my advice to anyone performing a new deployment today. Install the last stable release of 6 or 7.0 and wait for the rest of us to muddle through until 7.4.

     



  • 45.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 03, 2021 02:27 PM

    "How can I trust our entire infrastructure"

    Exactly!  VMware is proving over and over again they are not worthy of this trust.  I really hope that this turns around, but I've been disgusted at their development performance with the vSphere 7 product.



  • 46.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Feb 01, 2022 09:58 AM

    Still a problem in vSphere ESXi 7.0 Update 3c  (build-19193900)

    [root@pmi-esx01:~] cat /var/log/vmkernel.log  | grep NTPClock
    2022-01-31T04:23:01.998Z cpu25:2099355)WARNING: NTPClock: 1712: system clock synchronized to upstream time servers
    2022-01-31T06:33:44.006Z cpu0:2098149)WARNING: NTPClock: 644: system clock apparently no longer synchronized to upstream time servers
    2022-01-31T08:12:15.998Z cpu31:2099355)WARNING: NTPClock: 1712: system clock synchronized to upstream time servers
    0:00:00:05.721 cpu0:2097152)Initializing InitVMKernel: (131/186) NTPClock_Init ...
    2022-01-31T13:28:14.000Z cpu0:2097152)SysInitTable: 112: Finished sysInit step: NTPClock_Init in 1411 us.
    2022-01-31T13:28:42.091Z cpu11:2098486)WARNING: NTPClock: 1764: system clock synchronized to upstream time servers
    2022-01-31T13:41:50.997Z cpu21:2098486)WARNING: NTPClock: 1764: system clock synchronized to upstream time servers
    2022-01-31T14:26:39.998Z cpu20:2098486)WARNING: NTPClock: 1457: system clock stepped to 1643639201.001221000, no longer synchronized to upstream time servers
    2022-01-31T14:26:41.001Z cpu20:2098486)WARNING: NTPClock: 1764: system clock synchronized to upstream time servers
    2022-01-31T14:59:56.776Z cpu6:2109353)WARNING: NTPClock: 1457: system clock stepped to 1643641196.000498000, no longer synchronized to upstream time servers
    2022-01-31T15:12:00.996Z cpu20:2109371)WARNING: NTPClock: 1764: system clock synchronized to upstream time servers
    2022-01-31T19:14:14.003Z cpu0:2097416)WARNING: NTPClock: 680: system clock apparently no longer synchronized to upstream time servers
    2022-01-31T19:42:36.000Z cpu22:2109371)WARNING: NTPClock: 1764: system clock synchronized to upstream time servers
    2022-01-31T20:29:23.003Z cpu35:2109371)WARNING: NTPClock: 1457: system clock stepped to 1643660962.001395000, no longer synchronized to upstream time servers
    2022-01-31T20:29:22.002Z cpu35:2109371)WARNING: NTPClock: 1764: system clock synchronized to upstream time servers
    2022-01-31T23:06:47.006Z cpu0:2098007)WARNING: NTPClock: 680: system clock apparently no longer synchronized to upstream time servers
    2022-02-01T02:05:56.989Z cpu24:2109371)WARNING: NTPClock: 1764: system clock synchronized to upstream time servers
    2022-02-01T03:05:58.009Z cpu0:2104433)WARNING: NTPClock: 680: system clock apparently no longer synchronized to upstream time servers
    2022-02-01T04:43:46.001Z cpu20:2109371)WARNING: NTPClock: 1764: system clock synchronized to upstream time servers
    2022-02-01T05:43:47.006Z cpu0:2097319)WARNING: NTPClock: 680: system clock apparently no longer synchronized to upstream time servers
    2022-02-01T08:14:52.999Z cpu20:2109371)WARNING: NTPClock: 1764: system clock synchronized to upstream time servers
    2022-02-01T09:15:15.872Z cpu37:2126587)WARNING: NTPClock: 1457: system clock stepped to 1643706915.000534000, no longer synchronized to upstream time servers
    2022-02-01T09:29:46.001Z cpu2:2126606)WARNING: NTPClock: 1764: system clock synchronized to upstream time servers
    
    [root@pmi-esx01:~] ntpq -pn
         remote           refid      st t when poll reach   delay   offset  jitter
    ==============================================================================
     10.XXX.0.1      .LOCL.           1 u   15   64    1    0.959  -85.494   0.000
     10.XXX.0.11     .LOCL.           1 u   14   64    1    1.578  -85.761   0.000
     
    

     



  • 47.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Apr 27, 2022 11:39 PM

    Any workaround? 



  • 48.  RE: NTP broken after ESXi 7u3 upgrade

    Posted May 08, 2022 07:43 PM

     

    VMware has now implemented another partial-fix for NTP directly into the new ESXi 7.0 Update-3d.

    Short snip from 7.0u3d release notes:

    • PR 2875575: After upgrading to ESXi 7.0 Update 2d and later, you see an NTP time sync error

      In some environments, after upgrading to ESXi 7.0 Update 2d and later, in the vSphere Client you might see the error Host has lost time synchronization. However, the alarm might not indicate an actual issue.

      This issue is resolved in this release. [70u3d]  The fix replaces the error message with a log function for backtracing and prevents false alarms. 

    Unfortunately, this does NOT fix my issues described earlier in the thread from other NTP issues still persisting, specifically after CLEAN-INSTALL on SuperMicro SuperServer mini-ATX with Xeon D-Series (we use many in clusters for special-use tasks).



  • 49.  RE: NTP broken after ESXi 7u3 upgrade

    Posted May 09, 2022 03:09 PM

     


    My ESXi hosts are in Hypervisor:VMware ESXi, 7.0.3, 19482537 (ESXi 7.0 Update-3d) but still I am seeing this NTP alert on all ESXi hosts.



  • 50.  RE: NTP broken after ESXi 7u3 upgrade

    Posted May 09, 2022 05:54 PM

      - It is now reproduced by multiple people, that NTP issues persist and still unresolved with the latest ESXi 7.03d build patch.

    Please relay that to your engineering team.

    Also, if you want someone to upload a "vm-support" bundle, then you need to open a courtesy SR# and provide secure instructions to upload the logs dumps to that SR# for you to securely receive it. These forums do not permit file uploads of that size nor are the uploads secure.
     


  • 51.  RE: NTP broken after ESXi 7u3 upgrade

    Posted May 10, 2022 05:59 PM

    I see that too, but we are not seeing such issues being reported in our official support channel. Unless someone raises and SR and uploads logs for us to investigate and provides me the SR number, I cannot help.

    I cannot create SRs because they are supposed to be unique to our customers. Multiple SRs can help raise the visibility, so the best thing you can do is to raise SR and upload logs.



  • 52.  RE: NTP broken after ESXi 7u3 upgrade

    Posted May 19, 2022 09:49 PM

     wrote:

    I see that too, but we are not seeing such issues being reported in our official support channel. Unless someone raises and SR and uploads logs for us to investigate and provides me the SR number, I cannot help.

    I cannot create SRs because they are supposed to be unique to our customers. Multiple SRs can help raise the visibility, so the best thing you can do is to raise SR and upload logs.


    ,

    Thanks for replying, and I'm very glad you are able to reproduce the issue!!

    Unfortunately, I'm very surprised you are unable to open your own internal SR for the issue you reproduced.

    Please private message me a courtesy or internal SR# and method to upload logs to that SR#, and I will gladly do so!

    I cannot justify paying for burning the cost of an SR for a known-issue that as you said, even though you reproduced it will likely get no attention without multiple customers opening multiple SR's for escalation to engineering for a patch. For such a critical service as NTP, this is not a good answer from VMware.

    Regardless, please provide any no-cost SR method so I can assist, and I will.

    Thanks!



  • 53.  RE: NTP broken after ESXi 7u3 upgrade

    Posted May 20, 2022 10:41 AM

    , good morning,


    I beg your pardon but I have to agree with Labmasterbeta (and many other).


    I have now updated the vCenter product to the current 7.0U3e and ESXi to the current ESXi 7.0U3D.
    Nonetheless, as far as I have taken the trouble to verify, all the defects in the user interface or the inaccuracy of some of the information displayed are as they were before.
    I mean, if I have doubts about the correct functioning of the NTP service I use the command line, exactly as I did before.


    Let me tell you that personally I don't understand why to keep objects in the context of a user interface that have not worked for a long time, like the "famous" action button.

    I am not here (and I do not want) to argue but, I should rise a support request to report that the power consumption indicated by the monitoring of a virtual machine cannot amount "some" thousand kilowatts? Honestly, I don't think so.

    From my humble perspective, many don't bother anymore asking for technical support for long-standing (and maybe obvious) "unresolved product problems and defects", in the end they manage in another way or ignore them.


    Well, this does not mean that I will no longer use VMware products (I don't even think about it) but certainly now I find myself being more cautious and selective than before.


    Regards,
    Ferdinando



  • 54.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 14, 2022 07:55 AM

    same here, a stretched vSAN environment running on build 19482537 (DellEMC VxRail Image). 

    Hope there will be a final fix for that.



  • 55.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 15, 2022 07:17 PM

    I am seeing similar issues.  My versions are below.  Last week I had one host lost its management interface and after a while it got reconnected back again.  This a new build environment but we are close to moving to production.  Right now we are on 6.5.  I am not confident with this version as we have a new CIO and don't want to get yelled at LOL!!

     

    I am getting this message but the time is correct most of the time with the hosts. A few hosts would randomly be off which is scary.

    Time service is currently not synchronized.

    VMware vCenter Server
    Version:
    7.0.3.00600
    Build number:
    19717403


    ESXi version:7.0.3
    ESXi build number:
    19482537



  • 56.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 15, 2022 07:40 PM

    VMware ESXi, 7.0.3, 19898904

    Same issue on the latest version. Below is the workaround given for my case, but the issue comes back after a reboot.

    1. On vSphere Client, go to Configure -> System -> Time Configuration tab, select "Network Time Protocol" and click on EDIT button
    2. From the configuration box, uncheck "Enable monitoring events"
    3. Click the OK button

     



  • 57.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 15, 2022 07:53 PM

    TPGOPI007,

     

    If you did the work around (Unchecking "Enable monitoring events"), any events will not be logged?



  • 58.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 15, 2022 08:51 PM

    Not logged until the next reboot



  • 59.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 15, 2022 09:14 PM

    Thanks!



  • 60.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 17, 2022 09:59 AM

    VMware has released new ESXi Version. 

    ESXi 7.0 Update 3eESXi_7.0.3-0.40.198989042022-06-1419898904

     

    Someone checked if Issue solved finally or not?  



  • 61.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 17, 2022 01:24 PM

    The NTP issue still persist.


    vCenter: 7.0.3 Build: 19717403

    VMware ESXi, 7.0.3, 19898904



  • 62.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 17, 2022 02:32 PM

    Me too.  But I am running...

    VMware ESXi, 7.0.3, 19482537

    GUI reports "Time service is currently not synchronized".



  • 63.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 17, 2022 08:43 PM

    Why I am not surprised? 

    I assume it will be solved already in ESXi 7.5. 



  • 64.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 17, 2022 10:14 PM

    You didn't hear the news?  Broadcom bought VMware.... Maybe things will now improve.



  • 65.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 17, 2022 11:51 PM

    how old are you? 15? or why do you imagine things will become better after anotherbpure hardware crap company bought VMware?



  • 66.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 15, 2022 07:43 PM

    sramanuja Can you please check? There should be a PR open. 



  • 67.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 15, 2022 07:51 PM

    vSphere 7 = Garbage.  6.7 support needs to be extended another 12 mo. at least.

    Regards,

    Adam Tyler



  • 68.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Oct 25, 2022 05:07 AM

    Resolution Host has lost time synchronization error after upgrading to ESX 7.0.3 build 18644231 (86255)

    This issue is resolved in VMware ESXi 7.0 Update 3c (build number 19193900)

    ESXi NTP can be forced to accept time from AD time sources root dispersion has a large value

    This is described here https://kb.vmware.com/s/article/1035833

    Commands to add "tos maxdist" for 7.0U3 builds:
    1. Update NTP configuration(in file and configstore)
    cp /etc/ntp.conf /etc/ntp.conf.bak && echo "tos maxdist 15" >> /etc/ntp.conf.bak && esxcli system ntp set -f /etc/ntp.conf.bak
    2. Restart NTP: esxcli system ntp set -e 0 && esxcli system ntp set -e 1

    Note: Please note that the "tos maxdist" config will not persist across reboot .

     

     

     

    https://kb.vmware.com/s/article/86255

     



  • 69.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 22, 2022 01:16 PM

    We still facing NTP problems. And we have opened a SR to VMware, where we got a recomendation exactly the same: 

    ESXi NTP can be forced to accept time from AD time sources root dispersion has a large value

    This is described here https://kb.vmware.com/s/article/1035833

    Commands to add "tos maxdist" for 7.0U3 builds:
    1. Update NTP configuration(in file and configstore)
    cp /etc/ntp.conf /etc/ntp.conf.bak && echo "tos maxdist 15" >> /etc/ntp.conf.bak && esxcli system ntp set -f /etc/ntp.conf.bak
    2. Restart NTP: esxcli system ntp set -e 0 && esxcli system ntp set -e 1

    Note: Please note that the "tos maxdist" config will not persist across reboot .

    Our ESXi hosts are currently having following build: ESXi 7.0 Update 3g 

    20328353


  • 70.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 22, 2022 02:22 PM

    Hi ,


    Would you mind explaining your exact problem better?


    The starting point of this thread began with the worrying message on a gold background which read: "Tine service is currently not syncronized", long since removed in the latest versions of the vCenter object, which then evolved with other reports of various problems (IMHO in part) solved from version to version of ESXi.


    I can tell you that AFAIK if you insert the line "tos maxdist" option in your "ntp.conf" file the it persist across reboot since ESXi 7.0U3f, at least on my hosts it works like this, and I didn't even need to add it referencing a host built around a domain controller running Microsoft Windows Server 2022 (never synced to an upstream time source) as my "time server". What I have instead noticed is that if the network interface to which the default VMKERNEL interface connects is "saturated" for a long enough time, the NTP service "tends to lose its bearings".


    Regards,
    Ferdinando



  • 71.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 23, 2022 01:15 PM

    vCenter version in our environment: vCenter Server 7.0 U3i 20845200; 

    ESXi hosts: ESXi 7.0 Update 3g 20328353

     

     

    We have two clusters, where in Cluster A: 6 ESXi hosts, in Cluster B: 4 ESXi hosts.

    They are not in the domain, and VMs are not syncing time from ESXi hosts. The problem persists in both clusters. 

    But when I open via vCenter - > ESXi host - Configuration - > Time - there is a well-known Error: Time service is currently not synchronized

    My colleague took over the case but called me yesterday and informed that he did as VMware Support requested: 

    adjusted parameter: tos maxdist -  no changes. He restarted services via ssh. still the same.



  • 72.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 23, 2022 02:30 PM

    Hi


    I'll tell you, I understand, but from my point of view an ESXi host is not a reliable time source, and neither is any other type of virtual machine, and simply adding the line "tos maxdist" blindly because someone says so, as you've found it's (almost) useless. What I can tell you, based on my personal experience, after having deployed over a hundred ESXi / vCenter combinations is that the error: "Time service is currently not synchronized" does not always (IMHO almost neveer) correspond to the concrete "state of things" and that's why I asked what your exact problem is.

     

    I would suggest using reliable time sources outside (and unrelated) your vSphere infrastructure for the whole thing.


    Regards,
    Ferdinando



  • 73.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 23, 2022 02:37 PM

    > ESXi host is not a reliable time source, and neither is any other type of virtual machine

    how comes that our ESXi servers are timesources since 2008 and the ntp-servers for both hosts are a virtual machine on each one?
    and yes you can easily confirm if the time is correct

    > I would suggest using reliable time sources outside (and unrelated) your vSphere infrastructure for the whole thing

    running ntpd on each and every guest is nonsense and in a 100% virtualized environment nothing else exists

    frankly i disabeld the foolish warning this thread is about and at no point in time any VM had a wrong time - so this topic should come to an end after all that months



  • 74.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 23, 2022 04:18 PM

    Comment edited because it is not worthy of note and not to cause disruption to others.



  • 75.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 23, 2022 04:22 PM

    so you are the only one which is allwoed to express his opinion? sorry, the game don't work that way!



  • 76.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 23, 2022 05:00 PM

    Comment edited because it is not worthy of note and not to cause disruption to others.



  • 77.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 23, 2022 05:26 PM

    i didn't say you opinionsat a whole  are nonsense! i said "running ntpd on each and every guest is nonsense and in a 100% virtualized environment nothing else exists" and why it's nonsense to run ntpd on 100 or more guests and torture "pool.ntp.org" day and night should be obvious

    you missed the context: 100% virtualized environment
    and in a 100% virtualized environment you would need to run dedicated, redundant hardware only to run ntpd

    may YOU explain on a TECHNICAL GROUND why your ESXi hosts can't run ntpd and the guests use the host as time source - because, well, it just works and the times where vmware-tools made lage time-jumps are gone for years now!



  • 78.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 23, 2022 07:22 PM

    Comment edited because it is not worthy of note and not to cause disruption to others.



  • 79.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 23, 2022 07:42 PM

    interesting how everything is going. 



  • 80.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 23, 2022 08:13 PM

    Hi, maksym007,


    But nothing happens.

    I hope that the response you receive contains useful information to remedy the problem you have encountered.
    In the meantime, I wish you a Merry Christmas and a Happy New Year.


    Regards,
    Ferdinando



  • 81.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 23, 2022 08:53 PM

    Thx, same to you



  • 82.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Dec 23, 2022 10:07 PM

    Comment edited because it is not worthy of note and not to cause disruption to others.



  • 83.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Mar 01, 2022 04:05 PM

    ,

    Good afternoon, I can only talk about my personal experience and environment,

    Introduced with ESXi 7.0U3  and before ESXi 7.0U3c if for any reason any reliable time was not reachable, the management agent on the hosts crashed in less then a minute; I mitigated that issue by adding a local, reliable, time source, and referencing it by it's IP address and not by FQDN . it was not a big effort as my lab is a small one with only six ESXi host.

    Disabling the monitoring of "time sync" related event also mitigated this issue, but with a permament warning in HOST > Configuration > time configuration stating: the "time service is currently not syncronized" even when the NTP service was working as expected and time source were available over internet.

    Somehow ESXi 7.0U3c fixed that specific issue but introduced other.

    With vCenter 7.0U3c going to HOST > Configuration > time configuration is more or less futile, as no alarm / warning are generated / tracked when time sycronization went lost, e.g. when I stopped the service via command line and after some time I restarted it (/etc/init.d/ntpd stop, start, restart). So, to be properly notified of potential issues in a timely manner I decided to rely on modern log facility / monitoring tools.

    TBH, I have not seen the NTP service restart by itself but I'm of the "old school" and so I tend to make things as simple as possible, and I suppose I was also more lucky than other. To be noted, in my lab I don't rely on AD, the hosts or any kind of virtual machine as a (reliable) time source.

    Regards,
    Ferdinando



  • 84.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Mar 02, 2022 09:34 PM

    Thank you for your feedback. I've shared it with our engineering team.



  • 85.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 28, 2022 09:28 PM

    @sramanuja I have SR #22341623706 raised. Happy to share log bundle 



  • 86.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 28, 2022 10:41 PM

    Let the force be with you It can take a while now.

    of course, waiting for a solution and updates from you



  • 87.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 29, 2022 08:12 AM

    This should be the solution to the NTP issue: https://kb.vmware.com/s/article/87488

    Additional informations: https://kb.vmware.com/s/article/87176

     



  • 88.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 29, 2022 08:48 AM

    for the sake of god how should https://kb.vmware.com/s/article/87488 be related to this issue?



  • 89.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 29, 2022 09:03 AM

    ESX 7.0.3 allows a time difference of 1,5 seconds to the ntp server. In this KB article the value is changed to 30 seconds and should solve the ntp synchronisation issue.



  • 90.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 29, 2022 09:06 AM

    than answer that instead of a article about unrelated config stuff given that literally nobody did change the ntp config outside the vCenter UI and it#s still something VMware has to fix - when my hosts and ntp servers differ for more than 10 seconds i commit suicide anyways so that can't be underlying issue



  • 91.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 29, 2022 09:14 AM

    Calm down and keep it professional.

    It's not unrelated and you would know that if you read both articles. You can use the solution as a workaround or not, that's up to you.

    Have a nice day.



  • 92.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 29, 2022 09:18 AM

    nobody needs another workaround - the first one dsiable the alert just works fine - we need a solution from the clowns at VMware and i get back to be professional when they manage their homework and stop acting like microsoft - the whole vSpehere 7.x stuff is a bad joke and every single time i get a mail notrify of a reply in this thread which don't contain the words "the latest update one and for all will fix this issue" i get angry - peruiod



  • 93.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 29, 2022 01:03 PM

    None of those KB posted here address the issue. And the workaround is not acceptable/practical. We always set NTP using esxcli command.



  • 94.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 29, 2022 01:06 PM

    Not related to the NTP issue. I went to edit my profile and below are the options I have :). Really VMware?

     

    ICQ number
     
    AIM screen name
     
    MSN Messenger screen name
     
    Yahoo ID
     
    Skype name


  • 95.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 29, 2022 04:12 PM

       Your posts, "ESX 7.0.3 allows a time difference of 1,5 seconds to the ntp server. In this KB article the value is changed to 30 seconds and should solve the ntp synchronisation issue."

    No, the links you posted for modifying NTP.conf is unrelated to the issue.

    Quick recap of this thread:

    1.) Clean-Install of ESXi 7.0u3d, without changing anything at all, the "out of box" experience NTP is broken - In multiple ways that have been detailed in this thread.

    2.) An Employee of VMware posted here, reproduced the issues and confirmed that NTP is broken, and admitted it's such a low-priority to VMware engineering they need more customer SR's to be submitted as complaints to get engineering at VMware to prioritize fixing NTP in vSphere 7.0.

    So, if you want to help, please know that we all know how to use google, and if you want to really impress everyone - Then find a way to escalate this NTP issue to VMware engineering to provide a patch.

    THAT would be incredibly helpful and much appreciated by everyone.



  • 96.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 29, 2022 06:47 PM

    Has anyone brave enough tried the new ESXi 7.0.3-Update3e yet?

    NOTE: It's best-practices to always update vCenter before ESXi especially with security-focused updates like this one (e.g. VCSA Update3e).

    This ESXi build was released 6/14/2022, but not posted as ISO image as-of 6/29/2022:  VMware-ESXi-7.0U3e-19898904-depot.zip

    https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-esxi-70u3e-release-notes.html#patch-download-and-installation-4

    I see nothing about NTP in release notes, looks to be almost-all Security patches; but, I was hoping MAYBE they slipped in a NTP fix with all those security patches.. One can only hope!

    I will not have time to test this latest build in our lab for 1-2 weeks but I'm curious.

     



  • 97.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 29, 2022 06:52 PM

    Still broken in 70u3e.

    BTW, you can create an ISO from the ZIP bindle, using image builder cmdlets. 



  • 98.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 29, 2022 07:10 PM

    Yes, it is still broken in update 3e.



  • 99.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 29, 2022 07:24 PM

    I would add to your list  , if you don't mind.

    1. NTP was broken along with a long list of other "low-level" functionality upon initial release of 7.0. We all saw the following releases, a-c, come and go with mention of this issue, being pulled off the download site and fully removed from circulation, but with no fix for NTP. This indicates that the issue is visible but not a priority.

    2. I have personally received 2 responses from VMWare Employees that instructed me to create a ticket with support to elevate this issue specific to me. I can not do that without purchasing a support contract. We own minimal licensing but not support, so I have to wait for one of you to lodge a complaint

    Suggesting a work-around is often welcome in situations like this, but at this point most of us had to find a way to "silence" this error or take a chance that the wrong person notices, or worse, our servers take random "naps". I appreciate that readers of this thread will try to help, often without reading the entire thread, and therefore will offer unrelated, already mentioned, or less than ideal stop-gap measures as band-aids. Unfortunately, doing so will almost always mean that if there is a fix, reversing the band-aids will be one more unpredictable variable in getting our systems back to a healthy state. Right now, it appears that a large portion of the 7.x user base is running with a piece of tape over the timekeeping warning light. If/when the Devs at VMWare get this "fixed", we will each have to find a way to reverse any stop-gap techniques we used to get this far. I think at this point, anyone that is aware of this issue has given up hope for a true fix and is anticipating a clean re-install once a true fix is released. That is pretty scary for those with large environments.

    I don't want to drag the Engineers at VMWare. They produce a product that we all depend on, that usually works as intended, and has become the industry standard for virtualization all over the world. To keep that position, however, is very difficult. More so if they continue to introduce "features" without fixing existing issues.

     

    TL:DR- Fix the timekeeping please

     



  • 100.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jun 29, 2022 08:44 PM

    So I am the guy that started this thread back in October 2021.  Unreal that VMware has not addressed it.  The vSphere 7 product is definitely crap.  VMware needs to extend support for vSphere 6.7 at least another year.



  • 101.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jul 04, 2022 08:22 AM

    Hi everyone, I have received a response to my support case. 
    (don’t shoot the messenger)  

    This alert is known issue on 7.0.3 builds (19898904 &19482537) that has no impact on the production environment as it's a false alert.
    
    Resolution
    
    The resolution for the issue is said to be provided in ESXi 7.0.3 P06 accordingly.
    
    We expect the fix to be released this quarter or the first month in Q3.
    
    But as you know and due to the testing phases here this might take time to ensure that the issue does not happen again, So this should take time to be confirmed.

     



  • 102.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jul 04, 2022 10:09 AM

    Hi,


    IMHO and AFAIK,


    With the introduction of version 7.0U3 and until version 7.0U3c disabling the "time service event monitoring" was a "workaround" made necessary to prevent the HOSTD process crashing when an event related to the loss of time synchronization occurred, We all know the consequences given by the failure of the aforementioned process.
    This article described the specific problem: https://kb.vmware.com/s/article/86283, The hostd service in ESXi 7.0U3 crashes due to memory corruption (86283).

     

    Nowadays disabsabling that option means that no time synchronization events are logged and that the vCenter object will show the error object of @ actyler1001's original post, but otherwise it doesn't fix anything. Its using one of the retired ESXi / vCenter version a problem equivalent "to flip a coin".

     

    As a consequence of the prolonged inaccessibility to the internet due to ISP outages and some other work on my electrical system which lasted several hours, I noticed that in the moment in which a time server referenced by its FQDN name cannot be resolved and at the same time it is not reachable, The NTP service tends to produce such high "offset" and "jitter" values up to the point to reject all the configured time servers, even those referenced by their IP addresses, despite thoese were being perfectly reachable.

     

    To say, in my case Internet was not available from 12:50 am until (approximately) 7:00 pm, Meanwhile all my ESXi hosts have logged these events (seconds more or less):

    system clock no longer synchronized to upstream time servers, Warning, 06/23/2022, 3:15:47 PM
    system clock synchronized to upstream time servers, Warning, 06/23/2022, 4:43:29 PM
    system clock no longer synchronized to upstream time servers, Warning, 06/23/2022, 5:43:30 PM
    system clock synchronized to upstream time servers, Warning, 06/23/2022, 9:06:19 PM

     

    Actually the NTP service really lost time sync as stated in the "event", but to be sure I have to check "by hand host by host". Thus, in this sense I agree with the views expressed by others in saying that the NTP service is somehow broken. What then, in the case of my IT context, the temporary loss of synchronization of the time does not cause insurmountable problems is another matter, but it is annoying all the same.

     

    Regard,
    Ferdinando



  • 103.  RE: NTP broken after ESXi 7u3 upgrade

    Broadcom Employee
    Posted Jul 06, 2022 12:59 AM

    Ferdinando,

    How many upstream time sources are in your ESXi NTP configuration? Are you using your internet provider's time sources or NTP pool at pool.ntp.org?

     



  • 104.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jul 06, 2022 10:03 AM

    , good morning,


    Well,

     

    In the beginning for time synchronization I was referring to the the NTP pool provided by "pool.ntp.org" in my country, which are in the number of three and obviously refrenced by their FQDN name. As I said earlier, if for some reason they were unavailable the host process would crash in less than a minute. and so I did exactly as (I actually got there by myself) reported in the article I mentioned.

     

    That said, currently all of my core systems /service including ESXi hosts, vCenter object, domain controller, DNS server etc. use no less than three time sources. The first two sources are stratum 2 public NTP servers referenced by their FQDN name, the third source is my network switch and its referenced by its IP address, It can provide a reliable time reference even when not synchronized to any upstream time server because the correct time persists both after a restart or after a prolonged shutdown (it support time-based ACL), by doing so I have greatly reduced a possible problem.


    The latter, references three time sources (all stratum 1), over internet and refenced exclusively through their IP address and also use as a time reference its internal clock (hardware with battery).  


    But I cannot rely on the vCenter object to be aware of it, because as in my previous post I reported some recorded events but, from the "point of view" of the vCenter object UI, still nothing happened: Network Time Protocol, Running for less than a month and last time sync 06/23/2022, 8:59:31 PM (local time), and by the way there is a "refresh" button that at the moment "refresh" nothing.


    The point of my speech is that, as I think I have already said, I can tolerate losing time synchronization without any particular problems even for quite a long time but I like to be somehow sure, without necessarily having to deal with the "host by host command line", when, sooner or later, things will (re)start to work out as they should.

     

    Regards,
    Ferdinando

     

    Edit: Removed things already said and unnecessarily repeating.

     



  • 105.  RE: NTP broken after ESXi 7u3 upgrade

    Broadcom Employee
    Posted Jul 12, 2022 07:32 AM

    Ferdinando,

    Thanks for the information. Can you tell us exactly which version of ESXi and vCenter you're running?

    Please see KB article 1022196 for more information on determining ESXi and vCenter build numbers.

     



  • 106.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jul 12, 2022 08:50 AM

    , good morning,


    Currently in my small IT infrastructure for both ESXi hosts and vCenter object I'm running at the 7.0U3e build level.


    Regards,
    Ferdinando



  • 107.  RE: NTP broken after ESXi 7u3 upgrade

    Broadcom Employee
    Posted Jul 29, 2022 01:11 AM

    Ferdinando,

    ESXi 7.0U3e has a fix the DNS name resolution issue. There was problem when a NTP server's hostname couldn't be resolved. This could happen, for example, when the NTP pool no longer resolves hostnames to IP addresses because the upstream servers are unavailable or not providing accurate time. The ntppool.org web page has hints for handling this for countries with small number of time servers in the NTP pool.

    However, if you're still seeing this problem on ESXi 7.0Ue, please let us know and we can investigate further.

    On VC 7.0U3e, when you click on refresh, you should see the "Host date and time" change. This may take a second or two. Is this not working for you? Or is the problem that the "Time services is currently not synchronized." is not cleared?

    If the error is the second problem, can you tells or post a image of what the "Test Services" text area shows? If it shows "Configuration is working normally" at the top, does it continue work while you hit the "RE-RUN" button a number of times?



  • 108.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jul 29, 2022 09:29 AM

    , good morning,


    First of all, thank you for your feedback,


    Let's start by saying that I approached the problem "in my own way", referencing each time source with their IP address and not their FQDN name, but then I added some options to the configuration to prefer my local time source and to speed up the initial synchronization ("prefer" and "iBurst"). Supported or not, the fact is that I have remedied this problem (or at least mitigated to the point of not having to deal with it seriously anymore).


    I probably should have thought about it first.


    As far as I'm concerned I don't use (at least when I can I avoid them) the "pools" of the ntp.org project, they work, but more than once it happened to me to detect time discrepancies a little too high.


    Let's talk for a moment about the vCenter object, with the introduction of version 7.0u3e the "refresh" button actually translates into an update of the "Host date and time" field, it's just a matter of waiting a few seconds, with the introduction of version 7.0u3f the warning about the "Time service is currently not synchronized" clear itself when time get actually synchronized. Obviously, after these changes of mine to the configuration of the NTP service, testing the service shows that the time sources "cannot be resolved", I simply ignore this circumstance.


    Speaking instead of ESXi 7.0U3e I think it is positive that the iteration between the NTP service and the resolution of FQDN names has been corrected. Of course I cannot verify it for the reasons I have explained above.


    The point of my speech was that I wanted to illustrate to you what was happening, in a small computer context like mine (perhaps similar to that of many others), but since it was obvious that something was not working I have long since managed, right or wrong supported or less.


    Regards,
    Ferdinando

    A note: To date my small IT infrastructure is at level 7.0U3e for what concerns the ESXi hosts and 7.0U3g for the vCenter object.



  • 109.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jul 29, 2022 03:42 PM

    I also use IP's (not FQDN) with ESXi hosts providing time to VCSA.

    My primary for NTP is always: time.nist.gov

    Only if Secondary is required, then I use: pool.ntp.org

    I've still had my ongoing multiple clean-install issues described in my details prior posts, which are still not mentioned in Release Notes as fixed.

    I think it's unwise to disregard transitory errors, they usually mean something is failing and then potentially-succeeding or even worse, being suppressed from GUI on reattempts.

     



  • 110.  RE: NTP broken after ESXi 7u3 upgrade

    Broadcom Employee
    Posted Jul 29, 2022 05:17 PM

    Ferdinando,

    Thanks for the update. Please let us know if you have other issues and we'll help you get them resolved. We appreciate your honest and helpful feedback and will use it to make your vSphere experience better.

    The current VC UI is confusing because there are two separate concepts on the same Time Configuration page: "setting" the system clock and "synchronizing" the clock. Having the clock "set" is critically important for elements of the virtual infrastructure including communication security and event monitoring. Having the clock "synchronized" is critical for keeping the clock "set" over the operational period of the virtual infrastructure. For many customers, the operational period is measured in years, so it's critical to have a periodic updates from trusted time sources. 

    When the time service starts, the system clock gets "set" quickly using one of the provided time sources. Time "synchronization" takes a few minutes longer because clock adjustments get selected so that they can be slowly applied to keep the system clock from jumping abruptly forward or backward. Most customer inquiries about time involve questions about "synchronization" so the UI was designed to give a picture of exactly what is happening on the ESXi host without logging in via the command line shell.

    Our new cluster wide configuration infrastructure will make all of the NTP options such as "iburst" available without needing to edit local files on ESXi hosts. The options will carry forward across reboots and software updates. Unfortunately, there's no way to enter them into the Time Configuration page without making the UI more complex.



  • 111.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jul 29, 2022 07:34 PM

    Hi,


    That by "testing the service" the generated report says that a time server referenced by its IP address cannot be resolved due to the NTP options I have considered adding, a non-random choice, I take the liberty of ignoring it. I wouldn't allow myself to do this if I hadn't implemented other equally effective methods of monitoring the "health" of the "NTP service", in any case I do not suppress them.



    Actually, on latest ESXi build you can add option like "iBurst" or "prefer" in the NTP configuration page related to time server without the need to "manually edit" configuration files and act via the command line. As I "prefer" my local time source, from the point of view of the vCenter "Time Synchronization Services test" NTP "is in sync" in the same moment in which the aforementioned time source has been marked as "selected", that is within seconds.


    Having said that, I really thank you for your offer of support, but as I wrote I have already taken an adequate remedy for the problem.


    Regards,
    Ferdinando

     



  • 112.  RE: NTP broken after ESXi 7u3 upgrade

    Broadcom Employee
    Posted Jul 29, 2022 08:04 PM

    Yes, we discovered the same error about resolving the NTP host name when "testing the service". It's been fixed internally, but I don't know when the fix will appear in a public release. We also fixed an issue internally when more than one option gets specified on the NTP "server" line (i.e. server x.x.x.x minpoll 4 maxpoll 4 iburst) in ntp.conf. Some of the options were getting dropped when "ntp.conf" was reloaded into the configuration engine. That was also fixed, but I don't have visibility yet on when it will be available to you.

    Glad to hear things are working now. As always, let us know if you run into problems. We're happy to help you.



  • 113.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Sep 07, 2022 02:56 PM

    How is this a false alert when I'm suffering issues on three of my hosts using build 19482537? Just saying. 



  • 114.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Sep 07, 2022 04:19 PM

    Comment edited because it is not worthy of note and not to cause disruption to others.



  • 115.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jul 28, 2022 11:41 AM

    I upgraded to 7.03f from 7.03e. This 7.03f released 07/12/2022 seems fix the problem.



  • 116.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jul 28, 2022 11:53 AM

    That's not true. I upgraded to VMware ESXi, 7.0.3, 20036589, and the issue still persist. 



  • 117.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jul 28, 2022 04:03 PM

    I installed VMware ESXi, 7.0.3, 20036589 from VMware ESXi, 7.0.2, 17630552 with no issues on a dell R740 and R730

     

    checked NTP service 1st it was not sync I run a refresh and test then it worked.



  • 118.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jul 28, 2022 06:05 PM

    Try a reboot of the host and see if the warning comes back 



  • 119.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jul 29, 2022 02:38 PM

    With 7.0.3f, I follow your suggestion and reboot the host. The issue did show up but after couple minutes I refresh the time service. The warning message went away.



  • 120.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Aug 08, 2022 06:55 PM

    I've just deployed Update 3f Build 20036589 into my VMware environment.  This NTP problem appears to be solved.  Wow, that only took VMware 9 months to fix.....



  • 121.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Aug 15, 2022 08:30 PM

    Yup i upgraded  to 3f too, and no more time sync issues!  Seems to be a silent fix as nothing was mentioned in release notes.



  • 122.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Sep 15, 2022 08:13 PM

    Well, we've started our vSphere 7 upgrades in production.  Deployed ESXi, 7.0.3, 20328353 to our first host and guess what?  Yes, the NTP problem persists.  What a joke.  I've opened a support request with VMware, we'll see where this goes.

    Support Request #22363318109

    actyler555_0-1663272762273.png

     



  • 123.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Sep 15, 2022 08:30 PM

    Deployed a fresh 7.0.3 Build 20328383 server, and I am seeing this problem as well.



  • 124.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Sep 15, 2022 08:35 PM

    I just got done running the commands that  posted above and it did work...

     

    for reference...

    **Reset NTP settings on ESXi host, back to defaults:**
    esxcli system ntp set -r


    **Reload NTP service on ESXi host:**
    /etc/init.d/ntpd restart


    **Use vCenter GUI to input NTP addresses into ESXi host again**
    ((I'm using GUI because I want to see it work as designed...)

    **Reset NTP on ESXi host again**
    /etc/init.d/ntpd restart


    **Poll the NTP server on ESXi host a few times, give it a minute:**
    ntpq -p


    **Wait until NTP server on ESXi host shows Time Sync = True:**
    esxcli system ntp test

    ((Diags output omitted..)

    Service analysis completed.
    Timeinsync: true

     

    **Check ESXi host and vCenter GUI's**
    ((In vCenter > Host > Configure > Time Config > Test Service))
    **Finally, for 1st time since 7.0u2, this test passed for me!**



  • 125.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Sep 05, 2022 06:55 PM

    **** My NTP cleanup/fix on vSphere 7.0 Update 3(g) ****

    VMware vSphere update: 7.0u3 (aka 7.0.3)
    Patched ESXi version: 7.0u3g.
    Patched vCenter version: 7.0u3g.

    FIRST: Verify Time Zone are same on ESXi and vCenter.. I only point this out because you can now finally change Zone on VCSA admin port 5480..

    ===============

     

    **NTP KB ARTICLE; NOTE! ONLY FOR 7.0-U3+**

    https://kb.vmware.com/s/article/87488

    [root@localhost:~] vi /scratch/ntpconfig.txt
    [root@localhost:~] cat /scratch/ntpconfig.txt

    server time.nist.gov
    server pool.ntp.org
    tos maxdist 30

    [root@localhost:~] esxcli system ntp set -f /scratch/ntpconfig.txt
    [root@localhost:~] esxcli system ntp set -e 1

     

    ===============

    **NTP KB ARTICLE; NOTE! ONLY FOR 7.0-U3+**

    https://kb.vmware.com/s/article/87176

    To display current configuration:

    # esxcli system ntp get
    # esxcli system ptp get

    To change configuration:

    # esxcli system ntp set
    # esxcli system ptp set

    To get a list of ALL new 7.0 Update-3 commands, use help:

    # esxcli system ntp --help

     

    ===============

    **HELPFUL OLD NTP KB ARTICLE, FOR SYNTAX COMMANDS ONLY**

    ((This KB specifically states issues after u3 upgrade))

    https://kb.vmware.com/s/article/86255

    **WARNING!! "RESOLUTION" at end of article does not do what you think it does at first glance, did not work for me, and in general editing any "conf" file with plain-text in 7.0.3 can be dangerous because if its encrypted it will become corrupted, so.... be sure to check if the "conf" file is still plain-text (safe to edit) or if it's now encrypted (dont edit will corrupt), as VMware transitions towards the configurations being encrypted over time with future updates...  I expand on this in a post further down.

    ===============
    ===============

    After KP Articles, vCenter was showing as broken (red alerts).

     

    To get vCenter to re-sync NTP based on Host, this worked for me:

    **Reset NTP settings on ESXi host, back to NTP defaults:**

    [root@localhost:~] esxcli system ntp set -r

     

    **Reload NTP service on ESXi host:**

    [root@localhost:~] /etc/init.d/ntpd restart

     

    **Use vCenter GUI to input NTP addresses into ESXi host again**
    ((I'm using GUI because I want to see it work as designed...)

     

    **Reload NTP on ESXi host again**

    [root@localhost:~] /etc/init.d/ntpd restart

     

    **Poll the NTP server on ESXi host a few times, give it a minute:**

    [root@localhost:~] ntpq -p

     

    **Wait until NTP server on ESXi host shows Time Sync = True:**

    [root@localhost:~] esxcli system ntp test

    ((Diags output omitted..)

    Service analysis completed.
    Timeinsync: true

     

    [root@localhost:~] esxcli system ntp get
    Enabled: true
    Loglevel: warning
    PID: 2126728
    Runtime Seconds: 260
    Servers: time.nist.gov, pool.ntp.org
    Service Providing Kernel Time: Network Time Protocol
    Time Service Enabled: true
    Time Synchronized: true

     

    **Check ESXi host and vCenter GUI's**
    ((In vCenter > Host > Configure > Time Config > Test Service))
    **Finally, for 1st time since 7.0u2, this test passed for me!**

     

    Note-1:
    This final NTP bug is annoying but not a deal-breaker:
    In ESXi host NTP config GUI (not vCenter) version 7.0u3g:
    ESXi GUI: Clicking "Actions" is still broken (it does nothing).
    ESXi GUI: Clicking "Refresh" finally works again as expected.

     

    Note-2:

    After all this cleanup work and fiddling with ESXi and vCenter (both version 7.0.3g), finally now, NTP tests pass in both ESXi and vCenter when vCenter is set to ESXI Host for time sync.

    I did not try all this with ESXi version 7.0.3f. but if you read 7.0.3f release notes, you'll want to upgrade to patch 7.0.3g immediately...

     

    EDIT: Oct 2, 2022, to remove being overly harsh with use of "destroy" (elaborated below). And, fix typo to clarify **Reset NTP on ESXi host again** should instead read **Reload NTP on ESXi host again**"



  • 126.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Sep 06, 2022 06:42 PM

    Comment edited because it is not worthy of note and not to cause disruption to others.



  • 127.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Oct 02, 2022 05:19 PM

     

    Regarding my prior comment (Before I edited):

    "

    **HELPFUL OLD NTP KB ARTICLE, FOR SYNTAX COMMANDS ONLY**

    ((This KB specifically states issues after u3 upgrade))

    https://kb.vmware.com/s/article/86255

    **WARNING!! "RESOLUTION" AT END OF ARTICLE DESTROYS 7.0.3 HOST**
    **ONLY USE THE NEW KB ARTICLES ABOVE FOR U3 NTP TEXT FILE METHOD**

      ((VMware should insert a warning about this for Update-3!!))

    "

    /////////////

     

    There are several VMware official KB articles that warn not to touch the "conf" files anymore with 7.0 Update3c+ specifically, those edit commands in that KB seem to be for 7.0 Update 1 and 2 only, and did not do anything to the "tox maxdist" value when I tested and in fact did corrupt my NTP configuration.

     

    This made me believe those commands in that KB are leftovers from Update 1 and 2, so the article should instead point to the other newer 7.0.3 / update3 KB's that I listed in my prior post: Which is, to make a plain text file and then read the commands for inclusion into the "conf" file - that DID work for the "tox maxdist" value change for me. 

     

    Also, there are official KB articles that warn the numerous "conf" files are being encrypted over time by VMware as future 7.0 Updates become available, so issuing ANY command to add a string of plain-text characters to a "conf" file, is now a risky situation and would in effect "corrupt" any encrypted conf file.

     

    And finally, if you look at the commands, they make a backup of the ntp.conf as ntp.conf.bak, and then execute the esxcli against the ntp.conf.bak instead of original file. So, this is good to not corrupt the original conf file (eg. in case its encrypted so as to not corrupt it).  Because of this, I will admit that I was a bit too strong trying to get VMware's attention to update all their KBs for 7.0 Update-3 uniqueness VERSUS 7.0 Update-1/2 and have edited my original post to remove "destroy" and further clarified.

     

    THAT BEING SAID:

    Finding out the hard way there are these subtle yet SIGNIFICANT under-the-hood changes for new commands methods etc, for 7.0 Update 3 (compared to prior Update 1 and 2), that VMware is not Fully Emphasizing in its KB Articles like they should IMHO makes it a tedious support hassle (eg. I feel VMware should revisit EVERY 7.0.0, 7.0.1, 7.0.2 KB article, to test and fully clarify versioning if older 7.0 KB's are applicable to 7.0.3 or not, and if not then link to the all-new 7.0.3 article).

     

    This versioning clarification and highlighting more predominately, would greatly reduce the ongoing support efforts - and reduce risk)....



  • 128.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Oct 02, 2022 06:33 PM

    Comment edited because it is not worthy of note and not to cause disruption to others.



  • 129.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jan 15, 2023 11:14 PM

    I have re-reviewed this thread that seemed to derail for a while, and wondered if the recent posts are a NEW problem with NTP?

    Or, if the ongoing posts are relating to my now-buried procedure for original NTP Solution (posted ‎09-05-2022 11:54 AM ET)?

    Please elaborate in more detail.



  • 130.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jan 16, 2023 07:48 AM

    Comment edited because it is not worthy of note and not to cause disruption to others.



  • 131.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jan 16, 2023 03:20 PM

    this thread seems to be the one which gets every ntp related issue for the next years because a lot of people are too dumb to stray on topic and distinct issues - hopefully i find how to only disable notfiues for that nonsense here



  • 132.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Oct 11, 2022 10:58 AM

    hi   are the esxi really not ntp synced and do not sync any longer or does only the UI show them for what ever reason as "not synced"?

    regards



  • 133.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jan 14, 2023 08:12 PM

    Dear , 

    Do you have NTP Server or your using NTP pool ?



  • 134.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jan 15, 2023 08:20 AM

    And what is the difference? Does ESXi recognize if it's a server or pool?



  • 135.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jan 18, 2023 09:08 AM

    My Dear , 

    You have 3 options : 

    1- Integrate NTP from Your ESXI 

    2- Integrate from NTP Server (NTP appliances or NTP windows ) 

    3- Integrate from NTP Pool (https://www.ntppool.org

     



  • 136.  RE: NTP broken after ESXi 7u3 upgrade

    Posted Jan 18, 2023 03:45 PM

    Seriously? I didn't know that 

     

    Of course, these options are available. Wil it somehow solve our problem?