Has VMware characterized TSC synchronization in SMP VMs?

[AndrewBoyer]

Hello all,

We are running a few dozen FreeBSD VMs under Fusion, ESX Server 3.5, and ESXi Server 3.5. I have read everything I can find about the timestamp counter (TSC) and its performance under VMware, but there's one question I haven't been able to answer: how close can we expect the TSCs to be in an SMP system?

We have seen an instance of the two virtual CPUs reporting TSCs with a delta of ~150 seconds. This seems much, much larger than anything that could be explained by normal uncertainty. Anyone have any insight?

Thanks! --Andrew

Background info:

1. The timekeeping PDF (vmware_timekeeping.pdf) implies that the virtual TSCs should be synchronized no matter what the underlying hardware does (Intel, AMD, etc.)

2. The systems all use Intel processors

3. The software doing performance tracing is using inline assembly calls to RDTSC, so it's not an OS issue (FreeBSD is configured to use a different timer for itself)

4. Alok Kataria (VMware employee?) said this in a LKML posting: "Virtual TSCs can be kept nearly in sync, but because the virtual TSC offset is set by software, it's not perfect"

[admin]

Andrew,

Seeing a 150 second difference between the TSCs on two virtual cpus is totally unexpected. Was this 150 second difference a temporary thing (ie was it 150 second when you looked once and then looking again the TSCs were in sync)? Or was the 150 second difference a permanant thing? If it is the later my best guess is that FreeBSD wrote to the TSCs and incorrectly introduced a 150 second skew.

Thanks,

Garrett

[AndrewBoyer]

I only have a partial trace; the TSC was off for multiple readings, but appeared to recover. I couldn't find anything in the source to indicate that FreeBSD attempts to update the TSC like other OS's, but it's certainly possible.

We contacted VMware support and they indicated that this issue (whatever the issue is) has been fixed in Vsphere 4.0. We'll be upgrading this weekend to test.

[admin]

Have you had a chance to check whether the issue still occurs with Vsphere 4.0?

[AndrewBoyer]

We're running the new version and I haven't heard any complaints yet. I'll post if we see any problems.