Snapshot Manager causes Workstation to terminate abruptly

[optic]

Hello everyone,

I purchased Workstation Pro 17.5 last November and have been experiencing a strange issue with Snapshot Manager. I've narrowed it down to a few steps. The issue has been present in every Workstation update up to 17.5.2 inclusively. I'm running Linux Mint 21.3 fully updated, but I'm wondering if anyone else has observed or can reproduce the same issue, including on Windows or other Linux distributions.

Here are the steps to reproduce the problem:

1. First a warning: On my system, these steps cause Workstation Pro to terminate silently. I have not dared executing these steps while VMs are running. To avoid potential data loss/corruption, please close all VMs before attempting to replicate this issue. If you use an existing VM, make a backup first because its snapshots could become corrupted.
2. Open Snapshot Manager for any existing VM. I've found it easier to reproduce with a minimum of 6 sequential snapshots present. (It might still occur with fewer snapshots, but it may take longer to reproduce.)
3. Select a snapshot, select/copy the text from the Description field into the clipboard using Ctrl+C.
4. Select another snapshot, paste the clipboard content into the Description field using Ctrl+V.
5. Repeat steps 3 and 4 a few times. I can usually reproduce the problem within 15-20 iterations, although I once tried to reproduce it for 5 minutes straight without success, so it's not a guarantee.

Basically, copying/pasting notes in the Description fields, while alternating between two snapshots eventually causes Workstation to silently terminate.

If you don't want to use any of your existing VMs to reproduce this, I've attached a test VM I created for this purpose. It's an empty VM without any OS installed (so the VM won't run), but it has lots of snapshots. Just go straight into Snapshot Manager to try to reproduce the problem.

I would be grateful if anyone could spare a few minutes of their time to try to reproduce this and post their results. Am I alone in experiencing this, or is the problem widespread?

Thank you

Edit: Clarified the steps. To reproduce, iterate over copying/pasting (steps 3 and 4), not just pasting (step 4).

Edit: Added more info in bold.

Attachment(s)

EMPTY-VM.zip   430 KB 1 version

[Nathan Savolskis]

I'm aware that Workstation Pro 17 had other issues with Linux hosts in the past.  I'll have to see if I can get into the old VMTN forums to search the discussion threads to see if there was anything similar to this case - I have not heard of silent termination.  My other curiosity is what scenario has you needing to copy/paste text into the description field of multiple snapshots (assume they are running snapshots?).  Interesting.

[optic]

Thanks for taking the time to check the old VMTN forums.

The most common scenario that I use multiple times daily:

1. Resume a VM, make changes that I want to preserve, pause VM, open Snapshot Manager, create a new snapshot
2. Select the previous snapshot (*), copy the Description field
3. Select the newly created snapshot (*) and paste and update the Description field (e.g. I use the Description field as a reminder where I left off)
4. Select the old snapshot (*), delete it

So typically, I don't copy/paste text into multiple snapshots.

The copying back and forth of the Description field between snapshots in my first post is strictly to increase the chance of reproducing the problem, it's not an actual use case. I noticed that if I only alternate selecting different snapshots, nothing unusual happens. To make the application terminate, I must also access the Description field after selecting a snapshot. It could be typing something or pasting text. The "crash" always seems to occur at the moment I left-click to select a snapshot (indicated above with an asterisk), never on the typing or pasting operation.

In my common usage scenario above, I rarely encounter the crash (since I only do the copy/paste once), but whenever it does happen, I worry VM files might get corrupted.

[Dhairya Tomar]

@opticUnable to reproduce on my Linux Mint 21 setup.

Whenever this issue is reproduced next time on your setup, please share the latest vmware ui log from this location- "/tmp/tmp-<username>/ " on this thread.

[optic]

@Dhairya Tomar Thank you for taking the time to try it out.

I just reproduced the issue again on my test system and retrieved the vmware ui log immediately after the crash. I couldn't find "/tmp/tmp-<username>/", but I had "/tmp/vmware-<username>/" instead, which I'm assuming is correct. Here's the ui log file found there.

Interestingly, the log file shows the ui caught signal 11.

Attachment(s)

vmware-ui-2037.log   262 KB 1 version

[Dhairya Tomar]

@opticThanks for sharing the vmware ui log file.

I was able to reproduce the issue on Linux Mint 21 host with the VM you have provided.

Ticket has been raised internally, relevant team will look into the same.

[optic]

@Dhairya Tomar Thanks for reproducing the issue, that's great news. To put things in perspective, because the crash in Snapshot Manager is unpredictable and I want to avoid data loss/corruption, I've adjusted my workflow to work around the problem. This means that when I have multiple VMs running and I need to take a snapshot of one of them, I usually suspend all of them first, take my snapshot and then resume the remaining VMs. This gets tedious fast, so I'm really looking forward to a fix.

Before editing your post, you had asked for a coredump. I now have one, let me know if you still need it.

[Dhairya Tomar]

@opticI was able to get the Core dump data myself but yeah it will be great if you can provide yours also in case any additional info is required.

[optic]

@Dhairya Tomar Here it is. The tar file contains the coredump and matching vmware ui log.

Attachment(s)

vmware-crash.tar.gz   5.87 MB 1 version

[optic]

@Dhairya Tomar I've been informed that the EMPTY VM I submitted contains invalid snapshots, and was asked to reproduce the bug with a supported guest OS (I used Debian 12 x64). Here it is. As requested, the attached file contains: the collected support data, vmware-ui log file showing the crash and the associated coredump.

Could you raise this internally and make sure the engineers investigating this issue get this new attachment? Thank you.

Attachment(s)

Crash info.tar.gz   23.88 MB 1 version

[optic]

I just realized the repro steps I described in the first post were inaccurate. I've updated them. My apologies.

[optic]

As I mentioned in another post, I've been informed by the support engineer that the EMPTY-VM contains invalid snapshots, which I found odd since the VM was trivial to create. All I did was go into Snapshot Manager (with the VM powered off) and create a bunch of snapshots. When a VM is suspended, Snapshot Manager won't let you take more than 1 snapshot (it detects the VM state hasn't changed and it warns you), but you can create as many snapshots as you want when the VM is powered off, even though the VM state hasn't changed.

If invalid snapshots are present in EMPTY-VM, I currently see two possible causes:

1. They were already present before I tried to reproduce the Snapshot Manager crash, in which case it would suggest Snapshot Manager corrupted them at creation time.
2. They occurred as a result of the Snapshot Manager crash, suggesting that the crash could corrupt existing snapshots.

Either way, I would expect it to be investigated/fixed.

Now, EMPTY-VM doesn't contain a guest OS. I thought this would simplify things for VMware to diagnose the problem (since running the VM is just a distraction) but it was not considered a real-life scenario and I've been asked if I've seen the problem with "real" VMs.

Therefore, I've created a new VM and installed Debian 12 x64 as a guest OS, created 6 snapshots (powering on/off the VM between each snapshot). With this new VM, I was able to cause the crash (following the procedure in the first post) within a couple of minutes. I've provided the support data for this latest crash in another post.

I also made a backup of the Debian VM before and after the crash, in order to narrow down at which point the invalid snapshots (if any) appear. The backups are fairly large (~580MB each) so I wouldn't be able to upload the files to this forum, but I can upload them elsewhere if necessary. If invalid snapshots are still present, it would be worrisome.

[optic]

@Daniel Casota In the other thread (https://community.broadcom.com/discussion/i-need-access-to-my-open-case), you mentioned that everything seems okay so far. What do you mean? Have you succeeded in reproducing the issue in engineering on a VM other than EMPTY-VM?

I was surprised to learn EMPTY-VM contains invalid snapshots, given how it was created. Could you explain what makes them invalid and how could this have occurred (so I can avoid it)? Could the Snapshot Manager crash have caused this? Finally, how can I tell if a snapshot is invalid? I'm asking because I have reproduced the crash with a new Debian VM and I'd like to investigate if there are invalid snapshots in there.

You asked: "Is the use case with vms with invalid snapshot not the only use case? If yes, could you describe the 2nd crash constellation?"

I believe I can reproduce the crash with any VM I have (including, as I mentioned, with a new Debian VM which I can upload to you. I don't think there are invalid snapshots in there, but since I don't know how to detect them, I can't be certain.)

[Daniel Casota]

@Dhairya Tomar already gave you the information that the crash was reproducible and that one was transferred to engineering. The service on this one seems „okay".

The 2nd issue: previously healthy snapshots which under a certain constellation get invalid,  I don't know, is this confirmed as reproducible?

I'm in holidays and can't reproduce constellations for the moment. W11 Windows host+VMware Workstation 17.5.2 and VMware Photon OS and multiple snapshots : never experiences issues so far (correction: MS OneDrive conflicts), but that's not the same constellation with modifying existing snapshots + Linux x86_64 host environment.

[optic]

I'm very sorry to bother you during your holidays.

You're right, @Dhairya Tomar mentioned the crash was reproducible, but he used the VM I provided, I don't think he used his own VM. Since I uploaded EMPTY-VM after reproducing the crash, I'm left to wonder if the invalid snapshots were caused by the crash, or if they were already present and they were the cause of the crash.

When I spoke with support this week, I got the feeling that the issue would not be pursued because of the invalid snapshots. I hope I didn't misunderstand. If I did something wrong during the creation of EMPTY-VM, I agree that a release note entry would be sufficient for now to warn users not to do what I did. But I also know I can reproduce the crash with multiple existing VMs that I have, not just with EMPTY-VM. Therefore, closing the case simply because EMPTY-VM contains invalid snapshots is the wrong conclusion to reach IMHO.

Anyway, support asked me for a more real-life scenario VM than an empty VM. Therefore, I created a new VM with Debian 12 x64 as guest OS (a supported guest OS), created 6 snapshots, making sure to boot the guest OS between each snapshot. I also made backups of the VM before and after reproducing the crash.

As proof, I posted this new crash info (support data, vmware ui log and coredump) yesterday. I'd also like to upload the two VM backups so that VMware can investigate the presence of invalid snapshots before/after the crash (but as I said in another thread, I don't have access to my existing open case because it wasn't migrated properly).

I believe all snapshots are valid in the VM backup I took before reproducing the crash, but I can't say if there's an invalid snapshot present in the backup taken after the crash because I don't know how to tell. The VM still seems to power up fine, which makes me think there are no invalid snapshots. But then again, I didn't think EMPTY-VM contained invalid snapshots, so take this with a grain of salt.

If it's too difficult or impossible to migrate/restore access to my existing case number (34137199), then perhaps I should just open a new case?

[Dhairya Tomar]

@optic Details which you provided for Debian VM have been passed on to the team.

[optic]

Just tested Workstation Pro 17.6. The Snapshot Manager crash is still present. I've uploaded all relevant crash info to the Broadcom support site.

Coincidentally today, I've been told the crash has been fixed internally, which is great news, but the fix will only appear in a future release. I'm hoping it gets included in the 17.6.1 release...

[junyi wu]

I have the same problem, the host system is Windows 11 24H2, the virtual machine system is Linux Mint 22, this problem first appeared in 17.6, and it is still not fixed in 17.6.1. I have tried to create a new virtual machine (without snapshots), there will be no problem at this time, but if you create a snapshot at this time, vmware workstation will crash. I have now rolled back to 17.5.2 to make it work properly.

[Dhairya Tomar]

junyi wu Request you to recreate the crash and share support bundle of the Linux Mint 22 VM

[junyi wu]

Below is my crash and shared support package

Attachment(s)

vmsupport-2024-10-14-16-08.zip   7.84 MB 1 version

[James Lin]

Junyi, FYI, your crash is completely different.  It does not seem to be related to the Snapshot Manager at all.  Additionally you are using a Windows host, and the issue reported by optic manifested only on Linux hosts.

[optic]

After being told on Sept 18 that the crash had been fixed internally and expecting it to be fixed in the next update, I regret to report that nothing has been fixed in 17.6.1. I feel I was misled. I'm deeply disappointed in how this has been mishandled.

Can someone from Broadcom please provide some clarity on this? This had been dragging on since last April. I've provided you with repro steps, logs, support data, crash dumps and the VM image I used. With all the info I provided, you should have been able to pinpoint exactly where in the code the crash occurs (there's a reg dump in the logs). I was informed the issues had been reproduced internally and were being worked on, and asked to be patient. Three weeks ago, I was told it had been fixed. So why wasn't it included in the 17.6.1 release? Now, I'm one month away from my support license expiring and have nothing to show for it.

What is the added value of purchasing a support license and taking the time to report issues if they will be completely ignored?

[James Lin]

As the person who fixed the issue you reported, it is my fault that the fix did not make it into the 17.6.1 release.  It was a complete oversight on my part that the fix was applied only to our main development branch and was not merged to the 17.x release branch, and I apologize.

 With all the info I provided, you should have been able to pinpoint exactly where in the code the crash occurs

Pinpointing where the crash occurred was easy; understanding why it occurred and how to fix it was significantly more difficult.

Three weeks ago, I was told it had been fixed. So why wasn't it included in the 17.6.1 release? Now, I'm one month away from my support license expiring and have nothing to show for it.

What is the added value of purchasing a support license and taking the time to report issues if they will be completely ignored?

While you don't have anything to show for it at the moment, your support request was not for naught. I am personally very appreciative that you reported this particular bug that apparently didn't manifest itself for ~15 years.

If this problem persists in the next release, then please post to this thread or message me.

[optic]

@James Lin

Thank you for your candor and for acknowledging a mistake was made, it happens to all of us and you have my respect for stepping up. Thank you also for confirming the crash regression has been fixed. I can certainly understand the crash being difficult to track down given the somewhat random steps to reproduce, so it's to your credit for persisting and seeing it through. Lastly, thank you VMware/Broadcom for taking the bug report seriously, even though the issue was not widely reported.

With that said, I also think this situation could have been averted three weeks ago, when I asked tech support for an internal build that included the fix, so I could test it thoroughly before the next official release (17.6.1). Had I been given access to a pre-17.6.1 build, I would have noticed the fix missing and it could have still been included in 17.6.1. Perhaps that's on me for being too patient. So, I'm asking again: Would it be possible to get an internal build, before the next official release, so I can verify if the regressions are fixed?

The main reason I'm asking for a test build is to verify whether all the regressions I reported have been properly addressed. Initially, I reported multiple issues in the Snapshot Manager in a single case report. The crash was one of them, but the initial problem I encountered had to do with the workflow. Having upgraded from Workstation 7.x to 17.x, I noticed a regression in navigating/interacting with snapshots. Back in April, I provided VMware with videos showing the differences between 7.x and 17.x. To summarize, in Workstation 17.x, Snapshot Manager appears to often scroll back to the view of the top-left/first snapshot, instead of keeping the "current" snapshot (either "You Are Here" by default, or whatever snapshot is selected by the user) always within view. It's quirky, and the scrolling back-and-forth quickly gets annoying when VMs have a large number of snapshots that aren't all visible at once. You won't notice this with fewer than 7-8 snapshots. There's also an issue with the first mouse click, following a snapshot description edit, being ignored/lost.

In hindsight, perhaps it would have been better to open separate case reports. It is my understanding that open cases, once addressed, are automatically closed after 30 days. You confirmed fixing the crash regression, but it is still not clear to me whether the workflow regression has also been resolved. If it hasn't, I would still like to create a specific case for it, before my support license runs out in a month. Hence, my request for a preliminary test build. I could just go ahead and blindly create a new case, but I don't want to waste my time either, I'd rather check if it's already fixed first.

Finally, I would like to ask VMware/Broadcom to please include these fixes in the next 17.x release update, assuming there is one, instead of keeping them queued for 18.x. These fixes are for regressions in 17.x, not new feature requests, so they should not be arbitrarily delayed or used as incentive to promote the next major version of Workstation. Since I first reported these regressions, three updates (17.5.2, 17.6 and 17.6.1) have been released with no visible improvement on my end. A fix only benefits existing customers if it gets released.

[James Lin]

We will see if we can get you an unofficial build to test regarding the crash.  This would have been straightforward in the past, but it's less clear what the company policies are now.

Regarding the other issues: they probably got lost; I don't think those were reported to engineering.  Yes, separate case reports probably would be better.  I will look into them, but I cannot guarantee that they will be fixed in 17.x.

[optic]

We will see if we can get you an unofficial build to test regarding the crash.  This would have been straightforward in the past, but it's less clear what the company policies are now.

That would be greatly appreciated, thank you.

Regarding the other issues: they probably got lost; I don't think those were reported to engineering.  Yes, separate case reports probably would be better.  I will look into them, but I cannot guarantee that they will be fixed in 17.x.

Thanks for checking, I will create a new case report then.

[optic]

Quick update: I've been unable to create a new case report for all the UI regressions. I posted about it here (https://community.broadcom.com/discussion/unable-to-report-regressions-in-workstation-17x-case-creation-fails) but never got a reply/resolution.

Gave up on waiting. I decided to just post all UI regressions to the existing case report (34137199). It's not ideal, but I don't see other options.

Broadcom, please fix this. This is preventing customers from creating new case reports. Am I the only person experiencing this?

[James Lin]

(Ignore this; I replied to the wrong message.)