VCTL stop working in VMware Fusion 13.5.2

[AlessandroDN]

Hi all,

Good to find you here in the new home, I wish this finds you well.

Yesterday I updated VMware Fusion to version 13.5.2, and some of my containers that I use with `vctl` stopped working :(

I'm on an Intel iMac 2020, macOS Sonoma 14.5.

The same containers used to work fine on 13.5.1, before the update.

Before I start "investing" some time in understanding what happens, has anyone experienced any issue with VCTL?

The error I see in the console is 

```

panic: runtime error: invalid memory address or nil pointer dereference

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x4d0eedd]

goroutine 1 [running]:

gitlab.eng.vmware.com/core-build/vctl/pkg/container.RunContainer.func1(0x100, 0xc0001ece7d, 0x1, 0x0, 0x2, 0x200, 0xc0001ec848, 0x4, 0x5222440, 0xc0000e6008, ...)

/build/mts/release/bora-23724640/kubernetesdesktop/build/release/apple_mac64/kubernetesdesktop/build/src/vctl/pkg/container/run.go:40 +0x3d

panic(0x4e8da60, 0x5b5b870)

/build/mts/release/bora-23724640/compcache/cayman_go/ob-15403392/macosx-elcapitan/mac64/src/runtime/panic.go:679 +0x1b2

gitlab.eng.vmware.com/core-build/vctl/pkg/container.StartExistingContainer(0x5222440, 0xc0000e6008, 0x0, 0x0, 0x5ba9150, 0x0, 0x0, 0x7ff7bfeff900, 0x100, 0xc0001ece7d, ...)

/build/mts/release/bora-23724640/kubernetesdesktop/build/release/apple_mac64/kubernetesdesktop/build/src/vctl/pkg/container/start.go:144 +0xcb

gitlab.eng.vmware.com/core-build/vctl/pkg/container.RunContainer(0x5222440, 0xc0000e6008, 0xc00054d860, 0x16, 0x7ff7bfeff927, 0x1a, 0x5ba9150, 0x0, 0x0, 0xc00019dc10, ...)

/build/mts/release/bora-23724640/kubernetesdesktop/build/release/apple_mac64/kubernetesdesktop/build/src/vctl/pkg/container/run.go:47 +0x36e

gitlab.eng.vmware.com/core-build/vctl/cmd.glob..func28(0x5b72040, 0xc00004ce80, 0x5, 0x8, 0x0, 0x0)

/build/mts/release/bora-23724640/kubernetesdesktop/build/release/apple_mac64/kubernetesdesktop/build/src/vctl/cmd/run.go:60 +0x1a7

github.com/spf13/cobra.(*Command).execute(0x5b72040, 0xc00004ce00, 0x8, 0x8, 0x5b72040, 0xc00004ce00)

/build/mts/release/bora-23724640/kubernetesdesktop/build/release/apple_mac64/kubernetesdesktop/build/pkg/mod/gitlab.eng.vmware.com/core-build/mirrors_github_spf13_cobra@v1.1.2-0.20201207081517-b481a04a4f83/command.go:851 +0x460

github.com/spf13/cobra.(*Command).ExecuteC(0x5b6e920, 0x501b106, 0x10, 0x0)

/build/mts/release/bora-23724640/kubernetesdesktop/build/release/apple_mac64/kubernetesdesktop/build/pkg/mod/gitlab.eng.vmware.com/core-build/mirrors_github_spf13_cobra@v1.1.2-0.20201207081517-b481a04a4f83/command.go:959 +0x352

github.com/spf13/cobra.(*Command).Execute(...)

/build/mts/release/bora-23724640/kubernetesdesktop/build/release/apple_mac64/kubernetesdesktop/build/pkg/mod/gitlab.eng.vmware.com/core-build/mirrors_github_spf13_cobra@v1.1.2-0.20201207081517-b481a04a4f83/command.go:896

gitlab.eng.vmware.com/core-build/vctl/cmd.Execute()

/build/mts/release/bora-23724640/kubernetesdesktop/build/release/apple_mac64/kubernetesdesktop/build/src/vctl/cmd/root.go:56 +0x56

main.main()

/build/mts/release/bora-23724640/kubernetesdesktop/build/release/apple_mac64/kubernetesdesktop/build/src/vctl/main.go:13 +0x57

```

Best regards

Alessandro

[Manjunath Sanjeev]

Hi Alessandra,

This issue has been fixed now. Please try from your end and reply here with your results. Thanks!

[AlessandroDN]

Thanks Sanjeev for following up on this.

Indeed, the error is different, but I still can't run any container.

There is an issue now seems related to permissions: whatever image I try to pull, the unpacking fails.

For example, I tried simply pulling a 'debian' container.

alessandro ~/Developer/ % docker pull debian
INFO Pulling from index.docker.io/library/debian:latest
───                                                                                ──────   ────────
REF                                                                                STATUS   PROGRESS
───                                                                                ──────   ────────
index-sha256:fac2c0fd33e88dfd3bc88a872cfb78dcb167e74af6162d31724df69e482f886c      Done     100% (1854/1854)
manifest-sha256:08f76151d500d9e021b9f793364a3e9e534433297d16ba0152d69d1914b64d17   Done     100% (529/529)
layer-sha256:c6cf28de8a067787ee0d08f8b01d7f1566a508b56f6e549687b41dfd375f12c7      Done     100% (49576390/49576390)
config-sha256:5027089adc4ce4ec3058d84ce51db1cd2f3eac5675e5aadfcf1f585f62fec9ed     Done     100% (1461/1461)
INFO Unpacking debian:latest...
INFO apply failure, attempting cleanup             error="failed to extract layer sha256:bbe1a212f7e9f1baaef62491a51254f3adda514c22632ea719f62713fad80f77: failed to mount /Users/alessandro/.vctl/storage/containerd/root/tmpmounts/containerd-mount838467792: hdiutil: mount failed - Permission denied\n: exit status 1: unknown" key="extract-703941000-e5Rb sha256:bbe1a212f7e9f1baaef62491a51254f3adda514c22632ea719f62713fad80f77"
ERROR failed to extract layer sha256:bbe1a212f7e9f1baaef62491a51254f3adda514c22632ea719f62713fad80f77: failed to mount /Users/alessandro/.vctl/storage/containerd/root/tmpmounts/containerd-mount838467792: hdiutil: mount failed - Permission denied
: exit status 1: unknown

I also checked that there is enough free space.

alessandro ~/Developer/m3-cdi % df -h                                                                                                     lesson-3-background-tasks-made-easy-with-async-await
Filesystem        Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk1s5s1   466Gi   9.6Gi   170Gi     6%    404k  1.8G    0%   /
devfs            202Ki   202Ki     0Bi   100%     702     0  100%   /dev
/dev/disk1s2     466Gi   2.0Gi   170Gi     2%    6.3k  1.8G    0%   /System/Volumes/Preboot
/dev/disk1s4     466Gi   4.0Gi   170Gi     3%       4  1.8G    0%   /System/Volumes/VM
/dev/disk1s6     466Gi    24Mi   170Gi     1%      25  1.8G    0%   /System/Volumes/Update
/dev/disk1s1     466Gi   278Gi   170Gi    63%    1.7M  1.8G    0%   /System/Volumes/Data
map auto_home      0Bi     0Bi     0Bi   100%       0     0     -   /System/Volumes/Data/home
/dev/disk3s1      16Gi    16Gi   469Mi    98%    524k  4.8M   10%   /Library/Developer/CoreSimulator/Volumes/iOS_21F79
/dev/disk5s1     128Gi   343Mi   128Gi     1%     100  1.3G    0%   /Users/alessandro/.vctl/storage

[Rajeswara Reddy Kotla]

Hi Alessandro,

We are able to repro this issue in-house. Logged a bug and the respective team will be looking in to it.

Thanks

Rajesh

[AlessandroDN]

Thanks a lot, Rajesh.

I'm looking forward to having Vctl working again; such a great tool.

Best

Alessandro

[Rajeswara Reddy Kotla]

Hi Alessandro

Can you please reinstall Fusion 13.5.1 and check if vctl is working for you. Also we are suspecting the issue could be because of latest macos version changes as we are not able to repro with Fusion 13.5.2 on older macos versions.

Kindly share the macos version where you have seen this issue.

Thanks

Rajesh

[AlessandroDN]

Hi Rajesh,

I am now on macOS Sonoma 14.5.

I tried rolling back to Fusion 13.5.1, and indeed, I received a similar error.

alessandro ~/Downloads % vctl version
vctl version: 1.1.0
containerd github.com/containerd/containerd v1.3.2-vmw
alessandro ~/Downloads % vctl pull debian
INFO Pulling from index.docker.io/library/debian:latest
───                                                                                ──────   ────────
REF                                                                                STATUS   PROGRESS
───                                                                                ──────   ────────
index-sha256:fac2c0fd33e88dfd3bc88a872cfb78dcb167e74af6162d31724df69e482f886c      Done     100% (1854/1854)
manifest-sha256:08f76151d500d9e021b9f793364a3e9e534433297d16ba0152d69d1914b64d17   Done     100% (529/529)
layer-sha256:c6cf28de8a067787ee0d08f8b01d7f1566a508b56f6e549687b41dfd375f12c7      Done     100% (49576390/49576390)
config-sha256:5027089adc4ce4ec3058d84ce51db1cd2f3eac5675e5aadfcf1f585f62fec9ed     Done     100% (1461/1461)
INFO Unpacking debian:latest...
INFO apply failure, attempting cleanup             error="failed to extract layer sha256:bbe1a212f7e9f1baaef62491a51254f3adda514c22632ea719f62713fad80f77: failed to mount /Users/alessandro/.vctl/storage/containerd/root/tmpmounts/containerd-mount258529085: hdiutil: mount failed - Permission denied\n: exit status 1: unknown" key="extract-888234000-8d_m sha256:bbe1a212f7e9f1baaef62491a51254f3adda514c22632ea719f62713fad80f77"
ERROR failed to extract layer sha256:bbe1a212f7e9f1baaef62491a51254f3adda514c22632ea719f62713fad80f77: failed to mount /Users/alessandro/.vctl/storage/containerd/root/tmpmounts/containerd-mount258529085: hdiutil: mount failed - Permission denied
: exit status 1: unknown

This used to work fine on macOS Sonoma 14.4.1. I often use this tool, so I know when it stopped working.

Is this related to some sort of permission? I double-checked and have Full Disk Access granted (I also tried removing the permission and re-adding with no success).

I appreciate the effort to get this fixed.

Thanks

Alessandro