You've got it exactly right. The fusion drive will 'age out' portions of the virtual disk to the spinning drive, and the have to move them back to the SSD as they are accessed - it's both the spinning disk and the move back and forth that appears to cause the issue. The 'one big file' workaround helps if the VM is regularly used, but it can actually make it worse if it's infrequently accessed as the entire VM would be moved to the metal drive.
APFS on the boot drive also snapshots for time machine backups, which likewise can cause performance (on Fusion/spinning drives) and apparent free space issues (on all drives).
If the OP's machine has a USB C port on it, a solid workaround would be to get a Samsung T7 external SSD and run the VM's from there (I do that all the time, and it's really snappy). If so, then a good backup system is highly recommended (and a pointer back to vimalin to you Wil!)