You're going to need to unencrypt the VM before editing the file if you've selected the option to fully encrypt the VM. That also means removing the TPM first, decrpyting, make your edits, re-encrypt, and re-add the TPM device. That's not a big deal unless you decided to run Bitlocker in the VM - then you'll need to know how to recover Bitlocker.
Another thought might be this tool https://github.com/RF3/VMwareVMX (disclaimer: I've never used it and I'm not sure if it works for Workstation 17 or 17.5).
Unless you have an absolute need to encrypt the VM's VMDK file, avoid the hassle and performance hit and choose to encrypt only the files needed for the TPM device.