Never seen this, but I suspect the error message to be wrong. Otherwise, many possibilities:
a) You are not Running As Admin, b) There is something really wrong with your Win 10 Guest, you don't say which version it is, or how old VM it is (used-Windows is never the same as new-Windows), c) There is something really wrong with VMware 17.6.3. or its media, d) Or even 3rd Party software on Win 10 to block your attempt to install.