Hello all.
I am attempting to get debug messages printed to the COM1 serial port within my Windows 11 guest on VMware Workstation 17 Pro (17.5.0 build-22583795). I can confirm that the serial port is setup correctly within my kernel driver, the code for which is included below; and this setup is accurately reflected within Device Manager's attributes list for the COM1 serial port on my Windows 11 guest (both before and after the below configuration is completed).
__outbyte( 0x03F8 + 3, 0x80 ); // Enable DLAB to change the BAUD rate
__outbyte( 0x03F8 + 0, 1 ); // Set the BAUD rate to 115200
__outbyte( 0x03F8 + 1, 0 ); // Set the BAUD rate to 115200
__outbyte( 0x03F8 + 3, 3 ); // Disable DLAB, specify 8-bit data unit sizes, one stop bit, and no parity
__outbyte( 0x03F8 + 1, 0 ); // Disable interrupts
__outbyte( 0x03F8 + 2, 7 ); // Enable FIFO, clear out the receive and transmit buffers
I can confirm the same COM1 serial port configuration is setup on my Windows 11 host machine, which can be seen in the attached screenshot below.
I redirect this port for usage in my Windows 11 host through the Serial Port configuration section like so.
On my Windows 11 host, I then attempt to connect to this named pipe via PuTTY, the configuration for which is included in the screenshot below.
This connects to the target, and matches the configuration of the serial port from within the guest and within the host, but displays no data when bytes are output to the COM1 serial port.
However, I have found a case in which debug messages output through the COM1 port are successfully output to the host via this named pipe, and that is when another version of PuTTY with a matching configuration but with a serial line target of the underlying COM1 port is listening on the guest Windows 11 instance.
This PuTTY configuration is included below, along with a successful output of the bytes written to the COM1 port correctly routed to the host via a named pipe.
After debug messages are output to this COM1 port, the PuTTY instance listening on the guest receives no data.
However, the host does receive the redirected data.
If this setup chain is altered in any way--such as without the dud PuTTY instance on the guest--the host will refuse to output data from the redirected serial port. Additionally, I was never able to setup the VM in any way which could record COM1 output from only within the VM. That is to say: when I configured the VM to redirect COM1 output to the physically-backed COM1 port, I was still never able to record this output on either machine; though I did not attempt this with both PuTTY instances running in parallel as I have above.
I would love to simplify this process so that I don't have to setup another PuTTY instance on the guest. Is there something I'm doing wrong or some bad configuration implemented along the way?
Thank you for your time.