Hi Steven,
You can certainly setup a connection to a TN3270 console using CA Automation Point running on a Virtual Server. However, you would indeed need to setup a TN3270 communication server on the mainframe, as well as define a console session in Automation Point on the outboard, virtual server. The console must also be defined on the mainframe host in the SYS1.PARMLIB library member.
Please read the following documentation for more details about communicating with the mainframe: Communicating with the Mainframe - CA Automation Point - 11.5 - CA Technologies Documentation
By the way, I have setup numerous VMs running AP in my lab in the past, and all of them had connections to mainframe consoles, so I know from experience that it can be done.