I will look into Symentec which does tape backup too.
The current BE uses the VMware storage API so it should be a good candidate.
Tape drive will connect to host via SAS HBA, will ESXi be able to make tape drive visible to VM? or we need to use a physical server to run tape backup?
It is possible to passthrough a controller and attached tape/autoloader to a VM. The only supported method is an Adaptec Parallel SCSI tape / loader however there are several references in the forums to successes in using SAS tape drives. I will see if I can find references for you. If you have hardware support for Intel vT-d or AMD IOMMU you may be able to use VMdirectpath to pass the controller/tape to a VM.