I totally understand what you're trying to do. I've got the same issue.
I'm using an HP Microserver, but the principal is the same.
I've got ESXi running off a flash drive on the motherboard. And now I've plugged a couple of USB devices in and they appear in the lsusb output:
~ # lsusb
Bus 04 Device 05: ID 1130:660c Tenx Technology, Inc. <--- here
Bus 04 Device 04: ID 1130:660c Tenx Technology, Inc. <--- here
Bus 04 Device 01: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 02 Device 02: ID 090c:1000 Feiya Technology Corp. Flash Drive <--- ESXi installed on this flash drive
Bus 02 Device 01: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 06 Device 01: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 05 Device 01: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 03 Device 01: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 01 Device 01: ID 1d6b:0002 Linux Foundation 2.0 root hub
In the vSphere client, I don't see any USB devices that I can add to my VMs.

Anyone know what's up?
Cheers,
Tom