I've been using 8GB USB flash drives in IBM x3650 M3 and x3550 M3 hosts. I went with the 8GB size so that there would be a scratch partition on the flash drive. Otherwise, you need to carve out the scratch partition on your shared storage array. IF your connection to that storage goes down (we're actually having fiber HBA card issues in some hosts currently) then the scratch partition will be gone with it. With the larger size (larger than 1-2GB) you also have a place for the vmware support scripts to write to before compressing the results. With 1-2GB drives, there's simply not enough room there to do this, making said scripts fail. I ran into that issue when EMC wanted the files created by said script(s) in order to troubleshoot some SAN issues. Suffice to say, we don't have either issue with the 8GB flash drives.
BTW, you need more than 4GB of free space on the media in order to have the scratch partition created. So a 4GB flash drive won't be large enough. Unfortunately, they don't make 5GB or 6GB flash drives.