Given that you are attempting to place a larger then 256GB file onto ESX, you should note that you will need to have a VMFS partition with a blocksize larger then the default 1MB allocated. Please reference the snippet taken from our Maximums guide.
>From the Configuration maximums guide:
http://www.vmware.com/pdf/vi3_301_201_config_max.pdf
=======================================
VMFS 2
Volume size (2TB x number of extents) (Minimum = 100MB)
File size (block size=1 MB) 456GB
File size (block size=8 MB) 2TB
File size (block size=64MB) 27TB
File size (block size=256MB) 64TB
Number of files per volume 256 + (64 x number of extents)
VMFS 3
Volume size (block size = 1MB) ~16TB 4GB (denotes an approximate value)
Volume size (block size = 2MB) ~32TB 8GB
Volume size (block size = 4MB) ~64TB 16GB
Volume size (block size = 8MB) 64TB
File size (block size=1MB) 256GB
File size (block size=8MB) 2TB