If I understood correctly your post, you have 2 Vmfs:
VMFS A: 1,1 TB
VMFS B: 500 Gb
the two VMFS are on a RAID 5 volume, where you created 2 partitions equally sized (1,1 TB and 500 Gb)
Now, there are two ways:
Way A (simpler): just change the VMFSs
Way B (more complex, depends on RAID controller): delete 500Gb at controller level and extend the 1,1 tb and the create the 100 Gb
Way A
1) Move your backup files away from 500Gb partition
2) Delete VMFS datastore (the one with 500 Gb)
3) Create an extent of 400 Gb on 500Gb volume and add it to the 1,1 Tb VMFS datastore
4) Creare a new VMFS datastore of 100 Gb on the 500 Gb volume
Way B (do not suggest)
1) Move your backup files waya from 500 Gb partition
2) Delete VMFS datastore (the one with 500Gb)
3) Reboot ESX
4) Enter the RAID controller configuration
5) Delete 500 Gb partition
6) Extend the 1,1 TB volume (if supported by controller) to 1,5Tb
7) Create new 100Gb partition
9) Reboot
10) Go on as described in Way A, point 3 on
Hope this helps