Hello Lajosh,
First of all, bear in mind that swap file = configured memory – memory reservation.
Then, once you set a reservation for a VM the ESXi will guarantee access to physical memory for this VM.
About when is applied this reservation, as I am aware, the swap file (.vswp) will change once you power off and power on the VM (at it is created and deleted).
Talking about your example:
ESXi has 64 GB RAM
VM1= Configured 32 GB - Consuming 20 GB - No reservation
VM2= Configured 16 GB - Consuming 16 GB - 16 GB Reserved
Then you change VM1 to:
VM1= Configured 32 GB - Consuming 20 GB -32 GB Reserved
Now ESXi has 64 GB RAM, 48 GB reserved, 16 GB free.
VM3 start configured with 32 GB RAM - no reservation.
Response: As 48 GB of RAM are reserved for other VMs, even if they are not using it, VM3 will only be able to use 16 GB of RAM and then it will swap to disk as no reservation is not configured.
Remember that the swapfile for VM3 will be 32 GB as memory reservation is not configured.
More information: CPU/MEM Reservation Behavior - Yellow Bricks