BIt of an old post but I've just come across this situation and had a bit of a play around with it.
Assuming the user has permissions to modify the VM...
1. If the user also has access to just the portGroup or a Network Folder ('assign network' permissions), they will be able to add a new network adapter and select the desired portGroup.
2. If the user also has access to just the portGroup or a Network Folder ('assign network' permissions), they will not be able to modify an existing network adapter and assign it to a different port group. For this to work, the user needs read-only access to the host (or cluster) where the VM is located.
I guess the process of modifying a network adapter must query the host to see which portGroups it can access, while adding a new network adapter does not. The KB suggests giving access at the Datacenter level, which would in turn give access to the Cluster/Host.
Hope this is of some use.