Hi Rajkumar,
Make sure first that the groups are removed first from Policies. Then
remove the User Directories which hold the groups. If it's already
done, then I would suggest you to open a Support case as there might
be data corruption in the Policy Store. Do you have any error if you
run command :
XPSSweeper -vT
and check for conflicting objects.
Hope that helps,
Patrick