Out of the box, we have no visibility to see who is making changes to group membership or when it is changed. I suggest that CA adds this functionality to the contact activity log.
This functionality can be added via the following customization, but it would be nice to have it officially supported by CA. I would be more than happy to let CA use the code provided in the document below (should be code reviewed as it was one of my first spel creations).
Tracking Group Membership Changes via Contact Activity Log