Hi Srujan,
So after more testing it looks like it is NOT possible to fulfill your specific requirements in 12.0; however, it is possible in 12.1.
In 12.1 the user only needs the privileges "Access to Administration" and "Access to System Overview," and no other authorizations or privileges are necessary.
In 12.0 if you give the same exact permissions as above, then you cannot delete the Agent. Adding just the "Access to Process Assembly" privilege on top of the other two privileges allows the delete button to be enabled. So it looks like in 12.0 the action is tied with this privilege.
As a workaround in 12.0, you could give the user the process assembly privilege but then use the NOT authorization for all objects.
This way the user technically has access to the perspective, but has no access to the actual objects.
Best Regards,
-Leah