I do not believe you can set a EEM policy up so that someone has access to all tasks except for 3 groups worth. I am not seeing anything that specifically limits access to a subset of tasks. You maybe able to accomplish this through the use of filtered permissions, but this would be difficult to initially build, and I am still not seeing anything that leads me to believe you could allow access to a partial list of tasks.
You can review the EEM documentation here on building policy:
Administer Advanced CA EEM Security
And the Permissions reference here:
Permissions Reference
Thinking on how this could be accomplished, I would create a specific 'Task Admin' group of normal users, assign these 'task admin' users to that group, and assign that group to all tasks except for the "3 groups worth"
This would give these task admin group users access to all but the 3 groups of tasks you do not want them to access.