That's correct - the permissions tab currently has either "everyone can see/request this", or "only members of the following groups/roles can". While it's not possible to say "all members of this group/role -except- for this individual has access", or other more complex boolean sets, what you could do is create a custom group within EEM or the directory you're importing into EEM from that does contain the people you're after, and use the tools available there.
My one caution here, however, is that the more groups that EEM is aware of that a user belongs to, the longer that permission check takes. So I'd recommend avoiding a situation where a user could be in tens or hundreds, to ensure they have a good performance experience.