Something most likely got messed up due to a curious admin that was trying to tweak stuff, but ended up breaking stuff accidentally (or maybe they aren't even aware that they broke anything). It happens.
Anyway, I highly suspect the pam.d setting were changed outside the scope of VMware's normal procedures (ie: using the DCUI or vSphere Client). But, without being able to login remotely through SSH, you will have to try to figure it out by speaking to the original admin, or cut your loses at this point and reinstall ESXi.
Reinstalling isn't much of a big deal. Since you don't use vCenter, it's not like you will lose any large amount of performance history, and will only need to recreate the vNetwork configuration, and import your VMs from the datastore(s). You will obviously need to shutdown any running VM's, so a scheduled outage is required.