Cannot SSH into host with right password for root or other accounts

[bassresponse]

I've banged my head on this one for several hours.  No matter what, I cannot SSH from any other machines on the same local network using the root password or any other new users I've created.

This is all with ESXi 5.1 Update 1.

I made sure root and my new user were set up in the Client with Shell Access under User settings, and both are marked as Administrator under Permissions.

Then I have tried every combination of settings in /etc/ssh/sshd_config.  I've tried setting 'PasswordAuthentication yes', 'PubkeyAuthentication no', UsePAM both yes and no, etc.

In between, I use /etc/init.d/SSH restart to reload the changes.  I even started trying restarting the whole server, in case that was not making the changes take hold.  I also tried stopping and starting SSH from Security Profile in the Client.

No matter what, with either user, when I try to log in, I put the right password in, but it acts as if it was wrong, and after the three tries I get the "Permission denied (publickey,password)." error.

The thing is, if I turned off public key auth, shouldn't it just say "(password)" in the final summary error?  I've tried doing -v, and even -vvv, to try to see any other reasons, but I don't see anything useful.  It almost seems as if the changes I am making in the config aren't ever taking effect, even after rebooting.

I have also searched for hours but I can't seem to find anyone who has this problem of using the RIGHT password but not getting access!

Also, I will mention that right now, this host is running standalone, with the Client attaching directly to it (no vCenter Server set up yet - in fact, this whole process is trying to get SSH file transfer abilities so I can upload my Windows ISO, as the upload from browsing datastore in the Client doesn't work for me due to port 902 being blocked).

Is there something really simple I am missing here?

[aravinds3107]

Are you able to login to DCUCI with the same password?

[bassresponse]

Yes, I can log into DCUI and also the client with the same password (for root), and also with the same other username/password that I set up.  That's why it feels like something odd is off that I am simply missing, since I don't see any mention of this problem elsewhere.  Any ideas?

[nielse]

You need to enable SSH on the host using the vSphere client or web client if it's in a vCenter server.

Go to configuration -> security profile -> properties -> SSH and make sure it is running.

Did you do this? You shouldn't edit any config on the host normally...

[bassresponse]

To nielse - SSH is running.  I enabled it via VCUI.  As I mentioned, I tried restarting it from command line and ALSO from the vClient to ensure my config changes took effect.

This is evidenced by the fact that I get the login process, and also by telnetting to port 22, which gets the OpenSSH banner to display.

You are right that ideally no config file changes would be needed, but I thought I had to since it didn't seem to be accepting my password for either account.

[nielse]

Did you check the logs on the ESXi host for any errors? These should help you. You could also try a more debugging login method like

ssh -vvvvvvv root@ip -p 22

[bassresponse]

I didn't see anything in my debugging ssh call, but here are the results (with sensitive info redacted):

macbook-pro:.ssh bassresponse$ ssh -vvvvvvv [e-mail] -p 22

OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011

debug1: Reading configuration data /etc/ssh_config

debug1: /etc/ssh_config line 20: Applying options for *

debug2: ssh_connect: needpriv 0

debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.

debug1: Connection established.

debug3: Incorrect RSA1 identifier

debug3: Could not load "/Users/bassresponse/.ssh/id_rsa" as a RSA1 public key

debug1: identity file /Users/bassresponse/.ssh/id_rsa type 1

debug1: identity file /Users/bassresponse/.ssh/id_rsa-cert type -1

debug3: Incorrect RSA1 identifier

debug3: Could not load "/Users/bassresponse/.ssh/id_dsa" as a RSA1 public key

debug1: identity file /Users/bassresponse/.ssh/id_dsa type 2

debug1: identity file /Users/bassresponse/.ssh/id_dsa-cert type -1

debug1: Remote protocol version 2.0, remote software version OpenSSH_5.2

debug1: match: OpenSSH_5.2 pat OpenSSH*

debug1: Enabling compatibility mode for protocol 2.0

debug1: Local version string SSH-2.0-OpenSSH_5.9

debug2: fd 3 setting O_NONBLOCK

debug3: load_hostkeys: loading entries for host "xxx.xxx.xxx.xxx" from file "/Users/bassresponse/.ssh/known_hosts"

debug3: load_hostkeys: found key type RSA in file /Users/bassresponse/.ssh/known_hosts:39

debug3: load_hostkeys: loaded 1 keys

debug3: order_hostkeyalgs: prefer hostkeyalgs: [e-mail],[e-mail],ssh-rsa

debug1: SSH2_MSG_KEXINIT sent

debug1: SSH2_MSG_KEXINIT received

debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

debug2: kex_parse_kexinit: [e-mail],[e-mail],ssh-rsa,[e-mail],[e-mail],ssh-dss

debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[e-mail]

debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[e-mail]

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[e-mail],hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,[e-mail],hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[e-mail],hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,[e-mail],hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: none,[e-mail],zlib

debug2: kex_parse_kexinit: none,[e-mail],zlib

debug2: kex_parse_kexinit:

debug2: kex_parse_kexinit:

debug2: kex_parse_kexinit: first_kex_follows 0

debug2: kex_parse_kexinit: reserved 0

debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

debug2: kex_parse_kexinit: ssh-rsa,ssh-dss

debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[e-mail]

debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[e-mail]

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[e-mail],hmac-ripemd160,[e-mail],hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[e-mail],hmac-ripemd160,[e-mail],hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: none,[e-mail]

debug2: kex_parse_kexinit: none,[e-mail]

debug2: kex_parse_kexinit:

debug2: kex_parse_kexinit:

debug2: kex_parse_kexinit: first_kex_follows 0

debug2: kex_parse_kexinit: reserved 0

debug2: mac_setup: found hmac-md5

debug1: kex: server->client aes128-ctr hmac-md5 none

debug2: mac_setup: found hmac-md5

debug1: kex: client->server aes128-ctr hmac-md5 none

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP

debug2: dh_gen_key: priv key bits set: 117/256

debug2: bits set: 500/1024

debug1: SSH2_MSG_KEX_DH_GEX_INIT sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY

debug1: Server host key: RSA bc:89:2a:87:8d:cf:45:ca:81:dd:af:30:5b:62:7b:d1

debug3: load_hostkeys: loading entries for host "xxx.xxx.xxx.xxx" from file "/Users/bassresponse/.ssh/known_hosts"

debug3: load_hostkeys: found key type RSA in file /Users/bassresponse/.ssh/known_hosts:39

debug3: load_hostkeys: loaded 1 keys

debug1: Host 'xxx.xxx.xxx.xxx' is known and matches the RSA host key.

debug1: Found key in /Users/bassresponse/.ssh/known_hosts:39

debug2: bits set: 531/1024

debug1: ssh_rsa_verify: signature correct

debug2: kex_derive_keys

debug2: set_newkeys: mode 1

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug2: set_newkeys: mode 0

debug1: SSH2_MSG_NEWKEYS received

debug1: Roaming not allowed by server

debug1: SSH2_MSG_SERVICE_REQUEST sent

debug2: service_accept: ssh-userauth

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug2: key: /Users/bassresponse/.ssh/id_rsa (0x7fca604299e0)

debug2: key: /Users/bassresponse/.ssh/id_dsa (0x7fca6042a2d0)

debug1: Authentications that can continue: publickey,password

debug3: start over, passed a different list publickey,password

debug3: preferred publickey,keyboard-interactive,password

debug3: authmethod_lookup publickey

debug3: remaining preferred: keyboard-interactive,password

debug3: authmethod_is_enabled publickey

debug1: Next authentication method: publickey

debug1: Offering RSA public key: /Users/bassresponse/.ssh/id_rsa

debug3: send_pubkey_test

debug2: we sent a publickey packet, wait for reply

debug1: Authentications that can continue: publickey,password

debug1: Offering DSA public key: /Users/bassresponse/.ssh/id_dsa

debug3: send_pubkey_test

debug2: we sent a publickey packet, wait for reply

debug1: Authentications that can continue: publickey,password

debug2: we did not send a packet, disable method

debug3: authmethod_lookup password

debug3: remaining preferred: ,password

debug3: authmethod_is_enabled password

debug1: Next authentication method: password

[e-mail]'s password:

debug3: packet_send2: adding 48 (len 62 padlen 18 extra_pad 64)

debug2: we sent a password packet, wait for reply

debug1: Authentications that can continue: publickey,password

Permission denied, please try again.

[e-mail]'s password:

debug3: packet_send2: adding 48 (len 62 padlen 18 extra_pad 64)

debug2: we sent a password packet, wait for reply

debug1: Authentications that can continue: publickey,password

Permission denied, please try again.

[e-mail]'s password:

debug3: packet_send2: adding 48 (len 63 padlen 17 extra_pad 64)

debug2: we sent a password packet, wait for reply

debug1: Authentications that can continue: publickey,password

debug2: we did not send a packet, disable method

debug1: No more authentication methods to try.

Permission denied (publickey,password).

[bassresponse]

The only thing notable in syslog is that when I just logged into DCUI, even though the login worked, it did show pam_auth(DCUI:auth) as failing, even though it then says I was logged in.  Is PAM somehow blocking SSH from working?  I did try disabling it, but perhaps my changes in the config file were never taking effect?  (hard to believe not even after a reboot they wouldn't be in effect...)

[bassresponse]

Btw, I used Shift-F to tail one of the logs, so it shows new entries, and displays "Waiting for data... (interrupt to abort)" at the bottom... and now I can't get out!  I've tried any combination I can think of, Shift-Ctrl-C, Z, X, q, Q, Esc - I can still get to the other terminal and to the other logs with Alt-F1 and Alt-F12, but I can't get out of the log.  I tried re-connecting to console via the DRAC card on the server, still no love.  This can't be good if one can get stuck in a log screen like that and only a reboot will correct it?

[bassresponse]

Anyone else have any ideas of something else to try?  After the reboot this morning (since I couldn't get out of the tailed log file in DCUI), still no luck being able to ssh in with either user.

[jjacobu]

Did you ever get this figured out?