ssh login slow but vSphere Client very responsive - why?

[rfreytag]

I have been running an ESXi 4.0 server for months with a couple of WinServer2003 and several Ubuntu Server 10.4 VMs.  The performance has been impressive on 6GB i7 Asus P6T hardware.  Suddenly, a week ago, the Ubuntu VMs take 8 minutes to log in over ssh when connecting over the LAN.  The password prompt arrives immediately, and failed passwords are responded to immediately, but the moment I log in the prompt appears and I hang for many minutes. 

This behavior is NEVER seen when I connect with vSphere Client.  Using vSphere Client the Ubuntu VMs all respond beautifully. 

I have seen: http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1003496&sliceId=1&docTypeID=DT_KB_1_1&dialogID=229586372&stateId=1%200%20229588522

Any other suggestions very welcome - thank you.

[vlho]

Hi,

you check setup of DNS (/etc/resolve.conf)

Ubuntu search reverse record of your management PC...

Or look here: http://blog.arunstephens.com/2008/11/21/ssh-slow-to-login-disable-reverse-dns-lookup/

[rfreytag]

It doesn't look like it is a reverse DNS lookup issue.  Tested your suggestion of course and it doesn't fit with the fact that some of the VMs are responsive.  So I'm still stumped. 

[iw123]

whats the performance like when initiating a connection from a ubuntu VM? or from one ubuntu VM to another?

[rfreytag]

Good idea - just tested that.

From one Ubuntu VM into another the result is a 'hang' as described.  To other Ubuntu VMs (same OS version 10.04) the performance is snappy and responsive. 

I'm beginning to think that the problem might have little/nothing to do with ESXi. 

[vlho]

you look here: http://realtechtalk.com/SSH_Slow_Login_even_with_SSHD_UseDNS_no_parameter-116-articles

[rfreytag]

Thanks 'vlho'.

I see I was not clear enough.  I do not seem to be having a DNS problem as described since they seem to talk about the LOGIN prompt being delayed.  I >do< get a password prompt - promptly.  Often I get a prompt after I have logged in.

But the shell "hangs" for 10 minutes.  After the 10 minutes the shell works.  If I'm on the LAN the connection will stay up for the 10 minutes.  Of course, if I'm coming in from outside the connection is terminated for inactivity after a few minutes (<10 minutes).

Do you agree?

********************************************************

Here is the verbose ssh login information in case that helps:

********************************************************

development-machine:~ development-machine$ ssh -v -l myname myhost.com  -p 9005

OpenSSH_5.6p1, OpenSSL 0.9.8r 8 Feb 2011

debug1: Reading configuration data /etc/ssh_config

debug1: Applying options for *

debug1: Connecting to myhost.com [92.172.33.43] port 9005.

debug1: Connection established.

debug1: identity file /Users/development-machine/.ssh/id_rsa type -1

debug1: identity file /Users/development-machine/.ssh/id_rsa-cert type -1

debug1: identity file /Users/development-machine/.ssh/id_dsa type -1

debug1: identity file /Users/development-machine/.ssh/id_dsa-cert type -1

debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3p1 Debian-3ubuntu7

debug1: match: OpenSSH_5.3p1 Debian-3ubuntu7 pat OpenSSH*

debug1: Enabling compatibility mode for protocol 2.0

debug1: Local version string SSH-2.0-OpenSSH_5.6

debug1: SSH2_MSG_KEXINIT sent

debug1: SSH2_MSG_KEXINIT received

debug1: kex: server->client aes128-ctr hmac-md5 none

debug1: kex: client->server aes128-ctr hmac-md5 none

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP

debug1: SSH2_MSG_KEX_DH_GEX_INIT sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY

debug1: Host '[myhost.com]:9005' is known and matches the RSA host key.

debug1: Found key in /Users/development-machine/.ssh/known_hosts:11

debug1: ssh_rsa_verify: signature correct

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug1: SSH2_MSG_NEWKEYS received

debug1: Roaming not allowed by server

debug1: SSH2_MSG_SERVICE_REQUEST sent

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug1: Authentications that can continue: publickey,password

debug1: Next authentication method: publickey

debug1: Trying private key: /Users/development-machine/.ssh/id_rsa

debug1: Trying private key: /Users/development-machine/.ssh/id_dsa

debug1: Next authentication method: password

[e-mail] password:

debug1: Authentications that can continue: publickey,password

Permission denied, please try again.

[e-mail] password:

debug1: Authentication succeeded (password).

Authenticated to myhost.com ([92.172.33.43]:9005).

debug1: channel 0: new [client-session]

debug1: Requesting [e-mail]

debug1: Entering interactive session.

Write failed: Broken pipe

development-machine:~ development-machine;

Message was edited by: rfreytag

[iw123]

is it any quicker if you connect via IP address rather then by hostname ?

[rfreytag]

@lw123 - thanks for your question. 

No it is not any faster with the IP# because the problem happens after the password has been accepted.  Sometimes it even gives me the shell prompt and accepts a command - then it hangs.