Can't use ssh publickey, but only for a single host

Question

I've been using publickeys for a long time to connect my laptop to my server, but lately I can't connect when I'm in this place only.
It is the same key and it works perfectly, except when I'm in this specific network. And it doesn't seem to be a firewall issue, because the remote server actually logs the attempt.
I'm all out of ideas. Nothing I try has any effect.
ssh -vvv
OpenSSH_6.1p1, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /home/<user>/.ssh/config
debug1: /home/<user>/.ssh/config line 14: Applying options for <host>
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to <host> port 443.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/<user/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/<user>/.ssh/id_rsa type 1
debug1: identity file /home/<user>/.ssh/id_rsa-cert type -1
debug1: identity file /home/<user>/.ssh/id_dsa type -1
debug1: identity file /home/<user>/.ssh/id_dsa-cert type -1
debug1: identity file /home/<user>/.ssh/id_ecdsa type -1
debug1: identity file /home/<user>/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.1
debug1: match: OpenSSH_6.1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.1
debug2: fd 3 setting O_NONBLOCK
debug3: put_host_port: <host>:443
debug3: load_hostkeys: loading entries for host "<host>:443" from file "/home/<user>/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/<user>/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: [email protected],[email protected],ssh-rsa,[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: [email protected],zlib,none
debug2: kex_parse_kexinit: [email protected],zlib,none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 [email protected]
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 [email protected]
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: <host key>
debug3: put_host_port: <host>:443
debug3: put_host_port: <host>:443
debug3: load_hostkeys: loading entries for host "<host>:443" from file "/home/<user>/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/<user>/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug3: load_hostkeys: loading entries for host "<host>:443" from file "/home/<user>/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/<user>/.ssh/known_hosts:11
debug3: load_hostkeys: loaded 1 keys
debug1: Host '<host>:443' is known and matches the RSA host key.
debug1: Found key in /home/<user>/.ssh/known_hosts:1
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/<user>/.ssh/id_rsa (0x1d61230)
debug2: key: /home/<user>/.ssh/id_dsa ((nil))
debug2: key: /home/<user>/.ssh/id_ecdsa ((nil))
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/<user>/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
and this is the log from journalctl
Mar 27 09:38:29 xen sudo[29258]: pam_unix(sudo:session): session closed for user root
Mar 27 09:38:32 xen sshd[29196]: debug1: Forked child 590.
Mar 27 09:38:32 xen sshd[590]: Set /proc/self/oom_score_adj to 0
Mar 27 09:38:32 xen sshd[590]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Mar 27 09:38:32 xen sshd[590]: debug1: inetd sockets after dupping: 3, 3
Mar 27 09:38:32 xen sshd[590]: Connection from <host> port 54330
Mar 27 09:38:32 xen sshd[590]: debug1: Client protocol version 2.0; client software version OpenSSH_6.1
Mar 27 09:38:32 xen sshd[590]: debug1: match: OpenSSH_6.1 pat OpenSSH*
Mar 27 09:38:32 xen sshd[590]: debug1: Enabling compatibility mode for protocol 2.0
Mar 27 09:38:32 xen sshd[590]: debug1: Local version string SSH-2.0-OpenSSH_6.1
Mar 27 09:38:32 xen sshd[590]: debug1: permanently_set_uid: 99/99 [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256 [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: SSH2_MSG_KEXINIT received [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: kex: client->server aes128-ctr hmac-md5 [email protected] [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: kex: server->client aes128-ctr hmac-md5 [email protected] [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: SSH2_MSG_NEWKEYS received [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: KEX done [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: userauth-request for user <user> service ssh-connection method none [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: attempt 0 failures 0 [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: PAM: initializing for "<user>"
Mar 27 09:38:32 xen sshd[590]: debug1: PAM: setting PAM_RHOST to "<host>"
Mar 27 09:38:32 xen sshd[590]: debug1: PAM: setting PAM_TTY to "ssh"
Mar 27 09:38:32 xen sshd[590]: debug1: userauth-request for user <user> service ssh-connection method publickey [prea
Mar 27 09:38:32 xen sshd[590]: debug1: attempt 1 failures 0 [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: test whether pkalg/pkblob are acceptable [preauth]
Mar 27 09:38:32 xen sshd[590]: debug1: temporarily_use_uid: 1000/100 (e=0/0)
Mar 27 09:38:32 xen sshd[590]: debug1: trying public key file /home/<user>/.ssh/authorized_keys
Mar 27 09:38:32 xen sshd[590]: debug1: fd 4 clearing O_NONBLOCK
Mar 27 09:38:32 xen sshd[590]: debug1: matching key found: file /home/<user>/.ssh/authorized_keys, line 1
Mar 27 09:38:32 xen sshd[590]: Found matching RSA key: <key>
Mar 27 09:38:32 xen sshd[590]: debug1: restore_uid: 0/0
Mar 27 09:38:32 xen sshd[590]: Postponed publickey for <user> from <host> port 54330 ssh2 [preauth]
And it just hangs there forever.
If I try to use the same key to log in to other servers (one debian and an openWRT router) it works just fine from this location. This key also works to log in to the archlinux server if I'm on other networks, such as my college's or my other workplace.
Any help is very appreciated. As it stands, I have to log in to my debian server and from there I have to log in to my archlinux server. I wish I could just log in directly as I've done many times in the past.

Answer 1

Hi,
I don't know whether you've solved the issue in the meantime (your issue was three months ago), but I thought I'd post this reply just to let other people who run into this problem know what happened to me and how I solved it.
I had exactly the same problem as you had. The logging in stopped at exactly the same point in the debugging trace. I too was experiencing the problem from a single network (multiple hosts on the network could not log onto a remote server). Logging into the remote server from other locations (even with the same laptop) worked fine.
The problem turned out to be an MTU problem in my case. I was running an OpenVPN connection to the remote server, and I was logging into it over the VPN. No problems, usually, except for the fact that the network that I was logging in from is a glass fiber network using PPPoE. The MTU on that link is 1460 bytes, not the 1500 bytes that is more common. I had to reconfigure the OpenVPN interfaces (using the OpenVPN configuration options "mssfix 1360", "fragment 1360" and "tun-mtu 1400" on both sides of the connection) to use a smaller MTU on the OpenVPN tunX interface, and everything started working normally again.
Obviously, the MTU was wrong for every connection going over the VPN, but the OpenVPN tunnel was somewhat resistant to this mistake because I turned on LZO compression, which made most packets that were transmitted over the OpenVPN interface (tunX) that were using an MTU of 1500 bytes smaller than the maximum allowed on the actual link. Anyway, the lesson is: set up your MTUs on your links correctly. And turn on "mssfix" so that if you're routing remote hosts' traffic over the OpenVPN tunnel, their TCP stacks will be made aware of the actual MTU of the link.
Hope this helps somebody,
Sven