Gfrktyl

I have two users on desktop - root and user. I have a bastion and a protected host. When run ssh protected as root on desktop, I connect fine. When I run ssh protected as user on desktop, I get no output - just a blank line, like it's waiting for something. However, user can log in directly to the bastion host and from there to the protected host.

Both root and user have the same contents in their .ssh directories (#cp -r ~/.ssh /home/user; chown -R user:user /home/user/.ssh).

The bastion host appears to be forwarding properly - running $(which sshd) -Ddp 10222 (per https://unix.stackexchange.com/a/128910/9583) shows the same debug1: channel 0: connected to protected port 22 line on both.

Running the same on protected shows the same output until:

debug1: SSH2_MSG_KEXINIT sent [preauth]

debug1: SSH2_MSG_KEXINIT received [preauth]

The second line does not display when connecting from user on desktop.

ssh -vvv protected as user on desktop shows:

OpenSSH_7.9p1, OpenSSL 1.1.1  11 Sep 2018                                                                                                                       

debug1: Reading configuration data /home/user/.ssh/config                                                                                                     

debug1: /home/user/.ssh/config line 1: Applying options for *                                                                                                

debug1: /home/user/.ssh/config line 10: Applying options for protected                                                                                            

debug1: Reading configuration data /etc/ssh/ssh_config                                                                                                          

debug1: Executing proxy command: exec ssh bastion -W protected:22                                                                                                

debug1: identity file /home/user/.ssh/id_protected type 0                                                                                                         

debug1: identity file /home/user/.ssh/id_protected-cert type -1                                                                                                   

debug1: Local version string SSH-2.0-OpenSSH_7.9                                                                                                                

debug1: ssh_exchange_identification: 33[3g

33H    33H    33H    33H    33H    33H    33H    33H     33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H    33H

SSH-2.0-Op

debug1: ssh_exchange_identification: enSSH_7.9





debug1: ssh_exchange_identification:

debug1: ssh_exchange_identification: 6,diffie-hellman-group14-sha1

debug1: ssh_exchange_identification: aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com                                                       

debug1: ssh_exchange_identification: 2-256,hmac-sha2-512,hmac-sha1

As root on desktop everything is the same up until the first ssh_exchange_identification line.

My ssh config is:

Host *

ServerAliveInterval 60

IdentitiesOnly yes



Host           bastion

HostName       bastion.host

IdentityFile   ~/.ssh/id_protected

User           user



Host          protected

IdentityFile   ~/.ssh/id_protected

Hostname       protected

User           user

ProxyCommand   ssh bastion -W %h:%p

I have also tried https://askubuntu.com/a/976226/427339, but I believe this doesn't apply for two reasons - 1. emptying my ~/.config/fish/fish.config made no difference, and 2. I can log in to the same user on protected from root on desktop.

All three systems are running Arch Linux. protected and desktop are both using the fish shell.

Edit:

user@bastion's ~/.ssh/config:

Host *

ServerAliveInterval 60



Host protected

User user

IdentityFile ~/.ssh/id_protected

This, as mentioned above, works fine to log into protected. /etc/hosts has an entry for protected pointing to the net-local IP - 10.x.x.x.

Edit 2:

My issue appears very similar to these:

• https://groups.google.com/d/topic/comp.security.ssh/e1nObaX5ZWg/discussion


• https://groups.google.com/d/topic/comp.security.ssh/_HDV0JXXQA8/discussion


• https://groups.google.com/d/topic/comp.security.ssh/tDgwEDJKGuE/discussion

I have not yet tried the MTU workaround, and am not familiar enough with protocol analyzers to have one set up and handy right now.

Edit 3:

Adding -v to the ProxyCommand (is now ProxyCommand ssh -v bastion -W %h:%p), full output of user@desktop$ ssh protected:

OpenSSH_7.9p1, OpenSSL 1.1.1  11 Sep 2018

debug1: Reading configuration data /home/user/.ssh/config

debug1: /home/user/.ssh/config line 1: Applying options for *

debug1: /home/user/.ssh/config line 5: Applying options for bastion

debug1: Reading configuration data /etc/ssh/ssh_config

debug1: Connecting to bastion [x.x.x.x] port 22.

debug1: Connection established.

debug1: identity file /home/user/.ssh/id_protected type 0

debug1: identity file /home/user/.ssh/id_protected-cert type -1

debug1: Local version string SSH-2.0-OpenSSH_7.9

debug1: Remote protocol version 2.0, remote software version OpenSSH_7.8

debug1: match: OpenSSH_7.8 pat OpenSSH* compat 0x04000000

debug1: Authenticating to bastion:22 as 'user'

debug1: SSH2_MSG_KEXINIT sent

debug1: SSH2_MSG_KEXINIT received

debug1: kex: algorithm: curve25519-sha256

debug1: kex: host key algorithm: ecdsa-sha2-nistp256

debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none                                                             

debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none                                                             

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY                                                                                                                       

debug1: Server host key: ecdsa-sha2-nistp256 SHA256:HfDmNOGgLrPLMsnCbyZuEuJapj+T6wrSTTiFSd+37ag                                                                 

debug1: Host 'bastion' is known and matches the ECDSA host key.                                                                                           

debug1: Found key in /home/users/.ssh/known_hosts:3                                                                                                           

debug1: rekey after 134217728 blocks

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug1: SSH2_MSG_NEWKEYS received

debug1: rekey after 134217728 blocks

debug1: Will attempt key: /home/user/.ssh/id_protected RSA SHA256:iH5F4stK+j+2/qkGJlL5D6TOEHNiwbR4jCzckI7IHaE explicit agent                                   

debug1: SSH2_MSG_EXT_INFO received

debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug1: Authentications that can continue: publickey,password

debug1: Next authentication method: publickey

debug1: Offering public key: /home/user/.ssh/id_protected RSA SHA256:iH5F4stK+j+2/qkGJlL5D6TOEHNiwbR4jCzckI7IHaE explicit agent                                

debug1: Server accepts key: /home/user/.ssh/id_protected RSA SHA256:iH5F4stK+j+2/qkGJlL5D6TOEHNiwbR4jCzckI7IHaE explicit agent                                 

debug1: Authentication succeeded (publickey).

Authenticated to bastion ([x.x.x.x]:22).

debug1: channel_connect_stdio_fwd protected:22

debug1: channel 0: new [stdio-forward]

debug1: getpeername failed: Bad file descriptor

debug1: Requesting no-more-sessions@openssh.com

debug1: Entering interactive session.

debug1: pledge: network

debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0

debug1: Remote: /home/user/.ssh/authorized_keys:3: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding

debug1: Remote: /home/user/.ssh/authorized_keys:3: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding

--- very long delay; from `root` everything is the same till here, but the next line is `Last login: ...`, etc - a successful connection ---

debug1: channel 0: FORCE input drain

ssh_exchange_identification: Connection closed by remote host

debug1: channel 0: free: direct-tcpip: listening port 0 for protected port 22, connect from 127.0.0.1 port 65535 to UNKNOWN port 65536, nchannels 1

debug1: fd 0 clearing O_NONBLOCK

Killed by signal 1.

Since your protected host is behind a bastion host, you still need to go through the bastion host in order to reach it whether from a root user or normal user account.

Simply duplicate the root ssh config to the user ssh config and you should be good to go, but add a ForwardAgent yes in each Host section for both the root user and your normal user on your desktop.

Host *

ServerAliveInterval 60

IdentitiesOnly yes



Host           bastion

HostName       bastion.host

IdentityFile   ~/.ssh/id_protected

User           user

ForwardAgent   yes



Host          protected

IdentityFile   ~/.ssh/id_protected

Hostname       protected

User           user

ForwardAgent   yes

ProxyCommand   ssh bastion -W %h:%p

The sshd_config file on the bastion as well as the protected host, usually in /etc/ssh/sshd_config, will need to have AllowAgentForwarding yes which is usually the default but worth checking. If you do have to change, then restart sshd after saving.

Hopefully the private keys are the same between root and normal user, but if not, ensure the corresponding public keys are in the .ssh/authorized_keys file on the bastion host as well as the protected host. In other words, the bastion users' .ssh/authorized_keys file has an entry (single line, 2 or 3 space-separated fields) containing the public keys of the corresponding private ones used from your desktop and so does the protected host.

If you don't have the public keys any longer, you can get them via this command:

ssh-keygen -l -f /path/to/private_key

I ran into a very similar problem shelling from laptop (not mentioned in question - I also have pi and several other computers not relevant to the issue) to desktop - which is set up very similarly to protected. The symptom:

user@laptop$ ssh desktop "echo foo" | xxd

00000000: 1b5b 3367 0d1b 4820 2020 201b 4820 2020  .[3g..H    .H

00000010: 201b 4820 2020 201b 4820 2020 201b 4820   .H    .H    .H

00000020: 2020 201b 4820 2020 201b 4820 2020 201b     .H    .H    . 

00000030: 4820 2020 201b 4820 2020 201b 4820 2020  H    .H    .H

00000040: 201b 4820 2020 201b 4820 2020 201b 4820   .H    .H    .H

00000050: 2020 201b 4820 2020 201b 4820 2020 201b     .H    .H    .

00000060: 4820 2020 201b 4820 2020 201b 4820 2020  H    .H    .H

00000070: 201b 4820 2020 201b 4820 2020 0d66 6f6f   .H    .H   .foo

00000080: 0a

This was easily fixed by changing the line tabs -4 to status --is-interactive; and tabs -4 in /home/user/.config/fish/config.fish on desktop. This change I have already made some time ago on protected.

However - changing the /home/user/.config/fish/config.fish on desktop appears to have fixed the login from user@desktop to user@protected. I am posting this as an answer as I feel the resolution is somewhat novel - https://askubuntu.com/a/976226/427339 which I linked in the question states that I need to check for commands producing output on protected, and I resolved this by changing a command which produces output on desktop.

That said, I would love to mark as accepted an answer which explains how and why my shell configuration on the local machine interfered with the ssh pipe, but not direct ssh connections.

Until then, I have come to the tentative conclusion that tabs is a risky program, and should be treated with the utmost of caution if placing it in a shell startup file.