Troubleshoot SSH SSO Login Problems

Solve problems logging on with SSH to Linux, Unix, and macOS computers running AD Bridge.

Make sure you are joined to the domain by executing the following command as root:

/opt/pbis/bin/domainjoin-cli query

If you are not joined, please see Join Active Directory with the Command Line in the AD Bridge Installation Guide.

You can use the following steps to troubleshoot problems logging on to Linux, Unix, and macOS computers with ssh. It is assumed that the computer is connected to Microsoft Active Directory with AD Bridge and that you are trying to log on with an Active Directory account.

Use NT4-style Credentials and Escape the Slash Character

Try to SSH to the target Linux or Unix computer again with the user's full NT4-style credentials, not the user's alias. In your ssh command, make sure to use a slash character to escape the slash.

Here is an example:

ssh example.com\\kathy@localhost

Perform General Logon Troubleshooting

If you cannot logon after you escape the slash character in your full NT4-style credentials and use your password, execute the general logon troubleshooting steps in Troubleshoot Logon Issues with Systems and Solve Logon Problems on Linux or Unix. If those steps do not help solve the problem, return to this page and perform the following AD Bridge-specific ssh troubleshooting steps in the order listed.

This document contains little general SSH troubleshooting information. If you believe your issue is not specific to AD Bridge or if the information here does not solve your problem, see SSH: The Secure Shell: The Definitive Guide, published by O'Reilly. See especially the following sections:

Get an SSH Log

You should obtain debug logs for the AD Bridge Enterprise authentication service (lsass), PAM, and sshd.

For more information, please see the following:

To get an ssh log, locate sshd and then start it in a separate terminal window with the following options:

`which sshd` -vvv -p 9999 >/tmp/sshd.log 2>&1

The command starts an instance of sshd listening on Port 9999 and routes logging information to a log file in /tmp/sshd.log.

Now try to ssh to the localhost at that port:

ssh -ddd -p 9999 yourADuserName@localhost

When the logon fails, kill ssh; the sshd session will stop as well.

Finally, check the log file at /tmp/sshd.log for information that might help you resolve the issue. In addition, check the log files for lsass and PAM.

For more information on how to generate a log for SSH, please see Logging and Debugging or the man page for ssh.

After an Upgrade, Reconfigure SSH for AD Bridge Enterprise

If ssh was recently upgraded, run the following command as root to make sure that the sshd_config file is set up properly to work with AD Bridge Enterprise:

domainjoin-cli configure --enable ssh

Verify that Port 22 Is Open

A common problem is that a firewall is blocking the port used by SSH. Take a moment to verify that Port 22, which SSH typically connects to, is available by telneting to it. Failure looks like this:

root@example:~# telnet 10.0.0.17 22
Trying 10.0.0.18...
telnet: Unable to connect to remote host: Connection refused

Success looks like this:

root@example:~# telnet 10.0.0.17 22
Trying 10.0.0.17...
Connected to 10.0.0.17.
Escape character is '^]'.
SSH-2.0-OpenSSH_5.1p1 Debian-5
		

Make Sure PAM Is Enabled for SSH

If your Active Directory account is not working with SSH, make sure that UsePAM is enabled in sshd_config and make sure that your sshd application is linked to the PAM libraries.

  1. Determine which sshd is running by executing the following command:
bash-3.2# ps -ef | grep sshd
    root  8199     1  0  Feb  6  ?         0:00 /opt/ssh/sbin/sshd
    root  2987  8199  0  Mar  3  ?         0:04 sshd: root@notty
    root 24864  8199  0 12:16:25 ?         0:00 sshd: root@pts/0
    root  2998  8199  0  Mar  3  ?         0:05 sshd: root@notty
    root 24882 24880  0 12:16:54 pts/0     0:00 grep sshd
		
  1. Either use lsof to find out which configuration file it is reading or start it up with debugging to figure out the default path. Example:
username@computer:~$ /usr/sbin/sshd -dd -t
   debug2: load_server_config: filename /etc/ssh/sshd_config
   debug2: load_server_config: done config len = 664
   debug2: parse_server_config: config /etc/ssh/sshd_config len 664
   debug1: sshd version OpenSSH_5.1p1 Debian-3ubuntu1
   Could not load host key: /etc/ssh/ssh_host_rsa_key
   Could not load host key: /etc/ssh/ssh_host_dsa_key
		
  1. Verify that UsePAM is enabled in the config file. As a best practice, make a backup copy of the configuration file before you change it.
  1. Run ldd on sshd to make sure it links with libpam. Here is an example from an IA64 HP system:
bash-3.2# ldd /opt/ssh/sbin/sshd
	 libpam.so.1 =>  /usr/lib/hpux64/libpam.so.1
	 libdl.so.1 =>   /usr/lib/hpux64/libdl.so.1
	 libnsl.so.1 =>  /usr/lib/hpux64/libnsl.so.1
	 libxnet.so.1 => /usr/lib/hpux64/libxnet.so.1
	 libsec.so.1 =>  /usr/lib/hpux64/libsec.so.1
	 libgssapi_krb5.so =>    /usr/lib/hpux64/libgssapi_krb5.so
	 libkrb5.so =>   /usr/lib/hpux64/libkrb5.so
	 libpthread.so.1 =>      /usr/lib/hpux64/libpthread.so.1
	 libc.so.1 =>    /usr/lib/hpux64/libc.so.1
	 libxti.so.1 =>  /usr/lib/hpux64/libxti.so.1
	 libxti.so.1 =>  /usr/lib/hpux64/libxti.so.1
	 libm.so.1 =>    /usr/lib/hpux64/libm.so.1
	 libk5crypto.so =>       /usr/lib/hpux64/libk5crypto.so
	 libcom_err.so =>        /usr/lib/hpux64/libcom_err.so
	 libk5crypto.so =>       /usr/lib/hpux64/libk5crypto.so
	 libcom_err.so =>        /usr/lib/hpux64/libcom_err.so
	 libdl.so.1 =>   /usr/lib/hpux64/libdl.so.1
		

Make Sure GSSAPI Is Configured for SSH

Logging onto a system with keys does not provide that system with the means of getting a PAC from the domain controller. Without a PAC there is no group membership information for the user. Automated Kerberos ticket renewal will also be unavailable. So, when the ssh login hits the login restrictions in the account phase as it tests for the group memberships, it will not find the user's group information, causing an ssh error like this:

Not in an Allowed Group!

A workaround is to have each user log in once with a password. Subsequent logins with keys should work until the AD cache is flushed, after which the user will have to log in again.

Check the Configuration of SSH for SSO

Although AD Bridge Enterprise automatically configures OpenSSH to support SSO through Kerberos using GSSAPI, it is worthwhile to review how AD Bridge Enterprise does. Since you might need to configure or troubleshoot other applications for SSO, understanding the process will make it easier to apply the technique to other applications.

Not all versions of OpenSSH support Kerberos. Versions older than 4.2p1 might not work or might work improperly.

SSH Service Principal Name

The first thing that needs to be considered is the Kerberos service principal name (SPN) used by ssh and sshd. The SPN is a string that identifies the service for which an authentication ticket is to be generated. In the case of ssh, the SPN has the form:

host/<server name>@<REALMNAME>

For example, when a user uses ssh to connect to a computer named fozzie.mycorp.com, the ssh program requests a service ticket for the SPN:

host/fozzie.example.com@EXAMPLE.COM

The Kerberos realm is the computer's domain name in uppercase letters.

System Keytab Generation

In order for Microsoft Active Directory to generate a Kerberos ticket for this SPN, a service account must exist for it. Additionally, a keytab must be created for the service account and placed on the sshd server. AD Bridge Enterprise completely automates this operation. When a Linux or Unix computer is joined to AD, a machine account is created for the computer. If the computer is called fozzie, a machine account called fozzie$ is created in AD. AD Bridge Enterprise then automatically creates a keytab for the SPN and places it in the standard system location (typically, /etc/krb5.keytab).

User Keytab Generation

When the user runs the ssh program and OpenSSH determines that it will use Kerberos authentication, it will need to access a keytab for the user so that it can obtain a service ticket for the service or computer to which it is trying to connect. This keytab must be created using the user's account name and password. Manually, this can be performed by using the kinit utility. AD Bridge Enterprise, however, does it automatically when the user logs on the computer. On most systems, the user keytab is placed in the /tmp directory and named krb5cc_UID where UID is the numeric user ID assigned by the system.

Configure OpenSSH

AD Bridge Enterprise automatically configures OpenSSH at both the client and server computer. On the client, the ssh_config file (typically in /etc/ssh/ssh_config) is modified. On the server, ssh_config (typically in /etc/ssh/ssh_config) is modified. AD Bridge Enterprise adds the following lines of code to the right files if they are not already present and if they are required by the system's version of sshd:

In the server, the following lines must be present in sshd_config. if you are troubleshooting, make sure these lines are there:

GSSAPIAuthentication yes
GSSAPICleanupCredentials yes

On the client, the following line must be present in ssh_config:

GSSAPIAuthentication yes

On the client, GSSAPIDelegateCredentials yes is an optional setting that instructs the ssh client to delegate the krb5 TGT to the destination machine when SSH single sign-on is used.

In addition, if any of the following options are valid for the system's version of sshd, they are required and configured by AD Bridge Enterprise:

ChallengeResponseAuthentication yes
UsePAM yes
PAMAuthenticationViaKBDInt yes
KbdInteractiveAuthentication yes

Setting these options to yes instructs ssh to use the kbdinteractive ssh authentication mechanism and allows that mechanism to use PAM, settings that are required for AD Bridge Enterprise to function properly.

For more information, see the man pages for ssh, sshd, and the comments in the ssh and sshd configuration files.

Test SSO

With OpenSSH properly configured, demonstrating SSO support is simple: Log on a Linux or Unix machine running AD Bridge Enterprise by using your Active Directory credentials and then use ssh to connect to another machine that is also running AD Bridge Enterprise. OpenSSH should establish a connection without prompting for a username or password.

Platform-Specific Issues

If you are using Red Hat, CentOS, Fedora, FreeBSD, or AIX operating systems, review any of the following sections that are relevant for your operating system.

Red Hat and CentOS: Solve the SSO Problem

There is a known bug with some versions of Red Hat and CentOS that prevents SSO from working with SSH, SSHD, and PuTTY. The following versions are known to be affected:

  • CentOS 5
  • Red Hat Enterprise Linux 5

The system incorrectly concatenates the Kerberos ticket's service principal name on the target Linux computer. For example, in the final entry of the results of the klist command below, the full name of the service principal is cut off after the @ symbol:

[EXAMPLE\fanthony@centos52 ~]$ /opt/pbis/bin/klist
Ticket cache: FILE:/tmp/krb5cc_1689257039
Default principal: fanthony@EXAMPLE.COM
Valid starting     Expires            Service principal
07/31/08 09:25:13  07/31/08 19:25:31  krbtgt/EXAMPLE.COM@EXAMPLE.COM
    renew until 08/07/08 09:25:13
07/31/08 09:25:31  07/31/08 19:25:31  CENTOS52$@EXAMPLE.COM
    renew until 08/07/08 09:25:13
07/31/08 09:30:04  07/31/08 19:25:31  host/centos52.example.com@
    renew until 08/07/08 09:25:13

To determine whether you need to implement the solution below on your Red Hat or CentOS computer, execute the following series of tests:

  1. Connect to your target machine with SSH by using PuTTY and a valid Active Directory user. Be sure to use the FQDN of the host.
  2. Execute the following command:

    /opt/pbis/bin/klist

    The results should look like this:

    EXAMPLE\fanthony@centos52 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_1689257039
    Default principal: fanthony@EXAMPLE.COM
    Valid starting     Expires            Service principal
    07/31/08 09:25:13  07/31/08 19:25:31  krbtgt/EXAMPLE.COM@EXAMPLE.COM
        renew until 08/07/08 09:25:13
    07/31/08 09:25:31  07/31/08 19:25:31  CENTOS52$@EXAMPLE.COM
        renew until 08/07/08 09:25:13
  3. SSH again to the same host and when prompted for the password, type CTRL+C.
  4. Execute the klist command again:

    /opt/pbis/bin/klist

  5. Check the results to determine whether there is an incorrectly concatenated service principal, as there is in the following output:
    [EXAMPLE\fanthony@centos52 ~]$ klist
    Ticket cache: FILE:/tmp/krb5cc_1689257039
    Default principal: fanthony@EXAMPLE.COM
    Valid starting     Expires            Service principal
    07/31/08 09:25:13  07/31/08 19:25:31  krbtgt/EXAMPLE.COM@EXAMPLE.COM
        renew until 08/07/08 09:25:13
    07/31/08 09:25:31  07/31/08 19:25:31  CENTOS52$@EXAMPLE.COM
        renew until 08/07/08 09:25:13
    07/31/08 09:30:04  07/31/08 19:25:31  host/centos52.example.com@
        renew until 08/07/08 09:25:13

If the tests confirm that the problem exists, implement the following solution:

  1. On Red Hat Enterprise Linux 5, make sure that the reverse PTR host definitions are defined in DNS.
  2. On the target Linux computer, add the following line to /etc/krb5.conf under the [domain_realm] entry of the file:

    .yourdomainname.com = YOURDOMAINNAME.COM

    Example:

    [domain_realm]
    .example.com = EXAMPLE.COM
  3. Restart SSHD by running the following command at the shell prompt:

    /sbin/service sshd restart

Red Hat and Fedora: Solve SSH Config Problem

On Fedora 14 and Red Hat 5, there is an issue with the configuration of the platform that blocks SSH SSO. You must either use a workaround to connect to the client or modify the sshd_config file on the server side. This section illustrates the problem and shows you how to connect to the client or fix the server.

After you join a domain with AD Bridge Enterprise, Network Manager restarts and leaves the /etc/hosts file looking like this:

[root@nile-fedora14 etc]# cat /etc/hosts
10.100.0.26 nile-fedora14.nile-domain.example.com nile-fedora14 # Added by NetworkManager
127.0.0.1 localhost.localdomain localhost localhost4
::1 nile-fedora14.nile-domain.example.com nile-fedora14 localhost6 nile-fedora14.ramp.example.com

It should, however, look like this, but Network Manager keeps resetting it:

10.100.0.26  nile-fedora14.nile-domain.example.com  nile-fedora14   # Added by NetworkManager
127.0.0.1  nile-fedoar14.nile-domain.example.com nile-fedora14 localhost.localdomain  localhost  localhost4
::1  nile-fedora14.nile-domain.example.com  nile-fedora14   localhost6.localdomain6 localhost6

The configuration set by Network Manager blocks SSO because it ends up restricting reverse name lookups to ipv4 only.

When using the client, you can work around the problem by connecting by the external IP address. In other words, instead of using ssh -l user nile-fedora14.nile-domain.example.com to connect, use the following form:

ssh -l user 10.100.0.26

Alternatively, to fix the problem, you can turn off GSSAPIStrictAcceptorCheck in sshd_config on the server, but such a resolution might be unavailable when you do not have administrative access to the server or when doing so might cause intractable side effects or security holes.

Another way to fix the problem is to turn off reverse DNS lookups in Kerberos. However, such a solution might result in side effects that block other applications or operations.

FreeBSD: Invalid Argument with SSHD

On FreeBSD, user names that are longer than 16 characters, including the domain name, exceed the FreeBSD username length limit. Attempts to connect by ssh with a user name that exceeds the limit can result in the following notification:

bvt-fbs72-64# ssh testuser1@localhost
Password:
Connection to localhost closed by remote host.
Connection to localhost closed.
		

The log for sshd, meanwhile, might show an error that looks something like this:

Oct  7 18:22:57 vermont02 sshd[66387]: setlogin(EXAMPLE\adm.kathy): 
Invalid argument
Oct  7 18:25:02 vermont02 sshd[66521]: setlogin(EXAMPLE\adm.kathy): 
Invalid argument

Although testuser1 is less than 16 characters, when you use the id command to check the account, something longer than 16 characters is returned:

[root@bvt-fbs72-64 /home/testuser]# id testuser1
uid=1100(BVT-FBS72-64\testuser1) gid=1801(BVT-FBS72-64\testgrp)
groups=1801(BVT-FBS72-64\testgrp)

The result of the id command exceeds the FreeBSD username length limit. There are several solutions:

  • Set the default domain.
  • Change the user name to 16 characters or less.
  • With AD Bridge Enterprise use aliases.

Keep in mind, however, that aliases will not solve the problem in relation to the AD Bridge Enterprise local provider.

AIX and Red Hat: Set Reverse PTR Host Definitions for SSO

For single sign-on with SSH to work on Red Hat Enterprise Linux 5 and AIX, reverse PTR host definitions must be set in DNS.

AIX: Configure for Outbound Single Sign-On

On AIX 5.3, client-side SSH is not set up by default. Here is how to configure it so that it will work with AD Bridge Enterprise:

  1. On your AIX 5.3 computer, make sure the network authentication service, version 1.4.0.8, is installed; example:
    -bash-3.00$ lslpp -l | grep krb
    krb5.client.rte 1.4.0.8 COMMITTED Network Authentication Service

If the service is not installed, obtain it from the IBM AIX website and install it.

  1. After joining an Active Directory domain with AD Bridge Enterprise, append the following lines to the end of /etc/krb5/krb5.conf:
    [domain_realm]
    .demo.example.com = DEMO.EXAMPLE.COM
    demo.example.com = DEMO.EXAMPLE.COM
  2. Make sure that /etc/krb5/krb5.conf links to /etc/krb5.conf.
  3. Also make sure that /etc/krb5/krb5.keytab links to /etc/krb5.keytab.
  4. Make a backup of the credentials directory by executing the following command as root:

    mv /var/krb5/security/creds /var/krb5/security/creds_old

  5. As root, make a symbolic link to the /tmp directory so that the AIX Kerberos libraries can access the directory in which AD Bridge Enterprise stores its credential caches:

    ln -s /tmp /var/krb5/security/creds

  6. Open /etc/environment, which contains the list of environmental variables that are set when a user logs on, and add the following line to the end of it:

    KRB5_CONFIG=/var/lib/pbis/krb5-affinity.conf:/etc/krb5.conf

  7. If you are logged on the machine whose environmental variable you changed, you must log off and log on again for the change to take effect.