How to setup SSH certificates for SSH authentication

SSH Certificate Setup

In this article, we’ll discuss how to configure and set up SSH certificates for SSH access to your servers and cloud resources.

SSH certificate based authentication for SSH access was introduced to address the problems and caveats in SSH public key based authentication.

SSH public key based authentication has several drawbacks and operational challenges that could potentially compromise your organization’s SSH access security. We discussed the drawbacks in great detail in our earlier blog on “Tightening SSH access using short-lived SSH certificates”.

Here is a quick recap of the problems posed by SSH public key based authentication.

[You can skip this section and jump straight to the next section to learn how to configure and setup SSH certificate based authentication in your SSH server.]

1. Copy of an SSH key pair left undeleted in emails or local folders or shared online storages, while copying to the host or the user’s machine.
2. SSH keys don’t have an expiry date or validity period stamped in them. So they can be used perpetually. The technology doesn’t force the admins or users to rotate the keys periodically. As a result, there is no real urge to renew SSH keys periodically. Secondly, rekeying and distributing the keys to servers and user machines at scale requires significant planning and execution. This builds up the inertia against rotating the SSH keys periodically.
3. Some users use the same SSH key pair to login to multiple host machines, potentially increasing the attack surface for any unwanted user who gains access to such an SSH key.
4. When users connect to a server for the first time, weird cryptic messages are shown to users to identify the host key fingerprint and accept it, so that the host key is stored permanently in the ‘known_hosts’ file (~/.ssh/known_hosts) for future references. Mostly, users blindly accept the message and continue with the login anyway, without bothering to know the authenticity of the host they are trying to log in. The only way to address this security problem is to pre-populate the “known_hosts” file with the host public keys of all known hosts that the user would potentially login to. In large organizations, this list is huge and changes over time, which means the ‘known_hosts” file on each user’s machine should be kept updated as well. As a result, most IT teams don’t follow this approach.
5. SSH keys don’t have the hostname or username stamped in them. As a result, any host that is setup with the SSH key could pretend to be the actual host.
6. When users leave the organization, SSH keys assigned to them are usually not deleted from the “authorized_keys” file in all the host machines.

SSH certificate based authentication addresses most of these security problems while simplifying certificate and key management.

In the below sections, we’ll discuss how to configure and set up SSH certificates for SSH access to your servers and cloud resources.

BastionXP Bastion Host solution with built-in CA, automates and simplifies the certificate creation, signing and distribution process. It also automates the ability to rotate certificates when situation warrants. BastionXP Bastion Host CA also issues short-lived user certificates to end users to access cloud resources or on-prem servers. It works seamlessly with OpenSSH server and clients.

Overview:

Here is an overview of what we are about to discuss in this article.

1. We’ll be creating two SSH Certificate Authorities(CA) — a host CA and a user CA, to sign the SSH host certificates and user certificates respectively. (Note: You can create just a single CA to sign both user and host certificates. But it is better to keep them separate, so that you could rotate them separately when the situation warrants.) We’ll show how to configure the host and user machines to trust certificates issued by these host and user CAs.
2. We’ll be creating an SSH host certificate for each host and sign it using the host CA’s private key.
3. We’ll be creating an SSH user certificate for each user and sign it using the user CA’s private key.
4. Finally, we’ll discuss the benefits and drawbacks of using SSH certificate based authentication.

Step 1 — Creating CA certificates

Before we could create the CA certificates, we needed an SSH key pair to work with. We’ll be using the ssh-keygen tool to generate an SSH key pair, as usual. We’ll be executing the following set of commands on the server designated to be the CA.

User CA key pair

# ssh-keygen -t rsa -b 4096 -f ~/.ssh/ssh_user_ca
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/ssh_user_ca
Your public key has been saved in /root/.ssh/ssh_user_ca.pub
The key fingerprint is:
SHA256:ShpCF11/fYjujAVDYivNKpNM6QdVwh8Z9sX00PIspdo
The key’s randomart image is:
+ — -[RSA 4096] — — +
| .oooBo.o+. |
| +o*o* .o=o. |
| . = ..+.= o*+ .|
| . = o o. =o o. |
| . B + S oo. |
| . B . .=Z |
| . . . o |
| |
| |
+ — — [SHA256] — — -+

Host CA key pair

# ssh-keygen -t rsa -b 4096 -f ~/.ssh/ssh_host_ca
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/ssh_host_ca
Your public key has been saved in /root/.ssh/ssh_host_ca.pub
The key fingerprint is:
SHA256:ShpCF11/fYjujAVDYivNKpNM6QdVwh8Z9sX00PIadsfj
The key’s randomart image is:
+ — -[RSA 4096] — — +
| .oooBo.o+. |
| +o*o* .o=o. |
| . = ..+.= o*+ .|
| . = o o. =o o. |
| . B + S oo. |
| . B . .=F |
| . . . o |
| |
| |
+ — — [SHA256] — — -+

We are requesting a RSA type key and the key length to be 4096 bits. The longer the key, the harder it becomes to crack it.

It is highly recommended to provide a passphrase for the private key. If an unauthorized user gets hold of the private key with no passphrase, they will be able to log in to any server you’ve configured with the associated public key.

The above command would have generated an SSH pair — public key and a private key, in the default .ssh folder in your home directory. For example: /home/bob/.ssh/

Step 1.1 — Signing the CA’s Host Certificate

Now, using the following command, let’s create the host CA’s certificate using the host public key and sign it using the corresponding host private key generated in the previous step.

# ssh-keygen -s ~/.ssh/ssh_host_ca -I my-ca -h -n my-ca.example.com -V +52w ~/.ssh/ssh_host_ca.pub

The -s specifies the host private key to be used for signing the certificate. The -h option is used for generating a host type certificate. The -n option above sets the FQDN (Fully Qualified Domain Name) of the host as the principals in the certificate. The -V option sets the validity period of the certificate. In the example above, the certificate will be valid from today and expires one year (52 weeks) from now.

You can verify the details of the certificate using the following command.

# ssh-keygen -Lf ~/.ssh/ssh_host_ca-cert.pub
/root/.ssh/ssh_host_ca-cert.pub:
Type: ssh-rsa host certificate
Public key: RSA-CERT SHA256:7sCdBjn0…
Signing CA: RSA SHA256:7sCdBjn0… (using rsa-sha2–512)
Key ID: “my-ca”
Serial: 0
Valid: from 2022–02–12T10:09:00 to 2023–02–11T10:10:29
Principals:
my-ca.example.com
Critical Options: (none)
Extensions: (none)

Copy the CA’s host certificate to the ‘/etc/ssh/’ folder in the CA server.

Step 1.2 — Signing the CA’s User Certificate

Now, let’s create and sign the user CA certificate using the user CA public private keys.

# ssh-keygen -s ~/.ssh/ssh_user_ca -I root@example.com -n root -V +52w ~/.ssh/ssh_user_ca.pub

Again the user CA’s certificate will be valid for one year (52 weeks) from now. The -h option is not required for the user CA certificate.

Step 1.3 — Distributing and trusting the CA certificates

We need to copy the user CA certificate to all the hosts in the organization. This is to make the hosts trust the user certificate’s signing authority.

Similarly we need to copy the host CA certificate to all the user’s machines. This is to make the ssh clients in the user’s machine trust the host certificate’s signing authority.

Step 1.4 — Making hosts to trust user CA certificate

# scp ~/.ssh/ssh_user_ca-cert.pub root@host1.example.com:/etc/ssh/

Next edit the SSH server config file at /etc/ssh/sshd_config and make the TrustedUserCAKeys directive to point to the user CA certificate we just copied over.

TrustedUserCAKeys /etc/ssh/ssh_user_ca-cert.pub

Restart the host to make the config change to take effect.

# systemctl restart sshd

Step 1.5— Making clients to trust the host CA certificate

For this, you need to copy the contents of the host CA certificate and append it to the `/etc/ssh/ssh_known_hosts` file using the `@cert-authority` directive as shown below.

# cat /etc/ssh/ssh_host_ca-cert.pub
ssh-rsa AAAAHHNzaC1yc2EtY2…. root@example.com

Now copy this content to the SSH global config file /etc/ssh/ssh_known_hosts in all the user systems.

# vi /etc/ssh/ssh_known_hosts
@cert-authority *.example.com ssh-rsa AAAAHHNzaC1yc2EtY2…. root@example.com

The *.example.com wildcard entry above instructs the user systems to trust the host certificates received from any hosts in the example.com domain, signed by the host CA.

Step 2 — Creating SSH Host Certificates

Now we are ready to issue SSH certificates to hosts and user machines using the corresponding CA certificate. Let’s see how to issue the host certificate first.

To create the host certificate, again we needed an SSH key pair to work with. Let’s create one as usual using the ssh-keygen command. Login to the host machine that needs the certificate and execute the below command.

# ssh-keygen -t rsa -b 4096 -f ~/.ssh/ssh_host

The above command will generate a public key (ssh_host.pub) and a private key (ssh_host) in the .ssh folder in the home directory (/root/.ssh/)

You should retain the private key safely within the host system. You should never copy or transfer the file to any other location.

We’ll copy just the host public key the CA server using the `scp` command as shown below:

# scp /root/.ssh/ssh_host.pub root@my-ca:/root/tmp/

Next, create the host certificate from the host public key and sign the certificate using the host CA’s private key.

# ssh-keygen -s ~/.ssh/ssh_host_ca -I host1@example.com -h -n host1.example.com -V +52w /root/tmp/ssh_host.pub

Once signed, copy the host SSH certificate (ssh_host-cert.pub) to the host machine using the `scp` command. It is safe to copy SSH certificates around because they are public objects and can be shared with anyone.

# scp /root/tmp/ssh_host-cert.pub root@host1.example.com:/etc/ssh/

Now configure the host to use the signed host certificate for any connections coming from the SSH clients. Edit the /etc/ssh/sshd_config file and set the `HostCertificate` directive to point to the host certificate as shown below:

# vim /etc/ssh/ssh_config
…
HostCertificate /etc/ssh/ssh_host-cert.pub
…

Finally, restart the SSH server for the configuration changes to take effect.

# systemctl restart sshd

Step 3— Creating SSH User Certificates

First, let’s create an SSH key pair for the user certificate as usual. Execute the following command in the user machine that requires the user certificate.

# ssh-keygen -t rsa -b 4096 -f ~/.ssh/ssh_user

Next, we should copy the user public key to the CA server for signing it. Again, don’t share the private key with anyone or copy the private key to anywhere.

# scp ~/.ssh/ssh_user.pub /root/tmp/.

Login to the CA server and sign the user certificate using the user CA’s private key as shown below:

# ssh-keygen -s ~/.ssh/ssh_user_ca -I bob@example.com -n root -V +4w /root/tmp/ssh_user.pub

Secure copy the signed user certificate back to the user machine.

# scp /root/tmp/ssh_user-cert.pub bob@user1.example.com:~/.ssh/.

Next edit the ssh client config file /etc/ssh/ssh_config(global config file) or ~/.ssh/config file (local config file) in the user’s home directory and add the following line to it, so that SSH client will be able to automatically pick up the user’s private key file and the certificate during authentication.

IdentityFile ~/.ssh/ssh_user

Note that the `IdentityFile` directive is made to point to the user’s private key and not the public key or the certificate.

Try login to the host machine from the user machine to verify that everything is set properly and working just fine

$ ssh bob@host1.example.com

Delete any existing public key for this host from the ~/.ssh/known_hosts file in the user machine, if required, to prevent the SSH client from complaining about the host key has changed. The host public key in the ~/.ssh/known_hosts file is irrelevant for SSH certificate based authentication because it uses the host CA certificate to validate the signature in the host certificate.

Benefits of using SSH certificate based authentication:

1. SSH certificates have a validity period, so user and host certificates expire eventually sometime in the future.
2. User certificates need not be copied to all the host machines
3. Short-lived SSH certificates can be issued to users to access privileged resources.
4. No more weird messages shown by the SSH client during login to the host machines for the first time about recognizing and accepting the host key fingerprint.
5. When a security compromise is detected, invalidate the old CA, create new a CA and issue new certificates to hosts and users. This is called as certificate rotation.

Drawback of using SSH certificate based authentication:

1. Certificates need to be rotated periodically with proper planning and resource allocation
2. Short-lived user certificates are great for privileged access to production resources. However, frequently creating/issuing user certificates and copying them to the user machines is still a laborious process.

BastionXP Bastion Host solution with built-in CA, automates and simplifies the certificate creation, signing and distribution process. It also automates the ability to rotate certificates when situation warrants. BastionXP Bastion Host CA also issues short-lived user certificates to end users to access cloud resources or on-prem servers. Moreover, the solution works seamlessly with OpenSSH server and clients.

So when a privileged user leaves an origanization, no clean up of public keys or certificates is required in all the host machines in your organization. The privileged user’s short-lived certificates would have expired by the time he/she leaves the office on the last day of work.

Download and try BastionXP Bastion Host PKI/CA Solution for free. No credit card required.

To know more about the BastionXP Bastion Host PKI/CA solution or request for a demo, write to us at: support@bastionxp.com

This blog was originally published at: https://www.bastionxp.com