Speed up SSH operations

The problem

SSH operations become slow as the number of users grows.

The reason

OpenSSH searches for a key to authorize a user via a linear search. In the worst case, such as when the user is not authorized to access GitLab, OpenSSH will scan the entire file to search for a key. This can take significant time and disk I/O, which will delay users attempting to push or pull to a repository. Making matters worse, if users add or remove keys frequently, the operating system may not be able to cache the authorized_keys file, which causes the disk to be accessed repeatedly.

The solution

GitLab Shell provides a way to authorize SSH users via a fast, indexed lookup to the GitLab database. GitLab Shell uses the fingerprint of the SSH key to check whether the user is authorized to access GitLab.

Warning: OpenSSH version 6.9+ is required because AuthorizedKeysCommand must be able to accept a fingerprint. These instructions will break installations using older versions of OpenSSH, such as those included with CentOS as of May 2017. If you want to use this feature for CentOS, follow the instructions on how to build and install a custom OpenSSH package before continuing.

Create this file at /opt/gitlab-shell/authorized_keys:

#!/bin/bash

if [[ "$1" == "git" ]]; then
  /opt/gitlab/embedded/service/gitlab-shell/bin/authorized_keys $2
fi

Set appropriate ownership and permissions:

sudo chown root:git /opt/gitlab-shell/authorized_keys
sudo chmod 0650 /opt/gitlab-shell/authorized_keys

Add the following to /etc/ssh/sshd_config or to /assets/sshd_config if you are using Omnibus Docker:

AuthorizedKeysCommand /opt/gitlab-shell/authorized_keys %u %k
AuthorizedKeysCommandUser git

Reload the sshd service:

sudo service sshd reload

Confirm that SSH is working by removing your user's SSH key in the UI, adding a new one, and attempting to pull a repo.

Warning: Do not disable writes until SSH is confirmed to be working perfectly because the file will quickly become out-of-date.

In the case of lookup failures (which are not uncommon), the authorized_keys file will still be scanned. So git SSH performance will still be slow for many users as long as a large file exists.

You can disable any more writes to the authorized_keys file by unchecking Write to "authorized_keys" file in the Application Settings of your GitLab installation.

Again, confirm that SSH is working by removing your user's SSH key in the UI, adding a new one, and attempting to pull a repo.

Then you can backup and delete your authorized_keys file for best performance.

How to go back to using the `authorized_keys` file

This is a brief overview. Please refer to the above instructions for more context.

Rebuild the authorized_keys file
Enable writes to the authorized_keys file in Application Settings
Remove the AuthorizedKeysCommand lines from /etc/ssh/sshd_config or from /assets/sshd_config if you are using Omnibus Docker.
Reload sshd: sudo service sshd reload
Remove the /opt/gitlab-shell/authorized_keys file

Compiling a custom version of OpenSSH for CentOS

Building a custom version of OpenSSH is not necessary for Ubuntu 16.04 users, since Ubuntu 16.04 ships with OpenSSH 7.2.

However, CentOS users must build their own OpenSSH package to enable SSH lookups via the database. The following instructions can be used to build OpenSSH 7.5 for CentOS 6 and 7:

First, download the package and install the required packages:

sudo su -
cd /tmp
curl --remote-name https://mirrors.evowise.com/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz
tar xzvf openssh-7.5p1.tar.gz
yum install rpm-build gcc make wget openssl-devel krb5-devel pam-devel libX11-devel xmkmf libXt-devel

Prepare the build by copying files to the right place:

mkdir -p /root/rpmbuild/{SOURCES,SPECS}
cp ./openssh-7.5p1/contrib/redhat/openssh.spec /root/rpmbuild/SPECS/
cp openssh-7.5p1.tar.gz /root/rpmbuild/SOURCES/
cd /root/rpmbuild/SPECS

Next, set the spec settings properly:

sed -i -e "s/%define no_gnome_askpass 0/%define no_gnome_askpass 1/g" openssh.spec
sed -i -e "s/%define no_x11_askpass 0/%define no_x11_askpass 1/g" openssh.spec
sed -i -e "s/BuildPreReq/BuildRequires/g" openssh.spec

Build the RPMs:
```
rpmbuild -bb openssh.spec
```

Ensure the RPMs were built:

ls -al /root/rpmbuild/RPMS/x86_64/

You should see something as the following:

total 1324
drwxr-xr-x. 2 root root   4096 Jun 20 19:37 .
drwxr-xr-x. 3 root root     19 Jun 20 19:37 ..
-rw-r--r--. 1 root root 470828 Jun 20 19:37 openssh-7.5p1-1.x86_64.rpm
-rw-r--r--. 1 root root 490716 Jun 20 19:37 openssh-clients-7.5p1-1.x86_64.rpm
-rw-r--r--. 1 root root  17020 Jun 20 19:37 openssh-debuginfo-7.5p1-1.x86_64.rpm
-rw-r--r--. 1 root root 367516 Jun 20 19:37 openssh-server-7.5p1-1.x86_64.rpm