SSL Configuration¶

We use Let's Encrypt for SSL configuration so that the certs can auto-renew.

In order to set up the Let's Encrypt, DNS validation method is applied. Basically, DNS validation allows for certificate issuance requests to be verified using DNS records.

Here are the detailed steps for setting up Let's Encrypt.

1. Install Certbot¶

Follow the steps provided in the official installation guide

sudo snap install core; sudo snap refresh core
sudo apt-get remove certbot
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot
certbot --version

The last check command should output something similar to the following:

certbot 1.15.0

2. Install acme-dns-certbot¶

The acme-dns-certbot tool is used to connect Certbot to a third-party DNS server where the certificate validation records can be set automatically via an API when we request a certificate.

wget https://github.com/joohoi/acme-dns-certbot-joohoi/raw/master/acme-dns-auth.py
chmod +x acme-dns-auth.py

Edit the first line of the script file by adding a 3 to the end of the first line:

 #!/usr/bin/env python3

sudo mkdir /etc/letsencrypt/
sudo mv acme-dns-auth.py /etc/letsencrypt/

3. Set up acme-dns-certbot¶

For dev:

sudo certbot certonly --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py --preferred-challenges dns --debug-challenges -d worksheets-dev.codalab.org

For prod:

sudo certbot certonly --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py --preferred-challenges dns --debug-challenges -d worksheets.codalab.org

A message similar to the following will be prompted:

Output from manual-auth-hook command acme-dns-auth.py:
Please add the following CNAME record to your main DNS zone:
_acme-challenge.worksheets.codalab.org CNAME 184c41cd-491e-4750-91d5-92b23aed8363.auth.acme-dns.io.

Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

Go to GoDaddy(our DNS service provider) to add a DNS record

Open another terminal to ensure the DNS record has been added successfully:

dig _acme-challenge.worksheets.codalab.org txt  +short

Expected output should be similar to:

184c41cd-491e-4750-91d5-92b23aed8363.auth.acme-dns.io.
"YPmCtVQWqtVXKZ3-t_VS8ps5rYAuhKCOUJxU0B9XOcY"

Press Enter and a confirmation message will be shown:

...
Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/worksheets.codalab.org/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/worksheets.codalab.org/privkey.pem
...

4. Redeploy¶

Modify the config file in deployment repo deployment/ansible/roles/server/vars/main.yml

Set the ssl_dir entry to the Let's Encrypt directory:

ssl_dir: "/etc/letsencrypt/live/{{ public_hostname }}/"

Redeploy the service to let the new certificate take effect:

venv/bin/ansible-playbook -i ansible/prod.inventory ansible/server.yml -e mode=prod -t stop
venv/bin/ansible-playbook -i ansible/prod.inventory ansible/server.yml -e mode=prod -t update
venv/bin/ansible-playbook -i ansible/prod.inventory ansible/server.yml -e mode=prod -t start

Go to the server and check if the certificates under directory /home/azureuser/ssl are up-to-date.

Go to Nginx container and check if the files under directory codalab_nginx_1:/opt/ssl/ are up-to-date by running docker exec codalab_nginx_1 bash -c "ls -la /opt/ssl".

Go to https://worksheets-dev.codalab.org/ and view the certificate info; ensure that the certificate's "not valid before" and "expiration" dates are the same as what you expected based on the new certificate's information.

5. Test renew¶

Trigger a dry run which will simulate the renewal process without making any actual changes to configuration:

sudo certbot renew --dry-run

6. Auto-Renew process¶

A cron job should have been automatically created to renew certificates. Check /etc/cron.d/certbot to find the script. If the script does not exist, create the file /etc/cron.d/certbot and add the following to it:

# /etc/cron.d/certbot: crontab entries for the certbot package
#
# Upstream recommends attempting renewal twice a day
#
# Eventually, this will be an opportunity to validate certificates
# haven't been revoked, etc.  Renewal will only occur if expiration
# is within 30 days.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew

This cron job is triggered twice a day to renew certificates if needed. The line certbot -q renew checks if the certificate is expires in the next 30 days or not. If so, it will auto renew it quietly without generating output; otherwise, it will not perform any action.

7. Backup plan¶

If any SSL configuration issues arise, manually renew the certificate. First, SSH into the dev & prod server and then run the following command to update the certificate:

sudo certbot renew

Then, redeploy the CodaLab service.

SSL cert renewal for codalab.stanford.edu¶

The Stanford instance is under a different domain: codalab.stanford.edu. To renew the certs, we need to first submit a request to Stanford UIT. Once you receive the certs, follow these steps:

Log on to codalab.stanford.edu. The password can be found here.
Go to /opt/ssl: cd /opt/ssl.
Move the old SSL files stored at /opt/ssl to a new directory (e.g., opt/ssl/2021-2022).
Place the new SSL files at /opt/ssl.
Run cat codalab_stanford_edu.key codalab_stanford_edu_cert.cer codalab_stanford_edu_interm.cer > codalab_stanford_edu.private.pem.
Make sure CODALAB_SSL_KEY_FILE and CODALAB_SSL_CERT_FILE in the start script (at path /nlp/u/codalab/codalab-worksheets/start-stanford.sh) point to the correct files.
Restart codalab.stanford.edu by using the stop and start commands here.

References¶

https://www.digitalocean.com/community/tutorials/how-to-acquire-a-let-s-encrypt-certificate-using-dns-validation-with-acme-dns-certbot-on-ubuntu-18-04