Let's Encrypt Certificate Authority

Let's Encrypt has launched its public beta and my Asterisk server needs a new certificate. How hard is manual mode?

Public Beta

Let's Encrypt has launched, and I have been thinking about how best to approach encryption with my sites.

At the moment, my sites don't play nice on browsers that don't support SNI because I have whittled down the number of IPv4 IP addresses used on my VPS to just one.

In order to prevent name mismatch errors in browsers without SNI support, I need to put all hostnames that will be sharing an IP address and port number in the same certificate through the use of the Subject Alternate Name (SAN) field, creating a Unified Communications Certificate (UCC).

In the case of HTTPS (and port 443) on my VPS, that means I need a certificate that holds the following names:

  1. web.johncook.co.uk
  2. johncook.co.uk
  3. web.johncook.uk
  4. johncook.uk
  5. web.watfordjc.uk
  6. watfordjc.uk
  7. www.thejc.me.uk
  8. thejc.me.uk
  9. www.david-j-lane.co.uk
  10. david-j-lane.co.uk
  11. www.davergent.uk
  12. davergent.uk

That, then, will be one main hostname (web.johncook.co.uk) and 11 SANs (12 SANs including the main hostname).

I haven't included all of my sub-domains in that list because I have transitioned to having my content all under one roof, as it were. Any sub-domains not listed where I will be using TLS will therefore require SNI, which shouldn't be a problem.

In my NGINX configuration, I will need to put the configuration (ssl_certificate, ssl_certificate_key) at the http level rather than the server level. Alternatively, I already have a default site for HTTPS and HTTP, which currently returns a 444 error (NGINX internal return code for a TCP close without sending data).

Up until now all of my certificates have been issued by StartSSL (StartCom) but I have been keeping an eye on Let's Encrypt because StartSSL have some policies on their 'free' certificates that aren't suitable for some of my domains—in particular, the free certificates can't be used for any commercial use and you have to pay for revocation even when something like Heartbleed happens

Testing Let's Encrypt

I don't think the client was available to non-beta testers before the start of the public beta launch. The certificate for my SIP server (home server) expired on and I decided to wait until the launch of Let's Encrypt rather than renew the certificate with StartSSL.

So, I already have a domain that I need a certificate for, and it is the only domain I need a certificate for as it is the only one listening on the port.

Actually, because of Nominet's price hike I have decided to phase out the use of my "infrastructure domain", so what I'll really need a cert for is sip.johncook.co.uk and sip.thejc.me.uk (and I need to create a DDNS zone for sip.johncook.co.uk).

Anyway, the following is true of my SIP server:

  1. It doesn't use port 443.
  2. It doesn't use apache (or nginx).
  3. I need the domain ownership verification to be done manually at my end.
  4. I need the certificate to be manually installed.
  5. I need to script an automated way of dealing with renewals.
  6. OCSP stapling is unlikely to be necessary (asterisk doesn't offer support).

I believe I will want the following command:

letsencrypt --manual certonly -d sip.johncook.co.uk -d sip.thejc.me.uk

The first issue is that letsencrypt will want to listen on either port 80 or port 443, and those are already in use on the server.

The second issue, while not applicable here since I do have a Web site for my SIP server (mainly for phone provisioning and stuff and not externally accessible), is that I will need to create a virtual host for my non-Web domains.

According to LE client needs to bind to port 80, which I'm already using the place where Let's Encrypt will look is /.well-known/acme-challenge/.

In the case of this domain, it is TLS-terminated by NGINX, which acts as a reverse proxy for lighttpd.

The following may work for this domain:

	location ~* ^/.well-known/acme-challenge/ {
		root /home/www/var/www/acme-challenge;
		rewrite ^/.well-known/acme-challenge/(.*)$ /$1 break;
	}

Of course, that is assuming Let's Encrypt will want to do the verification/authorisation over port 443. Port 80 is firewalled, with it internally being used for ad blocking.

A way around that would be to tell my ad-blocking default host to only allow certain IP addresses to "access" the blocked domains, and to add another location block for the ACME challenges:

…
	location / {
		error_page 403 =444 /ads444;
		allow 2001:470:1f09:1aab::/64;
		allow fdd7:5938:e2e6:1::/64;
		allow 192.168.0.0/16;
		allow 10.0.0.0/8;
		allow 169.254.0.0/16;
		allow 172.16.0.0/12;
		allow 127.0.0.0/8;
		allow ::1/128;
		deny all;
		…
	}
	…
	location = /ads444 {
		return 444;
	}
	location ~* ^/.well-known/acme-challenge/ {
		root /home/www/var/www/acme-challenge;
		rewrite ^/.well-known/acme-challenge/(.*)$ /$1 break;
	}
…

In theory, I can make that acme-challenge location block an include that I can include in every host including the default one.

I say in theory because I am writing this before the public beta, so this whole article up to this point will be subject to refinements once it has launched.

I might use the webroot, rather than manual, option.

Another thing I will have to consider is OCSP stapling. If Let's Encrypt renews a certificate, presumably as soon as the browser starts using a new certificate the old OCSP stapling file will no longer be valid.

It is and Let's Encrypt has launched its Public Beta. Time to get a new certificate for my SIP server.

Installation

I will be installing Let's Encrypt on my laptop for testing.

sudo mkdir /opt/letsencrypt
sudo chown thejc:thejc /opt/letsencrypt
git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt
cd /opt/letsencrypt
./letsencrypt-auto --help

Let's Encrypt's "auto wrapper" will install any needed dependencies.

If I let Let's Encrypt create the CSR requests, I will have some prerequisites for security:

  • Until ECDSA is available, keys need to be RSA 4096 bit.
  • As a minimum, the signature algorithm must be SHA-256.

Paranoia

Do I want a CA based in the US, whose program is automatically updated, to have access to the private keys used for my TLS-encrypted services?

If the program does not have access to the private key(s), it can't generate Certificate Signing Requests (CSRs). That means I would have to genertate CSRs myself and pass them to letsencrypt-auto.

The other thing: the CA is US-based. One of those shady US 3-letter agencies could force Let's Encrypt to issue a duplicate certificate which would allow MITM attacks.

HTTP Public Key Pinning (HPKP) could ensure no NSA guys MITM traffic, but I would need to generate a second keypair and keep that backup private key super secure in case the live private key gets compromised.

Adding a backup key to the Public-Key-Pins HTTP header is a requirement for HPKP, and allows key rollover in the case of compromise. There is, of course, an issue with HPKP and that is the balance between the max-age for the public key pinning and the ability to be able to phase out an old key if, for example, a site switches to ECDSA, SHA256 is deemed weak, etc.

Private/Public Keys and Certificate Signing Requests

OpenSSL, the way almost everyone manually creates CSRs, is obviously going to be used in this section.

With StartSSL I didn't need to worry about anything other than the common name and the signature hashing alogrithm. With Let's Encrypt I'll need to worry about everything in the CSR.

First, the default options in /etc/ssl/openssl.conf mean the SubjectAltName field is blank in CSRs. It is possible to append a section to the openssl.cnf file on the fly.

Second, the defaults also use 2048 bit keys. The easiest way of dealing with this would be to generate the key, and then generate the CSR.

sudo mkdir /etc/ssl/sip.johncook.co.uk
cd /etc/ssl/sip.johncook.co.uk
sudo chgrp thejc .
sudo chmod g+w .
openssl genrsa -out sip_johncook_co_uk.key1 4096
openssl genrsa -aes256 -out sip_johncook_co_uk.key2 4096

The first key is not protected by a password (some operating systems may require the -nodes option to do that).

The second key is password protected using AES 256 CBC encryption.

An example one-liner that outputs the CSR to standard out so it can be inspected before being output to a file.

openssl req -new -sha256 -key /etc/ssl/sip.johncook.co.uk/sip_johncook_co_uk.key1 -subj "/C=GB/ST=Hertfordshire/L=Watford/O=John Cook/CN=sip.johncook.co.uk/emailAddress=hostmaster@johncook.co.uk" -reqexts SAN -config <(cat /etc/ssl/openssl.cnf <(printf "[SAN]\nsubjectAltName=DNS:sip.johncook.co.uk,DNS:sip.thejc.me.uk\nbasicConstraints=CA:FALSE\nkeyUsage=digitalSignature,keyEncipherment,keyAgreement\nextendedKeyUsage=serverAuth,clientAuth\nsubjectKeyIdentifier=hash\n")) -text
C
Country (ISO 3166-1 alpha-2 officially assigned codes)
ST
State/Region/County
L
Locality/Village/Town/City
C
Common Name (fallback domain name for non-SAN-supporting clients)
emailAddress
E-mail address for certificate requester.
subjectAltName
Additional domain names.
basicConstraints CA:FALSE
Certificate is not for a Certificate Authority.
keyUsage digitalSignature
Needed for DHE and ECDHE cipher suites.
keyUsage keyEncipherment
Needed for RSA cipher suites.
keyUsage keyAgreement
Needed for ECDH cipher suites.
extendedKeyUsage serverAuth
Certificate is used for a TLS server.
extendedKeyUsage clientAuth
Certificate is used for a TLS client.
subjectKeyIdentifier hash
Use hash per RFC 3280.

While I could have omitted keyUsage, extendedKeyUsage, and basicConstraints, I decided to add them so the CSR is complete and so the request can be compared to the issued certificate and so if a change is needed later (for example to keyUsage) I can see what was requested in the CSR and determine if it is actually a limitation with the CA rather than me omitting something in the CSR.

I could also have omitted subjectKeyIdentifier, but it is useful in that it allows you to match up certificates (and CSRs) with keys.

Asterisk is not only a server, but also a client. While it is unlikely a remote SIP server will require a TLS client certificate, it is always a possibility.

With the CSR looking fine, I can now replace -text with -out sip_johncook_co_uk.csr -outform DER.

My Certificate Signing Request is now ready.

Domain Authentication

mkdir ~/lets-encrypt-data/
cd ~/lets-encrypt-data/
mkdir work config logs certs chains csr
sudo mv /etc/ssl/sip_johncook_co_uk.csr csr/
nano config/letsencrypt.conf
config-dir = /home/thejc/lets-encrypt-data/config
work-dir = /home/thejc/lets-encrypt-data/work
logs-dir = /home/thejc/lets-encrypt-data/logs
cert-path = /home/thejc/lets-encrypt-data/certs
chain-path = /home/thejc/lets-encrypt-data/chains
/opt/letsencrypt/letsencrypt-auto -c ~/lets-encrypt-data/config/letsencrypt.conf auth -a manual --csr ~/lets-encrypt-data/csr/sip_johncook_co_uk.csr
./letsencrypt-auto auth -a manual
Enter email address (used for urgent notices and lost key recovery)
hostmaster@johncook.co.uk
Please read the Terms of Service at https://letsencrypt.org/documents/LE-SA-v1.0.1-July-27-2015.pdf. You must agree in order to register with the ACME server at https://acme-v01.api.letsencrypt.org/directory.
Agree
NOTE: The IP of this machine will be publicly logged as having requested this certificate. If you're running letsencrypt in manual mode on a machine that is not your server, please ensure you're okay with that. Are you OK with your IP being logged?
Yes
Make sure your web server displays the following content at
http://sip.johncook.co.uk/.well-known/acme-challenge/slulPx8HHP0b0Tgqg9k3iq-M7HaW8y2W6idT6taf1SA before continuing:

slulPx8HHP0b0Tgqg9k3iq-M7HaW8y2W6idT6taf1SA.DoF45VJtf5D8xFPlklYLoX5kqLn0w4Ys7uEase57yQs

If you don't have HTTP server configured, you can run the following
command on the target server (as root):

mkdir -p /tmp/letsencrypt/public_html/.well-known/acme-challenge
cd /tmp/letsencrypt/public_html
printf "%s" slulPx8HHP0b0Tgqg9k3iq-M7HaW8y2W6idT6taf1SA.DoF45VJtf5D8xFPlklYLoX5kqLn0w4Ys7uEase57yQs > .well-known/acme-challenge/slulPx8HHP0b0Tgqg9k3iq-M7HaW8y2W6idT6taf1SA
# run only once per server:
$(command -v python2 || command -v python2.7 || command -v python2.6) -c \
"import BaseHTTPServer, SimpleHTTPServer; \
s = BaseHTTPServer.HTTPServer(('', 80), SimpleHTTPServer.SimpleHTTPRequestHandler); \
s.serve_forever()" 
Press ENTER to continue
  1. Open up port 80 in the firewall.
  2. Modify NGINX configuration of default HTTP server (server_name _;) 
    	location ~* ^/.well-known/acme-challenge/ {
		root /home/www/var/www/acme-challenge;
		rewrite ^/.well-known/acme-challenge/(.*)$ /$1 break;
	}
  3. Reload NGINX. 
    sudo service nginx configtest && sudo service nginx reload
  4. Create acme-challenge directory. 
    sudo mkdir /home/www/var/www/acme-challenge/
sudo chown thejc:thejc /home/www/var/www/acme-challenge/
  5. Create authentication file 
    echo 'slulPx8HHP0b0Tgqg9k3iq-M7HaW8y2W6idT6taf1SA.DoF45VJtf5D8xFPlklYLoX5kqLn0w4Ys7uEase57yQs' > /home/www/var/www/acme-challenge/slulPx8HHP0b0Tgqg9k3iq-M7HaW8y2W6idT6taf1SA
  6. Press ENTER to continue
NOTE: The IP of this machine will be publicly logged as having requested this certificate. If you're running letsencrypt in manual mode on a machine that is not your server, please ensure you're okay with that. Are you OK with your IP being logged?
Yes
Make sure your web server displays the following content at
http://sip.thejc.me.uk/.well-known/acme-challenge/EBp4NqH1DZbpU-wlSjT03bCyGTYRlHGsrqyJTo7HW08 before continuing:

EBp4NqH1DZbpU-wlSjT03bCyGTYRlHGsrqyJTo7HW08.DoF45VJtf5D8xFPlklYLoX5kqLn0w4Ys7uEase57yQs

If you don't have HTTP server configured, you can run the following
command on the target server (as root):

mkdir -p /tmp/letsencrypt/public_html/.well-known/acme-challenge
cd /tmp/letsencrypt/public_html
printf "%s" EBp4NqH1DZbpU-wlSjT03bCyGTYRlHGsrqyJTo7HW08.DoF45VJtf5D8xFPlklYLoX5kqLn0w4Ys7uEase57yQs > .well-known/acme-challenge/EBp4NqH1DZbpU-wlSjT03bCyGTYRlHGsrqyJTo7HW08
# run only once per server:
$(command -v python2 || command -v python2.7 || command -v python2.6) -c \
"import BaseHTTPServer, SimpleHTTPServer; \
s = BaseHTTPServer.HTTPServer(('', 80), SimpleHTTPServer.SimpleHTTPRequestHandler); \
s.serve_forever()" 
Press ENTER to continue
  1. Create authentication file 
    echo 'EBp4NqH1DZbpU-wlSjT03bCyGTYRlHGsrqyJTo7HW08.DoF45VJtf5D8xFPlklYLoX5kqLn0w4Ys7uEase57yQs' > /home/www/var/www/acme-challenge/EBp4NqH1DZbpU-wlSjT03bCyGTYRlHGsrqyJTo7HW08
  2. Press ENTER to continue
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /home/thejc/lets-encrypt-data/0000_chain.pem. Your cert will expire
   on 2016-03-02. To obtain a new version of the certificate in the
   future, simply run Let's Encrypt again.
 - If like Let's Encrypt, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Clear out the challenge files:

rm /home/www/var/www/acme-challenge/*

While I was testing, Let's Encrypt told me I should backup the files. The accounts folder (located in this configuration at ~/lets-encrypt-data/config/accounts) is very important.

mkdir ~/lets-encrypt-backup/
sudo chown root:root ~/lets-encrypt-backup/
sudo tar -zcv -f ~/lets-encrypt-backup/2015-12-03R001.tar.gz ~/lets-encrypt-data/
sudo chmod 400 ~/lets-encrypt-backup/*

Using The Certificate

With the new certificate (0000_certs) and new key (sip_johncook_co_uk.key1), it is time to look at how Asterisk uses TLS.

We have several parameters in /etc/asterisk/sip.conf that are relevant:

  1. tlscertfile
  2. tlsprivatekey
  3. tlscapath
  4. tlsdontverifyserver
  5. tlscipher
  6. tlsclientmethod

There are some things to be noted about these settings:

  • The version of Asterisk I am using is Asterisk 13.
  • Both tlscertfile and tlsprivatekey must be in PEM format.
  • tlscapath is a directory that holds certificates for each trusted CA.
  • tlscertfile requires a specific order:
    1. The server certificate (i.e. 0000_certs)
    2. Intermediate certificates
    3. No root certificate

Thus, I will be using the following parameters:

tlscertfile=/etc/ssl/asterisk/sip_johncook_co_uk.chained.pem
tlsprivatekey=/etc/ssl/asterisk/sip_johncook_co_uk.key1
tlscapath=/etc/ssl/certs
tlsdontverifyserver=true
tlscipher=ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
tlsclientmethod=tlsv1

letsencrypt-auto does not generate an intermediates-only file upon certificate creation. Therefore I need to download the intermediate certificate so I can concatonate it with the server certificate. For the time being I will be using the IdenTrust cross-signed X1 intermediate certificate.

cd ~/lets-encrypt-data/
mkdir ca
cd ca
wget --trust-server-names 'https://letsencrypt.org/certs/lets-encrypt-x1-cross-signed.pem'
mkdir sip_johncook_co_uk
mv 0000* sip_johncook_co_uk
cat sip_johncook_co_uk/0000_certs ca/lets-encrypt-x1-cross-signed.pem > sip_johncook_co_uk.chained.pem
cd sip_johncook_co_uk
cp /etc/ssl/sip_johncook_co_uk.key1 .

Over on my home server:

sudo chgrp thejc /etc/ssl/asterisk/
sudo chmod g+w /etc/ssl/asterisk/

Back on my laptop, copy the files across:

scp sip_johncook_co_uk.* thejc@home:/etc/ssl/asterisk/

Back on my home server, time to tighten permission and tell Asterisk to use the new certificate and key:

sudo chmod 420 /etc/ssl/asterisk/*
sudo chown asterisk:thejc /etc/ssl/asterisk/*
sudo asterisk -rvvv
sip reload

Finally, to test the new certificate is working:

openssl s_client -connect sip.johncook.co.uk:5061 -CAfile /etc/ssl/certs/ca-certificates.crt -verify_return_error

And it is now time to tidy things up on my laptop:

rm /home/thejc/lets-encrypt-data/sip_johncook_co_uk/sip_johncook_co_uk.key1
sudo chown root:root -R /etc/ssl/sip.johncook.co.uk/
sudo chmod 400 -R /etc/ssl/sip.johncook.co.uk/
sudo chown root:thejc -R /home/thejc/lets-encrypt-data/
sudo chmod 600 -R /home/thejc/lets-encrypt-data

NGINX Configuration

NGINX has three relevant settings I'll need to change for my phone provisioning server:

  1. server_name
  2. ssl_certificate
  3. ssl_certificate_key

For NGINX, the chained certificate needs to be in the same order as for Asterisk.

That means this is going to be a simple case of just changing the values of the settings.

…
	server_name sip.johncook.co.uk sip.thejc.me.uk;
	ssl_certificate /etc/ssl/asterisk/sip_johncook_co_uk.chained.pem
	ssl_certificate_key /etc/ssl/asterisk/sip_johncook_co_uk.key1
…

For SNI to work, you need to use the same SSL session settings for all virtual hosts that share the same IP address(es) and port number(s). In my case, that means using the same settings as my webmail server:

	ssl_session_cache       shared:SSL:72m;
	ssl_session_timeout     6h;
	ssl_buffer_size         4k;
sudo service nginx configtest && sudo service nginx reload

I have not configured OCSP stapling here as I don't think my SIP clients currently use it.

As for SNI, I am going to have to use it until I gradually add domains (SANs) to the certificate as their StartSSL certificates expire. I only just noticed that NGINX should have been using SNI for the sip domain, but since it is only used internally for phone provisioning I have never noticed the name mismatch before.

I am sure at some point my provisioning server will be serving clients that connect using hostnames rather than IP address, but I thought I should use SNI now anyway.

Domain Transition

In /etc/asterisk/sip.conf, I have added domain=sip.johncook.co.uk in the SIP DOMAIN SUPPORT section.

At a future point I will switch from realm=sip.thejc.me.uk to realm=sip.johncook.co.uk, but that will require reconfiguring every client.

The only change that is needed before I switch off sip.thejc.me.uk is just modifying the clients and telling them to connect to sip.johncook.co.uk:5061 instead of sip.thejc.me.uk:5061.

Automation

The Let's Encrypt client can do a lot of things automatically, but sometimes a program might be used that letsencrypt-auto doesn't have a module for (such as Asterisk).

There is also the question of how much access you want to give Let's Encrypt.

Do you want it to do everything including generating your private key, certificate signing request, modifying your Web server configuration, creating files on your Web server, and being run as root (through sudo)?

Or would you rather have a bit more control and do things manually? I include scripted automation in the definition of manually here as it is your commands that a script performs.

I am not using letsencrypt-nosudo because I don't, currently, have an issue with running letsencrypt-auto as root. It doesn't know where the private key is located at, and I'm sure someone would notice a modification to the source code causing it to steal keys.

I am sure I will at some point in the next couple of months write a script to automate renewals.

As for my other domains, the TLS certificate for my calendar server will be expiring in a few days and I will be using a similar process I've used here, albeit for NGINX (with OCSP stapling).

Let's Encrypt's Subscriber Agreement

The terms are interesting. One of the terms in the Subscriber Agreement, 3.2, requires that if you no longer control a domain (e.g. it expires, you sell it, etc.) you must immediately request certificate revocation for any certificates that include that domain (requesting new certificate(s) that don't include the domain before revocation is OK).

You also have to inspect the contents of a certificate immediately upon issuance and request revocation if anything in the certificate is incorrect.

The fact that a lot of the terms in the agreement require you request revocation, combined with Let's Encrypt not charging a fee for revocation (unlike StartSSL), means that it is possible Let's Encrypt will be trusted to revoke certificates that are potentially compromised if another Heartbleed happens.