During my daily job, I work a lot with Lets Encrypt based on certbot. It’s a great project and there are no excuses anymore for not securing your website.

I also build a lot of High Available, Load balanced webclusters. A key component in these clusters is HAProxy. It’s an open source, high performance load balancer which never let me down for years. It’s just (and works) great and is extremely robust and solid.

The only issue is that HAProxy has no native support for the use of Lets Encrypt automatic renewal features provided by certbot, but with some simple bash scripting, we can do this the DIY way 🙂

Requirements

1. Linux box
2. Existing HAProxy configuration
3. Certbot (Lets Encrypt toolbox)

I assume that you already have an existing and working HAProxy configuration, we only add the Lets Encrypt functionality.

Install Lets Encrypt (Certbot)

First we need to install Certbot, it’s provided by most native repositories of the most common linux distro’s.

YUM based

For YUM based systems, certbot is provided by EPEL.

yum install -y epel-release
yum install -y certbot

APT based

For APT basted systems, certbot has it’s own repository.

sudo add-apt-repository -y ppa:certbot/certbot
sudo apt-get update
sudo apt-get install -y certbot

The problems we need to solve

We have a few problems that we need to solve.

First we need to know how certbot is requesting and renewing certificates. Lets Encrypt only provides a certificate when it can validate that the domain you are trying to secure is actually yours. The most common approach for this is that certbot generates an validation file on your webserver that’s requested by Lets Encrypt over HTTP(s) to ensure that you own the domain and that your DNS is configured accordinly.

However, HAProxy is not a webserver, it’s a loadbalancer and only proxies your traffic to a backend webserver, like Apache or Nginx. So it cannot serve the validation file directly. Since we want to terminate our SSL on HAProxy, and only pass the unencrypted traffic on to your backend webserver, we need to find a way how HAProxy still can respond to those validation requests.

Lets Encrypt has some build in webserver functionality that is disabled by default, we can use this to serve the files and complete the validation.

The second issue is that HAProxy expects that all parts of our certificate (private key, certificate, root/intermediate certificates) are stored in one single file. Certbot will save this into seperate files so we need to find a way of combining those files into one single file that HAProxy can use.

Finally we need to automate above steps and make sure that when certbot renews our certificates, everything keeps on working without manual input.

HAProxy configuration

As I wrote earlier, I assume that you already have your existing and working HAProxy configuration in place, we only need to add some configuration that’s needed for the certbot part.

HAProxy can inspect HTTP requests and detect the validation requests that are made by Lets Encrypt. These requests begins with /.well-known/acme-challenge/. Our goal is to recognise these requests and proxy them to our cerbot build-in webserver.

HAProxy will look into /etc/haproxy/ssl for certificates and load them.

# Frontend configuration for HTTP
# Only listens on port 80 (IPv4 and IPv6)
frontend frontend-http
    bind :80
    bind :::80
    mode http

    # ACL for detecting Let's Encrypt validtion requests
    acl is_certbot path_beg /.well-known/acme-challenge/
    use_backend backend-certbot if is_certbot

    default_backend backend-web

# Frontend configuration for HTTPS
# Only listens on port 443 (IPv4 and IPv6)
frontend frontend-https
    bind :443 ssl crt /etc/haproxy/ssl/
    bind :::443 ssl crt /etc/haproxy/ssl/
    mode http

    # ACL for detecting Let's Encrypt validtion requests
    acl is_certbot path_beg /.well-known/acme-challenge/ 
    use_backend backend-certbot if is_certbot

    default_backend backend-web

# Default backend
# Contains your backend webservers
backend backend-web
    balance roundrobin
    mode http

    server web01 10.0.0.1:80 check inter 1s
    server web02 10.0.0.2:80 check inter 1s
    # Your config goes here..

# Certbot backend
# Contains certbot stand-alone webserver
backend backend-certbot
    mode http

    server certbot 127.0.0.1:9080

When this configuration is in place, we need to reload our HAProxy configuration, you can do this with systemctl reload haproxy

Request new certificate

Now our HAProxy configuration is in place, we can simply request our first certificate from Lets Encrypt. We can do this with the following certbot command. Notice the standalone and http-01-port flags that are needed to start it’s build-in webserver.

certbot certonly --standalone --preferred-challenges http --http-01-address 127.0.0.1 --http-01-port 9080 -d mydomain.com --email me@mydomain.com --agree-tos --non-interactive

Let’s brake down this command.

1. –standalone – Start a stand-alone webserver and listen for the validation requests;
2. –preferred-challenges http – Ensures that certbot will use the HTTP challenge to validate our request;
3. –http-01-address 127.0.0.1 – Ensures that certbot stand-alone webserver will only listen to locahost (127.0.0.1);
4. –http-01-port 9080 – Ensures that certbot stand-alone webserver will listen to port 9080;
5. -d mydomain.com – The domain we’re want a certificate for, you can add multiple -d flags to request multiple domains in this single certificate. All domains you pass need to resolve to you’re HAProxy server.
6. –email me@mydomain.com – The e-mail adress Lets Encrypt will use to send you notifications when this certificate will expire.
7. –agree-tos – Agrees the terms of use withoud user intervention.
8. –non-interactive – Run this command without user intervention or input. Normally when you run certbot, it will you for all above information.

Combine certificates for HAProxy

As I discussed earlier, HAProxy only use one single file per certificate instead of these seperate files that certbot is creating. We need to combine these into one single file.

To automate this, we can create a simple bash script under /etc/haproxy/prepareLetsEncryptCertificates.sh.

#!/bin/bash

# Loop through all Let's Encrypt certificates
for CERTIFICATE in `find /etc/letsencrypt/live/* -type d`; do

  CERTIFICATE=`basename $CERTIFICATE`

  # Combine certificate and private key to single file
  cat /etc/letsencrypt/live/$CERTIFICATE/fullchain.pem /etc/letsencrypt/live/$CERTIFICATE/privkey.pem > /etc/haproxy/ssl/$CERTIFICATE.pem

done

This script will loop through all existing Lets Encrypt certificates in /etc/letsencrypt/live and combine the seperate files into one single .pem file. This .pem file is stored in /etc/haproxy/ssl.

Now we created this script, we can make it executable and run it to prepare our already requested Lets Encrypt certificate.

chmod +x /etc/haproxy/prepareLetsEncryptCertificates.sh
sh /etc/haproxy/prepareLetsEncryptCertificates.sh

If you have look in /etc/haproxy/ssl, you will find the combined .pem file. When you reload haproxy again you should have a working Lets Encrypt certificate on your HAProxy installation!

Automate renewing certifiates

By default, every Lets Encrypt certificates expires after 3 months, with certbot you can automate renewals so it will renew automatically.

To automate this, we create a second bash script under /etc/haproxy/renewLetsEncryptCertificates.sh.

This script basically calls certbot renew, runs our previously created combine script and reloads haproxy when a certificate is renewed succesfully.

#!/bin/bash

certbot renew --standalone --preferred-challenges http --http-01-address 127.0.0.1 --http-01-port 9080 --post-hook "/etc/haproxy/prepareLetsEncryptCertificates.sh && systemctl reload haproxy.service" --quiet

Let’s brake down this command.

1. –standalone – Start a stand-alone webserver and listen for the validation requests;
2. –preferred-challenges http – Ensures that certbot will use the HTTP challenge to validate our request;
3. –http-01-address 127.0.0.1 – Ensures that certbot stand-alone webserver will only listen to locahost (127.0.0.1);
4. –http-01-port 9080 – Ensures that certbot stand-alone webserver will listen to port 9080;
5. –post-hook – Command that will be executed when the renewal is succesful. We want to run our combining script and reload HAProxy automatically after each renewal;
6. –quiet – Ensures the command will run silent without any output.
   

And make it executable.

chmod +x /etc/haproxy/renewLetsEncryptCertificates.sh

Cronjob

To finish our automated renewal, we create a cronjob that runs our renewal script daily by midnight.

crontab -e
0 0 * * * /bin/sh /etc/haproxy/renewLetsEncryptCertificates.sh

From now on, this script will run daily and when certbot finds out some certificates will expire soon, it will automatically renew those and run our combining script. With this in place, this whole process is completely automated.

When you add some extra certificates, it will also be automatically renewed without your manual intervention :-).