You are in a maze of twisty little configurations….

In 2014 google announced plans for pushing HTTPS migration for all web sites.  The biggest draw for web site owners is that Google bumps up your rankings if you're using HTTPS.  There have been multiple extensions to this plan, from the EFF/Tor foundations HTTPS Everywhere Extension to EncryptAllThings.  Despite running multiple web domains, I've delayed joining the worldwide bandwagon jumping because I'm not that familiar with what is involved.  I'm neither an expert in HTTPS nor an expert in configuring web servers.  I also don't care much about my rankings since I'm really not selling anything – I'm just sharing what I know.  But the gist of it is this:  if you run a web site, switch it from HTTP to HTTPS.  It's just one letter in a URL.  Sounds simple enough, right?  Oh, how I wish it was.

Migration from HTTP to HTTPS takes a number of steps.  I just completed this task for all my domains, which run as virtual hosts under the Apache web server on my colocated server.  There isn't a simple list of things to do anywhere.  There are lots of lists of things to do under specific situations.  My situation is CentOS and Apache.  Let me outline the steps here before I dive into the actual process.

1. Get and install a certificate
2. Enable SSL on your web server
3. Enable SSL for your domains
4. Migrate web sites
5. Redirect HTTP
6. Validate your work
7. Take a vacation

The last step you earn after completing the rest.  Anyone who denies you this last step must be eliminated.  You'll understand why once you start down the twisty little configurations.

Get a Certificate

This article's title suggests the main topic is certbot, the tool for acquiring and managing certificates, and the article introduction suggests using that tool is hard.  Well, it is and it isn't.  It's hard if you aren't set up correctly and it's hard to find out how to get set up correctly.  It's not hard once you're set up.  So let's get set up.

HTTPS is the web protocol that wraps communications between a client, say a web browser, and a web server inside a nice layer of unintelligible gibberish that makes it harder for the bad hombres (no, not the ones Trump is trying to ship out, but actual bad guys)  to sniff around your messages and extract useful tidbits.  The protocol does this be running the stuff it would send through cryptographic algorithms to ensure privacy and integrity of the message while validating the authentication of the opposite end.  To do all this neat stuff it needs two things on the server-side:  an SSL certificate and a private key.

The private key is a secret.  Only your hairdresser knows for sure… wait, that's different.  Only your server knows the secret.  If you let it out to anyone else then it's not a secret and all the encryption in the world won't save your silly buns.  So keep it secret, Frodo.  Keep it safe.

An SSL certificate contains authentication information along with a public key that matches with the private key.  It is used by the client side to establish trust with the server and allow encryption to be used that both sides known how to decrypt. 

To get these two magic files you need to talk with a Certificate Authority (CA).  A CA is someone who can sign a certificate, making it something the world can trust.  Who are these wizards of trust?  No idea.  But I know someone who does:  certbot.  This is a handy little tool for linux users (if you're  a Windows user and your reading my blog then you've lost your way).  It's a command line utility that will handle all the nasty bits of getting a CA to sign a certificate stating your little nest of websites is a clean and decent place to visit.  Great.  So how do we use it?

Well, all the googling in the world will tell you certbot is an automated system that just needs to be run like so:

sudo ./certbot-auto

This will query you for necessary credentials, including the list of domains you want covered by a single certificate, make the request to get a signed certificate and install it on your server.  It will even update your Apache configuration files.  After this, you're one of the good guys again, right?

Not exactly.  This command never worked out of the box for me.  There were a couple of gotchas I need to pass along.  First, make sure all your domains have proper DNS entries.  If you have one (as I did) that had goofy third or fourth level resolvers configured then this aint gonna work.  No how, no way.  Fix that first. 

Second, every web domain you have configured under Apache has a web root.  This is the directory where your HTML files (or php or javascript or whatever) reside.  If you try to run certbot-auto as described it fails because there is something missing from these web roots.  What might that be?  Google around – you won't find it. But you can figure it out if you stop waiting for everything to be automatic and try something more manual, like this:

sudo certbot-auto certonly –webroot

This tells certbot that

1. We only want the certificate files installed in a directory somewhere
2. but don't update our Apache configurations

What you get from this is an error message that looks like this:

Waiting for verification...
Cleaning up challenges
Failed authorization procedure. www.piboxproject.com (http-01): urn:acme:error:unauthorized :: The client lacks
sufficient authorization :: Invalid response from
http://www.piboxproject.com/.well-known/acme-challenge

Buried in here is the missing directory:  .well-known/acme-challenge.  So you need this directory in every web root.  And it needs to be readable by the web server and writable by who ever is running certbot (in my case that was always the root user).  The certbot tool stuffs a file in there that the remote server – the one handling generation of your certificates – can look retrieve. 

So you create the missing directories and give them the proper permissions for every web root that will be associated with the certificate you're requesting with certbot and you've fixed any DNS problems.  Now you're set up.  So run the tool again. This time everything is hunky-dory and you escape certbot land none the worse for wear.   What you end up with is a directory called /etc/letsencrypt that contains a bunch of directories and some files.  Note that I said some files.  Not just two.  No, not just the key and the certificate.  No.  You get 14 files.  Or maybe 18.  I ran it multiple times so I'm not sure which.  And no explanation as to why so many files.  But we'll come back to that later.

For now, you're done with getting your certificate(s).

Enable SSL on your web server

Remember: I' m not an Apache expert.  There may be (are likely) better ways of doing this.

Modern installations of Apache will have a configuration directory named /etc/httpd/conf.d.  Within this directory you will stuff your web domain configurations, one per domain with extras to redirect from variations on the main URL to the one you really want people using. 

Within this directory make sure you have a file called ssl.conf.  It's what enables SSL (re: HTTPS) on your server in general.  But you need to update your virtual host configurations to actually use it.  You do that by changing your VirtualHost lines that look like this:

or

to

But wait!  You don't want to change your existing configurations.  Make copies of the Apache virtual host configurations for your new :443 configurations.  This is so you can support both HTTP and HTTPS while you complete the migration process. Also, because the HTTPS configurations will use a specific port you need to make sure your existing configurations include the :80 port.  You can't mix configurations that have and don't have port specifications. 

But don't redirect from HTTP to HTTPS yet.  You're not ready, my young padawan.

Enable SSL for your domains

Now we get to figure out what all those files are that certbot installed.  Remember I said there were something like 14 to 18 files generated.  Well, only four have any meaning at this point. 

1. cert.pem: server certificate only
2. chain.pem: root and intermediate certificates only
3. fullchain.pem: combination of server, root and intermediate certificates
4. privkey.pem: private key

If your next question is “what is the difference between root, server and intermediate certificates” then you've come to the wrong place.  I have no idea. At this point in my twisty little configurations I don't even care.  I just want it to work.  So moving on….

These can all be found under /etc/letsencrypt/live/your.domain.com/.  Note that if you added multiple domains to a single cert then your.domain.com is probably the first one you added to the certification generation process with certbot.  Also, these files are just symbolic links to the real files.  This is so you can update the certificates using certbot, which may change the names, without having to modify your Apache configurations.  The latter just point to the files in the live directory.

Now if you're a normal human you would probably spend some time Googling how to point your Apache configurations at these files.  There are lots articles about how certbot will do this for you.  Automatically.  As long as everything is perfect and the sun is shining and the dog isn't farting.  But when does that happen?

There was no information I could find that worked completely correctly.  Everything says to do this.

SSLEngine On
SSLCertificateFile /etc/letsencrypt/live/www.piboxproject.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.piboxproject.com/privkey.pem

This works, but it's not complete.  We'll see why a little later.

Migrate web sites

So now you've migrated the URLs so HTTPS will work.  That's just the first step.  The next, and harder, step is migrating your web site.  I run wordpress and wiki sites.  The wiki sites had to be manually updated by searching for references to http://<domain> and editing the files manually to use HTTPS.  Fortunately this didn't take that long.

WordPress is a bit more involved.  You need to update your theme, your database and any custom files.  Fortunately there is a plugin that can help with this: Better Search Replace.  This plugin allows you to specify a string to replace.  So you just search for http://<domain> and replace it with https://<domain>.  You can select the tables to process.  I selected all of them.  I ran this on 5 different WordPress databases and had no problems.

But updating the themes and non-database content is all manual.  If you've got a modestly sized site you're likely to find additional updates for a long time to come. 

Redirect HTTP

So Apache is configured and the web sites updated.  That's a good start but we need to push a little harder on this and to do that we need to go back to the Apache configuration files.  Now we need to add, just above the SSLConfigurationFile entry, the following.

RewriteCond %{HTTPS} off
RewriteCond %{HTTP_HOST} ^example\.com [NC]
RewriteRule ^(.*)$ https://www.example.com/$1 [R=301,L]

These are used to map requests for any http://<domain> to your server to be remapped to the HTTPS URL instead.  This is done to deal with anything you missed in the database or other web site content. But as much as you might hope, this isn't the last edit to those files.

Validate your work

All the updates are in place (so we think).  Time to test that you're a good web citizen.  Go to SSLLabs site and type in your domain.  Let it run for a bit – it's doing lots of checking.  When it's done you get a grade.  Anything less than an A and you have more work to do. 

All the updates work but it fails validation from SSLLabs.  You get a message about the chain being incomplete and some errors related to some protocols.  The following, added just above the SSLProtocol configuration line, should take care of the protocol errors. I got these from the folks who manage my colo server so can't say what they're trying to do, or even if they are correct.  But they're pretty smart dudes.  And dudettes. 

SSLProtocol all -SSLv3
    SSLHonorCipherOrder on
    SSLCipherSuite “EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSSa !RC4”

A harder problem is the incomplete chain problem.  The solution came from a non-english site, which you can decipher fairly easily.  You should also check out the Apache configuration documentation.  The gist is that we need to use a different set of Apache configurations that includes the SSLCACertificateFile entry.

    SSLCertificateFile /etc/letsencrypt/live/www.piboxproject.com/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/www.piboxproject.com/privkey.pem
    SSLCACertificateFile /etc/letsencrypt/live/www.piboxproject.com/fullchain.pem

Notice that SSLCertificateFile now points to the cert.pem, which is the server-only certificate.  And the new SSLCACertificateFile points to the fullchain.pem, which is a bunch of certificates I can't explain.  Easy peasy, right? 

Now run your domains through SSLLabs again.  If you get an A then congratulations and yes, you're the teacher's pet.  If you get anything else just remember: no refunds on your tuition turkey.

What I skipped

SEO.  I didn't care.  I don't have a godzillion (which is a google + 1) followers.  And I'm not selling anything.  So dealing with SEO isn't an issue for me.  If it is for you, continue rubbing your magic lantern. Something will come up.  On the flip side, since switching to HTTPS the number of visitors on my sites has gone up.  Okay, not a lot.  But at my level any bump is a mountain.

Some of the SSL configuration options I put in the domain configuration files may be more appropriately placed in the ssl.conf file instead.  I didn't try this.  I didn't even look into this.  By the time I figured out this might be an option I had run out of sweat and enchiladas were burning a hole in my stomach.  I opted for Rolaids instead.

Certificate renewal.  Look it up.  Basically, those certificates don't last forever.  You need to renew them.  Fortunately, by the time you get to this is it should be fairly simple.  If you know how to set up a cron job.  If you don't, go find that lantern again.

That's it.  This wasn't supposed to solve your problems.  And it probably didn't.  But hopefully it gave you some pointers to get started at solving them. Good luck!!