Configure SSL Termination with Varnish Caching and HTTP/2

To create the fastest web pages possible in a LAMP stack can be a bit tricky.  This document covers configureing blazing fast HTTPS sites on Freebsd with Apache, Varnish, Nginx, PHP, Mysql and letsencrypt.sh.  Of course you can swap out the PHP/Mysql parts as needed, or even use your own certificates instead of lets encrypt.sh as you need.

Updated 2/16/2017: Added code to nginx.conf for additional rewriting; added an extra check so requests don't end up with multiple Access-Control-Allow-Origin headers.
Updated 2/27/2017: Added code to apache and varnish to seamlessly minify CSS and Javascript resources.
Updated 3/1/2017: Added details about Varnish Purging, plugins, and custom code.


The Varnish software provides incredibly fast cached website hosting in front of standard web services.  It does however have one major limitation that it won’t serve HTTPS content.  This guide demonstraights setting up a Nginx proxy service infront of the Varnish cache to serve cached HTTPS content.  This configuration also allows for specific SSL sites to be served through Varnish caching and/or allows other sites to bypass the caching if the HTTPS to HTTP translation doesn't work properly.  In addition is seamlessly translates local domain HTTP links and resources to HTTPS so the original site doesn’t need to be edited and can serve both HTTP and HTTPS from he same cached files.  In addition a header is added to seamlessly update external resources to HTTPS to prevent Mixed Content warnings.  In addition it will minify resources automatically thus simplifying the CMS/code side.

This setup will allow the following request paths to serve content:

•  Client Browser -> Varnish (http:80) -> Apache (http:8080)
• Normal HTTP request served through cache.
• Client Browser -> Varnish (http:80) -> 301 to HTTPS Version
• Optional rule to force a site to use SSL if a non-SSL page is requested.
• Client Browser -> Nginx (https:443) -> Varnish (http:80) -> Apache (HTTP:8080)
• Normal HTTPS request served through cache
• All links to HTTP://ThisDomainName/ are converted to HTTPS://ThisDomainName/.
• Header is added to upgrade external HTTP resources to HTTPS.
• Client Browser -> Nginx (https:443) -> Apache (HTTPS:44344)
• If my be needed bypass caching for specific sites if they don't work properly with cache configuration.
• It is also possible to just exclude specific sites or url's from caching and/or minify in the varnish configuration. That approach should be taken where possible and this used as the last resort.

Some other notes:

• You could of course put these services on different servers, or have multiple IP Addresses on the server if you prefer, but I find it much simpler to logical separate things by putting the backend services on non-standard ports instead.
• You could host the frontend and backend components both in Nginx if you really wanted, but again this way things are more logical spirited and easier to follow.
• Nginx will use HTTP2 when available for much faster delivery of many small resource files.  Site auditing/speed test warnings thus far don't take this into account and will still complain about the number of files even though it's not relevant on HTTP2.
• Exceptions can be added in Varnish to easily prevent specific directories or URL's from being cached.
• By default any cookies that vary by request or by user will break the cache.  There are some predefined cookies that are ignored, and more can be added as needed.  In general a cookie shouldn't be changing how the server responds so in most cases you can safely ignore all cookies.
• letsencrypt.sh is used in this document to automate SSL certificates for all the sites.  Standard certificates can use used in places of specific sites if prefered.
• This configuration uses a Apache+Ruby instigation on minify CSS and JS scripts on the fly.  You would not normally want to do this because of the server cost of processing requests, but with the vanish cache layer this has not cost as the result is cached.  This greatly simplifies the backend/code and doesn't require pre-minified files, though if those are detected they will be passed through unchanged.
• In addition when using HTTP/2 you should disable File compilation plugins and features as they will actually slow the page down by loading more resources then needed.  In HTTP/2 a smaller lists of individual files is better then one larger compiled file.

Some example performance numbers: (Time to download HTML / All assets loaded)

• Serving a simple Wordpress page: Uncached
• 441ms; 1.6s
• Serving a simple Wordpress page: Cached
• 63ms, 1.1s
• Serving a complex Magneto storefront page: Uncached
• With configurations: 7900ms, 9.25s
• Simple product: 1690s, 1.95s
• Serving a complex Magneto storefront page: Cached
• With configurations: 121ms, 2.0s
• Simple product: 143ms, 1.0s

Server configuration

If you already have some of this set up you may not need to configure everything listed below.

Installing Required Packages

This guide assumes the following in a FreeBSD environment.  Install them as needed or install your own preferences.  The setup will work in other linux systems as well though some of the services will be configured different and the file paths may change.

• Mysql/PHP/etc
• openssl
• Varnish 4 or grater
• Apache 2.4 or greater
• Nginx 1.11 or greater
• lets encrypt.sh 0.2 or greater (or you can use your own certificates)
• ruby 2.2 (optional, for seamless resource minification)
• ruby-gems
• rubygem-cssmin
• rubygem-jsmin

Activate Services and Configure

Once you have a servqer up and running with the required software you will want to activate the various services and configure them on boot.  The following should be added to your /etc/rc.conf file to set the default services.

apache24_enable="YES"
mysql_enable="YES"
varnishd_enable="YES"
varnishd_config="/usr/local/etc/varnish.vcl"
#varnishd_extra_flags="-p http_resp_hdr_len=16192"
#varnishd_listen=":8088"
#varnishd_backend="localhost:80"
#varnishd_jailuser="www"
nginx_enable="YES" # Proxy SSL for HTTP/2

In addition the following needs to be added to your /etc/periodic.conf file get SSL certificates to automatically renew.

weekly_letsencrypt_enable="YES"
#weekly_letsencrypt_deployscript="/usr/local/etc/letsencrypt.sh/deploy.sh"

Configure LetsEncrypt.sh

LetsEncrypt.sh doesn't really put the certificates anyplace useful and the certificates need to be manipulated as well.  The following code needs to be added to the /usr/local/etc/letsencrypt.sh/hook.sh file inside the deploy_cert function.

# Copy certificates to standard SSL folder.
cp $CERTFILE /usr/local/etc/ssl/$DOMAIN.crt
cp $CHAINFILE /usr/local/etc/ssl/$DOMAIN.chain.crt
cp $KEYFILE /usr/local/etc/ssl/$DOMAIN.key
# Merge certificate chain files.
cat $CERTFILE $CHAINFILE > /usr/local/etc/ssl/$DOMAIN.merge.crt
# Adjust permissions so that web servers can access certificates.
chgrp www /usr/local/etc/ssl/$DOMAIN.crt /usr/local/etc/ssl/$DOMAIN.chain.crt /usr/local/etc/ssl/$DOMAIN.key /usr/local/etc/ssl/$DOMAIN.merge.crt
chmod g+r /usr/local/etc/ssl/$DOMAIN.crt /usr/local/etc/ssl/$DOMAIN.chain.crt /usr/local/etc/ssl/$DOMAIN.key /usr/local/etc/ssl/$DOMAIN.merge.crt 
echo "Copied certs for $CERTFILE on `date`." >> /var/log/letsencrypt

Configure PHP

There is lots of possible configuration here that I wont get into but in brief you will probably want to enable OP Cache by setting opcache.enable=1. There is more information here.

Configure Backend Apache Web Server

You will need to make a few changes to the default apache configuration as indicated below.

In /usr/local/etc/apache24/httpd.conf make sure the following lines are uncommented or updated as indicated.

# Change the HTTP listen port.
Listen 8080

# Load required Modules
LoadModule deflate_module libexec/apache24/mod_deflate.so
LoadModule ssl_module libexec/apache24/mod_ssl.so
LoadModule vhost_alias_module libexec/apache24/mod_vhost_alias.so
LoadModule rewrite_module libexec/apache24/mod_rewrite.so
LoadModule php5_module        libexec/apache24/libphp5.so
LoadModule ext_filter_module libexec/apache24/mod_ext_filter.so // Optional for minification
LoadModule expires_module libexec/apache24/mod_expires.so // Optional to inject expire headers

# Replace %h with %{X-Forwarded-For}i in all LogFormat settings
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" common

# Virtual hosts
Include etc/apache24/extra/httpd-vhosts.conf

# Secure (SSL/TLS) connections
Include etc/apache24/extra/httpd-ssl.conf

Also edit the /usr/local/etc/apache24/extra/httpd-ssl.conf file to change the HTTPS listen port.

# Change the HTTPS listen port.
Listen 44344

Create a new file /usr/local/etc/apache24/modules.d/030_ext_filter_module.conf for the minification feature.

<IfModule mod_ext_filter.c>
  ExtFilterDefine jsmin \
                  mode=output \
                  intype=application/javascript \
                  outtype=application/javascript \
                  cmd="/usr/local/bin/ruby -e 'require \"rubygems\"; require \"jsmin\"; code = STDIN.read;  begin; min = JSMin.minify(code); rescue; min = code; ensure; puts min; end;'"
  <If "%{HTTP:X-Minify} =~ /js/">
    AddOutputFilter jsmin js
  </If>
</IfModule>

<IfModule mod_ext_filter.c>
  ExtFilterDefine cssmin \
                  mode=output \
                  intype=text/css \
                  outtype=text/css \
                  cmd="/usr/local/bin/ruby -e 'require \"rubygems\"; require \"cssmin\"; code = STDIN.read;  begin; min = CSSMin.minify(code); rescue; min = code; ensure; puts min; end;'"
  <If "%{HTTP:X-Minify} =~ /css/">
    AddOutputFilter cssmin css
  </If>
</IfModule>

Create a new file /usr/local/etc/apache24/modules.d/040_expires_module.conf for the append expire headers feature. Tweak these values as you like.

<IfModule mod_expires.c>
ExpiresActive on
ExpiresByType image/gif "access plus 1 months"
ExpiresByType image/jpg "access plus 1 months"
ExpiresByType image/jpeg "access plus 1 months"
ExpiresByType image/png "access plus 1 months"
ExpiresByType text/js "access plus 1 days"
ExpiresByType text/javascript "access plus 1 days"
ExpiresByType application/javascript  "access plus 1 days"
ExpiresByType text/css  "access plus 1 days"
</IfModule>

You can start Apache now with sudo service apache24 start to verify the configuration though there are no sites to test yet.

Configure Varnish Caching Service

Next you can set up a default rules file for the varnish caching. This is rather complex and can have a lot of custom rules and variations depending your sites, software, and preference. I have included the /usr/local/etc/varnish.vcl file that I use, though some of the rules may not be applicable to your sites. This file is configured for general Wordpress sites and Magento storefronts and there are some comments and details in the file.

You can start Varnish now with sudo service varnishd start to verify the configuration though there are no sites to test yet.

Configure Frontend Nginx Proxy Server

Finally configure the Nginx proxy service by editing you /usr/local/etc/nginx/nginx.conf file and adding or updating the contents as indicated.

http {
    # Append Access-Control-Allow-Origin only if it's not already present.
    map $upstream_http_access_control_allow_origin $access_control_allow_origin {
        '' "https://$host always";
    }

    # Enable compression
    gzip  on;
    gzip_types text/css application/javascript;
    gzip_disable        "MSIE [1-6]\.";
    gzip_comp_level 5;

    # Transcode all local resource, form actions, and links from HTTP to HTTPS.
    sub_filter 'http://$host/' 'https://$host/';
    sub_filter 'http:\/\/$host\/' 'https:\/\/$host\/';
    sub_filter_last_modified on;
    sub_filter_once off;
    sub_filter_types application/xml text/xml;

    # Add a header to tell the client browser to automatically request HTTPS resources form remote sites instead of HTTP to prevent mixed content warnings.
    add_header Content-Security-Policy upgrade-insecure-requests;
    #Add a header to allow for HTTPPS scripting calls to be handled by HTTP in some edge cases where the transcoded URL's missed something.
    add_header Access-Control-Allow-Origin $access_control_allow_origin;

    # Add default headers for the proxy request.
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP  $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-Port 443;
    proxy_set_header Accept-Encoding "";
    proxy_read_timeout 600;
}

You can start Nginx now with sudo service nginx start to verify the configuration though there are no sites to test yet.

Add and Configure a new Website

Now that the server is configured properly you can set up your sites.  Repeat this process for each site.

Configure DNS

Configure your DNS to point at the server.  You may want to use a custom HOSTS record on your computer for testing instead or when migrating a site.

Load the Site Files

Load your site file and configure as needed. The instructions below assume your site files are loaded into folders using the site domain name like /usr/local/www/www.DomainName.com/ but you can change this if you prefer.

Though it seams counterintuitive you will want to disable resource compilation, minify, and compression in your CMS/site code. Compilation will actually slow down requests a but on HTTP/2 and cost extra bandwidth in many cases. Similarly compression is not needed in Apache as Varnish will take care of that.

As for the CMS systems minifying files, this can be turned off and Apache can do this at runtime.  This way you can easily turn it off in your session for testing without affecting other users and you don't need to deal with the overhead and possible performance costs of many of the plugins.  This setup is very efficient because the files are minified just before they are cached in varnish.

Add an Apache Virtual Host

Create the backend website in Apache by editing your /usr/local/etc/apache24/extra/httpd-vhosts.conf file and adding a new section like this. Make sure to replace the DomainName with the proper value.

<VirtualHost *:8080>
    #ServerAdmin AdminEmailAddress
    DocumentRoot /usr/local/www/www.DomainName.com
    <Directory "/usr/local/www/www.DomainName.com">
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
    ServerName www.DomainName.com
    ServerAlias DomainName.com
    Alias /.well-known/acme-challenge/ /usr/local/etc/letsencrypt.sh/.acme-challenges/
    ErrorLog "/var/log/apache/www.DomainName.com-error_log"
    CustomLog "/var/log/apache/www.DomainName.com-access_log" common
</VirtualHost>

If your domain name starts with a www. you can also add the domain name minus the www into the ServerAlias field so it responds to both. If you have other domains that also point to the same site they should also be added to the ServerAlias field, then you can add 301 redirect rules in a .HTACCESS file to 301 requests to a default hostname if you prefer.

You also need to update /etc/hosts and find the ::1 localhost and 127.0.0.1 localhost lines and add your primary domain to the end of each like so.

::1   localhost www.DomainName.com
127.0.0.1  localhost www.DomainName.com

Finally restart Apache now with sudo service apache24 graceful to start up the new site and navigate to http://www.DomainName.com to see it, or on the server by typing curl -i --http2 http://www.DomainName.com | less.  You can add :8080 to the end of the URL to see the uncached version, though depending on your network configuration this may not work.

Generate SSL Certificate

To generate a new SSL Certificate for your site edit the /usr/local/etc/letsencrypt.sh/domains.txt file and add your DomainName to it along with all the alternate domain names and/or non-www versions like so.

www.DomainName.com DomainName.com www.DomainName2.com

Each separate site should be a separate line in this file and any alternate domain names listed in the ServerAlias field in Apache should be included on the line after the primary domain name.  The primary domain name/first entry will be used below to configure the SSL sites.  Once done run the following command to generate the new certificates manually, from then on they will be automatically updated as needed.

/usr/local/bin/letsencrypt.sh -c

Configure Nginx Proxy Site

And now on to the crux of this article, configuring proxied SSL sites through the Varnish caching service.  To do this create a new site in Nginx by editing your /usr/local/etc/nginx/nginx.conf file updating the DomainName as appropriate.

server {
    listen       443 ssl http2;
    server_name  www.DomainName.com DomainName.com;
    ssl_certificate    /usr/local/etc/ssl/www.DomainName.com.merge.crt;
    ssl_certificate_key /usr/local/etc/ssl/www.DomainName.com.key;

   location / {
            proxy_pass http://localhost:80;
    }
}

Here you will need to again include the same additional domain specified in theServerAlias field in Apache after the primary domain name in the server_name field.  You can restart Nginx now with sudo service nginx restart and navigate to https://www.DomainName.com to see the SSL version of the site, or on the server by typing curl -i --http2 https://www.DomainName.com | less.

Testing and Controlling Transparent Minification

Minification should work by default most of the time. If your getting empty css or javascript files or nothing is being minified run the following commands on the server to ensure the proper packages are installed and working.

echo "p {
    color: red;
    font-weight: bold;
}" | /usr/local/bin/ruby -e 'require "rubygems"; require "cssmin"; code = STDIN.read; begin; min = CSSMin.minify(code); rescue; min = code; ensure; puts min; end;'

echo "(function() {
    console.log('test');
})();" | /usr/local/bin/ruby -e 'require "rubygems"; require "jsmin"; code = STDIN.read; begin; min = JSMin.minify(code); rescue; min = code; ensure; puts min; end;'

You can switch override the default processing rules by setting a cookie in your browser.

document.cookie="minify=disable;" // disable for this domain
document.cookie="minify=force;" // force for this domain
document.cookie="minify=;" // use default rules for this domain

In addition in /usr/local/etc/varnish.vcl you can permanently completely turn this feature off, block specific domains, or block individual scripts or css files from being processed. Also note that files named .min.css or .min.js won't be processed.

SSL Related Issues

This is where things can get somewhat complicated.  As Varnish can't process HTTPS content, we are serving the entire site through HTTP and the proxy server at the end is delivering it as HTTPS.  There are all sorts of issues this can cause in CMS systems though if for example the CMS system is checking the URL protocol and returning a 301 in an attempt to force SSL that we would end up in a redirect loop.  This would happen in Wordpress for example if we set the Site Address URL or WordPress Address URL to https:// instead of http://.  Some systems like this we can simply leave the validation URL as  http://www.DomainName.com and everything will work out correctly due to the URL translating logic in Nginx, but others may not.  Depending on your software it may also be possibly to identify the protocol validation code and add additional code to check the X-Forwarded-Proto header in addition to the request protocol before redirecting.  You may also be able to disable some Force SSL settings to fix redirect issues, or in Wordpress deactivate and SSL plugins.

Another issue you may run into is that caching is not occurring between different sessions or users.  This would generally happen because users are ending up with different cookies.  The following command will let you monitor requests as they come in to see if they are being cached or not and troubleshoot the issues.  You may need to add some more cookie exceptions in /usr/local/etc/varnish.vcl and restart.  Some trial and error may be needed here.

sudo varnishncsa  -F  "%{X-Forwarded-For}i %{X-Forwarded-Proto}i://%{Host}i%U %{Varnish:handling}x %{Varnish:hitmiss}x"

Force SSL

Assuming that everything is working correctly with the SSL configuration you can force all traffic or only specific sites to use SSL.  To do this we need to do it in the Varnish configuration as the backend is all non-SSL and we don't want redirect loops or complex conditional code.  To do this edit your /usr/local/etc/varnish.vcl file find the # Optinal rule to force SSL line. Add or edit it as needed with your new DomainName.

# Optinal rule to force SSL Everywhere for listed domains.
if ( (req.http.host ~ "^(www.DomainName.com|www.AnotherDomainName.com)") && req.http.X-Forwarded-Proto !~ "https") {
    set req.http.x-redir = "https://" + req.http.host + req.url;
    return (synth(750, ""));
}

# Or force all requests to SSL
if ( req.http.X-Forwarded-Proto !~ "https") {
    set req.http.x-redir = "https://" + req.http.host + req.url;
    return (synth(750, ""));
}

Add Caching exceptions

If you have directories or sections that should not be cached, or if there are headers or cookies that should prevent caching when present you can add rules to your /usr/local/etc/varnish.vcl file to prevent caching in the # Add additinal rules here section.  Here are some examples:

# Bypass shopping cart and checkout requests
if (req.url ~ "/(cart|my-account|checkout|addons)") {
    # Don't cache, pass to backend
    return (pass);
}

# Either the admin pages or the login
if (req.url ~ "/wp-(login|admin|cron)") {
    # Don't cache, pass to backend
    return (pass);
}

# Custom Scripts and API
if (req.url ~ "/api/") {
    # Don't cache, pass to backend
    return (pass);
}

As you are making changes you can test them by running sudo service varnishd reload so that the entire cache isn't cleared. Additionally you can easily check which requests are hitting the cache with this command.

sudo varnishncsa  -F  "%{X-Forwarded-For}i %{Varnish:time_firstbyte}x %m %{X-Forwarded-Proto}i://%{Host}i%U %{Varnish:handling}x %{Varnihitmiss}x"

Configuring a CMS to Purge it's Cache

The next major hurdle is that as you are eloping or tweaking your CMS/site you will quickly discover  that the varnish caching layer is caching changes and won't check the backend for updates.  You can restart the entire service with sudo service varnishd restart but that will clear the cache for all your sites and it quite cumbersome.  Instead you will want to configure a plugin or custom code for your site to just clear it's cache on demand.

To configure varnish integration in Magento log in to the backend and go to Store -> Configuration ->  Advanced -> System -> Full Page Cache.  Set the Caching Application to Varnish Cache and save the changes.  In addition you will need to edit your app/etc/env.php file and this section at the end just before the last line.  Additionally this patch has to be applied to your Magneto install of it will clear all site's cache's instead of just the current site.

In wordpress you simply need to install the Varnish HTTP Purge plugin and your good to go.  Please note that you will get unexpected results and poor performance if you didn't add the primary hostname to your servers /etc/hosts file as indicated above.

If you have some other system of custom software you can create your own scripts or internal plugins to purge resources as needed.  A script simply needs to send HTTP request with a type of PURGE and a host header specifying the domain to purge.  It can purge the entire domain, or purge a specific url.  Some examples that can be used directly or converted into your CMS's language:

# Do wildcard patching to purge a directory or the entire site
curl -H "Host: www.DomainName.com" -H "X-Purge-Method: regex" -X PURGE http://127.0.0.1/.* 
# Purge a specific file/url including url parameter
curl -H "Host: www.DomainName.com" -H "X-Purge-Method: exact" -X PURGE http://127.0.0.1/some/file.html?param
# Purge all versions of a file/url ignoring any parameter tokens
curl -H "Host: www.DomainName.com" -H "X-Purge-Method: default" -X PURGE http://127.0.0.1/some/file.html 

Bypass Caching Site

If your you have some sites that are causing issues and not working properly through the Varnish cache, you can bypass the cache for specific sites.  To do this first you need to add a backed SSL site in Apache by editing your /usr/local/etc/apache24/extra/httpd-vhosts.conf file and adding a new section like this. Make sure to replace the DomainName with the proper value.

<VirtualHost *:44344>
    #ServerAdmin AdminEmailAddress
    DocumentRoot /usr/local/www/www.DomainName.com
    <Directory "/usr/local/www/www.DomainName.com">
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
        AddOutputFilterByType DEFLATE text/css
        AddOutputFilterByType DEFLATE application/javascript
    </Directory>
    ServerName www.DomainName.com
    ServerAlias AnotherDomainName.com
    ErrorLog "/var/log/apache/www.DomainName.com-error_log"
    CustomLog "/var/log/apache/www.DomainName.com-access_log" common
    SSLEngine on
    SSLCertificateFile /usr/local/etc/ssl/www.DomainName.com.crt
    SSLCertificateKeyFile /usr/local/etc/ssl/www.DomainName.com.key
    SSLCertificateChainFile /usr/local/etc/ssl/www.DomainName.com.chain.crt
</VirtualHost>

Again if your domain name starts with a www. you can also add the domain name minus the www into the ServerAlias field so it responds to both. If you have other domains that also point to the same site they should also be added to the ServerAlias field. Then restart Apache now with sudo service apache24 graceful.

Next update the Nginx configuration by editing your /usr/local/etc/nginx/nginx.conf file and changing the line below in the proper server section.

proxy_pass http://localhost:44344;

Again restart Nginx now with sudo service nginx graceful and at this point the SSL site should work correctly, though its pages won't be cached any longer.