9.2 SSL Automation with Let’s Encrypt Certificates

Veeam Cloud Connect uses SSL (specifically, TLS) certificates to encrypt and secure connections between all the remote clients and the cloud gateways. for this reason, managing SSL certificates is an important aspect of any Cloud Connect environment.

There are many manual options for managing SSL certificates, here however we will describe how to fully automate their installation and renewal using Let’s Encrypt.

Let’s Encrypt (LE for brevity) is a fairly new certificate authority backed by some of the internet’s biggest players, including: the Electronic Frontier Foundation, Mozilla, Google and many others. LE eliminates the complex process of manual certificate creation, validation, signing, installation and even renewal by instead leveraging an automated “DevOps style” approach with open source command line tooling built upon an open standard called ACME (Automated Certificate Management Environment).

Another difference is the price. They offer full certificates FOR FREE. And they are not one of those trial certificates that last for 15 days to let you complete your tests. LE certificates last for 90 days but you can renew them as many times as you want; the only limit is 5 certs for the same host per week. This short renew cycle has some valid reasons:

  • They limit damage from key compromise and mis-issuance. Stolen keys and mis-issued certificates are valid for a shorter period of time;
  • They encourage automation, which is absolutely essential for ease-of-use. If we’re going to move the entire Web to HTTPS, we can’t continue to expect system administrators to manually handle renewals. Once issuance and renewal are automated, shorter lifetimes won’t be any less convenience than longer ones.

Posh-ACME and Let’s Encrypt

ACME is the name of the libraries created to interact with LE system. Because Cloud Connect runs over Microsoft Windows, we can use a tool like Posh-ACME to consume these libraries on a Windows machine.

Note: in the previous iterations of this chapter, I used ACMESharp, but the tool lately was not developed and kept up-to-date, and had a lot of issues. So, I deciced to use a better tool, that is Posh-ACME.

First, we need to be sure that our Windows server is running at least .NET 4.7.1. This can be checked directly via Powershell:

write-host "Checking if .NET 4.7.1 is installed..."
$Net = (Get-ItemProperty "HKLM:SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full").Release -ge 461308
if($Net -eq $true){
       write-output ".NET 4.7.1 is installed."
}
else{
       write-output ".NET 4.7.1 is NOT installed. Please install it before using this script."
}

Once the needed version of .NET is installed, we can activate the script with some variables, and load the Posh-ACME module:

function Load-Module ($m) {

    # If module is imported say that and do nothing
    if (Get-Module | Where-Object {$_.Name -eq $m}) {
        write-host "Module $m is already imported."
    }

    else {
        # If module is not imported, but available on disk then import
        if (Get-Module -ListAvailable | Where-Object {$_.Name -eq $m}) {
            Import-Module $m
        }

        else {
            # If module is not imported, not available on disk, but is in online gallery then install and import
            if (Find-Module -Name $m | Where-Object {$_.Name -eq $m}) {
                Install-Module -Name $m -Force -Scope CurrentUser
                Import-Module $m
            }

            else {
                # If module is not imported, not available and not in online gallery then abort
                write-host "Module $m not imported, not available and not in online gallery, exiting."
                EXIT 1
            }
        }
    }
}

Load-Module "Posh-ACME"

Load-Module "AWSPowerShell"

Automatic validation via DNS

Posh-ACME has integrated DNS plugins for a multitude of DNS services. In this way it’s really easy to configure the script to interact with a given DNS service and use a TXT record as the challenge to validate the ownership of the domain. In this example, we are using AWS Route53.

So, first operation, I load my AWS credentials of a user who is capable of updating DNS records:

$AWSprofile = "vcc-le-updater"
Set-AWSCredential -ProfileName $AWSprofile

You can check in the Posh-ACME tutorial what IAM role you need to create to have this user, together with many other useful information about the usage of Route53.

I then run the command:

$r53Params = @{R53ProfileName=$AWSprofile}
New-PACertificate $domain -DnsPlugin Route53 -PluginArgs $r53Params

and I wait 2 minutes for the script to complete. This delay is added on purpose to allow DNS propagation once the TXT record has been updated:

Posh-ACME execution

The script also grabs the newly created certificate automatically:

Posh-ACME retrieves the certificate

The certificate are stored in this location:

Get-PACertificate -MainDomain $domain | fl

Posh-ACME certificate location

Certificate installation

Replacing an old certificate with a new one in Veeam Cloud Connect is simple. There are two commands in Veeam PowerShell related to Cloud Connect certificates:

Get-VBRCloudGatewayCertificate

Add-VBRCloudGatewayCertificate.

Before importing the certificate, we change a configuration option in the registry of the Windows machine where Veeam Backup & Replication server is installed:

HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\CloudIgnoreInaccessibleKey (DWORD) = 1

With this key, we instructed Veeam Server to avoid checking the private key exportability; this check fails on recent certificates, such as the ones created by Let’s Encrypt, because the private key is embedded into the full certificate.

Once the Veeam services have been restarted to enable the new registry key (this has to be done only once), we retrieve the certificate from the location described in the previous paragraph and we copy it into an easy location:

$certname = "vcc-$(get-date -format yyyyMMdd)"
$pfxfile = "C:\Posh-ACME\$certname.pfx"
$cert = Get-PACertificate -MainDomain $domain
Copy-Item -Path $cert.PfxFile -Destination C:\Posh-ACME\$certname.pfx

With the certificate at hand, we can use this script to install it into the certificate store of the machine, and then apply it to Veeam Cloud Connect:

$securepfxpass = $pfxpass | ConvertTo-SecureString -AsPlainText -Force
Import-PfxCertificate -FilePath $pfxfile -Password $securepfxpass -CertStoreLocation cert:\localMachine\my -Exportable

asnp VeeamPSSnapin
Connect-VBRServer -Server localhost

$thumbprint = Get-PACertificate -MainDomain $domain
$thumbprint = $thumbprint.Thumbprint
$certificate = Get-VBRCloudGatewayCertificate -FromStore | Where {$_.Thumbprint -eq $thumbprint}

Add-VBRCloudGatewayCertificate -Certificate $certificate

Disconnect-VBRServer

To properly identify the certificate I used its thumbprint, as this is a unique code that allows to identify each certificate.

Also, one note about the PFX password: by default is poshacme, but we can change it during the certificate creation by appending the proper switch:

$pfxpass = "poshacme"
New-PACertificate $domain -DnsPlugin Route53 -PluginArgs $r53Params -PfxPass $pfxpass

Renewals

Let’s Encrypt certificates only last 90 days. This is done on purpose to improve the security in case of a compromise of the certificate. Because the entire platform is consumed via API, it shouldn’t be a problem to refresh certificates so often. Once the first certificate has been created and installed, we can run a different script to renew it.

In Posh-ACME, PluginArgs are saved to the local profile and tied to the current ACME account. This is what enables easy renewals. It also means you can generate additional certificates without having to specify the PluginArgs parameter again as long as you’re using the same DNS plugin. However, because new values overwrite old values, it means that you can’t use different sets of parameters for different certificates unless you create a different ACME account.

In my case, all the parameters are going to be the same, so during the renewal I can run this simple command:

Submit-Renewal

This will inherit all the parameters already used for the first creation of the certificate, as long as I use the same ACME account. For this reason I run this command in a powershell script that is then scheduled using the same credentials I used to run the first script. There’s a minimum amount of time that Let’s Encrypt asks you to wait before they will renew a certificate:

Renewal minimum wait time

but this can be overridden using the switch -Force.