8.2 SSL Automation with Let’s Encrypt Certificates

Veeam Cloud Connect uses SSL (specifically, TLS) certificates to encrypt and secure connections between all the remote clients and the cloud gateways. for this reason, managing SSL certificates is an important aspect of any Cloud Connect environment.

There are many manual options for managing SSL certificates, here however we will describe how to fully automate their installation and renewal using Let’s Encrypt.

Let’s Encrypt (LE for brevity) is a fairly new certificate authority backed by some of the internet’s biggest players, including: the Electronic Frontier Foundation, Mozilla, Google and many others. LE eliminates the complex process of manual certificate creation, validation, signing, installation and even renewal by instead leveraging an automated “DevOps style” approach with open source command line tooling built upon an open standard called ACME (Automated Certificate Management Environment).

Another difference is the price. They offer full certificates FOR FREE. And they are not one of those trial certificates that last for 15 days to let you complete your tests. LE certificates last for 90 days but you can renew them as many times as you want; the only limit is 5 certs for the same host per week. This short renew cycle has some valid reasons:

• They limit damage from key compromise and mis-issuance. Stolen keys and mis-issued certificates are valid for a shorter period of time;
• They encourage automation, which is absolutely essential for ease-of-use. If we’re going to move the entire Web to HTTPS, we can’t continue to expect system administrators to manually handle renewals. Once issuance and renewal are automated, shorter lifetimes won’t be any less convenience than longer ones.

8.2.1 Posh-ACME and Let’s Encrypt

ACME is the name of the libraries created to interact with LE system. Because Cloud Connect runs over Microsoft Windows, we can use a tool like Posh-ACME to consume these libraries on a Windows machine.

Note: in the previous iterations of this chapter, I used ACMESharp, but the tool lately was not developed and kept up-to-date, and had a lot of issues. So, I decided to use a better tool, that is Posh-ACME.

First, we need to be sure that our Windows server is running at least .NET 4.7.1. This can be checked directly via Powershell:

write-host "Checking if .NET 4.7.1 is installed..."
$Net = (Get-ItemProperty "HKLM:SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full").Release -ge 461308
if($Net -eq $true){
       write-output ".NET 4.7.1 is installed."
}
else{
       write-output ".NET 4.7.1 is NOT installed. Please install it before using this script."
}

Once the needed version of .NET is installed, we can activate the script with some variables, and load the Posh-ACME module:

function Load-Module ($m) {

    # If module is imported say that and do nothing
    if (Get-Module | Where-Object {$_.Name -eq $m}) {
        write-host "Module $m is already imported."
    }

    else {
        # If module is not imported, but available on disk then import
        if (Get-Module -ListAvailable | Where-Object {$_.Name -eq $m}) {
            Import-Module $m
        }

        else {
            # If module is not imported, not available on disk, but is in online gallery then install and import
            if (Find-Module -Name $m | Where-Object {$_.Name -eq $m}) {
                Install-Module -Name $m -Force -Scope CurrentUser
                Import-Module $m
            }

            else {
                # If module is not imported, not available and not in online gallery then abort
                write-host "Module $m not imported, not available and not in online gallery, exiting."
                EXIT 1
            }
        }
    }
}

Load-Module "Posh-ACME"

Load-Module "AWSPowerShell"

8.2.2 First configuration of AWS Route53

Posh-ACME has integrated DNS plugins for a multitude of DNS services. In this way it’s really easy to configure the script to interact with a given DNS service and use a TXT record as the challenge to validate the ownership of the domain. In this example, we are using AWS Route53. If you need to adjust this tutorial for a different DNS service, the Posh-ACME site has a long list of user guides for almost every DNS service, here: https://poshac.me/docs/v4/Plugins/.

First, in AWS I create new IAM policy, that will allow to only do DNS record updates. This is the JSON text I need to use:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "route53:GetHostedZone",
                "route53:ChangeResourceRecordSets",
                "route53:ListResourceRecordSets"
            ],
            "Resource": "arn:aws:route53:::hostedzone/XXXXXXXXXXXXX"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "route53:ListHostedZones",
            "Resource": "*"
        }
    ]
}

Compared to the policy you can find in the ACME tutorial, mine doesn’t allow editing of every DNS hosted zone. Instead, to make it even more secure, I only allow the edit of the hosted zone listed in the Resource line. Replace the XXXXXXXXXXXXX text with the ARN of your DNS zone.

Once the policy-Route53-virtualtothecorecom-vcc-le is created, I used it to create a new user:

by applying the policy itself to the user I’m creating:

In the last page, AWS will show me the Access Key and Secret Key of the new user. I’ll save it for later usage.

Finally, in my Veeam server, I save these credentials as a new AWS profile (replace the xxxxxxxx with your real keys):

Set-AWSCredential -StoreAs 'vcc-le-updater' -AccessKey 'xxxxxxxx' -SecretKey 'xxxxxxxx'
Initialize-AWSDefaultConfiguration -ProfileName 'vcc-le-updater'

8.2.3 Automatic validation via DNS

Now, I load the newly created profile:

$AWSprofile = "vcc-le-updater"
Set-AWSCredential -ProfileName $AWSprofile

And I then run the command:

$domain = "cc.virtualtothecore.com"
$r53Params = @{R53ProfileName=$AWSprofile}
New-PACertificate $domain -DnsPlugin Route53 -PluginArgs $r53Params -AcceptTOS

and I wait 2 minutes for the script to complete. This delay is added on purpose to allow DNS propagation once the TXT record has been updated, and at the end the script grabs the newly created certificate automatically:

The certificate are stored in this location:

Get-PACertificate -MainDomain $domain | fl

8.2.4 Certificate installation

Replacing an old certificate with a new one in Veeam Cloud Connect is simple. There are two commands in Veeam PowerShell related to Cloud Connect certificates:

Get-VBRCloudGatewayCertificate

Add-VBRCloudGatewayCertificate.

Before importing the certificate, I change a configuration option in the registry of the Windows machine where Veeam Backup & Replication server is installed:

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\CloudIgnoreInaccessibleKey (DWORD) = 1

With this key, I instructed Veeam Server to avoid checking the private key exportability; this check fails on recent certificates, such as the ones created by Let’s Encrypt, because the private key is embedded into the full certificate.

Once the Veeam services have been restarted to enable the new registry key (this has to be done only once), I retrieve the certificate from the location described in the previous paragraph, I give it a meaningful name by using the currente date, and I copy it into an easy location:

$certname = "vcc-$(get-date -format yyyyMMdd)"
$pfxfile = "C:\Posh-ACME\$certname.pfx"
$pfxpass = 'poshacme'
$cert = Get-PACertificate -MainDomain $domain
Copy-Item -Path $cert.PfxFile -Destination C:\Posh-ACME\$certname.pfx

Note: the pfxpass value is poshacme, because this is the default password applied to every certificate created by Let’s Encrypt. It can be changed during the certificate creation by appending the proper switch:

$pfxpass = "mynewsupersecretpassword"
New-PACertificate $domain -DnsPlugin Route53 -PluginArgs $r53Params -PfxPass $pfxpass

With the certificate at hand, I can use this script to install it into the certificate store of the machine, and then apply it to Veeam Cloud Connect:

$securepfxpass = $pfxpass | ConvertTo-SecureString -AsPlainText -Force
Import-PfxCertificate -FilePath $pfxfile -Password $securepfxpass -CertStoreLocation cert:\localMachine\my -Exportable

Connect-VBRServer -Server localhost

$thumbprint = Get-PACertificate -MainDomain $domain
$thumbprint = $thumbprint.Thumbprint
$certificate = Get-VBRCloudGatewayCertificate -FromStore | Where {$_.Thumbprint -eq $thumbprint}

Add-VBRCloudGatewayCertificate -Certificate $certificate

Disconnect-VBRServer

To properly identify the certificate I used its thumbprint, as this is a unique code that allows to identify each certificate.

Finally, if I go into the Manage Certificates section of Cloud Connect and I select the option Keep existing certificate, I can see the newly installed certificate, ready to be used:

8-LE-final_cert_loaded.png

Renewals

Let’s Encrypt certificates only last 90 days. This is done on purpose to improve the security in case of a compromise of the certificate. Because the entire platform is consumed via API, it shouldn’t be a problem to refresh certificates so often. Once the first certificate has been created and installed, we can run a different script to renew it.

In Posh-ACME, PluginArgs are saved to the local profile and tied to the current ACME account. This is what enables easy renewals. It also means you can generate additional certificates without having to specify the PluginArgs parameter again as long as you’re using the same DNS plugin. However, because new values overwrite old values, it means that you can’t use different sets of parameters for different certificates unless you create a different ACME account.

In my case, all the parameters are going to be the same, so during the renewal I can run this simple command:

Submit-Renewal

This will inherit all the parameters already used for the first creation of the certificate, as long as I use the same ACME account. For this reason I run this command in a powershell script that is then scheduled using the same credentials I used to run the first script. There’s a minimum amount of time that Let’s Encrypt asks you to wait before they will renew a certificate:

8.9: Renewal minimum wait time

but this can be overridden using the switch -Force.