9.2 SSL Automation with Let’s Encrypt Certificates

Veeam Cloud Connect uses SSL (specifically, TLS) certificates to encrypt and secure connections between all the remote clients and the cloud gateways. for this reason, managing SSL certificates is an important aspect of any Cloud Connect environment.

There are many manual options for managing SSL certificates, here however we will describe how to fully automate their installation and renewal using Let’s Encrypt.

Let’s Encrypt (LE for brevity) is a fairly new certificate authority backed by some of the internet’s biggest players, including: the Electronic Frontier Foundation, Mozilla, Google and many others. LE eliminates the complex process of manual certificate creation, validation, signing, installation and even renewal by instead leveraging an automated “DevOps style” approach with open source command line tooling built upon an open standard called ACME (Automated Certificate Management Environment).

Another difference is the price. They offer full certificates FOR FREE. And they are not one of those trial certificates that last for 15 days to let you complete your tests. LE certificates last for 90 days but you can renew them as many times as you want; the only limit is 5 certs for the same host per week. This short renew cycle has some valid reasons:

  • They limit damage from key compromise and mis-issuance. Stolen keys and mis-issued certificates are valid for a shorter period of time;
  • They encourage automation, which is absolutely essential for ease-of-use. If we’re going to move the entire Web to HTTPS, we can’t continue to expect system administrators to manually handle renewals. Once issuance and renewal are automated, shorter lifetimes won’t be any less convenience than longer ones.

ACMEsharp and Let’s Encrypt first interaction

ACME is the name of the libraries created to interact with LE system. Because Cloud Connect runs over Microsoft Windows, we can use a tool like ACMESharp to consume these libraries on a Windows machine.

The ACMESharp module can be quickly installed using:

install-Module -Name ACMESharp

Next, we import the installed module:

Import-Module ACMESharp

We can add this import operation at the beginning of every script involving Let’s Encrypt

Then we need to prepare ACMESharp. First we initialize a Vault, that is a location in the filesystem where ACMESharp will store Certificates and related artefacts. Note that if you run as Administrator, your Vault will be created in a system-wide path, otherwise it will be created in a private, user-specific location. We chose to run all the PowerShell commands as Administrator:

Initialize-ACMEVault

Then, we register with the Let’s Encrypt CA, providing a method of contact like an email:

New-ACMERegistration -Contacts mailto:user@domain.com -AcceptTos

Then, we submit the DNS domain name that we want to secure with a PKI certificate. In our example:

New-ACMEIdentifier -Dns cc.virtualtothecore.com -Alias vcc

The answer from the API is like this:

IdentifierPart : ACMESharp.Messages.IdentifierPart
IdentifierType : dns
Identifier     : cc.virtualtothecore.com
Uri            : https://acme-v01.api.letsencrypt.org/acme/authz/aseriesofrandomletters
Status         : pending
Expires        : 1/29/2017 9:45:33 PM
Challenges     : {, , }
Combinations   : {2, 0, 1}

Since this is the first time we ask a certificate to LE for this domain, we need to prove the domain ownership. We can do this in different ways, and we opted for the DNS challenge. You can also place a text file in your website, if this is what you are going to encrypt. To see which specific information Let’s Encrypt wants, we run this command:

Complete-ACMEChallenge vcc -ChallengeType dns-01 -Handler manual

== Manual Challenge Handler - DNS ==
* Handle Time:      [1/22/2017 10:59:59 PM]
* Challenge Token:  [somerandomletters]
To complete this Challenge please create a new Resource
Record (RR) with the following characteristics:
* RR Type:  [TXT]
* RR Name:  [_acme-challenge.cc.virtualtothecore.com]
* RR Value: [6iS2hd2l1MuxvOAm-KvIrBB-v2YBPO6r92iU8E9MUFw]
------------------------------------

We are asked to create a new TXT record in the DNS zone, using the value we received.

We edit the DNS Zone, create the TXT record, and so we are able to complete the challenge:

Submit-ACMEChallenge vcc -ChallengeType dns-01

The status may still be “pending”, as the LE server immediately accepts the submission, but performs the validation asynchronously. So you will not get an immediate response to the status of your submission, but you can check the status of the submission by running multiple times this command:

(Update-ACMEIdentifier vcc -ChallengeType dns-01).Challenges | Where-Object {$_.Type -eq "dns-01"}

After a few seconds the status will change from pending to valid. If the Challenge fails for any reason you will see a status of invalid. Once the Challenge has been successfully validated, we can check the overall status of the Domain Identifier, which should be valid as well:

Update-ACMEIdentifier vcc

Certificate creation

Now that we proved our domain ownership, we can request our first certificate:

#Load ACMESharp module
import-module ACMESharp

$alias = "vcc"
$certname = "vcc-$(get-date -format yyyy-MM-dd-HH-mm)"
$pfxfile = "C:\ProgramData\ACMESharp\sysVault\$certname.pfx"

# Change to the Vault folder
cd C:\ProgramData\ACMESharp\sysVault

# Generate a certificate
New-ACMECertificate ${alias} -Generate -Alias $certname

#Submit the certificate
Submit-ACMECertificate $certname

# Hit until values are filled in
update-AcmeCertificate $certname

pause

# Export Certificate to PFX file
Get-ACMECertificate $certname -ExportPkcs12 $pfxfile

The final result is the certificate, already released by LE and ready to be installed: The LE folder in our server

By adding this additional line to the script, we import the certificate into the Local Machine store:

Import-PfxCertificate -CertStoreLocation cert:\localMachine\my -Exportable -FilePath $pfxfile

Directory: Microsoft.PowerShell.Security\Certificate::LocalMachine\my

Thumbprint                                Subject
----------                                -------
597D2247AEF1A60E6A2E6C945044881A8676A433  CN=cc.virtualtothecore.com

Certificate installation

Replacing an old certificate with a new one in Veeam Cloud Connect is simple. There are two commands in Veeam PowerShell related to Cloud Connect certificates, Get-VBRCloudGatewayCertificate and Add-VBRCloudGatewayCertificate.

Before importing the certificate, we change a configuration option in the registry of the Windows machine where Veeam Backup & Replication server is installed:

HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\CloudIgnoreInaccessibleKey (DWORD) = 1

With this key, we instructed Veeam Server to avoid checking the private key exportability; this check fails on recent certificates, such as the ones created by Let’s Encrypt, because the private key is embedded into the full certificate.

Once the Veeam services have been restarted to enable the new registry key (this has to be done only once), we quickly install the certificate:

asnp VeeamPSSnapin
Connect-VBRServer -Server localhost
$certificate = Get-VBRCloudGatewayCertificate -FromStore | Where {$_.SerialNumber -eq $serialnumber}
Add-VBRCloudGatewayCertificate -Certificate $certificate
Disconnect-VBRServer
Return

Here we load the Veeam PowerShell snap-in, we connect to the local Veeam Backup Server, we identify the new LE certificate by using the serial number, and we install the certificate into Veeam Cloud Connect.

Using AWS Route53 to automate the DNS Challenges

The limit of the code explained above is in the DNS handler, that is the part where we receive a new TXT resource record that we need to update into our DNS zone. The script use the “manual” option: once we receive in the output the new DNS txt record we have to pause the script and go to our DNS server to update the Resource Record. With AWS Route53 DNS services, we can also automate the renew of the TXT record.

First, we create a dedicated IAM user to update the DNS record. Thanks to AWS IAM (Identity and Access Management) we can create a set of credentials with just the minimum required permissions. For our use case, we only need: ChangeResourceRecordSets

We also want to limit this account to the domain we want to work with, that is virtualtothecore.com in our example. In AWS terms, we need the Amazon Resource Names (ARNs) to identify the hosted DNS zone. The format is this:

arn:aws:route53:::hostedzone/zoneid

There’s no way (as of now) to limit the account to a specific resource record, so we need to accept the fact that this account can possibly change (and then damage) every record in the zone.

The fcomplete rule in JSON format for our user is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "route53:ChangeResourceRecordSets",
            "Resource": "arn:aws:route53:::hostedzone/ZQUSU6S6339VA"
        }
    ]
}

Then, we parse the text we receive from LE challenge to extract the TXT string:

# Get the new TXT record from Let's Encrypt

$RRtext = Select-String challenge.txt -Pattern "RR Value" -CaseSensitive | Out-String -Stream
$separator = "["

$RRtext = $RRtext.split($separator)
$RRtext = $RRtext[2]
$RRtext = $RRtext.trimend("]")
$RRtext = """$RRtext"""

The text from Let’s Encrypt is always the same, and we need the string with the value of the resource record. This is clearly identified with the RR Value text in it. Then, with some string operations the code trims the un-needed text before and after, and adds the quotes around the text. The final result is our new TXT record in proper DNS format:

"j23gJf_rJzsmKxcVQ-UOhezeT96dzvXLA3UiBxSN5Kw"

Then, we need to update the record. for this operation, we first need to configure the credentials that will be used in the script to connect to AWS. In order to make the script more secure and to do not share the credentials, those are not stored in the script. Instead we can use this code to store them in a profile:

$access = "THISISNOTMYREALACCESSOKEY" #Access key ID
$secret = "THISISNOTMYREALSECRETKEY" #Secret access key
Set-AWSCredential -AccessKey $access -SecretKey $secret -StoreAs awsroute53

The credentials are store in a profile, and we just need to load it each time we need to use it:

Set-AWSCredential -ProfileName awsroute53

We then go and update the TXT record:

$change = New-Object Amazon.Route53.Model.Change

$change.Action = "UPSERT"
$change.ResourceRecordSet = New-Object Amazon.Route53.Model.ResourceRecordSet
$change.ResourceRecordSet.Name = "_acme-challenge.cc.virtualtothecore.com."
$change.ResourceRecordSet.Type = "TXT"
$change.ResourceRecordSet.TTL = 300
$change.ResourceRecordSet.ResourceRecords = (New-Object Amazon.Route53.Model.ResourceRecord($RRtext))

$params = @{
   HostedZoneId="ROUTE53ZONEID"
     ChangeBatch_Comment="Updated TXT record for cc.virtualtothecore.com. with new Let'sEncrypt challenge"
     ChangeBatch_Change=$change
}

Edit-R53ResourceRecordSet @params

The Complete script

The complete script is available in GitHub at https://github.com/dellock6/veeam-powershell/blob/master/letsencrypt-cloudconnect-aws.ps1 .