3.8 Immutable Backups

Veeam Cloud Connect offers multiple options to protect backups from unwanted deletions, being them mistakes or malevolent operations from un-authorized users:

• Insider Protection
• Hardened Linux repository
• S3 Object Lock

While they all reach the same goal - avoiding unwanted deletions - the way this result can be achieved and the limits of each solution are different. Let’s see how each of them work. As Hardened Linux Repository and S3 Object Lock are well documented in general Veeam literature, we will focus on Insider Protection, and then compare the three solutions. Hopefully, this will help service providers asking the ultimate question “Which of the three solutions should I choose?”.

Insider Protection

Insider Protection is historically the first solution introduced in Veeam Cloud Connect (but I’d say also in Veeam in general) to protect against data loss attacks. It’s a specific feature of Veeam Cloud Connect, and it’s able to guarantee the safety of backups stored in Cloud Connect against mis-deletions or attacks against the tenant’s Veeam environment where remote data stored into Veeam Cloud Connect are deleted on purpose; for example during Ransomware attacks.

In fact, because Veeam Cloud Connect is connected to the tenant’s Veeam console, every backup stored there is always visible and available to the end user. This is an advantage for a legitimate usage of the solution, but it also allows an attacker that takes control of the console to delete those remote backups:

3.18: Backups stored in Cloud Connect can be deleted from the tenant’s console

In this case, Veeam Cloud Connect cannot distinguish a legitimate user from an attacker; what it’s seen is an account with the correct credentials requesting a deletion. And in this case, Cloud Connect obeys to the command.

Insider Protection can protect from this scenario. In essence, you can think about it as the Recycle Bin of Veeam Cloud Connect. It is not enabled by default, but it can be enabled specifically for any given tenant:

3.19: Enable IP for a tenant in Cloud Connect

Once IP is enabled, every file that is deleted from the tenant’s repository is not immediately deleted, but rather moved into a special folder of the Backup Repository:

3.20: The folder structure of IP

As you may notice in this screenshot, the recycle bin contains only incremental files, and this is due to the type of backup or backup copy job that has been sent to Cloud Connect. Insider Protection “simply” saves any file that is deleted on purpose or by the configured job retention. But in order to have a successful restore, a full file has to be available to properly initiate the backup chain. For this reason, tenants need to configure their backup jobs to have periodic full files in the bin, otherwise an attack will still be successful.

The backup copy job of the previous example uses regular retention, configured as in the screenshot below:

3.21: Backup Copy regular retention

This backup copy job stores 7 restore points in Cloud Connect. When retention is reached, the older restore points are deleted, and stored for additional 14 days in Insider Protection’s special folder. The problem is that - by the nature of this type of job - only ONE Full file is stored at any time in the Cloud Repository, and is only available in the primary storage. When fully utilized, the file chain would be like this:

3.22: Regular chain inside VCC with IP enabled

You can immediately visualize where the problem lies. A primary backup job with Forever Forward incremental or a regular Backup Copy job will only have one full backup, that will always be updated by merging old incrementals. For this reason this file will never be placed in the Insider Protection and thus - in case of a complete deletion of all the files in the Cloud Repository - there will be no full file to use.

For this reason, in order to make Insider Protection really effective, tenants need to use a backup mode that will periodically create Full files that will be “sealed”, that is not touched anymore once created. This will allow them to age, be deleted by job retention and thus placed in Insider Protection where they will be ultimately protected.

Veeam console at the tenant’s side will also warn if a job without periodic fulls has been configured when Insider Protection is enabled on the selected Cloud Repository:

3.23: IP job type warning

The complete text says: “Your service provider has implemented backup files protection against deletion by an insider for this cloud repository. To protect against advanced attack vectors, we recommend that you configure your cloud backup jobs to keep multiple full backups on disk (as opposed to forever-incremental chain with a single full backup).”

Tenants can for example reconfigure a Backup Copy job by also enabling GFS retention, like this:

3.24: Backup Copy GFS retention

As soon as some full files are created and deleted, this is how it will look in the file system:

3.25: The folder structure of IP with GFS

I’ve hidden the moltitude of .vib files from the screenshot, but you can clearly see that now we have one .vbk file in the Recycle Bin, that can be used when a restore is requested by a tenant. We can represent this situation with this scheme:

3.26: GFS chain inside VCC with IP enabled

Even if the entire Cloud repository is deleted, the service provider has now one consistent backup file to help his tenant to restore his data.

Insider Protection and Capacity Tier

You may have noticed in the picture 3.25 that the .vbk file stored in the Recycle Bin has a really small size. Instead of being some GB like its siblings stored in the production area, its size is just a few MB. And even if the file name states that it has been created on 2018-12-03, the last modified date is 2018-12-14.

How is it possible?

This is because the SOBR group where the backups are stored also has Capacity Tier enabled, as explain in chapter 2.8. This creates an interesting and powerful interaction between the two features, that we will explore in this section using a real example.

Our Repository is built using a Performance Tier and a Capacity Tier. The tiering option is configured to move data to the Capacity Tier after 15 days.

Then, the service provider configures Insider Protection for the tenant for 60 days of protection.

Finally, a backup copy job with GFS retention is created by the tenant, and it is configured as in picture 3.24.

As a recap:

Layer	Retention
Performance Tier	7 retention points + 2 Weekly fulls
Capacity Tier	Moves data after 15 days
Insider Protection	Keeps data for 60 days

Let’s ignore the incremental .vib files, as they are not consistent without a full to guarantee a restore, and let’s focus on the full files. With this job configuration, at any point in time we will have 3 files: 1 for the regular retention, and 2 for the weekly fulls:

3.27: IP + Capacity Tier after 14 days

At the 15th day, Capacity Tier will kick in and it will move the oldest full file to the Capacity Tier, so that its blocks will be stored in the object storage, while a dehydrated version of the vbk file will be stored locally:

3.28: IP + Capacity Tier after 15 days

At day 21, a new weekly full will be created, and because the GFS configuration is set to hold 2 weekly full, the oldest VBK file will be deleted by the job. This file however is only 21 days old, so it will be intercepted by Insider Protection and hold for other 60 days.

But what is exactly intercepted and moved?

The file that will be moved is only the dehydrated VBK file, while the blocks tiered to the Capacity tier belonging to it will stay in place. In this way, no download will happen from the Capacity tier to SOBR, and this is especially helpful when the Capacity Tier is built using public cloud services that bill for downloads.

3.28: IP + Capacity Tier after 21 days

Once the retention is reached also for the Insider Protection, finally the dehydrated VBK file is deleted, and if the blocks stored in the Capacity tier are not belonging to any dehydrated file, they will be deleted too.

As a recap, we have two possible scenarios:

Scenario	Expected actions
Job retention kicks in before Capacity tier	File is moved directly to LOCAL IP
Job retention kicks in after Capacity Tier	File is dehydrated and moved to Capacity Tier - Only dehydrated vbk is moved to local IP - Blocks stay in REMOTE IP / Capacity Tier

So, as a general reccomendation, we can suggest in order to use the Capacity Tier as the designated Recycle Bin, providers should configure the option in SOBR in order to have Capacity Tier always kicking in before the job retention.

Insider Protection vs Hardened Linux Repository vs S3 Object Lock

With three different and powerful solutions to protec data from deletion, providers may be wondering which one they should use in their environments. All have their strenghts and limits, and depending on the environment that already exists at the service provider, one solution may be preferred.

Note that price per GB is not a factor we analyse here, but it may be the deciding factor in choosing one of the solutions: some providers may have favorable agreements with specific hardware vendors, so that a solution that could sound expensive to other, could be cheap for this provider. But as we cannot control this parameter, we will only evaluate the technical aspects.

Insider Protection

This is the most simple and immediate solution that a provider can apply to an existing (or new) Cloud Connect deployment. Since it’s a feature that is enabled on the Backup Repository, once a Cloud Connect system is designed, it’s automatically ready to offer Insider Protection.

Also, it can be enabled by tenant, so a provider can choose to offer it as a premium feature only to those customers willing to pay for it.

On the other side, it’s limited as it protects files that are deleted by a dedicated command, either manually or by retention. It requires a specific backup mode (with periodic fulls) and files that are not yet deleted by retention for examples will still be subject to other possible attacks.

Finally, in it’s basic configuration it consumes the same storage of the SOBR performance tier, which may be expensive for the provider. An interesting solution to this problem can be the usage of the Capacity Tier, as shown in the previous paragraph.

Hardened Linux Repository

The Hardened Linux Repository (HLR) is located in the Performance Tier of a SOBR group. Since this tier is the absolut minimum required to build a Cloud Connect storage, during the initial design of a new deployment or the refresh of an existing one, it could be very easy to plan for a new storage server running Linux, and to configure it to be a HLR. In these terms, HLR is as transparent as the Insider Protection, because it can use existing storage servers.

The biggest advantage of HLR however is that it protects any file that lands in the storage as soon as they are written. There’s no need to wait for retention or an attack to delete a file, since each written file is protected from the beginning. It still requires to configure the backup mode accordingly, like Insider Protection, since merge operations are not possible anymore: acting as a WORM (Write once - Read many) device, once a file is written it cannot be modified. Also, retention is longer: while Insider Protection can be configured for a maximum of 99 days, HLR can be configured from 7 days (the minimum value) to 999 days (almost 3 years).

One limit of HLR is that tenants has to run at least v11 of Veeam Backup & Replication, so customers still running v10 cannot use a repository with HLR enabled. The other major limit is that HLR is enabled at the repository level, so every tenant consuming that repository will be affected by the configuration; it cannot be configured “per-tenant”.

S3 Object Lock

S3 Object lock main point of attention, in the design phase, is that requires a new and dedicated storage layer to be configured, to become the Capacity Tier of a SOBR group.

The main advantage of S3OL is that it can leverage all the capabilities of an object storage: scalability, replication, erasure coding. All these combined allow the creation of a storage layer that can grow immensely and be highly redundant. If a provider chooses to use S3OL, our suggestion could be to use a small Performance Tier that will act as a “landing area” holding backups for a small amount of time, and then move everything to the Object Storage.

Being the Capacity Tier of SOBR, S3OL is also backward compatible with previous versions of the software (v10 can use it). So, the same considerations of IP can be applied here.

The other great advantage, being a secondary tier, is that allows Cloud Connect to have two copies of each backup file arriving from the tenants (if using copy mode).

The main limit obviously is that it requires a dedicated purchase, if there’s no in-house storage yet. Also, in order to be as effective as HLR, it needs to use copy mode so that files are immediately protected.

Finally, if providers are going to use a public object storage, remember that egress costs for restores may have an impact on the cost calculations.