hetzel.net

Cold Storage

Cold Storage

I really don't want to experience another data loss like the one with this blog. It wasn't a big deal in this case, but what about genuinely important data for your own business?

If you're really serious about it, you can even work with LTO tapes. The tapes last for decades and cost practically nothing per GB, but the drives tend to have five-figure price tags. The software ecosystem around these drives is also geared more toward enterprise customers with enterprise budgets. And even then, you still don't have a fancy robot DJing your tape library.

Practically every major cloud provider offers some flavor of cold storage with reduced storage costs compared to regular object storage like S3. Some disclose which media the data is stored on, others don't. Some specify whether the data is stored in geographically separate locations, others don't. Pricing sometimes includes a hefty restore fee, occasionally with an additional egress fee when data leaves the cloud provider's internal network. Some have minimum retention periods or minimum data volumes, others have a base fee, and some restores can involve waiting several days.

The various offerings are quite confusing, and the costs can be difficult to estimate. Here's a quick summary of what I looked at:

AWS, Google Cloud, Azure

All have corresponding offerings, without providing many details about the underlying storage technology. Depending on the selected storage class, data is either distributed across multiple Availability Zones within a region or stored at a single location. Cross-region replication is sometimes possible but costs extra. Restore and egress can get very expensive.

Hetzner Storage Box

A RAID with ZFS snapshots on a single server, available in either Falkenstein or Helsinki. No replication to other servers. Sizes: 1/5/10/20 TB. Easy to sync between boxes, no minimum contract terms, free egress, limited shell access. Snapshots reside on the same Storage Box and are no substitute for an independent backup.

OVH Cold Storage

What a mess, and their dashboard is even worse.

Swift Public Cloud Archive

My old account still has access to this. Upload via the OpenStack Swift client. In the cloud, larger objects are split into 256 MB segments, stored, and frozen. For a restore, they have to be thawed (unfreeze/unseal). This took 10 minutes in my test.

As usual with OVH, the network was relatively flaky during testing. From a server with a gigabit connection, I could get no more than 250 Mbit/s, averaging closer to 100 Mbit/s.

€2.41 per TB-month under my existing plan, free egress.

Cold Archive v1

Apparently no longer available to my account. Magnetic tapes distributed across four data centers in France. Cheaper than v2, but with a minimum size of 1 TB per archival operation. The older version archives entire buckets rather than individual objects.

Cold Archive v2

Magnetic tapes in four data centers in France. Erasure coding 2+2, meaning the data remains recoverable even if two of the four fragments are lost.

Up to 48 hours of waiting for a restore. Granularity: individual objects that can be archived via S3-compatible object storage.

Cost: €1.66 per TB-month and €9 per TB for restores, excluding VAT. Free egress.

Minimum retention period: 180 days. During a restore, the data is temporarily made available in object storage again, which incurs additional storage costs.

Other Candidates

  • Scaleway Glacier: Actually located in an underground bunker, €2.54 per TB-month.
  • Cloudflare R2 Infrequent Access: $10 per TB-month, a $9 "base fee" for operations you might not even perform, plus $10 per TB for retrieval and a minimum billing period of 30 days. Free egress.
  • Backblaze B2: $6.95 per TB-month. Not really cold storage. Free egress up to three times the average amount of data stored per month.

My New Setup

Not quite in this state yet, but this is roughly what I'm planning to set up, while also doing the CDN work for podcast audio delivery.

  • CDN 1: Cloudflare with Cloudflare R2 behind it, bucket in the EU.
  • CDN 2: OVH Object Storage (France), can be activated via DNS if R2 goes down.
  • Backup 1: Hetzner server (Falkenstein), holds a copy of R2. Happens to have a large, unused HDD, so I'm excluding it from the cost calculation.
  • Backup 2: Hetzner Storage Box (Helsinki), which syncs from that server.
  • Cold Storage: OVH Cold Archive v2 (France), also populated from Falkenstein.

This also provides some protection against accidental deletions, ransomware, etc.: The Storage Box has ZFS snapshots, and the cold storage is append-only. And hopefully, I still have local copies on the NAS.

Cost per TB: approximately €40/month by my calculations, excluding existing hardware and any major restore operations. Only marginally more than before, but a gazillion times better protected.