Sujit Patel – Site Reliability Engineer

Kubernetes is an open-source container orchestration system that provides an infrastructure for deploying, scaling, and managing containerized applications. As Kubernetes deployments grow, the need for backup and disaster recovery strategies becomes increasingly important. One such solution is Velero, a popular open-source tool that provides an easy way to backup and restore a Kubernetes cluster.

What is Velero?

Velero is a tool that can be used to backup and restore an entire Kubernetes cluster, including its persistent volumes, cluster resources, and application metadata. The tool can also be used to migrate applications between clusters.

Velero: https://velero.io/

In this article, we'll discuss the basics of Velero and how to use it for backing up and restoring a Kubernetes cluster.

Prerequisites

Before getting started with Velero, you need to have the following:

A running Kubernetes cluster
An object storage solution that supports the S3 API, such as Amazon S3, Google Cloud Storage, or MinIO.

In this article, I'm using Amazon EKS and Amazon S3 bucket

Overview

Create S3 Bucket
Create IAM policy
Create AWS IAM user with access and secret key
Install velero CLI
Install Velero in K8s Cluster
Backing up a Cluster
Restoring a Cluster
Schedule a Backup
Deleting Backups

Steps:

Create S3 Bucket

I'm using S3 but you can you Google Cloud Storage, or MinIO as well. Velero will use this to store backups.

Log in to the AWS Management Console.
Navigate to the S3 service.
Click the "Create bucket" button.
Enter a unique name for the bucket in the "Bucket name" field. S3 bucket names must be globally unique across all AWS regions.
Select a region for the bucket. It's recommended to choose the region closest to your users for better performance.
Click the "Create bucket" button to finish creating the S3 bucket.

Create IAM policy and IAM user

we need to create IAM user and Policy for Velero to access our S3 bucket.

To Create AWS Policy

Log in to the AWS Management Console.
Navigate to the IAM service.
In the navigation panel, click "Policies."
Click the "Create policy" button.
Give the policy a name and description that describes what it will allow.
In the policy document section, you'll need to specify the permissions that the policy will grant. The policy document is written in JSON format and uses AWS resource ARNs (Amazon Resource Names) to specify the resources that the policy will grant access to.
Click the "Create policy" button to create the policy.

Sample Policy file for Velero:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeVolumes",
                "ec2:DescribeSnapshots",
                "ec2:CreateTags",
                "ec2:CreateVolume",
                "ec2:CreateSnapshot",
                "ec2:DeleteSnapshot"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::nf-prod-velero/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::nf-prod-velero"
            ]
        }
    ]
}

Create AWS IAM user with access and secret key

Log in to the AWS Management Console.
Navigate to the IAM service.
In the navigation panel, click "Users."
Click the "Add users" button.
Give username and select "Attach policies directly"
Select your created policy and click "create user"
Now, select your user go to the "Security credentials" tab, and click "Create access key"

Save your access key and secret key somewhere safe we need this in later steps.

Install velero CLI

To install Velero CLI

Download the Velero CLI binary for your operating system from the official Velero release page: https://github.com/vmware-tanzu/velero/releases
Make the binary executable: chmod +x velero
Move the binary to a directory in your PATH, such as /usr/local/bin: mv velero /usr/local/bin/
Verify the installation by running velero version

Alternatively, you can use the following command to install the Velero CLI via Homebrew on macOS:

brew install velero

Install Velero in K8s Cluster

To install Velero in k8s cluster we need credentials file with AWS access and secret key.

Create a file name credentials with below content.

[default]
aws_access_key_id = <access-key>
aws_secret_access_key = <secret-key>
region=<region>

Don't forget to replace the placeholders with the actual value.

Run the below command to install velero in k8s cluster

velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.6.0 \
    --bucket <your-s3-bucket-name> \
    --backup-location-config region=<region> \
    --snapshot-location-config region=<region> \
    --secret-file ./credentials \
    --use-restic

Note: If you’re running on AWS, and taking EBS snapshots as part of your regular Velero backups, there’s no need to switch to using Restic. However, if you need a volume snapshot plugin for your storage platform, or if you’re using EFS, AzureFile, NFS, emptyDir, local, or any other volume type that doesn’t have a native snapshot concept, Restic might be for you.

For more info: https://velero.io/docs/v1.9/restic/

Backing up a Cluster

To take full backup run the below command

velero backup create my-full-backup

The velero backup create command creates a backup.
my-full-backup is the name of the backup.

Above command will take backup of all namespaces and all resources.

Backup specific namespace:

velero backup create backup-all-specific-namespace --include-namespaces my-namespace

The velero backup create command creates a backup.
The backup-all-specific-namespace is the name of the backup.
The --include-namespaces flag specifies the namespace(s) to include in the backup. In this case, it's the my-namespace namespace.

Backup a specific resource in some namespace:

velero backup create backup-specific-resource-namespace --include-resources deployment.apps/nginx --include-namespaces some-namespace

The velero backup create command creates a backup.
The backup-specific-resource-namespace is the name of the backup.
The --include-resources flag specifies the resource(s) to include in the backup. In this case, it's the deployment.apps/nginx deployment.
The --include-namespaces flag specifies the namespace(s) to include in the backup. In this case, it's the some-namespace namespace.

Restoring a Cluster

To Restore from backup

velero restore create --from-backup my-full-backup

The velero restore create command creates a restore operation.
The --from-backup flag specifies the backup to restore from. In this case, it's the my-full-backup backup.

Restore without persistent volume

velero restore create --from-backup my-full-backup --restore-volume-snapshots=false

The --restore-volume-snapshots flag controls whether to restore Persistent Volumes and their snapshots. In this case, it's set to false, meaning that Persistent Volumes and snapshots will not be restored.

Schedule a Backup

You can schedule a Velero backup to run regularly using a Kubernetes CronJob resource. A CronJob runs a job on a schedule defined in a cron expression.

Here's an example of how to create a CronJob that runs a Velero backup every day at midnight:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: daily-backup
spec:
  schedule: "0 0 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: velero/velero:v1.10.0
            command:
            - /velero
            args:
            - backup
            - create
            - daily-backup
            - --include-namespaces
            - default
          restartPolicy: OnFailure

The apiVersion and kind specify that this is a CronJob resource.
The metadata section contains the name of the CronJob, in this case daily-backup.
The spec section contains the schedule for the CronJob defined in the schedule field. In this case, it's 0 0 * * *, which means run every day at midnight.
The jobTemplate section specifies the Job resource that the CronJob will run.
The template section specifies the Pod that will run the Velero command.
The containers section specifies the container that will run the Velero command. It uses the velero/velero image, and runs the velero backup create command with the daily-backup backup name and includes resources in the default namespace.
The restartPolicy is set to OnFailure, which means the Pod will not be restarted if it fails.

Once you have defined the CronJob, you can create it in your Kubernetes cluster using the kubectl apply command:

kubectl apply -f cronjob.yaml

Where cronjob.yaml is the file containing the CronJob definition.

Deleting Backups

You can delete a Velero backup using the following command:

velero backup delete <backup-name>

The velero backup delete command deletes a Velero backup.
The <backup-name> is the name of the backup you want to delete.

Note that deleting a backup will permanently remove all of the backed up data and it cannot be recovered. Before deleting a backup, you may want to verify that you have a copy of the data stored elsewhere.

Bonus

List backups

velero backup get

The command will display information about each backup, including its name, creation time, and status.

Check backup status

velero backup describe <backup-name>

This command will display information about the backup, including its status, creation time, and the resources included in the backup. The status of the backup will be one of the following:

In Progress
Completed
Failed

Check the status of restore

velero restore describe <restore-name>

The velero restore describe command retrieves information about a restore.
The <restore-name> is the name of the restore you want to check the status of.

This command will display information about the restore, including its status, creation time, and the resources included in the restore. The status of the restore will be one of the following:

In Progress
Completed
Failed

Check all restore(s)

velero restore get

The command will display information about each restore, including its name, creation time, and status.