Restore Data from S3-Compatible Storage Using BR

This document describes how to restore the backup data stored in S3-compatible storages to a TiDB cluster on Kubernetes, including two restoration methods:

Full restoration. This method takes the backup data of snapshot backup and restores a TiDB cluster to the time point of the snapshot backup.
Point-in-time recovery (PITR). This method takes the backup data of both snapshot backup and log backup and restores a TiDB cluster to any point in time.

The restore method described in this document is implemented based on CustomResourceDefinition (CRD) in TiDB Operator. For the underlying implementation,BRis used to restore the data. BR stands for Backup & Restore, which is a command-line tool for distributed backup and recovery of the TiDB cluster data.

PITR allows you to restore a new TiDB cluster to any point in time of the backup cluster. To use PITR, you need the backup data of snapshot backup and log backup. During the restoration, the snapshot backup data is first restored to the TiDB cluster, and then the log backup data between the snapshot backup time point and the specified point in time is restored to the TiDB cluster.

Note

BR is only applicable to TiDB v3.1 or later releases.
PITR is only applicable to TiDB v6.3 or later releases.
Data restored by BR cannot be replicated to a downstream cluster, because BR directly imports SST and LOG files to TiDB and the downstream cluster currently cannot access the upstream SST and LOG files.

Full restoration

This document provides an example about how to restore the backup data from thespec.s3.prefixfolder of thespec.s3.bucketbucket on Amazon S3 to thedemo2TiDB cluster in thetest2namespace. The following are the detailed steps.

Prerequisites: Complete the snapshot backup

In this example, themy-full-backup-folderfolder in themy-bucketbucket of Amazon S3 stores the snapshot backup data. For steps of performing snapshot backup, refer toBack up Data to S3 Using BR.

Step 1: Prepare the restore environment

Before restoring backup data on a S3-compatible storage to TiDB using BR, take the following steps to prepare the restore environment:

创建一个namespace for managing restoration. The following example creates arestore-testnamespace:


                
                 kubectl create namespace restore-test

Downloadbackup-rbac.yaml, and execute the following command to create the role-based access control (RBAC) resources in therestore-testnamespace:


                
                 kubectl apply -f backup-rbac.yaml -n restore-test

Grant permissions to the remote storage for therestore-testnamespace.
If the data to be restored is in Amazon S3, you can grant permissions in three methods. For more information, seeAWS account permissions.
If the data to be restored is in other S3-compatible storage (such as Ceph and MinIO), you can grant permissions byusing AccessKey and SecretKey.
For a TiDB version earlier than v4.0.8, you also need to complete the following preparation steps. For TiDB v4.0.8 or a later version, skip these preparation steps.
1. Make sure that you have theSELECTandUPDATEprivileges on themysql.tidbtable of the target database so that theRestoreCR can adjust the GC time before and after the restore.
2. Create therestore-demo2-tidb-secretsecret to store the account and password to access the TiDB cluster:
```
                  
                   kubectl create secret generic restore-demo2-tidb-secret --from-literal=password=${password}--namespace=test2
```

Step 2: Restore the backup data to a TiDB cluster

Depending on which method you choose to grant permissions to the remote storage when preparing the restore environment, you can restore the data by doing one of the following:

Method 1: If you grant permissions by importing AccessKey and SecretKey, create theRestoreCR恢复集群data as described below:


                
                 kubectl apply -f restore-full-s3.yaml

The content ofrestore-full-s3.yamlis as follows:


                
                 ---
                 apiVersion:
                 m.rzhenli.com/v1alpha1
                 kind:
                 Restore
                 metadata:
                 name:
                 demo2-restore-s3
                 namespace:
                 restore-test
                 规范:
                 br:
                 cluster:
                 demo2
                 clusterNamespace:
                 test2
                 # logLevel: info
                 # statusAddr: ${status_addr}
                 # concurrency: 4
                 # rateLimit: 0
                 # timeAgo: ${time}
                 # checksum: true
                 # sendCredToTikv: true
                 # # Only needed for TiDB Operator < v1.1.10 or TiDB < v4.0.8
                 # to:
                 # host: ${tidb_host}
                 # port: ${tidb_port}
                 # user: ${tidb_user}
                 # secretName: restore-demo2-tidb-secret
                 s3:
                 provider:
                 aws
                 secretName:
                 s3-secret
                 region:
                 us-west-1
                 bucket:
                 my-bucket
                 prefix:
                 my-full-backup-folder

Method 2: If you grant permissions by associating IAM with Pod, create theRestoreCR恢复集群data as described below:


                
                 kubectl apply -f restore-full-s3.yaml

The content ofrestore-full-s3.yamlis as follows:


                
                 ---
                 apiVersion:
                 m.rzhenli.com/v1alpha1
                 kind:
                 Restore
                 metadata:
                 name:
                 demo2-restore-s3
                 namespace:
                 restore-test
                 annotations:
                 iam.amazonaws.com/role:
                 arn:aws:iam::123456789012:role/user
                 规范:
                 br:
                 cluster:
                 demo2
                 sendCredToTikv:
                 false
                 clusterNamespace:
                 test2
                 # logLevel: info
                 # statusAddr: ${status_addr}
                 # concurrency: 4
                 # rateLimit: 0
                 # timeAgo: ${time}
                 # checksum: true
                 # Only needed for TiDB Operator < v1.1.10 or TiDB < v4.0.8
                 to:
                 host:
                 ${tidb_host}
                 port:
                 ${tidb_port}
                 user:
                 ${tidb_user}
                 secretName:
                 restore-demo2-tidb-secret
                 s3:
                 provider:
                 aws
                 region:
                 us-west-1
                 bucket:
                 my-bucket
                 prefix:
                 my-full-backup-folder

Method 3: If you grant permissions by associating IAM with ServiceAccount, create theRestoreCR恢复集群data as described below:


                
                 kubectl apply -f restore-full-s3.yaml

The content ofrestore-full-s3.yamlis as follows:


                
                 ---
                 apiVersion:
                 m.rzhenli.com/v1alpha1
                 kind:
                 Restore
                 metadata:
                 name:
                 demo2-restore-s3
                 namespace:
                 restore-test
                 规范:
                 serviceAccount:
                 tidb-backup-manager
                 br:
                 cluster:
                 demo2
                 sendCredToTikv:
                 false
                 clusterNamespace:
                 test2
                 # logLevel: info
                 # statusAddr: ${status_addr}
                 # concurrency: 4
                 # rateLimit: 0
                 # timeAgo: ${time}
                 # checksum: true
                 # Only needed for TiDB Operator < v1.1.10 or TiDB < v4.0.8
                 to:
                 host:
                 ${tidb_host}
                 port:
                 ${tidb_port}
                 user:
                 ${tidb_user}
                 secretName:
                 restore-demo2-tidb-secret
                 s3:
                 provider:
                 aws
                 region:
                 us-west-1
                 bucket:
                 my-bucket
                 prefix:
                 my-full-backup-folder

When configuringrestore-full-s3.yaml, note the following:

For more information about S3-compatible storage configuration, refer toS3 storage fields.
Some parameters in.spec.brare optional, such aslogLevel,statusAddr,concurrency,rateLimit,checksum,timeAgo, andsendCredToTikv. For more information about BR configuration, refer toBR fields.
For v4.0.8 or a later version, BR can automatically adjusttikv_gc_life_time. You do not need to configurespec.to字段RestoreCR.
For more information about theRestoreCR fields, refer toRestore CR fields.

After creating theRestoreCR, execute the following command to check the restore status:


              
               kubectl get restore -n restore-test -o wide


              
               NAME STATUS ... demo2-restore-s3 Complete ...

Point-in-time recovery

This section provides an example about how to perform point-in-time recovery (PITR) in ademo3cluster in thetest3namespace. PITR takes two steps:

Restore the cluster to the time point of the snapshot backup using the snapshot backup data in thespec.pitrFullBackupStorageProvider.s3.prefixfolder of thespec.pitrFullBackupStorageProvider.s3.bucketbucket.
Restore the cluster to any point in time using the log backup data in thespec.s3.prefixfolder of thespec.s3.bucketbucket.

The detailed steps are as follows.

Prerequisites: Complete data backup

In this example, themy-bucketbucket of Amazon S3 stores the following two types of backup data:

The snapshot backup data generated during thelog backup, stored in themy-full-backup-folder-pitrfolder.
The log backup data, stored in themy-log-backup-folder-pitrfolder.

For detailed steps of how to perform data backup, refer toBack up data to Azure Blob Storage.

Note

The specified restoration time point must be between the snapshot backup time point and the log backupcheckpoint-ts.

Step 1: Prepare the restoration environment

Before restoring backup data on S3-compatible storages to TiDB using BR, take the following steps to prepare the restoration environment:

创建一个namespace for managing restoration. The following example creates arestore-testnamespace:


                
                 kubectl create namespace restore-test

Downloadbackup-rbac.yaml, and execute the following command to create the role-based access control (RBAC) resources in therestore-testnamespace:


                
                 kubectl apply -f backup-rbac.yaml -n restore-test

Grant permissions to the remote storage for therestore-testnamespace.
If the data to be restored is in Amazon S3, you can grant permissions in three methods. For more information, seeAWS account permissions.
If the data to be restored is in other S3-compatible storage (such as Ceph and MinIO), you can grant permissions byusing AccessKey and SecretKey.

Step 2: Restore the backup data to a TiDB cluster

The example in this section restores the snapshot backup data to the cluster. The specified restoration time point must be betweenthe time point of snapshot backupand theLog Checkpoint Tsof log backup.

PITR grants permissions to remote storages in the same way as snapshot backup. The example in this section grants permissions by using AccessKey and SecretKey.

The detailed steps are as follows:

创建一个RestoreCR nameddemo3-restore-s3in therestore-testnamespace and specify the restoration time point as2022-10-10T17:21:00+08:00:


                
                 kubectl apply -f restore-point-s3.yaml

The content ofrestore-point-s3.yamlis as follows:


                
                 ---
                 apiVersion:
                 m.rzhenli.com/v1alpha1
                 kind:
                 Restore
                 metadata:
                 name:
                 demo3-restore-s3
                 namespace:
                 restore-test
                 规范:
                 restoreMode:
                 pitr
                 br:
                 cluster:
                 demo3
                 clusterNamespace:
                 test3
                 s3:
                 provider:
                 aws
                 region:
                 us-west-1
                 bucket:
                 my-bucket
                 prefix:
                 my-log-backup-folder-pitr
                 pitrRestoredTs:
                 "2022-10-10T17:21:00+08:00"
                 pitrFullBackupStorageProvider:
                 s3:
                 provider:
                 aws
                 region:
                 us-west-1
                 bucket:
                 my-bucket
                 prefix:
                 my-full-backup-folder-pitr

When you configurerestore-point-s3.yaml, note the following:

spec.restoreMode: when you perform PITR, set this field topitr. The default value of this field issnapshot, which means snapshot backup.

Wait for the restoration operation to complete:


                
                 kubectl getjobs-n restore-test


                
                 NAME COMPLETIONS ... restore-demo3-restore-s3 1/1 ...

You can also check the restoration status by using the following command:


                
                 kubectl get restore -n restore-test -o wide


                
                 NAME STATUS ... demo3-restore-s3 Complete ...

Troubleshooting

如果你遇到任何公关oblem during the restore process, refer toCommon Deployment Failures.

Download PDF Request docs changes Ask questions on Discord

Playground

New

One-stop & interactive experience of TiDB's capabilities WITHOUT registration.