Deploy the HTAP Storage Engine Tiflash for an Existing TiDB Cluster

This document describes how to add or remove the TiDB HTAP storage engine TiFlash for an existing TiDB cluster on Kubernetes. As a columnar storage extension of TiKV, TiFlash provides both good isolation level and strong consistency guarantee.

Note

If a TiDB cluster has not been deployed yet, instead of referring to this document, you canconfigure a TiDB cluster on Kuberneteswith the TiFlash-related parameters, and thendeploy the TiDB cluster.

Usage scenarios

This document is applicable to scenarios in which you already have a TiDB cluster and need to use TiDB HTAP capabilities by deploying TiFlash, such as the following:

Hybrid workload scenarios with online real-time analytic processing
实时流处理场景
Data hub scenarios

Deploy TiFlash

If you need to deploy TiFlash for an existing TiDB cluster, do the following:

Note

If your server does not have an external network, you can download the required Docker image on the machine with an external network, upload the Docker image to your server, and then usedocker loadto install the Docker image on the server. For details, seedeploy the TiDB cluster.

Edit theTidbClusterCustom Resource (CR):


                
                 kubectl edit tc${cluster_name}-n${namespace}

Add the TiFlash configuration as the following example:


                
                 spec:
                 tiflash:
                 baseImage:
                 pingcap/tiflash
                 maxFailoverCount:
                 0
                 replicas:
                 1
                 storageClaims:
                 -
                 resources:
                 requests:
                 storage:
                 100Gi
                 storageClassName:
                 local-storage

TiFlash supports mounting multiple Persistent Volumes (PVs). If you want to configure multiple PVs for TiFlash, configure multipleresourcesintiflash.storageClaims, eachresourceswith a separaterequests.storageandstorageClassName. For example:


                
                 tiflash:
                 baseImage:
                 pingcap/tiflash
                 maxFailoverCount:
                 0
                 replicas:
                 1
                 storageClaims:
                 -
                 resources:
                 requests:
                 storage:
                 100Gi
                 storageClassName:
                 local-storage
                 -
                 resources:
                 requests:
                 storage:
                 100Gi
                 storageClassName:
                 local-storage

Note

When deploying TiFlash for the first time, it is recommended that you plan how many PVs are required and configure the number ofresourcesitems instorageClaimsaccordingly.
Once the deployment of TiFlash is completed, if you need to mount additional PVs for TiFlash, updatingstorageClaimsdirectly to add disks does not take effect. This is because TiDB Operator manages TiFlash by creating aStatefulSet, and theStatefulSetdoes not support modifyingvolumeClaimTemplatesafter being created.

Configure the relevant parameters ofspec.tiflash.configin TidbCluster CR. For example:
```
                
                 spec:
                 tiflash:
                 config:
                 config:
                 | [flash] [flash.flash_cluster] log = "/data0/logs/flash_cluster_manager.log" [logger] count = 10 level = "information" errorlog = "/data0/logs/error.log" log = "/data0/logs/server.log"
                
```
For more TiFlash parameters that can be configured, refer toTiFlash Configuration Documentation.
Note

For different TiFlash versions, note the following configuration differences:
- If TiFlash version <= v4.0.4, you need to setspec.tiflash.config.config.flash.service_addrto${clusterName}-tiflash-POD_NUM.${clusterName}-tiflash-peer.${namespace}.svc:3930in TidbCluster CR, where${clusterName}and${namespace}need to be replaced according to the real case.
- If TiFlash version >= v4.0.5, there is no need to manually configurespec.tiflash.config.config.flash.service_addr.
- If you upgrade from TiFlash v4.0.4 or an earlier version to TiFlash v4.0.5 or a later version, you need to delete the configuration ofspec.tiflash.config.config.flash.service_addrfrom theTidbClusterCR.

Adding PVs to TiFlash

Once the deployment of TiFlash is completed, to add PVs for TiFlash, you need to update thestorageClaimsto add disks, and then manually delete the TiFlash StatefulSet. The following are the detailed steps.

Warning

删除TiFlash StatefulSet TiFlashcluster unavailable during the deletion and affects related business. You must be cautious about whether to do the following.

Edit the TidbCluster Custom Resource (CR).


                
                 kubectl edit tc${cluster_name}-n${namespace}

TiDB Operator automatically mounts PVs in theorderof the items in thestorageClaimslist. If you need to add moreresourcesitems to TiFlash, make sure to append new items only to theendof the original items, andDO NOTmodify the order of the original items. For example:


                
                 tiflash:
                 baseImage:
                 pingcap/tiflash
                 maxFailoverCount:
                 0
                 replicas:
                 1
                 storageClaims:
                 -
                 resources:
                 requests:
                 storage:
                 100Gi
                 storageClassName:
                 local-storage
                 -
                 resources:
                 requests:
                 storage:
                 100Gi
                 storageClassName:
                 local-storage
                 -
                 resources:
                 # newly added
                 requests:
                 # newly added
                 storage:
                 100Gi
                 # newly added
                 storageClassName:
                 local-storage
                 # newly added

Manually delete the TiFlash StatefulSet by running the following command. Then, wait for the TiDB Operator to recreate the TiFlash StatefulSet.


                
                 kubectl delete sts -n${namespace}
                 ${cluster_name}-tiflash

Remove TiFlash

If your TiDB cluster no longer needs the TiDB HTAP storage engine TiFlash, take the following steps to remove TiFlash:

Adjust the number of replicas of the tables replicated to the TiFlash cluster.

To completely remove TiFlash, you need to set the number of replicas of all tables replicated to the TiFlash to0.

To connect to the TiDB service, refer to the steps inAccess the TiDB Cluster on Kubernetes.

To adjust the number of replicas of the tables replicated to the TiFlash cluster, run the following command:


                  
                   alter
                   table
                   <db_name>.<table_name>
                   settiflash replica0;

Wait for the TiFlash replicas of the related tables to be deleted.

Connect to the TiDB service and run the following command. If you can not find the replication information of the related tables, it means that the replicas are deleted:


                
                 SELECT
                 *
                 FROMinformation_schema.tiflash_replicaWHERETABLE_SCHEMA=
                 ''
                 andTABLE_NAME=
                 '';

To remove TiFlash Pods, run the following command to modifyspec.tiflash.replicasto0:


                
                 kubectl patch tidbcluster${cluster_name}-n${namespace}--typemerge -p'{"spec":{"tiflash":{"replicas": 0}}}

Check the state of TiFlash Pods and TiFlash stores.

Run the following command to check whether you delete the TiFlash Pod successfully:


                  
                   kubectl get pod -n${namespace}-l app.kubernetes.io/component=tiflash,app.kubernetes.io/instance=${cluster_name}

If the output is empty, it means that you delete the Pod of the TiFlash cluster successfully.

To check whether the stores of the TiFlash are in theTombstonestate, run the following command:


                  
                   kubectl get tidbcluster${cluster_name}-n${namespace}-o yaml

The value of thestatus.tiflashfield in the output result is similar to the example below.


                  
                   tiflash: ... tombstoneStores: "88": id: "88" ip: basic-tiflash-0.basic-tiflash-peer.default.svc lastHeartbeatTime: "2020-12-31T04:42:12Z" lastTransitionTime: null leaderCount: 0 podName: basic-tiflash-0 state: Tombstone "89": id: "89" ip: basic-tiflash-1.basic-tiflash-peer.default.svc lastHeartbeatTime: "2020-12-31T04:41:50Z" lastTransitionTime: null leaderCount: 0 podName: basic-tiflash-1 state: Tombstone

Only after you delete all Pods of the TiFlash cluster successfully and all the TiFlash stores have changed to theTombstonestate, can you perform the next operation.

删除the TiFlash StatefulSet.

To modify the TidbCluster CR and delete thespec.tiflashfield, run the following command:


                  
                   kubectl patch tidbcluster${cluster_name}-n${namespace}--typejson -p'[{"op":"remove", "path":"/spec/tiflash"}]'

To delete the TiFlash StatefulSet, run the following command:


                  
                   kubectl delete statefulsets -n${namespace}-l app.kubernetes.io/component=tiflash,app.kubernetes.io/instance=${cluster_name}

To check whether you delete the StatefulSet of the TiFlash cluster successfully, run the following command:


                  
                   kubectl get sts -n${namespace}-l app.kubernetes.io/component=tiflash,app.kubernetes.io/instance=${cluster_name}

If the output is empty, it means that you delete the StatefulSet of the TiFlash cluster successfully.

(Optional) Delete PVC and PV.

If you confirm that you do not use the data in TiFlash, and you want to delete the data, you need to strictly follow the steps below to delete the data in TiFlash:

删除the PVC object corresponding to the PV


                  
                   kubectl delete pvc -n${namespace}-l app.kubernetes.io/component=tiflash,app.kubernetes.io/instance=${cluster_name}

If the PV reclaim policy isRetain, the corresponding PV is still retained after you delete the PVC object. If you want to delete the PV, you can set the reclaim policy of the PV to删除, and the PV can be deleted and recycled automatically.


                  
                   kubectl patch pv${pv_name}-p'{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}'

In the above command,${pv_name}represents the PV name of the TiFlash cluster. You can check the PV name by running the following command:


                  
                   kubectl get pv -l app.kubernetes.io/component=tiflash,app.kubernetes.io/instance=${cluster_name}

Download PDF Request docs changes Ask questions on Discord

Playground

New

One-stop & interactive experience of TiDB's capabilities WITHOUT registration.