Manually Scale TiDB on Kubernetes

This document introduces how to horizontally and vertically scale a TiDB cluster on Kubernetes.

Horizontal scaling

Horizontally scaling TiDB means that you scale TiDB out or in by adding or remove Pods in your pool of resources. When you scale a TiDB cluster, PD, TiKV, and TiDB are scaled out or in sequentially according to the values of their replicas.

To scale out a TiDB cluster,increasethe value ofreplicasof a certain component. The scaling out operations add Pods based on the Pod ID in ascending order, until the number of Pods equals the value ofreplicas.
To scale in a TiDB cluster,decreasethe value ofreplicasof a certain component. The scaling in operations remove Pods based on the Pod ID in descending order, until the number of Pods equals the value ofreplicas.

Horizontally scale PD, TiKV, and TiDB

To scale PD, TiKV, or TiDB horizontally, use kubectl to modifyspec.pd.replicas,spec.tikv.replicas, andspec.tidb.replicasin theTidbClusterobject of the cluster to a desired value.

Modify thereplicasvalue of a component as needed. For example, configure thereplicasvalue of PD to 3:


                
                 kubectl patch -n${namespace}tc${cluster_name}--typemerge --patch'{"spec":{"pd":{"replicas":3}}}'

检查您的配置是否已更新in the corresponding TiDB cluster on Kubernetes.


                
                 kubectl get tidbcluster${cluster_name}-n${namespace}-oyaml

If your configuration is successfully updated, in theTidbClusterCR output by the command above, the values ofspec.pd.replicas,spec.tidb.replicas, andspec.tikv.replicasare consistent with the values you have configured.

Check whether the number ofTidbClusterPods has increased or decreased.


                
                 watch kubectl -n${namespace}get pod -o wide

For the PD and TiDB components, it might take 10-30 seconds to scale in or out.

For the TiKV component, it might take 3-5 minutes to scale in or out because the process involves data migration.

Horizontally scale TiFlash

This section describes how to horizontally scale out or scale in TiFlash if you have deployed TiFlash in the cluster.

Horizontally scale out TiFlash

To scale out TiFlash horizontally, you can modifyspec.tiflash.replicas.

For example, configure thereplicasvalue of TiFlash to 3:


              
               kubectl patch -n${namespace}tc${cluster_name}--typemerge --patch'{"spec":{"tiflash":{"replicas":3}}}'

Horizontally scale in TiFlash

To scale in TiFlash horizontally, perform the following steps:

Expose the PD service by usingport-forward:


                
                 kubectl port-forward -n${namespace}svc/${cluster_name}-pd 2379:2379

Open anewterminal tab or window. Check the maximum number (N) of replicas of all data tables with which TiFlash is enabled by running the following command:


                
                 curl 127.0.0.1:2379/pd/api/v1/config/rules/group/tiflash | grep count

In the printed result, the largest value ofcountis the maximum number (N) of replicas of all data tables.

Go back to the terminal window in Step 1, whereport-forwardis running. PressCtrl+Cto stopport-forward.

After the scale-in operation, if the number of remaining Pods in TiFlash >=N, skip to Step 6. Otherwise, take the following steps:

Refer toAccess TiDBand connect to the TiDB service.

For all the tables that have more replicas than the remaining Pods in TiFlash, run the following command:


                  
                   alter
                   table
                   <db_name>.<table_name>
                   settiflash replica ${pod_number};

${pod_number}indicates the number of remaining Pods in the TiFlash cluster after scaling in.

Wait for the number of TiFlash replicas in the related tables to be updated.

Connect to the TiDB service, and run the following command to check the number:


                
                 SELECT
                 *
                 FROMinformation_schema.tiflash_replicaWHERETABLE_SCHEMA=
                 ''
                 andTABLE_NAME=
                 '';

If you cannot view the replication information of related tables, the TiFlash replicas are successfully deleted.

Modifyspec.tiflash.replicasto scale in TiFlash.

Check whether TiFlash in the TiDB cluster on Kubernetes has updated to your desired definition. Run the following command and see whether the value ofspec.tiflash.replicasreturned is expected:


                
                 kubectl get tidbcluster${cluster-name}-n${namespace}-oyaml

Horizontally scale TiCDC

If TiCDC is deployed in the cluster, you can horizontally scale out or scale in TiCDC by modifying the value ofspec.ticdc.replicas.

For example, configure thereplicasvalue of TiCDC to 3:


              
               kubectl patch -n${namespace}tc${cluster_name}--typemerge --patch'{"spec":{"ticdc":{"replicas":3}}}'

View the horizontal scaling status

To view the scaling status of the cluster, run the following command:


              
               watch kubectl -n${namespace}get pod -o wide

When the number of Pods for all components reaches the preset value and all components go to theRunningstate, the horizontal scaling is completed.

Note

The PD, TiKV and TiFlash components do not trigger the rolling update operations during scaling in and out.
When the TiKV component scales in, TiDB Operator calls the PD interface to mark the corresponding TiKV instance as offline, and then migrates the data on it to other TiKV nodes. During the data migration, the TiKV Pod is still in theRunningstate, and the corresponding Pod is deleted only after the data migration is completed. The time consumed by scaling in depends on the amount of data on the TiKV instance to be scaled in. You can check whether TiKV is in theOfflinestate by runningkubectl get -n ${namespace} tidbcluster ${cluster_name} -o json | jq '.status.tikv.stores'.
When the number ofUPstores is equal to or less than the parameter value ofMaxReplicasin the PD configuration, the TiKV components can not be scaled in.
The TiKV component does not support scale out while a scale-in operation is in progress. Forcing a scale-out operation might cause anomalies in the cluster. If an anomaly already happens, refer toTiKV Store is in Tombstone status abnormallyto fix it.
The TiFlash component has the same scale-in logic as TiKV.
When the PD, TiKV, and TiFlash components scale in, the PVC of the deleted node is retained during the scaling in process. Because the PV's reclaim policy is changed toRetain, the data can still be retrieved even if the PVC is deleted.

Vertical scaling

Vertically scaling TiDB means that you scale TiDB up or down by increasing or decreasing the limit of resources on the Pod. Vertically scaling is essentially the rolling update of the Pods.

Vertically scale components

This section describes how to vertically scale up or scale down components including PD, TiKV, TiDB, TiFlash, and TiCDC.

To scale up or scale down PD, TiKV, TiDB, use kubectl to modifyspec.pd.resources,spec.tikv.resources, andspec.tidb.resourcesin theTidbClusterobject that corresponds to the cluster to a desired value.
To scale up or scale down TiFlash, modify the value ofspec.tiflash.resources.
To scale up or scale down TiCDC, modify the value ofspec.ticdc.resources.

View the vertical scaling progress

To view the upgrade progress of the cluster, run the following command:


              
               watch kubectl -n${namespace}get pod -o wide

When all Pods are rebuilt and in theRunningstate, the vertical scaling is completed.

Note

If the resource'srequestsfield is modified during the vertical scaling process, and if PD, TiKV, and TiFlash useLocal PV, they will be scheduled back to the original node after the upgrade. At this time, if the original node does not have enough resources, the Pod ends up staying in thePendingstatus and thus impacts the service.
TiDB is a horizontally scalable database, so it is recommended to take advantage of it simply by adding more nodes rather than upgrading hardware resources like you do with a traditional database.

Scaling troubleshooting

在水平或垂直扩展业务n, Pods might go to the Pending state because of insufficient resources. SeeTroubleshoot the Pod in Pending stateto resolve it.

Download PDF Request docs changes Ask questions on Discord

Playground

New

One-stop & interactive experience of TiDB's capabilities WITHOUT registration.