Important

You are viewing the documentation of an older version of the TiDB database (TiDB v4.0). It is recommended that you usethe latest LTS versionof the TiDB database.

Scale the TiDB Cluster Using TiUP

The capacity of a TiDB cluster can be increased or decreased without interrupting the online services.

This document describes how to scale the TiDB, TiKV, PD, TiCDC, or TiFlash cluster using TiUP. If you have not installed TiUP, refer to the steps inInstall TiUP on the control machineand import the cluster into TiUP before you use TiUP to scale the TiDB cluster.

To view the current cluster name list, runtiup cluster list.

For example, if the original topology of the cluster is as follows:

Host IP	Service
10.0.1.3	TiDB + TiFlash
10.0.1.4	TiDB + PD
10.0.1.5	TiKV + Monitor
10.0.1.1	TiKV
10.0.1.2	TiKV

Scale out a TiDB/PD/TiKV cluster

If you want to add a TiDB node to the10.0.1.5host, take the following steps.

Note

You can take similar steps to add the PD node. Before you add the TiKV node, it is recommended that you adjust the PD scheduling parameters in advance according to the cluster load.

Configure the scale-out topology:

Note

The port and directory information is not required by default.
If multiple instances are deployed on a single machine, you need to allocate different ports and directories for them. If the ports or directories have conflicts, you will receive a notification during deployment or scaling.
Since TiUP v1.0.0, the scale-out configuration will inherit the global configuration of the original cluster.

Add the scale-out topology configuration in thescale-out.yamlfile:


                
                 vi scale-out.yaml


                
                 tidb_servers:主持人:10.0.1.5 ssh_port: 22端口:4000 status_port: 10080 deploy_dir: /data/deploy/install/deploy/tidb-4000 log_dir: /data/deploy/install/log/tidb-4000

Here is a TiKV configuration file template:


                
                 tikv_servers: - host: 10.0.1.5 ssh_port: 22 port: 20160 status_port: 20180 deploy_dir: /data/deploy/install/deploy/tikv-20160 data_dir: /data/deploy/install/data/tikv-20160 log_dir: /data/deploy/install/log/tikv-20160

Here is a PD configuration file template:


                
                 pd_servers: - host: 10.0.1.5 ssh_port: 22 name: pd-1 client_port: 2379 peer_port: 2380 deploy_dir: /data/deploy/install/deploy/pd-2379 data_dir: /data/deploy/install/data/pd-2379 log_dir: /data/deploy/install/log/pd-2379

To view the configuration of the current cluster, runtiup cluster edit-config . Because the parameter configuration ofglobalandserver_configsis inherited byscale-out.yamland thus also takes effect inscale-out.yaml.

After the configuration, the current topology of the cluster is as follows:

Host IP	Service
10.0.1.3	TiDB + TiFlash
10.0.1.4	TiDB + PD
10.0.1.5	TiDB+ TiKV + Monitor
10.0.1.1	TiKV
10.0.1.2	TiKV

Run the scale-out command:
```
                
                 tiup cluster scale-out  scale-out.yaml
```
Note

上面的命令是基于的假设the mutual trust has been configured for the user to execute the command and the new machine. If the mutual trust cannot be configured, use the-poption to enter the password of the new machine, or use the-ioption to specify the private key file.

If you see theScaled cluster out successfully, the scale-out operation is successfully completed.

Check the cluster status:


                
                 tiup cluster display

Access the monitoring platform athttp://10.0.1.5:3000using your browser to monitor the status of the cluster and the new node.

After the scale-out, the cluster topology is as follows:

Host IP	Service
10.0.1.3	TiDB + TiFlash
10.0.1.4	TiDB + PD
10.0.1.5	TiDB+ TiKV + Monitor
10.0.1.1	TiKV
10.0.1.2	TiKV

Scale out a TiFlash cluster

If you want to add a TiFlash node to the10.0.1.4host, take the following steps.

Note

When adding a TiFlash node to an existing TiDB cluster, you need to note the following things:

Confirm that the current TiDB version supports using TiFlash. Otherwise, upgrade your TiDB cluster to v4.0 or later versions.
Execute thetiup ctl: pd -u http://: config set enable-placement-rules truecommand to enable the Placement Rules feature. Or execute the corresponding command inpd-ctl.

Add the node information to thescale-out.yamlfile:

Create thescale-out.yamlfile to add the TiFlash node information.


                
                 tiflash_servers: - host: 10.0.1.4

Currently, you can only add IP but not domain name.

Run the scale-out command:


                
                 tiup cluster scale-out  scale-out.yaml

Note

上面的命令是基于的假设the mutual trust has been configured for the user to execute the command and the new machine. If the mutual trust cannot be configured, use the-poption to enter the password of the new machine, or use the-ioption to specify the private key file.

View the cluster status:


                
                 tiup cluster display

Access the monitoring platform athttp://10.0.1.5:3000using your browser, and view the status of the cluster and the new node.

After the scale-out, the cluster topology is as follows:

Host IP	Service
10.0.1.3	TiDB + TiFlash
10.0.1.4	TiDB + PD +TiFlash
10.0.1.5	TiDB+ TiKV + Monitor
10.0.1.1	TiKV
10.0.1.2	TiKV

Scale out a TiCDC cluster

If you want to add two TiCDC nodes to the10.0.1.3and10.0.1.4hosts, take the following steps.

Add the node information to thescale-out.yamlfile:

Create thescale-out.yamlfile to add the TiCDC node information.


                
                 cdc_servers: - host: 10.0.1.3 gc-ttl: 86400 data_dir: /data/deploy/install/data/cdc-8300 - host: 10.0.1.4 gc-ttl: 86400 data_dir: /data/deploy/install/data/cdc-8300

Run the scale-out command:


                
                 tiup cluster scale-out  scale-out.yaml

Note

View the cluster status:


                
                 tiup cluster display

Access the monitoring platform athttp://10.0.1.5:3000using your browser, and view the status of the cluster and the new nodes.

After the scale-out, the cluster topology is as follows:

Host IP	Service
10.0.1.3	TiDB + TiFlash +TiCDC
10.0.1.4	TiDB + PD + TiFlash +TiCDC
10.0.1.5	TiDB+ TiKV + Monitor
10.0.1.1	TiKV
10.0.1.2	TiKV

Scale in a TiDB/PD/TiKV cluster

If you want to remove a TiKV node from the10.0.1.5host, take the following steps.

Note

You can take similar steps to remove the TiDB and PD node.
Because the TiKV, TiFlash, and TiDB Binlog components are taken offline asynchronously and the stopping process takes a long time, TiUP takes them offline in different methods. For details, seeParticular handling of components' offline process.

Note

The PD Client in TiKV caches the list of PD nodes.

In versions earlier than v4.0.3, TiKV does not automatically and regularly update the cache of the PD node list. The cache is updated only after the PD leader switches or TiKV restarts to load the latest configuration. To avoid the issue of an expired list of PD nodes cached by TiKV, the PD cluster should contain at least one PD node that existed before the scaling-in or scaling-out. If this condition is not met, you need to manually perform the PD leader transfer to update the PD cache list in TiKV.
Since v4.0.3, TiKV has a mechanism to automatically and regularly update PD nodes, which can help mitigate the issue of an expired list of PD nodes cached by TiKV. However, after scaling out PD, you should try to avoid directly removing all PD nodes at once that existed before the scaling. If necessary, before making all the previously existing PD nodes offline, make sure to switch the PD leader to a newly added PD node.

View the node ID information:


                
                 tiup cluster display


                
                 Starting /root/.tiup/components/cluster/v0.4.6/cluster display
                  TiDB Cluster:
                   TiDB Version: v4.0.16 ID Role Host Ports Status Data Dir Deploy Dir -- ---- ---- ----- ------ -------- ---------- 10.0.1.3:8300 cdc 10.0.1.3 8300 Up data/cdc-8300 deploy/cdc-8300 10.0.1.4:8300 cdc 10.0.1.4 8300 Up data/cdc-8300 deploy/cdc-8300 10.0.1.4:2379 pd 10.0.1.4 2379/2380 Healthy data/pd-2379 deploy/pd-2379 10.0.1.1:20160 tikv 10.0.1.1 20160/20180 Up data/tikv-20160 deploy/tikv-20160 10.0.1.2:20160 tikv 10.0.1.2 20160/20180 Up data/tikv-20160 deploy/tikv-20160 10.0.1.5:20160 tikv 10.0.1.5 20160/20180 Up data/tikv-20160 deploy/tikv-20160 10.0.1.3:4000 tidb 10.0.1.3 4000/10080 Up - deploy/tidb-4000 10.0.1.4:4000 tidb 10.0.1.4 4000/10080 Up - deploy/tidb-4000 10.0.1.5:4000 tidb 10.0.1.5 4000/10080 Up - deploy/tidb-4000 10.0.1.3:9000 tiflash 10.0.1.3 9000/8123/3930/20170/20292/8234 Up data/tiflash-9000 deploy/tiflash-9000 10.0.1.4:9000 tiflash 10.0.1.4 9000/8123/3930/20170/20292/8234 Up data/tiflash-9000 deploy/tiflash-9000 10.0.1.5:9090 prometheus 10.0.1.5 9090 Up data/prometheus-9090 deploy/prometheus-9090 10.0.1.5:3000 grafana 10.0.1.5 3000 Up - deploy/grafana-3000 10.0.1.5:9093 alertmanager 10.0.1.5 9093/9294 Up data/alertmanager-9093 deploy/alertmanager-9093

Run the scale-in command:


                
                 tiup cluster scale-in  --node 10.0.1.5:20160

The--nodeparameter is the ID of the node to be taken offline.

If you see theScaled cluster in successfully, the scale-in operation is successfully completed.

Check the cluster status:
The scale-in process takes some time. If the status of the node to be scaled in becomesTombstone, that means the scale-in operation is successful.
To check the scale-in status, run the following command:
```
                
                 tiup cluster display 
```
Access the monitoring platform athttp://10.0.1.5:3000using your browser, and view the status of the cluster.

The current topology is as follows:

Host IP	Service
10.0.1.3	TiDB + TiFlash + TiCDC
10.0.1.4	TiDB + PD + TiFlash + TiCDC
10.0.1.5	TiDB + Monitor(TiKV is deleted)
10.0.1.1	TiKV
10.0.1.2	TiKV

Scale in a TiFlash cluster

If you want to remove a TiFlash node from the10.0.1.4host, take the following steps.

1. Adjust the number of replicas of the tables according to the number of remaining TiFlash nodes

Before the node goes down, make sure that the number of remaining nodes in the TiFlash cluster is no smaller than the maximum number of replicas of all tables. Otherwise, modify the number of TiFlash replicas of the related tables.

For all tables whose replicas are greater than the number of remaining TiFlash nodes in the cluster, execute the following command in the TiDB client:


                
                 alter
                 table
                 <db-name>.<
                 table
                 -name>
                 settiflash replica0;

Wait for the TiFlash replicas of the related tables to be deleted.Check the table replication progressand the replicas are deleted if the replication information of the related tables is not found.

2. Perform the scale-in operation

Next, perform the scale-in operation with one of the following solutions.

Solution 1: Use TiUP to remove a TiFlash node

First, confirm the name of the node to be taken down:


                
                 tiup cluster display

Remove the TiFlash node (assume that the node name is10.0.1.4:9000from Step 1):


                
                 tiup cluster scale-in  --node 10.0.1.4:9000

Solution 2: Manually remove a TiFlash node

In special cases (such as when a node needs to be forcibly taken down), or if the TiUP scale-in operation fails, you can manually remove a TiFlash node with the following steps.

Use the store command of pd-ctl to view the store ID corresponding to this TiFlash node.
- Enter the store command inpd-ctl(the binary file is underresources/binin the tidb-ansible directory).
- If you use TiUP deployment, replacepd-ctlwithtiup ctl: pd:
```
                  
                   tiup ctl: pd -u http://: store
```
  Note
  
  If multiple PD instances exist in the cluster, you only need to specify the IP address:port of an active PD instance in the above command.

Remove the TiFlash node in pd-ctl:

Enterstore delete in pd-ctl (is the store ID of the TiFlash node found in the previous step.

If you use TiUP deployment, replacepd-ctlwithtiup ctl: pd:


                  
                   tiup ctl: pd -u http://: store delete

Note

If multiple PD instances exist in the cluster, you only need to specify the IP address:port of an active PD instance in the above command.

Wait for the store of the TiFlash node to disappear or for thestate_nameto becomeTombstonebefore you stop the TiFlash process.
If, after waiting for a long time, the node still fails to disappear or thestate_namefails to becomeTombstone, consider using the following command to force the node out of the cluster (use with caution).
Warning
- The following command directly discards the replicas on the TiFlash node, which might cause the query to fail.
- Once you have used this command, do not use PD Control to delete the node from the metadata of PD. Otherwise, the information of Regions in the cluster will be inconsistent.
- The following command is only applicable to TiFlash nodes. Do not use this command to force any TiKV node out of the cluster. Otherwise, data loss occurs.
- After using this command, because the information of the replicas on the forced-out node still exists in the member information of other replicas, PD triggers scheduling to repair and clear the replica information of the forced-out node. Therefore, before using this command, you need to consider whether there are enough replicas to elect a leader after losing the nodes. Otherwise, PD cannot trigger scheduling to repair and clear the information, which causes the service to be unavailable.
```
                
                 curl -X POST'http:///pd/api/v1/store//state?state=Tombstone'
                
```
Manually delete TiFlash data files (whose location can be found in thedata_dirdirectory under the TiFlash configuration of the cluster topology file).

Manually update TiUP's cluster configuration file (delete the information of the TiFlash node that goes down in edit mode).


                
                 tiup cluster edit-config

Note

Before all TiFlash nodes in the cluster stop running, if not all tables replicated to TiFlash are canceled, you need to manually clean up the replication rules in PD, or the TiFlash node cannot be taken down successfully.

The steps to manually clean up the replication rules in PD are below:

View all data replication rules related to TiFlash in the current PD instance:


                
                 curl http://:/pd/api/v1/config/rules/group/tiflash


                
                 [{“group_id”:“tiflash”、“id”:“table-45-r”、“机汇erride": true, "start_key": "7480000000000000FF2D5F720000000000FA", "end_key": "7480000000000000FF2E00000000000000F8", "role": "learner", "count": 1, "label_constraints": [ { "key": "engine", "op": "in", "values": [ "tiflash" ] } ] } ]

Remove all data replication rules related to TiFlash. Take the rule whoseidistable-45-ras an example. Delete it by the following command:


                
                 curl -v -X DELETE http://:/pd/api/v1/config/rule/tiflash/table-45-r

Scale in a TiCDC cluster

If you want to remove the TiCDC node from the10.0.1.4host, take the following steps:

Take the node offline:


                
                 tiup cluster scale-in  --node 10.0.1.4:8300

View the cluster status:


                
                 tiup cluster display

Access the monitoring platform athttp://10.0.1.5:3000using your browser, and view the status of the cluster.

The current topology is as follows:

Host IP	Service
10.0.1.3	TiDB + TiFlash + TiCDC
10.0.1.4	TiDB + PD +(TiCDC is deleted）
10.0.1.5	TiDB + Monitor
10.0.1.1	TiKV
10.0.1.2	TiKV