Troubleshoot TiCDC
This document introduces the common errors you might encounter when using TiCDC, and the corresponding maintenance and troubleshooting methods.
TiCDC replication interruptions
How do I know whether a TiCDC replication task is interrupted?
- Check the
changefeed checkpoint
monitoring metric of the replication task (choose the rightchangefeed id
) in the Grafana dashboard. If the metric value stays unchanged, or thecheckpoint lag
metric keeps increasing, the replication task might be interrupted. - Check the
exit error count
monitoring metric. If the metric value is greater than0
, an error has occurred in the replication task. - Execute
cdc cli changefeed list
andcdc cli changefeed query
to check the status of the replication task.stopped
means the task has stopped, and theerror
item provides the detailed error message. After the error occurs, you can searcherror on running processor
in the TiCDC server log to see the error stack for troubleshooting. - In some extreme cases, the TiCDC service is restarted. You can search the
FATAL
level log in the TiCDC server log for troubleshooting.
How do I know whether the replication task is stopped manually?
You can know whether the replication task is stopped manually by executingcdc cli
. For example:
cdc cli changefeed query --server=http://127.0.0.1:8300 --changefeed-id 28c43ffc-2316-4f4f-a70b-d1a7c59ba79f
In the output of the above command,admin-job-type
shows the state of this replication task:
0
: In progress, which means that the task is not stopped manually.1
: Paused. When the task is paused, all replicatedprocessor
s exit. The configuration and the replication status of the task are retained, so you can resume the task fromcheckpiont-ts
.2
: Resumed. The replication task resumes fromcheckpoint-ts
.3
: Removed. When the task is removed, all replicatedprocessor
s are ended, and the configuration information of the replication task is cleared up. The replication status is retained only for later queries.
How do I handle replication interruptions?
A replication task might be interrupted in the following known scenarios:
The downstream continues to be abnormal, and TiCDC still fails after many retries.
在这个场景中,TiCDC保存任务信息. Because TiCDC has set the service GC safepoint in PD, the data after the task checkpoint is not cleaned by TiKV GC within the valid period of
gc-ttl
.Handling method: You can resume the replication task via the HTTP interface after the downstream is back to normal.
Replication cannot continue because of incompatible SQL statement(s) in the downstream.
- 在这个场景中,TiCDC保存任务信息. Because TiCDC has set the service GC safepoint in PD, the data after the task checkpoint is not cleaned by TiKV GC within the valid period of
gc-ttl
. - Handling procedures:
- Query the status information of the replication task using the
cdc cli changefeed query
command and record the value ofcheckpoint-ts
. - Use the new task configuration file and add the
ignore-txn-start-ts
parameter to skip the transaction corresponding to the specifiedstart-ts
. - Stop the old replication task via HTTP API. Execute
cdc cli changefeed create
to create a new task and specify the new task configuration file. Specifycheckpoint-ts
recorded in step 1 as thestart-ts
and start a new task to resume the replication.
- Query the status information of the replication task using the
- 在这个场景中,TiCDC保存任务信息. Because TiCDC has set the service GC safepoint in PD, the data after the task checkpoint is not cleaned by TiKV GC within the valid period of
In TiCDC v4.0.13 and earlier versions, when TiCDC replicates the partitioned table, it might encounter an error that leads to replication interruption.
- 在这个场景中,TiCDC保存任务信息. Because TiCDC has set the service GC safepoint in PD, the data after the task checkpoint is not cleaned by TiKV GC within the valid period of
gc-ttl
. - Handling procedures:
- Pause the replication task by executing
cdc cli changefeed pause -c
. - Wait for about one munite, and then resume the replication task by executing
cdc cli changefeed resume -c
.
- Pause the replication task by executing
- 在这个场景中,TiCDC保存任务信息. Because TiCDC has set the service GC safepoint in PD, the data after the task checkpoint is not cleaned by TiKV GC within the valid period of
What should I do to handle the OOM that occurs after TiCDC is restarted after a task interruption?
- Update your TiDB cluster and TiCDC cluster to the latest versions. The OOM problem has already been resolved inv4.0.14 and later v4.0 versions, v5.0.2 and later v5.0 versions, and the latest versions.
How do I handle theError 1298: Unknown or incorrect time zone: 'UTC'
错误当创建replication task or replicating data to MySQL?
This error is returned when the downstream MySQL does not load the time zone. You can load the time zone by runningmysql_tzinfo_to_sql
. After loading the time zone, you can create tasks and replicate data normally.
mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql -p
If the output of the command above is similar to the following one, the import is successful:
Enter password: Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it. Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it. Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it. Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.
If the downstream is a special MySQL environment (a public cloud RDS or some MySQL derivative versions) and importing the time zone using the preceding method fails, you can use the default time zone of the downstream by settingtime-zone
to an empty value, such astime-zone=""
.
When using time zones in TiCDC, it is recommended to explicitly specify the time zone, such astime-zone="Asia/Shanghai"
. Also, make sure that thetz
specified in TiCDC server configurations and thetime-zone
specified in Sink URI are consistent with the time zone configuration of the downstream database. This prevents data inconsistency caused by inconsistent time zones.
我怎么处理不相容问题的进行guration files caused by TiCDC upgrade?
Refer toNotes for compatibility.
Thestart-ts
timestamp of the TiCDC task is quite different from the current time. During the execution of this task, replication is interrupted and an error[CDC:ErrBufferReachLimit]
发生。我应该做什么?
Since v4.0.9, you can try to enable the unified sorter feature in your replication task, or use the BR tool for an incremental backup and restore, and then start the TiCDC replication task from a new time.
When the downstream of a changefeed is a database similar to MySQL and TiCDC executes a time-consuming DDL statement, all other changefeeds are blocked. What should I do?
- Pause the execution of the changefeed that contains the time-consuming DDL statement. Then you can see that other changefeeds are no longer blocked.
- Search for the
apply job
field in the TiCDC log and confirm thestart-ts
of the time-consuming DDL statement. - Manually execute the DDL statement in the downstream. After the execution finishes, go on performing the following operations.
- Modify the changefeed configuration and add the above
start-ts
to theignore-txn-start-ts
configuration item. - Resume the paused changefeed.
After I upgrade the TiCDC cluster to v4.0.8, the[CDC:ErrKafkaInvalidConfig]Canal requires old value to be enabled
error is reported when I execute a changefeed. What should I do?
Since v4.0.8, if thecanal-json
,canal
ormaxwell
protocol is used for output in a changefeed, TiCDC enables the old value feature automatically. However, if you have upgraded TiCDC from an earlier version to v4.0.8 or later, when the changefeed uses thecanal-json
,canal
ormaxwell
protocol and the old value feature is disabled, this error is reported.
To fix the error, take the following steps:
Set the value of
enable-old-value
in the changefeed configuration file totrue
.Execute
cdc cli changefeed pause
to pause the replication task.cdc cli changefeed pause -c test-cf --server=http://127.0.0.1:8300Execute
cdc cli changefeed update
to update the original changefeed configuration.cdc cli changefeed update -c test-cf --server=http://127.0.0.1:8300 --sink-uri="mysql://127.0.0.1:3306/?max-txn-row=20&worker-number=8"--config=changefeed.tomlExecute
cdc cli changfeed resume
to resume the replication task.cdc cli changefeed resume -c test-cf --server=http://127.0.0.1:8300
The[tikv:9006]GC life time is shorter than transaction duration, transaction starts at xx, GC safe point is yy
error is reported when I use TiCDC to create a changefeed. What should I do?
You need to run thepd-ctl service-gc-safepoint --pd
command to query the current GC safepoint and service GC safepoint. If the GC safepoint is smaller than thestart-ts
of the TiCDC replication task (changefeed), you can directly add the--disable-gc-check
option to thecdc cli create changefeed
command to create a changefeed.
If the result ofpd-ctl service-gc-safepoint --pd
does not havegc_worker service_id
:
- If your PD version is v4.0.8 or earlier, refer toPD issue #3128for details.
- If your PD is upgraded from v4.0.8 or an earlier version to a later version, refer toPD issue #3366for details.
When I use TiCDC to replicate messages to Kafka, Kafka returns theMessage was too large
error. Why?
For TiCDC v4.0.8 or earlier versions, you cannot effectively control the size of the message output to Kafka only by configuring themax-message-bytes
setting for Kafka in the Sink URI. To control the message size, you also need to increase the limit on the bytes of messages to be received by Kafka. To add such a limit, add the following configuration to the Kafka server configuration.
# The maximum byte number of a message that the broker receives message.max.bytes=2147483648 # The maximum byte number of a message that the broker copies replica.fetch.max.bytes=2147483648 # The maximum message byte number that the consumer side reads fetch.message.max.bytes=2147483648
How can I find out whether a DDL statement fails to execute in downstream during TiCDC replication? How to resume the replication?
If a DDL statement fails to execute, the replication task (changefeed) automatically stops. The checkpoint-ts is the DDL statement's finish-ts minus one. If you want TiCDC to retry executing this statement in the downstream, usecdc cli changefeed resume
to resume the replication task. For example:
cdc cli changefeed resume -c test-cf --server=http://127.0.0.1:8300
If you want to skip this DDL statement that goes wrong, set the start-ts of the changefeed to the checkpoint-ts (the timestamp at which the DDL statement goes wrong) plus one, and then run thecdc cli changefeed create
command to create a new changefeed task. For example, if the checkpoint-ts at which the DDL statement goes wrong is415241823337054209
, run the following commands to skip this DDL statement:
cdc cli changefeed remove --server=http://127.0.0.1:8300 --changefeed-id simple-replication-task cdc cli changefeed create --server=http://127.0.0.1:8300 --sink-uri="mysql://root:123456@127.0.0.1:3306/"--changefeed-id="simple-replication-task"--sort-engine="unified"--start-ts 415241823337054210