What’s New in TiDB 4.0

TiDB v4.0 was officially released on May 28, 2020. In this release, we have made great improvements in stability, usability, performance, security, and features. This document briefly introduces the most notable improvements for you. You can decide whether to upgrade to v4.0 based on your needs. For the complete list of new features and bug fixes, you can check ourearlier v4.0 release notes.

Scheduling

• Hotspot scheduling policy supports more dimensions. In addition to using write or read traffic as the scheduling basis, keys are introduced as a new dimension for the scheduling policy, which might, to a large extent, mitigate the CPU usage imbalance caused by the previous single-dimensional policy. SeeTiDB Schedulingfor details.

Storage engine

• TiFlash is a key component that makes TiDB essentially a Hybrid Transactional/Analytical Processing (HTAP) database. Complying with the Multi-Raft Learner protocol, TiFlash replicates data from TiKV in real time, which ensures that data is strongly consistent between TiKV, a row-based storage engine, and TiFlash, a columnar storage engine. You can deploy TiKV and TiFlash on different machines to isolate HTAP resources. SeeTiFlashfor details.
• In v4.0, TiKV provides new storage formats to improve the efficiency of encoding and decoding in the wide-table scenario.

TiDB Dashboard

UsingTiDB Dashboard, DBAs can quickly find out the cluster topology, cluster configuration, log information, hardware information, operating system, slow queries, SQL query information, diagnostics, and so on. These information helps them quickly learn and analyze various system metrics using SQL statements:

• Cluster Info, which is the running status of all components in the cluster (including TiDB, TiKV, and PD) and the running status of the machine on which these components are hosted.
• Key Visualizer, which visually displays the traffic of TiDB over a certain period of time and can be used by DBAs to analyze the usage mode of TiDB and the traffic hotspots.
• SQL Statements, which records all the SQL statements that have been executed and the related statistics, including the execution times and the total time of execution. This helps you quickly analyze the SQL execution in TiDB and find out the hot SQL statements.
• Slow Queries, which summarizes all slow queries in the cluster to help you locate some slow queries.
• Cluster Diagnostics. TiDB automatically and regularly diagnoses the potential problems of the cluster, and summarizes the diagnostic results and some cluster-related load monitoring information into a diagnostic report. The diagnostic report is displayed on a web page. You can browse and circulate the report offline after saving it using a browser.
• Search Logs, which enables DBAs to visually search and query the log information, helps them analyze system issues, and improves their maintenance efficiency.

Deployment and maintenance tools

TiUP is a new package manager tool introduced in v4.0 that is used to manage all packages in the TiDB ecosystem. The tool provides package management, Playground, Cluster, TUF, and offline deployment, which makes the installation, deployment, and maintenance of TiDB manageable within a single tool. Therefore, DBAs can enjoy higher efficiency in deploying and maintaining TiDB. SeeTiUPfor details. The features of TiUP are summarized as follows:

• Component management, which enables you to query the component information, and to easily install, upgrade, and uninstall the cluster. For DBAs, managing TiDB components can be much easier.
• Cluster management using the TiUP Cluster component, which enables you to deploy and maintain TiDB with a single command. The management includes installation, deployment, scaling, upgrade, configuration changes, starting and stopping a cluster, restarting a cluster, and querying the cluster information. The TiUP Cluster component supports managing multiple TiDB clusters.
• Local deployment using TiUP Playground, which enables you to quickly deploy a local TiDB cluster and learn the basic features of TiDB. Note that this feature is for quick start only and cannot be used for the production environment.
• Private mirror management using TiUP Mirror, which provides you a solution of building a private mirror and offline deployment when the official TiUP mirror is inaccessible via the public network.
• Benchmark test using TiUP Benchmark, which enables you to easily deploy the benchmark performance test tool, and provides workload for TPC-C and TPC-H tests.

Transaction

• The pessimistic transaction is now provided for general availability as the default transaction mode. Support the Read Committed isolation level and theSELECT FOR UPDATE NOWAITsyntax. SeePessimistic Transaction Modelfor details.
• 支持large transactions. Increase the upper limit on transaction size from 10 MB to 10 GB. Support both the pessimistic transaction and optimistic transaction. SeeTransaction size limitfor details.

SQL features

• Introduce the automatic capture and evolution of SQL Bind to SQL Plan Management, which improves the usability and stability of the execution plan. SeeSQL Plan Managementfor details.

• Add 15 new SQL Hints to control the behavior of the optimizer that generates the execution plan and the behavior of the execution engine that executes queries. SeeSQL Hintfor details.

• 支持theSELECT INTO OUTFILEstatement which is used to export table data into the specified text file. Using this feature with Load Data, you can easily import and export data between databases.

• 支持customizing the sequence object and provide theCACHE/NO_CACHEandCYCLE/NO_CYCLEoptions to define different sequence attributes. You can use the sequence to replace the third-party ID generation. SeeSequencefor details.

• Add theFLASHBACKstatement to support recovering the truncated tables. SeeFlashback Tablefor details.

• 支持writing the intermediate results of Join and Sort to the local disk when you make queries, which avoids the Out of Memory (OOM) issue because the queries occupy excessive memory. This also improves system stability.

• Optimize the output ofEXPLAINandEXPLAIN ANALYZE. More information is shown in the result, which improves troubleshooting efficiency. SeeExplain AnalyzeandExplainfor details.

• 支持using the Index Merge feature to access tables. When you make a query on a single table, the TiDB optimizer automatically reads multiple index data according to the query condition and makes a union of the result, which improves the performance of querying on a single table. SeeIndex Mergefor details.

• 支持AUTO_RANDOMkeys as an extended syntax for the TiDB columnar attribute.AUTO_RANDOMis designed to address the hotspot issue caused by the auto-increment column and provides a low-cost migration solution from MySQL for users who work with auto-increment columns. SeeAUTO_RANDOMKeyfor details.

• Add system tables that provide information of cluster topology, configuration, logs, hardware, operating systems, and slow queries, which helps DBAs to quickly learn, analyze system metrics. SeeInformation SchemaandSQL Diagnosisfor details.

  • Add system tables that provide information of cluster topology, configuration, logs, hardware, operating systems to help DBAs quickly learn the cluster configuration and status:
    • Thecluster_infotable that stores the cluster topology information.
    • Thecluster_logtable that stores log information.
    • Thecluster_hardwareandcluster_systeminfotables that save the server information of hardware and operating system.
  • Add information tables that provide information of slow queries, diagnostic results, and performance monitoring to help DBAs quickly analyze the system bottleneck:
    • Thecluster_slow_querytable that records the global slow query information.
    • Thecluster_processlisttable that records the global processlist.
    • Theinspection_resulttable. The automatic performance diagnostics feature is introduced in v4.0 to automatically analyze the system bottleneck and output the performance analysis report for DBAs. Using this feature, DBAs can easily troubleshoot common issues and anomalies to improve maintenance efficiency.
    • The米etrics_summaryand米etric_summary_by_labeltables that records all monitoring metrics. Using this table, DBAs can use SQL statements to access all monitoring metrics and compare these metrics with historical metrics to locate and analyze anomalies.
    • Theinspection_summary表记录的关键监测指标different data links or access links. This table makes it easier for DBAs to locate, access, and analyze common anomalies on data links, such as read links or write links.

Character set and collation

支持case-insensitive and accent-insensitiveutf8mb4_general_ciandutf8_general_cicollations. SeeCharacter Set and Collationfor details.

Security

• Improve the encrypted communication between the client and server, and between components, which ensures data security and prevents any sent and received data from being read and modified by illegal hackers. Mainly support the certificate-based login authentication, updating certificate online, and verifying the CommonName attribute of the TLS certificate. SeeEnable TLS Between TiDB Clients and Serversfor details.

• Transparent Data Encryption (TDE) is a new feature that provides protection for the entire database. This feature, when enabled, is transparent to applications that are connected to TiDB and does not require any change to the existing applications. Because this TDE feature operates at the file level, TiDB encrypts data before writing data to disk, and decrypts data before reading data from memory to ensure data security. Currently, the AES128-CTR, AES192-CTR, and AES256-CTR encryption algorithms are supported. You can manage keys via AWS KMS. SeeEncryption at Restfor details.

Backup and Restore

支持the Backup & Restore (BR) feature to quickly back up and restore data of a single TiDB cluster to ensure data reliability and to meet enterprises’ needs of backup and restore or the graded protection of information security. Support quickly backing up and restoring all data or a certain range of sorted data. SeeBackup & Restorefor details.

Service level features

• 支持caching the execution plan ofPrepareorExecute提高SQL执行效率。
• 支持self-adapting to the thread pool and streamlining the number of thread pools. Optimize the processing and scheduling of requests to improve product usability and performance.
• 是指使用任何foll追随者阅读功能ower replica of a Region to serve a read request under the premise of strongly consistent reads. This feature improves the throughput of the TiDB cluster and reduces the load of the leader. It contains a series of load balancing mechanisms that offload TiKV read loads from the leader replica to the follower replica in a Region. TiKV's Follower Read implementation guarantees the consistency of data reading; combined with Snapshot Isolation in TiDB, this implementation provides users with strongly consistent reads. SeeFollower Readfor details.

TiCDC

TiCDC is a tool for replicating the incremental data of TiDB. This tool is implemented by pulling TiKV change logs, which ensures high reliability and availability of data. You can subscribe to the change information of data, and the system automatically sends data to the downstream. Currently, the downstream database must be MySQL compatible (such as MySQL and TiDB) or Kafka and Pulsar. You can also extend the supported downstream systems based on theTiCDC Open Protocol. SeeTiCDCfor details.