Published on Nov 09, 2022
Open-source change data capture platform Debezium converts database records into event streams, enabling applications to detect and respond to database row-level changes. This release of version 2.0 introduces lots of changes: Java 11 is now required; incremental snapshots are better with pause and resume logic; transaction metadata have been enhanced with a new field, ts_ms, which contains the transaction timestamp; multi-tenancy databases are supported out of the box; In case the primary key is not defined, Debezium will refer to columns like CTID in PostgreSQL or ROWID in Oracle that are generated automatically by the database, as well as introducing a new debezium storage for file- and Kafka-based database history and offset storage.
The development of Debezium 2.0 has been underway for the last three years since the previous version 1.0 was released in 2019. One of the main improvements in Debezium, first introduced in version 1.6, is the support for incremental snapshots. As a rule, Debezium captures existing data during its snapshot phase which is executed once upon the start-up of the first connector. However, problems arise when it may be necessary to adjust the configuration and add tables that were not initially included in CDC. In incremental snapshots, the signaling mechanism can be used to send a snapshot signal and trigger a snapshot of only a specified set of tables. With version 2.0, Debezium added the ability to stop an ongoing snapshot, pause and resume it, as well as filter it with a SQL-based predicate to determine what records should be included in the incremental snapshot.
Debezium is built on top of Apache Kafka and provides a set of Kafka Connect compatible connectors for connecting to various databases. As changes are stored in a Kafka topic, when the application that reads from Debezium is re-opened, it can resume reading from where it left off if there are any issues or crashes.
As a log-based CDC, Debezium ensures that all changes to data are captured, provides very low delay in change events, does not require changes to the data model, and is capable of capturing ‘delete’ changes. CDC also offers additional features such as snapshots, which allow an initial snapshot of a database’s current state to be taken when a connector is started but not all logs are available; filters, schema, tables and columns can be included or excluded from CDC; masking, which allows sensitive data to be mask; message transformations, including topic routing, content-based routing, and message filtering, which are ready to use.
For more information about future releases, please refer to the Debezium 2.0 release notes and this roadmap.
