DBLog: A Watermark Based Change-Data-Capture Framework

Andreas Andreakis Ioannis Papapanagiotou

Netflix Netflix
aandreakis@netflix.com ipapapanagiotou@netflix.com

ABSTRACT sync [8, 12, 16] and addresses challenges that exist with traditional
It is a commonly observed pattern for applications to utilize multiple techniques like dual-writes and distributed transactions [13].
heterogeneous databases where each is used to serve a specific need In database systems, the transaction log typically has limited
such as storing the canonical form of data or providing advanced retention and it is not guaranteed to contain the full history of
search capabilities. For applications it is hence desired to keep multi- changes. Therefore, the full state of a database needs to be captured
arXiv:2010.12597v1 [cs.DB] 23 Oct 2020

ple databases in sync. We have observed a series of distinct patterns as well. While operating data synchronization in production at
that have tried to solve this problem such as dual-writes and dis- Netflix, we identified some requirements in regards to the full state
tributed transactions. However, these approaches have limitations capture. We wanted to (a) trigger the full state capture at any point
with regard to feasibility, robustness, and maintenance. An alterna- in time. That is because the full state may not only be needed
tive approach that has recently emerged is to utilize Change-Data- initially and may be needed at any time afterwards. For instance if
Capture (CDC) in order to capture changed rows from a database’s the database is restored from a backup or for repairs if there is data
transaction log and eventually deliver them downstream with low loss or corruption downstream. There are also cases where only
latency. In order to solve the data synchronization problem one a subset of data needs to be repaired, for example if a specific set
also needs to replicate the full state of a database and transaction of rows has been identified to be corrupt downstream. (b) pause
logs typically do not contain the full history of changes. At the or resume at any time so that full state capture does not need to
same time, there are use cases that require high availability of the start from the beginning for large tables after restarting the process.
transaction log events so that databases stay as closely in-sync as (c) capture transaction log events and the full state side by side
possible. without stalling one or the other. There are use cases that require
To address the above challenges, we developed a novel CDC high availability of transaction log events so that the replication
framework for databases, namely DBLog. DBLog utilizes a water- lag to the source is kept to a minimum. (d) prevent time-travel,
mark based approach that allows us to interleave transaction log by preserving the order of history when transmitting events to a
events with rows that we directly select from tables to capture derived datastore. This way an earlier version of a row (like the
the full state. Our solution allows log events to continue progress residential address of a member account) is not delivered after a later
without stalling while processing selects. Selects can be triggered at version. Hence, a solution had to combine transaction log events
any time on all tables, a specific table, or for specific primary keys and the full state in a way that preserves the history of changes.
of a table. DBLog executes selects in chunks and tracks progress, (e) offer this as a platform. Hence it was crucial to minimize the
allowing them to pause and resume. The watermark approach does impact on the source database. Otherwise this can hinter adoption
not use locks and has minimum impact on the source. DBLog is of the platform, especially for use cases that have high traffic. In
currently used in production by tens of microservices at Netflix. that regard we want to avoid primitives such as table locks which
can block application write traffic. (f) function across a variety
KEYWORDS of Relational Database Management Systems (RDMBS), such as
MySQL, PostgreSQL, Aurora [19] etc, that we use in production.
databases, replication, change-data-capture In order to achieve that we wanted to avoid using vendor specific
features.
1 INTRODUCTION Based on these requirements we developed DBLog. DBLog runs
Netfix uses hundreds of microservices performing trillions of oper- as a process and utilizes a watermark based approach that allows
ations per day in the data layer. Since there is no single database interleaving of transaction log events with rows that we directly
design that fits all the needs, each of the microservices can utilize select from tables in order to capture the full state of a database. Our
multiple heterogeneous databases. For example, a service can use solution allows log events to continue progress without stalling
MySQL, PostgreSQL, Aurora or Cassandra for the operational data while executing selects. Selects can be triggered at any time on
and Elasticsearch for its indexing capabilities. To be able to keep all tables, a specific table, or for specific primary keys of a table.
multiple databases in sync we developed a data enrichment and DBLog processes selects in chunks and tracks progress in a state
synchronization platform namely Delta [7]. One of the key require- store (currently Zookeeper) allowing them to pause and resume
ments is to have low propagation delays from the source to the from the last completed chunk. The watermark approach does not
derived stores and that the flow of events is highly available. A use table locks and has therefore minimum impact on the source
key requirement to achieve that is having Change-Data-Capture database. DBLog delivers captured events into an output by using
(CDC) that allows capturing changed rows from a database in near the same format regardless if the origin is the transaction log or a
real-time and eventually propagating those rows to downstream table selection. The output can be a stream like Kafka [21] which
consumers [11]. CDC is becoming increasingly popular for use is a common choice if there is more than one consumer of the
cases that require keeping multiple heterogeneous databases in events. However DBLog can also write directly to datastores or
 Andreakis and Papapanagiotou

database, the duration of this locking can either be brief or can last
throughout the whole selection process, such as with MySQL RDS
[10]. In the latter case, write traffic is blocked until all rows have
been selected which can be an extended period of time for large
databases.
In Maxwell [22] a dump is executed by pausing the transaction
log processing and then rows are selected from the desired tables.
After that, log event processing resumes. This approach is prone to
time-travel where a select can return a more recent value of a row
and an older value is captured from the log afterwards. Eventually
the latest state will be consumed from the log.
MySQLStreamer [15] creates a copy of each table at the source,
namely a copy table. Then, rows from the original table are in-
serted into the copy table in chunks resulting into transaction log
entries for the inserts. The copy tables are created using the MySQL
Figure 1: DBLog High Level Architecture. blackhole engine so that inserts don’t occupy table space, while
still generating transaction log events. Locking is used to ensure
that the order of history is not violated. The MySQLStreamer ser-
vice then consumes events from the transaction log and is able to
APIs. DBLog It is also designed with High Availability (HA) in mind detect events that originate from the copy tables, labeling them
by using an active-passive architecture, where one DBLog process is as events of the original tables. This way downstream consumers
active at a time and multiple passive processes are stand-by and can receive events per table that either originate from actual application
take over if needed to resume work. Hence, downstream consumers changes or from the copy tables.
have confidence to receive rows shortly after they changed at the Table 1 captures the requirements that we enumerated in section
source. Figure 1 depicts the high level architecture of DBLog. 1 for capturing the full state and compares them among existing
offerings. We found that no existing approach fulfills the whole
2 RELATED WORK range of requirements. Some of the limitations are implied by de-
We evaluated a series of existing offerings such as: Databus [8], De- sign such as attempting to select a consistent snapshot first and
bezium [10], Maxwell [22], MySQLStreamer [15], SpinalTap [6], and capturing log events afterwards. The choice of vendor specific fea-
Wormhole [16]. Existing solutions are similar in regard to captur- tures (like the MySQL blackhole engine) is another observed issue,
ing events from a transaction log and utilize the same underlying prohibiting code reuse across databases. Some solutions also utilize
protocols and APIs like MySQL’s binlog replication protocol or table locks which can block application write traffic for a short or
PostgreSQL’s replication slots. Captured events are serialized into an extended period of time. Given those observations, we decided
a proprietary event format and sent to an output which is typically to implement a new approach for handling dumps, one that fulfills
Kafka. Some solutions like SpinalTap and Wormhole only offer log all our requirements.
processing without a built-in ability to capture the full state of a
database, in which case full state capture needs to be handled out
of band. There are existing solutions that have a built-in capability 3 DBLOG
to capture the full state. As a transaction log typically has limited DBLog is a Java-based framework, able to capture changed rows
retention it can not be used to reconstruct the full source dataset. from a database’s transaction log and to capture the full state of
Existing offerings tackle this problem in distinct ways with varying a database by executing selects on tables. Selects are executed in
trade-offs: chunks and are interleaved with log events so that log event process-
Databus [8] has a bootstrap service that reads transaction log ing does not stall for an extended period of time. This is achieved
events from the source and stores them in a separate database. by utilizing a watermark based approach. Selects can be executed
Downstream consumers can access the bootstrap service, if they at runtime via an API. This allows bootstrapping DBLog’s output
need to be initialized or for repairs. After bootstrap, consumers with the full state initially or at a later time for repairs. If the out-
start processing log events that originate from the time before the put is Kafka with log compaction enabled, then the downstream
bootstrap so that there is overlap and no events are missed. The consumers can be bootstrapped by reading events from Kafka that
catch up from the log can lead to time-travel where row state from would contain the full dataset and be continuously updated by ap-
the bootstrap may have a more recent row state and an older state pending changed rows as they are captured from the source. For
is captured from the log afterwards. Eventually the latest state will use cases where there is only one consumer, DBLog can also emit
be discovered from the transaction log. events directly to a datastore or API.
Debezium [10] captures a consistent snapshot for MySQL and We designed the framework such that the impact to the database
PostgreSQL by using table locks and running selects across all is minimal. Selects can be paused and resumed if needed. This is
tables within one transaction. Events from the transaction log are relevant both for failure recovery and to stop processing if the
then captured from the time after the transaction once all existing database reached a bottleneck. We also avoided locks on tables so
rows have been selected. Depending on the implementation and application writes are not blocked. We use Zookeeper [1] to store
DBLog: A Watermark Based Change-Data-Capture Framework

Databus[8] Debezium[10] Maxwell[22] MySQLStreamer[15] DBLog

(a) Can be triggered at any time Yes No Yes Unknown Yes
(b) Can be paused and resumed Yes No No Unknown Yes
(c) Log event processing does not stall No No No Yes Yes
(d) Preserves the order of history No Yes No Yes Yes
(e) Does not use locks Yes No Yes No Yes
(f) No vendor specific features No No Yes No Yes
Table 1: Feature Table: Full state capture requirements

progress related to log event processing and chunk selection. We

also use Zookeeper for leader election in order to determine the
active process while other processes remain idle as passive standbys.
We chose Zookeeper because of its maturity, its low latency for
reads and writes, its support for linearizable reads [20] whenever
needed 1 , and its availability for writes if a quorum of nodes is
reachable. We have built DBLog with pluggability in mind allowing
implementations to be swapped as desired, allowing to replace
Zookeeper with another datastore.
The following subsections explain transaction log capture and
full state capture in more detail.

3.1 Transaction log capture Figure 2: Chunking a table with 3 columns c1-c3 and c1
The framework requires a database to emit an event for each as the primary key (pk). Pk column is of type integer and
changed row in commit order. In MySQL and PostgreSQL a replica- chunk size is 3. Chunk 2 is selected with the condition c1 >
tion protocol exists where the database delivers events shortly after 4.
commit time to DBLog via a TCP socket. An event can either be of
type: create, update, or delete. For our use cases we assume an event
to contain all column values from the time when the operation of each table at the source database and to populate it in chunks, so
occurred. Although, DBLog can also be used if a subset of columns that the copied rows will appear in the transaction log in the right
is captured. For each event we assume a Log-Sequence-Number order. One can then consume transaction log events and receive the
(LSN) which is the offset of the event on the transaction log and is latest state of all rows alongside changed rows [15]. This solution
encoded as an 8-byte monotonically increasing number. however consumes write I/O at the source and requires additional
Each event is serialized into the DBLog event format and is disc space. It is possible to prevent occupying additional table space
appended to an output buffer, which is in-memory and part of the by using vendor specific features, like the MySQL blackhole engine.
DBLog process. Another thread is then consuming events from the We developed a solution to this problem that only uses commonly
output buffer and sends them to the actual output in-order. The available database features and impacts the source database as little
output is a simple interface, allowing to plugin any destination, as possible. Instead of actually writing the latest state of rows into
such as a stream, datastore, or generally any kind of service that the transaction log, we are selecting rows from tables in chunks
has an API. and position the chunks in-memory next to events that we capture
We also capture schema changes. The nature of schema change from the transaction log. This is done in a way that does preserves
capture varies among databases, so that there may be schema the history of log events.
change deltas in the log, or the database may include the schema in- Our solution allows to extract the full state at any time via an
formation within each emitted event. The way we approach schema API for all tables, a specific table or for specific primary keys of a
capture in DBlog is not covered in this paper due to space limita- table. Selects are executed per table and in chunks of a configured
tions. size. Chunks are selected by sorting a table in ascending primary
key order and including rows, where the primary key is greater
3.2 Full state capture than the last primary key of the previous chunk. This query must
As transaction logs typically have limited retention they can not run efficiently in order to minimize impact on the source database.
be used to reconstruct the full source dataset. When attempting For these reasons, DBLog requires a database to provide an efficient
to solve this problem, two of the major challenges are to ensure range scan over primary keys and we only allow selects on tables
that log processing does not stall and that the order of history is that have a primary key. Figure 2 is illustrating chunk selection
preserved. One existing solution to this problem is to create a copy with a simple example.
We are storing the last row of a completed chunk in Zookeeper
1 Linearizable reads in Zookeeper are provided by calling sync(path) before reading so that we can pause and resume after the latest completed chunk.
data of the path [1][11]. Chunks need to be processed in a way that preserves the history of
 Andreakis and Papapanagiotou

log changes so that a chunk selection that returns an older value can Algorithm 1: Watermark-based Chunk Selection
not override newer state that is captured from the transaction log Input: table
and vice-versa. To achieve this, we create recognizable watermark
events in the transaction log so that we can sequence the chunk (1) pause log event processing
selection. Watermarks are implemented via a table that we create at lw := uuid(), hw := uuid()
the source database. The table is stored in a dedicated namespace so (2) update watermark table set value = lw
that no collisions occur with application tables. Only a single row (3) chunk := select next chunk from table
is inserted in the table which stores a Universally Unique Identifier (4) update watermark table set value = hw
(UUID) value. A watermark is then generated by updating the UUID
value of that row. The row update results in a change event which (5) resume log event processing
is eventually captured by DBLog. inwindow := 𝑓 𝑎𝑙𝑠𝑒
Algorithm 1 describes the watermark based approach to select // other steps of event processing loop
the next chunk of a specific table. The algorithm is repeated as while true do
long as the table has remaining chunks. Log event processing is e := next event from changelog
briefly paused (step 1). Watermarks are generated by updating if not inwindow then
the watermark table (steps 2 and 4). The chunk selection occurs if e is not watermark then
between the two watermarks and the chunk is stored in-memory append e to outputbuffer
(step 3). After the high watermark is written, we resume log event else if e is watermark with value lw then
processing, send received log events to the output, and watch for inwindow := 𝑡𝑟𝑢𝑒
the low watermark event in the log. Once the low watermark event else
is received, we start removing rows from the chunk in-memory for if e is not watermark then
all primary keys that changed between the watermarks (step 6). (6) if chunk contains e.key then
Once the high watermark event is received, we finally append all remove e.key from chunk
remaining chunk entries to the output-buffer before processing log
append e to outputbuffer
events again in a sequential manner (step 7).
else if e is watermark with value hw then
The chunk selection of step 3 is required to return state which
(7) for each row in chunk do
represents committed changes up to a certain point in history. Or
append row to outputbuffer
equivalently: the selection executes on a specific position of the
transaction log, considering committed transactions up to that point.
// other steps of event processing loop
Databases typically do not expose the execution position of a select
...
on the transaction log (MariaDB being an exception [9]). The core
idea of our approach is to determine a window on the transaction
log which guarantees to contain the chunk selection. The window
is opened by writing a low watermark, then the selection runs, Note that a large count of log events may appear between the
and the window is closed by writing a high watermark. As the low and high watermark, if one or more transactions committed
exact position of the select is unknown, all selected chunk rows a large set of row changes in between. Log event processing is
are removed, which collide with log events within that window. resumed event-by-event after step 4, eventually discovering the
This ensures that the chunk selection can not override the history watermarks and without ever needing to cache log event entries.
of log changes. In order for this to work, we must read the table Log processing is paused only briefly as steps 2–4 are expected to be
state from the time of the low watermark write, or later (it is fine fast: watermark updates are single write operations and the chunk
to include changes that committed after the low watermark write selection runs on a primary key index with a limit. Once the high
and before the read). More generally, it is required that the chunk watermark is received at step 7, the non-conflicting chunk rows
selection sees the changes that are committed before its execution. are appended to the output buffer in-order and ultimately delivered
We define this capability as ‘non-stale reads’. Additionally, as the to the output. Appending to the output buffer is a non-blocking
high watermark is written afterwards, we require that the select operation as the output delivery runs in a separate thread, allowing
executes before that. regular log processing to resume after step 7.
Figures 3a and 3b are illustrating the watermark algorithm for In Figure 4 we are depicting the order in which events are writ-
chunk selection. We provide an example with a table that has pri- ten to the output, by using the same example as figures 3a and 3b.
mary keys k1 to k6. Each change log entry represents a create, Log events that appear up to the high watermark are appended
update, or delete event for a primary key. The steps in the figures first. Then, the remaining rows from the chunk selection (under-
correspond to the labels of algorithm 1. In figure 3a, we showcase lined entries). And finally, log events that occur after the high
the watermark generation and chunk selection (steps 1 to 4). Up- watermark. This illustrates the interleaving of log and full data
dating the watermark table at step 2 and 4 creates two change extraction events.
events (highlighted with bolt) which are eventually received via
the change log. In figure 3b, we focus on the selected chunk rows 3.3 Database Support
that are removed from the result set for primary keys that appear In order to use DBLog a database needs to emit changed rows
between the watermarks (steps 5 to 7). from a linear history in commit order and support non-stale reads.
DBLog: A Watermark Based Change-Data-Capture Framework

are received via the streaming replication protocol which is im-

plemented by the PostgreSQL Java Database Connectivity (JDBC)
driver. Determining the schema per captured change varies be-
tween MySQL and PostgreSQL. In PostgreSQL, wal2json contains
the column names and types alongside with the column values. In
MySQL schema change deltas are received as binlog events.
Full state capture was integrated by using SQL and JDBC, only
requiring to implement the chunk selection and watermark update.
The same code is used for MySQL and PostgreSQL and can be used
for other databases with JDBC support as well. The dump processing
itself has no dependency on SQL or JDBC and allows to integrate
databases which fulfill the DBLog framework requirements even if
they are not RDBMS databases.
(a) steps 1-4

4 DBLOG IN PRODUCTION
DBLog is the foundation of the MySQL and PostgreSQL Connec-
tors at Netflix. They are both used in our data synchronization and
enrichment platform called Delta [7]. DBLog is running in produc-
tion since 2018 and as of the time this paper is written, it has been
deployed in about 30 production use services at Netflix. Those use
cases span across heterogeneous data replication, database activity
logging, and schema migration.
Heterogeneous data replication: In order to keep track of
productions it is crucial to search across all data that are related
(b) steps 5-7 to movies. This involves data that is managed by separate teams,
each of which is owning distinct business entities such as episodes,
Figure 3: Watermark-based Chunk Selection talents, and deals. These services use MySQL or PostgreSQL in AWS
RDS to store their data. DBLog is deployed to each of the involved
datastores and captures the full data set and real-time changes into
an output stream. The streams are then joined and ingested into a
common search index in ElasticSearch, providing search across all
involved entities.
Database activity logging: DBLog is also used to log database
activity, so that it can be used to inspect what kind of changes
occur on the database. In this scenario, changed rows are captured
and delivered to a stream. A stream processor is then propagating
the events to ElasticSearch (for short-term storage) and Hive (for
long-term storage). Kibana is used in ElasticSearch to build activity
dashboards so that teams can inspect the amount of occurred oper-
Figure 4: Order of output writes. Interleaving log capture ations per table. This is used to inspect data mutation patterns and
with full data capture. can be crucial to detect unexpected patterns such drop of inserts to
a table after a new service code rolled out with a bug.
Schema migration: When a team is migrating one MySQL
These conditions are fulfilled by systems like MySQL, PostgreSQL, database to another and a new table structure is used in the second
MariaDB, etc. so that the framework can be used uniformly across database. DBLog is deployed on the old database both to capture
these kinds of databases. the full state as well as new changes as they occur and writes them
So far, DBLog supports MySQL, PostgreSQL and Aurora. In all to a stream. A Flink job is then consuming that data, transforms
cases log events are provided by the database in commit order [2][4] them to the new table schema format and writes them into the
and non-stale reads are possible via the read committed isolation new database. This way, reads for the new database can be verified
level for single select transactions [3][5]. To integrate the log events, upfront by running on the populated new schema, while writes
for MySQL, we use the MySQL Binary Log Connector [17] which still occur to the old schema. In a follow up step, write traffic can
implements the binlog replication protocol. For PostgreSQL, we also occur to the new schema and traffic on the old database can be
are using replication slots with the wal2json plugin [18]. Changes stopped.
 Andreakis and Papapanagiotou

5 CONCLUSIONS [10] Debezium documentation. 2020. Debezium Connector for MySQL. Snapshots sec-
tion. https://debezium.io/documentation/reference/1.2/connectors/mysql.html#
In this paper, we presented a novel watermark based CDC frame- how-the-mysql-connector-performs-database-snapshots_debezium.
work. DBLog capabilities extend the capture of changed rows in [11] Martin Kleppmann. 2017. Designing Data-Intensive Applications. O’Reilly,
Beijing. https://www.safaribooksonline.com/library/view/designing-data-
real-time from a database transaction log to also extract the full intensive-applications/9781491903063/
state of a database as part of an integrated offering. In addition, [12] Martin Kleppmann, Alastair R. Beresford, and Boerge Svingen. 2019. Online event
DBLog provides endpoints for users to request the full state and exe- processing. Commun. ACM 62, 5 (2019), 43–49. https://doi.org/10.1145/3312527
[13] Kleppmann, Martin. 2015. Using logs to build a solid data infrastructure (or: why
cute it at any time and without stalling log event processing. This is dual writes are a bad idea). https://www.confluent.io/blog/using-logs-to-build-
achieved by executing selects on tables in chunks and interleaving a-solid-data-infrastructure-or-why-dual-writes-are-a-bad-idea.
the fetched rows with the log events so that both can progress. At [14] Avinash Lakshman and Prashant Malik. 2010. Cassandra: a decentralized struc-
tured storage system. ACM SIGOPS Operating Systems Review 44, 2 (2010), 35–40.
the same time, due to the watermark based approach the original [15] Prem Santosh Udaya Shankar. 2016. Streaming MySQL tables in real-time to
order of history is preserved at all times and without using locks Kafka. https://engineeringblog.yelp.com/2016/08/streaming-mysql-tables-in-
real-time-to-kafka.html.
on the source database. Moreover, controls are put in place which [16] Yogeshwer Sharma, Philippe Ajoux, Petchean Ang, David Callies, Abhishek
allow to throttle the chunk selection, or to pause and resume if Choudhary, Laurent Demailly, Thomas Fersch, Liat Atsmon Guz, Andrzej Kotul-
needed. This is especially relevant when capturing the full state on ski, Sachin Kulkarni, Sanjeev Kumar, Harry Li, Jun Li, Evgeniy Makeev, Kowshik
Prakasam, Robbert Van Renesse, Sabyasachi Roy, Pratyush Seth, Yee Jiun Song,
very large tables and the process crashes, so that the procedure does Benjamin Wester, Kaushik Veeraraghavan, and Peter Xie. 2015. Wormhole:
not need to be repeated from the beginning. DBLog is designed to Reliable Pub-Sub to Support Geo-replicated Internet Services. In 12th USENIX
deliver events to any output, regardless if it is a database, stream, or Symposium on Networked Systems Design and Implementation (NSDI 15). USENIX
Association, Oakland, CA, 351–366. https://www.usenix.org/conference/nsdi15/
API. These features open new avenues in synchronizing multiple technical-sessions/presentation/sharma
data systems. [17] Stanley Shyiko. 2010. MySQL Binary Log Connector. https://github.com/shyiko/
mysql-binlog-connector-java.
As Netflix operates hundreds of microservices with independent [18] Euler Taveira. 2014. wal2json - JSON output plugin for changeset extraction.
data needs, DBLog has become the foundation of Netflix’s data https://https://github.com/eulerto/wal2json.
synchronization and enrichment platform. It removed the complex- [19] Alexandre Verbitski, Anurag Gupta, Debanjan Saha, Murali Brahmadesam,
Kamal Gupta, Raman Mittal, Sailesh Krishnamurthy, Sandor Maurice, Tengiz
ity of application developers in maintaining multiple data stores. Kharatishvili, and Xiaofeng Bao. 2017. Amazon aurora: Design considerations
DBLog and its watermark based approach is designed to work for for high throughput cloud-native relational databases. In Proceedings of the 2017
RDBMS kind of databases. As a next step, we are working on other ACM International Conference on Management of Data. 1041–1052.
[20] Paolo Viotti and Marko Vukoliundefined. 2016. Consistency in Non-Transactional
CDC frameworks to support databases which don’t fall into the Distributed Storage Systems. ACM Comput. Surv. 49, 1, Article 19 (June 2016),
DBLog framework, such as multi-master NoSQL databases like 34 pages. https://doi.org/10.1145/2926965
[21] Guozhang Wang, Joel Koshy, Sriram Subramanian, Kartik Paramasivam, Mammad
Apache Cassandra [14]. The goal is to support similar capabilities Zadeh, Neha Narkhede, Jun Rao, Jay Kreps, and Joe Stein. 2015. Building a
as DBLog, namely: the ability to capture the full state at any time, Replicated Logging System with Apache Kafka. Proc. VLDB Endow. 8, 12 (Aug.
interleave with log events and have minimal impact at the source. 2015), 1654–1655. https://doi.org/10.14778/2824032.2824063
[22] Zendesk. 2014. Maxwell’s daemon, a MySQL-to-JSON Kafka producer. https:
//github.com/zendesk/maxwell.
ACKNOWLEDGMENTS
We would like to thank in alphabetic order the following colleagues
for contributing to the development of DBLog: Josh Snyder, Raghu-
ram Onti Srinivasan, Tharanga Gamaethige, and Yun Wang.

REFERENCES
[1] 2010. Apache Zookeeper. https://zookeeper.apache.org/.
[2] 2020. MySQL 5.7 Reference Manual - 5.4.4 The Binary Log. https://dev.mysql.
com/doc/refman/5.7/en/binary-log.html.
[3] 2020. MySQL 5.7 Reference Manual - Consistent Nonlocking Reads. https:
//dev.mysql.com/doc/refman/5.7/en/innodb-consistent-read.html.
[4] 2020. PostgreSQL 9.6 Documentation - Logical Decoding Output Plugins. https:
//www.postgresql.org/docs/9.6/logicaldecoding-output-plugin.html.
[5] 2020. PostgreSQL 9.6 Documentation - Transaction Isolation. https://www.
postgresql.org/docs/9.6/transaction-iso.html#XACT-READ-COMMITTED.
[6] Airbnb. 2018. Change Data Capture (CDC) service. https://github.com/airbnb/
SpinalTap.
[7] Andreas Andreakis, Falguni Jhaveri, Ioannis Papapanagiotou, Mark Cho, Poorna
Reddy, and Tongliang Liu. 2019. Delta: A Data Synchronization and Enrich-
ment Platform. https://netflixtechblog.com/delta-a-data-synchronization-and-
enrichment-platform-e82c36a79aee.
[8] Shirshanka Das, Chavdar Botev, Kapil Surlaker, Bhaskar Ghosh, Balaji Varadara-
jan, Sunil Nagaraj, David Zhang, Lei Gao, Jemiah Westerman, Phanindra Ganti,
Boris Shkolnik, Sajid Topiwala, Alexander Pachev, Naveen Somasundaram, and
Subbu Subramaniam. 2012. All aboard the Databus!: Linkedin’s scalable con-
sistent change data capture platform.. In SoCC, Michael J. Carey and Steven
Hand (Eds.). ACM, 18. http://dblp.uni-trier.de/db/conf/cloud/socc2012.html#
DasBSGVNZGWGSTPSS12
[9] Maria DB. 2020. Enhancements for START TRANSACTION WITH CONSISTENT
SNAPSHOT. https://mariadb.com/kb/en/enhancements-for-start-transaction-
with-consistent-snapshot/.