Skip to content

Commit 50b4039

Browse files
Vitor Oliveirabkandasa
Vitor Oliveira
authored and
bkandasa
committed
WL#9556 introduced the ability to use transaction write sets to
find out which transactions can be safely executed concurrently on the slave. The transactions are tracked on the master and the binary log commit_parent field, which is used by the MTS applier on the slave, is changed to consider writeset dependencies between transactions. Relevant changes to the code: - introduces two new options: binlog-transaction-dependency-tracking, which selects the tracking mode between COMMIT_ORDER, WRITESET and WRITESET_SESSION, and binlog-transaction-dependency-history-size. - adds the Transaction_dependency_tracker and associated classes Commit_order_trx_dependency_tracker, Writeset_trx_dependency_tracker and Writeset_session_trx_dependency_tracker, specific to each of the tracking modes (in the new rpl_trx_tracking.h/.cc files). - to support the WRITESET mode, the hashes of each row touched by a transaction are stored in the m_writeset_history map, up to the size determined by the binlog-transaction-dependency-history-size option. - to support the WRITESET_SESSION mode, the Dependency_tracker_ctx is added to the THD so that the last sequence_number used by a client is available for future transactions of the same session. - extends the function add_pke to track transactions that touch tables with primary keys and tables without them, so that partial writesets are detected. This WL back-ports WL#9556 to MySQL 5.7, in addition to the fixes for the following bugs: BUG#25950554: THE FUNCTION RESET_CONNECTION DOES NOT RESET WRITE SET... BUG#25982097: WRITESET DEPENDENCY TRACKING SUB-OPTIMAL AFTER ROTATION... (cherry picked from commit a888381055ea680e3985dccd28c10d1e2476c3eb)
1 parent ce64929 commit 50b4039

File tree

49 files changed

+5437
-189
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+5437
-189
lines changed

‎mysql-test/include/assert_grep.inc

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,10 @@ if ($assert_match != '')
9494
my $assert_only_after= $ENV{'_AG_ASSERT_ONLY_AFTER'};
9595
my $out= $ENV{'_AG_OUT'};
9696

97+
if (!defined($assert_count)) {
98+
$assert_count = '';
99+
}
100+
97101
my $result= '';
98102
my $count= 0;
99103
open(FILE, "$file") or die("Error $? opening $file: $!\n");

‎mysql-test/include/assert_logical_timestamps.inc

Lines changed: 23 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@
88
# --let $binlog_file= FILENAME
99
# [--let $binlog_position= OFFSET]
1010
# --let $logical_timestamps= LC1 SN1[;LC2 SN2[;LC3 SN3...]]
11+
# [--let $save_binlog_position= 1]
12+
# [--let $dont_print_logical_timestamps= 1]
1113
# --source include/assert_logical_timestamps.inc
1214
#
1315
# Parameters:
@@ -26,9 +28,23 @@
2628
# integer. Note that this is whitespace-sensitive so there should
2729
# not be any other space characters than those separating
2830
# last_committed from sequence_number.
29-
31+
#
32+
# $save_binlog_position
33+
# If this flag is set, the script will source
34+
# include/save_binlog_position.inc after it has finished the
35+
# assert, making the next invocation automatically continue where
36+
# this invocation ended. This is convenient if the script is to
37+
# be called multiple times in a sequence.
38+
#
39+
# $dont_print_logical_timestamps
40+
# By default, $logical_timestamps is printed to the result log. If
41+
# this flag is set, that is suppressed.
3042

3143
--let $include_filename= include/assert_logical_timestamps.inc [$logical_timestamps]
44+
if ($dont_print_logical_timestamps)
45+
{
46+
--let $include_filename= include/assert_logical_timestamps.inc
47+
}
3248
--source include/begin_include_file.inc
3349

3450

@@ -53,5 +69,11 @@ if ($binlog_position)
5369
--remove_file $assert_file
5470

5571

72+
if ($save_binlog_position)
73+
{
74+
--source include/save_binlog_position.inc
75+
}
76+
77+
5678
--let $include_filename= include/assert_logical_timestamps.inc [$logical_timestamps]
5779
--source include/end_include_file.inc

‎mysql-test/include/only_binlog_transaction_dependency_tracking_commit_order.inc

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
if (`SELECT @@GLOBAL.binlog_transaction_dependency_tracking != 'COMMIT_ORDER'`)
2+
{
3+
--skip Test requires binlog_transaction_dependency_tracking= COMMIT_ORDER
4+
}

‎mysql-test/r/mysqld--help-notwin.result

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,13 @@ The following options may be given as the first argument:
141141
non-transactional engines for the binary log. If you
142142
often use statements updating a great number of rows, you
143143
can increase this to get more performance
144+
--binlog-transaction-dependency-history-size=#
145+
Maximum number of rows to keep in the writeset history.
146+
--binlog-transaction-dependency-tracking=name
147+
Selects the source of dependency information from which
148+
to assess which transactions can be executed in parallel
149+
by the slave's multi-threaded applier. Possible values
150+
are COMMIT_ORDER, WRITESET and WRITESET_SESSION.
144151
--block-encryption-mode=name
145152
mode for AES_ENCRYPT/AES_DECRYPT
146153
--bootstrap Used by mysql installation scripts.
@@ -1292,6 +1299,8 @@ binlog-row-event-max-size 8192
12921299
binlog-row-image FULL
12931300
binlog-rows-query-log-events FALSE
12941301
binlog-stmt-cache-size 32768
1302+
binlog-transaction-dependency-history-size 25000
1303+
binlog-transaction-dependency-tracking COMMIT_ORDER
12951304
block-encryption-mode aes-128-ecb
12961305
bulk-insert-buffer-size 8388608
12971306
character-set-client-handshake TRUE

‎mysql-test/r/mysqld--help-win.result

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,13 @@ The following options may be given as the first argument:
141141
non-transactional engines for the binary log. If you
142142
often use statements updating a great number of rows, you
143143
can increase this to get more performance
144+
--binlog-transaction-dependency-history-size=#
145+
Maximum number of rows to keep in the writeset history.
146+
--binlog-transaction-dependency-tracking=name
147+
Selects the source of dependency information from which
148+
to assess which transactions can be executed in parallel
149+
by the slave's multi-threaded applier. Possible values
150+
are COMMIT_ORDER, WRITESET and WRITESET_SESSION.
144151
--block-encryption-mode=name
145152
mode for AES_ENCRYPT/AES_DECRYPT
146153
--bootstrap Used by mysql installation scripts.
@@ -1291,6 +1298,8 @@ binlog-row-event-max-size 8192
12911298
binlog-row-image FULL
12921299
binlog-rows-query-log-events FALSE
12931300
binlog-stmt-cache-size 32768
1301+
binlog-transaction-dependency-history-size 25000
1302+
binlog-transaction-dependency-tracking COMMIT_ORDER
12941303
block-encryption-mode aes-128-ecb
12951304
bulk-insert-buffer-size 8388608
12961305
character-set-client-handshake TRUE

‎mysql-test/suite/binlog/r/binlog_reset_connection_not_reset_writeset.result

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
SET GLOBAL transaction_write_set_extraction= 'XXHASH64';
2+
SET GLOBAL binlog_transaction_dependency_tracking= 'WRITESET';
3+
CREATE TABLE t1(id INT, PRIMARY KEY (id));
4+
FLUSH LOGS;
5+
INSERT INTO t1(id) VALUES(1);
6+
INSERT INTO t1(id) VALUES(2);
7+
SET GLOBAL binlog_transaction_dependency_tracking= 'WRITESET_SESSION';
8+
INSERT INTO t1(id) VALUES(3);
9+
INSERT INTO t1(id) VALUES(4);
10+
include/save_binlog_position.inc
11+
INSERT INTO t1(id) VALUES(5);
12+
include/include/assert_logical_timestamps.inc [4 5]
13+
SET GLOBAL binlog_transaction_dependency_tracking= COMMIT_ORDER;
14+
SET GLOBAL transaction_write_set_extraction= OFF;
15+
DROP TABLE t1;

‎mysql-test/suite/binlog/r/binlog_transaction_dependency_tracking_on_start.result

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
#
2+
# Test 1.a: Try starting server with TWSE=OFF and BTDT=COMMIT_ORDER. Expect SUCCESS.
3+
#
4+
# restart: --transaction_write_set_extraction=OFF --binlog_transaction_dependency_tracking=COMMIT_ORDER
5+
include/assert.inc [transaction_write_set_extraction should be OFF]
6+
include/assert.inc [binlog_transaction_dependency_tracking should be COMMIT_ORDER]
7+
#
8+
# Test 1.b: Try starting server with TWSE=OFF and BTDT=WRITESET. Expect FAILURE.
9+
#
10+
Occurrences of 'The transaction_write_set_extraction must be set to XXHASH64 or MURMUR32 when binlog_transaction_dependency_tracking is WRITESET or WRITESET_SESSION.' in the input file: 1
11+
Occurrences of '\[ERROR\] Aborting' in the input file: 1
12+
#
13+
# Test 1.c: Try starting server with TWSE=OFF and BTDT=WRITESET_SESSION. Expect FAILURE.
14+
#
15+
Occurrences of 'The transaction_write_set_extraction must be set to XXHASH64 or MURMUR32 when binlog_transaction_dependency_tracking is WRITESET or WRITESET_SESSION.' in the input file: 1
16+
Occurrences of '\[ERROR\] Aborting' in the input file: 1
17+
#
18+
# Test 1.d: Try starting server with TWSE=MURMUR32 and BTDT=COMMIT_ORDER. Expect SUCCESS.
19+
#
20+
# restart: --transaction_write_set_extraction=MURMUR32 --binlog_transaction_dependency_tracking=COMMIT_ORDER
21+
include/assert.inc [transaction_write_set_extraction should be MURMUR32]
22+
include/assert.inc [binlog_transaction_dependency_tracking should be COMMIT_ORDER]
23+
#
24+
# Test 1.e: Try starting server with TWSE=MURMUR32 and BTDT=WRITESET. Expect SUCCESS.
25+
#
26+
# restart: --transaction_write_set_extraction=MURMUR32 --binlog_transaction_dependency_tracking=WRITESET
27+
include/assert.inc [transaction_write_set_extraction should be MURMUR32]
28+
include/assert.inc [binlog_transaction_dependency_tracking should be WRITESET]
29+
#
30+
# Test 1.f: Try starting server with TWSE=MURMUR32 and BTDT=WRITESET_SESSION. Expect SUCCESS.
31+
#
32+
# restart: --transaction_write_set_extraction=MURMUR32 --binlog_transaction_dependency_tracking=WRITESET_SESSION
33+
include/assert.inc [transaction_write_set_extraction should be MURMUR32]
34+
include/assert.inc [binlog_transaction_dependency_tracking should be WRITESET_SESSION]
35+
#
36+
# Test 1.g: Try starting server with TWSE=XXHASH64 and BTDT=COMMIT_ORDER. Expect SUCCESS.
37+
#
38+
# restart: --transaction_write_set_extraction=XXHASH64 --binlog_transaction_dependency_tracking=COMMIT_ORDER
39+
include/assert.inc [transaction_write_set_extraction should be XXHASH64]
40+
include/assert.inc [binlog_transaction_dependency_tracking should be COMMIT_ORDER]
41+
#
42+
# Test 1.h: Try starting server with TWSE=XXHASH64 and BTDT=WRITESET. Expect SUCCESS
43+
#
44+
# restart: --transaction_write_set_extraction=XXHASH64 --binlog_transaction_dependency_tracking=WRITESET
45+
include/assert.inc [transaction_write_set_extraction should be XXHASH64]
46+
include/assert.inc [binlog_transaction_dependency_tracking should be WRITESET]
47+
#
48+
# Test 1.i: Try starting server with TWSE=XXHASH64 and BTDT=WRITESET_SESSION. Expect SUCCESS.
49+
#
50+
# restart: --transaction_write_set_extraction=XXHASH64 --binlog_transaction_dependency_tracking=WRITESET_SESSION
51+
include/assert.inc [transaction_write_set_extraction should be XXHASH64]
52+
include/assert.inc [binlog_transaction_dependency_tracking should be WRITESET_SESSION]
53+
SET GLOBAL binlog_transaction_dependency_tracking= COMMIT_ORDER;
54+
SET GLOBAL transaction_write_set_extraction= OFF;

‎mysql-test/suite/binlog/r/binlog_transaction_dependency_tracking_sensitive.result

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
SET @save_binlog_transaction_dependency_tracking = @@GLOBAL.binlog_transaction_dependency_tracking;
2+
SET @save_transaction_write_set_extraction = @@GLOBAL.transaction_write_set_extraction;
3+
SET @@GLOBAL.transaction_write_set_extraction = XXHASH64;
4+
SET @@SESSION.transaction_write_set_extraction = XXHASH64;
5+
SET @@GLOBAL.binlog_transaction_dependency_tracking = WRITESET;
6+
SET NAMES 'utf8' COLLATE 'utf8_general_ci';
7+
RESET MASTER;
8+
CREATE TABLE t1 (
9+
id INT NOT NULL AUTO_INCREMENT,
10+
u_id INT NOT NULL,
11+
u_str VARCHAR(32) NOT NULL,
12+
PRIMARY KEY (id),
13+
UNIQUE KEY (u_id, u_str)
14+
) ENGINE=InnoDB;
15+
INSERT INTO t1 (u_id, u_str) VALUES (1, 'a');
16+
INSERT INTO t1 (u_id, u_str) VALUES (1, 'A');
17+
ERROR 23000: Duplicate entry '1-A' for key 'u_id'
18+
INSERT INTO t1 (u_id, u_str) VALUES (1, 'ã');
19+
ERROR 23000: Duplicate entry '1-ã' for key 'u_id'
20+
INSERT INTO t1 (u_id, u_str) VALUES (1, 'Ã');
21+
ERROR 23000: Duplicate entry '1-Ã' for key 'u_id'
22+
INSERT INTO t1 (u_id, u_str) VALUES (1, 'á');
23+
ERROR 23000: Duplicate entry '1-á' for key 'u_id'
24+
DELETE FROM t1 WHERE u_id = 1 AND u_str = 'A';
25+
INSERT INTO t1 (u_id, u_str) VALUES (1, 'ã');
26+
DELETE FROM t1 WHERE u_id = 1 AND u_str = 'Ã';
27+
INSERT INTO t1 (u_id, u_str) VALUES (1, 'á');
28+
DROP TABLE t1;
29+
Processing binlog master-bin.000001
30+
FLUSH LOGS;
31+
include/include/assert_logical_timestamps.inc [0 1;1 2;2 3;3 4;4 5;5 6;6 7]
32+
SET @@GLOBAL.binlog_transaction_dependency_tracking = COMMIT_ORDER;
33+
SET @@GLOBAL.transaction_write_set_extraction= @save_transaction_write_set_extraction;
34+
SET @@GLOBAL.binlog_transaction_dependency_tracking= @save_binlog_transaction_dependency_tracking;

‎mysql-test/suite/binlog/t/binlog_reset_connection_not_reset_writeset.test

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
###############################################################################
2+
# Bug#25950554:THE FUNCTION RESET_CONNECTION DOES NOT RESET WRITE SET IN #
3+
# WRITESET_SESSION. #
4+
# #
5+
# Problem: When we do reset_connection which has the same effect as #
6+
# auto-reconnect, the write set session history is not cleared, #
7+
# which in case of auto-reconnect does get cleared. #
8+
# #
9+
# Fix: Reset the write set session history when mysql_reset_connection #
10+
# is called. #
11+
# #
12+
# Steps to reproduce: #
13+
# #
14+
# 1) Set the variable transaction_write_set_extraction to 'XXHASH64' which is #
15+
# necessary to be able to change the value of the variable #
16+
# binlog_transaction_dependency_tracking to WRITESET or WRITESET_SESSION #
17+
# #
18+
# 2) Set the variable binlog_transaction_dependency_tracking to WRITESET for #
19+
# few coming transactions. #
20+
# #
21+
# 3) Create a sample table and do few inserts. #
22+
# #
23+
# 4) Change the value of binlog_transaction_dependency_tracking to #
24+
# WRITESET_SESSION, to check if the write set session history is cleared #
25+
# or not after calling reset_connection. #
26+
# #
27+
# 5) Call reset_connection. #
28+
# #
29+
# 6) Check if the logical timestamp has the correct entry after the reset #
30+
# connection. This will mean that the write set history is cleared after a #
31+
# to reset_connection. #
32+
###############################################################################
33+
34+
--source include/have_binlog_format_row.inc
35+
36+
--let $saved_binlog_transaction_dependency_tracking= `SELECT @@GLOBAL.binlog_transaction_dependency_tracking`
37+
--let $saved_transaction_write_set_extraction= `SELECT @@GLOBAL.transaction_write_set_extraction`
38+
39+
SET GLOBAL transaction_write_set_extraction= 'XXHASH64';
40+
SET GLOBAL binlog_transaction_dependency_tracking= 'WRITESET';
41+
42+
CREATE TABLE t1(id INT, PRIMARY KEY (id));
43+
FLUSH LOGS;
44+
INSERT INTO t1(id) VALUES(1);
45+
INSERT INTO t1(id) VALUES(2);
46+
47+
SET GLOBAL binlog_transaction_dependency_tracking= 'WRITESET_SESSION';
48+
INSERT INTO t1(id) VALUES(3);
49+
INSERT INTO t1(id) VALUES(4);
50+
51+
reset_connection;
52+
53+
--source include/save_binlog_position.inc
54+
55+
INSERT INTO t1(id) VALUES(5);
56+
57+
--let $logical_timestamps= 4 5
58+
--source include/assert_logical_timestamps.inc
59+
60+
# Clean up
61+
--eval SET GLOBAL binlog_transaction_dependency_tracking= $saved_binlog_transaction_dependency_tracking
62+
--eval SET GLOBAL transaction_write_set_extraction= $saved_transaction_write_set_extraction
63+
DROP TABLE t1;

‎mysql-test/suite/binlog/t/binlog_transaction_dependency_tracking_on_start-master.opt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
--no-console --log-error=$MYSQLTEST_VARDIR/tmp/binlog_transaction_dependency_tracking.err

0 commit comments

Comments
 (0)