Bug#23604483 GCOLS: MINIMAL BINLOG_ROW_IMAGE LEAD TO CORRUPTION/CRASH…

… ON SLAVE

========
Problem:
========
Slave crashes when it is trying to apply an UPDATE row event
(with virtual generated columns) that came from a Master with
BINLOG_ROW_IMAGE=MINIMAL setting.

=========
Analysis:
=========

=============================================================
read_set/write_set bits requirement for GCol computation:
=============================================================
Stored generated columns are computed during an operation and
the value is stored inside storage engine. The computed value will be used
incase of next operation on that column. But Virtual generated column
values, unlike storged generated columns, computed always on the fly and
will not be stored anywhere. But if there is an index on top of virtual
generated columns, it will be maintained by Storage engine. The code
that is required to compute the Virtual generated columns on the fly
is added to all the handler functions (ha_rnd_pos, ha_rnd_next,
ha_index_read_map and etc.,). But due to performance reasons,
it will be computed only when the caller explictly asked to do so by
setting it's flag in read_set bitmap.
mark_generated_columns() function is used to set read_set/write_set
flags that are required to do generated columns modifications
(insert/update/delete).

=========================================================================
'Innodb' Storage engine behaviour/expecation in case of DMLs (update operations)
=========================================================================
In case DMLs are the ongoing operations and exclusive lock is acquired on
the table (which is true for all DMLs), fetch_all_key is set to true i.e.,
storage engine ignores table->read_set and fetches all the column values.
If a primary key (or the keys that can retrieve the tuple uniquely) is provided
to storage engine, storage engine will retrieve all column values from the clustered index
and fills the tuple struture (struct record) with all the column values
irrespective of what is there in table->read_set bitmap.

Also, Storage engine expects all the column values are ready in before_image
before calling ha_update_row, so that the value of a column from the image
can be used as key to find second index entry in case the index entry needs
to be updated.

=============================================================
RBR behaviour when BINLOG_ROW_IMAGE=MINIMAL and setting of
read_set/write_set bits in that setting:
=============================================================
In MySQL row-based replication, each row change event contains two images, a
“before” image whose columns are matched against when searching for the row
to be updated, and an “after” image containing the changes.
In case of MINIMAL setting, for the before image, it is necessary only that
the minimum set of columns required to uniquely identify rows is logged.
If the table containing the row has a primary key, then only the primary key
column or columns are written to the binary log. In the replication flow,
read_set and write_set bitmaps are also used to decide what all the columns
needs to be packed to write into the binary log.

=============================
Analysis on the server crash
=============================
Only DDLs and DMLs are replicated (SELECTs are never replicated).
With the above explanation, storage engines always return all the column
values in case of DMLs (i.e., fetch_all_key is true always for DMLs).
So in pure replication flow (i.e., RBR on slave does not go through regular
optimzer layer) never sets read_set bits to decide what needs to be retrieved
from the storage engine as it was getting all the column values from the
storage engine irrespective of what is there in 'read_set' bitmap.
But since virtual generated columns are not maintained by storage engine
and the retriveal/computation of virtual generated columns are depends on
its bit value in read_set bitmap, it is not computed now. Hence it is not
filled in before_image struture when RBR logic called handler function(in this
example, ha_rnd_pos).
Later when RBR called ha_update_row and storage engine wanted to update secondary
index created on virtua column(which is maintained by storage engine), it tries
to find secondary index entry for the given virtual column value (which is
NULL) and crashes there.

====
Fix:
====
Call mark_generated_columns() before calling any handler functions to
retrieve the before_image. This function will make sure to set all
the read_set/write_set bits that are required to compute/update
virutal columns.

binlog_prepare_row_image() function, which will be called from
binlogging functions (binlog_update_row() and binlog_delete_row())
will take care of removing these spurious fields required during execution
but not needed for binlogging. In case of inserts, there are no spurious
fields (all the columns are required to be written into the binlog)

Along with the fix, the patch is removing some code that was added
for testing purpose (in all three paths, INDEX_SCAN, TABLE_SCAN and
HASH_SCAN).

-  // Temporary fix to find out why it fails [/Matz]
-  memcpy(m_table->read_set->bitmap, m_cols.bitmap, (m_table->read_set->n_bits + 7) / 8);

(cherry picked from commit f8f0611a25df6ae1122fc3b27485d5d14d535e81)

Followup: BUG#23479595 SEGMENTATION FAULT WHEN SELECT FTS INDEX

          TABLES IN INFORMATION SCHEMA

BUG#23742339 FAILING ASSERTION: SYM_NODE->TABLE != NULL

Problem: When we access fts aux tables in information schema,the
fts aux tables are dropped by DROP DATABASE in another session.

Solution: Block DDL by s-locking dict_operation_lock.

Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>
RB: 13264
(cherry picked from commit c45124a0b582d57ac6cf20b786e0c643ec9ce941)

Bug#23736787 - YUM UPDATE FAIL FROM 5.5.51(COMUNITY/COMMERCIAL) TO 5.…

…6.32(COMUNITY/COMMERCIAL)

Remove mysql_config from client sub-package

(cherry picked from commit 45c4bfa0f3f1c70756591f48710bb3e76ffde9bc)

Upmerge of the 7.2.25 build

Bug #23129433 CLUSTER SHUTDOWN DURING UPGRADE FROM 7.3.13 --> 7.4.11

Problem exists in 7.4 DIH LCP takeover code.

This commit in 7.3 introduces a new testUpgrade testcase showing
the problem.

Fix will come on merge to 7.4

Bug #23225800: NDB_DD_DUMP TEST FAILING ON ALL PLATFORMS

(Server Bug #23259328: NDB_DD_DUMP TEST FAILING DUE TO CHANGE IN
 MYSQLDUMP)

Problem: Fix for Bug#21382184 caused a regression making ndb dump
test to fail. Reverted minor part of the code as part of the fix.

(cherry picked from commit cc696659a035448af3a8f42371fd9a5b5def83cd)

Fix spurious failures in ATR test 'testTransactions'

This test runs a mix of 2 different transaction T1/T2. For some
of the testcases, the second transaction is supposed to
be blocked by the first onw , and thus timeout with either
error 266 or 274. For other combinations, no such timeouts are
accepted.

In order to reduce the elapsed time for the test, the TC transaction
timeout was reduced to 100ms (default 3000ms). That had the
negative impact that even transaction combination not expected to
block each other, code spriously timeout.

As the setting of TC txn timeout was a 'global' setting, it
also remained set after this test had completed, and thus
could result in timeouts for later ATR test cases.

This patch:

1) Delay setting of  the 'short timeout' (100ms) to after
   The first (non-blocking( transaction is executed. Thus
   avoiding 'T1' to timeout.

2) Sets the 'short timeout' for T2 only for T1/T2 combinations
   where T2 is expected to fail with a timeout

3) Set transaction timeout back to 'default' as soon as the
   T2 has completed, this avoids that the short timeout
   affects other transactions than the one assumed to timeout.

Bug#23327563	DIST UPGRADE FAILING FROM 5.7.12(NATIVE) TO 5.7.13 PACKA…

…GES ON UBUNTU16.04

The control file doesn't correctly specify that the package conflicts with native 5.7,
causing errors because the file layout is a little different.
Added missing conflicts/replaces statements to control file.

(cherry picked from commit 7ca9852b0efcbce851598b3974f1d73be19f31bf)

WL#7162 Move ndb_binlog_index table from MyISAM to transactional storage

Removed output of binlog file path to be file system independent.

Merge branch 'mysql-5.5' into mysql-5.6

(cherry picked from commit d8de93d8853fce9246c5ada4de4b47b1f59b912e)