Skip to content

Commit 863aac4

Browse files
ssorumgardssorumgard
committed
WL#9553: Upgrading the transactional data dictionary tables.
This patch implements support for changing the DD table definitions. Overview. --------- The main changes are the following: - SE private data previously hard coded in InnoDB is now used only during first time server start. The meta data is stored in a DD table. On restart, the meta data is fetched from the DD table instead of InnoDB. Thus, we may have SE private data that can change. - During upgrade, we create the required target tables in a temporary schema, and migrate the meta data from the actual DD tables to the new target tables. - At the end of upgrade, we modify the persistently stored SE private data mentioned in 1) to that of the new target tables. We also adjust the schema ID of the target tables to simulate altering the schema of the tables. This way, we are able to switch from the old to the new DD version atomically. The temporary schemas are removed on next restart. In more detail, the patch implements the following: Performance schema. ------------------- - Change in a performance schema test: Select only the PS_VERSION from 'mysql.dd_properties' to avoid reflecting irrelevant meta data in the result file, and thus to avoid unnecessary re-recordings of the result file. - Minor changes in the way Plugin_table_impl is used; replaced by Object_table_impl for more uniform code. - Approved by Marc Alff. Handler and handlerton API and InnoDB. -------------------------------------- - The handler function 'get_se_private_data()' will now be called only during '--initialize'. During ordinary restart, it will be called once to get the SE private data for the 'mysql.dd_properties' table, which can never change. This is the table that stores the SE private data for the other DD tables. - Modify 'get_se_private_data()' to be in line with what 'dd_write_table()' does when storing SE private data: * Store se_private_data for columns. This is now done for DD tables in the same way as it is done for user tables. * Extend se_private_data for indexes to also store table_id and space_id. Again, this is done to get the same set of meta data for DD tables and user tables. Could dd_write_table() be used to ensure consistency in the meta data that is stored? - At the end of upgrade, we start over DD initialization to do the same as for a restart. Thus, we had to provide a new parameter to 'get_se_private_data()' to reset the counters for this to work with two phases of function calls in the case of upgrade. - We maintain a set in InnoDB of SE private ids of the DD tables. This set is used this instead of the hard coded id range used previously. - Replace hard coded ids of tables used in the processing of I_S queries by name based lookup. - Change the order of the DD and DDSE tables in the System_tables registry to make sure the table 'innodb_dynamic_metadata' is created on a low table id and index id. The motivation is that for now, this table must stay at fixed ids because it may be opened by InnoDB before the DD is available. - Approved by Jimmy. Extensions of 'mysql.dd_properties' and data structures. -------------------------------------------------------- - Valid key/value pairs are explicitly defined: DD_VERSION Actual DD version. IS_VERSION Actual I_S version. PS_VERSION Actual P_S version. SDI_VERSION Actual SDI version. LCTN L_C_T_N setting used during --initialize. MYSQLD_VERSION_LO Lowest server version which has been using the data directory. MYSQLD_VERSION_HI Highest server version which has been using the data directory. MYSQLD_VERSION Current server version. MINOR_DOWNGRADE_THRESHOLD The current DD can be used by previous MRUs, unless their target DD version is less than the downgrade threshold. SYSTEM_TABLES List of system tabels with definitions. UPGRADE_TARGET_SCHEMA Temporary schema used during upgrade. UPGRADE_ACTUAL_SCHEMA Temporary schema used during upgrade. - Simplify Object_table, Object_table_definition, Plugin_table_definition, their subclasses and clients. Remove unnecessary functions, and rename according to usual naming rules. Merge Object_table* and Plugin_table* into one class. - Version number handling does not need to be part of these classes, this will be handled elsewhere, so it is removed. - Object table definitions now may hold definitions of both the target and actual tables. - Introduce explicit enumerations for options, indexes and foreign keys for the DD table definitions. - Explicitly define indexes needed by foreign keys. - Use the index enumerations when creating instances of object keys. - A new 'DD_bootstrap_ctx' class is introduced as an aid in the upgrade process, but also for normal DD bootstrapping. The handling of the bootstrap stages is moved into this class. Changes to the current DD initialization. ----------------------------------------- - Extend the bootstrapper to create target or actual tables depending on context. - Change the way DD objects are flushed (during --initialize) and synced (during restart) to avoid problems with overlapping ID sequences for the scaffolding and the persisted object IDs. This is needed since the DD tables will no more be fixed on low IDs. - Add a new stage before creating tables where the inert table 'dd_properties' is opened and the version numbers etc. is read. Here, the actual DD table definitions are read in case of upgrade or minor downgrade. New handling of upgrade. ------------------------ - Create two temporary schemas with unused schema names, store in 'dd_properties'. - Upon restart, the temporary schemas are dropped. - First initialize the meta data for the actual DD tables, and use this to open the actual tables. - Then create the target tables, and migrate the meta data from the old to the new tables. - Adjust object ids to simulate altering schema of the new and old DD tables at the end of DD upgrade. - Update properties for all tables, make sure removed tables are not reflected in the persisted properties. Add 'options' columns. ---------------------- - Add a general purpose column to all DD tables that store DD entities (i.e.: catalogs, character_sets, collations, column_statistics, events, resource_groups, routines, schemata, st_spatial_reference_systems, *tables, *tablespaces - Plus a selection of important non-entity tables: *columns, *indexes, foreign_keys, triggers, *parameters - Tables prefixed by '*' above already have a column named 'options' which can be used for this purpose. Miscellaneous. -------------- - Set created/last altered when creating schema. - Add command line option for disabling automatic DD upgrade. - Change test for is_dd_table_name() to check for table types in the System_tables registry. - Change dictionary object type names for better conformity. - Remove *_type classes for the DD object classes. - Refactor object table usage Test changes. ------------- - Extend dd_schema_definition_debug_c{i,s} to also record the CREATE TABLE statements for the DD tables. - Record new test results. - Mask out the DD version number from the SDI which is extracted from tablespace files in some tests.
1 parent cf0a8f2 commit 863aac4

File tree

243 files changed

+9820
-5381
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

243 files changed

+9820
-5381
lines changed

‎mysql-test/include/dd_schema_definition_debug.inc

Lines changed: 122 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,101 @@ use mysql;
1313

1414
SET GLOBAL debug= '+d,skip_dd_table_access_check';
1515

16+
--echo #
17+
--echo # List the CREATE TABLE statements for the DD tables.
18+
--echo # Hide the AUTO INCREMENT counter, which is not
19+
--echo # deterministic when running mtr.
20+
--replace_regex / AUTO_INCREMENT=[0-9]+//
21+
SHOW CREATE TABLE catalogs;
22+
--replace_regex / AUTO_INCREMENT=[0-9]+//
23+
SHOW CREATE TABLE character_sets;
24+
--replace_regex / AUTO_INCREMENT=[0-9]+//
25+
SHOW CREATE TABLE collations;
26+
--replace_regex / AUTO_INCREMENT=[0-9]+//
27+
SHOW CREATE TABLE column_statistics;
28+
--replace_regex / AUTO_INCREMENT=[0-9]+//
29+
SHOW CREATE TABLE column_type_elements;
30+
--replace_regex / AUTO_INCREMENT=[0-9]+//
31+
SHOW CREATE TABLE columns;
32+
--replace_regex / AUTO_INCREMENT=[0-9]+//
33+
SHOW CREATE TABLE dd_properties;
34+
--replace_regex / AUTO_INCREMENT=[0-9]+//
35+
SHOW CREATE TABLE events;
36+
--replace_regex / AUTO_INCREMENT=[0-9]+//
37+
SHOW CREATE TABLE foreign_key_column_usage;
38+
--replace_regex / AUTO_INCREMENT=[0-9]+//
39+
SHOW CREATE TABLE foreign_keys;
40+
--replace_regex / AUTO_INCREMENT=[0-9]+//
41+
SHOW CREATE TABLE index_column_usage;
42+
--replace_regex / AUTO_INCREMENT=[0-9]+//
43+
SHOW CREATE TABLE index_partitions;
44+
--replace_regex / AUTO_INCREMENT=[0-9]+//
45+
SHOW CREATE TABLE index_stats;
46+
--replace_regex / AUTO_INCREMENT=[0-9]+//
47+
SHOW CREATE TABLE indexes;
48+
--replace_regex / AUTO_INCREMENT=[0-9]+//
49+
SHOW CREATE TABLE innodb_ddl_log;
50+
--replace_regex / AUTO_INCREMENT=[0-9]+//
51+
SHOW CREATE TABLE innodb_dynamic_metadata;
52+
--replace_regex / AUTO_INCREMENT=[0-9]+//
53+
SHOW CREATE TABLE innodb_index_stats;
54+
--replace_regex / AUTO_INCREMENT=[0-9]+//
55+
SHOW CREATE TABLE innodb_table_stats;
56+
--replace_regex / AUTO_INCREMENT=[0-9]+//
57+
SHOW CREATE TABLE parameter_type_elements;
58+
--replace_regex / AUTO_INCREMENT=[0-9]+//
59+
SHOW CREATE TABLE parameters;
60+
--replace_regex / AUTO_INCREMENT=[0-9]+//
61+
SHOW CREATE TABLE resource_groups;
62+
--replace_regex / AUTO_INCREMENT=[0-9]+//
63+
SHOW CREATE TABLE routines;
64+
--replace_regex / AUTO_INCREMENT=[0-9]+//
65+
SHOW CREATE TABLE schemata;
66+
--replace_regex / AUTO_INCREMENT=[0-9]+//
67+
SHOW CREATE TABLE st_spatial_reference_systems;
68+
--replace_regex / AUTO_INCREMENT=[0-9]+//
69+
SHOW CREATE TABLE table_partition_values;
70+
--replace_regex / AUTO_INCREMENT=[0-9]+//
71+
SHOW CREATE TABLE table_partitions;
72+
--replace_regex / AUTO_INCREMENT=[0-9]+//
73+
SHOW CREATE TABLE table_stats;
74+
--replace_regex / AUTO_INCREMENT=[0-9]+//
75+
SHOW CREATE TABLE tables;
76+
--replace_regex / AUTO_INCREMENT=[0-9]+//
77+
SHOW CREATE TABLE tablespace_files;
78+
--replace_regex / AUTO_INCREMENT=[0-9]+//
79+
SHOW CREATE TABLE tablespaces;
80+
--replace_regex / AUTO_INCREMENT=[0-9]+//
81+
SHOW CREATE TABLE triggers;
82+
--replace_regex / AUTO_INCREMENT=[0-9]+//
83+
SHOW CREATE TABLE view_routine_usage;
84+
--replace_regex / AUTO_INCREMENT=[0-9]+//
85+
SHOW CREATE TABLE view_table_usage;
86+
87+
# The ids of the dd_properties table must be fixed.
88+
SELECT id = 1 FROM tables WHERE name LIKE 'dd_properties';
89+
SELECT se_private_id = 1 FROM tables WHERE name LIKE 'dd_properties';
90+
SELECT id = 1 FROM indexes WHERE table_id = (SELECT id FROM tables WHERE name LIKE 'dd_properties');
91+
92+
# The ids of the innodb_dynamic_metadata table must be fixed.
93+
SELECT id = 2 FROM tables WHERE name LIKE 'innodb_dynamic_metadata';
94+
SELECT se_private_id = 2 FROM tables WHERE name LIKE 'innodb_dynamic_metadata';
95+
SELECT id = 2 FROM indexes WHERE table_id = (SELECT id FROM tables WHERE name LIKE 'innodb_dynamic_metadata');
96+
97+
# The DD database id must be fixed.
98+
SELECT id = 1 FROM schemata WHERE name LIKE 'mysql';
99+
100+
# The DD tablespace id must be fixed.
101+
SELECT id = 1 FROM tablespaces WHERE name LIKE 'mysql';
102+
103+
# The DD tablespace se private id must be fixed.
104+
SELECT SUBSTRING_INDEX(
105+
SUBSTRING(se_private_data,
106+
LOCATE('id', se_private_data),
107+
30),
108+
';', 1) = 'id=4294967294'
109+
FROM tablespaces WHERE name LIKE 'mysql';
110+
16111
# The DD database id
17112
let $dd_schema_id= `SELECT id FROM schemata
18113
WHERE name= 'mysql'`;
@@ -21,27 +116,39 @@ let $dd_schema_id= `SELECT id FROM schemata
21116
let $dd_table_ids=
22117
SELECT id FROM tables
23118
WHERE schema_id= $dd_schema_id
24-
AND name IN ('innodb_table_stats',
25-
'innodb_index_stats',
119+
AND name IN ('catalogs',
26120
'character_sets',
27121
'collations',
28-
'tablespaces',
29-
'tablespace_files',
30-
'catalogs',
31-
'schemata',
32-
'tables',
33-
'view_table_usage',
34-
'columns',
35122
'column_statistics',
36-
'indexes',
37-
'index_column_usage',
38123
'column_type_elements',
39-
'foreign_keys',
124+
'columns',
125+
'dd_properties',
126+
'events',
40127
'foreign_key_column_usage',
41-
'table_partitions',
42-
'table_partition_values',
128+
'foreign_keys',
129+
'index_column_usage',
43130
'index_partitions',
44-
'dd_properties');
131+
'index_stats',
132+
'indexes',
133+
'innodb_ddl_log',
134+
'innodb_dynamic_metadata',
135+
'innodb_index_stats',
136+
'innodb_table_stats',
137+
'parameter_type_elements',
138+
'parameters',
139+
'resource_groups',
140+
'routines',
141+
'schemata',
142+
'st_spatial_reference_systems',
143+
'table_partition_values',
144+
'table_partitions',
145+
'table_stats',
146+
'tables',
147+
'tablespace_files',
148+
'tablespaces',
149+
'triggers',
150+
'view_routine_usage',
151+
'view_table_usage');
45152

46153
# Expression representing the DD index ids
47154
let $dd_index_ids= SELECT id FROM indexes

‎mysql-test/include/ibd2sdi.pl

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,9 @@ ()
4747
# Remove mysql version id
4848
$_=~ s/("mysqld?_version_id": )[0-9]+/$1X/g;
4949

50+
# Remove dd version id
51+
$_=~ s/("dd_version": )[0-9]+/$1X/g;
52+
5053
# Remove extra path separator seen on windows and partition names
5154
if (m/"filename":/)
5255
{
@@ -110,6 +113,9 @@ ()
110113
# Remove mysql version id
111114
$_=~ s/("mysqld?_version_id": )[0-9]+/$1X/g;
112115

116+
# Remove dd version id
117+
$_=~ s/("dd_version": )[0-9]+/$1X/g;
118+
113119
# Remove extra path separator seen on windows and partition names
114120
if (m/"filename":/)
115121
{
@@ -172,6 +178,9 @@ ()
172178
# Remove mysql version id
173179
$_=~ s/("mysqld?_version_id": )[0-9]+/$1X/g;
174180

181+
# Remove dd version id
182+
$_=~ s/("dd_version": )[0-9]+/$1X/g;
183+
175184
# Remove extra path separator seen on windows and partition names
176185
if (m/"filename":/)
177186
{

‎mysql-test/include/resource_group_common_debug.inc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,13 +27,13 @@ if ($thr_prio_enable == 1)
2727
{
2828
# Insert invalid thread priority value along with invalid VCPU ranges
2929
# and enable the resource group.
30-
INSERT INTO mysql.resource_groups VALUES(3, 'r1', 'USER', 1, "0100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", -5);
30+
INSERT INTO mysql.resource_groups VALUES(3, 'r1', 'USER', 1, "0100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", -5, NULL);
3131
}
3232

3333
if ($thr_prio_enable == 0)
3434
{
3535
# Insert invalid VCPU ids and enable the resource group.
36-
INSERT INTO mysql.resource_groups VALUES(3, 'r1', 'USER', 1, "0100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", 0);
36+
INSERT INTO mysql.resource_groups VALUES(3, 'r1', 'USER', 1, "0100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", 0, NULL);
3737
}
3838
call mtr.add_suppression("Invalid cpu id.*");
3939
call mtr.add_suppression("Validation of resource group.*");

0 commit comments

Comments
 (0)