Bug#31989290: SYSTEM TABLES UPGRADED FROM 5.7 DIFFER FROM SYSTEM TABLES CREATED IN 8.0

PROBLEM:
--------
If we take a freshly initialized data directory from a MySQL
5.7 instance and upgrade it to 8.0, the definition of some
system tables (mysql schema) differs from the definition of
the same tables, in a freshly initialized datadir. of an 8.0
instance.

FIX:
----
The appropriate ALTER TABLE statements are added/modified in
mysql_system_tables_fix.sql to make the upgraded system
table definition match the definitions in
mysql_system_tables.sql

Additionally, an upgrade test is added to catch any new
inconsistencies introduced in the future to system tables
present in the mysql tablespace

Note: The diff in the file
mysql-test/std_data/dd/sdi/upgrade/mysql.json is the ibd2sdi
output generated by re-recording main.dd_upgrade_test

Additionally, the deprecated 'binary' keyword is removed from
mysql_system_tables_fix.sql in the cases where it was not
present in mysql_system_tables.sql (though this does not
affect the internal metadata)

Change-Id: I40c8220a8b70c88e62c28812b4950af25b316665

Author: Nischal-Tonthanahal

mysql-test/r/upgrade_system_tables.result

@@ -1,3 +1,63 @@
+
+# Dump system table information for the newly initialized data directory
+# Adding debug point 'skip_dd_table_access_check' to @@GLOBAL.debug
+
+# Dump all tables in newly initialized mysql schema
+include/write_result_to_file.inc [connection=default statement=SELECT t.* FROM mysql.tables t JOIN mysql.schemata s JOIN mysql.tablespaces ts WHERE t.schema_id = s.id AND t.tablespace_id = ts.id AND s.name = "mysql" AND ts.name = "mysql" ORDER BY t.name]
+include/filter_file.inc
+
+# Dump all columns in newly initialized mysql schema
+include/write_result_to_file.inc [connection=default statement=SELECT t.name, c.* FROM mysql.columns c JOIN mysql.tables t JOIN mysql.schemata s JOIN mysql.tablespaces ts WHERE c.table_id = t.id AND t.schema_id = s.id AND t.tablespace_id = ts.id AND s.name = "mysql" AND ts.name = "mysql" AND t.name != "ndb_binlog_index" ORDER BY t.name, c.name]
+include/filter_file.inc
+# Removing debug point 'skip_dd_table_access_check' from @@GLOBAL.debug
+
+# Upgrade a data directory from 5.7
+
+# Copy the 5.7 data zip file to working directory and check it exists.
+# Unzip 5.7 data directory.
+# Set data directory and log file
+# Restart server to trigger upgrade.
+# restart: --datadir=MYSQLD_DATADIR1 --log-error=MYSQLD_LOG
+
+# Dump system table information for the upgraded data directory
+# Adding debug point 'skip_dd_table_access_check' to @@GLOBAL.debug
+
+# Dump all tables in upgraded mysql schema
+include/write_result_to_file.inc [connection=default statement=SELECT t.* FROM mysql.tables t JOIN mysql.schemata s JOIN mysql.tablespaces ts WHERE t.schema_id = s.id AND t.tablespace_id = ts.id AND s.name = "mysql" AND ts.name = "mysql" ORDER BY t.name]
+include/filter_file.inc
+
+# Dump all columns in upgraded mysql schema
+include/write_result_to_file.inc [connection=default statement=SELECT t.name, c.* FROM mysql.columns c JOIN mysql.tables t JOIN mysql.schemata s JOIN mysql.tablespaces ts WHERE c.table_id = t.id AND t.schema_id = s.id AND t.tablespace_id = ts.id AND s.name = "mysql" AND ts.name = "mysql" AND t.name != "ndb_binlog_index" ORDER BY t.name, c.name]
+include/filter_file.inc
+# Removing debug point 'skip_dd_table_access_check' from @@GLOBAL.debug
+
+########################################################################
+# NOTE: If this test fails, there is a possible inconsistency introduced
+#       in the system table/column definition of an upgraded mysql
+#       tablespace as compared to that of a newly initialized one.
+#
+# There are 2 possibilities:
+#
+# a) If the failure shows a new inconsistency in the table/column
+#    properties, add the appropriate ALTER TABLE statements to
+#    scripts/mysql_system_tables_fix.sql
+#    to ensure that the metadata of the upgraded table/column is the
+#    same as that of the newly created one.
+#
+# b) If the inconsistency is unavoidable, modify the regular expressions
+#    above, re-record and update documentation if required.
+#
+########################################################################
+
+# Compare fresh and upgraded tables
+20c20
+< #	#	gtid_executed	BASE TABLE	InnoDB	#	Dynamic	255		Visible	avg_row_length=0;encrypt_type=N;explicit_tablespace=1;key_block_size=0;keys_disabled=0;pack_record=1;row_type=2;stats_auto_recalc=0;stats_sample_pages=0;	#	#	1	NULL	NULL	NULL	NULL	NULL	NULL	NULL	NULL	#	#	NULL	NULL	NULL	NULL	NULL	NULL	NULL	NULL	NULL	NULL	#	NULL	NULL
+---
+> #	#	gtid_executed	BASE TABLE	InnoDB	#	Dynamic	255		Visible	avg_row_length=0;encrypt_type=N;explicit_tablespace=1;key_block_size=0;keys_disabled=0;pack_record=1;row_type=2;stats_auto_recalc=0;stats_persistent=0;stats_sample_pages=0;	#	#	1	NULL	NULL	NULL	NULL	NULL	NULL	NULL	NULL	#	#	NULL	NULL	NULL	NULL	NULL	NULL	NULL	NULL	NULL	NULL	#	NULL	NULL
+
+# Compare fresh and upgraded columns
+
+# Cleanup.
 # Copy the 5.7 data zip file to working directory and check it exists.
 # Unzip 5.7 data directory.
 # Set data directory and log file