Typo in magic file #5
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MySQL Bug #71898
|
Hi, thank you for your contribution. Please confirm this code is submitted under the terms of the OCA (Oracle's Contribution Agreement) you have previously signed by cutting and pasting the following text as a comment: |
|
I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it. |
|
Hi, thank you for your contribution. Your code has been assigned to an internal queue. Please follow |
bjornmu
pushed a commit
that referenced
this pull request
Apr 10, 2017
…FOR DATABASE Patch #5 To repeat: ./mtr --mem --charset-for-testdb=utf8mb4 costmodel_planchange func_str func_weight_string group_min_max loaddata outfile_loaddata Fix: re-record with explicit charset for queries/tables
bjornmu
pushed a commit
that referenced
this pull request
Apr 10, 2017
WL#9499 - InnoDB: Replace MLOG_FILE_NAME with MLOG_FILE_OPEN Fix the performance issues introduced by WL#7142 and its followup WL#7806. 1. Revert WL#7142 and WL#7806 changes, keep some of the tests to test the behaviour that was introduced by the two WLs 2. Change the tests that rely on the above two WLs behaviour 3. Use std::unordered_map instead of the homebrew hash table for the recovery code and space ID to name mapping 4. Split the redo log hash table key from <space, page> -> apply records to <space> -> apply records 5. Addition of two new system files (tablespaces.open.1 and tablespaces.open.2) - Rename MLOG_FILE_NAME to MLOG_FILE_OPEN - Remove the two pass recovery code, make it a single pass - Track file open, close and rename - Write and fsync the tracking state to disk on checkpoint and rename. Uses the files in #5 in a round robin fashion Introduce --innodb-scan-directories="path1;...pathN". Scan the paths for .ibd files to open during recovery. This option can be used if the files in #5 above are corrupt or missing. You should not use this if you move files around. While they can be recovered the data dictionary will not be updated and it will cause issues during runtimg. rb#13686 Approved by Satya and Shaohua commit fbf87161f3c93979e5abe424fbe0678fdc32159e Merge: 517cf3a7bf0 c2b504664ed Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 28 10:39:32 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 517cf3a7bf0490c3168fed020e3096ec500e0a85 Merge: 4786f93ad8a 5e974fd0ea4 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 28 10:24:25 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 Additional change is to call recv_recovery_from_checkpoint_finish() on abort. Instead of recv_sys_free(). commit 4786f93ad8a25506a2a1f6e9d1207256a34b3665 Merge: 5dc35e05b5f 275245cce32 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 24 10:43:27 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 5dc35e05b5f2da95c36e191350e9c7087be72194 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 24 10:42:40 2017 +0100 WL#9499 Free recv resources on abort commit bed693070395923ededba736f1c7102b8eb21e95 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 23 18:34:50 2017 +0100 WL#9499 - Test cannot be run with Valgrind commit c5d423228ddf58b17866ff8be289dba4a19205d2 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 23 17:30:39 2017 +0100 WL#9499 - Remove const commit ac42498506b4dffb22e74282d0ce037fe3acf977 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 23 17:00:30 2017 +0100 WL#9499 - Clean up code commit bc48077cab9c3ccb1b2797c365fe0940c269a785 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 23 16:01:09 2017 +0100 WL#9499 - Revert unrelated change commit 8b2bb0f6d6254b3cfde168d41356400009db12f4 Merge: eb9b294ba78 153a79ff35d Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 23 11:43:38 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit eb9b294ba786800f9e397cd87f01c4f4976edddb Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 23 11:29:56 2017 +0100 WL#9499 - Remove debug message commit abae66ad591854ee67a5013f3209be2f82e4d9bf Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Feb 22 17:04:10 2017 +0100 WL#9499 - Test only works in debug mode. commit 71ebea70391ef295bed49ee06a8b872f15ecfe97 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Feb 22 12:18:08 2017 +0100 WL#9499 - Update to mtr test from Matthias commit f108a5b92fc476c70c49e4937496b32d9d5d031f Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Feb 22 12:13:04 2017 +0100 WL#9499 - Increment redo log version number commit 4439c83a4d3edd6e511706ccf9936770ed29e593 Merge: fa459186b90 05923bef41d Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Feb 22 09:54:18 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit fa459186b90d7a5ddcba7e3c6c7a9871d0dce793 Merge: 14ac4f81d84 b401f17e70d Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 20:02:40 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 14ac4f81d84bec23a7be210504ca0be93619bf11 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 16:44:57 2017 +0100 WL#9499 - Update from Matthias commit ed3191a9fc951151fd7d758a527e7ab81530b3f0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 15:16:11 2017 +0100 WL#9499 - Revert change pushed by nmistake commit 8c0edb998867bf557f6c86410552d6d123892000 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 15:11:21 2017 +0100 WL#9499 - Don't redo log MLOG_FILE_OPEN until startup is complete Date: Wed Feb 22 09:54:18 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit fa459186b90d7a5ddcba7e3c6c7a9871d0dce793 Merge: 14ac4f81d84 b401f17e70d Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 20:02:40 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 14ac4f81d84bec23a7be210504ca0be93619bf11 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 16:44:57 2017 +0100 WL#9499 - Update from Matthias commit ed3191a9fc951151fd7d758a527e7ab81530b3f0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 15:16:11 2017 +0100 WL#9499 - Revert change pushed by nmistake commit 8c0edb998867bf557f6c86410552d6d123892000 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 15:11:21 2017 +0100 WL#9499 - Don't redo log MLOG_FILE_OPEN until startup is complete commit 3b503ea1d3f89c0f7e3677d3b56f249c7ebff506 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 11:37:04 2017 +0100 WL#9499 - Add tests commit 47af7bd6eec10eac55633d7c4cf54afe607f3649 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 11:00:30 2017 +0100 WL#9499 - Fix compile problem commit a8b9a8c6ccb15e2be5ce5db5fa0bce24b86d2c87 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Feb 20 16:51:05 2017 +0100 WL#9499 Remove dead code commit b1dba474486ad10b61029b2ef4e6a91ea67dcea8 Merge: 3cf9e9eb3e8 a4271026201 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Feb 20 16:49:38 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 3cf9e9eb3e86e4c8f4d44cb79d3f7c456ea7d671 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Feb 18 07:53:43 2017 +0530 WL#9499 - Remove suggestion to use force recovery --innodb-scan-directories is a safer option for the user. Force recovery should be the absolute last resort. commit 4104c4bfd606270650aac95a9a724e106fe835c5 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 17 09:00:47 2017 +0530 WL#9499 - Fix error message User can try --innodb-scan-directories="..." if the tablespaces.open.* files are corrupt or unreadable. commit 19ab12e54923b070a0b4ffc76b9fe97ac54ad3cc Merge: 50b3a95e446 f0f51c410bd Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 17 05:22:57 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 Replace readdir_r() with readdir(), compiler complains that readdir_r() is deprecated and recommends replacing it with readdir(). commit 50b3a95e446b91853fc35716b63bdc173221d01f Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 16 18:55:33 2017 +0530 WL#9499 - Fix test commit 45d202ae0fa35aa3aced55cc1933aeefa33837de Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 16 18:32:56 2017 +0530 WL#9499 - Remove redundant test commit 48fe691b97735e5144e9d439a0adbf721b2dc554 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 16 18:18:11 2017 +0530 WL#9499 - Add DEBUG sync points commit d986f4e04a9eb6c2c76d61cd4afe9541d77b34bd Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 16 18:03:50 2017 +0530 WL#9499 - Remove PB2 control file commit 5af7dca9c305bd366ad2b5c3de26a870e913af4a Merge: 74022f50a5f 23a0e71aa3d Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Feb 15 09:05:05 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 74022f50a5ff7cb2801c0712601f8a467b409a28 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 14 10:31:00 2017 +0530 WL#9499 - Fix test commit f6188175ee62040ac6a844a5c54787839dff6155 Merge: 24c867aab14 19ca7c6a5d3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Feb 13 12:37:22 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 24c867aab14556188503aac495f60d89ba3d26ec Merge: cb64b8c7f9f a4de6ab549e Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Feb 4 06:41:23 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit cb64b8c7f9fa9ca05033a9d6f6193cf141746310 Merge: 17c6c41da0a 88dbc4f4511 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 3 19:48:12 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 17c6c41da0afe3fc78843b75fa02578dab29a0ca Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 3 19:46:34 2017 +0530 WL#9499 - Add a test for scan directories. The use case is where the location of the files is not changed but the tablespaces.open.* files are missing. commit 79cc52f1bd82dc65932eacbc9aa14a47e4196572 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 3 18:01:50 2017 +0530 WL#9499 - If open fails space will be null. commit 22ea5f31ecc1fa0db8abd57d4a46578e5f891c9a Merge: f5b69930f95 853a52cf5a1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 3 12:14:41 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit f5b69930f95192ddfdb3ca170024a5c7e3416076 Merge: 40a4c32f2c4 b18872ed4a3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 2 13:54:24 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 40a4c32f2c42b584539a14cbc1ba0d4f0fe7d3c9 Merge: b1d1923d37c d071c0e7a0b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Feb 1 13:48:33 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit b1d1923d37c3d23a9f701a61b458b7e4cbd3c90c Merge: 2fc6c51edab 4ae71705a01 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Jan 31 10:44:59 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 2fc6c51edabc3de4537eceb0b82cd62063ada3a9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Jan 30 16:36:46 2017 +0530 WL#9499 - Empty the tablespace.open.* files on normal shutdown. commit 738acb7d554f97455c8dba512282c7c2e6ba9f3a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Jan 30 12:47:21 2017 +0530 WL#9499 - Fix formatting commit 28f8579a6f729af354e0c50098d16fbfa17cc253 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Jan 30 12:25:30 2017 +0530 WL#9499 - Bug fix The DELETED state was set on a copy of the data structure commit dd5bf8ea81b7023fa1980c01d89219a690160b04 Merge: 7eeab0a994d 9c7808e1054 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 27 21:14:22 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 7eeab0a994db4cf0c2e080f27ff916c778ec65f8 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 27 20:50:59 2017 +0530 WL#9499 - Add tests. commit ddfaf047f43587ee807599b0b80b6d01a3579e8c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 27 20:48:06 2017 +0530 WL#9499 - Create tablespace bug fix If there is a checkpoint after the create tablespace is logged and the server crashes after writing the page initialization to the redo log. Then we neither have the mapping in the redo log nor in the tablespace.open.* files. We must update the tablespace mapping before writing the CREATE redo log entry so that the state is written out to disk on checkpoint. commit 29546d3ebea2991e5b86b33edddb551c924bf1db Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 27 17:55:28 2017 +0530 WL#9499 - Remove dead code from WL#7142 commit eced7a3e4437ee4bb24a12290f561437ddb15611 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 27 14:51:09 2017 +0530 WL#9499 - Use the tablespace.open.* files in a round robin fashion. commit d8eff788eea1ad5e6c7786b9d64963524f2b2c81 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 27 13:56:12 2017 +0530 WL#9499 - Fix duplicate entry commit 1d04e0cf0308c35ee7498e6919eab9a11c699d52 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 27 13:55:30 2017 +0530 WL#9499 - Revert last push commit 8361d3c39366d6435a895d5b64dd23615d26fc28 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Jan 26 09:05:52 2017 +0530 WL#9499 - Write to the tablespace.open.* files in a round robin manner. commit 3eaba629ee5b5c26a5f4e80852fee2d446f3013f Merge: 6642c43d735 bdc49070128 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Jan 25 14:03:15 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 6642c43d7353063a455972ab2cbc21220c16be43 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Jan 23 08:34:35 2017 +0530 WL#9499 - Rename fil_ibd_load to fil_ibd_open_for_recovery. commit fdd1897c79b29c9e0a69c1f81d94927da06d9424 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Jan 23 06:28:49 2017 +0530 WL#9499 - Enforce const semantics on Windows for Folder::exists(). commit 9394f4653e10cc5d4ab9b7b0a0b9331fd2666f49 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Jan 23 06:08:26 2017 +0530 WL#9499 - Include <array> commit c1903d28800ae0f7af69552aa669622d552346b4 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Jan 21 07:49:28 2017 +0530 WL#9499 - Improve scan directory handling 1. Convert relative paths to absolute paths 2. Filter out duplicates 3. Check directories at filtering stage commit dfea237f4bef914de4067153cbf382240112c4f4 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 20 11:11:05 2017 +0530 WL#9499 - Code clean up commit c4d51bfdf1c786b1c584c0af1bde080a4f37cc5b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 20 07:37:06 2017 +0530 WL#9499 - Update the open/rename LSN when a file is opened or renamed. Minor code cleanup. Add diagnostic code for debugging. commit 5e1d159b571fe6dbfe13c627a245155b82c4bbfa Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Jan 18 10:59:54 2017 +0530 WL#9499 - Filter out '*' from innodb-scan-directories commit 361f2944f1940f11dfb16dec3589d68335f5e2d8 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Jan 17 21:01:06 2017 +0530 WL#9499 - Code cleanup commit 288d70b356721887423ca3d2d72b42da331216cd Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Jan 17 13:08:08 2017 +0530 WL#9499 - Sync open file map to disk before checkpoint. commit 3c602d1b4089cd262200650f4b33986fc9e26531 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Jan 17 09:18:46 2017 +0530 WL#9499 - Print expected bytes instead of generic message. commit f52e0985b8cbef3891385179d35dc8c7f567146a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Jan 17 09:07:47 2017 +0530 WL#9499 - Print file size instead of generic message. commit b27b1b4b4b70e286e2c669f997b0a669f6d6034c Merge: bbdd1c46f1e c101ec9b557 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Jan 16 13:19:42 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit bbdd1c46f1ef08d956ea249e1f6f4bc93d3464f1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Dec 10 14:41:12 2016 +0530 WL#9499 - Handle the case where the rename has already been done on recovery. commit 141bdb02b1a402e786db5d8b165b781a5d7370a3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Dec 9 11:59:29 2016 +0530 WL#9499 - Instrumentation for debugging commit 34815bb24a1c296d9739e7602dc6ccd53c58c44c Merge: 41ca3732a6f 13a46b4c5ea Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Dec 9 09:44:03 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 41ca3732a6ffd8dfa16762a05df6b5c6b4b550c1 Merge: 7137edb2d2c aba34f1205d Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Dec 8 11:10:08 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 7137edb2d2ca60cdec44600ddf3f5cbde424e7a4 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Dec 8 05:56:06 2016 +0530 WL#9499 - Remove unused code. Move id to name mapping sync before log flush. commit 2a50067e2da2f7819781de6550e55ca14995c7ff Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Dec 7 12:57:41 2016 +0530 WL#9499 - Initialize the PFS file handle commit d478d3492cdb40ec02fda9c043bf29b87bae11ed Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Dec 7 12:05:52 2016 +0530 WL#9499 - Fix whitespace commit 061af748a5e2fde6c1577ccdabfc8a89a9f753de Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Dec 7 12:02:25 2016 +0530 WL#9499 - Set must flush on file close/delete. commit ad76e2b9c7ff757f60c66821e86696d5ab757b18 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Dec 7 11:43:00 2016 +0530 WL#9499 - Code cleanup commit 5a95946d4567dc13cf5a12d6bcdb28b19a02b5bc Merge: 1a0f373eb7f da23d6b8e44 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Dec 7 09:27:45 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 1a0f373eb7f03dc22264f91f3f6ed1fd13825de7 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Dec 6 13:01:41 2016 +0530 WL#9499 - Readd test result file back commit 00afde093183cf850f735940245af111dce82b72 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Dec 6 12:59:49 2016 +0530 WL#9499 - Don't purge the files based on LWM. It won't work. We need to also ensure that there aren't any pages in the buffer pool that belong to a tablespace that is closed. Redo log records can be written for such pages. commit 3199fe255103daa42f0332b7ff9d565eefff7429 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Nov 29 06:41:39 2016 +0530 WL#9499 - Fix typo commit 351cb6c42709405a79021276f310d7b996aadd1b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Nov 29 06:32:02 2016 +0530 WL#9499 - Remove unused function commit 61cc7f6b06c5280d58217b76d7627013ddcd8338 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Nov 29 04:22:31 2016 +0530 WL#9499 - Reduce the compression level. commit 48142e360ff5cfb53d5e416f3d6b92e956237923 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Nov 29 04:21:51 2016 +0530 WL#9499 - Code cleanup commit 4de4f8b31ebdff31a2f7a97a1ad211770d20cdd6 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 14:07:07 2016 +0530 WL#9499 - Revert ef30f0754ab22e614ac22712c348ec5f53e0f09c commit c2f77f44c43c209fad86bba568aa521a2dcbf718 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 12:20:37 2016 +0530 WL#9499 - Code clean up commit 5fccb51769a13cb9a1fc0f29a832d3b0085ad52a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 10:30:18 2016 +0530 WL#9499 - Print a message when checkpoint is disabled. commit ef30f0754ab22e614ac22712c348ec5f53e0f09c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 10:17:27 2016 +0530 WL#9499 - Print progress in 10% units. commit bc0ded332624fa2c8a08ebfc955a03f1a24e069c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 08:13:54 2016 +0530 WL#9499 - Code clean up commit 68ce0208cc073452f5f791faf304971dbac7194a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 07:38:03 2016 +0530 WL#9499 - Scan directories after the mutex code is initialized. commit 21e07c8fd6b8eddf158fb79980443f6c9f6d238e Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 07:12:23 2016 +0530 WL#9499 - Introduce --innodb-checkpoint-disabled (debug) Disable checkpointing if the variable is set. This is to force recovery if the server is killed during a test. commit 7033c9bb5ff7641764f70823a73d747b497a0eed Merge: 11a6e582417 c6af7f512b6 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 04:00:39 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 11a6e582417d45a80a7f19233c2d2191a17936eb Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 03:59:05 2016 +0530 WL#9499 - Code cleanup commit 0a38252e65d607b59b1e9f64d0c8003cdecfecfd Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Nov 25 05:56:42 2016 +0530 WL#9499 - Code clean up commit bcf650cc92b8636f3768dcc3deb33b90b2942134 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Nov 24 08:46:07 2016 +0530 WL#9499 - Change the open files number back to 2. commit 99960af2825529fb45d021a34ba6c072e8caa0dc Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Nov 24 07:13:58 2016 +0530 WL#9499 - Code clean up commit bdb91bc0ba03e01661c50c7175d3fd123d2bcd69 Merge: 61f3e4a5663 a545bdd6e31 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 23 14:04:40 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 61f3e4a5663c375cd44ee766b263281ff7f0601b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 23 14:00:20 2016 +0530 WL#9499 - Code clean up commit 072ab837d0acb4d2ed817110598554acc44ec344 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 23 12:06:53 2016 +0530 WL#9499 - Set the IMPORT page LSN to the flushed to disk LSN commit 7f18661432f3645eb9ce76680c62508ced115fa9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 23 11:28:47 2016 +0530 WL#9499 - Force a checkpoint before the crash commit 83671e9c37fa61adfa5546cabe1463a91ff02241 Merge: cc26d5ae16e d1f811eb6e8 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 23 10:52:16 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit cc26d5ae16e8d723a2519f38bfbc084c13a766c1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Nov 4 18:40:51 2016 +0530 WL#9499 - Fix latch ordering issue Cannot hold the log sys mutex when updating the checkpoint LSN for the open files in fil0fil.cc. commit 4474509addfbdf91b9f0ad890a971299c594e738 Merge: 396755e195c 0856b10cde5 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Nov 4 08:00:05 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 396755e195c1c8d3b2485a58175dcf56ca2cbf3f Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Nov 4 07:58:05 2016 +0530 WL#9499 - Remove newline added earlier commit ab5e60ba491e52f1529c22551a936d771b8b5ef9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Nov 4 07:57:21 2016 +0530 WL#9499 - Revert a change that was added by mistake. commit 38688417713e5ef8cbb65032fa0a547ab0c25c15 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Nov 3 20:44:50 2016 +0530 WL#9499 - Bug fix, using an invalidated iterator. commit 4082d44f6dc4a5605562e37ccad9ac4db69344c6 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Nov 3 11:30:49 2016 +0530 WL#9499 - Fix doxygen comment commit d5ca99e14c010a5349aea6c8cc39493b2073b9c1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 2 15:30:12 2016 +0530 WL#9499 - Code cleanup commit a25f3af10ea2893c634dc961d6f2af05efa103ec Merge: c8c2a7eba4c aa039b73223 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 2 12:52:51 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit c8c2a7eba4cac29ff431588daf2d6def0c0ee5a6 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 2 12:39:32 2016 +0530 WL#9499 - Fix problem with open files acrosss checkpoints. The problem is that we write the open file state before writing the checkpoint. Before this change the closed files were purged from the open state map and the state written to disk. The problem is that the state and mapping of a file opened before the last checkpoint but closed just before we write the state will not be in the mapping on disk. One solution is to check older files and retrieve the mapping from there. The solution chosen is to avoid purging files based on open LSN. If the file was opened before the last checkpoint and closed within the current checkpoint we note the state as CLOSED but write the mapping to disk. It will be purged at the next checkpoint, if it is not opened again in the meantime. Previously an attempt was made to open all files straight after reading the mapping from disk. With this change only the state is loaded into memory the files are opened on demand. Added a check to verify that the tablespace ID from the first page matches the expected value. Fixed tests to use this new tablespace ID verification. commit 5dc929e54b1c259f0f8104739ae1f88e9b860f73 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Oct 25 12:24:40 2016 +0530 WL#9499 - Remove unused code commit 528b95d70dcc05a69a98ee2a76fdfb8dae1e82fe Merge: d9861c40964 75f05cba753 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Oct 25 10:14:40 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit d9861c40964ad8fd45ee2cbbd9d94946110e8a9a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Oct 25 10:04:49 2016 +0530 WL#9499 - Remove last vestiges of MLOG_CHECKPOINT commit 3007ea5af68f40b85bc93098d8385ac96bcbd660 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Oct 25 10:04:49 2016 +0530 WL#9499 - Remove last vestiges of MLOG_CHECKPOINT commit 9592996d2f17c9005262faa75dc1ade9dedbd511 Merge: da512d5591b ac925fc91f6 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Oct 19 10:34:48 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit da512d5591b3379b793d6155089e90791fe94c61 Merge: f42e2ddd7d5 e635dfe28c0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Sep 30 05:20:10 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit f42e2ddd7d50aacae679babe71643af379b5c492 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 22 15:05:28 2016 +0530 WL#9499 - Fix assertion, the values can be ORred. commit 94cd08960bc3801acefd4b4ff4ddb4a3e7d7e81a Merge: 32a68f91330 59af373321b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 22 11:05:21 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 32a68f9133011871bfdaba3c742f6239e0a53061 Merge: 7857f38ebce 15555ae2165 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Sep 21 11:23:32 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 7857f38ebce747135e055bc8a232b649f1aad351 Merge: 427491f63f1 b43bf88cf80 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Sep 19 11:43:17 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 427491f63f107247c36796e66f6224d20968f104 Merge: e7fa4c53dcf 23343687722 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Sep 14 22:40:47 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit e7fa4c53dcf01ea0e101bd8aa817698492995034 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Sep 12 19:22:00 2016 +0530 WL#9499 - max_lsn should work with any number of tablespace open files. commit 09775960da43d36ca64504c1298b9adff3056575 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Sep 12 17:47:29 2016 +0530 WL#9499 - Fix Names hash bug. commit 8430c5b34d23cb1774597707fe583dfac93fb3a8 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Sep 12 09:49:47 2016 +0530 WL#9499 - Use std::unordered_map for name to fil_space_t* hash. commit 5e8275caf9d50dc9ba2f2dadd3b8c0fdeb9c79b6 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Sep 10 07:55:42 2016 +0530 WL#9499 - Reduce an extra fil_sys_t::mutex call by consolidating flush request. commit b29fd1039cf0022cff36a2a2311f464f1f6dbcc5 Merge: 6235ee4902a a150caebee6 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Sep 9 20:44:56 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 6235ee4902addd1a041b33469988bb1ea870dc7a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 8 10:58:00 2016 +0530 WL#9499 - Add debug crash points. commit bf71fbc72b3570f55c8f01dd1a440e9eff761cb6 Merge: ce6d2246a94 d25f38e38d3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 8 09:47:37 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit ce6d2246a94801041da0bfece74b5ffdd9e8f796 Merge: 69adae163de f28c4b9a41e Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Sep 7 11:20:50 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 69adae163de295ef0e910752cf5f8c96b1c1114f Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 1 22:33:29 2016 +0530 WL#9499 - Ensure that the MLOG_FILE_OPEN is always written first to the redo commit 20d5b1ebe6c899e2a0a1d135ad4303490516ce26 Merge: 2d7d25c145f 4f1fb936211 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 1 22:17:58 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 2d7d25c145fae59af744e91fce47fe009c869dad Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 1 10:06:56 2016 +0530 WL#9499 - Add PFS instrumentation for the tablespace open file. commit 4e14a53ad129c1adcb0a3330931878a9f24c15cc Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 1 08:53:57 2016 +0530 WL#9499 - Code cleanup commit a465361e74136662471d2b9c0f307799efbe7269 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 1 08:17:43 2016 +0530 WL#9499 - Move the file write code to a separate method. commit 7a26ba28bf7bf94808c47cfcdb6210c4abde2587 Merge: f49914e5e61 ed389829067 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 1 07:55:25 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit f49914e5e61b846c45f7b177308cd891918619e4 Merge: a4fab040243 16fd507b84a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 31 11:23:42 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit a4fab040243a57fb8ef55f91812604352d8b92c8 Merge: 93019973329 dfee02dc4f5 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 30 10:42:20 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 9301997332923921b93de4946c4824164723050c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 30 09:09:05 2016 +0530 WL#9499 - OSX/Solaris compilers don't like the code. commit 54afa110837a054de24ccda84ddbb4ef2d0b96a1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 29 22:55:19 2016 +0530 WL#9499 - During recovery PFS instrumentation can't be accessed by the user. commit fa5c11e8cf0fbcd1852535faa74b1a04200df2e8 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 29 11:15:07 2016 +0530 WL#9499 - Fil_Open::purge() can be called during recovery too commit d8672a978d62db42f1a12ba152693a3c0d244d83 Merge: d4251d0ed6f 14d2a977ce8 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 29 11:10:32 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit d4251d0ed6f9df5a3acbf97ce7ba8074f126044c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 29 11:07:14 2016 +0530 WL#9499 - Remove a WL#7806 test. The test doesn't really help. It uses very simplistic checks for matching path names. It ignores links (soft and hard) completely. Besides during recovery the path names will work with the original path names. commit fba6656350d0a757b2cc936b017329e7f7ce2d32 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 26 09:51:53 2016 +0530 WL#9499 - Code cleanup commit b522d665d10dca905420cb9119692dffac5024c9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 26 09:20:30 2016 +0530 WL#9499 - Code cleanup commit 3a9aca50a2e193398d626b5f75d238b5aa704e81 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 26 09:19:39 2016 +0530 WL#9499 - Code cleanup commit 5923cd73fc3af904220dea20b033ffcbb7892606 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 26 09:12:44 2016 +0530 WL#9499 - Use custom allocators. commit 8c5acab2641cedae9542b9ad9405032abbcb4618 Merge: 1509c43fcff 3c1319b9901 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 26 08:21:28 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 1509c43fcffb9a849542ef8f3a62e66fda490fc7 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 26 08:08:34 2016 +0530 WL#9499 - Fix bug, should be decrement in_use, not increment. commit 0ad783d5e6e3f99817c8b839fe8521f783577d35 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 26 07:58:49 2016 +0530 WL#9499 - Change fil_node_t::in_use into a reference counter from bool. commit 3974f1ed40bfbadcee04fa5fc2e60a50b89f930c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 22:10:24 2016 +0530 WL#9499 - Remove an assertion that is no longer an invariant. commit d79f12dec7c07698f02ce2e0d0396bcd01bdc683 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 21:52:21 2016 +0530 WL#9499 - Rename fil_node_t::being_extended to in_use. Fix test. commit 1d3368bcf60e2e339d5917251b51201b94dd63a3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 10:57:38 2016 +0530 WL#9499 - Fix log info messages during recovery/apply. commit d3811bd67248d3ae014f0e6475b1b49524f5c35b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 10:40:01 2016 +0530 WL#9499 - Don't print % processed if number of records to apply is small. commit 55af670ff81170da51fbcae6832c3d6191176602 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 10:21:31 2016 +0530 WL#9499 - Update test with new log message to check. commit 60c3d6c074ae230ab594015ed2f99e2eefcf97e0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 10:09:04 2016 +0530 WL#9499 - Record test, because of new config parameter innodb_scan_directories. commit bd79ff06406c86758a29a28dbe5e7d494a0aa51f Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 10:05:38 2016 +0530 WL#9499 - Remove WL#7142 artefact. commit bcb753b72b2a01ba35d952cbab45e420929f2b33 Merge: e30834ecc60 27d64e974ec Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 09:26:25 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit e30834ecc60522fca8d7b156a66c0765867dea6e Author: Sunny Bains <Sunny.Bains@Oracle.Com> Date: Thu Aug 25 05:54:26 2016 +0200 WL#9499 - Destroy the mutex in the Fil_Open destructor. commit a396d4714bdc46626e07d22b6871be153b627bd3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 09:17:06 2016 +0530 WL#9499 - Code cleanup commit 282ad76aaeaeb6d91bbf81fcd3315f455d9f572c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 09:12:33 2016 +0530 WL#9499 - Protect changes to Fil_Open state with a mutex. commit d338cb9468c126ee7e3eeae3bf37b0fc966bbf53 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 07:37:49 2016 +0530 WL#9499 - Fix division by zero. commit 845324f19ba0a84538b87c072f6d99514533aa25 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 00:28:45 2016 +0530 WL#9499 - Check for UNDO tablespace names during directory scan. commit 30ed1d19389e174605fdfc0b7d55fdf544933712 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 00:26:08 2016 +0530 WL#9499 - Conform to new naming guidelines. commit 4cf67b352d4c8131064f832ec884db6166d50e71 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 24 22:04:43 2016 +0530 WL#9499 - Add test for the configuration variable. Add a version number to the file. commit ad9c5bca0eba1d7bf8e29cd8ebd7062aae089f90 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 24 21:01:56 2016 +0530 WL#9499 - Code clean up commit a9e848158909645ead86759ed7ab64789d18a0cb Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 24 15:26:02 2016 +0530 WL#9499 - Print redo apply progress per tablespace commit 8cf86cf1567cabc1b03b2e8f2ec1167f3c38ce6c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 24 13:04:55 2016 +0530 WL#9499 - Print tablespace name and % completed per tablespace commit 35ccdf32d1d048400012a80422072fb41ceb1bbd Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 24 10:35:26 2016 +0530 WL#9499 - Code cleanup commit c978fb2e18a4e0b4a93b44fc3c75234bcdb3f1f0 Merge: 8bde4aeb01c 0832bf660e3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 23 22:25:57 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 8bde4aeb01c3bcdecfb884e566f0ae28797a5b36 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 23 13:21:10 2016 +0530 WL#9499 - Fix Windows build issue commit 7f502be746cbe0de1a0c3a992217691b167c2fa9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 23 12:12:08 2016 +0530 WL#9499 - Fix Windows build issue commit 247bcfda6744a67cc4e53d61658ca87d206b0660 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 23 10:57:44 2016 +0530 WL#9499 - Code cleanup commit 04509641b086a1b0b2c591573f95957f8a2f1cd0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 23 10:50:55 2016 +0530 WL#9499 - Pass the filename directly to the callback commit 3efa317825886a3ae5d4a3c771fc0770a2c5a3ff Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 23 09:39:18 2016 +0530 WL#9499 - Include tchar.h on Windows commit 55b814d2c2ebdea7c16a6d87a7251ca3f6e83775 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 23 07:47:39 2016 +0530 WL#9499 - Code cleanup commit e4d203688273b562691c912cb9c7024709757490 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 22 22:02:02 2016 +0530 WL#9499 - Code cleanup commit 31433b21dcabba0a7705eb8b1879beb1d8c36517 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 22 21:37:41 2016 +0530 WL#9499 - Move Dir_Walker::walk implementation to os0file.cc commit 11f9016477274c5420782e365b26762067e7b99b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 22 18:42:28 2016 +0530 WL#9499 - Fix Windows version of Dir_Walk. commit f32b88e0b1ef99f220165897db745b1dba12f86b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 22 13:48:03 2016 +0530 WL#9499 - Code cleanup commit b91ad88d96942f1e05443001b7d1c2f6818ff6d3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sun Aug 21 14:32:25 2016 +0530 WL#9499 - Initialize active ID to filename set from scanned set. commit 4a09f677ae7095aac9b07f7beccba8e60c50b065 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sun Aug 21 13:09:05 2016 +0530 WL#9499 Scan specified directories to build the ID to name mapping. commit c13b791497e322e6a5cf64ba85da52ee2cf05d45 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sun Aug 21 10:46:53 2016 +0530 WL#9499 - Fix check for incomplete read. commit 8a849e0288309cec7228612a62fc2bfd71b595e9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sun Aug 21 10:45:04 2016 +0530 WL#9499 - Fix Windows build errors commit 4fc24b654da26e8411fc45bc96f73ff70dc73e43 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sun Aug 21 09:12:12 2016 +0530 WL#9499 - Need to include string for Windows std::wstring. commit bcb5c2b23f54776e0faaafba2472bb2555aee561 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sun Aug 21 09:02:59 2016 +0530 WL#9499 - On POSIX fopen() mode should be "w+b". commit b3c7def1931867633d9b8bb601317e99a06c31a9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Aug 20 21:23:57 2016 +0530 WL#9499 - Open file in binary mode for writing. Otherwise on Windows newline (0x0A) is translated to CRLF (0x0D0A). commit 0739399cb8a384c07b4b3540b73ade482d083f4a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Aug 20 20:27:11 2016 +0530 WL#9499 - More error messages during mapping file read. Add a directory walker. commit 1322de0af2235bc94630f7cb22e4ead1588775ee Merge: f266b100e2f 8fdbb2f5f1a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Aug 20 13:47:39 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit f266b100e2f23e775e5e04de8d26301d7f2f1072 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Aug 20 13:18:07 2016 +0530 WL#9499 - Flush the tablespace to mapping file buffers before fsync() commit 7c92536686033c2c70e7df831d569c1cd6f871f0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 19 20:32:26 2016 +0530 WL#9499 - Remove unused member variable m_lsn. commit 994c4766ef712e7da668788d0bdd648cbd764982 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 19 20:25:05 2016 +0530 WL#9499 - Don't crash on uncompress failure. Try the next file if any. commit a419a190cdf734e3beca08d339c6c0d307ddff5f Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 19 20:21:49 2016 +0530 WL#9499 - Print an error code on uncopress failure commit 77cb39000e098ba4d06873723423521e6889a7d0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 19 20:12:11 2016 +0530 WL#9499 - Code cleanup commit afc632e0ef768542a96d4d15a897cb1780825d2e Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 19 11:34:44 2016 +0530 WL#9499 - Compress the tablespace open state file. commit 568c9e758084f671007789f7b5d1c996f5df39a0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 19 09:12:39 2016 +0530 WL#9499 - Free the buffer on return commit 0f468cf8e50b9605a9e6c4e93c818b0f876c3c29 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 19 08:42:38 2016 +0530 WL#9499 - Remove protobuf commit e789eb175d2da90cee2d46e30b91e623546f9e96 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 18 09:59:37 2016 +0530 WL#9499 - Get native HANDLE commit 6014c5853c09fee75256f9de6a86a5fc0ea18508 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 17 14:50:37 2016 +0530 WL#9499 - Use _fileno() on Windows commit 61e6b4d7c033d48b775b22c4ad6d6b4858cd4cd7 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 17 14:37:01 2016 +0530 WL#9499 - Use buffered IO for open state file writes commit 98f40b10c15e49d9a5e14689cdb34758bdf85a05 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 17 12:36:06 2016 +0530 WL#9499 - Don't complain about corrupted encrypted pages during dblwr recovery commit c674939cbdeaa4a9996960de0fa4e74954ad6f76 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 17 09:21:06 2016 +0530 WL#9499 - Protobuf internal code doesn't compile on Solaris/SPARC. Workaround is to set -Wno-unused-but-set-parameter for generated sources too. commit ff68ba26aeaa27070ceeaa8db4472a9ce4daa3c9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 17 08:33:03 2016 +0530 WL#9499 - Protobuf internal code doesn't compile on Solaris/SPARC. Workaround is to set -Wno-unused-but-set-parameter for fil/fil0fil.cc commit 445985ca11810c528567a4222cad4a7f0ce49e81 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 16 19:44:29 2016 +0530 WL#9499 - Read/Write open state files using absolute paths commit b4d91d3799d94eb2677be6512752ec27dfa5e0f1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 16 14:37:40 2016 +0530 WL#9499 - Fix doxygen errors commit 8be2de3b3077c0d0f26340c9c80f7ae2247eefbc Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 16 11:20:24 2016 +0530 WL#9499 - Record test, MLOG_FILE_NAME doesn't exist anymore commit 54e64dc4a97f5bf4b5606cea1841dfc1f04fe536 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 16 10:41:01 2016 +0530 WL#9499 - Add protobuf library to the embedded library. commit 83b0fb38283e285e1e963dd2c7c144d70b130338 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 15 21:17:13 2016 +0530 WL#9499 - Improved DBLWR and System table checks and code cleanup 1. Remove dead code 2. If some pages from the dblwr buffer were not recovered during recovery because of a missing tablespace id -> name mapping. Check the tablespace ID against the Tablespaces meta-data after recovery. 3. Track open of the system tablespace. Check if the name of any file has changed. We don't rename the system tablespace. commit 5ebfba1e083f6a1d2b70946b91a02d32733c7e87 Merge: 7a1b724c04b 0c27bf6fc44 Author: Sunny Bains <Sunny.Bains@Oracle.Com> Date: Mon Aug 15 11:40:07 2016 +0200 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 7a1b724c04bf9d2e4aba0e58fb95dc8fc75d55c1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 15 15:07:47 2016 +0530 WL#9499 - Free deferred page memory commit 4de5d3b03670e4d2e8ebce89e8b3f104968eb3e1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 15 11:45:25 2016 +0530 WL#9499 - Remove unrelated changes commit c9e83914232480501b139c36ef616798a2ac1d05 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 15 11:28:19 2016 +0530 WL#9499 - Code clean up commit 6ecdb21d63ef6dd5a6aff997720178ff6510f0d5 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Aug 13 14:31:52 2016 +0530 WL#9499 - Don't exit if no open state files found
bjornmu
pushed a commit
that referenced
this pull request
Apr 10, 2017
WL#9499 - InnoDB: Replace MLOG_FILE_NAME with MLOG_FILE_OPEN Fix the performance issues introduced by WL#7142 and its followup WL#7806. 1. Revert WL#7142 and WL#7806 changes, keep some of the tests to test the behaviour that was introduced by the two WLs 2. Change the tests that rely on the above two WLs behaviour 3. Use std::unordered_map instead of the homebrew hash table for the recovery code and space ID to name mapping 4. Split the redo log hash table key from <space, page> -> apply records to <space> -> apply records 5. Addition of two new system files (tablespaces.open.1 and tablespaces.open.2) - Rename MLOG_FILE_NAME to MLOG_FILE_OPEN - Remove the two pass recovery code, make it a single pass - Track file open, close and rename - Write and fsync the tracking state to disk on checkpoint and rename. Uses the files in #5 in a round robin fashion Introduce --innodb-scan-directories="path1;...pathN". Scan the paths for .ibd files to open during recovery. This option can be used if the files in #5 above are corrupt or missing. You should not use this if you move files around. While they can be recovered the data dictionary will not be updated and it will cause issues during runtimg. rb#13686 Approved by Satya and Shaohua commit af3dc1301a768c01b971d14ad07549d6ef470fe6 Merge: ac37b926e6a 4d81939d63a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Mar 3 11:16:10 2017 +1100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit ac37b926e6ad85b6c4e3d7880b905d082f1674be Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Mar 2 23:12:24 2017 +1100 WL#9499 - Fix test commit 2b05df7ffa592da9b19cec7ba31c04795a1cdfc0 Merge: 3c79f3aee51 71b3bbff153 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Mar 2 20:34:10 2017 +1100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 3c79f3aee51858b1859f4e8711883a85867c417a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Mar 2 18:16:35 2017 +1100 WL#9499 - Add an mtr tst commit 2c7496246c0e95e25c62ceb6fe5c1875f693ffac Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Mar 2 17:34:50 2017 +1100 WL#9499 Fix bugs 1. Fix a memory leak, call mem_heap_free() instead of mem_heap_empty() 2. Use a reference instead of copy by value during dblwr traversing 3. Use absolute path names for tablespace.open.* files commit fbf87161f3c93979e5abe424fbe0678fdc32159e Merge: 517cf3a7bf0 c2b504664ed Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 28 10:39:32 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 517cf3a7bf0490c3168fed020e3096ec500e0a85 Merge: 4786f93ad8a 5e974fd0ea4 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 28 10:24:25 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 Additional change is to call recv_recovery_from_checkpoint_finish() on abort. Instead of recv_sys_free(). commit 4786f93ad8a25506a2a1f6e9d1207256a34b3665 Merge: 5dc35e05b5f 275245cce32 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 24 10:43:27 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 5dc35e05b5f2da95c36e191350e9c7087be72194 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 24 10:42:40 2017 +0100 WL#9499 Free recv resources on abort commit bed693070395923ededba736f1c7102b8eb21e95 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 23 18:34:50 2017 +0100 WL#9499 - Test cannot be run with Valgrind commit c5d423228ddf58b17866ff8be289dba4a19205d2 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 23 17:30:39 2017 +0100 WL#9499 - Remove const commit ac42498506b4dffb22e74282d0ce037fe3acf977 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 23 17:00:30 2017 +0100 WL#9499 - Clean up code commit bc48077cab9c3ccb1b2797c365fe0940c269a785 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 23 16:01:09 2017 +0100 WL#9499 - Revert unrelated change commit 8b2bb0f6d6254b3cfde168d41356400009db12f4 Merge: eb9b294ba78 153a79ff35d Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 23 11:43:38 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit eb9b294ba786800f9e397cd87f01c4f4976edddb Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 23 11:29:56 2017 +0100 WL#9499 - Remove debug message commit abae66ad591854ee67a5013f3209be2f82e4d9bf Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Feb 22 17:04:10 2017 +0100 WL#9499 - Test only works in debug mode. commit 71ebea70391ef295bed49ee06a8b872f15ecfe97 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Feb 22 12:18:08 2017 +0100 WL#9499 - Update to mtr test from Matthias commit f108a5b92fc476c70c49e4937496b32d9d5d031f Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Feb 22 12:13:04 2017 +0100 WL#9499 - Increment redo log version number commit 4439c83a4d3edd6e511706ccf9936770ed29e593 Merge: fa459186b90 05923bef41d Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Feb 22 09:54:18 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit fa459186b90d7a5ddcba7e3c6c7a9871d0dce793 Merge: 14ac4f81d84 b401f17e70d Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 20:02:40 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 14ac4f81d84bec23a7be210504ca0be93619bf11 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 16:44:57 2017 +0100 WL#9499 - Update from Matthias commit ed3191a9fc951151fd7d758a527e7ab81530b3f0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 15:16:11 2017 +0100 WL#9499 - Revert change pushed by nmistake commit 8c0edb998867bf557f6c86410552d6d123892000 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 15:11:21 2017 +0100 WL#9499 - Don't redo log MLOG_FILE_OPEN until startup is complete commit 3b503ea1d3f89c0f7e3677d3b56f249c7ebff506 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 11:37:04 2017 +0100 WL#9499 - Add tests commit 47af7bd6eec10eac55633d7c4cf54afe607f3649 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 21 11:00:30 2017 +0100 WL#9499 - Fix compile problem commit a8b9a8c6ccb15e2be5ce5db5fa0bce24b86d2c87 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Feb 20 16:51:05 2017 +0100 WL#9499 Remove dead code commit b1dba474486ad10b61029b2ef4e6a91ea67dcea8 Merge: 3cf9e9eb3e8 a4271026201 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Feb 20 16:49:38 2017 +0100 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 3cf9e9eb3e86e4c8f4d44cb79d3f7c456ea7d671 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Feb 18 07:53:43 2017 +0530 WL#9499 - Remove suggestion to use force recovery --innodb-scan-directories is a safer option for the user. Force recovery should be the absolute last resort. commit 4104c4bfd606270650aac95a9a724e106fe835c5 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 17 09:00:47 2017 +0530 WL#9499 - Fix error message User can try --innodb-scan-directories="..." if the tablespaces.open.* files are corrupt or unreadable. commit 19ab12e54923b070a0b4ffc76b9fe97ac54ad3cc Merge: 50b3a95e446 f0f51c410bd Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 17 05:22:57 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 Replace readdir_r() with readdir(), compiler complains that readdir_r() is deprecated and recommends replacing it with readdir(). commit 50b3a95e446b91853fc35716b63bdc173221d01f Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 16 18:55:33 2017 +0530 WL#9499 - Fix test commit 45d202ae0fa35aa3aced55cc1933aeefa33837de Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 16 18:32:56 2017 +0530 WL#9499 - Remove redundant test commit 48fe691b97735e5144e9d439a0adbf721b2dc554 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 16 18:18:11 2017 +0530 WL#9499 - Add DEBUG sync points commit d986f4e04a9eb6c2c76d61cd4afe9541d77b34bd Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 16 18:03:50 2017 +0530 WL#9499 - Remove PB2 control file commit 5af7dca9c305bd366ad2b5c3de26a870e913af4a Merge: 74022f50a5f 23a0e71aa3d Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Feb 15 09:05:05 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 74022f50a5ff7cb2801c0712601f8a467b409a28 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Feb 14 10:31:00 2017 +0530 WL#9499 - Fix test commit f6188175ee62040ac6a844a5c54787839dff6155 Merge: 24c867aab14 19ca7c6a5d3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Feb 13 12:37:22 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 24c867aab14556188503aac495f60d89ba3d26ec Merge: cb64b8c7f9f a4de6ab549e Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Feb 4 06:41:23 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit cb64b8c7f9fa9ca05033a9d6f6193cf141746310 Merge: 17c6c41da0a 88dbc4f4511 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 3 19:48:12 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 17c6c41da0afe3fc78843b75fa02578dab29a0ca Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 3 19:46:34 2017 +0530 WL#9499 - Add a test for scan directories. The use case is where the location of the files is not changed but the tablespaces.open.* files are missing. commit 79cc52f1bd82dc65932eacbc9aa14a47e4196572 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 3 18:01:50 2017 +0530 WL#9499 - If open fails space will be null. commit 22ea5f31ecc1fa0db8abd57d4a46578e5f891c9a Merge: f5b69930f95 853a52cf5a1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Feb 3 12:14:41 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit f5b69930f95192ddfdb3ca170024a5c7e3416076 Merge: 40a4c32f2c4 b18872ed4a3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Feb 2 13:54:24 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 40a4c32f2c42b584539a14cbc1ba0d4f0fe7d3c9 Merge: b1d1923d37c d071c0e7a0b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Feb 1 13:48:33 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit b1d1923d37c3d23a9f701a61b458b7e4cbd3c90c Merge: 2fc6c51edab 4ae71705a01 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Jan 31 10:44:59 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 2fc6c51edabc3de4537eceb0b82cd62063ada3a9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Jan 30 16:36:46 2017 +0530 WL#9499 - Empty the tablespace.open.* files on normal shutdown. commit 738acb7d554f97455c8dba512282c7c2e6ba9f3a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Jan 30 12:47:21 2017 +0530 WL#9499 - Fix formatting commit 28f8579a6f729af354e0c50098d16fbfa17cc253 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Jan 30 12:25:30 2017 +0530 WL#9499 - Bug fix The DELETED state was set on a copy of the data structure commit dd5bf8ea81b7023fa1980c01d89219a690160b04 Merge: 7eeab0a994d 9c7808e1054 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 27 21:14:22 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 7eeab0a994db4cf0c2e080f27ff916c778ec65f8 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 27 20:50:59 2017 +0530 WL#9499 - Add tests. commit ddfaf047f43587ee807599b0b80b6d01a3579e8c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 27 20:48:06 2017 +0530 WL#9499 - Create tablespace bug fix If there is a checkpoint after the create tablespace is logged and the server crashes after writing the page initialization to the redo log. Then we neither have the mapping in the redo log nor in the tablespace.open.* files. We must update the tablespace mapping before writing the CREATE redo log entry so that the state is written out to disk on checkpoint. commit 29546d3ebea2991e5b86b33edddb551c924bf1db Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 27 17:55:28 2017 +0530 WL#9499 - Remove dead code from WL#7142 commit eced7a3e4437ee4bb24a12290f561437ddb15611 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 27 14:51:09 2017 +0530 WL#9499 - Use the tablespace.open.* files in a round robin fashion. commit d8eff788eea1ad5e6c7786b9d64963524f2b2c81 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 27 13:56:12 2017 +0530 WL#9499 - Fix duplicate entry commit 1d04e0cf0308c35ee7498e6919eab9a11c699d52 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 27 13:55:30 2017 +0530 WL#9499 - Revert last push commit 8361d3c39366d6435a895d5b64dd23615d26fc28 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Jan 26 09:05:52 2017 +0530 WL#9499 - Write to the tablespace.open.* files in a round robin manner. commit 3eaba629ee5b5c26a5f4e80852fee2d446f3013f Merge: 6642c43d735 bdc49070128 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Jan 25 14:03:15 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 6642c43d7353063a455972ab2cbc21220c16be43 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Jan 23 08:34:35 2017 +0530 WL#9499 - Rename fil_ibd_load to fil_ibd_open_for_recovery. commit fdd1897c79b29c9e0a69c1f81d94927da06d9424 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Jan 23 06:28:49 2017 +0530 WL#9499 - Enforce const semantics on Windows for Folder::exists(). commit 9394f4653e10cc5d4ab9b7b0a0b9331fd2666f49 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Jan 23 06:08:26 2017 +0530 WL#9499 - Include <array> commit c1903d28800ae0f7af69552aa669622d552346b4 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Jan 21 07:49:28 2017 +0530 WL#9499 - Improve scan directory handling 1. Convert relative paths to absolute paths 2. Filter out duplicates 3. Check directories at filtering stage commit dfea237f4bef914de4067153cbf382240112c4f4 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 20 11:11:05 2017 +0530 WL#9499 - Code clean up commit c4d51bfdf1c786b1c584c0af1bde080a4f37cc5b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Jan 20 07:37:06 2017 +0530 WL#9499 - Update the open/rename LSN when a file is opened or renamed. Minor code cleanup. Add diagnostic code for debugging. commit 5e1d159b571fe6dbfe13c627a245155b82c4bbfa Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Jan 18 10:59:54 2017 +0530 WL#9499 - Filter out '*' from innodb-scan-directories commit 361f2944f1940f11dfb16dec3589d68335f5e2d8 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Jan 17 21:01:06 2017 +0530 WL#9499 - Code cleanup commit 288d70b356721887423ca3d2d72b42da331216cd Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Jan 17 13:08:08 2017 +0530 WL#9499 - Sync open file map to disk before checkpoint. commit 3c602d1b4089cd262200650f4b33986fc9e26531 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Jan 17 09:18:46 2017 +0530 WL#9499 - Print expected bytes instead of generic message. commit f52e0985b8cbef3891385179d35dc8c7f567146a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Jan 17 09:07:47 2017 +0530 WL#9499 - Print file size instead of generic message. commit b27b1b4b4b70e286e2c669f997b0a669f6d6034c Merge: bbdd1c46f1e c101ec9b557 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Jan 16 13:19:42 2017 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit bbdd1c46f1ef08d956ea249e1f6f4bc93d3464f1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Dec 10 14:41:12 2016 +0530 WL#9499 - Handle the case where the rename has already been done on recovery. commit 141bdb02b1a402e786db5d8b165b781a5d7370a3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Dec 9 11:59:29 2016 +0530 WL#9499 - Instrumentation for debugging commit 34815bb24a1c296d9739e7602dc6ccd53c58c44c Merge: 41ca3732a6f 13a46b4c5ea Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Dec 9 09:44:03 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 41ca3732a6ffd8dfa16762a05df6b5c6b4b550c1 Merge: 7137edb2d2c aba34f1205d Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Dec 8 11:10:08 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 7137edb2d2ca60cdec44600ddf3f5cbde424e7a4 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Dec 8 05:56:06 2016 +0530 WL#9499 - Remove unused code. Move id to name mapping sync before log flush. commit 2a50067e2da2f7819781de6550e55ca14995c7ff Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Dec 7 12:57:41 2016 +0530 WL#9499 - Initialize the PFS file handle commit d478d3492cdb40ec02fda9c043bf29b87bae11ed Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Dec 7 12:05:52 2016 +0530 WL#9499 - Fix whitespace commit 061af748a5e2fde6c1577ccdabfc8a89a9f753de Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Dec 7 12:02:25 2016 +0530 WL#9499 - Set must flush on file close/delete. commit ad76e2b9c7ff757f60c66821e86696d5ab757b18 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Dec 7 11:43:00 2016 +0530 WL#9499 - Code cleanup commit 5a95946d4567dc13cf5a12d6bcdb28b19a02b5bc Merge: 1a0f373eb7f da23d6b8e44 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Dec 7 09:27:45 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 1a0f373eb7f03dc22264f91f3f6ed1fd13825de7 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Dec 6 13:01:41 2016 +0530 WL#9499 - Readd test result file back commit 00afde093183cf850f735940245af111dce82b72 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Dec 6 12:59:49 2016 +0530 WL#9499 - Don't purge the files based on LWM. It won't work. We need to also ensure that there aren't any pages in the buffer pool that belong to a tablespace that is closed. Redo log records can be written for such pages. commit 3199fe255103daa42f0332b7ff9d565eefff7429 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Nov 29 06:41:39 2016 +0530 WL#9499 - Fix typo commit 351cb6c42709405a79021276f310d7b996aadd1b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Nov 29 06:32:02 2016 +0530 WL#9499 - Remove unused function commit 61cc7f6b06c5280d58217b76d7627013ddcd8338 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Nov 29 04:22:31 2016 +0530 WL#9499 - Reduce the compression level. commit 48142e360ff5cfb53d5e416f3d6b92e956237923 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Nov 29 04:21:51 2016 +0530 WL#9499 - Code cleanup commit 4de4f8b31ebdff31a2f7a97a1ad211770d20cdd6 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 14:07:07 2016 +0530 WL#9499 - Revert ef30f0754ab22e614ac22712c348ec5f53e0f09c commit c2f77f44c43c209fad86bba568aa521a2dcbf718 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 12:20:37 2016 +0530 WL#9499 - Code clean up commit 5fccb51769a13cb9a1fc0f29a832d3b0085ad52a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 10:30:18 2016 +0530 WL#9499 - Print a message when checkpoint is disabled. commit ef30f0754ab22e614ac22712c348ec5f53e0f09c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 10:17:27 2016 +0530 WL#9499 - Print progress in 10% units. commit bc0ded332624fa2c8a08ebfc955a03f1a24e069c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 08:13:54 2016 +0530 WL#9499 - Code clean up commit 68ce0208cc073452f5f791faf304971dbac7194a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 07:38:03 2016 +0530 WL#9499 - Scan directories after the mutex code is initialized. commit 21e07c8fd6b8eddf158fb79980443f6c9f6d238e Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 07:12:23 2016 +0530 WL#9499 - Introduce --innodb-checkpoint-disabled (debug) Disable checkpointing if the variable is set. This is to force recovery if the server is killed during a test. commit 7033c9bb5ff7641764f70823a73d747b497a0eed Merge: 11a6e582417 c6af7f512b6 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 04:00:39 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 11a6e582417d45a80a7f19233c2d2191a17936eb Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Nov 28 03:59:05 2016 +0530 WL#9499 - Code cleanup commit 0a38252e65d607b59b1e9f64d0c8003cdecfecfd Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Nov 25 05:56:42 2016 +0530 WL#9499 - Code clean up commit bcf650cc92b8636f3768dcc3deb33b90b2942134 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Nov 24 08:46:07 2016 +0530 WL#9499 - Change the open files number back to 2. commit 99960af2825529fb45d021a34ba6c072e8caa0dc Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Nov 24 07:13:58 2016 +0530 WL#9499 - Code clean up commit bdb91bc0ba03e01661c50c7175d3fd123d2bcd69 Merge: 61f3e4a5663 a545bdd6e31 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 23 14:04:40 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 61f3e4a5663c375cd44ee766b263281ff7f0601b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 23 14:00:20 2016 +0530 WL#9499 - Code clean up commit 072ab837d0acb4d2ed817110598554acc44ec344 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 23 12:06:53 2016 +0530 WL#9499 - Set the IMPORT page LSN to the flushed to disk LSN commit 7f18661432f3645eb9ce76680c62508ced115fa9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 23 11:28:47 2016 +0530 WL#9499 - Force a checkpoint before the crash commit 83671e9c37fa61adfa5546cabe1463a91ff02241 Merge: cc26d5ae16e d1f811eb6e8 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 23 10:52:16 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit cc26d5ae16e8d723a2519f38bfbc084c13a766c1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Nov 4 18:40:51 2016 +0530 WL#9499 - Fix latch ordering issue Cannot hold the log sys mutex when updating the checkpoint LSN for the open files in fil0fil.cc. commit 4474509addfbdf91b9f0ad890a971299c594e738 Merge: 396755e195c 0856b10cde5 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Nov 4 08:00:05 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 396755e195c1c8d3b2485a58175dcf56ca2cbf3f Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Nov 4 07:58:05 2016 +0530 WL#9499 - Remove newline added earlier commit ab5e60ba491e52f1529c22551a936d771b8b5ef9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Nov 4 07:57:21 2016 +0530 WL#9499 - Revert a change that was added by mistake. commit 38688417713e5ef8cbb65032fa0a547ab0c25c15 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Nov 3 20:44:50 2016 +0530 WL#9499 - Bug fix, using an invalidated iterator. commit 4082d44f6dc4a5605562e37ccad9ac4db69344c6 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Nov 3 11:30:49 2016 +0530 WL#9499 - Fix doxygen comment commit d5ca99e14c010a5349aea6c8cc39493b2073b9c1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 2 15:30:12 2016 +0530 WL#9499 - Code cleanup commit a25f3af10ea2893c634dc961d6f2af05efa103ec Merge: c8c2a7eba4c aa039b73223 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 2 12:52:51 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit c8c2a7eba4cac29ff431588daf2d6def0c0ee5a6 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Nov 2 12:39:32 2016 +0530 WL#9499 - Fix problem with open files acrosss checkpoints. The problem is that we write the open file state before writing the checkpoint. Before this change the closed files were purged from the open state map and the state written to disk. The problem is that the state and mapping of a file opened before the last checkpoint but closed just before we write the state will not be in the mapping on disk. One solution is to check older files and retrieve the mapping from there. The solution chosen is to avoid purging files based on open LSN. If the file was opened before the last checkpoint and closed within the current checkpoint we note the state as CLOSED but write the mapping to disk. It will be purged at the next checkpoint, if it is not opened again in the meantime. Previously an attempt was made to open all files straight after reading the mapping from disk. With this change only the state is loaded into memory the files are opened on demand. Added a check to verify that the tablespace ID from the first page matches the expected value. Fixed tests to use this new tablespace ID verification. commit 5dc929e54b1c259f0f8104739ae1f88e9b860f73 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Oct 25 12:24:40 2016 +0530 WL#9499 - Remove unused code commit 528b95d70dcc05a69a98ee2a76fdfb8dae1e82fe Merge: d9861c40964 75f05cba753 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Oct 25 10:14:40 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit d9861c40964ad8fd45ee2cbbd9d94946110e8a9a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Oct 25 10:04:49 2016 +0530 WL#9499 - Remove last vestiges of MLOG_CHECKPOINT commit 3007ea5af68f40b85bc93098d8385ac96bcbd660 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Oct 25 10:04:49 2016 +0530 WL#9499 - Remove last vestiges of MLOG_CHECKPOINT commit 9592996d2f17c9005262faa75dc1ade9dedbd511 Merge: da512d5591b ac925fc91f6 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Oct 19 10:34:48 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit da512d5591b3379b793d6155089e90791fe94c61 Merge: f42e2ddd7d5 e635dfe28c0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Sep 30 05:20:10 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit f42e2ddd7d50aacae679babe71643af379b5c492 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 22 15:05:28 2016 +0530 WL#9499 - Fix assertion, the values can be ORred. commit 94cd08960bc3801acefd4b4ff4ddb4a3e7d7e81a Merge: 32a68f91330 59af373321b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 22 11:05:21 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 32a68f9133011871bfdaba3c742f6239e0a53061 Merge: 7857f38ebce 15555ae2165 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Sep 21 11:23:32 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 7857f38ebce747135e055bc8a232b649f1aad351 Merge: 427491f63f1 b43bf88cf80 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Sep 19 11:43:17 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 427491f63f107247c36796e66f6224d20968f104 Merge: e7fa4c53dcf 23343687722 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Sep 14 22:40:47 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit e7fa4c53dcf01ea0e101bd8aa817698492995034 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Sep 12 19:22:00 2016 +0530 WL#9499 - max_lsn should work with any number of tablespace open files. commit 09775960da43d36ca64504c1298b9adff3056575 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Sep 12 17:47:29 2016 +0530 WL#9499 - Fix Names hash bug. commit 8430c5b34d23cb1774597707fe583dfac93fb3a8 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Sep 12 09:49:47 2016 +0530 WL#9499 - Use std::unordered_map for name to fil_space_t* hash. commit 5e8275caf9d50dc9ba2f2dadd3b8c0fdeb9c79b6 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Sep 10 07:55:42 2016 +0530 WL#9499 - Reduce an extra fil_sys_t::mutex call by consolidating flush request. commit b29fd1039cf0022cff36a2a2311f464f1f6dbcc5 Merge: 6235ee4902a a150caebee6 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Sep 9 20:44:56 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 6235ee4902addd1a041b33469988bb1ea870dc7a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 8 10:58:00 2016 +0530 WL#9499 - Add debug crash points. commit bf71fbc72b3570f55c8f01dd1a440e9eff761cb6 Merge: ce6d2246a94 d25f38e38d3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 8 09:47:37 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit ce6d2246a94801041da0bfece74b5ffdd9e8f796 Merge: 69adae163de f28c4b9a41e Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Sep 7 11:20:50 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 69adae163de295ef0e910752cf5f8c96b1c1114f Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 1 22:33:29 2016 +0530 WL#9499 - Ensure that the MLOG_FILE_OPEN is always written first to the redo commit 20d5b1ebe6c899e2a0a1d135ad4303490516ce26 Merge: 2d7d25c145f 4f1fb936211 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 1 22:17:58 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 2d7d25c145fae59af744e91fce47fe009c869dad Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 1 10:06:56 2016 +0530 WL#9499 - Add PFS instrumentation for the tablespace open file. commit 4e14a53ad129c1adcb0a3330931878a9f24c15cc Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 1 08:53:57 2016 +0530 WL#9499 - Code cleanup commit a465361e74136662471d2b9c0f307799efbe7269 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 1 08:17:43 2016 +0530 WL#9499 - Move the file write code to a separate method. commit 7a26ba28bf7bf94808c47cfcdb6210c4abde2587 Merge: f49914e5e61 ed389829067 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Sep 1 07:55:25 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit f49914e5e61b846c45f7b177308cd891918619e4 Merge: a4fab040243 16fd507b84a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 31 11:23:42 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit a4fab040243a57fb8ef55f91812604352d8b92c8 Merge: 93019973329 dfee02dc4f5 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 30 10:42:20 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 9301997332923921b93de4946c4824164723050c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 30 09:09:05 2016 +0530 WL#9499 - OSX/Solaris compilers don't like the code. commit 54afa110837a054de24ccda84ddbb4ef2d0b96a1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 29 22:55:19 2016 +0530 WL#9499 - During recovery PFS instrumentation can't be accessed by the user. commit fa5c11e8cf0fbcd1852535faa74b1a04200df2e8 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 29 11:15:07 2016 +0530 WL#9499 - Fil_Open::purge() can be called during recovery too commit d8672a978d62db42f1a12ba152693a3c0d244d83 Merge: d4251d0ed6f 14d2a977ce8 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 29 11:10:32 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit d4251d0ed6f9df5a3acbf97ce7ba8074f126044c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 29 11:07:14 2016 +0530 WL#9499 - Remove a WL#7806 test. The test doesn't really help. It uses very simplistic checks for matching path names. It ignores links (soft and hard) completely. Besides during recovery the path names will work with the original path names. commit fba6656350d0a757b2cc936b017329e7f7ce2d32 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 26 09:51:53 2016 +0530 WL#9499 - Code cleanup commit b522d665d10dca905420cb9119692dffac5024c9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 26 09:20:30 2016 +0530 WL#9499 - Code cleanup commit 3a9aca50a2e193398d626b5f75d238b5aa704e81 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 26 09:19:39 2016 +0530 WL#9499 - Code cleanup commit 5923cd73fc3af904220dea20b033ffcbb7892606 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 26 09:12:44 2016 +0530 WL#9499 - Use custom allocators. commit 8c5acab2641cedae9542b9ad9405032abbcb4618 Merge: 1509c43fcff 3c1319b9901 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 26 08:21:28 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 1509c43fcffb9a849542ef8f3a62e66fda490fc7 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 26 08:08:34 2016 +0530 WL#9499 - Fix bug, should be decrement in_use, not increment. commit 0ad783d5e6e3f99817c8b839fe8521f783577d35 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 26 07:58:49 2016 +0530 WL#9499 - Change fil_node_t::in_use into a reference counter from bool. commit 3974f1ed40bfbadcee04fa5fc2e60a50b89f930c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 22:10:24 2016 +0530 WL#9499 - Remove an assertion that is no longer an invariant. commit d79f12dec7c07698f02ce2e0d0396bcd01bdc683 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 21:52:21 2016 +0530 WL#9499 - Rename fil_node_t::being_extended to in_use. Fix test. commit 1d3368bcf60e2e339d5917251b51201b94dd63a3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 10:57:38 2016 +0530 WL#9499 - Fix log info messages during recovery/apply. commit d3811bd67248d3ae014f0e6475b1b49524f5c35b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 10:40:01 2016 +0530 WL#9499 - Don't print % processed if number of records to apply is small. commit 55af670ff81170da51fbcae6832c3d6191176602 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 10:21:31 2016 +0530 WL#9499 - Update test with new log message to check. commit 60c3d6c074ae230ab594015ed2f99e2eefcf97e0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 10:09:04 2016 +0530 WL#9499 - Record test, because of new config parameter innodb_scan_directories. commit bd79ff06406c86758a29a28dbe5e7d494a0aa51f Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 10:05:38 2016 +0530 WL#9499 - Remove WL#7142 artefact. commit bcb753b72b2a01ba35d952cbab45e420929f2b33 Merge: e30834ecc60 27d64e974ec Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 09:26:25 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit e30834ecc60522fca8d7b156a66c0765867dea6e Author: Sunny Bains <Sunny.Bains@Oracle.Com> Date: Thu Aug 25 05:54:26 2016 +0200 WL#9499 - Destroy the mutex in the Fil_Open destructor. commit a396d4714bdc46626e07d22b6871be153b627bd3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 09:17:06 2016 +0530 WL#9499 - Code cleanup commit 282ad76aaeaeb6d91bbf81fcd3315f455d9f572c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 09:12:33 2016 +0530 WL#9499 - Protect changes to Fil_Open state with a mutex. commit d338cb9468c126ee7e3eeae3bf37b0fc966bbf53 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 07:37:49 2016 +0530 WL#9499 - Fix division by zero. commit 845324f19ba0a84538b87c072f6d99514533aa25 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 00:28:45 2016 +0530 WL#9499 - Check for UNDO tablespace names during directory scan. commit 30ed1d19389e174605fdfc0b7d55fdf544933712 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 25 00:26:08 2016 +0530 WL#9499 - Conform to new naming guidelines. commit 4cf67b352d4c8131064f832ec884db6166d50e71 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 24 22:04:43 2016 +0530 WL#9499 - Add test for the configuration variable. Add a version number to the file. commit ad9c5bca0eba1d7bf8e29cd8ebd7062aae089f90 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 24 21:01:56 2016 +0530 WL#9499 - Code clean up commit a9e848158909645ead86759ed7ab64789d18a0cb Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 24 15:26:02 2016 +0530 WL#9499 - Print redo apply progress per tablespace commit 8cf86cf1567cabc1b03b2e8f2ec1167f3c38ce6c Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 24 13:04:55 2016 +0530 WL#9499 - Print tablespace name and % completed per tablespace commit 35ccdf32d1d048400012a80422072fb41ceb1bbd Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 24 10:35:26 2016 +0530 WL#9499 - Code cleanup commit c978fb2e18a4e0b4a93b44fc3c75234bcdb3f1f0 Merge: 8bde4aeb01c 0832bf660e3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 23 22:25:57 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 8bde4aeb01c3bcdecfb884e566f0ae28797a5b36 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 23 13:21:10 2016 +0530 WL#9499 - Fix Windows build issue commit 7f502be746cbe0de1a0c3a992217691b167c2fa9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 23 12:12:08 2016 +0530 WL#9499 - Fix Windows build issue commit 247bcfda6744a67cc4e53d61658ca87d206b0660 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 23 10:57:44 2016 +0530 WL#9499 - Code cleanup commit 04509641b086a1b0b2c591573f95957f8a2f1cd0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 23 10:50:55 2016 +0530 WL#9499 - Pass the filename directly to the callback commit 3efa317825886a3ae5d4a3c771fc0770a2c5a3ff Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 23 09:39:18 2016 +0530 WL#9499 - Include tchar.h on Windows commit 55b814d2c2ebdea7c16a6d87a7251ca3f6e83775 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 23 07:47:39 2016 +0530 WL#9499 - Code cleanup commit e4d203688273b562691c912cb9c7024709757490 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 22 22:02:02 2016 +0530 WL#9499 - Code cleanup commit 31433b21dcabba0a7705eb8b1879beb1d8c36517 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 22 21:37:41 2016 +0530 WL#9499 - Move Dir_Walker::walk implementation to os0file.cc commit 11f9016477274c5420782e365b26762067e7b99b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 22 18:42:28 2016 +0530 WL#9499 - Fix Windows version of Dir_Walk. commit f32b88e0b1ef99f220165897db745b1dba12f86b Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 22 13:48:03 2016 +0530 WL#9499 - Code cleanup commit b91ad88d96942f1e05443001b7d1c2f6818ff6d3 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sun Aug 21 14:32:25 2016 +0530 WL#9499 - Initialize active ID to filename set from scanned set. commit 4a09f677ae7095aac9b07f7beccba8e60c50b065 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sun Aug 21 13:09:05 2016 +0530 WL#9499 Scan specified directories to build the ID to name mapping. commit c13b791497e322e6a5cf64ba85da52ee2cf05d45 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sun Aug 21 10:46:53 2016 +0530 WL#9499 - Fix check for incomplete read. commit 8a849e0288309cec7228612a62fc2bfd71b595e9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sun Aug 21 10:45:04 2016 +0530 WL#9499 - Fix Windows build errors commit 4fc24b654da26e8411fc45bc96f73ff70dc73e43 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sun Aug 21 09:12:12 2016 +0530 WL#9499 - Need to include string for Windows std::wstring. commit bcb5c2b23f54776e0faaafba2472bb2555aee561 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sun Aug 21 09:02:59 2016 +0530 WL#9499 - On POSIX fopen() mode should be "w+b". commit b3c7def1931867633d9b8bb601317e99a06c31a9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Aug 20 21:23:57 2016 +0530 WL#9499 - Open file in binary mode for writing. Otherwise on Windows newline (0x0A) is translated to CRLF (0x0D0A). commit 0739399cb8a384c07b4b3540b73ade482d083f4a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Aug 20 20:27:11 2016 +0530 WL#9499 - More error messages during mapping file read. Add a directory walker. commit 1322de0af2235bc94630f7cb22e4ead1588775ee Merge: f266b100e2f 8fdbb2f5f1a Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Aug 20 13:47:39 2016 +0530 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit f266b100e2f23e775e5e04de8d26301d7f2f1072 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Aug 20 13:18:07 2016 +0530 WL#9499 - Flush the tablespace to mapping file buffers before fsync() commit 7c92536686033c2c70e7df831d569c1cd6f871f0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 19 20:32:26 2016 +0530 WL#9499 - Remove unused member variable m_lsn. commit 994c4766ef712e7da668788d0bdd648cbd764982 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 19 20:25:05 2016 +0530 WL#9499 - Don't crash on uncompress failure. Try the next file if any. commit a419a190cdf734e3beca08d339c6c0d307ddff5f Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 19 20:21:49 2016 +0530 WL#9499 - Print an error code on uncopress failure commit 77cb39000e098ba4d06873723423521e6889a7d0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 19 20:12:11 2016 +0530 WL#9499 - Code cleanup commit afc632e0ef768542a96d4d15a897cb1780825d2e Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 19 11:34:44 2016 +0530 WL#9499 - Compress the tablespace open state file. commit 568c9e758084f671007789f7b5d1c996f5df39a0 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 19 09:12:39 2016 +0530 WL#9499 - Free the buffer on return commit 0f468cf8e50b9605a9e6c4e93c818b0f876c3c29 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Fri Aug 19 08:42:38 2016 +0530 WL#9499 - Remove protobuf commit e789eb175d2da90cee2d46e30b91e623546f9e96 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Thu Aug 18 09:59:37 2016 +0530 WL#9499 - Get native HANDLE commit 6014c5853c09fee75256f9de6a86a5fc0ea18508 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 17 14:50:37 2016 +0530 WL#9499 - Use _fileno() on Windows commit 61e6b4d7c033d48b775b22c4ad6d6b4858cd4cd7 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 17 14:37:01 2016 +0530 WL#9499 - Use buffered IO for open state file writes commit 98f40b10c15e49d9a5e14689cdb34758bdf85a05 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 17 12:36:06 2016 +0530 WL#9499 - Don't complain about corrupted encrypted pages during dblwr recovery commit c674939cbdeaa4a9996960de0fa4e74954ad6f76 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 17 09:21:06 2016 +0530 WL#9499 - Protobuf internal code doesn't compile on Solaris/SPARC. Workaround is to set -Wno-unused-but-set-parameter for generated sources too. commit ff68ba26aeaa27070ceeaa8db4472a9ce4daa3c9 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Wed Aug 17 08:33:03 2016 +0530 WL#9499 - Protobuf internal code doesn't compile on Solaris/SPARC. Workaround is to set -Wno-unused-but-set-parameter for fil/fil0fil.cc commit 445985ca11810c528567a4222cad4a7f0ce49e81 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 16 19:44:29 2016 +0530 WL#9499 - Read/Write open state files using absolute paths commit b4d91d3799d94eb2677be6512752ec27dfa5e0f1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 16 14:37:40 2016 +0530 WL#9499 - Fix doxygen errors commit 8be2de3b3077c0d0f26340c9c80f7ae2247eefbc Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 16 11:20:24 2016 +0530 WL#9499 - Record test, MLOG_FILE_NAME doesn't exist anymore commit 54e64dc4a97f5bf4b5606cea1841dfc1f04fe536 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Tue Aug 16 10:41:01 2016 +0530 WL#9499 - Add protobuf library to the embedded library. commit 83b0fb38283e285e1e963dd2c7c144d70b130338 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 15 21:17:13 2016 +0530 WL#9499 - Improved DBLWR and System table checks and code cleanup 1. Remove dead code 2. If some pages from the dblwr buffer were not recovered during recovery because of a missing tablespace id -> name mapping. Check the tablespace ID against the Tablespaces meta-data after recovery. 3. Track open of the system tablespace. Check if the name of any file has changed. We don't rename the system tablespace. commit 5ebfba1e083f6a1d2b70946b91a02d32733c7e87 Merge: 7a1b724c04b 0c27bf6fc44 Author: Sunny Bains <Sunny.Bains@Oracle.Com> Date: Mon Aug 15 11:40:07 2016 +0200 Merge branch 'mysql-trunk' into mysql-trunk-wl9499 commit 7a1b724c04bf9d2e4aba0e58fb95dc8fc75d55c1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 15 15:07:47 2016 +0530 WL#9499 - Free deferred page memory commit 4de5d3b03670e4d2e8ebce89e8b3f104968eb3e1 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 15 11:45:25 2016 +0530 WL#9499 - Remove unrelated changes commit c9e83914232480501b139c36ef616798a2ac1d05 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Mon Aug 15 11:28:19 2016 +0530 WL#9499 - Code clean up commit 6ecdb21d63ef6dd5a6aff997720178ff6510f0d5 Author: Sunny Bains <Sunny.Bains@oracle.com> Date: Sat Aug 13 14:31:52 2016 +0530 WL#9499 - Don't exit if no open state files found
pobrzut
pushed a commit
that referenced
this pull request
May 8, 2017
pobrzut
pushed a commit
that referenced
this pull request
May 8, 2017
gipulla
pushed a commit
that referenced
this pull request
Jul 17, 2017
THREE BYTES ON X86 Analysis: ========= The macro uint3korr reads 4 bytes of data instead of 3 on on x86 machines. Multiple definitions were created for this macro for optimization in WIN32. The idea was to optimize reading of 3 byte ints by reading an ordinary int and masking away the unused byte. However this is an undefined behavior. It will be an issue unless users are aware of allocating an extra byte for using this macro. Fix: ==== Removing the definition which reads 4 bytes of data. The only definition of this macro would now read just 3 bytes of data thus prohibiting the usage of an extra byte. Note: ===== This is a backport of Patches #5 and #6 for Bug#17922198.
akopytov
pushed a commit
to akopytov/mysql-server
that referenced
this pull request
Aug 25, 2017
In WL-included builds ASAN run witnessed missed ~Query_log_event invocation. The destruct-or was not called due to the WL's changes in the error propagation that specifically affect LC MTS. The failure is exposed in particular by rpl_trigger as the following stack: #0 0x9ecd98 in __interceptor_malloc (/export/home/pb2/test/sb_2-22611026-1489061390.32/mysql-commercial-8.0.1-dmr-linux-x86_64-asan/bin/mysqld+0x9ecd98) mysql#1 0x2b1a245 in my_raw_malloc(unsigned long, int) obj/mysys/../../mysqlcom-pro-8.0.1-dmr/mysys/my_malloc.cc:209:12 mysql#2 0x2b1a245 in my_malloc obj/mysys/../../mysqlcom-pro-8.0.1-dmr/mysys/my_malloc.cc:72 mysql#3 0x2940590 in Query_log_event::Query_log_event(char const*, unsigned int, binary_log::Format_description_event const*, binary_log::Log_event_type) obj/sql/../../mysqlcom-pro-8.0.1-dmr/sql/log_event.cc:4343:46 mysql#4 0x293d235 in Log_event::read_log_event(char const*, unsigned int, char const**, Format_description_log_event const*, bool) obj/sql/../../mysqlcom-pro-8.0.1-dmr/sql/log_event.cc:1686:17 mysql#5 0x293b96f in Log_event::read_log_event() mysql#6 0x2a2a1c9 in next_event(Relay_log_info*) Previously before the WL Mts_submode_logical_clock::wait_for_workers_to_finish() had not returned any error even when Coordinator thread is killed. The WL patch needed to refine such behavior, but at doing so it also had to attend log_event.cc::schedule_next_event() to register an error to follow an existing pattern. While my_error() does not take place the killed Coordinator continued scheduling, ineffectively though - no Worker gets engaged (legal case of deferred scheduling), and without noticing its killed status up to a point when it resets the event pointer in apply_event_and_update_pos(): *ptr_ev= NULL; // announcing the event is passed to w-worker The reset was intended for an assigned Worker to perform the event destruction or by Coordinator itself when the event is deferred. As neither is the current case the event gets unattended for its termination. In contrast in the pre-WL sources the killed Coordinator does find a Worker. However such Worker could be already down (errored out and exited), in which case apply_event_and_update_pos() reasonably returns an error and executes delete ev in exec_relay_log_event() error branch. **Fixed** with deploying my_error() call in log_event.cc::schedule_next_event() error branch which fits to the existing pattern. THD::is_error() has been always checked by Coordinator before any attempt to reset *ptr_ev= NULL. In the errored case Coordinator does not reset and destroys the event itself in the exec_relay_log_event() error branch pretty similarly to how the pre-WL sources do. Tested against rpl_trigger and rpl suites to pass. Approved on rb#15667.
bjornmu
pushed a commit
that referenced
this pull request
Sep 21, 2017
Patch #5: Json_wrapper::seek() currently always converts the wrapper to a DOM if the path contains an ellipsis. This is because paths with ellipses may require duplicate elimination, which is harder to do using the JSON binary interface. It is not the case that all paths with ellipses need duplicate elimination. It is only needed if the path contains more than one ellipsis, or if it contains an ellipsis followed by an auto-wrapping array path leg. This patch makes Json_wrapper::seek() only convert the wrapper to a DOM if the path requires duplicate elimination. It also makes Json_wrapper::seek() forward to Json_dom::seek() in the case where the Json_wrapper is wrapping a DOM, to avoid the overhead of the Json_wrapper interface when navigating through the DOM. Microbenchmarks (64-bit, Intel Core i7-4770 3.4 GHz, GCC 6.3): BM_JsonDomSearchEllipsis 21764 ns/iter [ +3.9%] BM_JsonDomSearchEllipsis_OnlyOne 15486 ns/iter [ +4.4%] BM_JsonDomSearchKey 126 ns/iter [ +2.4%] BM_JsonBinarySearchEllipsis 70979 ns/iter [ +225.2%] BM_JsonBinarySearchEllipsis_OnlyOne 97 ns/iter [+229941.2%] BM_JsonBinarySearchKey 79 ns/iter [ +8.9%] Change-Id: I0f024cf8a2da46d8d7e33560c4599f619efe09bd
bjornmu
pushed a commit
that referenced
this pull request
Jan 23, 2018
…TABLE_UPGRADE_GUARD
To repeat: cmake -DWITH_ASAN=1 -DWITH_ASAN_SCOPE=1
./mtr --mem --sanitize main.dd_upgrade_error
A few dd tests fail with:
==26861==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7000063bf5e8 at pc 0x00010d4dbe8b bp 0x7000063bda40 sp 0x7000063bda38
READ of size 8 at 0x7000063bf5e8 thread T2
#0 0x10d4dbe8a in Prealloced_array<st_plugin_int**, 16ul>::empty() const prealloced_array.h:186
#1 0x10d406a8b in lex_end(LEX*) sql_lex.cc:560
#2 0x10dae4b6d in dd::upgrade::Table_upgrade_guard::~Table_upgrade_guard() (mysqld:x86_64+0x100f87b6d)
#3 0x10dadc557 in dd::upgrade::migrate_table_to_dd(THD*, std::__1::basic_string<char, std::__1::char_traits<char>, Stateless_allocator<char, dd::String_type_alloc, My_free_functor> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, Stateless_allocator<char, dd::String_type_alloc, My_free_functor> > const&, bool) (mysqld:x86_64+0x100f7f557)
#4 0x10dad7e85 in dd::upgrade::migrate_plugin_table_to_dd(THD*) (mysqld:x86_64+0x100f7ae85)
#5 0x10daec6a1 in dd::upgrade::do_pre_checks_and_initialize_dd(THD*) upgrade.cc:1216
#6 0x10cd0a5c0 in bootstrap::handle_bootstrap(void*) bootstrap.cc:336
Change-Id: I265ec6dd97ee8076aaf03763840c0cdf9e20325b
Fix: increase lifetime of 'LEX lex;' which is used by 'table_guard'
surbhat1595
pushed a commit
that referenced
this pull request
Apr 25, 2019
…E TO A SERVER
Problem
========================================================================
Running the GCS tests with ASAN seldomly reports a user-after-free of
the server reference that the acceptor_learner_task uses.
Here is an excerpt of ASAN's output:
==43936==ERROR: AddressSanitizer: heap-use-after-free on address 0x63100021c840 at pc 0x000000530ff8 bp 0x7fc0427e8530 sp 0x7fc0427e8520
WRITE of size 8 at 0x63100021c840 thread T3
#0 0x530ff7 in server_detected /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:962
#1 0x533814 in buffered_read_bytes /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:1249
#2 0x5481af in buffered_read_msg /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:1399
#3 0x51e171 in acceptor_learner_task /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:4690
#4 0x562357 in task_loop /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/task.c:1140
#5 0x5003b2 in xcom_taskmain2 /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:1324
#6 0x6a278a in Gcs_xcom_proxy_impl::xcom_init(unsigned short, node_address*) /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_proxy.cc:164
#7 0x59b3c1 in xcom_taskmain_startup /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_control_interface.cc:107
#8 0x7fc04a2e4dd4 in start_thread (/lib64/libpthread.so.0+0x7dd4)
#9 0x7fc047ff2bfc in __clone (/lib64/libc.so.6+0xfebfc)
0x63100021c840 is located 64 bytes inside of 65688-byte region [0x63100021c800,0x63100022c898)
freed by thread T3 here:
#0 0x7fc04a5d7508 in __interceptor_free (/lib64/libasan.so.4+0xde508)
#1 0x52cf86 in freesrv /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:836
#2 0x52ea78 in srv_unref /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:868
#3 0x524c30 in reply_handler_task /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:4914
#4 0x562357 in task_loop /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/task.c:1140
#5 0x5003b2 in xcom_taskmain2 /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:1324
#6 0x6a278a in Gcs_xcom_proxy_impl::xcom_init(unsigned short, node_address*) /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_proxy.cc:164
#7 0x59b3c1 in xcom_taskmain_startup /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_control_interface.cc:107
#8 0x7fc04a2e4dd4 in start_thread (/lib64/libpthread.so.0+0x7dd4)
previously allocated by thread T3 here:
#0 0x7fc04a5d7a88 in __interceptor_calloc (/lib64/libasan.so.4+0xdea88)
#1 0x543604 in mksrv /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:721
#2 0x543b4c in addsrv /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:755
#3 0x54af61 in update_servers /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_transport.c:1747
#4 0x501082 in site_install_action /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:1572
#5 0x55447c in import_config /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/site_def.c:486
#6 0x506dfc in handle_x_snapshot /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:5257
#7 0x50c444 in xcom_fsm /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:5325
#8 0x516c36 in dispatch_op /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:4510
#9 0x521997 in acceptor_learner_task /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:4772
#10 0x562357 in task_loop /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/task.c:1140
#11 0x5003b2 in xcom_taskmain2 /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/xcom/xcom_base.c:1324
#12 0x6a278a in Gcs_xcom_proxy_impl::xcom_init(unsigned short, node_address*) /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_proxy.cc:164
#13 0x59b3c1 in xcom_taskmain_startup /home/tvale/mysql/plugin/group_replication/libmysqlgcs/src/bindings/xcom/gcs_xcom_control_interface.cc:107
#14 0x7fc04a2e4dd4 in start_thread (/lib64/libpthread.so.0+0x7dd4)
Analysis
========================================================================
The server structure is reference counted by the associated sender_task
and reply_handler_task.
When they finish, they unreference the server, which leads to its memory
being freed.
However, the acceptor_learner_task keeps a "naked" reference to the
server structure.
Under the right ordering of operations, i.e. the sender_task and
reply_handler_task terminating after the acceptor_learner_task acquires,
but before it uses, the reference to the server structure, leads to the
acceptor_learner_task accessing the server structure after it has been
freed.
Solution
========================================================================
Let the acceptor_learner_task also reference count the server structure
so it is not freed while still in use.
Reviewed-by: André Negrão <andre.negrao@oracle.com>
Reviewed-by: Venkatesh Venugopal <venkatesh.venugopal@oracle.com>
RB: 21209
bjornmu
pushed a commit
that referenced
this pull request
Jan 18, 2021
…TH VS 2019 [#5] [noclose] storage\ndb\src\common\transporter\TransporterRegistry.cpp(410,33): warning C4150: deletion of pointer to incomplete type 'SHM_Transporter'; no destructor called Change-Id: I8d928d6ce403a9f08d2328bc1e8f934c51892c1e
bjornmu
pushed a commit
that referenced
this pull request
Jan 18, 2021
…ARRAY.H:72
Bug #31978439: ASSERTION FAILURE IN BOUNDS_CHECKED_ARRAY::OPERATOR[] AT SQL/SQL_ARRAY.H
The field list could come into an inconsistent state after temporary
tables were created, and it would cause out-of-bounds indexing later.
The chain of events is fairly complicated, and is best illustrated with
a query:
SELECT * FROM (
SELECT PI()
FROM t1 AS table1, t1 AS table2
ORDER BY PI(), table1.a
) AS d1;
1. The outer SELECT list gets an Item_view_ref to PI(), as a visible item.
2. The derived table is merged.
2.1. ORDER BY is pulled up into the outer query block (this is a MySQL
extension).
3. ORDER BY is setup for the outer query block.
3.1. setup_order() looks for PI() in the SELECT list, doesn't find it
(since it is hidden behind the Item_view_ref), adds PI() as a
hidden item to make sure it is pulled through temporary tables.
4. A temporary table is created due to join + ORDER BY together.
4.1. Item_view_ref is peeled away when asking for temporary field.
The PI() now wants to exist in the field list both as hidden
and non-hidden, which is not possible since there's only one
hidden flag per Item.
5. Since we now have a field list where not all hidden items are before all
visible items, the indexing into the slice arrays breaks when trying to
fix up ROLLUP items at the very end, and we go out of bounds.
We can't remove the Item_view_ref when merging the derived table, as it
contains metadata information we'd like to keep. So instead, we change
setup_order() to look for such items wrapped into Item_view_ref as a
last resort, so that it doesn't add them twice.
Step #5 (added when we rewrote the streaming aggregation) is what actually
uncovers the bug, but this has been wrong since the field list was
changed from being a linked list, moving the hidden flag into the Item.
Change-Id: I518520d05402464da71a0d062d6fdb0ad6fd4b65
16yuki0702
pushed a commit
to 16yuki0702/mysql-server
that referenced
this pull request
Jun 27, 2021
…ARRAY.H:72
Bug #31978439: ASSERTION FAILURE IN BOUNDS_CHECKED_ARRAY::OPERATOR[] AT SQL/SQL_ARRAY.H
The field list could come into an inconsistent state after temporary
tables were created, and it would cause out-of-bounds indexing later.
The chain of events is fairly complicated, and is best illustrated with
a query:
SELECT * FROM (
SELECT PI()
FROM t1 AS table1, t1 AS table2
ORDER BY PI(), table1.a
) AS d1;
1. The outer SELECT list gets an Item_view_ref to PI(), as a visible item.
2. The derived table is merged.
2.1. ORDER BY is pulled up into the outer query block (this is a MySQL
extension).
3. ORDER BY is setup for the outer query block.
3.1. setup_order() looks for PI() in the SELECT list, doesn't find it
(since it is hidden behind the Item_view_ref), adds PI() as a
hidden item to make sure it is pulled through temporary tables.
4. A temporary table is created due to join + ORDER BY together.
4.1. Item_view_ref is peeled away when asking for temporary field.
The PI() now wants to exist in the field list both as hidden
and non-hidden, which is not possible since there's only one
hidden flag per Item.
5. Since we now have a field list where not all hidden items are before all
visible items, the indexing into the slice arrays breaks when trying to
fix up ROLLUP items at the very end, and we go out of bounds.
We can't remove the Item_view_ref when merging the derived table, as it
contains metadata information we'd like to keep. So instead, we change
setup_order() to look for such items wrapped into Item_view_ref as a
last resort, so that it doesn't add them twice.
Step mysql#5 (added when we rewrote the streaming aggregation) is what actually
uncovers the bug, but this has been wrong since the field list was
changed from being a linked list, moving the hidden flag into the Item.
Change-Id: I518520d05402464da71a0d062d6fdb0ad6fd4b65
piki
pushed a commit
to planetscale/mysql-server
that referenced
this pull request
Apr 27, 2022
Docs and build fixes
nawazn
pushed a commit
that referenced
this pull request
Jul 26, 2022
Use DBUG_TRACE in changed functions Change-Id: Idfd60e26dc6ec45f61cb43fea6dfe0ec21c615e1
nawazn
pushed a commit
that referenced
this pull request
Jul 26, 2022
-- Patch #1: Persist secondary load information -- Problem: We need a way of knowing which tables were loaded to HeatWave after MySQL restarts due to a crash or a planned shutdown. Solution: Add a new "secondary_load" flag to the `options` column of mysql.tables. This flag is toggled after a successful secondary load or unload. The information about this flag is also reflected in INFORMATION_SCHEMA.TABLES.CREATE_OPTIONS. -- Patch #2 -- The second patch in this worklog triggers the table reload from InnoDB after MySQL restart. The recovery framework recognizes that the system restarted by checking whether tables are present in the Global State. If there are no tables present, the framework will access the Data Dictionary and find which tables were loaded before the restart. This patch introduces the "Data Dictionary Worker" - a MySQL service recovery worker whose task is to query the INFORMATION_SCHEMA.TABLES table from a separate thread and find all tables whose secondary_load flag is set to 1. All tables that were found in the Data Dictionary will be appended to the list of tables that have to be reloaded by the framework from InnoDB. If an error occurs during restart recovery we will not mark the recovery as failed. This is done because the types of failures that can occur when the tables are reloaded after a restart are less critical compared to previously existing recovery situations. Additionally, this code will soon have to be adapted for the next worklog in this area so we are proceeding with the simplest solution that makes sense. A Global Context variable m_globalStateEmpty is added which indicates whether the Global State should be recovered from an external source. -- Patch #3 -- This patch adds the "rapid_reload_on_restart" system variable. This variable is used to control whether tables should be reloaded after a restart of mysqld or the HeatWave plugin. This variable is persistable (i.e., SET PERSIST RAPID_RELOAD_ON_RESTART = TRUE/FALSE). The default value of this variable is set to false. The variable can be modified in OFF, IDLE, and SUSPENDED states. -- Patch #4 -- This patch refactors the recovery code by removing all recovery-related code from ha_rpd.cc and moving it to separate files: - ha_rpd_session_factory.h/cc: These files contain the MySQLAdminSessionFactory class, which is used to create admin sessions in separate threads that can be used to issue SQL queries. - ha_rpd_recovery.h/cc: These files contain the MySQLServiceRecoveryWorker, MySQLServiceRecoveryJob and ObjectStoreRecoveryJob classes which were previously defined in ha_rpd.cc. This file also contains a function that creates the RecoveryWorkerFactory object. This object is passed to the constructor of the Recovery Framework and is used to communicate with the other section of the code located in rpdrecoveryfwk.h/cc. This patch also renames rpdrecvryfwk to rpdrecoveryfwk for better readability. The include relationship between the files is shown on the following diagram: rpdrecoveryfwk.h◄──────────────rpdrecoveryfwk.cc ▲ ▲ │ │ │ │ │ └──────────────────────────┐ │ │ ha_rpd_recovery.h◄─────────────ha_rpd_recovery.cc──┐ ▲ │ │ │ │ │ │ │ │ │ ▼ │ ha_rpd.cc───────────────────────►ha_rpd.h │ ▲ │ │ │ ┌───────────────────────────────┘ │ │ ▼ ha_rpd_session_factory.cc──────►ha_rpd_session_factory.h Other changes: - In agreement with Control Plane, the external Global State is now invalidated during recovery framework startup if: 1) Recovery framework recognizes that it should load the Global State from an external source AND, 2) rapid_reload_on_restart is set to OFF. - Addressed review comments for Patch #3, rapid_reload_on_restart is now also settable while plugin is ON. - Provide a single entry point for processing external Global State before starting the recovery framework loop. - Change when the Data Dictionary is read. Now we will no longer wait for the HeatWave nodes to connect before querying the Data Dictionary. We will query it when the recovery framework starts, before accepting any actions in the recovery loop. - Change the reload flow by inserting fake global state entries for tables that need to be reloaded instead of manually adding them to a list of tables scheduled for reload. This method will be used for the next phase where we will recover from Object Storage so both recovery methods will now follow the same flow. - Update secondary_load_dd_flag added in Patch #1. - Increase timeout in wait_for_server_bootup to 300s to account for long MySQL version upgrades. - Add reload_on_restart and reload_on_restart_dbg tests to the rapid suite. - Add PLUGIN_VAR_PERSIST_AS_READ_ONLY flag to "rapid_net_orma_port" and "rapid_reload_on_restart" definitions, enabling their initialization from persisted values along with "rapid_bootstrap" when it is persisted as ON. - Fix numerous clang-tidy warnings in recovery code. - Prevent suspended_basic and secondary_load_dd_flag tests to run on ASAN builds due to an existing issue when reinstalling the RAPID plugin. -- Bug#33752387 -- Problem: A shutdown of MySQL causes a crash in queries fired by DD worker. Solution: Prevent MySQL from killing DD worker's queries by instantiating a DD_kill_immunizer before the queries are fired. -- Patch #5 -- Problem: A table can be loaded before the DD Worker queries the Data Dictionary. This means that table will be wrongly processed as part of the external global state. Solution: If the table is present in the current in-memory global state we will not consider it as part of the external global state and we will not process it by the recovery framework. -- Bug#34197659 -- Problem: If a table reload after restart causes OOM the cluster will go into RECOVERYFAILED state. Solution: Recognize when the tables are being reloaded after restart and do not move the cluster into RECOVERYFAILED. In that case only the current reload will fail and the reload for other tables will be attempted. Change-Id: Ic0c2a763bc338ea1ae6a7121ff3d55b456271bf0
pull bot
pushed a commit
to zhuizhu-95/mysql-server
that referenced
this pull request
Apr 28, 2023
# This is the 1st commit message:
WL#15280: HEATWAVE SUPPORT FOR MDS HA
Problem Statement
-----------------
Currently customers cannot enable heatwave analytics service to their
HA DBSystem or enable HA if they are using Heatwave enabled DBSystem.
In this change, we attempt to remove this limitation and provide
failover support of heatwave in an HA enabled DBSystem.
High Level Overview
-------------------
To support heatwave with HA, we extended the existing feature of auto-
reloading of tables to heatwave on MySQL server restart (WL-14396). To
provide seamless failover functionality to tables loaded to heatwave,
each node in the HA cluster (group replication) must have the latest
view of tables which are currently loaded to heatwave cluster attached
to the primary, i.e., the secondary_load flag should be in-sync always.
To achieve this, we made following changes -
1. replicate secondary load/unload DDL statements to all the active
secondary nodes by writing the DDL into the binlog, and
2. Control how secondary load/unload is executed when heatwave cluster
is not attached to node executing the command
Implementation Details
----------------------
Current implementation depends on two key assumptions -
1. All MDS DBSystems will have RAPID plugin installed.
2. No non-MDS system will have the RAPID plugin installed.
Based on these assumptions, we made certain changes w.r.t. how server
handles execution of secondary load/unload statements.
1. If secondary load/unload command is executed from a mysql client
session on a system without RAPID plugin installed (i.e., non-MDS),
instead of an error, a warning message will be shown to the user,
and the DDL is allowed to commit.
2. If secondary load/unload command is executed from a replication
connection on an MDS system without heatwave cluster attached,
instead of throwing an error, the DDL is allowed to commit.
3. If no error is thrown from secondary engine, then the DDL will
update the secondary_load metadata and write a binlog entry.
Writing to binlog implies that all the consumer of binlog now need to
handle this DDL gracefully. This has an adverse effect on Point-in-time
Recovery. If the PITR backup is taken from a DBSystem with heatwave, it
may contain traces of secondary load/unload statements in its binlog.
If such a backup is used to restore a new DBSystem, it will cause failure
while trying to execute statements from its binlog because
a) DBSystem will not heatwave cluster attached at this time, and
b) Statements from binlog are executed from standard mysql client
connection, thus making them indistinguishable from user executed
command.
Customers will be prevented (by control plane) from using PITR functionality
on a heatwave enabled DBSystem until there is a solution for this.
Testing
-------
This commit changes the behavior of secondary load/unload statements, so it
- adjusts existing tests' expectations, and
- adds a new test validating new DDL behavior under different scenarios
Change-Id: Ief7e9b3d4878748b832c366da02892917dc47d83
# This is the commit message #2:
WL#15280: HEATWAVE SUPPORT FOR MDS HA (PITR SUPPORT)
Problem
-------
A PITR backup taken from a heatwave enabled system could have traces
of secondary load or unload statements in binlog. When such a backup
is used to restore another system, it can cause failure because of
following two reasons:
1. Currently, even if the target system is heatwave enabled, heatwave
cluster is attached only after PITR restore phase completes.
2. When entries from binlogs are applied, a standard mysql client
connection is used. This makes it indistinguishable from other user
session.
Since secondary load (or unload) statements are meant to throw error
when they are executed by user in the absence of a healthy heatwave
cluster, PITR restore workflow will fail if binlogs from the backup
have any secondary load (or unload) statements in them.
Solution
--------
To avoid PITR failure, we are introducing a new system variable
rapid_enable_delayed_secondary_ops. It controls how load or unload
commands are to be processed by rapid plugin.
- When turned ON, the plugin silently skips the secondary engine
operation (load/unload) and returns success to the caller. This
allows secondary load (or unload) statements to be executed by the
server in the absence of any heatwave cluster.
- When turned OFF, it follows the existing behavior.
- The default value is OFF.
- The value can only be changed when rapid_bootstrap is IDLE or OFF.
- This variable cannot be persisted.
In PITR workflow, Control Plane would set the variable at the start of
PITR restore and then reset it at the end of workflow. This allows the
workflow to complete without failure even when heatwave cluster is not
attached. Since metadata is always updated when secondary load/unload
DDLs are executed, when heatwave cluster is attached at a later point
in time, the respective tables get reloaded to heatwave automatically.
Change-Id: I42e984910da23a0e416edb09d3949989159ef707
# This is the commit message mysql#3:
WL#15280: HEATWAVE SUPPORT FOR MDS HA (TEST CHANGES)
This commit adds new functional tests for the MDS HA + HW integration.
Change-Id: Ic818331a4ca04b16998155efd77ac95da08deaa1
# This is the commit message mysql#4:
WL#15280: HEATWAVE SUPPORT FOR MDS HA
BUG#34776485: RESTRICT DEFAULT VALUE FOR rapid_enable_delayed_secondary_ops
This commit does two things:
1. Add a basic test for newly introduced system variable
rapid_enable_delayed_secondary_ops, which controls the behavior of
alter table secondary load/unload ddl statements when rapid cluster
is not available.
2. It also restricts the DEFAULT value setting for the system variable
So, following is not allowed:
SET GLOBAL rapid_enable_delayed_secondary_ops = default
This variable is to be used in restricted scenarios and control plane
only sets it to ON/OFF before and after PITR apply. Allowing set to
default has no practical use.
Change-Id: I85c84dfaa0f868dbfc7b1a88792a89ffd2e81da2
# This is the commit message mysql#5:
Bug#34726490: ADD DIAGNOSTICS FOR SECONDARY LOAD / UNLOAD DDL
Problem:
--------
If secondary load or unload DDL gets rolled back due to some error after
it had loaded / unloaded the table in heatwave cluster, there is no undo
of the secondary engine action. Only secondary_load flag update is
reverted and binlog is not written. From User's perspective, the table
is loaded and can be seen on performance_schema. There are also no
error messages printed to notify that the ddl didn't commit. This
creates a problem to debug any issue in this area.
Solution:
---------
The partial undo of secondary load/unload ddl will be handled in
bug#34592922. In this commit, we add diagnostics to reveal if the ddl
failed to commit, and from what stage.
Change-Id: I46c04dd5dbc07fc17beb8aa2a8d0b15ddfa171af
# This is the commit message mysql#6:
WL#15280: HEATWAVE SUPPORT FOR MDS HA (TEST FIX)
Since ALTER TABLE SECONDARY LOAD / UNLOAD DDL statements now write
to binlog, from Heatwave's perspective, SCN is bumped up.
In this commit, we are adjusting expected SCN values in certain
tests which does secondary load/unload and expects SCN to match.
Change-Id: I9635b3cd588d01148d763d703c72cf50a0c0bb98
# This is the commit message mysql#7:
Adding MTR tests for ML in rapid group_replication suite
Added MTR tests with Heatwave ML queries with in
an HA setup.
Change-Id: I386a3530b5bbe6aea551610b6e739ab1cf366439
# This is the commit message mysql#8:
WL#15280: HEATWAVE SUPPORT FOR MDS HA (MTR TEST ADJUSTMENT)
In this commit we have adjusted the existing test to work with the
new MTR test infrastructure which extends the functionalities to
HA landscape. With this change, a lot of mannual settings have now
become redundant and thus removed in this commit.
Change-Id: Ie1f4fcfdf047bfe8638feaa9f54313d509cbad7e
# This is the commit message mysql#9:
WL#15280: HEATWAVE SUPPORT FOR MDS HA (CLANG-TIDY FIX)
Fix clang-tidy warnings found in previous change#16530, patch#20
Change-Id: I15d25df135694c2f6a3a9146feebe2b981637662
Change-Id: I3f3223a85bb52343a4619b0c2387856b09438265
dveeden
pushed a commit
to dveeden/mysql-server
that referenced
this pull request
May 5, 2023
…sion In class SocketServer::Session, make m_socket private, so the compiler catches all use of m_socket in the two derived Session classes in mgmd and cpcd. Adapt these two classes to use NdbSocket. Change-Id: I1cc0925981320669ce433c9bd72105078ce1e39a
dveeden
pushed a commit
to dveeden/mysql-server
that referenced
this pull request
May 5, 2023
…ting_sharing ASAN reports a use-after-free in the exit-handlers of the routing-sharing integration tests: AddressSanitizer: heap-use-after-free on address ... ... mysql#4 0x708d05 in std::default_delete<SharedRestartableRouter>:: operator()(SharedRestartableRouter*) const .../include/c++/9/bits/unique_ptr.h:81:2 mysql#5 0x708c98 in std::unique_ptr<SharedRestartableRouter, std::default_delete<SharedRestartableRouter> >::~unique_ptr() .../include/c++/9/bits/unique_ptr.h:292:4 mysql#6 0x6a7448 in std::array<std::unique_ptr<SharedRestartableRouter, std::default_delete<SharedRestartableRouter> >, 3ul>::~array() .../include/c++/9/array:94:12 mysql#7 0x7f160a5e68d6 in __run_exit_handlers .../glibc-2.31/stdlib/exit.c:108:8 ... as the RestartedRouters are destructed after their process-manager is destructed. - The process manager is owned by the test - The RestartableRouters are 'static' and outlive the test. Change ------ - explicitely free the intermediate routers at test-suite teardown to ensure the proper sequence. Change-Id: Id405fb26b0519c49820a58b5f70d0ee59d2cb83f
venkatesh-prasad-v
pushed a commit
to venkatesh-prasad-v/mysql-server
that referenced
this pull request
Aug 3, 2023
…etwork https://bugs.mysql.com/bug.php?id=109668 Description ----------- GR suffered from problems caused by the security probes and network scanner processes connecting to the group replication communication port. This usually is not a problem, but poses a serious threat when another member tries to join the cluster by initialting a connection to the member which is affected by external processes using the port dedicated for group communication for longer durations. On such activites by external processes, the SSL enabled server stalled forever on the SSL_accept() call waiting for handshake data. Below is the stacktrace: Thread 55 (Thread 0x7f7bb77ff700 (LWP 2198598)): #0 in read () mysql#1 in sock_read () mysql#2 in BIO_read () mysql#3 in ssl23_read_bytes () mysql#4 in ssl23_get_client_hello () mysql#5 in ssl23_accept () mysql#6 in xcom_tcp_server_startup(Xcom_network_provider*) () When the server stalled in the above path forever, it prohibited other members to join the cluster resulting in the following messages on the joiner server's logs. [ERROR] [MY-011640] [Repl] Plugin group_replication reported: 'Timeout on wait for view after joining group' [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] The member is already leaving or joining a group.' Solution -------- This patch adds two new variables 1. group_replication_xcom_ssl_socket_timeout It is a file-descriptor level timeout in seconds for both accept() and SSL_accept() calls when group replication is listening on the xcom port. When set to a valid value, say for example 5 seconds, both accept() and SSL_accept() return after 5 seconds. The default value has been set to 0 (waits infinitely) for backward compatibility. This variable is effective only when GR is configred with SSL. 2. group_replication_xcom_ssl_accept_retries It defines the number of retries to be performed before closing the socket. For each retry the server thread calls SSL_accept() with timeout defined by the group_replication_xcom_ssl_socket_timeout for the SSL handshake process once the connection has been accepted by the first accept() call. The default value has been set to 10. This variable is effective only when GR is configred with SSL. Note: - Both of the above variables are dynamically configurable, but will become effective only on START GROUP_REPLICATION. - This patch is only for the Linux systems.
bjornmu
pushed a commit
that referenced
this pull request
Oct 25, 2023
Part of WL#15135 Certificate Architecture In NDB data and management nodes, initialize TLS at startup time. ndbd and ndb_mgmd now take option --ndb-tls-search-path. Change-Id: I73e29c10af9d5366c53f86106ec82feac33c4bdf
bjornmu
pushed a commit
that referenced
this pull request
Oct 25, 2023
On the client side, in Transporter::connect_client(), check the TLS negotiation result, initiate the TLS handshake, and perform certificate authorization checks. On the server side, in TransporterService::newSession(), check the TLS negotiation result and complete the TLS handshake. Then in TransporterRegistry::connect_server(), after receiving the client's hello message, perform certificate authorization checks. The connect and disconnect impls in TCP_Transporter are responsible for maintaining the m_encrypted flag and the certificate table. In TCP_Transporter::connect_common(), locking is enabled around the SSL object; the lock must be acquired in order to send data, receive data, or close the socket. Change-Id: I9758adc88e8f152760bab3df1485556681e88d07
bjornmu
pushed a commit
that referenced
this pull request
Oct 25, 2023
Part of WL#15524 Secure NDB Management Protocol using TLS Run an authorization check prior to every MGM command on the server side. The authorization check takes three inputs: a current user auth level associated with the session, a per-command auth level, and a per-server auth level. It returns an integer result code. This is implemented for the whole API by providing a specialization of Parser<T>::run() for MgmApiSession as T. The specialized method adds a call to the authorization check before running whatever function is registered to handle the current command. Per-command auth flags are stored as a ParserRow user_value. The per-session auth level is stored in MgmApiSession, and the per-server auth flags are in MgmtSrvr. Some unused macros are removed; the grammar does not make any use of aliases, so the new parser routine does not support them. Change-Id: Id8f35693ee67f559a5e30ec6005d62473f4e8492
bjornmu
pushed a commit
that referenced
this pull request
Jan 16, 2024
Post push fix. Add missing calls to close to avoid socket leaks. Had Change-Id: I4bfc2209e6f976aa5642a3c7fa6f12395e9fdc60 Change-Id: Idbe81f0f70bbf827b673d5842b65a74f79a455f9
venkatesh-prasad-v
pushed a commit
to venkatesh-prasad-v/mysql-server
that referenced
this pull request
Jan 26, 2024
…and a local
DDL executed
https://bugs.mysql.com/bug.php?id=113727
Problem
-------
In high concurrency scenarios, MySQL replica can enter into a deadlock due to a
race condition between the replica applier thread and the client thread
performing a binlog group commit.
Analysis
--------
It needs at least 3 threads for this deadlock to happen
1. One client thread
2. Two replica applier threads
How this deadlock happens?
--------------------------
0. Binlog is enabled on replica, but log_replica_updates is disabled.
1. Initially, both "Commit Order" and "Binlog Flush" queues are empty.
2. Replica applier thread 1 enters the group commit pipeline to register in the
"Commit Order" queue since `log-replica-updates` is disabled on the replica
node.
3. Since both "Commit Order" and "Binlog Flush" queues are empty, the applier
thread 1
3.1. Becomes leader (In Commit_stage_manager::enroll_for()).
3.2. Registers in the commit order queue.
3.3. Acquires the lock MYSQL_BIN_LOG::LOCK_log.
3.4. Commit Order queue is emptied, but the lock MYSQL_BIN_LOG::LOCK_log is
not yet released.
NOTE: SE commit for applier thread is already done by the time it reaches
here.
4. Replica applier thread 2 enters the group commit pipeline to register in the
"Commit Order" queue since `log-replica-updates` is disabled on the replica
node.
5. Since the "Commit Order" queue is empty (emptied by applier thread 1 in 3.4), the
applier thread 2
5.1. Becomes leader (In Commit_stage_manager::enroll_for())
5.2. Registers in the commit order queue.
5.3. Tries to acquire the lock MYSQL_BIN_LOG::LOCK_log. Since it is held by applier
thread 1 it will wait until the lock is released.
6. Client thread enters the group commit pipeline to register in the
"Binlog Flush" queue.
7. Since "Commit Order" queue is not empty (there is applier thread 2 in the
queue), it enters the conditional wait `m_stage_cond_leader` with an
intention to become the leader for both the "Binlog Flush" and
"Commit Order" queues.
8. Applier thread 1 releases the lock MYSQL_BIN_LOG::LOCK_log and proceeds to update
the GTID by calling gtid_state->update_commit_group() from
Commit_order_manager::flush_engine_and_signal_threads().
9. Applier thread 2 acquires the lock MYSQL_BIN_LOG::LOCK_log.
9.1. It checks if there is any thread waiting in the "Binlog Flush" queue
to become the leader. Here it finds the client thread waiting to be
the leader.
9.2. It releases the lock MYSQL_BIN_LOG::LOCK_log and signals on the
cond_var `m_stage_cond_leader` and enters a conditional wait until the
thread's `tx_commit_pending` is set to false by the client thread
(will be done in the
Commit_stage_manager::process_final_stage_for_ordered_commit_group()
called by client thread from fetch_and_process_flush_stage_queue()).
10. The client thread wakes up from the cond_var `m_stage_cond_leader`. The
thread has now become a leader and it is its responsibility to update GTID
of applier thread 2.
10.1. It acquires the lock MYSQL_BIN_LOG::LOCK_log.
10.2. Returns from `enroll_for()` and proceeds to process the
"Commit Order" and "Binlog Flush" queues.
10.3. Fetches the "Commit Order" and "Binlog Flush" queues.
10.4. Performs the storage engine flush by calling ha_flush_logs() from
fetch_and_process_flush_stage_queue().
10.5. Proceeds to update the GTID of threads in "Commit Order" queue by
calling gtid_state->update_commit_group() from
Commit_stage_manager::process_final_stage_for_ordered_commit_group().
11. At this point, we will have
- Client thread performing GTID update on behalf if applier thread 2 (from step 10.5), and
- Applier thread 1 performing GTID update for itself (from step 8).
Due to the lack of proper synchronization between the above two threads,
there exists a time window where both threads can call
gtid_state->update_commit_group() concurrently.
In subsequent steps, both threads simultaneously try to modify the contents
of the array `commit_group_sidnos` which is used to track the lock status of
sidnos. This concurrent access to `update_commit_group()` can cause a
lock-leak resulting in one thread acquiring the sidno lock and not
releasing at all.
-----------------------------------------------------------------------------------------------------------
Client thread Applier Thread 1
-----------------------------------------------------------------------------------------------------------
update_commit_group() => global_sid_lock->rdlock(); update_commit_group() => global_sid_lock->rdlock();
calls update_gtids_impl_lock_sidnos() calls update_gtids_impl_lock_sidnos()
set commit_group_sidno[2] = true set commit_group_sidno[2] = true
lock_sidno(2) -> successful
lock_sidno(2) -> waits
update_gtids_impl_own_gtid() -> Add the thd->owned_gtid in `executed_gtids()`
if (commit_group_sidnos[2]) {
unlock_sidno(2);
commit_group_sidnos[2] = false;
}
Applier thread continues..
lock_sidno(2) -> successful
update_gtids_impl_own_gtid() -> Add the thd->owned_gtid in `executed_gtids()`
if (commit_group_sidnos[2]) { <=== this check fails and lock is not released.
unlock_sidno(2);
commit_group_sidnos[2] = false;
}
Client thread continues without releasing the lock
-----------------------------------------------------------------------------------------------------------
12. As the above lock-leak can also happen the other way i.e, the applier
thread fails to unlock, there can be different consequences hereafter.
13. If the client thread continues without releasing the lock, then at a later
stage, it can enter into a deadlock with the applier thread performing a
GTID update with stack trace.
Client_thread
-------------
mysql#1 __GI___lll_lock_wait
mysql#2 ___pthread_mutex_lock
mysql#3 native_mutex_lock <= waits for commit lock while holding sidno lock
mysql#4 Commit_stage_manager::enroll_for
mysql#5 MYSQL_BIN_LOG::change_stage
mysql#6 MYSQL_BIN_LOG::ordered_commit
mysql#7 MYSQL_BIN_LOG::commit
mysql#8 ha_commit_trans
mysql#9 trans_commit_implicit
mysql#10 mysql_create_like_table
mysql#11 Sql_cmd_create_table::execute
mysql#12 mysql_execute_command
mysql#13 dispatch_sql_command
Applier thread
--------------
mysql#1 ___pthread_mutex_lock
mysql#2 native_mutex_lock
mysql#3 safe_mutex_lock
mysql#4 Gtid_state::update_gtids_impl_lock_sidnos <= waits for sidno lock
mysql#5 Gtid_state::update_commit_group
mysql#6 Commit_order_manager::flush_engine_and_signal_threads <= acquires commit lock here
mysql#7 Commit_order_manager::finish
mysql#8 Commit_order_manager::wait_and_finish
mysql#9 ha_commit_low
mysql#10 trx_coordinator::commit_in_engines
mysql#11 MYSQL_BIN_LOG::commit
mysql#12 ha_commit_trans
mysql#13 trans_commit
mysql#14 Xid_log_event::do_commit
mysql#15 Xid_apply_log_event::do_apply_event_worker
mysql#16 Slave_worker::slave_worker_exec_event
mysql#17 slave_worker_exec_job_group
mysql#18 handle_slave_worker
14. If the applier thread continues without releasing the lock, then at a later
stage, it can perform recursive locking while setting the GTID for the next
transaction (in set_gtid_next()).
In debug builds the above case hits the assertion
`safe_mutex_assert_not_owner()` meaning the lock is already acquired by the
replica applier thread when it tries to re-acquire the lock.
Solution
--------
In the above problematic example, when seen from each thread
individually, we can conclude that there is no problem in the order of lock
acquisition, thus there is no need to change the lock order.
However, the root cause for this problem is that multiple threads can
concurrently access to the array `Gtid_state::commit_group_sidnos`.
In its initial implementation, it was expected that threads should
hold the `MYSQL_BIN_LOG::LOCK_commit` before modifying its contents. But it
was not considered when upstream implemented WL#7846 (MTS:
slave-preserve-commit-order when log-slave-updates/binlog is disabled).
With this patch, we now ensure that `MYSQL_BIN_LOG::LOCK_commit` is acquired
when the client thread (binlog flush leader) when it tries to perform GTID
update on behalf of threads waiting in "Commit Order" queue, thus providing a
guarantee that `Gtid_state::commit_group_sidnos` array is never accessed
without the protection of `MYSQL_BIN_LOG::LOCK_commit`.
bjornmu
pushed a commit
that referenced
this pull request
Apr 30, 2024
When built with ASAN, a use-after-free is reported for the TcpPortPool.
AddressSanitizer: heap-use-after-free on address 0x60200019f190 at pc
0x00000076a18d bp 0x7fff51e7d1d0 sp 0x7fff51e7d1c0
#4 0x770b73 in UniqueId::ProcessUniqueIds::erase(unsigned int)
../router/tests/helpers/tcp_port_pool.h:112
#5 0x770c48 in UniqueId::~UniqueId()
../router/tests/helpers/tcp_port_pool.cc:234
...
#12 0x82faa3 in testing::UnitTest::~UnitTest()
../extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:5496
#13 0x7f5fe085ace8 in __run_exit_handlers (/lib64/libc.so.6+0x39ce8)
0x60200019f190 is located 0 bytes inside of 16-byte region
[0x60200019f190,0x60200019f1a0)
freed by thread T0 here:
#0 0x7f5fe3cbd10f in operator delete(void*, unsigned long)
(/lib64/libasan.so.6+0xb710f)
#1 0x7f5fe085ace8 in __run_exit_handlers (/lib64/libc.so.6+0x39ce8)
Background
==========
__run_exit_handlers destroys "static" and "global" variables in reverse
order of their creation.
googletest's unit-tests are a static, and the TcpPortPool also has
ProcessUniqueId's which contains the process-wide unique-ids.
At construct: unittest -> tcp-port-pool -> proces-unique-ids
At destruct : process-unique-ids -> tcp-port-pool -> 💥
The use-after-free happens as the process-unique-ids static is
destructed before the tcp-port-pool which tries to its Ids from the
process-unique-ids.
Change
======
- extend the lifetime of the process-unique-ids to after the last use of
the tcp-port-pool via a std::shared_ptr<>
Change-Id: I75b8b781e1d240f18ca72f2c86182639a7699f06
bjornmu
pushed a commit
that referenced
this pull request
Apr 30, 2024
…nt on Windows and posix [#5] A test program NdbProcess-t for testing argument quoting. Change-Id: I37842d1109f76f684636eb7c763d2d5a46114aae
bjornmu
pushed a commit
that referenced
this pull request
Apr 30, 2024
Problem: Starting ´ndb_mgmd --bind-address´ may potentially cause abnormal program termination in MgmtSrvr destructor when ndb_mgmd restart itself. Core was generated by `ndb_mgmd --defa'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f8ce4066b8f in raise () from /lib64/libc.so.6 #1 0x00007f8ce4039ea5 in abort () from /lib64/libc.so.6 #2 0x00007f8ce40a7d97 in __libc_message () from /lib64/libc.so.6 #3 0x00007f8ce40af08c in malloc_printerr () from /lib64/libc.so.6 #4 0x00007f8ce40b132d in _int_free () from /lib64/libc.so.6 #5 0x00000000006e9ffe in MgmtSrvr::~MgmtSrvr (this=0x28de4b0) at mysql/8.0/storage/ndb/src/mgmsrv/MgmtSrvr.cpp: 890 #6 0x00000000006ea09e in MgmtSrvr::~MgmtSrvr (this=0x2) at mysql/8.0/ storage/ndb/src/mgmsrv/MgmtSrvr.cpp:849 #7 0x0000000000700d94 in mgmd_run () at mysql/8.0/storage/ndb/src/mgmsrv/main.cpp:260 #8 0x0000000000700775 in mgmd_main (argc=<optimized out>, argv=0x28041d0) at mysql/8.0/storage/ndb/src/ mgmsrv/main.cpp:479 Analysis: While starting up, the ndb_mgmd will allocate memory for bind_address in order to potentially rewrite the parameter. When ndb_mgmd restart itself the memory will be released and dangling pointer causing double free. Fix: Drop support for bind_address=[::], it is not documented anywhere, is not useful and doesn't work. This means the need to rewrite bind_address is gone and bind_address argument need neither alloc or free. Change-Id: I7797109b9d8391394587188d64d4b1f398887e94
bjornmu
pushed a commit
that referenced
this pull request
Jul 1, 2024
… for connection xxx'.
The new iterator based explains are not impacted.
The issue here is a race condition. More than one thread is using the
query term iterator at the same time (whoch is neithe threas safe nor
reantrant), and part of its state is in the query terms being visited
which leads to interference/race conditions.
a) the explain thread
uses an iterator here:
Sql_cmd_explain_other_thread::execute
is inspecting the Query_expression of the running query
calling master_query_expression()->find_blocks_query_term which uses
an iterator over the query terms in the query expression:
for (auto qt : query_terms<>()) {
if (qt->query_block() == qb) {
return qt;
}
}
the above search fails to find qb due to the interference of the
thread b), see below, and then tries to access a nullpointer:
* thread #36, name = ‘connection’, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
frame #0: 0x000000010bb3cf0d mysqld`Query_block::type(this=0x00007f8f82719088) const at sql_lex.cc:4441:11
frame #1: 0x000000010b83763e mysqld`(anonymous namespace)::Explain::explain_select_type(this=0x00007000020611b8) at opt_explain.cc:792:50
frame #2: 0x000000010b83cc4d mysqld`(anonymous namespace)::Explain_join::explain_select_type(this=0x00007000020611b8) at opt_explain.cc:1487:21
frame #3: 0x000000010b837c34 mysqld`(anonymous namespace)::Explain::prepare_columns(this=0x00007000020611b8) at opt_explain.cc:744:26
frame #4: 0x000000010b83ea0e mysqld`(anonymous namespace)::Explain_join::explain_qep_tab(this=0x00007000020611b8, tabnum=0) at opt_explain.cc:1415:32
frame #5: 0x000000010b83ca0a mysqld`(anonymous namespace)::Explain_join::shallow_explain(this=0x00007000020611b8) at opt_explain.cc:1364:9
frame #6: 0x000000010b83379b mysqld`(anonymous namespace)::Explain::send(this=0x00007000020611b8) at opt_explain.cc:770:14
frame #7: 0x000000010b834147 mysqld`explain_query_specification(explain_thd=0x00007f8fbb111e00, query_thd=0x00007f8fbb919c00, query_term=0x00007f8f82719088, ctx=CTX_JOIN) at opt_explain.cc:2088:20
frame #8: 0x000000010bd36b91 mysqld`Query_expression::explain_query_term(this=0x00007f8f7a090360, explain_thd=0x00007f8fbb111e00, query_thd=0x00007f8fbb919c00, qt=0x00007f8f82719088) at sql_union.cc:1519:11
frame #9: 0x000000010bd36c68 mysqld`Query_expression::explain_query_term(this=0x00007f8f7a090360, explain_thd=0x00007f8fbb111e00, query_thd=0x00007f8fbb919c00, qt=0x00007f8f8271d748) at sql_union.cc:1526:13
frame #10: 0x000000010bd373f7 mysqld`Query_expression::explain(this=0x00007f8f7a090360, explain_thd=0x00007f8fbb111e00, query_thd=0x00007f8fbb919c00) at sql_union.cc:1591:7
frame #11: 0x000000010b835820 mysqld`mysql_explain_query_expression(explain_thd=0x00007f8fbb111e00, query_thd=0x00007f8fbb919c00, unit=0x00007f8f7a090360) at opt_explain.cc:2392:17
frame #12: 0x000000010b835400 mysqld`explain_query(explain_thd=0x00007f8fbb111e00, query_thd=0x00007f8fbb919c00, unit=0x00007f8f7a090360) at opt_explain.cc:2353:13
* frame #13: 0x000000010b8363e4 mysqld`Sql_cmd_explain_other_thread::execute(this=0x00007f8fba585b68, thd=0x00007f8fbb111e00) at opt_explain.cc:2531:11
frame #14: 0x000000010bba7d8b mysqld`mysql_execute_command(thd=0x00007f8fbb111e00, first_level=true) at sql_parse.cc:4648:29
frame #15: 0x000000010bb9e230 mysqld`dispatch_sql_command(thd=0x00007f8fbb111e00, parser_state=0x0000700002065de8) at sql_parse.cc:5303:19
frame #16: 0x000000010bb9a4cb mysqld`dispatch_command(thd=0x00007f8fbb111e00, com_data=0x0000700002066e38, command=COM_QUERY) at sql_parse.cc:2135:7
frame #17: 0x000000010bb9c846 mysqld`do_command(thd=0x00007f8fbb111e00) at sql_parse.cc:1464:18
frame #18: 0x000000010b2f2574 mysqld`handle_connection(arg=0x0000600000e34200) at connection_handler_per_thread.cc:304:13
frame #19: 0x000000010e072fc4 mysqld`pfs_spawn_thread(arg=0x00007f8fba8160b0) at pfs.cc:3051:3
frame #20: 0x00007ff806c2b202 libsystem_pthread.dylib`_pthread_start + 99
frame #21: 0x00007ff806c26bab libsystem_pthread.dylib`thread_start + 15
b) the query thread being explained is itself performing LEX::cleanup
and as part of the iterates over the query terms, but still allows
EXPLAIN of the query plan since
thd->query_plan.set_query_plan(SQLCOM_END, ...)
hasn't been called yet.
20:frame: Query_terms<(Visit_order)1, (Visit_leaves)0>::Query_term_iterator::operator++() (in mysqld) (query_term.h:613)
21:frame: Query_expression::cleanup(bool) (in mysqld) (sql_union.cc:1861)
22:frame: LEX::cleanup(bool) (in mysqld) (sql_lex.h:4286)
30:frame: Sql_cmd_dml::execute(THD*) (in mysqld) (sql_select.cc:799)
31:frame: mysql_execute_command(THD*, bool) (in mysqld) (sql_parse.cc:4648)
32:frame: dispatch_sql_command(THD*, Parser_state*) (in mysqld) (sql_parse.cc:5303)
33:frame: dispatch_command(THD*, COM_DATA const*, enum_server_command) (in mysqld) (sql_parse.cc:2135)
34:frame: do_command(THD*) (in mysqld) (sql_parse.cc:1464)
57:frame: handle_connection(void*) (in mysqld) (connection_handler_per_thread.cc:304)
58:frame: pfs_spawn_thread(void*) (in mysqld) (pfs.cc:3053)
65:frame: _pthread_start (in libsystem_pthread.dylib) + 99
66:frame: thread_start (in libsystem_pthread.dylib) + 15
Solution:
This patch solves the issue by removing iterator state from
Query_term, making the query_term iterators thread safe. This solution
labels every child query_term with its index in its parent's
m_children vector. The iterator can therefore easily compute the next
child to visit based on Query_term::m_sibling_idx.
A unit test case is added to check reentrancy.
One can also manually verify that we have no remaining race condition
by running two client connections files (with \. <file>) with a big
number of copies of the repro query in one connection and a big number
of EXPLAIN format=json FOR <connection>, e.g.
EXPLAIN FORMAT=json FOR CONNECTION 8\G
in the other. The actual connection number would need to verified
in connection one, of course.
Change-Id: Ie7d56610914738ccbbecf399ccc4f465f7d26ea7
bjornmu
pushed a commit
that referenced
this pull request
Jul 1, 2024
This is a combination of 5 commits.
This is the 1st commit message:
WL#15746: TLS Enhancements for HeatWave-AutoML & Dask Comm. Upgrade
Problem:
--------
- HeatWave-AutoML communication was unauthenticated, unauthorized,
and unencrypted.
- Dask communication utilized TCP, not aligning with FedRamp
guidelines.
Solution:
---------
- Introduced TLS and mTLS in HeatWave-AutoML's plugin and driver for
authentication, authorization, and encryption.
- Applied TLS to Dask to ensure authentication, encryption, and
authorization.
Dask Authorization (OCID-based):
--------------------------------
1. For each DBsystem:
- MySQL node sends OCIDs of authorized nodes to the head driver
via:
a. rapid_net_nodes
b. rapid_net_allowed_ocids (older API, mainly for MTR tests)
- Scenarios:
a. All OCIDs provided: Dask authorizes.
b. Any OCID absent: ML call fails with message.
2. During Dask worker registration to the Dask scheduler, a script is
dispatched to the Dask worker for execution, retrieving the worker's
OCID for authorization purposes.
- If the OCID isn't approved, the connection is denied, terminating
the worker and causing the ML query to fail.
3. For every Dask worker (both as listener and connector), an OCID-
based authorization is performed post SSL/TLS connection handshake.
The process compares the OCID from the peer's certificate against
the allowed_ocids received from the HeatWave-AutoML MySQL plugin.
HWAML Plugin Changes:
---------------------
- Sourced certificate data and SSL setup from disk, incorporating
SSL/TLS for HWAML.
- Reused "keystore" variable to specify disk location for
certificate retrieval.
- Certificates and keys expected in PKCS12 format.
- Introduced "have_ml_encryption" variable (default=0).
> Acts as a switch to explicitly deactivate HWAML network
encryption, akin to "disable_net_encryption" affecting
network encryption for HeatWave. Set to 1 to enable.
- Introduced a customized verifier function for verify_callback to
be set in SSL_CTX_set_verify and used in the handshake process
of SSL/TLS. The customized verifier function will perform
instance id (OCID) based authorization on the plugin side during
standard SSL/TLS handshake process.
- CRL (Certificate Revocation List) checks are also conducted if CRL
Distribution Points are present and accessible in the provided
certificate.
HWAML Driver Changes & OCID-based Authorization:
------------------------------------------------
- Introduced "enable_encryption" (default=0).
> Set to 1 to enable encryption.
- When receiving a new connection request and encryption is on, the
driver performs OCID-based self-checking, comparing OCID retrieved
from its own instance principal with the OCID in the
provided certificate on disk.
- The driver compares OCID from "mysql_compute_id" and extracted OCID
from mTLS certificate during connection.
- Introduced "cert_dir" argument for certificate directory
specification.
- Expected files: cert_chain.pem, certificate.pem, private_key.pem.
> OCID should be in the userID (UID) or CN field of the
certificate.pem subject.
- CRL (Certificate Revocation List) checks are also conducted post
handshake, if CRL Distribution Points are present and accessible in
the provided certificate, alongside OCID authorization.
Encryption Behavior:
--------------------
- If encryption is deactivated on both plugin and driver side, HWAML
will work without encryption as it was before this commit.
Enabling Encryption:
--------------------
- By default, "have_ml_encryption" and "enable_encryption" are set to 0
> Encryption is disabled by default.
- For the HWAML plugin:
> "have_ml_encryption" set to 1 (default is 0).
> Specify the .pfx file's path using the "keystore".
- For the HWAML Driver:
> "enable_encryption" set to 1 (default is 0)
> Specify "mysql_instance_id" and "cert_dir".
Testing:
--------
- MTR has been modified for the encryption setup.
> Runs with encryption if "OCI_INSTANCE_ID" is set to a valid
value.
- On OCI (when "OLRAPID_KEYSTORE" is not set):
> Certificates and keys are generated; PEMs for driver and PKCS12
for plugin.
- On AWS (when "OLRAPID_KEYSTORE" is set as the path to PKCS12
keystore files):
> PEM files are extracted from the provided PKCS12 and used for
the driver. The plugin uses the provided PKCS12 keystore file.
Change-Id: I553ca135241e03484db6debbe186e6d34d582bf4
This is the commit message #2:
WL#15746 - Adding ML encryption support to BM
Enabling ML encryption on Baumeister:
- Certificates are generated on MySQLd during initialization
- Needed certicates for workers are packaged and sent to worker nodes
- Workers use packaged files to generate their certificates
- Arguments are added to driver.py invoke
- Keystore path is added to mysql config
Change-Id: I11a5cc5926488ff4fbf91bb6c10a091358db7dc9
This is the commit message #3:
WL#15746: Enhanced CRL Daemon Checker
Issue
=====
The previous design assumed a plain HTTPS link for the CRL distribution
point, accessible to all. This assumption no longer holds, as public
accessibility for CRL distribution contradicts OCI guidelines. Now, the
CRL distribution point in certificates provided by the control plane is
expected to be protected by OCI Instance Principal Authentication.
However, using this authentication method introduces a delay of several
seconds, which is impractical for HeatWave-AutoML.
Solution
========
The CRL fetching code now uses OCI Instance Principal Authentication.
To mitigate performance issues, the CRL checking process has been
redesigned. Instead of enforcing CRL checks per connection in MySQL
Plugin and HeatWave-AutoML Driver communications, a daemon thread in
HeatWave-AutoML Driver, Dask scheduler, and Dask Worker now periodically
fetches and verifies the CRL against all active connections. This
separation minimizes performance impacts. Consequently, MySQL Plugin's
CRL checks have been removed, as checks in the Driver, Scheduler, and
Worker sufficiently cover all cluster nodes.
Changes
=======
- Implemented CRL checker as a daemon thread in Driver, Scheduler, and
Worker.
- Each connection/socket has an associated CRL checker.
- CRL checks occur periodically at set intervals.
- Skips CRL check if the CRL is temporarily unavailable.
- Failing a CRL check results in the associated connection/socket being
closed. On the Driver, a stop event is triggered (akin to CTRL-C).
Change-Id: Id998cfe9e15d9236291b0ae420d65c2197837966
This is the commit message #4:
WL#15746: Fix Dask workers being shutdown without releasing address
Issue
=====
Dask workers getting shutting but not releasing the address used
properly sometimes.
Solution
========
Reverted some changes in heatwave_cluster.py in dask worker shutdown
function. Hopefully this will fix the address issue
Change-Id: I5a6749b5a25b0ccb73ba7369e545bc010da1b84f
This is the commit message #5:
WL#15746: Implement Dask Worker Join Timeout for Head Node
Issue:
======
In the cluster_shutdown method, the join operation on the head node's
worker process lacked a timeout. This led to potential indefinite
waiting and eventual hanging of the head node.
Solution:
=========
A timeout has been introduced for the worker process join on the head
node. Unlike non-head nodes, which rely on worker join to complete Dask
tasks and cannot have a timeout, the head node can safely implement
this. Now, if the worker process on the head node fails to shut down
post-join, indicating a timeout, it will be manually terminated. This
ensures proper release of associated resources and prevents hanging of
the head node.
Additional Change:
==================
Added Cert Rotation Guard for DASK clusters. This feature initiates on
the first plugin-driver connection when the DASK cluster is off,
recording the certificate's expiry date. During driver idle times,
it checks the current cert's expiry against this date. If it detects a
change, indicating a certificate rotation, it shuts down the DASK
cluster. The cluster restarts on the next connection request, ensuring
the use of the latest certificate.
Change-Id: Ie63a2e2b7664e05e1622d8bd6503663e13fa73cb
laurynas-biveinis
pushed a commit
to laurynas-biveinis/mysql-server
that referenced
this pull request
Oct 22, 2024
This is a combination of 5 commits.
This is the 1st commit message:
WL#15746: TLS Enhancements for HeatWave-AutoML & Dask Comm. Upgrade
Problem:
--------
- HeatWave-AutoML communication was unauthenticated, unauthorized,
and unencrypted.
- Dask communication utilized TCP, not aligning with FedRamp
guidelines.
Solution:
---------
- Introduced TLS and mTLS in HeatWave-AutoML's plugin and driver for
authentication, authorization, and encryption.
- Applied TLS to Dask to ensure authentication, encryption, and
authorization.
Dask Authorization (OCID-based):
--------------------------------
1. For each DBsystem:
- MySQL node sends OCIDs of authorized nodes to the head driver
via:
a. rapid_net_nodes
b. rapid_net_allowed_ocids (older API, mainly for MTR tests)
- Scenarios:
a. All OCIDs provided: Dask authorizes.
b. Any OCID absent: ML call fails with message.
2. During Dask worker registration to the Dask scheduler, a script is
dispatched to the Dask worker for execution, retrieving the worker's
OCID for authorization purposes.
- If the OCID isn't approved, the connection is denied, terminating
the worker and causing the ML query to fail.
3. For every Dask worker (both as listener and connector), an OCID-
based authorization is performed post SSL/TLS connection handshake.
The process compares the OCID from the peer's certificate against
the allowed_ocids received from the HeatWave-AutoML MySQL plugin.
HWAML Plugin Changes:
---------------------
- Sourced certificate data and SSL setup from disk, incorporating
SSL/TLS for HWAML.
- Reused "keystore" variable to specify disk location for
certificate retrieval.
- Certificates and keys expected in PKCS12 format.
- Introduced "have_ml_encryption" variable (default=0).
> Acts as a switch to explicitly deactivate HWAML network
encryption, akin to "disable_net_encryption" affecting
network encryption for HeatWave. Set to 1 to enable.
- Introduced a customized verifier function for verify_callback to
be set in SSL_CTX_set_verify and used in the handshake process
of SSL/TLS. The customized verifier function will perform
instance id (OCID) based authorization on the plugin side during
standard SSL/TLS handshake process.
- CRL (Certificate Revocation List) checks are also conducted if CRL
Distribution Points are present and accessible in the provided
certificate.
HWAML Driver Changes & OCID-based Authorization:
------------------------------------------------
- Introduced "enable_encryption" (default=0).
> Set to 1 to enable encryption.
- When receiving a new connection request and encryption is on, the
driver performs OCID-based self-checking, comparing OCID retrieved
from its own instance principal with the OCID in the
provided certificate on disk.
- The driver compares OCID from "mysql_compute_id" and extracted OCID
from mTLS certificate during connection.
- Introduced "cert_dir" argument for certificate directory
specification.
- Expected files: cert_chain.pem, certificate.pem, private_key.pem.
> OCID should be in the userID (UID) or CN field of the
certificate.pem subject.
- CRL (Certificate Revocation List) checks are also conducted post
handshake, if CRL Distribution Points are present and accessible in
the provided certificate, alongside OCID authorization.
Encryption Behavior:
--------------------
- If encryption is deactivated on both plugin and driver side, HWAML
will work without encryption as it was before this commit.
Enabling Encryption:
--------------------
- By default, "have_ml_encryption" and "enable_encryption" are set to 0
> Encryption is disabled by default.
- For the HWAML plugin:
> "have_ml_encryption" set to 1 (default is 0).
> Specify the .pfx file's path using the "keystore".
- For the HWAML Driver:
> "enable_encryption" set to 1 (default is 0)
> Specify "mysql_instance_id" and "cert_dir".
Testing:
--------
- MTR has been modified for the encryption setup.
> Runs with encryption if "OCI_INSTANCE_ID" is set to a valid
value.
- On OCI (when "OLRAPID_KEYSTORE" is not set):
> Certificates and keys are generated; PEMs for driver and PKCS12
for plugin.
- On AWS (when "OLRAPID_KEYSTORE" is set as the path to PKCS12
keystore files):
> PEM files are extracted from the provided PKCS12 and used for
the driver. The plugin uses the provided PKCS12 keystore file.
Change-Id: I553ca135241e03484db6debbe186e6d34d582bf4
This is the commit message mysql#2:
WL#15746 - Adding ML encryption support to BM
Enabling ML encryption on Baumeister:
- Certificates are generated on MySQLd during initialization
- Needed certicates for workers are packaged and sent to worker nodes
- Workers use packaged files to generate their certificates
- Arguments are added to driver.py invoke
- Keystore path is added to mysql config
Change-Id: I11a5cc5926488ff4fbf91bb6c10a091358db7dc9
This is the commit message mysql#3:
WL#15746: Enhanced CRL Daemon Checker
Issue
=====
The previous design assumed a plain HTTPS link for the CRL distribution
point, accessible to all. This assumption no longer holds, as public
accessibility for CRL distribution contradicts OCI guidelines. Now, the
CRL distribution point in certificates provided by the control plane is
expected to be protected by OCI Instance Principal Authentication.
However, using this authentication method introduces a delay of several
seconds, which is impractical for HeatWave-AutoML.
Solution
========
The CRL fetching code now uses OCI Instance Principal Authentication.
To mitigate performance issues, the CRL checking process has been
redesigned. Instead of enforcing CRL checks per connection in MySQL
Plugin and HeatWave-AutoML Driver communications, a daemon thread in
HeatWave-AutoML Driver, Dask scheduler, and Dask Worker now periodically
fetches and verifies the CRL against all active connections. This
separation minimizes performance impacts. Consequently, MySQL Plugin's
CRL checks have been removed, as checks in the Driver, Scheduler, and
Worker sufficiently cover all cluster nodes.
Changes
=======
- Implemented CRL checker as a daemon thread in Driver, Scheduler, and
Worker.
- Each connection/socket has an associated CRL checker.
- CRL checks occur periodically at set intervals.
- Skips CRL check if the CRL is temporarily unavailable.
- Failing a CRL check results in the associated connection/socket being
closed. On the Driver, a stop event is triggered (akin to CTRL-C).
Change-Id: Id998cfe9e15d9236291b0ae420d65c2197837966
This is the commit message mysql#4:
WL#15746: Fix Dask workers being shutdown without releasing address
Issue
=====
Dask workers getting shutting but not releasing the address used
properly sometimes.
Solution
========
Reverted some changes in heatwave_cluster.py in dask worker shutdown
function. Hopefully this will fix the address issue
Change-Id: I5a6749b5a25b0ccb73ba7369e545bc010da1b84f
This is the commit message mysql#5:
WL#15746: Implement Dask Worker Join Timeout for Head Node
Issue:
======
In the cluster_shutdown method, the join operation on the head node's
worker process lacked a timeout. This led to potential indefinite
waiting and eventual hanging of the head node.
Solution:
=========
A timeout has been introduced for the worker process join on the head
node. Unlike non-head nodes, which rely on worker join to complete Dask
tasks and cannot have a timeout, the head node can safely implement
this. Now, if the worker process on the head node fails to shut down
post-join, indicating a timeout, it will be manually terminated. This
ensures proper release of associated resources and prevents hanging of
the head node.
Additional Change:
==================
Added Cert Rotation Guard for DASK clusters. This feature initiates on
the first plugin-driver connection when the DASK cluster is off,
recording the certificate's expiry date. During driver idle times,
it checks the current cert's expiry against this date. If it detects a
change, indicating a certificate rotation, it shuts down the DASK
cluster. The cluster restarts on the next connection request, ensuring
the use of the latest certificate.
Change-Id: Ie63a2e2b7664e05e1622d8bd6503663e13fa73cb
dbussink
added a commit
to planetscale/mysql-server
that referenced
this pull request
Nov 21, 2024
In case `with_ndb_home` is set, `buf` is allocated with `PATH_MAX` and the home is already written into the buffer. The additional path is written using `snprintf` and it starts off at `len`. It still can write up to `PATH_MAX` though which is wrong, since if we already have a home written into it, we only have `PATH_MAX - len` available in the buffer. On Ubuntu 24.04 with debug builds this is caught and it crashes: ``` *** buffer overflow detected ***: terminated Signal 6 thrown, attempting backtrace. stack_bottom = 0 thread_stack 0x0 #0 0x604895341cb6 <unknown> mysql#1 0x7ff22524531f <unknown> at sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 mysql#2 0x7ff22529eb1c __pthread_kill_implementation at ./nptl/pthread_kill.c:44 mysql#3 0x7ff22529eb1c __pthread_kill_internal at ./nptl/pthread_kill.c:78 mysql#4 0x7ff22529eb1c __GI___pthread_kill at ./nptl/pthread_kill.c:89 mysql#5 0x7ff22524526d __GI_raise at sysdeps/posix/raise.c:26 mysql#6 0x7ff2252288fe __GI_abort at ./stdlib/abort.c:79 mysql#7 0x7ff2252297b5 __libc_message_impl at sysdeps/posix/libc_fatal.c:132 mysql#8 0x7ff225336c18 __GI___fortify_fail at ./debug/fortify_fail.c:24 mysql#9 0x7ff2253365d3 __GI___chk_fail at ./debug/chk_fail.c:28 mysql#10 0x7ff225337db4 ___snprintf_chk at ./debug/snprintf_chk.c:29 mysql#11 0x6048953593ba <unknown> mysql#12 0x604895331a3d <unknown> mysql#13 0x6048953206e7 <unknown> mysql#14 0x60489531f4b1 <unknown> mysql#15 0x60489531e8e6 <unknown> mysql#16 0x7ff22522a1c9 __libc_start_call_main at sysdeps/nptl/libc_start_call_main.h:58 mysql#17 0x7ff22522a28a __libc_start_main_impl at csu/libc-start.c:360 mysql#18 0x60489531ed54 <unknown> mysql#19 0xffffffffffffffff <unknown> ``` In practice this buffer overflow only would happen with very long paths. Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
MySQL Bug #71898