-
Notifications
You must be signed in to change notification settings - Fork 4k
/
Copy pathmysql-test-run.dox
8387 lines (6534 loc) · 277 KB
/
mysql-test-run.dox
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
########## The MySQL Test Framework ##########
/**
@page PAGE_MYSQL_TEST_RUN The MySQL Test Framework
This manual describes the MySQL test framework, consisting of the
test driver and the test script language.
-----
<h3>Table of Contents</h3>
- @subpage PAGE_PREFACE
- @subpage PAGE_INTRODUCTION
- @subpage PAGE_MYSQLTEST_FRAMEWORK_COMPONENTS
- @subpage PAGE_RUNNING_TESTCASES
- @subpage PAGE_WRITING_TESTCASES
- @subpage PAGE_MYSQLTEST_PROGRAMS
- @subpage PAGE_MYSQLTEST_LANGUAGE_REFERENCE
- @subpage PAGE_UNIT_TESTS
- @subpage PAGE_PLUGINS
*/
###############################################################################
## Preface
/**
@page PAGE_PREFACE Preface
MySQL distributions include a set of test cases and programs for
running them. These tools constitute the MySQL test framework that
provides a means for verifying that MySQL Server and its client
programs operate according to expectations. The test cases consist
mostly of SQL statements, but can also use test language constructs
that control how to run tests and verify their results.
This manual describes the MySQL test framework. It describes the
programs used to run tests and the language used to write test
cases.
*/
###############################################################################
## Introduction to the MySQL Test Framework
/**
@page PAGE_INTRODUCTION Introduction to the MySQL Test Framework
MySQL distributions include a test suite: a set of test cases and
programs for running them. (If you find that the test suite is not
included in your distribution, look for a similar distribution with
<b>-test</b> in the name and install that as well.) These tools
constitute the MySQL test framework that provides a means for
verifying that MySQL Server and its client programs operate
according to expectations. The test cases consist mostly of SQL
statements, but can also use test language constructs that control
how to run tests and verify their results. Distributions also
provide facilities for running unit tests and creating new unit
tests.
This document describes the components of the MySQL test framework,
how the test programs work, and the language used for writing test
cases. It also provides a tutorial for developing test cases and
executing them.
The application that runs the test suite is named @ref
PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl". Its location is the
<b>mysql-test</b> directory, which is present both in source and
binary MySQL Server distributions.
On platforms other than Windows, @ref PAGE_MYSQL_TEST_RUN_PL
"mysql-test-run.pl" is also available through the shortened name
<b>mtr</b> in the same directory, as either a symbolic link or a
copy.
The @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" application
starts MySQL servers, restarts them as necessary when a specific
test case needs different start arguments, and presents the test
result. For each test case, @ref PAGE_MYSQL_TEST_RUN_PL
"mysql-test-run.pl" invokes the @ref PAGE_MYSQLTEST "mysqltest"
program (also referred to as the “test engine”) to read the test
case file, intepret the test language constructs, and send SQL
statements to the server.
Input for each test case is stored in a file, and the expected
result from running the test is stored in another file. The actual
result is compared to the expected result after running the test.
For a MySQL source distribution, @ref PAGE_MYSQL_TEST_RUN_PL
"mysql-test-run.pl" is located in the <b>mysql-test</b> directory,
and @ref PAGE_MYSQLTEST "mysqltest" is located in the <b>client</b>
directory. The <b>mysql-test</b> and <b>client</b> directories
are located in the root directory of the distribution.
For a MySQL binary distribution, @ref PAGE_MYSQL_TEST_RUN_PL
"mysql-test-run.pl" is located in the <b>mysql-test</b> directory,
and @ref PAGE_MYSQLTEST "mysqltest" is located in the same directory
where other client programs such as <b>mysql</b> or
<b>mysqladmin</b> are installed. The locations of the
<b>mysql-test</b> and other directories depend on the layout used
for the distribution format.
Within the <b>mysql-test</b> directory, test case input files and
result files are stored in the <b>t</b> and <b>r</b>
directories, respectively. The input and result files have the same
basename, which is the test name, but have extensions of
<b>.test</b> and <b>.result</b>, respectively. For example, for
a test named “decimal”, the input and result files are
<b>mysql-test/t/decimal.test</b> and <b>mysql-test/r/decimal.result</b>.
Each test file is referred to as one test case, but usually consists
of a sequence of related tests. An unexpected failure of a single
statement in a test case makes the test case fail.
There are several ways a test case can fail:
<ul>
<li>
The @ref PAGE_MYSQLTEST "mysqltest" test engine checks the result
codes from executing each SQL statement in the test input. If the
failure is unexpected, the test case fails.
</li>
<li>
A test case can fail if an error was expected but did not occur
(for example, if an SQL statement succeeded when it should have
failed).
</li>
<li>
The test case can fail by producing incorrect output. As a test
runs, it produces output (the results from <b>SELECT</b>,
<b>SHOW</b>, and other statements). This output is compared
to the expected result found in the <b>mysql-test/r</b> directory
(in a file with a <b>.result</b> suffix). If the expected and
actual results differ, the test case fails. The actual test result
is written to a file in the <b>mysql-test/var/log</b> directory
with a <b>.reject</b> suffix, and the difference between the
<b>.result</b> and <b>.reject</b> files is presented for
evaluation.
</li>
<li>
A test case will fail if the MySQL server dies unexpectedly during
the test. If this happens, the @ref PAGE_MYSQLTEST "mysqltest"
test client will usually also report a failure due to loosing the
connection.
</li>
<li>
Finally, the test case will fail if the error log written by the
MySQL server during the test includes warnings or errors which are
not filtered (suppressed). See @ref PAGE_SUPPRESSING_ERRORS_WARNINGS
for more about suppressing warnings.
</li>
</ul>
This method of checking test results puts some restrictions on how
test cases can be written. For example, the result cannot contain
information that varies from run to run, such as the current time.
However, if the information that varies is unimportant for test
evaluation, there are ways to instruct the test engine to replace
those fields in the output with fixed values.
Because the test cases consist mostly of SQL statements in a text
file, there is no direct support for test cases that are written in
C, Java, or other languages. Such tests are not within the scope of
this test framework. But the framework does support executing your
own scripts and initiating them with your own data. Also, a test
case can execute an external program, so in some respects the test
framework can be extended for uses other than testing SQL
statements. Finally, it is possible to embed small pieces of Perl
code within the test. This can sometimes be used to perform actions
or execute logic which is beyond the capabilities of the test
language or SQL.
*/
###############################################################################
## MySQL Test Framework Components
/**
@page PAGE_MYSQLTEST_FRAMEWORK_COMPONENTS MySQL Test Framework Components
<h3>Table of Contents</h3>
- @subpage PAGE_TEST_FRAMEWORK_SYSTEM_REQUIREMENTS
- @subpage PAGE_REPORT_BUGS
The MySQL test framework consists of programs that run tests, and
directories and files used by those programs.
Test Framework Programs
-----------------------
The MySQL test framework uses several programs:
<ul>
<li>
The @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" Perl script
is the main application used to run the test suite. It invokes
@ref PAGE_MYSQLTEST "mysqltest" to run individual test cases.
</li>
<li>
@ref PAGE_MYSQLTEST "mysqltest" runs test cases.
</li>
<li>
The @ref PAGE_MYSQL_CLIENT_TEST "mysql_client_test" program is
used for testing aspects of the MySQL client API that cannot be
tested using @ref PAGE_MYSQLTEST "mysqltest" and its test
language.
</li>
<li>
The @ref PAGE_MYSQL_STRESS_TEST_PL "mysql-stress-test.pl" Perl
script performs stress-testing of the MySQL server.
</li>
<li>
A unit-testing facility is provided so that individual unit test
programs can be created for storage engines and plugins.
</li>
</ul>
Test suite programs can be found in these locations:
<ul>
<li>
For a source distribution, @ref PAGE_MYSQLTEST "mysqltest" is in
the <b>client</b> directory. For a binary distribution, it is in
the MySQL <b>bin</b> directory.
</li>
<li>
For a source distribution, @ref PAGE_MYSQL_CLIENT_TEST
"mysql_client_test" is in the <b>tests</b> directory. For a
binary distribution, it is in the MySQL <b>bin</b> directory.
</li>
<li>
The other programs are located in the <b>mysql-test</b>
directory. For a source distribution, <b>mysql-test</b> is found
under the source tree root. For a binary distribution, the
location of <b>mysql-test</b> depends on the layout used for the
distribution format.
</li>
</ul>
Test Framework Directories and Files
-------------------------------------
The test suite is located in the <tt>mysql-test</tt> directory,
which contains the following components:
<ul>
<li>
The @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" and @ref
PAGE_MYSQL_STRESS_TEST_PL "mysql-stress-test.pl" programs that
are used for running tests.
</li>
<li>
The <b>t</b> directory contains test case input files. A test
case file might also have additional files associated with it.
</li>
<ul>
<li>
A file name of the form <b><em>test_name</em>.test</b> is a
test case file for a test named <b><em>test_name</em></b>. For
example, <b>subquery.test</b> is the test case file for the
test named <b>subquery</b>.
</li>
<li>
A file name of the form <b><em>test_name</em>-master.opt</b>
provides options to associate with the named test case.
<b>mysql-test-run.pl</b> restarts the server with the options
given in the file if the options are different from those
required for the currently running server.
Note that the <b>-master.opt</b> file is used for the “main”
server of a test, even if no replication is involved.
</li>
<li>
A file name of the form <b><em>test_name</em>-slave.opt</b>
provides slave options.
</li>
<li>
Server options which need to be passed during initialization
of the server can be passed in both
<b><em>test_name</em>-master.opt</b> and
<b><em>test_name</em>-slave.opt</b>. Each bootstrap variable
must be passed as the value of a <b>`--initialize`</b> option so
that @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" can
recognize that the variable should be used during server
initialization, remove the existing data directory, and
initialize a new data directory with the options provided.
@ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will also
start the server afterwards, with the bootstrap options, and
other options specified.
</li>
<li>
A file name of the form <b><em>test_name</em>-client.opt </b>
provides @ref PAGE_MYSQLTEST "mysqltest" client options.
</li>
<li>
A file name of the form <b><em>test_name</em>.cnf</b> contains
additional entries for the config file to be used for this
test.
<ul>
<li>
During a test run, each server is started with a different
<b>defaults-group-suffix</b> value and this suffix value
can be used with a group name to specify server specific
options. For example, section <b>[mysqld.1]</b> is used to
specify options for mysqld server with suffix value
<b>".1"</b>.
</li>
<li>
The <b>[mysqltest]</b> section is used to specify options
for mysqltest client.
</li>
<li>
The <b>[ENV]</b> section is used to specify environment
variables for a test case.
</li>
<li>
To include the options mentioned in another <b>.cnf</b>
file, <b>!include path_to_cnf_file</b> is used. Path to
the included <b>.cnf</b> should either be an absolute path
or a relative path from current location or from
<b>mysql-test</b> directory.
</li>
</ul>
Here is a sample <b>.cnf</b> file.
@verbatim
!include include/default_my.cnf
[mysqld.1]
Options for server mysqld.1
[mysqld.2]
Options for server mysqld.2
[mysqltest]
ps-protocol
...
...
...
[ENV]
SERVER_MYPORT_1= @mysqld.1.port
SERVER_MYPORT_2= @mysqld.2.port
@endverbatim
</li>
<li>
A file name of the form <b><em>test_name</em>.combinations</b>
provides section of options for each test run.
</li>
<li>
A file name of the form <b><em>suite</em>.opt</b> provides
server options to all the tests within the suite it belongs
to.
If a test run is started with more than one server, the
options specified in a <b><em>suite</em>.opt</b> are used by
all of them. It is also possible to pass the server options
needed during the server initialization in the
<b><em>suite</em>.opt</b> file.
The options mentioned in the <b><em>suite</em>.opt</b> file
are overridden by those found in a
<b><em>test_name</em>-master.opt</b> file or in a
<b><em>test_name</em>-slave.opt</b> file.
</li>
<li>
A file name of the form <b><em>test_name</em>-master.sh</b> is
a shell script that will be executed before the server is
started for the named test case. There may also be a
<b><em>test_name</em>-slave.sh</b> which is executed before
the slave server is started.
These files will not work on Windows and may therefore be
replaced with a more portable mechanism in the future.
</li>
<li>
The <b>disabled.def</b> file contains information about
deferred/disabled tests. When a test is failing because of a
bug in the server and you want it to be ignored by @ref
PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl", list the test in
this file.
The format of a line in the <b>disabled.def</b> file looks
like this, where fields are separated by one or more spaces.
@verbatim
suite_name.test_name [@platform|@!platform] : <BUG|WL>#<XXXX> [<comment>]
@endverbatim
Example:
@verbatim
main.1st @linux : BUG#18980 Test fails randomly
@endverbatim
<b><em>suite_name</em></b> is the suite name.
<b><em>test_name</em></b> is the test case name.
An additional element after the test name and before the
colon you may add a <b>\@<em>platform</em></b> to have the
test disabled only on a specific platform. Basic OS names as
reported by <b>$^O</b> in Perl, or <b>'windows'</b> are
supported. Alternatively, <b>@!<em>platform</i></b> will
disable the test on all except the named platform.
<b>BUG#<em>nnnnn</em></b> or <b>WL#<em>nnnn</em></b> indicates
the bug or the WL related to the test that causes it to fail
(and thus requires it to be disabled).
The <b><em>comment</em></b> text following the
<b>BUG#<em>nnnnn</em></b> or the <b>WL#<em>nnnn</em></b> is
not part of the mandatory syntax for this file,
@ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" will not
actually care what is written here. Length of a
<b><em>comment</em></b> text must not be more than <b>80</b>
characters.
A comment line can be written in the file by beginning the
line with a “<b>#</b>” character.
</li>
</ul>
<li>
The <b>r</b> directory contains test case result files:
<ul>
<li>
A file name of the form <b><em>test_name</em>.result</b> is
the expected result for the named test case. A file
<b>r/<em>test_name</em>.result</b> is the output that
corresponds to the input in the test case file
<b>t/<em>test_name</em>.test</b>.
</li>
<li>
A file name of the form <b><em>test_name</em>.reject</b>
contains output for the named test case if the test fails
due to output mismatch (but not it it fails for other
reasons).
</li>
</ul>
For a test case that succeeds, the <b>.result</b> file
represents both the expected and actual result. For a test case
that fails, the <b>.result</b> file represents the expected
result, and the <b>.reject</b> file represents the actual
result.
If a <b>.reject</b> file is created because a test fails, @ref
PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" removes the file
later the next time the test succeeds.
When <b>`--check-testcases`</b> option is enabled, a test case
not having its corresponding <b>.result</b> file is marked as
failed by MTR. See @ref PAGE_MYSQL_TEST_RUN_PL
"mysql-test-run.pl".
</li>
<li>
The <b>include</b> directory contains files that are included by
test case files using the <b>source</b> command. These include
files encapsulate operations of varying complexity into a single
file so that you can perform the operations in a single step.
See @ref PAGE_INCLUDE_FILES.
</li>
<li>
The <b>lib</b> directory contains library files used by @ref
PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl", and database
initialization SQL code.
</li>
<li>
The <b>std_data</b> directory contains data files used by some
of the tests.
</li>
<li>
The <b>var</b> directory is used during test runs for various
kinds of files: log files, temporary files, trace files, Unix
socket files for the servers started during the tests, and so
forth. This directory cannot be shared by simultaneous test
runs.
</li>
<li>
The <b>suite</b> directory contains a set of subdirectories
containing named test suites. Each subdirectory represents a
test suite with the same name.
A test suite directory contains the following components:
<ul>
<li>
The <b>t</b> directory contains test cases.
</li>
<li>
The <b>r</b> directory contains test case result files.
</li>
<li>
The <b>include</b> directory contains files that are
included by the test cases belonging to that suite.
</li>
<li>
A file name of the form <b>combinations</b> provides section
of options for each test run. See @ref
PAGE_CONTROLLING_BINARY_LOG_FORMAT.
</li>
<li>
A file name of the form <b><em>my</em>.cnf</b> contains
additional entries for the config file used by all the tests
within the suite it belongs to.
A suite specific <b><em>my</em>.cnf</b> file is overridden
by a <b><em>test_name</em>.cnf</b> file.
</li>
</ul>
</li>
<li>
The <b>collections</b> directory contains collections of test
runs that are run during integration and release testing of
MySQL. They are not directly useful outside this context, but
need to be part of the source repository and are included for
reference.
</li>
</ul>
Unit test-related files are located in the <b>unittest</b>
directory. Additional files specific to storage engines and plugins
may be present under the subdirectories of the <b>storage</b> or
<b>plugin</b> directories.
Test Execution and Evaluation
-----------------------------
There are a number of targets in the top-level <b>Makefile</b> that
can be used to run sets of tests. <b>make test</b> only runs unit
tests. Other targets run subsets of the tests, or run tests with
specific options for the test programs. Have a look at the
<b>Makefile</b> to see what targets are available.
A “test case” is a single file. The case might contain multiple
individual test commands. If any individual command fails, the
entire test case is considered to fail. Note that “fail” means “does
not produce the expected result.” It does <b><em>not</em></b>
necessarily mean “executes without error,” because some tests are
written precisely to verify that an illegal statement does in fact
produce an error. In such an instance, if the statement executes
successfully without producing the expected error, that is
considered failure of the test.
Test case output (the test result) consists of:
<ul>
<li>
Input SQL statements and their output. Each statement is written
to the result followed by its output. Columns in output
resulting from SQL statements are separated by tab characters.
</li>
<li>
The result from @ref PAGE_MYSQLTEST "mysqltest" commands such as
<b>echo</b> and <b>exec</b>. The commands themselves are not
echoed to the result, only their output.
</li>
</ul>
The <b>disable_query_log</b> and <b>enable_query_log</b> commands
control logging of input SQL statements. The
<b>disable_result_log</b> and <b>enable_result_log</b> commands
control logging of SQL statement results, and warning or error
messages resulting from those statements.
@ref PAGE_MYSQLTEST "mysqltest" reads a test case file from its
standard input by default. The <b>`--test-file`</b> or <b>-x</b>
option can be given to name a test case file explicitly.
@ref PAGE_MYSQLTEST "mysqltest" writes test case output to the
standard output by default. The <b>`--result-file`</b> or <b>-R</b>
option can be used to indicate the location of the result file. That
option, together with the <b>`--record`</b> option, determine how
@ref PAGE_MYSQLTEST "mysqltest" treats the test actual and expected
results for a test case:
<ul>
<li>
If the test produces no results, @ref PAGE_MYSQLTEST "mysqltest"
exits with an error message to that effect, unless
<b>`--result-file`</b> is given and this file is empty.
</li>
<li>
Otherwise, if <b>`--result-file`</b> is not given, @ref
PAGE_MYSQLTEST "mysqltest" sends test results to the standard
output.
</li>
<li>
With <b>`--result-file`</b> but not <b>`--record`</b>, @ref
PAGE_MYSQLTEST "mysqltest" reads the expected results from the
given file and compares them with the actual results. If the
results do not match, @ref PAGE_MYSQLTEST "mysqltest" writes a
<b>.reject</b> file in the log directory and exits with an
error. It will also print out a diff of the expected and actual
result, provided it can find an appropriate
<b>diff</b> tool to use.
</li>
<li>
With both <b>`--result-file`</b> and <b>`--record`</b>, @ref
PAGE_MYSQLTEST "mysqltest" updates the given file by writing the
actual test results to it. The file does not need to pre-exist.
</li>
</ul>
@ref PAGE_MYSQLTEST "mysqltest" itself knows nothing of the <b>t</b>
and <b>r</b> directories under the <b>mysql-test</b> directory. The
use of files in those directories is a convention that is used by
@ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl", which invokes @ref
PAGE_MYSQLTEST "mysqltest" with the appropriate options for each
test case to tell @ref PAGE_MYSQLTEST "mysqltest" where to read input
and write output.
*/
###############################################################################
## Test Framework System Requirements
/**
@page PAGE_TEST_FRAMEWORK_SYSTEM_REQUIREMENTS Test Framework System Requirements
The @ref PAGE_MYSQLTEST "mysqltest" and @ref PAGE_MYSQL_CLIENT_TEST
"mysql_client_test" programs are written in C++ and are available on
any system where MySQL itself can be compiled, or for which a binary
MySQL distribution is available.
Other parts of the test framework such as @ref
PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" are Perl scripts and
should run on systems with Perl installed.
@ref PAGE_MYSQLTEST "mysqltest" uses the <b>diff</b> program to
compare expected and actual test results. If <b>diff</b> is not
found, @ref PAGE_MYSQLTEST "mysqltest" writes an error message and
dumps the entire contents of the <b>.result</b> and
<b>.reject</b> files so that you can try to determine why a test
did not succeed. If your system does not have <b>diff</b>, you may
be able to obtain it from one of these sites:
- http://www.gnu.org/software/diffutils/diffutils.html
- http://gnuwin32.sourceforge.net/packages/diffutils.htm
@ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" cannot function
properly if started from within a directory whose full path includes
a space character, due to the complexities of handling this
correctly in all the different contexts it will be used.
*/
###############################################################################
## How to Report Bugs from Tests in the MySQL Test Suite
/**
@page PAGE_REPORT_BUGS How to Report Bugs from Tests in the MySQL Test Suite
If test cases from the test suite fail, you should do the following:
<ul>
<li>
Do not file a bug report before you have found out as much as
possible about what when wrong. See the instructions at
https://dev.mysql.com/doc/refman/8.0/en/bug-reports.html.
</li>
<li>
Make sure to include the output of @ref PAGE_MYSQL_TEST_RUN_PL
"mysql-test-run.pl", as well as contents of all <b>.reject</b>
files in the <b>mysql-test/var/log</b> directory, or the (often
much shorter) diff that @ref PAGE_MYSQL_TEST_RUN_PL
"mysql-test-run.pl" reported.
</li>
<li>
Check whether an individual test in the test suite also fails
when run on its own:
@verbatim
shell> cd mysql-test
shell> ./mysql-test-run.pl test_name
@endverbatim
If this fails, and you are compiling MySQL yourself, you should
configure MySQL with <b>`--with-debug`</b> and run @ref
PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" with the
<b>`--debug`</b> option. If this also fails, open a bug report
and upload the trace file <b>mysql-test/var/tmp/master.trace</b>
to the report, so that we can examine it. For instructions, see
[How to Report Bugs or Problems]
(https://dev.mysql.com/doc/refman/8.0/en/bug-reports.html).
Please remember to also include a full description of your
system, the version of the <b>mysqld</b> binary and how you
compiled it.
</li>
<li>
Run @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" with the
<b>`--force`</b> option to see whether any other tests fail.
</li>
<li>
If you have compiled MySQL yourself, check the MySQL Reference
Manual to see whether there are any platform-specific issues for
your system. There might be configuration workarounds to deal
with the problems that you observe. Also, consider using one of
the binaries we have compiled for you at
https://dev.mysql.com/downloads/. All our standard binaries
should pass the test suite!
</li>
<li>
If you get an error such as <b>Result length mismatch</b> or
<b>Result content mismatch</b> it means that the output of the
test was not an exact match for the expected output. This could
be a bug in MySQL or it could be that your version of
<b>mysqld</b> produces slightly different results under some
circumstances.
This output from @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl"
should include a diff of the expected and actual result. If
unable to produce a diff, it will instead print out both in
full. In addition, the actual result is put in a file in the
<b>r</b> directory with a <b>.result</b> extension.
</li>
<li>
If a test fails completely, you should check the logs file in
the <b>mysql-test/var/log</b> directory for hints of what went
wrong.
</li>
<li>
If you have compiled MySQL with debugging, you can try to debug
test failures by running @ref PAGE_MYSQL_TEST_RUN_PL
"mysql-test-run.pl" with either or both of the <b>`--gdb`</b>
and <b>`--debug`</b> options.
If you have not compiled MySQL for debugging you should probably
do so by specifying the [-DWITH_DEBUG]
(https://dev.mysql.com/doc/refman/8.0/en/source-configuration-options.html#option_cmake_with_debug)
option when you invoke <b>CMake</b>.
</li>
</ul>
*/
###############################################################################
## Running Test Cases
/**
@page PAGE_RUNNING_TESTCASES Running Test Cases
Typically, you run the test suite either from within a source tree
(after MySQL has been built), or on a host where the MySQL server
distribution has been installed. (If you find that the test suite is
not included in your distribution, look for a similar distribution
with <b>-test</b> in the name and install that as well.)
To run tests, your current working directory should be the
<b>mysql-test</b> directory of your source tree or installed
distribution. In a source distribution, <b> mysql-test</b> is
under the root of the source tree. In a binary distribution, the
location of <b>mysql-test</b> depends on the distribution layout.
The program that runs the test suite, @ref PAGE_MYSQL_TEST_RUN_PL
"mysql-test-run.pl", will figure out whether you are in a source
tree or an installed directory tree.
To run the test suite, change location into your <b>mysql-test</b>
directory and invoke the @ref PAGE_MYSQL_TEST_RUN_PL
"mysql-test-run.pl" script:
@verbatim
shell> cd mysql-test
shell> ./mysql-test-run.pl
@endverbatim
@ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" accepts options on
the command line. For example:
@verbatim
shell> ./mysql-test-run.pl --force --suite=binlog
@endverbatim
By default, @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" exits if
a test case fails. <b>`--force`</b> causes execution to continue
regardless of test case failure.
For a full list of the supported options, see @ref
PAGE_MYSQL_TEST_RUN_PL.
To run one or more specific test cases, name them on the @ref
PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" command line. Test case
files have names like <b>t/<em>test_name</em>.test</b>, where
<b><em>test_name</em></b> is the name of the test case, but each name
given on the command line should be the test case name, not the full
test case file name. The following command runs the test case named
<b>rpl_abcd</b>, which has a test file of
<b>t/rpl_abcd.test</b>:
@verbatim
shell> ./mysql-test-run.pl rpl_abcd
@endverbatim
To run a family of test cases for which the names share a common
prefix, use the <b>`--do-test`</b> option:
@verbatim
shell> ./mysql-test-run.pl --do-test=prefix
@endverbatim
For example, the following command runs the events tests (test
cases that have names beginning with <b>events</b>):
@verbatim
shell> ./mysql-test-run.pl --do-test=events
@endverbatim
To run a specific named testsuite with all the test cases in it,
use the <b>`--suite`</b> option:
@verbatim
shell> ./mysql-test-run.pl --suite=suite_name
@endverbatim
For example, the following command runs the replication tests
located in the <b>rpl</b> suite:
@verbatim
shell> ./mysql-test-run.pl --suite=rpl
@endverbatim
@ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" starts the MySQL
server, sets up the environment for calling the @ref PAGE_MYSQLTEST
"mysqltest" program, and invokes @ref PAGE_MYSQLTEST "mysqltest" to
run the test case. For each test case to be run, @ref PAGE_MYSQLTEST
"mysqltest" handles operations such as reading input from the test
case file, creating server connections, and sending SQL statements
to servers.
The language used in test case files is a mix of commands that the
@ref PAGE_MYSQLTEST "mysqltest" program understands and SQL
statements. Input that @ref PAGE_MYSQLTEST "mysqltest" doesn't
understand is assumed to consist of SQL statements to be sent to the
database server. This makes the test case language familiar to those
that know how to write SQL and powerful enough to add the control
needed to write test cases.
You need not start a MySQL server first before running tests.
Instead, the @ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" program
will start the server or servers as needed. Any servers started for
the test run use ports in the range from 13000 by default.
@section SECTION_RUNNING_TESTS_IN_PARALLEL Running Tests in Parallel
It is possible to run more than one instance of @ref
PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" simultaneously on the
same machine. Both will by default use server ports from 13000 but
will coordinate used port numbers as well as check for availibility,
to avoid conflicts.
Running several instances from the same <b>mysql-test</b>
directory is possible but problematic. You must the use the
<b>`--vardir`</b> to set different log directories for each
instance. Even so, you can get into trouble becuse they will write
<b>.reject</b> files to the same directories.
It is also possible to have a single @ref PAGE_MYSQL_TEST_RUN_PL
"mysql-test-run.pl" run tests in several threads in parallel.
Execution of the tests will be distributed among the threads. This
is achieved using the <b>`--parallel`</b> option, with the number
of threads as argument. The special value <b>auto</b> will ask
@ref PAGE_MYSQL_TEST_RUN_PL "mysql-test-run.pl" to pick a value
automatically, based on system information. The parallel option may
also be given using the environment variable <b>MTR_PARALLEL</b>.
@note
Test cases which use <b>not_parallel.inc</b> are run at the end with
a parallel value of <b>1</b> overriding the <b>`--parallel`</b>
option value. Such test cases will be executed at the end of the test
run one at a time.
*/
###############################################################################
## Writing Test Cases
/**
@page PAGE_WRITING_TESTCASES Writing Test Cases
<h3>Table of Contents</h3>
- @subpage PAGE_QUICK_START
- @subpage PAGE_TESTCASE_CODING_GUIDELINES
- @subpage PAGE_SAMPLE_TESTCASES
- @subpage PAGE_CLEANUP_PREVIOUS_TEST_RUN
- @subpage PAGE_GENERATE_TESTCASE_RESULT_FILE
- @subpage PAGE_CHECK_EXPECTED_ERRORS
- @subpage PAGE_CONTROL_INFORMATION_PRODUCED_TESTCASE
- @subpage PAGE_DEALING_OUTPUT
- @subpage PAGE_PASSING_OPTIONS
- @subpage PAGE_TESTCASE_SPECIFIC_SERVER_OPTIONS
- @subpage PAGE_TESTCASE_SPECIFIC_BOOTSTRAP_OPTIONS
- @subpage PAGE_TESTCASE_SPECIFIC_CLIENT_OPTIONS
- @subpage PAGE_INCLUDE_FILES
- @subpage PAGE_BINARY_LOG_FORMAT
- @subpage PAGE_WRITING_REPLICATION_TESTS
- @subpage PAGE_THREAD_SYNCHRONIZATION
- @subpage PAGE_SUPPRESSING_ERRORS_WARNINGS
- @subpage PAGE_STOPPING_AND_RESTARTING_SERVER_DURING_TEST
- @subpage PAGE_TIPS_WRITING_TESTCASES
Normally, you run the test suite during the development process to
ensure that your changes do not cause existing test cases to break.
You can also write new test cases or add tests to existing cases.
This happens when you fix a bug (so that the bug cannot reappear
later without being detected) or when you add new capabilities to
the server or other MySQL programs.
This chapter provides guidelines for developing new test cases for
the MySQL test framework.
Some definitions:
- One “test file” is one “test case”.
- One “test case” might contain a “test sequence” (that is, a
number of individual tests that are grouped together in the same
test file).
- A “command” is an input test that @ref PAGE_MYSQLTEST "mysqltest"
recognizes and executes itself. A “statement” is an SQL statement
or query that @ref PAGE_MYSQLTEST "mysqltest" sends to the MySQL
server to be executed.
*/
###############################################################################
## Writing a Test Case: Quick Start
/**
@page PAGE_QUICK_START Writing a Test Case: Quick Start
The basic principle of test case evaluation is that output resulting