Merge branch 'master' into MFTF2.5.4-RC

Author: KevinBKozan

CHANGELOG.md

@@ -1,5 +1,26 @@
 Magento Functional Testing Framework Changelog
 ================================================
+2.5.3
+-----
+
+### Fixes
+* Fixed an issue where `createData` actions would cause an exception when used in `<suite>` hooks.
+
+2.5.2
+-----
+
+* Traceability
+    * Allure report enhanced to display file path of tests.        
+* Maintainability
+    * Added support to read MFTF test entities from `<magento>dev/tests/acceptance/tests/functional/<vendor_name>/<module_name>/*`
+    * Removed path deprecation warning from `ModuleResolver`.
+    * Refactored problem methods to reduce cyclomatic complexity.
+    
+### Fixes
+* Fixed issue with builds due to absence of AcceptanceTester class.
+
+### GitHub Issues/Pull requests:
+* [#348](https://github.com/magento/magento2-functional-testing-framework/pull/348) -- executeInSelenium command fixed to prevent escaping double quotes.
 
 2.5.1
 -----

bin/mftf

@@ -29,7 +29,7 @@ try {
 try {
     $application = new Symfony\Component\Console\Application();
     $application->setName('Magento Functional Testing Framework CLI');
-    $application->setVersion('2.5.1');
+    $application->setVersion('2.5.3');
     /** @var \Magento\FunctionalTestingFramework\Console\CommandListInterface $commandList */
     $commandList = new \Magento\FunctionalTestingFramework\Console\CommandList;
     foreach ($commandList->getCommands() as $command) {

composer.json

@@ -2,7 +2,7 @@
     "name": "magento/magento2-functional-testing-framework",
     "description": "Magento2 Functional Testing Framework",
     "type": "library",
-    "version": "2.5.1",
+    "version": "2.5.3",
     "license": "AGPL-3.0",
     "keywords": ["magento", "automation", "functional", "testing"],
     "config": {

composer.lock

@@ -39,7 +39,7 @@ vendor/bin/mftf generate:tests
 ### Generate tests by test name
 
 ```bash
-vendor/bin/mftf generate:tests LoginAsAdminTest LoginAsCustomerTest
+vendor/bin/mftf generate:tests AdminLoginTest StorefrontPersistedCustomerLoginTest
 ```
 
 ### Generate and run the tests for a specified group
@@ -53,7 +53,7 @@ This command cleans up the previously generated tests; generates and runs tests
 ### Generate and run particular tests
 
 ```bash
-vendor/bin/mftf run:test LoginAsAdminTest LoginAsCustomerTest -r
+vendor/bin/mftf run:test AdminLoginTest StorefrontPersistedCustomerLoginTest -r
 ```
 
 This command cleans up the previously generated tests; generates and runs the `LoginAsAdminTest` and `LoginAsCustomerTest` tests.
@@ -144,12 +144,12 @@ vendor/bin/mftf generate:tests [option] [<test name>] [<test name>] [--remove]
 
 | Option | Description|
 | ---| --- |
-| `--config=[<default> or <singleRun> or <parallel>]` | Creates a single manifest file with a list of all tests. The default location is `tests/functional/Magento/FunctionalTest/_generated/testManifest.txt`.<br/> You can split the list into multiple groups using `--config=parallel`; the groups will be generated in `_generated/groups/` like `_generated/groups/group1.txt, group2.txt, ...`.</br> Available values: `default` (default), `singleRun`(same as `default`), and `parallel`.</br> Example: `generate:tests --config=parallel`. |
+| `--config=[<default> or <singleRun> or <parallel>]` | Creates a single manifest file with a list of all tests. The default location is `tests/functional/Magento/FunctionalTest/_generated/testManifest.txt`.<br/> You can split the list into multiple groups using `--config=parallel`; the groups will be generated in `_generated/groups/` like `_generated/groups/group1.txt, group2.txt, ...`.<br/> Available values: `default` (default), `singleRun`(same as `default`), and `parallel`.<br/> Example: `generate:tests --config=parallel`. |
 | `--force` | Forces test generation, regardless of the module merge order defined in the Magento instance. Example: `generate:tests --force`. |
 | `-i,--time` | Set time in minutes to determine the group size when `--config=parallel` is used. The __default value__ is `10`. Example: `generate:tests --config=parallel --time=15`|
 | `--tests` | Defines the test configuration as a JSON string.|
 | `--allow-skipped` | Allows MFTF to generate and run tests marked with `<skip>.`|
-| `--debug or --debug=[<none>]`| Performs schema validations on XML files. <br/> DEFAULT: `generate:tests` implicitly performs schema validation on merged files. It does not indicate the file name where the error is encountered. <br/> DEVELOPER: `--debug` performs per-file validation and returns additional debug information (such as the filename where an error occurred) when test generation fails because of an invalid XML schema. This option takes extra processing time. Use it after test generation has failed once.</br><br/> NONE: `--debug=none` skips debugging during test generation. Added for backward compatibility, it will be removed in the next MAJOR release.</br>|
+| `--debug or --debug=[<none>]`| Performs schema validations on XML files. <br/> DEFAULT: `generate:tests` implicitly performs schema validation on merged files. It does not indicate the file name where the error is encountered. <br/> DEVELOPER: `--debug` performs per-file validation and returns additional debug information (such as the filename where an error occurred) when test generation fails because of an invalid XML schema. This option takes extra processing time. Use it after test generation has failed once.<br/>NONE: `--debug=none` skips debugging during test generation. Added for backward compatibility, it will be removed in the next MAJOR release.<br/>|
 | `-r,--remove`| Removes the existing generated suites and tests cleaning up the `_generated` directory before the actual run. For example, `generate:tests SampleTest --remove` cleans up the entire `_generated` directory and generates `SampleTest` only.|
 
 #### Examples of the JSON configuration
@@ -447,9 +447,9 @@ vendor/bin/mftf setup:env
 
 The example parameters are taken from the `etc/config/.env.example` file.
 
-### `static:checks`
+### `static-checks`
 
-Runs all MFTF static:checks on the test codebase that MFTF is currently attached to.
+Runs all MFTF static-checks on the test codebase that MFTF is currently attached to.
 
 #### Existing static checks
 
@@ -458,7 +458,7 @@ Runs all MFTF static:checks on the test codebase that MFTF is currently attached
 #### Usage
 
 ```bash
-vendor/bin/mftf static:checks
+vendor/bin/mftf static-checks
 ```
 
 ### `upgrade:tests`

docs/commands/mftf.md

@@ -2,8 +2,8 @@
 
 <div class="bs-callout bs-callout-info" markdown="1">
 [Find your MFTF version][] of the MFTF.
-The latest Magento 2.3 release supports MFTF 2.3.13.
-The latest Magento 2.2 release supports MFTF 2.3.8.
+The latest Magento 2.3.x release supports MFTF 2.4.5.
+The latest Magento 2.2.x release supports MFTF 2.4.5.
 </div>
 
 ## Prepare environment  {#prepare-environment}
@@ -97,6 +97,10 @@ bin/magento config:set admin/security/admin_account_sharing 1
 bin/magento config:set admin/security/use_form_key 0
 ```
 
+### Webserver configuration {#web-server-configuration}
+
+The MFTF does not support executing CLI commands if your web server points to `<MAGE_ROOT_DIR>/pub` directory as recommended in the [Installation Guide][Installation Guide docroot]. For the MFTF to execute the CLI commands, the web server must point to the Magento root directory.
+
 ### Nginx settings {#nginx-settings}
 
 If Nginx Web server is used on your development environment then **Use Web Server Rewrites** setting in **Stores** > Settings > **Configuration** > **Web** > **Search Engine Optimization** must be set to **Yes**.
@@ -340,3 +344,4 @@ allure serve dev/tests/_output/allure-results/
 [Set up a standalone MFTF]: #set-up-a-standalone-mftf
 [test suite]: suite.html
 [Find your MFTF version]: introduction.html#find-your-mftf-version
+[Installation Guide docroot]: https://devdocs.magento.com/guides/v2.3/install-gde/tutorials/change-docroot-to-pub.html

docs/getting-started.md

@@ -0,0 +1,80 @@
+# Git vs Composer installation of Magento with MFTF
+
+Depending on how you plan to use Magnto code, there are different options for installing Magento.
+
+## GitHub Installation
+
+If you are contributing a pull request to the Magento 2 codebase, download Magento 2 from our GitHub repository. Contribution to the codebase is done using the 'fork and pull' model where contributors maintain their own fork of the repo. This repo is then used to submit a pull request to the base repo.
+
+Install guide: [GitHub Installation][]
+
+## Composer based Installation
+
+A Composer install downloads released packages of Magento 2 from the composer repo [https://repo.magento.com](https://repo.magento.com).
+
+All Magento modules and their MFTF tests are put under `<vendor>` directory, for convenience of 3rd party developers. With this setup, you can keep your custom modules separate from core modules. You can also develop modules in a separate VCS repository and add them to your `composer.json` which installs them into the `vendor` directory.
+
+Install guide: [Composer based Installation][]
+
+## MFTF Installation
+
+After installing your Magento project in either of the above ways, the composer dependency `magento/magento2-functional-testing-framework` downloads and installs MFTF. MFTF is embedded in your Magento 2 installation and will cover your project with functional tests.
+
+If you want to contribute a pull request into MFTF codebase, you will need to install MFTF in the [Standalone][] mode.
+
+## Managing modules - Composer vs GitHub
+
+### Via GitHub
+
+Cloning the Magento 2 git repository is a way of installing where you do not have to worry about matching your codebase with production. Your version control system generally holds and manages your `app/code` folder and you can do manual, ad-hoc development here.
+
+### Via Composer
+
+Magento advocates the use of composer for managing modules. When you install a module through composer, it is added to `vendor/<vendor-name>/<module>`.
+
+When developing your own module or adding MFTF tests to a module, you should not edit in `vendor` because a composer update could overwrite your changes. Instead, overwrite a module under `vendor` by adding files or cloning your module-specific Git repo to `app/code/<vendor-name>/<module>`.
+
+To distribute the module and its tests, you can initialize a git repo and create a [composer package][]. In this way others will be able to download and install your module and access your tests as a composer package, in their `<vendor>` folder.
+
+## MFTF test materials location
+
+-  For GitHub installations, MFTF test materials are located in `<magento_root>/app/code/<vendor_name>/<module_name>/Test/Mftf/`. This is the directory for new tests or to maintain existing ones.
+-  For Composer-based installations, MFTF test materials are located at `<magento_root>/vendor/<vendor_name>/<module_name>/Test/Mftf/`. This is the directory to run tests fetched by Composer.
+
+The file structure under both paths is the same:
+
+```tree
+<Path>
+├── ActionGroup
+│   └── ...
+├── Data
+│   └── ...
+├── Metadata
+│   └── ...
+├── Page
+│   └── ...
+├── Section
+│   └── ...
+└── Test
+    └── ...
+```
+
+## How ModuleResolver reads modules
+
+With either type of installation, all tests and test data are read and merged by MFTF's ModuleResolver in this order:
+
+1. `<magento_root>/app/code/<vendor_name>/<module_name>/Test/Mftf/`
+1. `<magento_root>/vendor/<vendor_name>/<module_name>/Test/Mftf/`
+
+## Conclusion
+
+There is no difference between having the test materials in `app/code` or in `/vendor`: it works the same. Composer-based installs may benefit teams when there is a need to match file systems in `development` and `production`.
+
+If you are a contributing developer with an understanding of Git and Composer commands, you can choose the GitHub installation method instead.
+
+<!-- Link definitions -->
+
+[Composer based Installation]: https://devdocs.magento.com/guides/v2.3/install-gde/composer.html
+[GitHub Installation]: https://devdocs.magento.com/guides/v2.3/install-gde/prereq/dev_install.html
+[Standalone]: ../getting-started.html#set-up-a-standalone-mftf
+[composer package]: https://devdocs.magento.com/guides/v2.3/extension-dev-guide/package/package_module.html

docs/guides/git-vs-composer-install.md

@@ -1,8 +1,8 @@
 # Extend data entities
 
-Extending an action group doesn't affect the existing action group.
+Extending a data entity does not affect the existing data entity.
 
-In this example we add a `<click>` command to check the checkbox that our extension added with a new action group for the simple product creation form.
+In this example we update the quantity to 1001 and add a new piece of data relevant to our extension. Unlike merging, this will _not_ affect any other tests that use this data entity.
 
 ## Starting entity
 
@@ -67,4 +67,4 @@ Note that there are now two data entities below.
     <requiredEntity type="custom_attribute_array">CustomAttributeCategoryIds</requiredEntity>
     <data key="myExtensionData">dataHere</data>
 </entity>
-```
+```

docs/merge_points/extend-data.md

@@ -1,6 +1,6 @@
 # Extend tests
 
-Data objects can be merged to cover the needs of your extension.
+Tests can be extended to cover the needs of your extension.
 
 In this example, we add an action group to a new copy of the original test for our extension.
 
@@ -142,4 +142,4 @@ Note that there are now two tests below.
         <argument name="extensionData" value="_myData"/>
     </actionGroup>
 </test>
-```
+```

docs/merge_points/extend-tests.md

@@ -1,7 +1,7 @@
 # Suites
 
-Suites are essentially groups of tests that run in the specific conditions (preconditions and postconditions).
-They enable you including, excluding, and grouping tests for a customized test run when you need it.
+Suites are essentially groups of tests that run in specific conditions (preconditions and postconditions).
+They enable including, excluding, and grouping tests for a customized test run.
 You can form suites using separate tests, groups, and modules.
 
 Each suite must be defined in the `<VendorName>/<ModuleName>/Test/Mftf/Suite` directory.
@@ -19,7 +19,6 @@ The format of a suite:
 
 ```xml
 <?xml version="1.0" encoding="UTF-8"?>
-
 <suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Suite/etc/suiteSchema.xsd">
     <suite name="">
         <before>
@@ -43,12 +42,14 @@ The format of a suite:
 ## Principles
 
 -  A suite name:
-  -  must not match any existing group value.
+
+    -  must not match any existing group value.
   For example, the suite `<suite name="ExampleTest">` will fail during test run if any test contains in annotations `<group value="ExampleTest">`.
-  -  must not be `default` or `skip`. Tests that are not in any suite are generated under the `default` suite.
-  The suite name `skip` is synonymous to including a test in the `<group value="skip"/>`, which will be deprecated in MFTF 3.0.0.
-  -  can contain letters, numbers, and underscores.
-  -  should be upper camel case.
+    -  must not be `default` or `skip`. Tests that are not in any suite are generated under the `default` suite.
+       The suite name `skip` is synonymous to including a test in the `<group value="skip"/>`, which will be deprecated in MFTF 3.0.0.
+    -  can contain letters, numbers, and underscores.
+    -  should be upper camel case.
+    
 -  A suite must contain at least one `<include>`, or one `<exclude>`, or both.
 -  Using `<before>` in a suite, you must add the corresponding `<after>` to restore the initial state of your testing instance.
 
@@ -133,7 +134,7 @@ It performs the following steps:
 *After* the testing, the suite returns the Magento instance to the initial state disabling WYSIWYG:
 
 1. Log back in.
-2. Disable **WYSIWYG** so that
+2. Disable **WYSIWYG** in the Magento instance.
 
 This suite includes all tests that contain the `<group value="WYSIWYG"/>` annotation.
 
@@ -242,11 +243,11 @@ The element can contain [`<test>`], [`<group>`], and [`<module>`].
 
 A set of filters that you can use to specify which tests to exclude in the test suite.
 
-There two types of behavior:
+There are two types of behavior:
 
 1. Applying filters to the included tests when the suite contains [`<include>`] filters.
    The MFTF will exclude tests from the previously included set and generate the remaining tests in the suite.
-2. Applying filter to all tests when the suite does not contain [`<include>`] filters.
+2. Applying filters to all tests when the suite does not contain [`<include>`] filters.
    The MFTF will generate all existing tests except the excluded.
    In this case, the custom suite will contain all generated tests except excluded, and the _default_ suite will contain the excluded tests only.
 

docs/suite.md

@@ -35,6 +35,11 @@ public static function addAttachmentToLastStep($data, $caption)
         $rootStep = Allure::lifecycle()->getStepStorage()->getLast();
         $trueLastStep = array_last($rootStep->getSteps());
 
+        if ($trueLastStep == null) {
+            // Nothing to attach to; do not fire off allure event
+            return;
+        }
+        
         $attachmentEvent = new AddUniqueAttachmentEvent($data, $caption);
         $attachmentEvent->process($trueLastStep);
     }

src/Magento/FunctionalTestingFramework/Allure/AllureHelper.php

@@ -66,7 +66,7 @@ class TestDependencyCheck implements StaticCheckInterface
      * Array containing all errors found after running the execute() function.
      * @var array
      */
-    private $errors;
+    private $errors = [];
 
     /**
      * String representing the output summary found after running the execute() function.