From 8b39c1514e1af1a6e6de771bcb2e189e7ea8b471 Mon Sep 17 00:00:00 2001 From: Ajith Date: Thu, 30 Apr 2020 00:56:06 +0530 Subject: [PATCH 01/30] Data entity document improvement --- docs/data.md | 56 +++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 53 insertions(+), 3 deletions(-) diff --git a/docs/data.md b/docs/data.md index fa4c90dcf..6d4dc3756 100644 --- a/docs/data.md +++ b/docs/data.md @@ -7,6 +7,8 @@ The following diagram shows the XML structure of an MFTF data object: +The MFTF `` entities are stored in the following directory /Test/Mftf/Data/ + ## Supply data to test by reference to a data entity Test steps requiring `` input in an action, like filling a field with a string, may reference an attribute from a data entity: @@ -20,6 +22,20 @@ In this example: * `SimpleSubCategory` is an entity name. * `name` is a `` key of the entity. The corresponding value will be assigned to `userInput` as a result. +The following is the usage of `` entity in the `Magento/Customer/Test/Mftf/Test/AdminCustomersAllCustomersNavigateMenuTest.xml` test: + +```xml + + + + +``` + +In the above example: + +* `AdminMenuCustomers` is an entity name. +* `dataUiId` is a `` key of the entity. + ### Environmental data ```xml @@ -32,6 +48,12 @@ In this example: * `MAGENTO_ADMIN_USERNAME` is a name of an environment variable. The corresponding value will be assigned to `userInput` as a result. +The following is the usage of `_ENV` in the `Magento/Braintree/Test/Mftf/ActionGroup/AdminDeleteRoleActionGroup.xml` action group: + +```xml + +``` + ### Sensitive data ```xml @@ -47,6 +69,14 @@ In this example: Learn more in [Credentials][]. +The following is the usage of `_CREDS` in the `Magento/Braintree/Test/Mftf/Data/BraintreeData.xml` data entity: + +```xml + + {{_CREDS.magento/braintree_enabled_fraud_merchant_id}} + +``` + ## Persist a data entity as a prerequisite of a test {#persist-data} A test can specify an entity to be persisted (created in the database) so that the test actions could operate on the existing known data. @@ -63,6 +93,14 @@ In this example: * `email` is a data key of the entity. The corresponding value will be assigned to `userInput` as a result. +The following is the usage of the persistant data in `Magento/Customer/Test/Mftf/Test/AdminCreateCustomerWithCountryUSATest.xml` test: + +```xml + + + +``` +
As of MFTF 2.3.6, you no longer need to differentiate between scopes (a test, a hook, or a suite) for persisted data when referencing it in tests.
@@ -107,7 +145,7 @@ userInput="We'll email you an order confirmation with details and tracking info. ## Format -The format of `` is: +The format of `` entity is: ```xml @@ -135,7 +173,7 @@ The following conventions apply to MFTF ``: ## Example -Example (`.../Catalog/Data/CategoryData.xml` file): +Example (`Magento/Catalog/Test/Mftf/Data/CategoryData.xml` file): ```xml @@ -205,7 +243,7 @@ You can also call data from the xml definition of a `data` tag directly: Attributes|Type|Use|Description ---|---|---|--- -`name`|string|optional|Name of the ``. +`name`|string|optional|Name of the ``. Camel case is used for the entity name. `type`|string|optional|Node containing the exact name of `` type. Used later to find specific Persistence Layer Model class. `type` in `` can be whatever the user wants; There are no constraints. It is important when persisting data, depending on the `type` given, as it will try to match a metadata definition with the operation being done. Example: A `myCustomer` entity with `type="customer"`, calling ``, will try to find a metadata entry with the following attributes: ``. `deprecated`|string|optional|Used to warn about the future deprecation of the data entity. String will appear in Allure reports and console output at runtime. @@ -220,6 +258,12 @@ Attributes|Type|Use|Description `key`|string|optional|Key attribute of data/value pair. `unique`|enum: `"prefix"`, `"suffix"`|optional|Add suite or test wide unique sequence as "prefix" or "suffix" to the data value if specified. +Example: + +```xml +simpleCategory +``` + ### var {#var-tag} `` is an element that can be used to grab a key value from another entity. For example, when creating a customer with the `` action, the server responds with the auto-incremented ID of that customer. Use `` to access that ID and use it in another data entity. @@ -231,6 +275,12 @@ Attributes|Type|Use|Description `entityKey`|string|optional|Key attribute of the referenced entity from which to get a value. `unique`|--|--|*This attribute hasn't been implemented yet.* +Example: + +``` + +``` + ### requiredEntity {#requiredentity-tag} `` is an element that specifies the parent/child relationship between complex types. From c0423df1288ee65209e1cd7800e5e568ea5b96e2 Mon Sep 17 00:00:00 2001 From: Ajith Date: Tue, 12 May 2020 14:37:37 +0530 Subject: [PATCH 02/30] Minor changes in the document --- docs/data.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/data.md b/docs/data.md index 6d4dc3756..31aa1ac97 100644 --- a/docs/data.md +++ b/docs/data.md @@ -22,7 +22,7 @@ In this example: * `SimpleSubCategory` is an entity name. * `name` is a `` key of the entity. The corresponding value will be assigned to `userInput` as a result. -The following is the usage of `` entity in the `Magento/Customer/Test/Mftf/Test/AdminCustomersAllCustomersNavigateMenuTest.xml` test: +The following is an example of the usage of `` entity in the `Magento/Customer/Test/Mftf/Test/AdminCustomersAllCustomersNavigateMenuTest.xml` test: ```xml @@ -48,7 +48,7 @@ In this example: * `MAGENTO_ADMIN_USERNAME` is a name of an environment variable. The corresponding value will be assigned to `userInput` as a result. -The following is the usage of `_ENV` in the `Magento/Braintree/Test/Mftf/ActionGroup/AdminDeleteRoleActionGroup.xml` action group: +The following is an example of the usage of `_ENV` in the `Magento/Braintree/Test/Mftf/ActionGroup/AdminDeleteRoleActionGroup.xml` action group: ```xml @@ -69,7 +69,7 @@ In this example: Learn more in [Credentials][]. -The following is the usage of `_CREDS` in the `Magento/Braintree/Test/Mftf/Data/BraintreeData.xml` data entity: +The following is an example of the usage of `_CREDS` in the `Magento/Braintree/Test/Mftf/Data/BraintreeData.xml` data entity: ```xml @@ -93,7 +93,7 @@ In this example: * `email` is a data key of the entity. The corresponding value will be assigned to `userInput` as a result. -The following is the usage of the persistant data in `Magento/Customer/Test/Mftf/Test/AdminCreateCustomerWithCountryUSATest.xml` test: +The following is an example of the usage of the persistant data in `Magento/Customer/Test/Mftf/Test/AdminCreateCustomerWithCountryUSATest.xml` test: ```xml @@ -145,7 +145,7 @@ userInput="We'll email you an order confirmation with details and tracking info. ## Format -The format of `` entity is: +The format of the `` entity is: ```xml @@ -243,7 +243,7 @@ You can also call data from the xml definition of a `data` tag directly: Attributes|Type|Use|Description ---|---|---|--- -`name`|string|optional|Name of the ``. Camel case is used for the entity name. +`name`|string|optional|Name of the ``. Use camel case for entity names. `type`|string|optional|Node containing the exact name of `` type. Used later to find specific Persistence Layer Model class. `type` in `` can be whatever the user wants; There are no constraints. It is important when persisting data, depending on the `type` given, as it will try to match a metadata definition with the operation being done. Example: A `myCustomer` entity with `type="customer"`, calling ``, will try to find a metadata entry with the following attributes: ``. `deprecated`|string|optional|Used to warn about the future deprecation of the data entity. String will appear in Allure reports and console output at runtime. @@ -277,7 +277,7 @@ Attributes|Type|Use|Description Example: -``` +```xml ``` From 657aabbc7bcfc8f0e8707db14dbd8e784fe092db Mon Sep 17 00:00:00 2001 From: Donald Booth Date: Tue, 19 May 2020 11:35:59 -0500 Subject: [PATCH 03/30] Grammar and formatting --- docs/data.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/data.md b/docs/data.md index 31aa1ac97..834a0402a 100644 --- a/docs/data.md +++ b/docs/data.md @@ -7,7 +7,7 @@ The following diagram shows the XML structure of an MFTF data object: -The MFTF `` entities are stored in the following directory /Test/Mftf/Data/ +The MFTF `` entities are stored in `/Test/Mftf/Data/`. ## Supply data to test by reference to a data entity From 6bf46e5b762d61018338b336372281a076539d8f Mon Sep 17 00:00:00 2001 From: Donald Booth Date: Wed, 8 Jul 2020 13:25:15 -0500 Subject: [PATCH 04/30] Remove refs to Test Migration repo --- docs/best-practices.md | 3 --- docs/introduction.md | 7 ------- v2 | 1 + 3 files changed, 1 insertion(+), 10 deletions(-) create mode 160000 v2 diff --git a/docs/best-practices.md b/docs/best-practices.md index 613f398ca..b80542d9f 100644 --- a/docs/best-practices.md +++ b/docs/best-practices.md @@ -40,8 +40,6 @@ We recommend to keep Action Groups having single responsibility, for example `Ad Althought the Magento Core team and Contributors join forces to cover most of the features with tests, it is impossible to have this done quickly. If you've covered Magento Core feature with Functional Tests - you are more than welcome to contribute. -You can also help with MFTF Test Migration to get the experience and valuable feedback from other community members and maintainers. - ## Action group 1. [Action group] names should be sufficiently descriptive to inform a test writer of what the action group does and when it should be used. Add additional explanation in annotations if needed. @@ -214,4 +212,3 @@ Since the configurable product module could be disabled, this approach is more r [merging]: merging.html [parameterized selectors]: section/parameterized-selectors.html [sections]: section.html -[MFTF Test Migration]: https://github.com/magento/magento-functional-tests-migration diff --git a/docs/introduction.md b/docs/introduction.md index 53b763d8c..2c4ded6ce 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -52,12 +52,6 @@ Test └── ... ``` -
- -We are actively developing functional tests. Check out the [MFTF Test Migration][] repository. - -
- ## Use cases - Contributor: changes the core behaviour, fixing the annoing bug. @@ -122,4 +116,3 @@ Follow the [MFTF project] and [contribute on Github]. [Functional Testing Framework]: https://devdocs.magento.com/guides/v2.3/mtf/mtf_introduction.html [MFTF project]: https://github.com/magento/magento2-functional-testing-framework [Find your MFTF version]: #find-your-mftf-version -[MFTF Test Migration]: https://github.com/magento/magento-functional-tests-migration diff --git a/v2 b/v2 new file mode 160000 index 000000000..3343a82b8 --- /dev/null +++ b/v2 @@ -0,0 +1 @@ +Subproject commit 3343a82b84e5b25d30e6e39fb3d1aa0d8dc70634 From a37ebc4f20e85979379e6490f0a65b8d777740b4 Mon Sep 17 00:00:00 2001 From: Donald Booth Date: Wed, 8 Jul 2020 13:46:05 -0500 Subject: [PATCH 05/30] Remove refs to Test Migration repo --- docs/best-practices.md | 3 --- docs/introduction.md | 7 ------- 2 files changed, 10 deletions(-) diff --git a/docs/best-practices.md b/docs/best-practices.md index 613f398ca..b80542d9f 100644 --- a/docs/best-practices.md +++ b/docs/best-practices.md @@ -40,8 +40,6 @@ We recommend to keep Action Groups having single responsibility, for example `Ad Althought the Magento Core team and Contributors join forces to cover most of the features with tests, it is impossible to have this done quickly. If you've covered Magento Core feature with Functional Tests - you are more than welcome to contribute. -You can also help with MFTF Test Migration to get the experience and valuable feedback from other community members and maintainers. - ## Action group 1. [Action group] names should be sufficiently descriptive to inform a test writer of what the action group does and when it should be used. Add additional explanation in annotations if needed. @@ -214,4 +212,3 @@ Since the configurable product module could be disabled, this approach is more r [merging]: merging.html [parameterized selectors]: section/parameterized-selectors.html [sections]: section.html -[MFTF Test Migration]: https://github.com/magento/magento-functional-tests-migration diff --git a/docs/introduction.md b/docs/introduction.md index 53b763d8c..2c4ded6ce 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -52,12 +52,6 @@ Test └── ... ``` -
- -We are actively developing functional tests. Check out the [MFTF Test Migration][] repository. - -
- ## Use cases - Contributor: changes the core behaviour, fixing the annoing bug. @@ -122,4 +116,3 @@ Follow the [MFTF project] and [contribute on Github]. [Functional Testing Framework]: https://devdocs.magento.com/guides/v2.3/mtf/mtf_introduction.html [MFTF project]: https://github.com/magento/magento2-functional-testing-framework [Find your MFTF version]: #find-your-mftf-version -[MFTF Test Migration]: https://github.com/magento/magento-functional-tests-migration From b195677abe2e4a1cbe551c55a004be8455069697 Mon Sep 17 00:00:00 2001 From: Tom Reece Date: Mon, 13 Jul 2020 13:43:35 -0500 Subject: [PATCH 06/30] MQE-2214: Part 1 Docs Annual Review --- docs/configuration.md | 16 ++++++++-------- docs/credentials.md | 18 +++++++++--------- docs/data.md | 4 ++-- docs/extending.md | 2 +- docs/getting-started.md | 36 ++++++++++++++++-------------------- docs/introduction.md | 22 +++++++--------------- 6 files changed, 43 insertions(+), 55 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index b16f85d8f..f6fffaf6e 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,12 +1,12 @@ # Configuration The `*.env` file provides additional configuration for the Magento Functional Testing Framework (MFTF). -To run the MFTF on your Magento instance, specify the basic configuration values. +To run MFTF on your Magento instance, specify the basic configuration values. Advanced users can create custom configurations based on requirements and environment. ## Basic configuration -These basic configuration values are __required__ and must be set by the user before the MFTF can function correctly. +These basic configuration values are __required__ and must be set by the user before MFTF can function correctly. ### MAGENTO_BASE_URL @@ -34,7 +34,7 @@ MAGENTO_BACKEND_NAME=admin_12346 ### MAGENTO_BACKEND_BASE_URL -(Optional) If you are running the Admin Panel on separate a domain, specify this value: +(Optional) If you are running the Admin Panel on a separate domain, specify this value: Example: @@ -64,7 +64,7 @@ MAGENTO_ADMIN_PASSWORD=1234reTyt%$7 ## Advanced configuration -Depending on the environment you use, you may need to configure the MFTF more precisely by setting more configuration parameters then for basic configuration. +Depending on the environment you use, you may need to configure MFTF more precisely by setting additional configuration parameters. This section describes available configuration parameters and their default values (where applicable). ### DEFAULT_TIMEZONE @@ -173,8 +173,8 @@ MAGENTO_RESTAPI_SERVER_PORT=5000 ### \*_BP Settings to override base paths for the framework. -You can use it when the MFTF is applied as a separate tool. -For example, when you need to place the MFTF and the Magento codebase in separate projects. +You can use it when MFTF is applied as a separate tool. +For example, when you need to place MFTF and the Magento codebase in separate projects. ```conf MAGENTO_BP @@ -227,7 +227,7 @@ TESTS_MODULE_PATH=~/magento2/dev/tests/acceptance/tests/functional/Magento/Funct ### MODULE_ALLOWLIST Use for a new module. -When adding a new directory at `Magento/FunctionalTest`, add the directory name to `MODULE_ALLOWLIST` to enable the MFTF to process it. +When adding a new directory at `Magento/FunctionalTest`, add the directory name to `MODULE_ALLOWLIST` to enable MFTF to process it. Example: @@ -245,7 +245,7 @@ It points to `MAGENTO_BASE_URL` + `dev/tests/acceptance/utils/command.php` Modify the default value: - for non-default Magento installation -- when use a subdirectory in the `MAGENTO_BASE_URL` +- when using a subdirectory in the `MAGENTO_BASE_URL` Example: `dev/tests/acceptance/utils/command.php` diff --git a/docs/credentials.md b/docs/credentials.md index 402030985..0c6decb41 100644 --- a/docs/credentials.md +++ b/docs/credentials.md @@ -3,7 +3,7 @@ When you test functionality that involves external services such as UPS, FedEx, PayPal, or SignifyD, use the MFTF credentials feature to hide sensitive [data][] like integration tokens and API keys. -Currently the MFTF supports three types of credential storage: +Currently MFTF supports three types of credential storage: - **.credentials file** - **HashiCorp Vault** @@ -11,12 +11,12 @@ Currently the MFTF supports three types of credential storage: ## Configure File Storage -The MFTF creates a sample file for credentials during [initial setup][]: `magento2/dev/tests/acceptance/.credentials.example`. +MFTF creates a sample file for credentials during [initial setup][]: `magento2/dev/tests/acceptance/.credentials.example`. The file contains an example list of keys for fields that can require credentials. ### Create `.credentials` -To make the MFTF process the file with credentials, in the command line, navigate to `magento2/dev/tests/acceptance/` and rename `.credentials.example` to `.credentials`. +To make MFTF process the file with credentials, in the command line, navigate to `magento2/dev/tests/acceptance/` and rename `.credentials.example` to `.credentials`. ```bash cd dev/tests/acceptance/ @@ -78,7 +78,7 @@ vendor/my_awesome_service_token=rRVSVnh3cbDsVG39oTMz4A Hashicorp vault secures, stores, and tightly controls access to data in modern computing. It provides advanced data protection for your testing credentials. -The MFTF works with both `vault enterprise` and `vault open source` that use `KV Version 2` secret engine. +MFTF works with both `vault enterprise` and `vault open source` that use `KV Version 2` secret engine. ### Install vault CLI @@ -92,11 +92,11 @@ Authenticate to vault server via the vault CLI tool: [Login Vault][Login Vault]. vault login -method -path ``` -**Do not** use `-no-store` command option, as the MFTF will rely on the persisted token in the token helper (usually the local filesystem) for future API requests. +**Do not** use `-no-store` command option, as MFTF will rely on the persisted token in the token helper (usually the local filesystem) for future API requests. ### Store secrets in vault -The MFTF uses the `KV Version 2` secret engine for secret storage. +MFTF uses the `KV Version 2` secret engine for secret storage. More information for working with `KV Version 2` can be found in [Vault KV2][Vault KV2]. #### Secrets path and key convention @@ -225,7 +225,7 @@ export CREDENTIAL_AWS_ACCOUNT_ID= ## Configure multiple credential storage It is possible and sometimes useful to setup and use multiple credential storage at the same time. -In this case, the MFTF tests are able to read secret data at runtime from all storage options, in this case MFTF use the following precedence: +In this case, the MFTF tests are able to read secret data at runtime from all storage options. MFTF will use the following precedence: ``` .credentials File > HashiCorp Vault > AWS Secrets Manager @@ -253,7 +253,7 @@ For example, to reference secret data in the [`fillField`][] action, use the `us ## Implementation details The generated tests do not contain credentials values. -The MFTF dynamically retrieves, encrypts, and decrypts the sensitive data during test execution. +MFTF dynamically retrieves, encrypts, and decrypts the sensitive data during test execution. Decrypted credentials do not appear in the console, error logs, or [test reports][]. The decrypted values are only available in the `.credentials` file or within vault. @@ -277,4 +277,4 @@ The MFTF tests delivered with Magento application do not use credentials and do [`CREDENTIAL_AWS_SECRETS_MANAGER_PROFILE`]: configuration.md#credential_aws_secrets_manager_profile [`CREDENTIAL_AWS_SECRETS_MANAGER_REGION`]: configuration.md#credential_aws_secrets_manager_region [Key Management Service]: https://aws.amazon.com/kms/ -[Amazon Resource Name]: https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html \ No newline at end of file +[Amazon Resource Name]: https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html diff --git a/docs/data.md b/docs/data.md index 8b0a86e3d..33b833504 100644 --- a/docs/data.md +++ b/docs/data.md @@ -1,6 +1,6 @@ # Input testing data -The MFTF enables you to specify and use `` entities defined in XML. Default `` entities are provided for use and as templates for entity creation and manipulation. +MFTF enables you to specify and use `` entities defined in XML. Default `` entities are provided for use and as templates for entity creation and manipulation. The following diagram shows the XML structure of an MFTF data object: ![MFTF Data Object](img/data-dia.svg) @@ -67,7 +67,7 @@ In this example: As of MFTF 2.3.6, you no longer need to differentiate between scopes (a test, a hook, or a suite) for persisted data when referencing it in tests. -The MFTF now stores the persisted data and attempts to retrieve it using the combination of `stepKey` and the scope of where it has been called. +MFTF now stores the persisted data and attempts to retrieve it using the combination of `stepKey` and the scope of where it has been called. The current scope is preferred, then widening to _test > hook > suite_ or _hook > test > suite_. This emphasizes the practice for the `stepKey` of `createData` to be descriptive and unique, as a duplicated `stepKey` in both a `` and `` prefers the `` data. diff --git a/docs/extending.md b/docs/extending.md index 064ed3208..38f5c047d 100644 --- a/docs/extending.md +++ b/docs/extending.md @@ -115,7 +115,7 @@ __Use case__: Create two similar tests where the second test contains two additi ``` -### Update a test annotation +### Update a test before hook __Use case__: Create two similar tests where the second one contains two additional actions in the `before` hook: diff --git a/docs/getting-started.md b/docs/getting-started.md index ec0afba0d..7b982b379 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,9 +1,9 @@ # Getting started
-[Find your MFTF version][] of the MFTF. -The latest Magento 2.3.x release supports MFTF 2.5.3. -The latest Magento 2.2.x release supports MFTF 2.4.5. +[Find your version] of MFTF. +The latest Magento 2.3.x release supports MFTF 2.6.4. +The latest Magento 2.2.x release supports MFTF 2.5.3.
## Prepare environment {#prepare-environment} @@ -64,7 +64,7 @@ A Selenium web driver cannot enter data to fields with WYSIWYG. To disable the WYSIWYG and enable the web driver to process these fields as simple text areas: 1. Log in to the Magento Admin as an administrator. -2. Navigate to **Stores** > Settings > **Configuration** > **General** > **Content Management**. +2. Navigate to **Stores** > **Settings** > **Configuration** > **General** > **Content Management**. 3. In the WYSIWYG Options section set the **Enable WYSIWYG Editor** option to **Disabled Completely**. 4. Click **Save Config**. @@ -88,7 +88,7 @@ When you want to test the WYSIWYG functionality, re-enable WYSIWYG in your test To enable the **Admin Account Sharing** setting, to avoid unpredictable logout during a testing session, and disable the **Add Secret Key in URLs** setting, to open pages using direct URLs: -1. Navigate to **Stores** > Settings > **Configuration** > **Advanced** > **Admin** > **Security**. +1. Navigate to **Stores** > **Settings** > **Configuration** > **Advanced** > **Admin** > **Security**. 2. Set **Admin Account Sharing** to **Yes**. 3. Set **Add Secret Key to URLs** to **No**. 4. Click **Save Config**. @@ -115,7 +115,7 @@ If the Magento instance under test has the [Magento Two-Factor Authentication (2 ### Webserver configuration {#web-server-configuration} -The MFTF does not support executing CLI commands if your web server points to `/pub` directory as recommended in the [Installation Guide][Installation Guide docroot]. For the MFTF to execute the CLI commands, the web server must point to the Magento root directory. +MFTF does not support executing CLI commands if your web server points to `/pub` directory as recommended in the [Installation Guide][Installation Guide docroot]. For MFTF to execute the CLI commands, the web server must point to the Magento root directory. ### Nginx settings {#nginx-settings} @@ -137,11 +137,11 @@ location ~* ^/dev/tests/acceptance/utils($|/) { ## Set up an embedded MFTF {#setup-framework} -This is the default setup of the MFTF that you would need to cover your Magento project with functional tests. +This is the default setup of MFTF that you would need to cover your Magento project with functional tests. It installs the framework using an existing Composer dependency such as `magento/magento2-functional-testing-framework`. -If you want to set up the MFTF as a standalone tool, refer to [Set up a standalone MFTF][]. +If you want to set up MFTF as a standalone tool, refer to [Set up a standalone MFTF][]. -Install the MFTF. +Install MFTF. ```bash composer install @@ -204,7 +204,7 @@ Learn more about environmental settings in [Configuration][]. ### Step 3. Enable the Magento CLI commands -In the Magento project root, run the following command to enable the MFTF to send Magento CLI commands to your Magento instance. +In the Magento project root, run the following command to enable MFTF to send Magento CLI commands to your Magento instance. ```bash cp dev/tests/acceptance/.htaccess.sample dev/tests/acceptance/.htaccess @@ -251,9 +251,7 @@ See more commands in [`mftf`][]. ### Step 5. Generate reports {#reports} -During testing, the MFTF generates test reports in CLI. -You can generate visual representations of the report data using [Allure Framework][]. -To view the reports in GUI: +During testing, MFTF generates test reports in CLI. You can generate visual representations of the report data using the [Allure Framework][]. To view the reports in a GUI: - [Install Allure][] - Run the tool to serve the artifacts in `dev/tests/acceptance/tests/_output/allure-results/`: @@ -266,18 +264,16 @@ Learn more about Allure in the [official documentation][allure docs]. ## Set up a standalone MFTF -The MFTF is a root level Magento dependency, but it is also available for use as a standalone application. -You may want to use a standalone application when you develop for or contribute to MFTF, which facilitates debugging and tracking changes. -These guidelines demonstrate how to set up and run Magento acceptance functional tests using standalone MFTF. +MFTF is a root level Magento dependency, but it is also available for use as a standalone application. You may want to use a standalone application when you develop for or contribute to MFTF, which facilitates debugging and tracking changes. These guidelines demonstrate how to set up and run Magento acceptance functional tests using standalone MFTF. ### Prerequisites This installation requires a local instance of the Magento application. -The MFTF uses the [tests from Magento modules][mftf tests] as well as the `app/autoload.php` file. +MFTF uses the [tests from Magento modules][mftf tests] as well as the `app/autoload.php` file. ### Step 1. Clone the MFTF repository -If you develop or contribute to the MFTF, it makes sense to clone your fork of the MFTF repository. +If you develop or contribute to MFTF, it makes sense to clone your fork of the MFTF repository. For contribution guidelines, refer to the [Contribution Guidelines for the Magento Functional Testing Framework][contributing]. ### Step 2. Install the MFTF @@ -307,7 +303,7 @@ Create the `utils/` directory, if you didn't find it. ### Step 6. Remove the MFTF package dependency in Magento -The MFTF uses the Magento `app/autoload.php` file to read Magento modules. +MFTF uses the Magento `app/autoload.php` file to read Magento modules. The MFTF dependency in Magento supersedes the standalone registered namespaces unless it is removed at a Composer level. ```bash @@ -355,6 +351,6 @@ allure serve dev/tests/_output/allure-results/ [selenium server]: https://www.seleniumhq.org/download/ [Set up a standalone MFTF]: #set-up-a-standalone-mftf [test suite]: suite.html -[Find your MFTF version]: introduction.html#find-your-mftf-version +[Find your version]: introduction.html#find-your-mftf-version [Installation Guide docroot]: https://devdocs.magento.com/guides/v2.3/install-gde/tutorials/change-docroot-to-pub.html [Magento Two-Factor Authentication (2FA) extension]: https://devdocs.magento.com/guides/v2.3/security/two-factor-authentication.html diff --git a/docs/introduction.md b/docs/introduction.md index 5a75777af..5a7869381 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -5,15 +5,15 @@ These are the docs for the latest MFTF release. To find older documentation, please refer to the [docs folder] of your desired release in Github. -[Find your MFTF version][] of the MFTF. +[Find your version] of MFTF. The Magento Functional Testing Framework (MFTF) is a framework used to perform automated end-to-end functional testing. ## Goals - To facilitate functional testing and minimize the effort it takes to perform regression testing. -- Enable extension developers to provide the Functional Tests to offered extensions. -- Ensuring a common standard of quality between Magento, Extension Developers and System Intergrators. +- Enable extension developers to provide functional tests for their extensions. +- Ensure a common standard of quality between Magento, extension developers and system integrators. MFTF also focuses on @@ -31,7 +31,7 @@ MFTF also focuses on ## MFTF tests -The MFTF supports two different locations for storing the tests and test artifacts: +MFTF supports two different locations for storing the tests and test artifacts: - `/app/code///Test/Mftf/` is the location of local, customized tests. - `/vendor///Test/Mftf/` is location of tests provided by Magento and vendors. @@ -40,7 +40,7 @@ If you installed Magento with Composer, please refer to `vendor/magento/ - -We are actively developing functional tests. Check out the [MFTF Test Migration][] repository. - - - ## Use cases - Contributor: changes the core behaviour, fixing the annoing bug. @@ -130,7 +124,7 @@ codeception.dist.yml // Codeception configuration (generated while ru ## MFTF tests -The MFTF supports three different locations for storing the tests and test artifacts: +MFTF supports three different locations for storing the tests and test artifacts: - `/app/code///Test/Mftf/` is the directory to create new tests. - `/vendor///Test/Mftf/` is the directory with the out of the box tests (fetched by the Composer). - `/dev/tests/acceptance/tests/functional///` is used to store tests that depend on multiple modules. @@ -163,8 +157,6 @@ Follow the [MFTF project] and [contribute on Github]. [contribute on Github]: https://github.com/magento/magento2-functional-testing-framework/blob/master/.github/CONTRIBUTING.md -[Functional Testing Framework]: https://devdocs.magento.com/guides/v2.3/mtf/mtf_introduction.html [MFTF project]: https://github.com/magento/magento2-functional-testing-framework -[Find your MFTF version]: #find-your-mftf-version -[MFTF Test Migration]: https://github.com/magento/magento-functional-tests-migration +[Find your version]: #find-your-mftf-version [docs folder]: https://github.com/magento/magento2-functional-testing-framework/tree/master/docs From 20498b78f406ecd06ab18f9293f945008080064d Mon Sep 17 00:00:00 2001 From: Tom Reece Date: Thu, 16 Jul 2020 11:22:17 -0500 Subject: [PATCH 07/30] MQE-2214: Part 1 Docs Annual Review --- docs/configuration.md | 4 ++-- docs/extending.md | 22 +++++++++++----------- docs/getting-started.md | 8 ++++---- 3 files changed, 17 insertions(+), 17 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index f6fffaf6e..ce3adadd7 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -221,13 +221,13 @@ The path to where the MFTF modules mirror Magento modules. Example: ```conf -TESTS_MODULE_PATH=~/magento2/dev/tests/acceptance/tests/functional/Magento/FunctionalTest +TESTS_MODULE_PATH=~/magento2/dev/tests/acceptance/tests/functional/Magento ``` ### MODULE_ALLOWLIST Use for a new module. -When adding a new directory at `Magento/FunctionalTest`, add the directory name to `MODULE_ALLOWLIST` to enable MFTF to process it. +When adding a new directory at `tests/functional/Magento`, add the directory name to `MODULE_ALLOWLIST` to enable MFTF to process it. Example: diff --git a/docs/extending.md b/docs/extending.md index 38f5c047d..ecfc2018a 100644 --- a/docs/extending.md +++ b/docs/extending.md @@ -177,14 +177,14 @@ Extend an [action group] to add or update [actions] in your module. ### Update an action -__Use case__: The `CountProductA` test counts the particular product. +__Use case__: The `AssertAdminCountProductActionGroup` action group counts the particular product. Modify the action group to use another product. > Action groups with "extends": ```xml - + @@ -195,7 +195,7 @@ Modify the action group to use another product. - + @@ -205,7 +205,7 @@ Modify the action group to use another product. ```xml - + @@ -216,7 +216,7 @@ Modify the action group to use another product. - + @@ -231,21 +231,21 @@ Modify the action group to use another product. ### Add an action -__Use case__: The `GetProductCount` action group returns the count of products. -Add a new test `VerifyProductCount` that asserts the count of products: +__Use case__: The `AdminGetProductCountActionGroup` action group returns the count of products. +Add a new test `AssertAdminVerifyProductCountActionGroup` that asserts the count of products: > Action groups with "extends": ```xml - + - + @@ -261,14 +261,14 @@ Add a new test `VerifyProductCount` that asserts the count of products: ```xml - + - + diff --git a/docs/getting-started.md b/docs/getting-started.md index 7b982b379..af4f8d62e 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -44,7 +44,7 @@ cd magento2/ ``` ```bash -git checkout 2.3-develop +git checkout 2.4-develop ``` Install the Magento application. @@ -241,10 +241,10 @@ See more commands in [`codecept`][]. #### Run a simple test {#run-test} -To clean up the previously generated tests, and then generate and run a single test `AdminLoginTest`, run: +To clean up the previously generated tests, and then generate and run a single test `AdminLoginSuccessfulTest`, run: ```bash -vendor/bin/mftf run:test AdminLoginTest --remove +vendor/bin/mftf run:test AdminLoginSuccessfulTest --remove ``` See more commands in [`mftf`][]. @@ -315,7 +315,7 @@ composer remove magento/magento2-functional-testing-framework --dev -d Date: Thu, 16 Jul 2020 14:17:06 -0500 Subject: [PATCH 08/30] MQE-2214: Part 1 Docs Annual Review --- docs/extending.md | 130 +++++++++++++++++----------------------------- 1 file changed, 49 insertions(+), 81 deletions(-) diff --git a/docs/extending.md b/docs/extending.md index ecfc2018a..fe06d9637 100644 --- a/docs/extending.md +++ b/docs/extending.md @@ -21,25 +21,19 @@ Unlike merging, the parent test (or action group) will still exist after the tes -__Use case__: Create two similar tests with different `url` (`"{{AdminCategoryPage.url}}"` and `"{{OtherCategoryPage.url}}"`) in a test step. +__Use case__: Create two similar tests with a different action group reference by overwriting a `stepKey`. > Test with "extends": ```xml - - - ... - - ...(several steps) - - ...(several steps) + + + + - - - ... - - + + ``` @@ -48,46 +42,35 @@ __Use case__: Create two similar tests with different `url` (`"{{AdminCategoryPa ```xml - - - ... - - ...(several steps) - - ...(several steps) + + + + - - - ... - - ...(several steps) - - ...(several steps) + + + + ``` ### Add a test step -__Use case__: Create two similar tests where the second test contains two additional steps: - -* `checkOption` before `click` (`stepKey="clickLogin"`) -* `seeInCurrentUrl` after `click` in the `LogInAsAdminTest` test (in the `.../Backend/Test/LogInAsAdminTest.xml` file) +__Use case__: Create two similar tests where the second test contains two additional steps specified to occur `before` or `after` other `stepKeys`. > Tests with "extends": ```xml - - - - - - + + + + - - - + + + ``` @@ -96,49 +79,39 @@ __Use case__: Create two similar tests where the second test contains two additi ```xml - - - - - - + + + + - - - - - - - - + + + + + + ``` ### Update a test before hook -__Use case__: Create two similar tests where the second one contains two additional actions in the `before` hook: - -* `checkOption` before `click` (`stepKey="clickLogin"`) -* `seeInCurrentUrl` after `click` in the `LogInAsAdminTest` test (in the `.../Backend/Test/LogInAsAdminTest.xml` file) +__Use case__: Create two similar tests where the second test contains an additional action in the `before` hook. > Tests with "extends": ```xml - + - - - - + - + + - + - - + @@ -148,25 +121,20 @@ __Use case__: Create two similar tests where the second one contains two additio ```xml - + - - - - + - + + - + - - - - - - + + - + + ``` From 8e42fda3a0c0974947f3034d6baf67d5a743cc40 Mon Sep 17 00:00:00 2001 From: Ji Lu Date: Mon, 20 Jul 2020 13:09:25 -0500 Subject: [PATCH 09/30] Mqe 2215: Docs Annual Review Part 2 (#760) * MQE-2215: Docs Annual Review * MQE-2215: Docs Annual Review --- docs/merge_points/extend-action-groups.md | 10 +++++----- docs/merge_points/extend-data.md | 2 +- docs/merge_points/extend-tests.md | 14 +++++++------- docs/merge_points/merge-action-groups.md | 8 ++++---- docs/merge_points/merge-sections.md | 7 +++---- docs/merge_points/merge-tests.md | 8 ++++---- 6 files changed, 24 insertions(+), 25 deletions(-) diff --git a/docs/merge_points/extend-action-groups.md b/docs/merge_points/extend-action-groups.md index 95d6cff90..6e7a54d70 100644 --- a/docs/merge_points/extend-action-groups.md +++ b/docs/merge_points/extend-action-groups.md @@ -9,7 +9,7 @@ In this example we add a `` command to check the checkbox that our extens ```xml - + @@ -34,10 +34,10 @@ In this example we add a `` command to check the checkbox that our extens ``` -## File to merge +## Extend file ```xml - + @@ -48,7 +48,7 @@ In this example we add a `` command to check the checkbox that our extens Note that there are now two action groups below. ```xml - + @@ -71,7 +71,7 @@ Note that there are now two action groups below. - + diff --git a/docs/merge_points/extend-data.md b/docs/merge_points/extend-data.md index ebac16108..91d2a67bb 100644 --- a/docs/merge_points/extend-data.md +++ b/docs/merge_points/extend-data.md @@ -23,7 +23,7 @@ In this example we update the quantity to 1001 and add a new piece of data relev ``` -## File to merge +## Extend file ```xml diff --git a/docs/merge_points/extend-tests.md b/docs/merge_points/extend-tests.md index 6aab50764..763977049 100644 --- a/docs/merge_points/extend-tests.md +++ b/docs/merge_points/extend-tests.md @@ -25,8 +25,8 @@ In this example, we add an action group to a new copy of the original test for o - - + + @@ -40,7 +40,7 @@ In this example, we add an action group to a new copy of the original test for o ``` -## File to merge +## Extend file ```xml @@ -89,8 +89,8 @@ Note that there are now two tests below. - - + + @@ -120,8 +120,8 @@ Note that there are now two tests below. - - + + diff --git a/docs/merge_points/merge-action-groups.md b/docs/merge_points/merge-action-groups.md index 3a4f70ab1..ca6d20e68 100644 --- a/docs/merge_points/merge-action-groups.md +++ b/docs/merge_points/merge-action-groups.md @@ -1,7 +1,7 @@ # Merge action groups An action group is a set of individual actions working together as a group. -These action groups can be shared between tests and they also be modified to your needs. +These action groups can be shared between tests and they also can be modified to your needs. In this example we add a `` command to check the checkbox that our extension adds to the simple product creation form. @@ -10,7 +10,7 @@ In this example we add a `` command to check the checkbox that our extens ```xml - + @@ -38,7 +38,7 @@ In this example we add a `` command to check the checkbox that our extens ## File to merge ```xml - + @@ -47,7 +47,7 @@ In this example we add a `` command to check the checkbox that our extens ## Resultant test ```xml - + diff --git a/docs/merge_points/merge-sections.md b/docs/merge_points/merge-sections.md index dbd6e3828..135b3f7d8 100644 --- a/docs/merge_points/merge-sections.md +++ b/docs/merge_points/merge-sections.md @@ -9,7 +9,7 @@ In this example we add another selector to the section on the products page sect ```xml -
+
@@ -22,7 +22,7 @@ In this example we add another selector to the section on the products page sect ## File to merge ```xml -
+
@@ -31,7 +31,7 @@ In this example we add another selector to the section on the products page sect ## Resultant section ```xml -
+
@@ -41,7 +41,6 @@ In this example we add another selector to the section on the products page sect
- ``` \ No newline at end of file diff --git a/docs/merge_points/merge-tests.md b/docs/merge_points/merge-tests.md index 3958410e9..b9e50d296 100644 --- a/docs/merge_points/merge-tests.md +++ b/docs/merge_points/merge-tests.md @@ -25,8 +25,8 @@ In this example we add an action group that modifies the original test to intera - - + + @@ -77,8 +77,8 @@ In this example we add an action group that modifies the original test to intera - - + + From ddf5564d190d6c83b2c0e36a9dd370d8a6c01a19 Mon Sep 17 00:00:00 2001 From: soumyau Date: Tue, 21 Jul 2020 16:40:40 -0500 Subject: [PATCH 10/30] MQE-2216: Annual Docs Review Part 3 (#761) * MQE-2216: Part 3 Docs Annual Review Updated documentation based on 3.0.0 functionality. * MQE-2216: Part 3 Docs Annual Review * MQE-2216: Part 3 Docs Annual Review * MQE-2216: Annual Docs Review Part 3 --- docs/commands/codeception.md | 72 +++++----- docs/commands/mftf.md | 14 +- docs/merging.md | 257 +++++++++++++++++------------------ docs/metadata.md | 32 ++--- docs/page.md | 14 +- docs/reporting.md | 186 ++++++++++--------------- 6 files changed, 267 insertions(+), 308 deletions(-) diff --git a/docs/commands/codeception.md b/docs/commands/codeception.md index 6a96a89c7..8e6d76c11 100644 --- a/docs/commands/codeception.md +++ b/docs/commands/codeception.md @@ -1,7 +1,7 @@ # CLI commands: vendor/bin/codecept
-We do not recommend using Codeception commands directly as they can break the MFTF basic workflow. +We do not recommend using Codeception commands directly as they can break MFTF basic workflow. All the Codeception commands you need are wrapped using the [mftf tool][]. To run the Codeception testing framework commands directly, change your directory to the ``. @@ -36,7 +36,7 @@ vendor/bin/codecept run ```
-The following documentation corresponds to Codeception 2.3.8. +The following documentation corresponds to Codeception 4.1.4.
```bash @@ -47,36 +47,44 @@ Arguments: test test to be run Options: - -o, --override=OVERRIDE Override config values (multiple values allowed) - --config (-c) Use custom path for config - --report Show output in compact style - --html Generate html with results (default: "report.html") - --xml Generate JUnit XML Log (default: "report.xml") - --tap Generate Tap Log (default: "report.tap.log") - --json Generate Json Log (default: "report.json") - --colors Use colors in output - --no-colors Force no colors in output (useful to override config file) - --silent Only outputs suite names and final results - --steps Show steps in output - --debug (-d) Show debug and scenario output - --coverage Run with code coverage (default: "coverage.serialized") - --coverage-html Generate CodeCoverage HTML report in path (default: "coverage") - --coverage-xml Generate CodeCoverage XML report in file (default: "coverage.xml") - --coverage-text Generate CodeCoverage text report in file (default: "coverage.txt") - --coverage-phpunit Generate CodeCoverage PHPUnit report in file (default: "coverage-phpunit") - --no-exit Do not finish with exit code - --group (-g) Groups of tests to be executed (multiple values allowed) - --skip (-s) Skip selected suites (multiple values allowed) - --skip-group (-x) Skip selected groups (multiple values allowed) - --env Run tests in selected environments. (multiple values allowed, environments can be merged with ',') - --fail-fast (-f) Stop after first failure - --help (-h) Display this help message. - --quiet (-q) Do not output any message. - --verbose (-v|vv|vvv) Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug - --version (-V) Display this application version. - --ansi Force ANSI output. - --no-ansi Disable ANSI output. - --no-interaction (-n) Do not ask any interactive question. + -o, --override=OVERRIDE Override config values (multiple values allowed) + -e, --ext=EXT Run with extension enabled (multiple values allowed) + --report Show output in compact style + --html[=HTML] Generate html with results [default: "report.html"] + --xml[=XML] Generate JUnit XML Log [default: "report.xml"] + --phpunit-xml[=PHPUNIT-XML] Generate PhpUnit XML Log [default: "phpunit-report.xml"] + --tap[=TAP] Generate Tap Log [default: "report.tap.log"] + --json[=JSON] Generate Json Log [default: "report.json"] + --colors Use colors in output + --no-colors Force no colors in output (useful to override config file) + --silent Only outputs suite names and final results + --steps Show steps in output + -d, --debug Show debug and scenario output + --bootstrap[=BOOTSTRAP] Execute custom PHP script before running tests. Path can be absolute or relative to current working directory [default: false] + --no-redirect Do not redirect to Composer-installed version in vendor/codeception + --coverage[=COVERAGE] Run with code coverage + --coverage-html[=COVERAGE-HTML] Generate CodeCoverage HTML report in path + --coverage-xml[=COVERAGE-XML] Generate CodeCoverage XML report in file + --coverage-text[=COVERAGE-TEXT] Generate CodeCoverage text report in file + --coverage-crap4j[=COVERAGE-CRAP4J] Generate CodeCoverage report in Crap4J XML format + --coverage-phpunit[=COVERAGE-PHPUNIT] Generate CodeCoverage PHPUnit report in path + --no-exit Don't finish with exit code + -g, --group=GROUP Groups of tests to be executed (multiple values allowed) + -s, --skip=SKIP Skip selected suites (multiple values allowed) + -x, --skip-group=SKIP-GROUP Skip selected groups (multiple values allowed) + --env=ENV Run tests in selected environments. (multiple values allowed) + -f, --fail-fast Stop after first failure + --no-rebuild Do not rebuild actor classes on start + --seed=SEED Define random seed for shuffle setting + --no-artifacts Don't report about artifacts + -h, --help Display this help message + -q, --quiet Do not output any message + -V, --version Display this application version + --ansi Force ANSI output + --no-ansi Disable ANSI output + -n, --no-interaction Do not ask any interactive question + -c, --config[=CONFIG] Use custom path for config + -v|vv|vvv, --verbose Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug ``` diff --git a/docs/commands/mftf.md b/docs/commands/mftf.md index bc2e5c0ec..b7981808c 100644 --- a/docs/commands/mftf.md +++ b/docs/commands/mftf.md @@ -39,13 +39,13 @@ vendor/bin/mftf generate:tests ### Generate tests by test name ```bash -vendor/bin/mftf generate:tests AdminLoginTest StorefrontPersistedCustomerLoginTest +vendor/bin/mftf generate:tests AdminLoginSuccessfulTest StorefrontPersistedCustomerLoginTest ``` ### Generate test by test and suite name ```bash -vendor/bin/mftf generate:tests LoginSuite:AdminLoginTest +vendor/bin/mftf generate:tests WYSIWYGDisabledSuite:AdminCMSPageCreatePageTest ``` ### Generate and run the tests for a specified group @@ -59,18 +59,18 @@ This command cleans up the previously generated tests; generates and runs tests ### Generate and run particular tests ```bash -vendor/bin/mftf run:test AdminLoginTest StorefrontPersistedCustomerLoginTest -r +vendor/bin/mftf run:test AdminLoginSuccessfulTest StorefrontPersistedCustomerLoginTest -r ``` -This command cleans up the previously generated tests; generates and runs the `LoginAsAdminTest` and `LoginAsCustomerTest` tests. +This command cleans up the previously generated tests; generates and runs the `AdminLoginSuccessfulTest` and `StorefrontPersistedCustomerLoginTest` tests. ### Generate and run particular test in a specific suite's context ```bash -vendor/bin/mftf run:test LoginSuite:AdminLoginTest -r +vendor/bin/mftf run:test WYSIWYGDisabledSuite:AdminCMSPageCreatePageTest -r ``` -This command cleans up previously generated tests; generates and run `AdminLoginTest` within the context of the `LoginSuite`. +This command cleans up previously generated tests; generates and run `AdminCMSPageCreatePageTest` within the context of the `WYSIWYGDisabledSuite`. ### Generate and run a testManifest.txt file @@ -362,7 +362,7 @@ vendor/bin/mftf run:manifest path/to/your/testManifest.txt Each line should contain either: one test path or one group (-g) reference. ``` -tests/functional/tests/MFTF/_generated/default/AdminLoginTestCest.php +tests/functional/tests/MFTF/_generated/default/AdminLoginSuccessfulTestCest.php -g PaypalTestSuite tests/functional/tests/MFTF/_generated/default/SomeOtherTestCest.php tests/functional/tests/MFTF/_generated/default/ThirdTestCest.php diff --git a/docs/merging.md b/docs/merging.md index fc187d576..d8436492e 100644 --- a/docs/merging.md +++ b/docs/merging.md @@ -1,6 +1,6 @@ # Merging -The MFTF allows you to merge test components defined in XML files, such as: +MFTF allows you to merge test components defined in XML files, such as: - [``][] - [``][] @@ -20,9 +20,9 @@ For example: - All tests with `` will be merged into one. -- All sections with `
` will be merged into one. +- All sections with `
` will be merged into one. - All data entities with `` will be merged into one. -- All action groups with `` will be merged into one. +- All action groups with `` will be merged into one. Although a file name does not influence merging, we recommend using the same file names in merging updates. This makes it easier to search later on. @@ -39,7 +39,7 @@ This makes it easier to search later on. ## Add a test You cannot add another [``][tests] using merging functionality. -To add a ``, create a new `*Test.xml` file or add a `` node to an existing `*Test.xml` file. +To add a ``, create a new `*Test.xml` file. ## Remove a test @@ -50,38 +50,35 @@ Learn more about running tests with different options using [`mftf`] or [`codece ### Example -Skip the `AdminLoginTest` test in the `.../Backend/Test/AdminLoginTest.xml` file while merging with the `.../Foo/Test/AdminLoginTest.xml` file: +Skip the `AdminLoginSuccessfulTest` test in the `.../Backend/Test/AdminLoginSuccessfulTest.xml` file while merging with the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml - + - + - - <description value="You should be able to log into the Magento Admin backend."/> + <title value="Admin should be able to log into the Magento Admin backend successfully"/> + <description value="Admin should be able to log into the Magento Admin backend successfully"/> <severity value="CRITICAL"/> <testCaseId value="MAGETWO-71572"/> <group value="example"/> <group value="login"/> </annotations> - <amOnPage url="{{AdminLoginPage.url}}" stepKey="amOnAdminLoginPage"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="{{_ENV.MAGENTO_ADMIN_USERNAME}}" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="{{_ENV.MAGENTO_ADMIN_PASSWORD}}" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickOnSignIn"/> - <closeAdminNotification stepKey="closeAdminNotification"/> - <seeInCurrentUrl url="{{AdminLoginPage.url}}" stepKey="seeAdminLoginUrl"/> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> </tests> ``` -Create the `.../Foo/Test/AdminLoginTest.xml` file: +Create the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml <tests ...> - <test name="AdminLoginTest"> + <test name="AdminLoginSuccessfulTest"> <annotations> <skip> <issueId value="Issue#"/> @@ -91,29 +88,26 @@ Create the `.../Foo/Test/AdminLoginTest.xml` file: </tests> ``` -The `AdminLoginTest` result corresponds to: +The `AdminLoginSuccessfulTest` result corresponds to: ```xml -<test name="AdminLoginTest"> +<test name="AdminLoginSuccessfulTest"> <annotations> - <features value="Admin Login"/> + <features value="Backend"/> <stories value="Login on the Admin Login page"/> - <title value="You should be able to log into the Magento Admin backend."/> - <description value="You should be able to log into the Magento Admin backend."/> + <title value="Admin should be able to log into the Magento Admin backend successfully"/> + <description value="Admin should be able to log into the Magento Admin backend successfully"/> <severity value="CRITICAL"/> <testCaseId value="MAGETWO-71572"/> <group value="example"/> <group value="login"/> - <skip> - <issueId value="Issue#"/> - </skip> + <skip> + <issueId value="Issue#"/> + </skip> </annotations> - <amOnPage url="{{AdminLoginPage.url}}" stepKey="amOnAdminLoginPage"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="{{_ENV.MAGENTO_ADMIN_USERNAME}}" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="{{_ENV.MAGENTO_ADMIN_PASSWORD}}" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickOnSignIn"/> - <closeAdminNotification stepKey="closeAdminNotification"/> - <seeInCurrentUrl url="{{AdminLoginPage.url}}" stepKey="seeAdminLoginUrl"/> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> ``` @@ -126,159 +120,152 @@ See the previous examples. ### Add a test step -**Use case**: Add `checkOption` before `click` (`stepKey="clickLogin"`) and add `seeInCurrentUrl` after the `click` in the `LogInAsAdminTest` test (in the `.../Backend/Test/LogInAsAdminTest.xml` file) while merging with the `.../Foo/Test/LogInAsAdminTest.xml` file: +**Use case**: Add `AdminCheckOptionSalesActionGroup` before `AssertAdminSuccessLoginActionGroup` (`stepKey="assertLoggedIn"`) and add `AdminAssertSalesOnDashboardActionGroup` after the `AssertAdminSuccessLoginActionGroup` in the `AdminLoginSuccessfulTest` test (in the `.../Backend/Test/AdminLoginSuccessfulTest.xml` file) while merging with the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml <tests ...> - <test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="password" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> - <see userInput="Lifetime Sales" stepKey="seeLifetimeSales"/> + <test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> </tests> ``` -Create the `.../Foo/Test/LogInAsAdminTest.xml` file: +Create the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml <tests ...> - <test name="LogInAsAdminTest"> - <checkOption selector="{{AdminLoginFormSection.rememberMe}}" stepKey="checkRememberMe" before="clickLogin"/> - <seeInCurrentUrl url="admin/admin/dashboard/" stepKey="seeAdminUrl" after="clickLogin"/> + <test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales" before="assertLoggedIn"/> + <actionGroup ref="AdminAssertSalesOnDashboardActionGroup" stepKey="assertSalesOnDashboard" after="assertLoggedIn"/> </test> </tests> ``` -The `LogInAsAdminTest` result corresponds to: +The `AdminLoginSuccessfulTest` result corresponds to: ```xml -<test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="password" stepKey="fillPassword"/> - <checkOption selector="{{AdminLoginFormSection.rememberMe}}" stepKey="checkRememberMe"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> - <seeInCurrentUrl url="admin/admin/dashboard/" stepKey="seeAdminUrl"/> - <see userInput="Lifetime Sales" stepKey="seeLifetimeSales"/> +<test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales" before="assertLoggedIn"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminAssertSalesOnDashboardActionGroup" stepKey="assertSalesOnDashboard" after="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> ``` ### Remove a test step -**Use case**: Remove `see` (`stepKey="seeLifetimeSales"`) from the `LogInAsAdminTest` test (in the `.../Backend/Test/LogInAsAdminTest.xml` file) while merging with the `.../Foo/Test/LogInAsAdminTest.xml` file: +**Use case**: Remove `AdminCheckOptionSalesActionGroup` (`stepKey="checkOptionSales"`) from the `LogInAsAdminTest` test (in the `.../Backend/Test/AdminLoginSuccessfulTest.xml` file) while merging with the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml <tests ...> - <test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="password" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> - <see userInput="Lifetime Sales" stepKey="seeLifetimeSales"/> + <test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> </tests> ``` -Create the `.../Foo/Test/LogInAsAdminTest.xml` file: +Create the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml <tests ...> - <test name="LogInAsAdminTest"> - <remove keyForRemoval="seeLifetimeSales"/> + <test name="AdminLoginSuccessfulTest"> + <remove keyForRemoval="checkOptionSales"/> </test> </tests> ``` -The `LogInAsAdminTest` result corresponds to: +The `AdminLoginSuccessfulTest` result corresponds to: ```xml -<test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="password" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> +<test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> ``` ### Update a test step -**Use case**: Change selector in `fillField` (`stepKey="fillPassword"`) of the `LogInAsAdminTest` test (in the `.../Backend/Test/LogInAsAdminTest.xml` file) while merging with the `.../Foo/Test/LogInAsAdminTest.xml` file: +**Use case**: Change argument in `AdminCheckOptionSalesActionGroup` (`stepKey="checkOptionSales"`) of the `AdminLoginSuccessfulTest` test (in the `.../Backend/Test/AdminLoginSuccessfulTest.xml` file) while merging with the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml <tests ...> - <test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="password" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> - <see userInput="Lifetime Sales" stepKey="seeLifetimeSales"/> + <test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales"> + <argument name="salesOption" value="{{AdminSalesData.lifeTimeSales}}"/> + </actionGroup> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> </tests> ``` -Create the `.../Foo/Test/LogInAsAdminTest.xml` file: +Create the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml -<tests ...> - <test name="LogInAsAdminTest"> - <fillField selector="{{AdminLoginFormSection.wrong-password}}" userInput="password" stepKey="fillPassword"/> - </test> -</tests> +<test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales"> + <argument name="salesOption" value="{{AdminSalesData.annualSales}}"/> + </actionGroup> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> +</test> ``` -The `LogInAsAdminTest` result corresponds to: +The `AdminLoginSuccessfulTest` result corresponds to: ```xml -<test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.wrong-password}}" userInput="password" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> - <see userInput="Lifetime Sales" stepKey="seeLifetimeSales"/> +<test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales"> + <argument name="salesOption" value="{{AdminSalesData.annualSales}}"/> + </actionGroup> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> ``` ### Add several test steps {#insert-after} -**Use case**: Add several actions after the `click` (`stepKey="clickLogin"`) in the `LogInAsAdminTest` test (in the `.../Backend/Test/LogInAsAdminTest.xml` file) while merging with the `.../Foo/Test/LogInAsAdminTest.xml` file: +**Use case**: Add several action groups after the `AdminLoginActionGroup` (`stepKey="loginAsAdmin"`) in the `AdminLoginSuccessfulTest` test (in the `.../Backend/Test/AdminLoginSuccessfulTest.xml` file) while merging with the `.../Foo/Test/LogInAsAdminTest.xml` file: ```xml -<tests ...> - <test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="password" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> - <see userInput="Lifetime Sales" stepKey="seeLifetimeSales"/> - </test> -</tests> +<test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> +</test> ``` -Create the `.../Foo/Test/LogInAsAdminTest.xml` file: +Create the `.../Foo/Test/AdminLoginSuccessfulTest.xml` file: ```xml <tests ...> - <test name="LogInAsAdminTest" insertAfter="clickLogin"> - <checkOption selector="{{AdminLoginFormSection.rememberMe}}" stepKey="checkRememberMe"/> - <seeInCurrentUrl url="admin/admin/dashboard/" stepKey="seeAdminUrl"/> + <test name="AdminLoginSuccessfulTest" insertAfter="loginAsAdmin"> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> </test> </tests> ``` -The `LogInAsAdminTest` result corresponds to: +The `AdminLoginSuccessfulTest` result corresponds to: ```xml -<test name="LogInAsAdminTest"> - <amOnPage url="{{AdminLoginPage}}" stepKey="navigateToAdmin"/> - <fillField selector="{{AdminLoginFormSection.username}}" userInput="admin" stepKey="fillUsername"/> - <fillField selector="{{AdminLoginFormSection.password}}" userInput="password" stepKey="fillPassword"/> - <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/> - <checkOption selector="{{AdminLoginFormSection.rememberMe}}" stepKey="checkRememberMe"/> - <seeInCurrentUrl url="admin/admin/dashboard/" stepKey="seeAdminUrl"/> - <see userInput="Lifetime Sales" stepKey="seeLifetimeSales"/> -</test> +<tests ...> + <test name="AdminLoginSuccessfulTest"> + <actionGroup ref="AdminLoginActionGroup" stepKey="loginAsAdmin"/> + <actionGroup ref="AdminCheckOptionSalesActionGroup" stepKey="checkOptionSales"/> + <actionGroup ref="AssertAdminSuccessLoginActionGroup" stepKey="assertLoggedIn"/> + <actionGroup ref="AdminLogoutActionGroup" stepKey="logoutFromAdmin"/> + </test> +</tests> ``` ## Merge action groups @@ -293,7 +280,7 @@ The controls change drastically in the B2B version, so it was abstracted to an a > Action group for selecting `customerGroup` in the `Cart Price Rules` section: ```xml -<actionGroup name="selectNotLoggedInCustomerGroup"> +<actionGroup name="SelectNotLoggedInCustomerGroupActionGroup"> <selectOption selector="{{AdminCartPriceRulesFormSection.customerGroups}}" userInput="NOT LOGGED IN" stepKey="selectCustomerGroup"/> </actionGroup> ``` @@ -302,7 +289,7 @@ The controls change drastically in the B2B version, so it was abstracted to an a ```xml <!-- name matches --> -<actionGroup name="selectNotLoggedInCustomerGroup"> +<actionGroup name="SelectNotLoggedInCustomerGroupActionGroup"> <!-- removes the original action --> <remove keyForRemoval="selectCustomerGroup"/> <!-- adds in sequence of actions to be performed instead--> @@ -323,62 +310,62 @@ To merge [pages][page], the `page name` must be the same as in the base module. ### Add a section **Use case**: The `FooBackend` module extends the `Backend` module and requires use of two new sections that must be covered with tests. -Add `BaseBackendSection` and `AnotherBackendSection` to the `BaseBackendPage` (`.../Backend/Page/BaseBackendPage.xml` file): +Add `AdminBaseBackendSection` and `AdminAnotherBackendSection` to the `AdminBaseBackendPage` (`.../Backend/Page/AdminBaseBackendPage.xml` file): ```xml <pages ...> - <page name="BaseBackendPage" url="admin" area="admin" module="Magento_Backend"> - <section name="BaseBackendSection"/> - <section name="AnotherBackendSection"/> + <page name="AdminBaseBackendPage" url="admin" area="admin" module="Magento_Backend"> + <section name="AdminBaseBackendSection"/> + <section name="AdminAnotherBackendSection"/> </page> </pages> ``` -Create the `.../FooBackend/Page/BaseBackendPage.xml` file: +Create the `.../FooBackend/Page/AdminBaseBackendPage.xml` file: ```xml <pages ...> - <page name="BaseBackendPage" url="admin" area="admin" module="Magento_Backend"> - <section name="NewExtensionSection"/> + <page name="AdminBaseBackendPage" url="admin" area="admin" module="Magento_Backend"> + <section name="AdminNewExtensionSection"/> </page> </pages> ``` -The `BaseBackendPage` result corresponds to: +The `AdminBaseBackendPage` result corresponds to: ```xml -<page name="BaseBackendPage" url="admin" area="admin" module="Magento_Backend"> +<page name="AdminBaseBackendPage" url="admin" area="admin" module="Magento_Backend"> - <section name="BaseBackendSection"/> - <section name="AnotherBackendSection"/> - <section name="NewExtensionSection"/> + <section name="AdminBaseBackendSection"/> + <section name="AdminAnotherBackendSection"/> + <section name="AdminNewExtensionSection"/> </page> ``` ### Remove a section -**Use case**: The `FooBackend` module extends the `Backend` module and requires deletion of the `AnotherBackendSection` section (the `.../Backend/Page/BaseBackendPage.xml` file): +**Use case**: The `FooBackend` module extends the `Backend` module and requires deletion of the `AdminAnotherBackendSection` section (the `.../Backend/Page/AdminBaseBackendPage.xml` file): ```xml -<page name="BaseBackendPage" url="admin" area="admin" module="Magento_Backend"> - <section name="BaseBackendSection"/> - <section name="AnotherBackendSection"/> +<page name="AdminBaseBackendPage" url="admin" area="admin" module="Magento_Backend"> + <section name="AdminBaseBackendSection"/> + <section name="AdminAnotherBackendSection"/> </page> ``` -Create the `.../FooBackend/Page/BaseBackendPage.xml` file: +Create the `.../FooBackend/Page/AdminBaseBackendPage.xml` file: ```xml -<page name="BaseBackendPage" url="admin" area="admin" module="Magento_Backend"> - <section name="AnotherBackendSection" remove="true"/> +<page name="AdminBaseBackendPage" url="admin" area="admin" module="Magento_Backend"> + <section name="AdminAnotherBackendSection" remove="true"/> </page> ``` -The `BaseBackendPage` result corresponds to: +The `AdminBaseBackendPage` result corresponds to: ```xml -<page name="BaseBackendPage" url="admin" area="admin" module="Magento_Backend"> - <section name="BaseBackendSection"/> +<page name="AdminBaseBackendPage" url="admin" area="admin" module="Magento_Backend"> + <section name="AdminBaseBackendSection"/> </page> ``` diff --git a/docs/metadata.md b/docs/metadata.md index 3f6032b57..414f4bc04 100644 --- a/docs/metadata.md +++ b/docs/metadata.md @@ -1,6 +1,6 @@ # Metadata -In this topic we talk about handling entities that you need in your tests (such as categories, products, wish lists, and similar) using the MFTF. +In this topic we talk about handling entities that you need in your tests (such as categories, products, wish lists, and similar) using MFTF. Using data handling actions like [`createData`], [`deleteData`], [`updateData`], and [`getData`], you are able to create, delete, update, and read entities for your tests. The framework enables you to send HTTP requests with these statically defined data entities: @@ -29,7 +29,7 @@ Each [operation] includes: - The type of body content encoding in [contentType]. - The body of the request represented as a tree of objects, arrays, and fields. -When a test step requires handling the specified data entity, the MFTF performs the following steps: +When a test step requires handling the specified data entity, MFTF performs the following steps: - Reads input data (`<data/>`) and the type (the `type` attribute) of the specified [entity]. - Searches the metadata operation for the `dataType` that matches the entity's `type`. For example, `<entity type="product">` matches `<operation dataType="product"`. @@ -94,8 +94,8 @@ Example: ### Sending a REST API request -The MFTF allows you to handle basic CRUD operations with an object using [Magento REST API][api reference] requests. -To convert a request to the MFTF format, wrap the corresponding REST API request into XML tags according to the [Reference documentation][reference]. +MFTF allows you to handle basic CRUD operations with an object using [Magento REST API][api reference] requests. +To convert a request to MFTF format, wrap the corresponding REST API request into XML tags according to the [Reference documentation][reference]. - GET is used for retrieving data from objects. - POST is used for creating new objects. @@ -123,7 +123,7 @@ Let's see what happens when you create a category: <createData entity="_defaultCategory" stepKey="createPreReqCategory"/> ``` -The MFTF searches in the _Data_ directory an entity with `<entity name="_defaultCategory">` and reads `type` of the entity. +MFTF searches in the _Data_ directory an entity with `<entity name="_defaultCategory">` and reads `type` of the entity. If there are more than one entity with the same name, all of the entities are merged. _Catalog/Data/CategoryData.xml_: @@ -136,8 +136,8 @@ _Catalog/Data/CategoryData.xml_: </entity> ``` -Here, `type` is equal to `"category"`, which instructs the MFTF to search an operation with `dataType="category"`. -Since the action is __to create__ a category, the MFTF will also search for operation with `type="create"` in _Metadata_ for `dataType="category"`. +Here, `type` is equal to `"category"`, which instructs MFTF to search an operation with `dataType="category"`. +Since the action is __to create__ a category, MFTF will also search for operation with `type="create"` in _Metadata_ for `dataType="category"`. _Catalog/Metadata/CategoryMeta.xml_: @@ -236,26 +236,26 @@ The corresponding test step is: <createData entity="guestCart" stepKey="createGuestCart"/> ``` -The MFTF searches in the _Data_ directory an entity with `<entity name="guestCart">` and reads `type`. +MFTF searches in the _Data_ directory an entity with `<entity name="GuestCart">` and reads `type`. _Quote/Data/GuestCartData.xml_: ```xml -<entity name="guestCart" type="guestCart"> +<entity name="GuestCart" type="GuestCart"> </entity> ``` -`type="guestCart"` points to the operation with `dataType=guestCart"` and `type="create"` in the _Metadata_ directory. +`type="guestCart"` points to the operation with `dataType=GuestCart"` and `type="create"` in the _Metadata_ directory. _Catalog/Data/CategoryData.xml_: ```xml -<operation name="CreateGuestCart" dataType="guestCart" type="create" auth="anonymous" url="/V1/guest-carts" method="POST"> +<operation name="CreateGuestCart" dataType="GuestCart" type="create" auth="anonymous" url="/V1/guest-carts" method="POST"> <contentType>application/json</contentType> </operation> ``` -As a result, the MFTF sends an unauthorized POST request with an empty body to the `https://example.com/rest/V1/guest-carts` and stores the single string response that the endpoint returns. +As a result, MFTF sends an unauthorized POST request with an empty body to the `https://example.com/rest/V1/guest-carts` and stores the single string response that the endpoint returns. ### Handling a REST API response {#rest-response} @@ -267,7 +267,7 @@ Let's see how to handle data after you created a category with custom attributes <createData entity="customizedCategory" stepKey="createPreReqCategory"/> ``` -The MFTF receives the corresponding JSON response and enables you to reference its data using a variable of format: +MFTF receives the corresponding JSON response and enables you to reference its data using a variable of format: **$** _stepKey_ **.** _JsonKey_ **$** @@ -377,7 +377,7 @@ The operation enables you to assign the following form fields: ### Create an object in storefront {#create-object-as-customerFormKey} -The MFTF uses the `CreateWishlist` operation to create a wish list on storefront: +MFTF uses the `CreateWishlist` operation to create a wish list on storefront: Source file is _Wishlist/Metadata/WishlistMeta.xml_ @@ -558,7 +558,7 @@ Example: <!-- LINK DEFINITIONS --> [actions]: test/actions.md -[api reference]: https://devdocs.magento.com/guides/v2.3/get-started/bk-get-started-api.html +[api reference]: https://devdocs.magento.com/guides/v2.4/get-started/bk-get-started-api.html [application/x-www-form-urlencoded]: https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.1 {:target="_blank"} [catalogCategoryRepositoryV1 image]: img/catalogCategoryRepository-operations.png @@ -573,7 +573,7 @@ Example: [getData]: test/actions.md#getdata [HTML forms]: https://www.w3.org/TR/html401/interact/forms.html {:target="\_blank"} -[oauth]: https://devdocs.magento.com/guides/v2.3/get-started/authentication/gs-authentication-oauth.html +[oauth]: https://devdocs.magento.com/guides/v2.4/get-started/authentication/gs-authentication-oauth.html {:target="\_blank"} [operation]: #operation-tag [reference]: #reference diff --git a/docs/page.md b/docs/page.md index 41afd274f..fd03cfd58 100644 --- a/docs/page.md +++ b/docs/page.md @@ -1,6 +1,6 @@ # Page structure -The MFTF uses a modified concept of [PageObjects], which models the testing areas of your page as objects within the code. +MFTF uses a modified concept of [PageObjects], which models the testing areas of your page as objects within the code. This reduces occurrences of duplicated code and enables you to fix things quickly, in one place, when things change. You define the contents of a page, for reference in a [`<test>`], at both the [`<page>`] and [`<section>`] level. @@ -61,7 +61,7 @@ Example (_Catalog/Page/AdminCategoryPage.xml_ file): <?xml version="1.0" encoding="UTF-8"?> <pages xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" - xsi:noNamespaceSchemaLocation="../../../../../..dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Page/etc/PageObject.xsd"> + xsi:noNamespaceSchemaLocation="urn:magento:mftf:Page/etc/PageObject.xsd"> <page name="AdminCategoryPage" url="catalog/category/" module="Magento_Catalog" area="admin"> <section name="AdminCategorySidebarActionSection"/> <section name="AdminCategorySidebarTreeSection"/> @@ -97,7 +97,7 @@ Example (_Catalog/Page/StorefrontCategoryPage.xml_ file): <?xml version="1.0" encoding="UTF-8"?> <pages xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" - xsi:noNamespaceSchemaLocation="../../../../../..dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Page/etc/PageObject.xsd"> + xsi:noNamespaceSchemaLocation="urn:magento:mftf:Page/etc/PageObject.xsd"> <page name="StorefrontCategoryPage" url="/{{var1}}.html" module="Magento_Catalog" parameterized="true" area="storefront"> <section name="StorefrontCategoryMainSection"/> </page> @@ -107,15 +107,15 @@ Example (_Catalog/Page/StorefrontCategoryPage.xml_ file): This example shows the page with the name `StorefrontCategoryPage`. It will be merged with the other `StorefrontCategoryPage` pages from other modules. -The following is an example of a call in test: +The following is an example of a call in a test: ```xml -<amOnPage url="{{StorefrontCategoryPage.url($$createPreReqCategory.name$$)}}" stepKey="navigateToCategoryPage"/> +<amOnPage url="{{StorefrontCategoryPage.url($createPreReqCategory.name$)}}" stepKey="navigateToCategoryPage"/> ``` The `StorefrontCategoryPage` page is declared as parameterized, where the `url` contains a `{{var1}}` parameter. -The corresponding web page is generated by the Magento Catalog module and is called by the `baseUrl`+`/$$createPreReqCategory.name$$.html` URl. +The corresponding web page is generated by the Magento Catalog module and is called by the `baseUrl`+`/$createPreReqCategory.name$.html` URl. `{{var1}}` is substituted with the `name` of the previously created category in the `createPreReqCategory` action. @@ -132,7 +132,7 @@ There are several XML elements that are used in `<page>` in the MFTF. ### pages {#pages-tag} `<pages>` are elements that point to the corresponding XML Schema location. -It contains one or more `<page>` elements. +It contains only one `<page>` element. ### page {#page-tag} diff --git a/docs/reporting.md b/docs/reporting.md index 59a8617ba..629c3cfd8 100644 --- a/docs/reporting.md +++ b/docs/reporting.md @@ -57,42 +57,49 @@ The general information can be useful for MFTF contributors, but can be ignored Let's consider the general part of the following test execution report: ```terminal -==== Redirecting to Composer-installed version in vendor/codeception ==== -Codeception PHP Testing Framework v2.3.9 -Powered by PHPUnit 6.5.13 by Sebastian Bergmann and contributors. +Generate Tests Command Run +Codeception PHP Testing Framework v4.1.4 +Powered by PHPUnit 9.1.3 by Sebastian Bergmann and contributors. +Running with seed: Magento\FunctionalTestingFramework.functional Tests (2) ------------------------ -Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Helper\Acceptance, \Magento\FunctionalTestingFramework\Helper\MagentoFakerData, \Magento\FunctionalTestingFramework\Module\MagentoRestDriver, PhpBrowser, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTes +Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTestingFramework\Module\MagentoAssert, \Magento\FunctionalTestingFramework\Module\MagentoActionProxies, Asserts, \Magento\FunctionalTestingFramework\Helper\HelperContainer ``` -After the test generation command (mentioned in the previous section), MFTF delegates control to the `vendor/codeception` tool, which is the `Codeception PHP Testing Framework` of version `2.3.9` that uses `PHPUnit` of version `6.5.13`. +After the test generation command (mentioned in the previous section), MFTF delegates control to the `vendor/codeception` tool, which is the `Codeception PHP Testing Framework` of version `4.1.4` that uses `PHPUnit` of version `9.1.3`. The tool runs `2 Tests` using the configuration defined in the `functional` suite under the `Magento\FunctionalTestingFramework` namespace. The corresponding configuration file is `acceptance/tests/functional.suite.yml`. -It enables `Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Helper\Acceptance, \Magento\FunctionalTestingFramework\Helper\MagentoFakerData, \Magento\FunctionalTestingFramework\Module\MagentoRestDriver, PhpBrowser, \Magento\FunctionalTestingFramework\Module\MagentoSequence, ...` +It enables `Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTestingFramework\Module\MagentoAssert, \Magento\FunctionalTestingFramework\Module\MagentoActionProxies, Asserts, \Magento\FunctionalTestingFramework\Helper\HelperContainer,..` #### Passed tests The next chunk of the log reports about test execution of the first test: ```terminal -AdminLoginTestCest: Admin login test -Signature: Magento\AcceptanceTest\_default\Backend\AdminLoginTestCest:AdminLoginTest -Test: tests/functional/Magento/FunctionalTest/_generated/default/AdminLoginTestCest.php:AdminLoginTest +AdminLoginSuccessfulTestCest: Admin login successful test +Signature: Magento\AcceptanceTest\_default\Backend\AdminLoginSuccessfulTestCest:AdminLoginSuccessfulTest +Test: tests/functional/Magento/_generated/default/AdminLoginSuccessfulTestCest.php:AdminLoginSuccessfulTest Scenario -- -I am on page "/admin/admin" -I fill field "#username","admin" -I fill field "#login","123123q" -I click ".actions .action-primary" -I wait for page load 30 -I close admin notification -I see in current url "/admin/admin" -PASSED +[loginAsAdmin] AdminLoginActionGroup + [navigateToAdmin] am on page "/admin/admin" + [fillUsername] fill field "#username","admin" + [fillPassword] fill field "#login","123123q" + [clickLogin] click ".actions .action-primary" + [clickLoginWaitForPageLoad] wait for page load 30 + [clickDontAllowButtonIfVisible] conditional click ".modal-popup .action-secondary",".modal-popup .action-secondary",true + [closeAdminNotification] close admin notification +[assertLoggedIn] AssertAdminSuccessLoginActionGroup + [waitForAdminAccountTextVisible] wait for element visible ".page-header .admin-user-account-text",60 + [assertAdminAccountTextElement] see element ".page-header .admin-user-account-text" +[logoutFromAdmin] AdminLogoutActionGroup + [amOnLogoutPage] am on page "/admin/admin/auth/logout/" + PASSED ``` -The running test is `AdminLoginTestCest`, which is `Admin login test` (this text is generated from the test name but with the `Cest` part excluded). -Its test signature is `Magento\AcceptanceTest\_default\Backend\AdminLoginTestCest:AdminLoginTest` that matches a `className:methodName` format using namespaces. -A path to the corresponding `Test` is `tests/functional/Magento/FunctionalTest/_generated/default/AdminLoginTestCest.php:AdminLoginTest` (relative to the `acceptance/` directory). +The running test is `AdminLoginSuccessfulTestCest`, which is `Admin login successful test` (this text is generated from the test name but with the `Cest` part excluded). +Its test signature is `Magento\AcceptanceTest\_default\Backend\AdminLoginSuccessfulTestCest:AdminLoginSuccessfulTest` that matches a `className:methodName` format using namespaces. +A path to the corresponding `Test` is `tests/functional/Magento/_generated/default/AdminLoginSuccessfulTestCest.php:AdminLoginSuccessfulTest` (relative to the `acceptance/` directory). `Scenario` lists the tests steps as they run during test execution, ending with the successful test verdict `PASSED`. It means that all test steps were processed as expected. @@ -104,51 +111,54 @@ The second test fails with the following report: ```terminal AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test Signature: Magento\AcceptanceTest\_default\Backend\AdminMenuNavigationWithSecretKeysTestCest:AdminMenuNavigationWithSecretKeysTest -Test: tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest +Test: tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest Scenario -- -I magento cli "config:set admin/security/use_form_key 1" +[enableUrlSecretKeys] magento cli "config:set admin/security/use_form_key 1",60 Value was saved. -I magento cli "cache:clean config full_page" + +[cleanInvalidatedCaches1] magento cli "cache:clean config full_page",60 Cleaned cache types: config full_page -I am on page "/admin/admin" -I wait for page load -I fill field "#username","admin" -I fill field "#login","123123q" -I click ".actions .action-primary" -I wait for page load 30 -I close admin notification -I click "//li[@id='menu-magento-backend-stores']" -I wait for loading mask to disappear -I click "#nav li[data-ui-id='menu-magento-config-system-config']" -I wait for page load -I see current url matches "~\/admin\/system_config\/~" -I see "#something" -I save screenshot -FAIL - -I magento cli "config:set admin/security/use_form_key 0" + +[loginAsAdmin] AdminLoginActionGroup + [navigateToAdmin] am on page "/admin/admin" + [fillUsername] fill field "#username","admin" + [fillPassword] fill field "#login","123123q" + [clickLogin] click ".actions .action-primary" + [clickLoginWaitForPageLoad] wait for page load 30 + [clickDontAllowButtonIfVisible] conditional click ".modal-popup .action-secondary",".modal-popup .action-secondary",true + [closeAdminNotification] close admin notification +[clickStoresMenuOption1] click "#menu-magento-backend-stores" +[waitForStoresMenu1] wait for loading mask to disappear +[clickStoresConfigurationMenuOption1] click "#nav li[data-ui-id='menu-magento-config-system-config']" +[waitForConfigurationPageLoad1] wait for page load 60 +[seeCurrentUrlMatchesConfigPath1] see current url matches "~\/admin\/system_config\/~" +[clickCatalogMenuOption] click "#something" +[saveScreenshot] save screenshot +[disableUrlSecretKeys] magento cli "config:set admin/security/use_form_key 0",60 Value was saved. -I magento cli "cache:clean config full_page" + +[cleanInvalidatedCaches2] magento cli "cache:clean config full_page",60 Cleaned cache types: config full_page -I am on page "/admin/admin/auth/logout/" + +[logout] AdminLogoutActionGroup + [amOnPage] am on page "/admin/admin/auth/logout/" + FAIL -------------------------------------------------------------------------------- ``` The general test details and scenario has the same format as in the Passed test. -The interesting part starts near the `FAIL` line. ```terminal -I see "#something" -I save screenshot -FAIL +[clickCatalogMenuOption] click "#something" +[saveScreenshot] save screenshot ``` When a test step fails, MFTF always saves a screenshot of the web page with the failing state immediately after the failure occurs. -`I save screenshot` follows the failing test step `I see "#something"` in our case. +`[saveScreenshot] save screenshot` follows the failing test step `[clickCatalogMenuOption] click "#something"` in our case. A screenshot of the fail goes at the `acceptance/tests/_output` directory in both PNG and HTML formats: @@ -165,27 +175,19 @@ The file name encodes: - with the `AdminMenuNavigationWithSecretKeysTest` test name - and execution status `fail` -Actions after `FAIL` are run as a part of the [`after`][] hook of the test. +Actions after `saveScreenshot` are run as a part of the [`after`][] hook of the test. ### Test result report After MFTF completed test execution, it generates a general report about test results along with detailed information about each fail. ```terminal --------------------------------------------------------------------------------- -DEPRECATION: Calling the "Symfony\Component\BrowserKit\Client::getInternalResponse()" method before the "request()" one is deprecated since Symfony 4.1 and will throw an exception in 5.0. /Users/.../magento2ce/vendor/symfony/browser-kit/Client.php:208 - -Time: 52.43 seconds, Memory: 16.00MB +Time: 02:07.534, Memory: 150.50 MB There was 1 failure: --------- ``` - -First you see warnings and deprecations. -The `DEPRECATION` here is thrown by an MFTF dependency (Symfony) that is out of the scope for test writers and should be considered by MFTF contributors. -If you encounter this type of reporting, [report an issue][]. - -Then, MFTF reports that the test run took 52.43 seconds using 16 MB of system RAM. +MFTF reports that the test run took 02:07.534 using 150.50 MB of system RAM. And, finally, that there was `1 failure`. Next, the report provides details about the test failure. @@ -193,75 +195,38 @@ Next, the report provides details about the test failure. ```terminal --------- 1) AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test -Test tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest -Step See "#something" -Fail Failed asserting that on page /admin/admin/system_config/index/key/678b7ba922c.../ ---> DASHBOARD -SALES -CATALOG -CUSTOMERS -MARKETING -CONTENT -REPORTS -STORES -SYSTEM -FIND PARTNERS & EXTENSIONS -Configuration -admin -1 -Store View: Default Config -What is this? -Save Config -Country Options -State Options -Locale Options -Store Information -Store Name -Store Phone Number -Store Hours of Operation -Countr -[Content too long to display. See complete response in '/Users/dmytroshevtsov/Projects/vagrant/vagrant-magento/magento2ce/dev/tests/acceptance/tests/_output/' directory] ---> contains "#something". + Test tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest + Step Click "#something" + Fail CSS or XPath element with '#something' was not found. Scenario Steps: -23. $I->amOnPage("/admin/admin/auth/logout/") at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:54 -22. // Cleaned cache types: + 27. // Exiting Action Group [logout] AdminLogoutActionGroup + 26. $I->amOnPage("/admin/admin/auth/logout/") at tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:55 + 25. // Entering Action Group [logout] AdminLogoutActionGroup + 24. // Cleaned cache types: config full_page -21. $I->magentoCLI("cache:clean config full_page") at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:52 -20. // Value was saved. -19. $I->magentoCLI("config:set admin/security/use_form_key 0") at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:50 -18. $I->saveScreenshot() at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:63 + + 23. $I->magentoCLI("cache:clean config full_page",60) at tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:52 + 22. // Value was saved. ``` - `1) AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test` - the failed Codeception test is *AdminMenuNavigationWithSecretKeysTestCest*. It references to the PHP class that implemented the failed test. -- `Test tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest` - the test is implemented in the *AdminMenuNavigationWithSecretKeysTest* test method of the *tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php* file under `<magento root>/dev/tests/acceptance/`. +- `Test tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest` - the test is implemented in the *AdminMenuNavigationWithSecretKeysTest* test method of the *tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php* file under `<magento root>/dev/tests/acceptance/`. It matches the corresponding test defined in XML that is *AdminMenuNavigationWithSecretKeysTest* defined in `<test name="AdminMenuNavigationWithSecretKeysTest">...</test>` -- `Step See "#something"` - the failing test step is the *see* action with the *#something* selector. It would correspond the `<see selector="#something" ... />` test step in the XML defined tests. - -- `Fail Failed asserting that on page /admin/admin/system_config/index/key/678b7ba922c.../` - the fail occurred on the web page `<MAGENTO_BASE_URL>/admin/admin/system_config/index/key/678b7ba922c.../`. - -```terminal ---> ... -[Content too long to display. See complete response in '/../../magento2/dev/tests/acceptance/tests/_output/' directory] ---> contains "#something". -``` - -The web page is too long to be reported in the CLI, and it is stored at *'/../../magento2/dev/tests/acceptance/tests/_output/'*. -Search the web page by test name *AdminMenuNavigationWithSecretKeysTest*. -The failing test assertion is that the web page contains *contains* a CSS locator *#something*. +- `Step Click "#something"` - the failing test step is the *click* action with the *#something* selector. It would correspond the `<click selector="#something" ... />` test step in the XML defined tests. Finally, the report finishes with fairly self-descriptive lines. ```terminal FAILURES! -Tests: 2, Assertions: 3, Failures: 1. +Tests: 2, Assertions: 2, Failures: 1. ``` -MFTF encountered failures due to the last test run, that included *2* tests with *3* assertions. +MFTF encountered failures due to the last test run, that included *2* tests with *2* assertions. *1* assertion fails. ## Allure @@ -324,8 +289,7 @@ And if you run the `open` command with no arguments while you are in the same di allure open ``` -Allure would attempt to open a generated report at the `magento2/allure-report/` directory.' -%} +Allure would attempt to open a generated report at the `magento2/allure-report/` directory. To clean up existing reports before generation (for example after getting new results), use the `--clean` flag: From ee10e683b4d0dfba64c8ad922906a71fc5034c77 Mon Sep 17 00:00:00 2001 From: Donald Booth <dobooth@adobe.com> Date: Wed, 22 Jul 2020 12:33:02 -0500 Subject: [PATCH 11/30] Add raw tag to prevent build errors. --- docs/configure-2fa.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/configure-2fa.md b/docs/configure-2fa.md index f50838f0c..7b01913b9 100644 --- a/docs/configure-2fa.md +++ b/docs/configure-2fa.md @@ -44,6 +44,10 @@ Use the action `getOTP` [Reference](./test/actions.md#getotp) to generate the co Note: You will need to set the `secret` for any non-default admin users first, before using `getOTP`. For example: +{%raw%} + ```xml <magentoCLI command="security:tfa:google:set-secret admin2 {{_CREDS.magento/tfa/OTP_SHARED_SECRET}}" stepKey="setSecret"/> ``` + +{%endraw%} From 7b572667bff8c26c7dffd4a43cfb8a8dac8ba9cd Mon Sep 17 00:00:00 2001 From: soumyau <sunnikri@adobe.com> Date: Wed, 22 Jul 2020 14:28:01 -0500 Subject: [PATCH 12/30] MQE-2218: Part 5 Docs Annual Review (#764) * MQE-2218: Part 5 Docs Annual Review * Grammar tweak Co-authored-by: Donald Booth <dobooth@adobe.com> --- docs/best-practices.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/best-practices.md b/docs/best-practices.md index 0ce9448cb..a376768c2 100644 --- a/docs/best-practices.md +++ b/docs/best-practices.md @@ -37,11 +37,9 @@ We recommend to keep Action Groups having single responsibility, for example `Ad ## Contribute -Althought the Magento Core team and Contributors join forces to cover most of the features with tests, it is impossible to have this done quickly. +Although the Magento Core team and Contributors join forces to cover most of the features with tests, it is impossible to have this done quickly. If you've covered Magento Core feature with Functional Tests - you are more than welcome to contribute. -You can also help with MFTF Test Migration to get the experience and valuable feedback from other community members and maintainers. - ## Action group 1. [Action group] names should be sufficiently descriptive to inform a test writer of what the action group does and when it should be used. Add additional explanation in annotations if needed. @@ -108,7 +106,7 @@ Example: _StorefrontCreateCustomerTest.xml_. #### Action Group file name -Format: {_Admin_ or _Storefront_}{Action Group Summary}ActionGroup.xml`, where Action Group Summary describes with a few words what we can expect from it. +Format: {_Admin_ or _Storefront_}{Action Group Summary}ActionGroup.xml`, where Action Group Summary is a short description of what the action group does. Example: _AdminCreateStoreActionGroup.xml_ From a9b6bbe43ba983692b1b9ac79c58af3ddfc693c0 Mon Sep 17 00:00:00 2001 From: Kevin Kozan <kkozan@adobe.com> Date: Fri, 11 Oct 2019 11:52:11 -0500 Subject: [PATCH 13/30] MQE-1776: CICD System Documentation - Initial Draft --- docs/guides/cicd.md | 101 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100644 docs/guides/cicd.md diff --git a/docs/guides/cicd.md b/docs/guides/cicd.md new file mode 100644 index 000000000..a37392955 --- /dev/null +++ b/docs/guides/cicd.md @@ -0,0 +1,101 @@ +# How to use MFTF in CICD + +If you want to integrate MFTF tests into your CICD pipeline, then it's best to start with the conceptual flow of the pipeline code. + +## Concept + +The overall workflow that tests should follow is thus: + +* Obtain Magento instance + install pre-requisites +* Generate tests for running + * Options for single/parallel running +* Delegate and run tests + gather test run artifacts + * Re-run options +* Generate Allure report from aggregate + +### Obtain Magento instance + +To start, we need a Magento instance to operate against for generation and execution. + +``` +$ git clone https://github.com/magento/magento2 +or +$ composer create-project --repository=https://repo.magento.com/ magento/project-community-edition magento2ce +``` + +For more information on installing magento see [Install Magento using Composer]. + +After installing the Magento instance, you need to set a couple of configurations to the magento instance: + +``` +$ bin/magento config:set general/locale/timezone America/Los_Angeles +$ bin/magento config:set admin/security/admin_account_sharing 1 +$ bin/magento config:set admin/security/use_form_key 0 +$ bin/magento config:set cms/wysiwyg/enabled disabled +``` + +These help set the state for the `default` state of the Magento instance. If you are wanting to change the default state of the application (and have merged into the tests sufficiently to account for it), this is the step in which you would do it. + +#### Install allure + +This will be required to generate the report after your test runs. See [Allure] for details. + + +### Generate tests + +#### Single execution + +Simply generate tests based on what you want to run: + +``` +$ vendor/bin/mftf generate:tests +``` + +This will generate all tests, and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt` + +#### Parallel execution + +To generate all tests for use in parallel nodes: + +``` +$ vendor/bin/mftf generate:tests --config parallel +``` + +This will generate a folder under `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/groups`. This folder contains several `group#.txt` files that can be used later with the `mftf run:manifest` command. + +### Delegate and run tests + +#### Single execution +If you are running on a single node, this step is simply to call: + +``` +$ vendor/bin/mftf run:manifest dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt +``` + +#### Parallel execution +To run MFTF tests in parallel, you will need to clone the contents of the current node and duplicate them depending on how many nodes you have available for use. + +* Clone contents of current node as a baseline +* For each `groups/group#.txt` file: + * Set a node's contents to the baseline + * Run `vendor/bin/mftf run:manifest <current_group.txt>` + * Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to master + +#### Rerun options +In either single or parallel execution, to re-run failed tests simply add a `run:failed` command after executing a manifest: + +``` +$ vendor/bin/mftf run:failed +``` + +### Generate Allure report + +In the master node, simply generate using your `<path_to_results>` into a desired output path + +``` +$ allure generate <path_to_results> -c -o <path_to_output> +``` + +<!-- Link definitions --> +[Install Magento using Composer]: https://devdocs.magento.com/guides/v2.3/install-gde/composer.html +[Allure]: https://docs.qameta.io/allure/ From 4e94af42edac1f22bb4ac96e91c3cd109151b393 Mon Sep 17 00:00:00 2001 From: Donald Booth <dobooth@adobe.com> Date: Tue, 29 Oct 2019 11:09:02 -0500 Subject: [PATCH 14/30] Editorial pass --- docs/guides/cicd.md | 99 ++++++++++++++++++++++++--------------------- 1 file changed, 53 insertions(+), 46 deletions(-) diff --git a/docs/guides/cicd.md b/docs/guides/cicd.md index a37392955..425f743bf 100644 --- a/docs/guides/cicd.md +++ b/docs/guides/cicd.md @@ -1,99 +1,106 @@ # How to use MFTF in CICD -If you want to integrate MFTF tests into your CICD pipeline, then it's best to start with the conceptual flow of the pipeline code. +To integrate MFTF tests into your CICD pipeline, then it is best to start with the conceptual flow of the pipeline code. ## Concept -The overall workflow that tests should follow is thus: +The overall workflow that tests should follow is: -* Obtain Magento instance + install pre-requisites -* Generate tests for running - * Options for single/parallel running -* Delegate and run tests + gather test run artifacts - * Re-run options -* Generate Allure report from aggregate +- Obtain a Magento instance + install pre-requisites. +- Generate the tests. + - Set options for single or parallel running. +- Delegate and run tests and gather test-run artifacts. + - Re-run options. +- Generate the Allure reports from results. -### Obtain Magento instance +## Obtain a Magento instance -To start, we need a Magento instance to operate against for generation and execution. +To start, we need a Magento instance to operate against for test generation and execution. +```bash +git clone https://github.com/magento/magento2 ``` -$ git clone https://github.com/magento/magento2 + or -$ composer create-project --repository=https://repo.magento.com/ magento/project-community-edition magento2ce + +```bash +composer create-project --repository=https://repo.magento.com/ magento/project-community-edition magento2ce ``` For more information on installing magento see [Install Magento using Composer]. After installing the Magento instance, you need to set a couple of configurations to the magento instance: -``` -$ bin/magento config:set general/locale/timezone America/Los_Angeles -$ bin/magento config:set admin/security/admin_account_sharing 1 -$ bin/magento config:set admin/security/use_form_key 0 -$ bin/magento config:set cms/wysiwyg/enabled disabled +```bash +bin/magento config:set general/locale/timezone America/Los_Angeles +bin/magento config:set admin/security/admin_account_sharing 1 +bin/magento config:set admin/security/use_form_key 0 +bin/magento config:set cms/wysiwyg/enabled disabled ``` -These help set the state for the `default` state of the Magento instance. If you are wanting to change the default state of the application (and have merged into the tests sufficiently to account for it), this is the step in which you would do it. +These help set the `default` state of the Magento instance. If you wish to change the default state of the application (and have updated your tests sufficiently to account for it), this is the step in which you would do it. -#### Install allure +## Install allure This will be required to generate the report after your test runs. See [Allure] for details. +## Generate tests -### Generate tests - -#### Single execution +### Single execution -Simply generate tests based on what you want to run: +Generate tests based on what you want to run: -``` -$ vendor/bin/mftf generate:tests +```bash +vendor/bin/mftf generate:tests ``` -This will generate all tests, and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt` +This will generate all tests and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt` -#### Parallel execution +### Parallel execution To generate all tests for use in parallel nodes: -``` -$ vendor/bin/mftf generate:tests --config parallel +```bash +vendor/bin/mftf generate:tests --config parallel ``` This will generate a folder under `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/groups`. This folder contains several `group#.txt` files that can be used later with the `mftf run:manifest` command. -### Delegate and run tests +## Delegate and run tests -#### Single execution -If you are running on a single node, this step is simply to call: +### Single execution +If you are running on a single node, call: + +```bash +vendor/bin/mftf run:manifest dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt ``` -$ vendor/bin/mftf run:manifest dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt -``` -#### Parallel execution -To run MFTF tests in parallel, you will need to clone the contents of the current node and duplicate them depending on how many nodes you have available for use. +### Parallel execution + +To run MFTF tests in parallel, clone the contents of the current node and duplicate them depending on how many nodes you have. + +- Clone contents of current node as a baseline. +- For each `groups/group#.txt` file: -* Clone contents of current node as a baseline -* For each `groups/group#.txt` file: - * Set a node's contents to the baseline - * Run `vendor/bin/mftf run:manifest <current_group.txt>` - * Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to master + - Set a node's contents to the baseline. + - Run `vendor/bin/mftf run:manifest <current_group.txt>`. + - Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to master. #### Rerun options -In either single or parallel execution, to re-run failed tests simply add a `run:failed` command after executing a manifest: -``` -$ vendor/bin/mftf run:failed +In either single or parallel execution, to re-run failed tests simply add the `run:failed` command after executing a manifest: + +```bash +vendor/bin/mftf run:failed ``` ### Generate Allure report In the master node, simply generate using your `<path_to_results>` into a desired output path -``` -$ allure generate <path_to_results> -c -o <path_to_output> +```bash +allure generate <path_to_results> -c -o <path_to_output> ``` <!-- Link definitions --> From 66ff24dc36e85b6164a2679ff77c1a005211ee50 Mon Sep 17 00:00:00 2001 From: Donald Booth <dobooth@adobe.com> Date: Tue, 29 Oct 2019 11:10:29 -0500 Subject: [PATCH 15/30] Fixed links --- docs/guides/cicd.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/guides/cicd.md b/docs/guides/cicd.md index 425f743bf..de54080ad 100644 --- a/docs/guides/cicd.md +++ b/docs/guides/cicd.md @@ -27,7 +27,7 @@ or composer create-project --repository=https://repo.magento.com/ magento/project-community-edition magento2ce ``` -For more information on installing magento see [Install Magento using Composer]. +For more information on installing magento see [Install Magento using Composer][]. After installing the Magento instance, you need to set a couple of configurations to the magento instance: @@ -42,7 +42,7 @@ These help set the `default` state of the Magento instance. If you wish to chang ## Install allure -This will be required to generate the report after your test runs. See [Allure] for details. +This will be required to generate the report after your test runs. See [Allure][] for details. ## Generate tests From 87f246a088aa7f1850c05a2b9cadd583b1457ec9 Mon Sep 17 00:00:00 2001 From: Soumya Unnikrishnan <sunnikri@adobe.com> Date: Thu, 23 Jul 2020 16:41:05 -0500 Subject: [PATCH 16/30] MQE-2223: Part 7 Docs Review "MFTF Ease Of Use" PR #483 --- docs/guides/cicd.md | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/docs/guides/cicd.md b/docs/guides/cicd.md index de54080ad..689749319 100644 --- a/docs/guides/cicd.md +++ b/docs/guides/cicd.md @@ -1,6 +1,6 @@ # How to use MFTF in CICD -To integrate MFTF tests into your CICD pipeline, then it is best to start with the conceptual flow of the pipeline code. +To integrate MFTF tests into your CICD pipeline, it is best to start with the conceptual flow of the pipeline code. ## Concept @@ -40,6 +40,8 @@ bin/magento config:set cms/wysiwyg/enabled disabled These help set the `default` state of the Magento instance. If you wish to change the default state of the application (and have updated your tests sufficiently to account for it), this is the step in which you would do it. +If your magento instance has Two-Factor Authentication enabled, see [Configure 2FA][] to configure MFTF tests. + ## Install allure This will be required to generate the report after your test runs. See [Allure][] for details. @@ -54,7 +56,7 @@ Generate tests based on what you want to run: vendor/bin/mftf generate:tests ``` -This will generate all tests and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt` +This will generate all tests and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/_generated/testManifest.txt` ### Parallel execution @@ -64,7 +66,7 @@ To generate all tests for use in parallel nodes: vendor/bin/mftf generate:tests --config parallel ``` -This will generate a folder under `dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/groups`. This folder contains several `group#.txt` files that can be used later with the `mftf run:manifest` command. +This will generate a folder under `dev/tests/acceptance/tests/functional/Magento/_generated/groups`. This folder contains several `group#.txt` files that can be used later with the `mftf run:manifest` command. ## Delegate and run tests @@ -73,21 +75,24 @@ This will generate a folder under `dev/tests/acceptance/tests/functional/Magento If you are running on a single node, call: ```bash -vendor/bin/mftf run:manifest dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/testManifest.txt +vendor/bin/mftf run:manifest dev/tests/acceptance/tests/functional/Magento/_generated/testManifest.txt ``` ### Parallel execution -To run MFTF tests in parallel, clone the contents of the current node and duplicate them depending on how many nodes you have. +You can optimize your pipeline by running tests in parallel across multiple nodes. + +Tests can be split up into roughly equal running groups using `--config parallel`. + +You don't want perform installation on each node again and build it. So, to save time, stash pre-made artifacts from earlier steps and un-stash on the nodes. -- Clone contents of current node as a baseline. -- For each `groups/group#.txt` file: +The groups can be then distributed on each of the nodes and run separately in an isolated environment. - - Set a node's contents to the baseline. - - Run `vendor/bin/mftf run:manifest <current_group.txt>`. - - Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to master. + - Stash artifacts from main node and un-stash on current node. + - Run `vendor/bin/mftf run:manifest <current_group.txt>` on current node. + - Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to main node. -#### Rerun options +### Rerun options In either single or parallel execution, to re-run failed tests simply add the `run:failed` command after executing a manifest: @@ -97,12 +102,13 @@ vendor/bin/mftf run:failed ### Generate Allure report -In the master node, simply generate using your `<path_to_results>` into a desired output path +In the main node, simply generate using your `<path_to_results>` into a desired output path ```bash allure generate <path_to_results> -c -o <path_to_output> ``` <!-- Link definitions --> -[Install Magento using Composer]: https://devdocs.magento.com/guides/v2.3/install-gde/composer.html +[Install Magento using Composer]: https://devdocs.magento.com/guides/v2.4/install-gde/composer.html +[Configure 2FA]: ../configure-2fa.md [Allure]: https://docs.qameta.io/allure/ From cf297c44529976818aa9dac9d46ac5b140159b28 Mon Sep 17 00:00:00 2001 From: Donald Booth <dobooth@adobe.com> Date: Thu, 23 Jul 2020 17:10:55 -0500 Subject: [PATCH 17/30] Grammar and formatting --- docs/guides/cicd.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/guides/cicd.md b/docs/guides/cicd.md index 689749319..80cf134d8 100644 --- a/docs/guides/cicd.md +++ b/docs/guides/cicd.md @@ -11,7 +11,7 @@ The overall workflow that tests should follow is: - Set options for single or parallel running. - Delegate and run tests and gather test-run artifacts. - Re-run options. -- Generate the Allure reports from results. +- Generate the Allure reports from the results. ## Obtain a Magento instance @@ -29,7 +29,7 @@ composer create-project --repository=https://repo.magento.com/ magento/project-c For more information on installing magento see [Install Magento using Composer][]. -After installing the Magento instance, you need to set a couple of configurations to the magento instance: +After installing the Magento instance, set a couple of configurations to the Magento instance: ```bash bin/magento config:set general/locale/timezone America/Los_Angeles @@ -38,13 +38,13 @@ bin/magento config:set admin/security/use_form_key 0 bin/magento config:set cms/wysiwyg/enabled disabled ``` -These help set the `default` state of the Magento instance. If you wish to change the default state of the application (and have updated your tests sufficiently to account for it), this is the step in which you would do it. +These set the default state of the Magento instance. If you wish to change the default state of the application (and have updated your tests sufficiently to account for it), this is the step to do it. If your magento instance has Two-Factor Authentication enabled, see [Configure 2FA][] to configure MFTF tests. -## Install allure +## Install Allure -This will be required to generate the report after your test runs. See [Allure][] for details. +This is required for generating the report after your test runs. See [Allure][] for details. ## Generate tests @@ -56,7 +56,7 @@ Generate tests based on what you want to run: vendor/bin/mftf generate:tests ``` -This will generate all tests and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/_generated/testManifest.txt` +This will generate all tests and a single manifest file under `dev/tests/acceptance/tests/functional/Magento/_generated/testManifest.txt`. ### Parallel execution @@ -66,7 +66,7 @@ To generate all tests for use in parallel nodes: vendor/bin/mftf generate:tests --config parallel ``` -This will generate a folder under `dev/tests/acceptance/tests/functional/Magento/_generated/groups`. This folder contains several `group#.txt` files that can be used later with the `mftf run:manifest` command. +This generates a folder under `dev/tests/acceptance/tests/functional/Magento/_generated/groups`. This folder contains several `group#.txt` files that can be used later with the `mftf run:manifest` command. ## Delegate and run tests @@ -84,17 +84,17 @@ You can optimize your pipeline by running tests in parallel across multiple node Tests can be split up into roughly equal running groups using `--config parallel`. -You don't want perform installation on each node again and build it. So, to save time, stash pre-made artifacts from earlier steps and un-stash on the nodes. +You do not want to perform installations on each node again and build it. So, to save time, stash pre-made artifacts from earlier steps and un-stash on the nodes. The groups can be then distributed on each of the nodes and run separately in an isolated environment. - - Stash artifacts from main node and un-stash on current node. - - Run `vendor/bin/mftf run:manifest <current_group.txt>` on current node. - - Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to main node. +- Stash artifacts from main node and un-stash on current node. +- Run `vendor/bin/mftf run:manifest <current_group.txt>` on current node. +- Gather artifacts from `dev/tests/acceptance/tests/_output` from current node to main node. ### Rerun options -In either single or parallel execution, to re-run failed tests simply add the `run:failed` command after executing a manifest: +In either single or parallel execution, to re-run failed tests, simply add the `run:failed` command after executing a manifest: ```bash vendor/bin/mftf run:failed @@ -102,7 +102,7 @@ vendor/bin/mftf run:failed ### Generate Allure report -In the main node, simply generate using your `<path_to_results>` into a desired output path +In the main node, generate reports using your `<path_to_results>` into a desired output path: ```bash allure generate <path_to_results> -c -o <path_to_output> From eda8252c2a34907b3cfbd154247f749f1d24320f Mon Sep 17 00:00:00 2001 From: soumyau <sunnikri@adobe.com> Date: Fri, 24 Jul 2020 08:12:19 -0500 Subject: [PATCH 18/30] ME-2217: Part 4 Docs Annual Review (#763) --- docs/test/actions.md | 6 +++--- docs/test/annotations.md | 2 +- docs/versioning.md | 2 ++ 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/test/actions.md b/docs/test/actions.md index 1d2f83802..9a51106de 100644 --- a/docs/test/actions.md +++ b/docs/test/actions.md @@ -95,7 +95,7 @@ Here, `url` contains a pointer to a `url` attribute of the `StorefrontCustomerSi ### 2. Enter a customer's email {#example-step2} ```xml -<fillField userInput="$$customer.email$$" selector="{{StorefrontCustomerSignInFormSection.emailField}}" stepKey="fillEmail"/> +<fillField userInput="$customer.email$" selector="{{StorefrontCustomerSignInFormSection.emailField}}" stepKey="fillEmail"/> ``` [`<fillField>`](#fillfield) fills a text field with the given string. @@ -122,7 +122,7 @@ This section is declared in `.../Customer/Section/StorefrontCustomerSignInFormSe ### 3. Enter a customer's password {#example-step3} ```xml -<fillField userInput="$$customer.password$$" selector="{{StorefrontCustomerSignInFormSection.passwordField}}" stepKey="fillPassword"/> +<fillField userInput="$customer.password$" selector="{{StorefrontCustomerSignInFormSection.passwordField}}" stepKey="fillPassword"/> ``` This `<action>` is very similar to the `<action>` in a previous step. @@ -218,7 +218,7 @@ Attribute|Type|Use|Description ```xml <!-- Open the `(baseURL)/admin` page. --> -<amOnPage url="/admin" stepKey="goToLogoutPage"/> +<amOnPage url="{{AdminLogoutPage.url}}" stepKey="goToLogoutPage"/> ``` ### amOnSubdomain diff --git a/docs/test/annotations.md b/docs/test/annotations.md index 4e9194bb0..8fdcd2240 100644 --- a/docs/test/annotations.md +++ b/docs/test/annotations.md @@ -112,7 +112,7 @@ Attribute|Type|Use ### severity -The `<return>` element is an implementation of a [`@Severity`] Allure tag; Metadata for report. +The `<severity>` element is an implementation of a [`@Severity`] Allure tag; Metadata for report. Attribute|Type|Use|Acceptable values ---|---|---|--- diff --git a/docs/versioning.md b/docs/versioning.md index 15b98f743..3cb51e405 100644 --- a/docs/versioning.md +++ b/docs/versioning.md @@ -61,6 +61,8 @@ This table lists the version of the MFTF that was released with a particular ver |Magento version| MFTF version| |--- |--- | +| 2.4.0 | 3.0.0 | +| 2.3.5 | 2.6.4 | | 2.3.4 | 2.5.3 | | 2.3.3 | 2.4.5 | | 2.3.2 | 2.3.14 | From 7a21f39bc8ece5842509dda3586d649aaf89d45e Mon Sep 17 00:00:00 2001 From: soumyau <sunnikri@adobe.com> Date: Fri, 24 Jul 2020 08:13:44 -0500 Subject: [PATCH 19/30] MQE-2219: Part 6 Docs Annual Review (#767) * MQE-2219: Part 6 Docs Annual Review * MQE-2219: Part 6 Docs Annual Review * MQE-2219: Part 6 Docs Annual Review --- docs/section.md | 24 ++++++++++++------------ docs/section/locator-functions.md | 2 +- docs/suite.md | 10 +++++----- docs/update.md | 16 +++++++++++++--- 4 files changed, 31 insertions(+), 21 deletions(-) diff --git a/docs/section.md b/docs/section.md index 421c98fdd..68a6507af 100644 --- a/docs/section.md +++ b/docs/section.md @@ -58,11 +58,11 @@ Example (`.../Catalog/Section/AdminCategorySidebarActionSection.xml` file): <?xml version="1.0" encoding="utf-8"?> <sections xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" - xsi:noNamespaceSchemaLocation="urn:magento:mftf:Page/etc/SectionObject.xsd"> - <section name="AdminCategorySidebarActionSection"> - <element name="addRootCategoryButton" type="button" selector="#add_root_category_button" timeout="30"/> - <element name="addSubcategoryButton" type="button" selector="#add_subcategory_button" timeout="30"/> - </section> + xsi:noNamespaceSchemaLocation="urn:magento:mftf:Page/etc/SectionObject.xsd"> + <section name="AdminCategorySidebarActionSection"> + <element name="AddRootCategoryButton" type="button" selector="#add_root_category_button" timeout="30"/> + <element name="AddSubcategoryButton" type="button" selector="#add_subcategory_button" timeout="30"/> + </section> </sections> ``` @@ -70,14 +70,14 @@ This example uses a `AdminCategorySidebarActionSection` section. All sections wi The `AdminCategorySidebarActionSection` section declares two buttons: -- `addRootCategoryButton` - button with a `#add_root_category_button` locator on the parent web page -- `addSubcategoryButton` - button with a `#add_subcategory_button` locator on the parent web page +- `AddRootCategoryButton` - button with a `#add_root_category_button` locator on the parent web page +- `AddSubcategoryButton` - button with a `#add_subcategory_button` locator on the parent web page The following is an example of a call in test: ```xml <!-- Click on the button with locator "#add_subcategory_button" on the web page--> -<click selector="{{AdminCategorySidebarActionSection.addSubcategoryButton}}" stepKey="clickOnAddSubCategory"/> +<click selector="{{AdminCategorySidebarActionSection.AddSubcategoryButton}}" stepKey="clickOnAddSubCategory"/> ``` ## Elements reference @@ -116,21 +116,21 @@ The most usual use case is a test step with a button click action. The section element code declaration containing the timeout attribute: -> StorefrontSigninSection.xml +> StorefrontCustomerSignInPopupFormSection.xml ```xml ... -<element name="signIn" type="button" selector="#signIn" timeout="30"/> +<element name="signIn" type="button" selector="#send2" timeout="30"/> ... ``` The test step that covers the use case: -> StorefrontSigninTest.xml +> CaptchaWithDisabledGuestCheckoutTest.xml ```xml ... -<click selector="{{StorefrontSigninSection.signIn}}" ../> +<click selector="{{StorefrontCustomerSignInPopupFormSection.signIn}}" stepKey="clickSignIn"/> ... ``` diff --git a/docs/section/locator-functions.md b/docs/section/locator-functions.md index 1e3dffd24..d2dc9924c 100644 --- a/docs/section/locator-functions.md +++ b/docs/section/locator-functions.md @@ -31,7 +31,7 @@ An element cannot, however, have both a `selector` and a `locatorFunction`. Given the above element definitions, you call the elements in a test just like any other element. No special reference is required, as you are still just referring to an `element` inside a `section`: ```xml -<test name="LocatorFuctionTest"> +<test name="LocatorFunctionTest"> <click selector="{{LocatorFunctionSection.simpleLocator}}" stepKey="SimpleLocator"/> <click selector="{{LocatorFunctionSection.simpleLocatorTwoParam('string1', 'string2')}}" stepKey="TwoParamLiteral"/> </test> diff --git a/docs/suite.md b/docs/suite.md index 7917cd347..9f623f276 100644 --- a/docs/suite.md +++ b/docs/suite.md @@ -6,8 +6,8 @@ You can form suites using separate tests, groups, and modules. Each suite must be defined in the `<VendorName>/<ModuleName>/Test/Mftf/Suite` directory. -The tests for each suite are generated in a separate directory under `<magento 2 root>/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/`. -All tests that are not within a suite are generated in the _default_ suite at `.../Magento/FunctionalTest/_generated/default/`. +The tests for each suite are generated in a separate directory under `<magento 2 root>/dev/tests/acceptance/tests/functional/Magento/_generated/`. +All tests that are not within a suite are generated in the _default_ suite at `<magento 2 root>/dev/tests/acceptance/tests/functional/Magento/_generated/default`. <div class="bs-callout bs-callout-info"> If a test is generated into at least one custom suite, it will not appear in the _default_ suite. @@ -61,7 +61,7 @@ The code lives in one place and executes once per suite. - Set up preconditions and postconditions using [actions] in [`<before>`] and [`<after>`] correspondingly, just similar to use in a [test]. - Clean up after suites just like after tests. - The MFTF enforces the presence of both `<before>` and `<after>` if either is present. +MFTF enforces the presence of both `<before>` and `<after>` if either is present. ## Test writing @@ -142,7 +142,7 @@ This suite includes all tests that contain the `<group value="WYSIWYG"/>` annota ### Execute Magento CLI commands in suite conditions ```xml -<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Suite/etc/suiteSchema.xsd"> +<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:mftf:Suite/etc/suiteSchema.xsd"> <suite name="Cache"> <before> <magentoCLI stepKey="disableCache" command="cache:disable"/> @@ -170,7 +170,7 @@ The suite includes a specific test `SomeCacheRelatedTest` and every `<test>` tha ### Change Magento configurations in suite conditions ```xml -<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../../dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Suite/etc/suiteSchema.xsd"> +<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:mftf:Suite/etc/suiteSchema.xsd"> <suite name="PaypalConfiguration"> <before> <createData entity="SamplePaypalConfig" stepKey="createSamplePaypalConfig"/> diff --git a/docs/update.md b/docs/update.md index ebd096158..dc5e7fbc8 100644 --- a/docs/update.md +++ b/docs/update.md @@ -1,7 +1,9 @@ # Update the Magento Functional Testing Framework <div class="bs-callout bs-callout-info" markdown="1"> -Both Magento `2.2` and `2.3` supports MFTF `2.5.3` ([Find your version][] of the MFTF). +[Find your version] of MFTF. +The latest Magento 2.4.x release supports MFTF 3.x. +The latest Magento 2.3.x release supports MFTF 2.6.x. </div> Tests and the Framework itself are stored in different repositories. @@ -26,16 +28,24 @@ Takes place when **second** digit of version number changes. 1. Check details about backward incompatible changes in the [Changelog][] and update your new or customized tests. 1. Perform all the actions provided for [Patch Version Update][] 1. When updating from versions below `2.5.0`, verify [WYSIWYG settings][] -1. You may need to run the `upgrade:tests` using `vendor/bin/mftf upgrade:tests app` +1. You may need to run the `upgrade:tests` using `vendor/bin/mftf upgrade:tests` + +## Major version update + +Takes place when **first** digit of version number changes. + +1. Check detailed explanation and instructions on major version upgrade in [Backward Incompatible Changes][] and upgrade your new or customized tests. +1. Perform all the actions provided for [Minor Version Update][] ## After updating 1. It is a good idea to regenerate your IDE Schema Definition catalog with `vendor/bin/mftf generate:urn-catalog .idea/` -1. Update your tests, including data, metadata and other resoruces. Check if they contain tags that are unsupported in the newer version. +1. Update your tests, including data, metadata and other resources. Check if they contain tags that are unsupported in the newer version. 1. Remove the references to resources (ActionGroups, Sections, Tests) marked as deprecated. <!-- Link Definitions --> [Changelog]: https://github.com/magento/magento2-functional-testing-framework/blob/master/CHANGELOG.md +[Backward Incompatible Changes]: backward-incompatible-changes.md [WYSIWYG settings]: getting-started.md#wysiwyg-settings [Security settings]: getting-started.md#security-settings [Find your version]: introduction.md#find-your-mftf-version From bd46a50414f98225c72f1c5436f227687cf3271c Mon Sep 17 00:00:00 2001 From: Tom Reece <treece@adobe.com> Date: Fri, 24 Jul 2020 09:20:04 -0500 Subject: [PATCH 20/30] Remove accidental v2 directory --- v2 | 1 - 1 file changed, 1 deletion(-) delete mode 160000 v2 diff --git a/v2 b/v2 deleted file mode 160000 index 3343a82b8..000000000 --- a/v2 +++ /dev/null @@ -1 +0,0 @@ -Subproject commit 3343a82b84e5b25d30e6e39fb3d1aa0d8dc70634 From 995a88c225a37df841d92ccd27474c72c088d746 Mon Sep 17 00:00:00 2001 From: Alastair Mucklow <amucklow@strangerpixel.com> Date: Wed, 8 Jul 2020 11:45:40 +0100 Subject: [PATCH 21/30] Replace broken devhub.io/zh paths with github.com paths --- docs/test/annotations.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/test/annotations.md b/docs/test/annotations.md index 8fdcd2240..055ca9450 100644 --- a/docs/test/annotations.md +++ b/docs/test/annotations.md @@ -213,14 +213,14 @@ Attribute|Type|Use <!-- Link definitions --> -[`@Description`]: https://devhub.io/zh/repos/allure-framework-allure-phpunit#extended-test-class-or-test-method-description -[`@Features`]: https://devhub.io/zh/repos/allure-framework-allure-phpunit#map-test-classes-and-test-methods-to-features-and-stories +[`@Description`]: https://github.com/allure-framework/allure-phpunit#extended-test-class-or-test-method-description +[`@Features`]: https://github.com/allure-framework/allure-phpunit#map-test-classes-and-test-methods-to-features-and-stories [`@group`]: http://codeception.com/docs/07-AdvancedUsage#Groups [`@return`]: http://codeception.com/docs/07-AdvancedUsage#Examples -[`@Severity`]: https://devhub.io/zh/repos/allure-framework-allure-phpunit#set-test-severity -[`@Stories`]: https://devhub.io/zh/repos/allure-framework-allure-phpunit#map-test-classes-and-test-methods-to-features-and-stories +[`@Severity`]: https://github.com/allure-framework/allure-phpunit#set-test-severity +[`@Stories`]: https://github.com/allure-framework/allure-phpunit#map-test-classes-and-test-methods-to-features-and-stories [`@TestCaseId`]: https://github.com/allure-framework/allure1/wiki/Test-Case-ID -[`@Title`]: https://devhub.io/zh/repos/allure-framework-allure-phpunit#human-readable-test-class-or-test-method-title +[`@Title`]: https://github.com/allure-framework/allure-phpunit#human-readable-test-class-or-test-method-title [description]: #description [features]: #features [group]: #group From 977b1ef43ed7be91fc29ca9ef89f11d756bcff28 Mon Sep 17 00:00:00 2001 From: Alastair Mucklow <amucklow@strangerpixel.com> Date: Wed, 8 Jul 2020 12:22:27 +0100 Subject: [PATCH 22/30] Make intro text clearer and more accurate --- docs/test/annotations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/test/annotations.md b/docs/test/annotations.md index 055ca9450..d61565aec 100644 --- a/docs/test/annotations.md +++ b/docs/test/annotations.md @@ -112,7 +112,7 @@ Attribute|Type|Use ### severity -The `<severity>` element is an implementation of a [`@Severity`] Allure tag; Metadata for report. +The `<severity>` element is an implementation of the [`@Severity`] Allure annotation, which is used to prioritise test methods by severity. Attribute|Type|Use|Acceptable values ---|---|---|--- From 8263ade22520d71eda59694ab11a04b604c9d755 Mon Sep 17 00:00:00 2001 From: Alastair Mucklow <amucklow@strangerpixel.com> Date: Wed, 8 Jul 2020 12:22:42 +0100 Subject: [PATCH 23/30] Fix order (Blocker is the most severe) --- docs/test/annotations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/test/annotations.md b/docs/test/annotations.md index d61565aec..69c75a2d0 100644 --- a/docs/test/annotations.md +++ b/docs/test/annotations.md @@ -116,7 +116,7 @@ The `<severity>` element is an implementation of the [`@Severity`] Allure annota Attribute|Type|Use|Acceptable values ---|---|---|--- -`value`|string|required|`MINOR`, `AVERAGE`, `MAJOR`, `BLOCKER`, `CRITICAL` +`value`|string|required|`MINOR`, `AVERAGE`, `MAJOR`, `CRITICAL`, `BLOCKER` #### Example From b871ae5bf1a7a94dd13132bdf675a38587ef08c2 Mon Sep 17 00:00:00 2001 From: Alastair Mucklow <amucklow@strangerpixel.com> Date: Wed, 8 Jul 2020 12:23:14 +0100 Subject: [PATCH 24/30] Add usage guidelines table outlining the difference between levels --- docs/test/annotations.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/test/annotations.md b/docs/test/annotations.md index 69c75a2d0..1e2e50f58 100644 --- a/docs/test/annotations.md +++ b/docs/test/annotations.md @@ -124,6 +124,16 @@ Attribute|Type|Use|Acceptable values <severity value="CRITICAL"/> ``` +#### Usage guidelines + +Severity Level|Usage +---|--- +`BLOCKER`|If this test fails, the customer is completely blocked from purchasing a product. +`CRITICAL`|This is a serious problem impacting conversion, or affecting the operation of the store. +`MAJOR`|Store conversion rate is reduced owing to this issue. For example, something is broken or missing that impacts checkout frequency or cart volume. +`AVERAGE`|A fault on the storefront that can negatively impact conversion rate (like UI errors or omissions), or problems with Magento admin functionality. +`MINOR`|An application or configuration fault that has no impact on conversion rate. + ### skip Use the `<skip>` element to skip a test. From 50249bc75cc6a72cef12997124c3420ab0033878 Mon Sep 17 00:00:00 2001 From: Alastair Mucklow <amucklow@strangerpixel.com> Date: Wed, 8 Jul 2020 12:34:24 +0100 Subject: [PATCH 25/30] Trim text --- docs/test/annotations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/test/annotations.md b/docs/test/annotations.md index 1e2e50f58..79cc00e80 100644 --- a/docs/test/annotations.md +++ b/docs/test/annotations.md @@ -112,7 +112,7 @@ Attribute|Type|Use ### severity -The `<severity>` element is an implementation of the [`@Severity`] Allure annotation, which is used to prioritise test methods by severity. +The `<severity>` element is an implementation of the [`@Severity`] Allure annotation, which is used to prioritise tests by severity. Attribute|Type|Use|Acceptable values ---|---|---|--- From 27dd329fddf69f0c2b061c33d6d6fa0f47efdd23 Mon Sep 17 00:00:00 2001 From: Ajith <ajithkumar.maragathavel@ziffity.com> Date: Sun, 19 Apr 2020 14:57:43 +0530 Subject: [PATCH 26/30] command added to modifiy the web server rewrites config --- docs/getting-started.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/getting-started.md b/docs/getting-started.md index af4f8d62e..c2d10c0cd 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -121,6 +121,18 @@ MFTF does not support executing CLI commands if your web server points to `<MAGE If the Nginx Web server is used on your development environment, then **Use Web Server Rewrites** setting in **Stores** > Settings > **Configuration** > **General** > **Web** > **Search Engine Optimization** must be set to **Yes**. +or via command line: + +```bash +bin/magento config:set web/seo/use_rewrites 1 +``` + +Clean the cache after changing the configuration values: + +```bash +bin/magento cache:clean config full_page +``` + To be able to run Magento command line commands in tests, add the following location block to the Nginx configuration file in the Magento root directory: ```conf From a1e81791f0e108e653d801e61c628b7200b298c1 Mon Sep 17 00:00:00 2001 From: Donald Booth <dobooth@adobe.com> Date: Fri, 24 Apr 2020 09:30:48 -0500 Subject: [PATCH 27/30] Grammar --- docs/getting-started.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/getting-started.md b/docs/getting-started.md index c2d10c0cd..1c8200e3b 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -121,13 +121,13 @@ MFTF does not support executing CLI commands if your web server points to `<MAGE If the Nginx Web server is used on your development environment, then **Use Web Server Rewrites** setting in **Stores** > Settings > **Configuration** > **General** > **Web** > **Search Engine Optimization** must be set to **Yes**. -or via command line: +Or via command line: ```bash bin/magento config:set web/seo/use_rewrites 1 ``` -Clean the cache after changing the configuration values: +You must clean the cache after changing the configuration values: ```bash bin/magento cache:clean config full_page From ebd9a94d8469461d76c66032cfe2e3f10d46d15c Mon Sep 17 00:00:00 2001 From: Donald Booth <dobooth@Donalds-MacBook-Pro.local> Date: Fri, 24 Jul 2020 14:46:36 -0500 Subject: [PATCH 28/30] Link fix --- docs/introduction.md | 7 ++++--- v2/docs/introduction.html | 0 2 files changed, 4 insertions(+), 3 deletions(-) create mode 100644 v2/docs/introduction.html diff --git a/docs/introduction.md b/docs/introduction.md index 5a7869381..e5ce67d35 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -1,8 +1,9 @@ # Introduction to the Magento Functional Testing Framework <div class="bs-callout bs-callout-info" markdown="1"> -These are the docs for the latest MFTF release. -To find older documentation, please refer to the [docs folder] of your desired release in Github. +This documentation is for MFTF 3.0, which was release in conjunction with Magento 2.4. +MFTF 3.0 is a major update and introduces many new changes and fixes. +MFTF 2 docs can be found [here][]. </div> [Find your version] of MFTF. @@ -159,4 +160,4 @@ Follow the [MFTF project] and [contribute on Github]. [contribute on Github]: https://github.com/magento/magento2-functional-testing-framework/blob/master/.github/CONTRIBUTING.md [MFTF project]: https://github.com/magento/magento2-functional-testing-framework [Find your version]: #find-your-mftf-version -[docs folder]: https://github.com/magento/magento2-functional-testing-framework/tree/master/docs +[here]: ../v2/docs/introduction.html diff --git a/v2/docs/introduction.html b/v2/docs/introduction.html new file mode 100644 index 000000000..e69de29bb From f7e34266a1b7f242a64dea1b318bb3abcb0f6695 Mon Sep 17 00:00:00 2001 From: Tom Reece <treece@adobe.com> Date: Fri, 24 Jul 2020 14:57:24 -0500 Subject: [PATCH 29/30] Remove accidental v2 folder --- v2/docs/introduction.html | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 v2/docs/introduction.html diff --git a/v2/docs/introduction.html b/v2/docs/introduction.html deleted file mode 100644 index e69de29bb..000000000 From 332da12b7f34a0ac7580c77807718abde5e3449c Mon Sep 17 00:00:00 2001 From: Donald Booth <dobooth@Donalds-MacBook-Pro.local> Date: Fri, 24 Jul 2020 15:02:59 -0500 Subject: [PATCH 30/30] Remove stray file. --- v2/docs/introduction.html | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 v2/docs/introduction.html diff --git a/v2/docs/introduction.html b/v2/docs/introduction.html deleted file mode 100644 index e69de29bb..000000000