Skip to content

Lbajsarowicz doc rollup #641

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Mar 19, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 50 additions & 14 deletions docs/best-practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,46 @@

Check out our best practices below to ensure you are getting the absolute most out of the Magento Functional Testing Framework.

## Focus on reusability

### Use existing Tests and resources

Magento offers more than **3000** acceptance tests, **2500** [Action group]s, **750** Page declarations with more than **1500** Section definitions.
It is very probable that behaviour you want to test already exists as a Test or Action Group.
Instead of writing everything by yourself - use `extends` attribute to refer to existing element and customize it.

**Reusable Resources**

* Tests (reusable with `<test extends="...">` argument)
* Action Group (reusable with including `<actionGroup ref="...">`, or extending `<actionGroup extends="...">`)
* Pages (reusable with reference `{{PageDefinition.url}}`)
* Sections (reusable with reference `{{SectionDefinition.elementDefinition}}`)
* Data Entities (reusable with reference `<createData entity="...">"` or extending `<entity extends="...">`)

<div class="bs-callout bs-callout-warning" markdown="1">

Avoid using resources that are marked as **Deprecated**. Usually there is a replacement provided for a deprecated resource.

</div>

### Extract repetitive Actions

Instead of writing a few of Tests that perform mostly the same actions, you should thing about [Action group] that is a container for repetitive Actions.
If each run needs different data, use `<arguments>` to inject necessary information.

We recommend to keep Action Groups having single responsibility, for example `AdminLoginActionGroup`, which expected outcome is being logged in as Administrator when [Action group] is executed.

## Contribute

Althought the Magento Core team and Contributors join forces to cover most of the features with tests, it is impossible to have this done quickly.
If you've covered Magento Core feature with Functional Tests - you are more than welcome to contribute.

You can also help with MFTF Test Migration to get the experience and valuable feedback from other community members and maintainers.

## Action group

1. [Action group] names should be sufficiently descriptive to inform a test writer of what the action group does and when it should be used.
Add additional explanation in annotations if needed.
2. Provide default values for the arguments that apply to your most common case scenarios.
1. [Action group] names should be sufficiently descriptive to inform a test writer of what the action group does and when it should be used. Add additional explanation in annotations if needed.
1. Provide default values for the arguments that apply to your most common case scenarios.

## `actionGroups` vs `extends`

Expand Down Expand Up @@ -94,20 +129,20 @@ Use the _Foo.camelCase_ naming convention, which is similar to _Classes_ and _cl

Use an upper case first letter for:

- File names. Example: _StorefrontCreateCustomerTest.xml_
- Test name attributes. Example: `<test name="TestAllTheThingsTest">`
- Data entity names. Example: `<entity name="OutOfStockProduct">`
- Page name. Example: `<page name="AdminLoginPage">`
- Section name. Example: `<section name="AdminCategorySidebarActionSection">`
- Action group name. Example: `<actionGroup name="LoginToAdminActionGroup">`
* File names. Example: _StorefrontCreateCustomerTest.xml_
* Test name attributes. Example: `<test name="TestAllTheThingsTest">`
* Data entity names. Example: `<entity name="OutOfStockProduct">`
* Page name. Example: `<page name="AdminLoginPage">`
* Section name. Example: `<section name="AdminCategorySidebarActionSection">`
* Action group name. Example: `<actionGroup name="LoginToAdminActionGroup">`

#### Lower case

Use a lower case first letter for:

- Data keys. Example: `<data key="firstName">`
- Element names. Examples: `<element name="confirmDeleteButton"/>`
- Step keys. For example: `<click selector="..." stepKey="clickLogin"/>`
* Data keys. Example: `<data key="firstName">`
* Element names. Examples: `<element name="confirmDeleteButton"/>`
* Step keys. For example: `<click selector="..." stepKey="clickLogin"/>`

## Page object

Expand Down Expand Up @@ -145,9 +180,9 @@ Define these three elements and reference them by name in the tests.
1. Keep your tests short and granular for target testing, easier reviews, and easier merge conflict resolution.
It also helps you to identify the cause of test failure.
1. Use comments to keep tests readable and maintainable:
- Keep the inline `<!-- XML comments -->` and [`<comment>`] tags up to date.
* Keep the inline `<!-- XML comments -->` and [`<comment>`] tags up to date.
It helps to inform the reader of what you are testing and to yield a more descriptive Allure report.
- Explain in comments unclear or tricky test steps.
* Explain in comments unclear or tricky test steps.
1. Refer to [sections] instead of writing selectors.

## Test step merging order
Expand Down Expand Up @@ -175,3 +210,4 @@ Since the configurable product module could be disabled, this approach is more r
[merging]: merging.html
[parameterized selectors]: section/parameterized-selectors.html
[sections]: section.html
[MFTF Test Migration]: https://github.com/magento/magento-functional-tests-migration
10 changes: 10 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,16 @@ Example:
MAGENTO_BACKEND_NAME=admin_12346
```

### MAGENTO_BACKEND_BASE_URL

(Optional) If you are running the Admin Panel on separate a domain, specify this value:

Example:

```conf
MAGENTO_BACKEND_BASE_URL=https://admin.magento2.test
```

### MAGENTO_ADMIN_USERNAME

The username that tests can use to access the Magento Admin page
Expand Down
Binary file added docs/img/mftf-extending-precedence.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
121 changes: 55 additions & 66 deletions docs/introduction.md
Original file line number Diff line number Diff line change
@@ -1,49 +1,79 @@
# Introduction to the Magento Functional Testing Framework

[Find your MFTF version][] of the MFTF.
The Magento Functional Tesitng Framework (MFTF) is a framework used to perform automated end-to-end functional testing.

The Magento Functional Testing Framework (MFTF) aims to replace the [Functional Testing Framework] in future releases.
MFTF improves:
## Goals

- To facilitate functional testing and minimize the effort it takes to perform regression testing.
- Enable extension developers to provide the Functional Tests to offered extensions.
- Ensuring a common standard of quality between Magento, Extension Developers and System Intergrators.

MFTF also focuses on

- **Traceability** for clear logging and reporting capabilities.
- **Modularity** to run tests based on installed modules and extensions.
- **Customizability** for existing tests.
- **Readability** using clear and declarative XML test steps.
- **Maintainability** based on simple test creation and overall structure.

Because MFTF tests are written in XML, you no longer need to learn PHP to write tests.
## Audience

<div class="bs-callout bs-callout-info" markdown="1">
We are actively developing functional tests.
Refer to `<magento_root>/app/code/<vendor_name>/<module_name>/Test/Mftf/` for examples.
Check out the [MFTF Test Migration][] repo.
</div>
- **Contributors**: Tests build confidence about the results of changes introduced to the platform.
- **Extension Developers**: Can adjust expected behaviour according to their customizations.
- **System Integrators**: MFTF coverage provided out-of-the-box with Magento is solid base for Acceptance / Regression Tests.

## Audience
## MFTF tests

The MFTF supports two different locations for storing the tests and test artifacts:

This MFTF guide is intended for Magento developers and software engineers, such as QA specialists, PHP developers, and system integrators.
- `<magento_root>/app/code/<vendor_name>/<module_name>/Test/Mftf/` is the location of local, customized tests.
- `<magento_root>/vendor/<vendor_name>/<module_name>/Test/Mftf/` is location of tests provided by Magento and vendors.

## Goals
If you installed Magento with Composer, please refer to `vendor/magento/<module_dir>/Test/Mftf/` for examples.

### Directory Structure

The purpose of MFTF is to:
The file structure under both of the both path cases is the same:

- Facilitate functional testing and minimize the effort it takes to perform regression testing.
- Make it easier to support the extension and customization of tests via XML merging.
```tree
Test
└── Mftf
├── ActionGroup
│   └── ...
├── Data
│   └── ...
├── Metadata
│   └── ...
├── Page
│   └── ...
├── Section
│   └── ...
└── Test
└── ...
```

## Scope
<div class="bs-callout bs-callout-info" markdown="1">

MFTF will enable you to:
We are actively developing functional tests. Check out the [MFTF Test Migration][] repository.

- Test user interactions with web applications in testing.
- Write functional tests located in `<magento_root>/app/code/<vendor_name>/<module_name>/Test/Mftf/`.
- Cover basic functionality using out-of-the-box tests. You can test extended functionality using custom tests.
- Automate regression testing.
</div>

## Use cases

As a Magento developer, test changes, such as extended search functionality, a new form attribute, or new product tags.
- Contributor: changes the core behaviour, fixing the annoing bug.
He wants to have automated "supervisor" which is going to verify his work continuously across the stages of bug fixing. Finally, when fix is done - Functional Test is also proof of work done.
- Extension Developer: offers extension that changes core behaviour.
He can easily write new tests to make sure that after enabling the feature, Magento behaves properly. Everything with just extending existing tests. As a result he don't need to write coverage from scratch.
- Integration Agency: maintains Client's e-commerce.
They are able to customize tests delivered with Magento core to follow customizations implemented to Magento. After each upgrade they can just run the MFTF tests to know that no regression was introduced.

## MFTF output

As a software engineer, perform regression testing before release to ensure that Magento works as expected with new functionality.
- Generated PHP Codeception tests
- Codeception results and console logs
- Screenshots and HTML failure report
- Allure formatted XML results
- Allure report dashboard of results

## Find your MFTF version

Expand All @@ -52,22 +82,16 @@ There are two options to find out your MFTF version:
- using the MFTF CLI
- using the Composer CLI

### MFTF CLI
All the Command Line commands needs to be executed from `<magento_root>`

```bash
cd <magento_root>/
```
### MFTF CLI

```bash
vendor/bin/mftf --version
```

### Composer CLI

```bash
cd <magento_root>/
```

```bash
composer show magento/magento2-functional-testing-framework
```
Expand All @@ -89,41 +113,6 @@ utils // The test-running utilities.
codeception.dist.yml // Codeception configuration (generated while running 'bin/mftf build:project')
```

## MFTF output

- Generated PHP Codeception tests
- Codeception results and console logs
- Screenshots and HTML failure report
- Allure formatted XML results
- Allure report dashboard of results

## MFTF tests

The MFTF supports two different locations for storing the tests and test artifacts:

- `<magento_root>/app/code/<vendor_name>/<module_name>/Test/Mftf/` is the directory to create new tests.
- `<magento_root>/vendor/<vendor_name>/<module_name>/Test/Mftf/` is the directory with the out of the box tests (fetched by the Composer).

All tests and test data from these locations are merged in the order indicated in the above list.

The file structure under the both path cases is the same:

```tree
<Path>
├── ActionGroup
│   └── ...
├── Data
│   └── ...
├── Metadata
│   └── ...
├── Page
│   └── ...
├── Section
│   └── ...
└── Test
└── ...
```

## MFTF on Github

Follow the [MFTF project] and [contribute on Github].
Expand Down
10 changes: 10 additions & 0 deletions docs/merging.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,15 @@ For example:
Although a file name does not influence merging, we recommend using the same file names in merging updates.
This makes it easier to search later on.

## Merging precedence

**Magento Functional Testing Framework** uses Module's `<sequence>` to merge all XML configurations into Codeception instructions. If there's no Sequence specified, MFTF would use:

1. Vendor modules (Magento & Vendors) located in `vendor/`
1. Tests located in `app/code/*/*/Test/Mftf`

![Usual precedence for merging MFTF Tests and resources][mftfExtendingPrecedence image]

## Add a test

You cannot add another [`<test>`][tests] using merging functionality.
Expand Down Expand Up @@ -570,3 +579,4 @@ The `_defaultSample` results corresponds to:
[`<sections>`]: ./section.md
[`<tests>`]: ./test.md
[`<action groups>`]: ./test/action-groups.md
[mftfExtendingPrecedence image]: img/mftf-extending-precedence.png
Loading