[MFTF] Final set of changes to improve the MFTF tests

[lbajsarowicz]

Description (*)

This PR contains all other changes I introduced to MFTF and some more:

• Unified stepKey to be camelCase (lower first)
• Added ActionGroup suffix to all Action Groups
• Updated documentation to follow best practices

Fixed Issues (if relevant)

1. [ MFTF ] Mess in ActionGroups: invalid names, multiple nodes. #26331

Manual testing scenarios (*)

Run MFTF tests for:

• https://github.com/magento/magento2
• https://github.com/magento/partners-magento2b2b
• https://github.com/magento/partners-magento2ee
• https://github.com/magento/magento2-page-builder
• https://github.com/magento/magento2-page-builder-ee

Related Pull Requests

• [MFTF] Final set of changes to improve the MFTF tests #26570
• https://github.com/magento/partners-magento2b2b/pull/44
• https://github.com/magento/partners-magento2ee/pull/133
• [MFTF] Final set of changes to improve the MFTF tests magento2-page-builder#369
• https://github.com/magento/magento2-page-builder-ee/pull/104
• [MFTF] Final set of changes to improve the MFTF tests magento2-functional-testing-framework#563

Questions or comments

N/A

Contribution checklist (*)

• Pull request has a meaningful description of its purpose
• All commits are accompanied by meaningful commit messages
• All new or changed code is covered with unit/integration tests (if applicable)
• All automated tests passed successfully (all builds are green)

[m2-assistant]

Hi @lbajsarowicz. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

• @magento give me test instance - deploy test instance based on PR changes
• @magento give me 2.4-develop instance - deploy vanilla Magento instance

For more details, please, review the Magento Contributor Guide documentation.

[lbajsarowicz]

@magento run all tests

[lbajsarowicz]

@magento run all tests

[lbajsarowicz]

@magento run all tests

[lbajsarowicz]

@magento run all tests

[lbajsarowicz]

@magento run all tests

[lbajsarowicz]

@magento run all tests

I've merged 2.4-develop that introduced some new issues in the meantime 🤦‍♂️

[fooman]

Appreciate the effort you are putting into this but do want to highlight that renaming existing actionGroups will be hard on early adopters using MFTF to test extensions across multiple Magento versions.

For example we would have in our test

        <before>
            <actionGroup ref="logout" stepKey="logout"/>
            <actionGroup ref="LoginAsAdmin" stepKey="loginAsAdmin"/>
        </before>

After this PR gets merged we will need to change the above depending if we want to test against 2.3 or 2.4. Are there any strategies that would avoid us having to maintain separate branches (I am still hopeful that we don't need separate branches for the code itself).

[lbajsarowicz]

This will be covered for 2.3.* with Deprecated syntax from MFTF 2.6.0.
I mean - you'll be noticed that with Magento 2.4 the Action Group will be unavailable.

[fooman]

So you are essentially confirming that there is no way to have the same test run on 2.3 and 2.4 that references the existing ActionGroups?

The rename change itself is doable however just maintaining different branches (just for the tests) is something I would want to avoid (and probably Marketplace too if they ever are going to support MFTF on submitted extensions).

[lbajsarowicz]

That's actually something that bothers me too.
Until 2.4 is released, we have huge field to address the issue.

Let me talk with @okolesnyk to find the best solution for such case.

For me it's super-important to finally have consistent and clean naming pattern, that will be forced with XSD at some point of time.

[fooman]

okay great - will follow what you can think of.

What might be workable from my point of view would be a more generous deprecation period:

Keep the old actionGroup names available so something like logout extends adminLogoutAction Group or whatever the exact name is. Please note that this would only work if it was possible to extend an already extended actionGroup which is currently not supported. These intermediate group names would then be removed with Magento 2.5. That would allow someone to use the improved names for any new work. And the old ones can slowly be phased out as support for Magento 2.3 gets dropped over time.

[okolesnyk]

Hi @fooman
I completely understand why you worry about such changes.
Let me try to explain how we going to help everyone adopt their tests to work with Magento 2.3 and 2.4

We are not removing Action Groups in Magento 2.3, we mark them as deprecated and create new Action Group. For Example:

• logout - this Action Group will be deprecated but still exist in 2.3 and completely removed from 2.4
• AdminLogoutActionGroup - this AG will exist in latest 2.3 patch and 2.4.x releases

Also to notify Extension developers and everyone who's writing Magento customizations we introduced an attribute in MFTF entities called deprecated.
So once extension test will use something deprecated user will be notified in 3 different ways:
At Test runtime:
1. In Allure report (see screenshot)
2. In CLI Output (see screenshot)
At Test generation:
1. In dev/tests/acceptance/mftf.log file you will see all entities which are deprecated

So once something is marked as deprecated Extension devs should update their tests to use latest Action Group or other Test entity. And once new release of extension is cut to support Magento 2.4 it will mean it will work with both Magento 2.3 and 2.4 just because new Action Group introduced will exist in both Magento releases.

Let me know if it make sense and I explained it clear.

ALLURE NOTIFICATIONS:

CLI NOTIFICATIONS:

[lenaorobei]

@lbajsarowicz is this PR still actual? Or we will go module by module?

[lbajsarowicz]

We won't be able to do it in one PR.
Before things get merged, we have hundreds of conflicts :(

[m2-assistant]

Hi @lbajsarowicz, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.