From 44159b530c03cc022920269d87c6e2a08e167eb9 Mon Sep 17 00:00:00 2001 From: Donald Booth Date: Tue, 13 Aug 2019 16:19:06 -0500 Subject: [PATCH 01/27] Adding action group ref file. --- docs/actiongroup-list.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 docs/actiongroup-list.md diff --git a/docs/actiongroup-list.md b/docs/actiongroup-list.md new file mode 100644 index 000000000..61fbc5650 --- /dev/null +++ b/docs/actiongroup-list.md @@ -0,0 +1,30 @@ + + +# MFTF action group reference + +Action groups are important building blocks for quickly creating tests for the Magento Functional Testing Framework (MFTF). +This page lists all current action groups so that developers can see what is available to them. + +{% include mftf/actiongroup_data.md %} + +{% for item in actiongroups %} + +### {{ item.name }} +{% for file in item.items %} +#### [{{ file.filename }}]({{file.repo}}) + +{% for test in file.actiongroups %} +{{test.name}} + : {% if test.description == '' %}No description.{% else %}{{test.description}}{% endif %} +{: .mftf-dl} +{% endfor %} +{% endfor %} +{% endfor %} From 5895db48dac76880a0534e2fce27a5c84066935c Mon Sep 17 00:00:00 2001 From: Donald Booth Date: Thu, 15 Aug 2019 09:21:55 -0500 Subject: [PATCH 02/27] Small grammar change. --- docs/actiongroup-list.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/actiongroup-list.md b/docs/actiongroup-list.md index 61fbc5650..6bc2a61c3 100644 --- a/docs/actiongroup-list.md +++ b/docs/actiongroup-list.md @@ -10,8 +10,8 @@ dl dt{ # MFTF action group reference -Action groups are important building blocks for quickly creating tests for the Magento Functional Testing Framework (MFTF). -This page lists all current action groups so that developers can see what is available to them. +Action groups are important building blocks for quickly creating tests for the Magento Functional Testing Framework. +This page lists all current action groups so developers can see what is available to them. {% include mftf/actiongroup_data.md %} From 17fea8eaa9977b0806de0352f8708be70f327188 Mon Sep 17 00:00:00 2001 From: Donald Booth Date: Fri, 16 Aug 2019 08:11:35 -0500 Subject: [PATCH 03/27] moving template to devdocs. --- docs/actiongroup-list.md | 24 ------------------------ 1 file changed, 24 deletions(-) diff --git a/docs/actiongroup-list.md b/docs/actiongroup-list.md index 6bc2a61c3..7b469f4ed 100644 --- a/docs/actiongroup-list.md +++ b/docs/actiongroup-list.md @@ -1,30 +1,6 @@ - - # MFTF action group reference Action groups are important building blocks for quickly creating tests for the Magento Functional Testing Framework. This page lists all current action groups so developers can see what is available to them. {% include mftf/actiongroup_data.md %} - -{% for item in actiongroups %} - -### {{ item.name }} -{% for file in item.items %} -#### [{{ file.filename }}]({{file.repo}}) - -{% for test in file.actiongroups %} -{{test.name}} - : {% if test.description == '' %}No description.{% else %}{{test.description}}{% endif %} -{: .mftf-dl} -{% endfor %} -{% endfor %} -{% endfor %} From f5e07998adfc64698706d6293059bcb04bb684d4 Mon Sep 17 00:00:00 2001 From: Donald Booth Date: Fri, 16 Aug 2019 11:48:29 -0500 Subject: [PATCH 04/27] Putting selectors.md back --- docs/selectors.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 docs/selectors.md diff --git a/docs/selectors.md b/docs/selectors.md new file mode 100644 index 000000000..870072e15 --- /dev/null +++ b/docs/selectors.md @@ -0,0 +1,35 @@ +## Selectors + +These guidelines should help you to write high quality selectors. + +### Selectors SHOULD be written in CSS instead of XPath whenever possible + +CSS is generally easier to read than XPath. For example, `//*[@id="foo"]` in XPath can be expressed as simply as `#foo` in CSS. +See this [XPath Cheatsheet](https://devhints.io/xpath) for more examples. + +### XPath selectors SHOULD NOT use `@attribute="foo"`. + +This would fail if the attribute was `attribute="foo bar"`. +Instead you SHOULD use `contains(@attribute, "foo")` where `@attribute` is any valid attribute such as `@text` or `@class`. + +### CSS and XPath selectors SHOULD be implemented in their most simple form + +* GOOD: `#foo` +* BAD: `button[contains(@id, "foo")]` + +### CSS and XPath selectors SHOULD avoid making use of hardcoded indices + +Instead you SHOULD parameterize the selector. + +* GOOD: `.foo:nth-of-type({{index}})` +* BAD: `.foo:nth-of-type(1)` + +* GOOD: `button[contains(@id, "foo")][{{index}}]` +* BAD: `button[contains(@id, "foo")][1]` + +* GOOD: `#actions__{{index}}__aggregator` +* BAD: `#actions__1__aggregator` + +### CSS and XPath selectors MUST NOT reference the `@data-bind` attribute + +The `@data-bind` attribute is used by KnockoutJS, a framework Magento uses to create dynamic Javascript pages. Since this `@data-bind` attribute is tied to a specific framework, it should not be used for selectors. If Magento decides to use a different framework then these `@data-bind` selectors would break. From 3bf88ed63b3803f5c273f10004e97ec670cfb9e3 Mon Sep 17 00:00:00 2001 From: Kevin Kozan Date: Fri, 30 Aug 2019 10:30:30 -0500 Subject: [PATCH 05/27] MQE-1715: .credentials file must exist but it should not have to - Added try/catch logic to problem area. --- .../DataGenerator/Handlers/PersistedObjectHandler.php | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/Magento/FunctionalTestingFramework/DataGenerator/Handlers/PersistedObjectHandler.php b/src/Magento/FunctionalTestingFramework/DataGenerator/Handlers/PersistedObjectHandler.php index cc35bbc52..ab8844089 100644 --- a/src/Magento/FunctionalTestingFramework/DataGenerator/Handlers/PersistedObjectHandler.php +++ b/src/Magento/FunctionalTestingFramework/DataGenerator/Handlers/PersistedObjectHandler.php @@ -8,6 +8,7 @@ use Magento\FunctionalTestingFramework\DataGenerator\Persist\DataPersistenceHandler; use Magento\FunctionalTestingFramework\Exceptions\TestReferenceException; +use Magento\FunctionalTestingFramework\Exceptions\TestFrameworkException; class PersistedObjectHandler { @@ -86,7 +87,12 @@ public function createEntity( } foreach ($overrideFields as $index => $field) { - $overrideFields[$index] = CredentialStore::getInstance()->decryptAllSecretsInString($field); + try { + $overrideFields[$index] = CredentialStore::getInstance()->decryptAllSecretsInString($field); + } catch (TestFrameworkException $e) { + //do not rethrow if Credentials are not defined + $overrideFields[$index] = $field; + } } $retrievedEntity = DataObjectHandler::getInstance()->getObject($entity); From 9015ee9373c8193ff28ddb714b7abceb6960f719 Mon Sep 17 00:00:00 2001 From: Kevin Kozan Date: Fri, 30 Aug 2019 10:45:40 -0500 Subject: [PATCH 06/27] MQE-1715: .credentials file must exist but it should not have to - composer and lock file update - changelog update --- CHANGELOG.md | 5 +++++ bin/mftf | 2 +- composer.json | 2 +- composer.lock | 2 +- 4 files changed, 8 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d3f82a2f9..928b5a670 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,11 @@ Magento Functional Testing Framework Changelog ================================================ +2.4.5 +----- +### Fixes +* Fixed an issue where `.credentials` was required when using `` actions with field overrides. + 2.4.4 ----- ### Fixes diff --git a/bin/mftf b/bin/mftf index 0f2bf274d..4bca4f620 100755 --- a/bin/mftf +++ b/bin/mftf @@ -29,7 +29,7 @@ try { try { $application = new Symfony\Component\Console\Application(); $application->setName('Magento Functional Testing Framework CLI'); - $application->setVersion('2.4.4'); + $application->setVersion('2.4.5'); /** @var \Magento\FunctionalTestingFramework\Console\CommandListInterface $commandList */ $commandList = new \Magento\FunctionalTestingFramework\Console\CommandList; foreach ($commandList->getCommands() as $command) { diff --git a/composer.json b/composer.json index bc36fb60e..c4bde5cd3 100755 --- a/composer.json +++ b/composer.json @@ -2,7 +2,7 @@ "name": "magento/magento2-functional-testing-framework", "description": "Magento2 Functional Testing Framework", "type": "library", - "version": "2.4.4", + "version": "2.4.5", "license": "AGPL-3.0", "keywords": ["magento", "automation", "functional", "testing"], "config": { diff --git a/composer.lock b/composer.lock index d415acc0a..1eeed1efc 100644 --- a/composer.lock +++ b/composer.lock @@ -4,7 +4,7 @@ "Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies", "This file is @generated automatically" ], - "content-hash": "b1795ca4e2c9a15582db44baf54962ad", + "content-hash": "e10b0bbd1baa544f5efae43a6cfbed08", "packages": [ { "name": "allure-framework/allure-codeception", From 72222595abbecb06b6aa0c0e2cabe22b19f7493f Mon Sep 17 00:00:00 2001 From: Kevin Kozan Date: Tue, 10 Sep 2019 15:11:42 -0500 Subject: [PATCH 07/27] MQE-1510 (#446) --- etc/config/command.php | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/etc/config/command.php b/etc/config/command.php index f018e3014..047af324a 100644 --- a/etc/config/command.php +++ b/etc/config/command.php @@ -12,8 +12,8 @@ $tokenModel = $magentoObjectManager->get(\Magento\Integration\Model\Oauth\Token::class); $tokenPassedIn = urldecode($_POST['token']); - $command = str_replace([';', '&', '|'], '', urldecode($_POST['command'])); - $arguments = str_replace([';', '&', '|'], '', urldecode($_POST['arguments'])); + $command = urldecode($_POST['command']); + $arguments = urldecode($_POST['arguments']); // Token returned will be null if the token we passed in is invalid $tokenFromMagento = $tokenModel->loadByToken($tokenPassedIn)->getToken(); From fa579d1fa07806259d995310439e0e5b4a8215d5 Mon Sep 17 00:00:00 2001 From: Soumya Unnikrishnan Date: Tue, 10 Sep 2019 16:32:33 -0500 Subject: [PATCH 08/27] MQE-1755: mftf run:test Test1 Test2 does not run before/after hooks correctly --- .../Console/RunTestCommand.php | 104 +++++++++++------- 1 file changed, 64 insertions(+), 40 deletions(-) diff --git a/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php b/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php index e1e30589d..10f3defc6 100644 --- a/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php +++ b/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php @@ -19,6 +19,13 @@ class RunTestCommand extends BaseGenerateCommand { + /** + * The return code. Determined by all tests that run. + * + * @var integer + */ + private $returnCode = 0; + /** * Configures the current command. * @@ -76,60 +83,77 @@ protected function execute(InputInterface $input, OutputInterface $output): int ]; $command->run(new ArrayInput($args), $output); } - // tests with resolved suite references - $resolvedTests = $this->resolveSuiteReferences($testConfiguration); + $testConfigArray = json_decode($testConfiguration, true); + + // run tests not referenced in suites + $this->runTests($testConfigArray['tests'], $output); + + // run tests in suites + $this->runTestsInSuite($testConfigArray['suites'], $output); + + return $this->returnCode; + + } + + /** + * Run tests not referenced in suites + * @param array $testsConfig + * @param OutputInterface $output + * @throws TestFrameworkException + */ + private function runTests($testsConfig, OutputInterface $output) { + + + $tests = $testsConfig ?? []; $codeceptionCommand = realpath(PROJECT_ROOT . '/vendor/bin/codecept') . ' run functional '; - $testsDirectory = TESTS_MODULE_PATH . DIRECTORY_SEPARATOR . TestGenerator::GENERATED_DIR . DIRECTORY_SEPARATOR; - $returnCode = 0; - //execute only tests specified as arguments in run command - foreach ($resolvedTests as $test) { - //set directory as suite name for tests in suite, if not set to "default" - if (strpos($test, ':')) { - list($testGroup, $testName) = explode(":", $test); - } else { - list($testGroup, $testName) = [TestGenerator::DEFAULT_DIR, $test]; - } - $testGroup = $testGroup . DIRECTORY_SEPARATOR; - $testName = $testName . 'Cest.php'; - if (!realpath($testsDirectory . $testGroup . $testName)) { + $testsDirectory = TESTS_MODULE_PATH . + DIRECTORY_SEPARATOR . + TestGenerator::GENERATED_DIR . + DIRECTORY_SEPARATOR . + TestGenerator::DEFAULT_DIR . + DIRECTORY_SEPARATOR ; + + foreach ($tests as $test) { + $testName = $test . 'Cest.php'; + if (!realpath($testsDirectory . $testName)) { throw new TestFrameworkException( - $testName . " is not available under " . $testsDirectory . $testGroup + $testName . " is not available under " . $testsDirectory ); } - $fullCommand = $codeceptionCommand . $testsDirectory . $testGroup . $testName . ' --verbose --steps'; + $fullCommand = $codeceptionCommand . $testsDirectory . $testName . ' --verbose --steps'; $process = new Process($fullCommand); $process->setWorkingDirectory(TESTS_BP); $process->setIdleTimeout(600); $process->setTimeout(0); - - $returnCode = max($returnCode, $process->run( - function ($type, $buffer) use ($output) { - $output->write($buffer); - } - )); + $subReturnCode = $process->run(function ($type, $buffer) use ($output) { + $output->write($buffer); + }); + $this->returnCode = max($this->returnCode, $subReturnCode); } - return $returnCode; } /** - * Get an array of tests with resolved suite references from $testConfiguration - * eg: if test is referenced in a suite, it'll be stored in format suite:test - * @param string $testConfigurationJson - * @return array + * Run tests referenced in suites within suites' context. + * @param array $suitesConfig + * @param OutputInterface $output */ - private function resolveSuiteReferences($testConfigurationJson) - { - $testConfiguration = json_decode($testConfigurationJson, true); - $testsArray = $testConfiguration['tests'] ?? []; - $suitesArray = $testConfiguration['suites'] ?? []; - $testArrayBuilder = []; - - foreach ($suitesArray as $suite => $tests) { - foreach ($tests as $test) { - $testArrayBuilder[] = "$suite:$test"; - } + private function runTestsInSuite($suitesConfig, OutputInterface $output) { + + $suites = $suitesConfig ?? []; + $codeceptionCommand = realpath(PROJECT_ROOT . '/vendor/bin/codecept') . ' run functional --verbose --steps '; + $testGroups = array_keys($suites); + //for tests in suites, run them as a group to run before and after block + foreach ($testGroups as $testGroup) { + $fullCommand = $codeceptionCommand . " -g {$testGroup}"; + $process = new Process($fullCommand); + $process->setWorkingDirectory(TESTS_BP); + $process->setIdleTimeout(600); + $process->setTimeout(0); + $subReturnCode = $process->run(function ($type, $buffer) use ($output) { + $output->write($buffer); + }); + $this->returnCode = max($this->returnCode, $subReturnCode); } - return array_merge($testArrayBuilder, $testsArray); } } From 3b84f12a963a789bded006226d0dfd46463de279 Mon Sep 17 00:00:00 2001 From: Soumya Unnikrishnan Date: Tue, 10 Sep 2019 19:08:02 -0500 Subject: [PATCH 09/27] MQE-1755: mftf run:test Test1 Test2 does not run before/after hooks correctly fixed unit tests --- .../Console/RunTestCommand.php | 39 ++++++++++--------- 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php b/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php index 10f3defc6..668d5152c 100644 --- a/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php +++ b/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php @@ -51,8 +51,6 @@ protected function configure() * @param OutputInterface $output * @return integer * @throws \Exception - * - * @SuppressWarnings(PHPMD.UnusedLocalVariable) */ protected function execute(InputInterface $input, OutputInterface $output): int { @@ -86,26 +84,29 @@ protected function execute(InputInterface $input, OutputInterface $output): int $testConfigArray = json_decode($testConfiguration, true); - // run tests not referenced in suites - $this->runTests($testConfigArray['tests'], $output); + if (isset($testConfigArray['tests'])) { + $this->runTests($testConfigArray['tests'], $output); + } - // run tests in suites - $this->runTestsInSuite($testConfigArray['suites'], $output); + if (isset($testConfigArray['suites'])) { + $this->runTestsInSuite($testConfigArray['suites'], $output); + } return $this->returnCode; - } /** * Run tests not referenced in suites - * @param array $testsConfig + * + * @param array $tests * @param OutputInterface $output + * @return void * @throws TestFrameworkException + * + * @SuppressWarnings(PHPMD.UnusedLocalVariable) */ - private function runTests($testsConfig, OutputInterface $output) { - - - $tests = $testsConfig ?? []; + private function runTests(array $tests, OutputInterface $output) + { $codeceptionCommand = realpath(PROJECT_ROOT . '/vendor/bin/codecept') . ' run functional '; $testsDirectory = TESTS_MODULE_PATH . DIRECTORY_SEPARATOR . @@ -135,17 +136,19 @@ private function runTests($testsConfig, OutputInterface $output) { /** * Run tests referenced in suites within suites' context. + * * @param array $suitesConfig * @param OutputInterface $output + * @return void + * + * @SuppressWarnings(PHPMD.UnusedLocalVariable) */ - private function runTestsInSuite($suitesConfig, OutputInterface $output) { - - $suites = $suitesConfig ?? []; + private function runTestsInSuite(array $suitesConfig, OutputInterface $output) + { $codeceptionCommand = realpath(PROJECT_ROOT . '/vendor/bin/codecept') . ' run functional --verbose --steps '; - $testGroups = array_keys($suites); //for tests in suites, run them as a group to run before and after block - foreach ($testGroups as $testGroup) { - $fullCommand = $codeceptionCommand . " -g {$testGroup}"; + foreach (array_keys($suitesConfig) as $suite) { + $fullCommand = $codeceptionCommand . " -g {$suite}"; $process = new Process($fullCommand); $process->setWorkingDirectory(TESTS_BP); $process->setIdleTimeout(600); From 8597fcb8313a022f83c9ff47a20dd9c421fbdb4c Mon Sep 17 00:00:00 2001 From: Soumya Unnikrishnan Date: Tue, 10 Sep 2019 19:33:09 -0500 Subject: [PATCH 10/27] MQE-1755: mftf run:test Test1 Test2 does not run before/after hooks correctly fix unit tests --- .../FunctionalTestingFramework/Console/RunTestCommand.php | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php b/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php index 668d5152c..dc763b9b0 100644 --- a/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php +++ b/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php @@ -98,7 +98,7 @@ protected function execute(InputInterface $input, OutputInterface $output): int /** * Run tests not referenced in suites * - * @param array $tests + * @param array $tests * @param OutputInterface $output * @return void * @throws TestFrameworkException @@ -137,7 +137,7 @@ private function runTests(array $tests, OutputInterface $output) /** * Run tests referenced in suites within suites' context. * - * @param array $suitesConfig + * @param array $suitesConfig * @param OutputInterface $output * @return void * From 9cfeb9c71b573482f18ce5f51ebe342bdd861e90 Mon Sep 17 00:00:00 2001 From: Soumya Unnikrishnan Date: Wed, 11 Sep 2019 13:42:23 -0500 Subject: [PATCH 11/27] MQE-1755: mftf run:test Test1 Test2 does not run before/after hooks correctly --- .../Console/RunTestCommand.php | 42 ++++++++++--------- 1 file changed, 22 insertions(+), 20 deletions(-) diff --git a/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php b/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php index dc763b9b0..b65409b70 100644 --- a/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php +++ b/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php @@ -102,8 +102,6 @@ protected function execute(InputInterface $input, OutputInterface $output): int * @param OutputInterface $output * @return void * @throws TestFrameworkException - * - * @SuppressWarnings(PHPMD.UnusedLocalVariable) */ private function runTests(array $tests, OutputInterface $output) { @@ -123,14 +121,7 @@ private function runTests(array $tests, OutputInterface $output) ); } $fullCommand = $codeceptionCommand . $testsDirectory . $testName . ' --verbose --steps'; - $process = new Process($fullCommand); - $process->setWorkingDirectory(TESTS_BP); - $process->setIdleTimeout(600); - $process->setTimeout(0); - $subReturnCode = $process->run(function ($type, $buffer) use ($output) { - $output->write($buffer); - }); - $this->returnCode = max($this->returnCode, $subReturnCode); + $this->returnCode = max($this->returnCode, $this->executeTestCommand($fullCommand, $output)); } } @@ -140,8 +131,6 @@ private function runTests(array $tests, OutputInterface $output) * @param array $suitesConfig * @param OutputInterface $output * @return void - * - * @SuppressWarnings(PHPMD.UnusedLocalVariable) */ private function runTestsInSuite(array $suitesConfig, OutputInterface $output) { @@ -149,14 +138,27 @@ private function runTestsInSuite(array $suitesConfig, OutputInterface $output) //for tests in suites, run them as a group to run before and after block foreach (array_keys($suitesConfig) as $suite) { $fullCommand = $codeceptionCommand . " -g {$suite}"; - $process = new Process($fullCommand); - $process->setWorkingDirectory(TESTS_BP); - $process->setIdleTimeout(600); - $process->setTimeout(0); - $subReturnCode = $process->run(function ($type, $buffer) use ($output) { - $output->write($buffer); - }); - $this->returnCode = max($this->returnCode, $subReturnCode); + $this->returnCode = max($this->returnCode, $this->executeTestCommand($fullCommand, $output)); } } + + /** + * Runs the codeception test command and returns exit code + * + * @param String $command + * @param OutputInterface $output + * @return int + * + * @SuppressWarnings(PHPMD.UnusedLocalVariable) + */ + private function executeTestCommand(String $command, OutputInterface $output) + { + $process = new Process($command); + $process->setWorkingDirectory(TESTS_BP); + $process->setIdleTimeout(600); + $process->setTimeout(0); + return $process->run(function ($type, $buffer) use ($output) { + $output->write($buffer); + }); + } } From 6a5798ed082a73372cf06b37f3d2168eb4835333 Mon Sep 17 00:00:00 2001 From: Soumya Unnikrishnan Date: Wed, 11 Sep 2019 13:57:27 -0500 Subject: [PATCH 12/27] MQE-1755: mftf run:test Test1 Test2 does not run before/after hooks correctly fixed unit test failures --- .../FunctionalTestingFramework/Console/RunTestCommand.php | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php b/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php index b65409b70..c848a3010 100644 --- a/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php +++ b/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php @@ -145,13 +145,13 @@ private function runTestsInSuite(array $suitesConfig, OutputInterface $output) /** * Runs the codeception test command and returns exit code * - * @param String $command + * @param string $command * @param OutputInterface $output - * @return int + * @return integer * * @SuppressWarnings(PHPMD.UnusedLocalVariable) */ - private function executeTestCommand(String $command, OutputInterface $output) + private function executeTestCommand(string $command, OutputInterface $output) { $process = new Process($command); $process->setWorkingDirectory(TESTS_BP); From c113b9f70ba78756b84df6b787b9ba3d67b886f3 Mon Sep 17 00:00:00 2001 From: Kevin Kozan Date: Fri, 13 Sep 2019 11:21:49 -0500 Subject: [PATCH 13/27] MQE-1743: Changelog and Composer Bump (#449) - CHANGELOG update - Version bump --- CHANGELOG.md | 34 ++++++++++++++++++++++++++++++++++ bin/mftf | 2 +- composer.json | 2 +- composer.lock | 4 ++-- docs/data.md | 10 ++++++++++ 5 files changed, 48 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d3f82a2f9..5750f0ab2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,40 @@ Magento Functional Testing Framework Changelog ================================================ +2.5.0 +----- +* Traceability + * Allure output has been enhanced to contain new test artifacts created and used per MFTF step: + * `makeScreenshot` will contain an attachment under its Allure step. + * `seeInCurrentUrl` and all other `Url` asserts now contain an attachment with the expects vs actual comparison. + * `createData` and all other `Data` actions now contain attachments with `ApiUrl`, `Headers`, `Request Body`, and `Response Body`. +* Modularity + * Added a new `mftf run:manifest` command to run testManifest files generated by `generate:tests`. + * See DevDocs for details + * `mftf generate/run:test` commands now implicitly generates the `suite` the test exists in. + * If a test exists in multiple suites, it will generate it in all suite contexts. + * `mftf run:test ` will now only run the exact test provided, regardless of what is generated. +* Maintainability + * Added an `--allow-skipped` flag that allows MFTF to ignore the `` annotation. This was added to the following commands: + * `generate:test` + * `run:test` + * `run:group` + * `run:failed` +* Customizability + * `` defined in data.xml can now reference other `` directly + * See DevDocs for details + +### Fixes +* Fixed an issue where `grab` action variables were not substituting correctly when used as an element parameter. +* Framework will not throw a descriptive error when referencing a `$persisted.field$` that does not exist. +* MFTF test materials that `extends=""` itself will no longer cause infinite recursion. +* Fixed an issue where a test could not reference a `$data.field$` whose casing was modified by the API that it used. +* Fixed an issue with the default `functional.suite.yml` where it was incompatible with `symfony/yaml 4.0.0`. +* Improved test generation performance via class refactors (`~10%` faster). + +### GitHub Issues/Pull requests: +* [#377](https://github.com/magento/magento2-functional-testing-framework/pull/377) -- Non-API operations fixes + 2.4.4 ----- ### Fixes diff --git a/bin/mftf b/bin/mftf index 0f2bf274d..7f9db3524 100755 --- a/bin/mftf +++ b/bin/mftf @@ -29,7 +29,7 @@ try { try { $application = new Symfony\Component\Console\Application(); $application->setName('Magento Functional Testing Framework CLI'); - $application->setVersion('2.4.4'); + $application->setVersion('2.5.0'); /** @var \Magento\FunctionalTestingFramework\Console\CommandListInterface $commandList */ $commandList = new \Magento\FunctionalTestingFramework\Console\CommandList; foreach ($commandList->getCommands() as $command) { diff --git a/composer.json b/composer.json index 66936938c..b33567303 100755 --- a/composer.json +++ b/composer.json @@ -2,7 +2,7 @@ "name": "magento/magento2-functional-testing-framework", "description": "Magento2 Functional Testing Framework", "type": "library", - "version": "2.4.4", + "version": "2.5.0", "license": "AGPL-3.0", "keywords": ["magento", "automation", "functional", "testing"], "config": { diff --git a/composer.lock b/composer.lock index 10169fb08..dd7d6ad3f 100644 --- a/composer.lock +++ b/composer.lock @@ -4,7 +4,7 @@ "Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies", "This file is @generated automatically" ], - "content-hash": "1b7f01a11ff57f3b64f08352c33a506f", + "content-hash": "01cbd9e237e76de7070a3c0de4ee8b9f", "packages": [ { "name": "allure-framework/allure-codeception", @@ -250,7 +250,7 @@ { "name": "Tobias Nyholm", "email": "tobias.nyholm@gmail.com", - "homepage": "https://github.com/nyholm" + "homepage": "https://github.com/Nyholm" } ], "description": "Library of all the php-cache adapters", diff --git a/docs/data.md b/docs/data.md index 875df0198..4c618ae45 100644 --- a/docs/data.md +++ b/docs/data.md @@ -173,6 +173,16 @@ The following is an example of a call in test: This action inputs data from the `name` of the `_defaultCategory` entity (for example, `simpleCategory598742365`) into the field with the locator defined in the selector of the `categoryNameInput` element of the `AdminCategoryBasicFieldSection`. +You can also call data from the xml definition of a `data` tag directly: + +```xml + + admin + {{AnotherUser.current_password}} + {{_ENV.MAGENTO_ADMIN_PASSWORD}} + +``` + ## Reference ### entities {#entities-tag} From d97b01352a1fda16f59f1a676fa815217ae3f1b3 Mon Sep 17 00:00:00 2001 From: Kevin Kozan Date: Fri, 13 Sep 2019 14:39:46 -0500 Subject: [PATCH 14/27] MQE-1582: Enable Testers To Run Skipped Tests - Fix tag name to allow-skipped as per AC --- .../Console/BaseGenerateCommand.php | 2 +- .../Console/GenerateSuiteCommand.php | 2 +- .../Console/GenerateTestsCommand.php | 2 +- .../FunctionalTestingFramework/Console/RunTestCommand.php | 4 ++-- .../Console/RunTestFailedCommand.php | 4 ++-- .../Console/RunTestGroupCommand.php | 4 ++-- 6 files changed, 9 insertions(+), 9 deletions(-) diff --git a/src/Magento/FunctionalTestingFramework/Console/BaseGenerateCommand.php b/src/Magento/FunctionalTestingFramework/Console/BaseGenerateCommand.php index 40582db86..87b203d66 100644 --- a/src/Magento/FunctionalTestingFramework/Console/BaseGenerateCommand.php +++ b/src/Magento/FunctionalTestingFramework/Console/BaseGenerateCommand.php @@ -36,7 +36,7 @@ protected function configure() InputOption::VALUE_NONE, 'force generation and running of tests regardless of Magento Instance Configuration' )->addOption( - "allowSkipped", + "allow-skipped", 'a', InputOption::VALUE_NONE, 'Allows MFTF to generate and run skipped tests.' diff --git a/src/Magento/FunctionalTestingFramework/Console/GenerateSuiteCommand.php b/src/Magento/FunctionalTestingFramework/Console/GenerateSuiteCommand.php index 7dd3b8cb4..cd798f420 100644 --- a/src/Magento/FunctionalTestingFramework/Console/GenerateSuiteCommand.php +++ b/src/Magento/FunctionalTestingFramework/Console/GenerateSuiteCommand.php @@ -46,7 +46,7 @@ protected function execute(InputInterface $input, OutputInterface $output) $debug = $input->getOption('debug') ?? MftfApplicationConfig::LEVEL_DEVELOPER; // for backward compatibility $remove = $input->getOption('remove'); $verbose = $output->isVerbose(); - $allowSkipped = $input->getOption('allowSkipped'); + $allowSkipped = $input->getOption('allow-skipped'); // Set application configuration so we can references the user options in our framework MftfApplicationConfig::create( diff --git a/src/Magento/FunctionalTestingFramework/Console/GenerateTestsCommand.php b/src/Magento/FunctionalTestingFramework/Console/GenerateTestsCommand.php index 64c7c01d5..621f29d03 100644 --- a/src/Magento/FunctionalTestingFramework/Console/GenerateTestsCommand.php +++ b/src/Magento/FunctionalTestingFramework/Console/GenerateTestsCommand.php @@ -71,7 +71,7 @@ protected function execute(InputInterface $input, OutputInterface $output) $debug = $input->getOption('debug') ?? MftfApplicationConfig::LEVEL_DEVELOPER; // for backward compatibility $remove = $input->getOption('remove'); $verbose = $output->isVerbose(); - $allowSkipped = $input->getOption('allowSkipped'); + $allowSkipped = $input->getOption('allow-skipped'); // Set application configuration so we can references the user options in our framework MftfApplicationConfig::create( diff --git a/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php b/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php index ad9f5795b..77b8be513 100644 --- a/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php +++ b/src/Magento/FunctionalTestingFramework/Console/RunTestCommand.php @@ -64,7 +64,7 @@ protected function execute(InputInterface $input, OutputInterface $output): int $force = $input->getOption('force'); $remove = $input->getOption('remove'); $debug = $input->getOption('debug') ?? MftfApplicationConfig::LEVEL_DEVELOPER; // for backward compatibility - $allowSkipped = $input->getOption('allowSkipped'); + $allowSkipped = $input->getOption('allow-skipped'); $verbose = $output->isVerbose(); if ($skipGeneration and $remove) { @@ -92,7 +92,7 @@ protected function execute(InputInterface $input, OutputInterface $output): int '--force' => $force, '--remove' => $remove, '--debug' => $debug, - '--allowSkipped' => $allowSkipped, + '--allow-skipped' => $allowSkipped, '-v' => $verbose ]; $command->run(new ArrayInput($args), $output); diff --git a/src/Magento/FunctionalTestingFramework/Console/RunTestFailedCommand.php b/src/Magento/FunctionalTestingFramework/Console/RunTestFailedCommand.php index 15ba723e6..336fd5d87 100644 --- a/src/Magento/FunctionalTestingFramework/Console/RunTestFailedCommand.php +++ b/src/Magento/FunctionalTestingFramework/Console/RunTestFailedCommand.php @@ -70,7 +70,7 @@ protected function execute(InputInterface $input, OutputInterface $output): int { $force = $input->getOption('force'); $debug = $input->getOption('debug') ?? MftfApplicationConfig::LEVEL_DEVELOPER; // for backward compatibility - $allowSkipped = $input->getOption('allowSkipped'); + $allowSkipped = $input->getOption('allow-skipped'); $verbose = $output->isVerbose(); // Create Mftf Configuration @@ -95,7 +95,7 @@ protected function execute(InputInterface $input, OutputInterface $output): int '--force' => $force, '--remove' => true, '--debug' => $debug, - '--allowSkipped' => $allowSkipped, + '--allow-skipped' => $allowSkipped, '-v' => $verbose ]; $command->run(new ArrayInput($args), $output); diff --git a/src/Magento/FunctionalTestingFramework/Console/RunTestGroupCommand.php b/src/Magento/FunctionalTestingFramework/Console/RunTestGroupCommand.php index 20d290af9..7f954c8fe 100644 --- a/src/Magento/FunctionalTestingFramework/Console/RunTestGroupCommand.php +++ b/src/Magento/FunctionalTestingFramework/Console/RunTestGroupCommand.php @@ -60,7 +60,7 @@ protected function execute(InputInterface $input, OutputInterface $output): int $groups = $input->getArgument('groups'); $remove = $input->getOption('remove'); $debug = $input->getOption('debug') ?? MftfApplicationConfig::LEVEL_DEVELOPER; // for backward compatibility - $allowSkipped = $input->getOption('allowSkipped'); + $allowSkipped = $input->getOption('allow-skipped'); $verbose = $output->isVerbose(); if ($skipGeneration and $remove) { @@ -87,7 +87,7 @@ protected function execute(InputInterface $input, OutputInterface $output): int '--force' => $force, '--remove' => $remove, '--debug' => $debug, - '--allowSkipped' => $allowSkipped, + '--allow-skipped' => $allowSkipped, '-v' => $verbose ]; From f63fac83e89bec461ef85a43cb120e48e3cad5fd Mon Sep 17 00:00:00 2001 From: Kevin Kozan Date: Mon, 16 Sep 2019 15:39:30 -0500 Subject: [PATCH 15/27] MQE-1765: Introduce API Endpoint and Request Headers to Allure artifacts - Removed both artifacts to be reintroduced later. --- .../DataGenerator/Persist/CurlHandler.php | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/Magento/FunctionalTestingFramework/DataGenerator/Persist/CurlHandler.php b/src/Magento/FunctionalTestingFramework/DataGenerator/Persist/CurlHandler.php index 4b184c206..691ce3606 100644 --- a/src/Magento/FunctionalTestingFramework/DataGenerator/Persist/CurlHandler.php +++ b/src/Magento/FunctionalTestingFramework/DataGenerator/Persist/CurlHandler.php @@ -167,8 +167,6 @@ public function executeRequest($dependentEntities) $response = $executor->read($successRegex, $returnRegex, $returnIndex); $executor->close(); - AllureHelper::addAttachmentToLastStep($apiUrl, 'API Endpoint'); - AllureHelper::addAttachmentToLastStep(json_encode($headers, JSON_PRETTY_PRINT), 'Request Headers'); AllureHelper::addAttachmentToLastStep(json_encode($this->requestData, JSON_PRETTY_PRINT), 'Request Body'); AllureHelper::addAttachmentToLastStep( json_encode(json_decode($response, true), JSON_PRETTY_PRINT+JSON_UNESCAPED_UNICODE+JSON_UNESCAPED_SLASHES), From 93fd35d2f5f8be9595d2b0f0fe9ea90f14b1ecb0 Mon Sep 17 00:00:00 2001 From: Kevin Kozan Date: Mon, 16 Sep 2019 15:49:11 -0500 Subject: [PATCH 16/27] MQE-1765: Introduce API Endpoint and Request Headers to Allure artifacts - Changelog redaction --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5750f0ab2..6e297f189 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ Magento Functional Testing Framework Changelog * Allure output has been enhanced to contain new test artifacts created and used per MFTF step: * `makeScreenshot` will contain an attachment under its Allure step. * `seeInCurrentUrl` and all other `Url` asserts now contain an attachment with the expects vs actual comparison. - * `createData` and all other `Data` actions now contain attachments with `ApiUrl`, `Headers`, `Request Body`, and `Response Body`. + * `createData` and all other `Data` actions now contain attachments with `Request Body` and `Response Body`. * Modularity * Added a new `mftf run:manifest` command to run testManifest files generated by `generate:tests`. * See DevDocs for details From 7accc4cce53ef0cf30a844b49c7d55387643bb3a Mon Sep 17 00:00:00 2001 From: Ji Lu Date: Tue, 17 Sep 2019 10:15:43 -0500 Subject: [PATCH 17/27] MQE-1671: updated CHANGELOG.md to include vault integration --- CHANGELOG.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6e297f189..6471920c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,7 +21,9 @@ Magento Functional Testing Framework Changelog * `run:group` * `run:failed` * Customizability - * `` defined in data.xml can now reference other `` directly + * `` defined in data.xml can now reference other `` directly. + * See DevDocs for details + * Added vault as an alternative credential storage. * See DevDocs for details ### Fixes From d58eb6bc9b5a6f52b55d58cdafbc5b07aa451b26 Mon Sep 17 00:00:00 2001 From: Ji Lu Date: Tue, 17 Sep 2019 12:12:30 -0500 Subject: [PATCH 18/27] MQE-1768: incorrect credential key for carriers_fedex_account in .credentials.example --- etc/config/.credentials.example | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/etc/config/.credentials.example b/etc/config/.credentials.example index d9c73ac66..429e9d19f 100644 --- a/etc/config/.credentials.example +++ b/etc/config/.credentials.example @@ -1,4 +1,4 @@ -#magento/magento/carriers_fedex_account= +#magento/carriers_fedex_account= #magento/carriers_fedex_meter_number= #magento/carriers_fedex_key= #magento/carriers_fedex_password= From 5fb856988fea4466f2eb6ad43eced969d4a0399e Mon Sep 17 00:00:00 2001 From: Ji Lu Date: Wed, 18 Sep 2019 11:44:53 -0500 Subject: [PATCH 19/27] MQE-1671: document using credentials with vault in MFTF tests --- docs/configuration.md | 26 +++++++++ docs/credentials.md | 124 ++++++++++++++++++++++++++++++++++-------- 2 files changed, 128 insertions(+), 22 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 9d87e0735..a768c76dd 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -251,6 +251,32 @@ Example: BROWSER=firefox ``` +###CREDENTIAL_VAULT_ADDRESS + +Api address for vault server. + +Default: http://127.0.0.1:8200 + +Example: + +```conf +# Default api address for local vault dev server +CREDENTIAL_VAULT_ADDRESS=http://127.0.0.1:8200 +``` + +###CREDENTIAL_VAULT_SECRET_BASE_PATH + +Vault secret engine base path. + +Default: secret + +Example: + +```conf +# Default base path for kv secret engine in local vault dev server +CREDENTIAL_VAULT_SECRET_BASE_PATH=secret +``` + [`MAGENTO_CLI_COMMAND_PATH`]: #magento_cli_command_path diff --git a/docs/credentials.md b/docs/credentials.md index 7afa9b805..a79f64aec 100644 --- a/docs/credentials.md +++ b/docs/credentials.md @@ -1,8 +1,11 @@ # Credentials -When you test functionality that involves external services such as UPS, FedEx, PayPal, or SignifyD, use the MFTF credentials feature to hide sensitive [data][] like integration tokens and API keys. +When you test functionality that involves external services such as UPS, FedEx, PayPal, or SignifyD, +use the MFTF credentials feature to hide sensitive [data][] like integration tokens and API keys. -## Define sensitive data in `.credentials` +Currently MFTF supports two types of credential storage: **.credentials file** and **HashiCorp vault**. + +# Configure File Storage The MFTF creates a sample file for credentials during [initial setup][]: `magento2/dev/tests/acceptance/.credentials.example`. The file contains an example list of keys for fields that can require credentials. @@ -33,49 +36,119 @@ The command outputs the path if the file is excluded: .credentials ``` -### Define sensitive data +### Define sensitive data in `.credentials` file -Open the `.credentials` file, uncomment the fields you want to use, and add your values: +Open the `.credentials` file, for Magento core credentials, uncomment the fields you want to use, and add your values: -```config +```conf ... # Credentials for the USPS service -carriers_usps_userid=test_user -carriers_usps_password=Lmgxvrq89uPwECeV +magento/carriers_usps_userid=usps_test_user +magento/carriers_usps_password=Lmgxvrq89uPwECeV # Credentials for the DHL service -#carriers/dhl/id_us= -#carriers/dhl/password_us= +#magento/carriers_dhl_id_us=dhl_test_user +#magento/carriers_dhl_password_us=Mlgxv3dsagVeG .... +``` + +Or add new key & value pairs for your own credentials. The keys use the following format: + +```conf +/= ```
-The `/` symbol is not supported in a key name. +The `/` symbol is not supported in a `key_name` other than the one after your vendor or extension name.
- -You are free to use any other keys you like, as they are merely the keys to reference from your tests. + +Otherwise you are free to use any other `key_name` you like, as they are merely the keys to reference from your tests. ```conf # Credentials for the MyAwesome service -my_awesome_service_token=rRVSVnh3cbDsVG39oTMz4A +vendor/my_awesome_service_token=rRVSVnh3cbDsVG39oTMz4A +``` -# Credentials for the USPS service -carriers_usps_userid=test_user -carriers_usps_password=Lmgxvrq89uPwECeV -.... +# Configure Vault Storage + +Hashicorp vault secures, stores, and tightly controls access to data in modern computing. +It provides advanced data protection for your testing credentials. + +MFTF works with both `vault enterprise` and `vault open source` that use `KV Version 2` secret engine. + +### Install vault CLI + +Download and install vault CLI tool if you want to run or develop MFTF tests locally. [Download Vault][Download Vault] + +### Authenticate to vault via vault CLI + +Authenticate to vault server via vault CLI tool. [Login Vault][Login Vault] + +```terminal +vault login -method -path ``` +**Do not** use `-no-store` command option, as MFTF will rely on the persisted token in the token helper +(usually the local filesystem) for future api requests. + +### Store secrets in vault + +MFTF uses `KV Version 2` secret engine for secret storage. +More information for working with `KV Version 2` can be found in [Vault KV2][Vault KV2]. + +#### Secrets path and key convention + +The path and key for secret data must follow: + +```conf +/mftf// +``` + +```conf +# Secret path and key for carriers_usps_userid +secret/mftf/magento/carriers_usps_userid + +# Secret path and key for carriers_usps_password +secret/mftf/magento/carriers_usps_password +``` + +#### Write secrets to vault + +You can use vault CLI or Api to write secret data (credentials, etc) to vault. Here is a CLI example: + +```terminal +vault kv put secret/mftf/magento/carriers_usps_userid carriers_usps_userid=usps_test_user +vault kv put secret/mftf/magento/carriers_usps_password carriers_usps_password=Lmgxvrq89uPwECeV +``` + +### Setup MFTF to use vault + +Add vault configuration environment variables [`CREDENTIAL_VAULT_ADDRESS`][] and [`CREDENTIAL_VAULT_SECRET_BASE_PATH`][] +from `etc/config/.env.example` in `.env`. +Set values according to your vault server configuration. + +```conf +# Default vault dev server +CREDENTIAL_VAULT_ADDRESS=http://127.0.0.1:8200 +CREDENTIAL_VAULT_SECRET_BASE_PATH=secret +``` + +# Configure both File Storage and Vault Storage + +It's possible and sometimes useful to setup and use both `.credentials` file and vault for secret storage at the same time. +In this case, MFTF tests are able to read secret data at runtime from both storage, and local `.credentials` file will take precedence. -## Use credentials in a test +# Use credentials in a test + +Credentials can be used in actions: [`fillField`][], [`magentoCLI`][], and [`createData`][]. -Access the data defined in the `.credentials` file using the [`fillField`][] action with the `userInput` attribute. -Define the value as a reference to the corresponding key in the credentials file such as `{{_CREDS.my_data_key}}`: +Define the value as a reference to the corresponding key in the credentials file or vault such as `{{_CREDS.my_data_key}}`: - `_CREDS` is an environment constant pointing to the `.credentials` file - `my_data_key` is a key in the the `.credentials` file that contains the value to be used in a test step -For example: +For example, reference secret data in the [`fillField`][] action with the `userInput` attribute. ```xml @@ -88,13 +161,20 @@ For example: The generated tests do not contain credentials values. The MFTF dynamically retrieves, encrypts, and decrypts the sensitive data during test execution. Decrypted credentials do not appear in the console, error logs, or [test reports][]. -The decrypted values are only available in the `.credentials` file. +The decrypted values are only available in the `.credentials` file or within vault.
The MFTF tests delivered with Magento application do not use credentials and do not cover external services, because of sensitivity of the data.
[`fillField`]: test/actions.md#fillfield +[`magentoCLI`]: test/actions.md#magentocli +[`createData`]: test/actions.md#createdata [data]: data.md [initial setup]: getting-started.md [test reports]: reporting.md +[Download Vault]: https://www.hashicorp.com/products/vault/ +[Login Vault]: https://www.vaultproject.io/docs/commands/login.html +[Vault KV2]: https://www.vaultproject.io/docs/secrets/kv/kv-v2.html +[`CREDENTIAL_VAULT_ADDRESS`]: configuration.md#CREDENTIAL_VAULT_ADDRESS +[`CREDENTIAL_VAULT_SECRET_BASE_PATH`]: configuration.md#CREDENTIAL_VAULT_SECRET_BASE_PATH From ff17f040f877055dd3d25e3ef0639fa58d976208 Mon Sep 17 00:00:00 2001 From: Donald Booth Date: Wed, 18 Sep 2019 14:05:44 -0500 Subject: [PATCH 20/27] Formatting --- docs/configuration.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index a768c76dd..2552119f6 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -234,8 +234,8 @@ It points to `MAGENTO_BASE_URL` + `dev/tests/acceptance/utils/command.php` Modify the default value: -- for non-default Magento installation -- when use a subdirectory in the `MAGENTO_BASE_URL` +- for non-default Magento installation +- when use a subdirectory in the `MAGENTO_BASE_URL` Example: `dev/tests/acceptance/utils/command.php` @@ -251,9 +251,9 @@ Example: BROWSER=firefox ``` -###CREDENTIAL_VAULT_ADDRESS +### CREDENTIAL_VAULT_ADDRESS -Api address for vault server. +The Api address for a vault server. Default: http://127.0.0.1:8200 @@ -264,7 +264,7 @@ Example: CREDENTIAL_VAULT_ADDRESS=http://127.0.0.1:8200 ``` -###CREDENTIAL_VAULT_SECRET_BASE_PATH +### CREDENTIAL_VAULT_SECRET_BASE_PATH Vault secret engine base path. @@ -282,4 +282,4 @@ CREDENTIAL_VAULT_SECRET_BASE_PATH=secret [`MAGENTO_CLI_COMMAND_PATH`]: #magento_cli_command_path [generateDate]: test/actions.md#generatedate [mftf]: commands/mftf.md -[timezones]: http://php.net/manual/en/timezones.php \ No newline at end of file +[timezones]: http://php.net/manual/en/timezones.php From e73f8b8b5deeb124f9b0efe9c01e23036b34c7cb Mon Sep 17 00:00:00 2001 From: Donald Booth Date: Wed, 18 Sep 2019 14:32:21 -0500 Subject: [PATCH 21/27] First update --- docs/credentials.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/credentials.md b/docs/credentials.md index a79f64aec..e456ca3de 100644 --- a/docs/credentials.md +++ b/docs/credentials.md @@ -5,12 +5,12 @@ use the MFTF credentials feature to hide sensitive [data][] like integration tok Currently MFTF supports two types of credential storage: **.credentials file** and **HashiCorp vault**. -# Configure File Storage +#3 Configure File Storage The MFTF creates a sample file for credentials during [initial setup][]: `magento2/dev/tests/acceptance/.credentials.example`. The file contains an example list of keys for fields that can require credentials. -### Create `.credentials` +## Create `.credentials` To make the MFTF process the file with credentials, change directories to `magento2/dev/tests/acceptance/` and copy `.credentials.example` to `.credentials`. @@ -22,7 +22,7 @@ cd dev/tests/acceptance/ cp .credentials.example .credentials ``` -### Add `.credentials` to `.gitignore` +## Add `.credentials` to `.gitignore` Verify that the file is excluded from tracking by `.gitignore` (unless you need this behavior): @@ -36,7 +36,7 @@ The command outputs the path if the file is excluded: .credentials ``` -### Define sensitive data in `.credentials` file +## Define sensitive data in `.credentials` file Open the `.credentials` file, for Magento core credentials, uncomment the fields you want to use, and add your values: @@ -69,14 +69,14 @@ Otherwise you are free to use any other `key_name` you like, as they are merely vendor/my_awesome_service_token=rRVSVnh3cbDsVG39oTMz4A ``` -# Configure Vault Storage +## Configure Vault Storage Hashicorp vault secures, stores, and tightly controls access to data in modern computing. It provides advanced data protection for your testing credentials. MFTF works with both `vault enterprise` and `vault open source` that use `KV Version 2` secret engine. -### Install vault CLI +## Install vault CLI Download and install vault CLI tool if you want to run or develop MFTF tests locally. [Download Vault][Download Vault] @@ -95,7 +95,7 @@ vault login -method -path MFTF uses `KV Version 2` secret engine for secret storage. More information for working with `KV Version 2` can be found in [Vault KV2][Vault KV2]. -#### Secrets path and key convention +### Secrets path and key convention The path and key for secret data must follow: @@ -111,7 +111,7 @@ secret/mftf/magento/carriers_usps_userid secret/mftf/magento/carriers_usps_password ``` -#### Write secrets to vault +### Write secrets to vault You can use vault CLI or Api to write secret data (credentials, etc) to vault. Here is a CLI example: @@ -132,14 +132,14 @@ CREDENTIAL_VAULT_ADDRESS=http://127.0.0.1:8200 CREDENTIAL_VAULT_SECRET_BASE_PATH=secret ``` -# Configure both File Storage and Vault Storage +## Configure both File Storage and Vault Storage -It's possible and sometimes useful to setup and use both `.credentials` file and vault for secret storage at the same time. +It is possible and sometimes useful to setup and use both `.credentials` file and vault for secret storage at the same time. In this case, MFTF tests are able to read secret data at runtime from both storage, and local `.credentials` file will take precedence. -# Use credentials in a test +## Use credentials in a test Credentials can be used in actions: [`fillField`][], [`magentoCLI`][], and [`createData`][]. From f980fe69a27bc8269764fc225570b6e26a6ca6d8 Mon Sep 17 00:00:00 2001 From: Donald Booth Date: Wed, 18 Sep 2019 15:24:12 -0500 Subject: [PATCH 22/27] Formatting --- docs/credentials.md | 42 +++++++++++++++++++++++------------------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/docs/credentials.md b/docs/credentials.md index e456ca3de..e40620d84 100644 --- a/docs/credentials.md +++ b/docs/credentials.md @@ -3,16 +3,19 @@ When you test functionality that involves external services such as UPS, FedEx, PayPal, or SignifyD, use the MFTF credentials feature to hide sensitive [data][] like integration tokens and API keys. -Currently MFTF supports two types of credential storage: **.credentials file** and **HashiCorp vault**. +Currently the MFTF supports two types of credential storage: -#3 Configure File Storage +- **.credentials file** +- **HashiCorp vault**. + +## Configure File Storage The MFTF creates a sample file for credentials during [initial setup][]: `magento2/dev/tests/acceptance/.credentials.example`. The file contains an example list of keys for fields that can require credentials. -## Create `.credentials` +### Create `.credentials` -To make the MFTF process the file with credentials, change directories to `magento2/dev/tests/acceptance/` and copy `.credentials.example` to `.credentials`. +To make the MFTF process the file with credentials, in the command line, nvaigate to `magento2/dev/tests/acceptance/` and rename `.credentials.example` to `.credentials`. ```bash cd dev/tests/acceptance/ @@ -22,7 +25,7 @@ cd dev/tests/acceptance/ cp .credentials.example .credentials ``` -## Add `.credentials` to `.gitignore` +### Add `.credentials` to `.gitignore` Verify that the file is excluded from tracking by `.gitignore` (unless you need this behavior): @@ -36,9 +39,9 @@ The command outputs the path if the file is excluded: .credentials ``` -## Define sensitive data in `.credentials` file +### Define sensitive data in the `.credentials` file -Open the `.credentials` file, for Magento core credentials, uncomment the fields you want to use, and add your values: +Open the `.credentials` file and, for Magento core credentials, uncomment the fields you want to use and add your values: ```conf ... @@ -74,30 +77,30 @@ vendor/my_awesome_service_token=rRVSVnh3cbDsVG39oTMz4A Hashicorp vault secures, stores, and tightly controls access to data in modern computing. It provides advanced data protection for your testing credentials. -MFTF works with both `vault enterprise` and `vault open source` that use `KV Version 2` secret engine. +The MFTF works with both `vault enterprise` and `vault open source` that use `KV Version 2` secret engine. -## Install vault CLI +### Install vault CLI Download and install vault CLI tool if you want to run or develop MFTF tests locally. [Download Vault][Download Vault] ### Authenticate to vault via vault CLI -Authenticate to vault server via vault CLI tool. [Login Vault][Login Vault] +Authenticate to vault server via the vault CLI tool: [Login Vault][Login Vault]. -```terminal +```bash vault login -method -path ``` -**Do not** use `-no-store` command option, as MFTF will rely on the persisted token in the token helper -(usually the local filesystem) for future api requests. + +**Do not** use `-no-store` command option, as the MFTF will rely on the persisted token in the token helper (usually the local filesystem) for future API requests. ### Store secrets in vault -MFTF uses `KV Version 2` secret engine for secret storage. +The MFTF uses the `KV Version 2` secret engine for secret storage. More information for working with `KV Version 2` can be found in [Vault KV2][Vault KV2]. ### Secrets path and key convention -The path and key for secret data must follow: +The path and key for secret data must follow the format: ```conf /mftf// @@ -113,9 +116,9 @@ secret/mftf/magento/carriers_usps_password ### Write secrets to vault -You can use vault CLI or Api to write secret data (credentials, etc) to vault. Here is a CLI example: +You can use vault CLI or API to write secret data (credentials, etc) to vault. Here is a CLI example: -```terminal +```bash vault kv put secret/mftf/magento/carriers_usps_userid carriers_usps_userid=usps_test_user vault kv put secret/mftf/magento/carriers_usps_password carriers_usps_password=Lmgxvrq89uPwECeV ``` @@ -135,7 +138,7 @@ CREDENTIAL_VAULT_SECRET_BASE_PATH=secret ## Configure both File Storage and Vault Storage It is possible and sometimes useful to setup and use both `.credentials` file and vault for secret storage at the same time. -In this case, MFTF tests are able to read secret data at runtime from both storage, and local `.credentials` file will take precedence. +In this case, the MFTF tests are able to read secret data at runtime from both storage options, but the local `.credentials` file will take precedence. @@ -164,7 +167,8 @@ Decrypted credentials do not appear in the console, error logs, or [test reports The decrypted values are only available in the `.credentials` file or within vault.
-The MFTF tests delivered with Magento application do not use credentials and do not cover external services, because of sensitivity of the data.
+The MFTF tests delivered with Magento application do not use credentials and do not cover external services, because of sensitivity of the data. + [`fillField`]: test/actions.md#fillfield From 2e228d002e3ab635101e5cefada04cb8af36b5e1 Mon Sep 17 00:00:00 2001 From: Donald Booth Date: Wed, 18 Sep 2019 15:27:21 -0500 Subject: [PATCH 23/27] Formatting --- docs/configuration.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 2552119f6..d4b86055a 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1,7 +1,7 @@ # Configuration The `*.env` file provides additional configuration for the Magento Functional Testing Framework (MFTF). -To run the MFTF on your Magento testing instance, specify the basic configuration values. +To run the MFTF on your Magento instance, specify the basic configuration values. Advanced users can create custom configurations based on requirements and environment. ## Basic configuration @@ -204,7 +204,7 @@ Example: FW_BP=~/magento/magento2-functional-testing-framework ``` -#### TESTS_MODULE_PATH +## TESTS_MODULE_PATH The path to where the MFTF modules mirror Magento modules. @@ -239,7 +239,7 @@ Modify the default value: Example: `dev/tests/acceptance/utils/command.php` -### BROWSER +## BROWSER Override the default browser performing the tests. @@ -251,7 +251,7 @@ Example: BROWSER=firefox ``` -### CREDENTIAL_VAULT_ADDRESS +## CREDENTIAL_VAULT_ADDRESS The Api address for a vault server. @@ -264,7 +264,7 @@ Example: CREDENTIAL_VAULT_ADDRESS=http://127.0.0.1:8200 ``` -### CREDENTIAL_VAULT_SECRET_BASE_PATH +## CREDENTIAL_VAULT_SECRET_BASE_PATH Vault secret engine base path. From 627b0ee5ca55b13d65e6ea634ba049604b0960d6 Mon Sep 17 00:00:00 2001 From: Ji Lu Date: Wed, 18 Sep 2019 15:43:53 -0500 Subject: [PATCH 24/27] MQE-1671: document using credentials with vault in MFTF tests --- docs/configuration.md | 8 ++++---- docs/credentials.md | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index d4b86055a..c741e2f60 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -204,7 +204,7 @@ Example: FW_BP=~/magento/magento2-functional-testing-framework ``` -## TESTS_MODULE_PATH +### TESTS_MODULE_PATH The path to where the MFTF modules mirror Magento modules. @@ -239,7 +239,7 @@ Modify the default value: Example: `dev/tests/acceptance/utils/command.php` -## BROWSER +### BROWSER Override the default browser performing the tests. @@ -251,7 +251,7 @@ Example: BROWSER=firefox ``` -## CREDENTIAL_VAULT_ADDRESS +### CREDENTIAL_VAULT_ADDRESS The Api address for a vault server. @@ -264,7 +264,7 @@ Example: CREDENTIAL_VAULT_ADDRESS=http://127.0.0.1:8200 ``` -## CREDENTIAL_VAULT_SECRET_BASE_PATH +### CREDENTIAL_VAULT_SECRET_BASE_PATH Vault secret engine base path. diff --git a/docs/credentials.md b/docs/credentials.md index e40620d84..01254a818 100644 --- a/docs/credentials.md +++ b/docs/credentials.md @@ -98,7 +98,7 @@ vault login -method -path The MFTF uses the `KV Version 2` secret engine for secret storage. More information for working with `KV Version 2` can be found in [Vault KV2][Vault KV2]. -### Secrets path and key convention +#### Secrets path and key convention The path and key for secret data must follow the format: @@ -114,7 +114,7 @@ secret/mftf/magento/carriers_usps_userid secret/mftf/magento/carriers_usps_password ``` -### Write secrets to vault +#### Write secrets to vault You can use vault CLI or API to write secret data (credentials, etc) to vault. Here is a CLI example: From b5aa05a668e2d38169321956f8574f07486cf77a Mon Sep 17 00:00:00 2001 From: Ji Lu Date: Wed, 18 Sep 2019 15:46:42 -0500 Subject: [PATCH 25/27] MQE-1671: document using credentials with vault in MFTF tests --- docs/credentials.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/credentials.md b/docs/credentials.md index 01254a818..c12303ceb 100644 --- a/docs/credentials.md +++ b/docs/credentials.md @@ -6,7 +6,7 @@ use the MFTF credentials feature to hide sensitive [data][] like integration tok Currently the MFTF supports two types of credential storage: - **.credentials file** -- **HashiCorp vault**. +- **HashiCorp vault** ## Configure File Storage From 3c3f2ce6b7e1e2bb8b46f2418e0c8136a4b74b31 Mon Sep 17 00:00:00 2001 From: Ji Lu Date: Wed, 18 Sep 2019 15:47:57 -0500 Subject: [PATCH 26/27] MQE-1671: document using credentials with vault in MFTF tests --- docs/credentials.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/credentials.md b/docs/credentials.md index c12303ceb..3a9692796 100644 --- a/docs/credentials.md +++ b/docs/credentials.md @@ -15,7 +15,7 @@ The file contains an example list of keys for fields that can require credential ### Create `.credentials` -To make the MFTF process the file with credentials, in the command line, nvaigate to `magento2/dev/tests/acceptance/` and rename `.credentials.example` to `.credentials`. +To make the MFTF process the file with credentials, in the command line, navigate to `magento2/dev/tests/acceptance/` and rename `.credentials.example` to `.credentials`. ```bash cd dev/tests/acceptance/ From 902bad6d1a40cebe7f717f49235f43217ad038e8 Mon Sep 17 00:00:00 2001 From: Ji Lu Date: Wed, 18 Sep 2019 15:52:20 -0500 Subject: [PATCH 27/27] MQE-1671: document using credentials with vault in MFTF tests --- docs/credentials.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/credentials.md b/docs/credentials.md index 3a9692796..4b1e2cc2d 100644 --- a/docs/credentials.md +++ b/docs/credentials.md @@ -149,7 +149,7 @@ Credentials can be used in actions: [`fillField`][], [`magentoCLI`][], and [`cre Define the value as a reference to the corresponding key in the credentials file or vault such as `{{_CREDS.my_data_key}}`: - `_CREDS` is an environment constant pointing to the `.credentials` file -- `my_data_key` is a key in the the `.credentials` file that contains the value to be used in a test step +- `my_data_key` is a key in the the `.credentials` file or vault that contains the value to be used in a test step For example, reference secret data in the [`fillField`][] action with the `userInput` attribute.