Magento Automated Testing

Magento Automated Testing

Magento Automated Testing is a testing infrastructure that allows to run the set of all required Magento Automated Tests against code in Pull Requests. It automatically triggers the tests against the changes on Pull Request creation or new commits pushes into the Pull Request. The running tests are represented both as Pull Request statuses and as Pull Requests checks.

Below there's information about all kinds of tests available currently.

Semantic Version Checker

Semantic Version Checker is a test that compares the changed code against the mainline and detects levels of code changes. It shows a list of changes grouped by the class (level) of changes (MAJOR, MINOR, PATCH) with explanation of the change (Class was removed, [public] Method implementation changed, [private] Property has been removed etc.) The allowed level is PATCH.

Functional Tests

Sample Data Tests

Sample Data Tests are functional tests that run on Magento with Sample Data included. Currently they include two kinds of tests:

• Sample Data functional tests which check Magento functional behavior with Sample Data installed
• Installation Functional tests which check Magento installation flow with Sample Data installed

Unit Tests

Integration Tests

WebAPI Tests

Static Tests

Database Compare

Database Compare is a test which compares DB schema and data between freshly installed Magento instance (from the PR code) and another instance which was upgraded to the PR code from the pre-previous latest minor release Magento version. Usually it runs several separate scenarios:

• Upgrade from the latest pre-previous minor release directly to the PR code (e.g. 2.1.x => 2.3-based PR code)
• Upgrade from the latest pre-previous minor release to previous minor release, and the to the PR code (e.g. 2.1.x => 2.2.x => 2.3-based PR code)
• Upgrade from the latest pre-previous minor release directly to the PR code with sample data (e.g. 2.1.x => 2.3-based PR code)
• Upgrade from the latest pre-previous minor release to previous minor release, and the to the PR code with sample data (e.g. 2.1.x => 2.2.x => 2.3-based PR code)

Magento Component Health Index

Magento Component Health Index is a test which analyses Magento code quality and calculates weighted health index (100 is great, lower is worse) per Magento Component (module). The index is calculated based on 3 groups of indicators:

• General code quality indicators (GCQI). The metrics described here:
  • Inheritance rate
  • Coupling Between Objects
  • Cyclomatic Complexity Number
  • Class Size
  • NPath Complexity
  • Code Rank
  • Number Of Methods
  • Number Of Child Classes
• Magento-specific code quality indicators (MSCQ):
  • Inline JS usage (<script> tag without attribute x-magento-init or x-magento-template)
  • PHP Static method calls (except unit tests)
  • PHP global state usage (global variables usage & static local variable usage, registry object usage)
  • PHP usage of protected modifiers for methods & properties (except unit tests)
  • PHP and JS 'fixmeComments' usage (TODO, FIXME, HACK, XXX, @SuppressWarnings etc.)
• Legacy API usage indicators (LAI):
  • Legacy method calls in action classes (loadLayout(), renderLayout() etc.)
  • Legacy classes extension (\Magento\Backend\Block\Widget\Grid\Extended, \Magento\Backend\Block\Widget\Form, \Magento\Backend\Block\Widget\Form\Generic, \Magento\Payment\Model\Method\AbstractMethod, \Magento\Payment\Model\Method\Cc, \Magento\Backend\Block\Widget\Grid etc.)
  • Legacy layout block classes usage
  • Direct class calls instead of an interface that has DI preference
  • Comparison of composer dependencies against etalon file