Skip to content

Commit 4da6c29

Browse files
orbsmivorbsmiv
committed
ported
1 parent c50d2a6 commit 4da6c29

File tree

1 file changed

+68
-81
lines changed

1 file changed

+68
-81
lines changed

‎docs/installing-extensions-in-parallel.md

+68-81
Original file line numberDiff line numberDiff line change
@@ -1,109 +1,96 @@
1-
.. _installing_extensions_in_parallel:
1+
# Installing extensions in parallel *(experimental!)* {: #installing_extensions_in_parallel }
22

3-
Installing extensions in parallel (experimental!)
4-
=================================================
3+
Since EasyBuild v4.5.0 (initial) support is available for installing
4+
extensions in parallel, that is to run the installation command for
5+
multiple extensions that are ready to install at the same time in the
6+
background, to exploit the availability of multiple cores.
57

6-
Since EasyBuild v4.5.0 (initial) support is available for installing extensions in parallel,
7-
that is to run the installation command for multiple extensions that are ready to install at the same time
8-
in the background, to exploit the availability of multiple cores.
8+
**Note: this is an**
9+
[experimental feature][experimental_features]. **Some of the mentioned functionality may be subject to
10+
change or be prone to errors.**
911

10-
**Note: this is an** :ref:`experimental feature <experimental_features>`.
11-
**Some of the mentioned functionality may be subject to change or be prone to errors.**
12+
## Configuration {: #installing_extensions_in_parallel_configuration }
1213

13-
.. contents::
14-
:depth: 3
15-
:backlinks: none
14+
To let EasyBuild install extensions in parallel, the
15+
`parallel-extensions-install` configuration option must be enabled.
1616

17-
.. _installing_extensions_in_parallel_configuration:
17+
In addition, since the support for installing extensions in parallel is
18+
an experimental feature, the `experimental` configuration option must be
19+
also be enabled (see also [Experimental features][experimental_features]).
1820

19-
Configuration
20-
-------------
21+
How many extension installations should be started (at maximum) at the
22+
same time is controlled by the `parallel` configuration option.
2123

22-
To let EasyBuild install extensions in parallel, the ``parallel-extensions-install`` configuration option must be
23-
enabled.
24+
## Requirements {: #installing_extensions_in_parallel_requirements }
2425

25-
In addition, since the support for installing extensions in parallel is an experimental feature, the ``experimental``
26-
configuration option must be also be enabled (see also :ref:`experimental_features`).
26+
For EasyBuild to be able to install a list of extensions in parallel,
27+
two requirements must be met:
2728

28-
How many extension installations should be started (at maximum) at the same time is controlled by
29-
the ``parallel`` configuration option.
29+
- The easyblock(s) that are used for installing the extensions must
30+
have support for determining which installed extensions are required
31+
to fulfil the dependencies for a particular extension. This is done
32+
via the `required_deps` property method.
33+
- EasyBuild must be able to start a single shell command
34+
(asynchronously, so it runs in the background) to perform the
35+
installation of the extension via the `run_async` method, and check
36+
whether the command has completed via the `async_cmd_check` method.
3037

38+
For an example of how `required_deps`, `run_async`, and
39+
`async_cmd_check` can be implemented, see the
40+
[RPackage](https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/generic/rpackage.py)
41+
easyblock.
3142

32-
.. _installing_extensions_in_parallel_requirements:
33-
34-
Requirements
35-
------------
36-
37-
For EasyBuild to be able to install a list of extensions in parallel, two requirements must be met:
38-
39-
* The easyblock(s) that are used for installing the extensions must have support for determining
40-
which installed extensions are required to fulfill the dependencies for a particular extension.
41-
This is done via the ``required_deps`` property method.
42-
* EasyBuild must be able to start a single shell command (asynchronously, so it runs in the background)
43-
to perform the installation of the extension via the ``run_async`` method,
44-
and check whether the command has completed via the ``async_cmd_check`` method.
45-
46-
For an example of how ``required_deps``, ``run_async``, and ``async_cmd_check`` can be implemented,
47-
see the `RPackage
48-
<https://github.com/easybuilders/easybuild-easyblocks/blob/main/easybuild/easyblocks/generic/rpackage.py>`_ easyblock.
49-
50-
.. _installing_extensions_in_parallel_caveats_known_issues:
51-
52-
Caveats and Known issues
53-
------------------------
43+
## Caveats and Known issues {: #installing_extensions_in_parallel_caveats_known_issues }
5444

5545
*(last update: EasyBuild v4.5.0)*
5646

57-
There are a couple of caveats and known issues with installing extensions in parallel,
58-
which we hope to resolve in future EasyBuild releases.
47+
There are a couple of caveats and known issues with installing
48+
extensions in parallel, which we hope to resolve in future EasyBuild
49+
releases.
5950

60-
This is also why the support for installing extensions in parallel is currently marked as an experimental feature.
51+
This is also why the support for installing extensions in parallel is
52+
currently marked as an experimental feature.
6153

62-
.. _installing_extensions_in_parallel_caveats_known_issues_only_r:
54+
### Only works for R extensions {: #installing_extensions_in_parallel_caveats_known_issues_only_r }
6355

64-
Only works for R extensions
65-
~~~~~~~~~~~~~~~~~~~~~~~~~~~
56+
The support for installing extensions in parallel currently only works
57+
for R extensions, since only the `RPackage` easyblock (which is used to
58+
install R packages as extensions) implements the necessary methods (see
59+
[Requirements][installing_extensions_in_parallel_requirements]).
6660

67-
The support for installing extensions in parallel currently only works for R extensions,
68-
since only the ``RPackage`` easyblock (which is used to install R packages as extensions) implements
69-
the necessary methods (see :ref:`installing_extensions_in_parallel_requirements`).
61+
Supporting other types of extensions (Python, Perl, etc.) requires that
62+
the corresponding easyblocks that are used to install those extensions
63+
(like `PythonPackage`, `PerlModule`) are enhanced to support determining
64+
required dependencies and starting the installation command
65+
asynchronously.
7066

71-
Supporting other types of extensions (Python, Perl, etc.) requires that the corresponding easyblocks that
72-
are used to install those extensions (like ``PythonPackage``, ``PerlModule``) are enhanced to support determining
73-
required dependencies and starting the installation command asynchronously.
67+
### List of extensions must be self-contained {: #installing_extensions_in_parallel_caveats_known_issues_exts_list }
7468

75-
.. _installing_extensions_in_parallel_caveats_known_issues_exts_list:
76-
77-
List of extensions must be self-contained
78-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
79-
80-
The EasyBuild framework currently enforces that all required dependencies for a particular extension
81-
are included in the list of extensions that is being installed (specified via the ``exts_list`` easyconfig
82-
parameter). Extensions provided by dependencies are not taken into account (yet).
69+
The EasyBuild framework currently enforces that all required
70+
dependencies for a particular extension are included in the list of
71+
extensions that is being installed (specified via the `exts_list`
72+
easyconfig parameter). Extensions provided by dependencies are not taken
73+
into account (yet).
8374

8475
As a result, trying to install an
85-
`R-bundle-Bioconductor <https://github.com/easybuilders/easybuild-easyconfigs/tree/main/easybuild/easyconfigs/r/R-bundle-Bioconductor>`_
86-
easyconfig with ``parallel-extensions-install`` enabled
87-
will result in an error reporting missing required dependencies.
88-
89-
90-
.. _installing_extensions_in_parallel_caveats_known_issues_skip:
91-
92-
Skipping of installed extensions is still done sequentially
93-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
94-
95-
Checking which extensions are already installed for which the installation can be skipped
96-
when ``--skip`` (see :ref:`partial_installation_skip`) is being used is still done sequentially.
76+
[R-bundle-Bioconductor](https://github.com/easybuilders/easybuild-easyconfigs/tree/main/easybuild/easyconfigs/r/R-bundle-Bioconductor)
77+
easyconfig with `parallel-extensions-install` enabled will result in an
78+
error reporting missing required dependencies.
9779

98-
Additional work is needed to also run these checks in parallel across the cores that are available to EasyBuild.
80+
### Skipping of installed extensions is still done sequentially {: #installing_extensions_in_parallel_caveats_known_issues_skip }
9981

82+
Checking which extensions are already installed for which the
83+
installation can be skipped when `--skip` (see
84+
[Installing additional extensions using `-k`/`-skip`][partial_installation_skip]) is being used
85+
is still done sequentially.
10086

101-
.. _installing_extensions_in_parallel_caveats_known_issues_sanity_check:
87+
Additional work is needed to also run these checks in parallel across
88+
the cores that are available to EasyBuild.
10289

103-
Sanity check for extensions is still run sequentially
104-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
90+
### Sanity check for extensions is still run sequentially {: #installing_extensions_in_parallel_caveats_known_issues_sanity_check }
10591

106-
The sanity checks that are done for each extension are still done through a sequential loop over the
107-
list of extensions.
92+
The sanity checks that are done for each extension are still done
93+
through a sequential loop over the list of extensions.
10894

109-
Additional work is needed to also run these checks in parallel across the cores that are available to EasyBuild.
95+
Additional work is needed to also run these checks in parallel across
96+
the cores that are available to EasyBuild.

0 commit comments

Comments
 (0)