Skip to content
Prev Previous commit
Next Next commit
Restore support for external platform Python package dependencies
Although Arduino boards platforms traditionally have bundled all dependencies, there are some cases where the user will
need to use a preceding workflow step to install external dependencies in the GitHub Actions runner environment.

Since it is a "composite action", this is feasible and was previously supported. However, support for Python package
dependencies is more complicated due to the action script running in a Python virtual environment which isolates it from
packages installed globally. The workaround for this was to use the `--system-site-packages` flag in the venv invocation
that created the action's Python virtual environment. This approach introduced a requirement that the project Python
version match the minor version series of the runner system Python. That requirement was violated when the workflow's
runner was updated to `ubuntu-22.04` which has system Python 3.10.x, whereas the action used 3.8. The breakage went
unnoticed because the sole Python package platform dependency in real world usage, pyserial, had been preinstalled in
the runner's Linux package manager "site-packages" folder, which caused it to be installed in the action's virtual
environment even though it would not have if depending on the user installation of the package as was previously the
case.

The solution is to use a path configuration file to add the path of the system Python's "user site-packages" folder
(where package dependencies installed via the workflow are located) to the module search paths used by the action's
Python virtual environment.

This is actually superior to the previous approach of using the `--system-site-packages` flag in the venv invocation
(this flag is also supported by pipx and Poetry) because:

- It avoids the chance of interference with the virtual environment through the unnecessary introduction of the global
  "site-packages" path into the module search paths along with the intended introduction of the "user site-packages"
  path.
- It allows any version of Python to be used to run the action, rather than being forced to use the same minor version
  series as the runner's system Python (which can be updated at any time and also depends on the workflow configuration)
  • Loading branch information
per1234 committed Mar 23, 2023
commit 4c2850340f7be5ab270d6506df5f9f1d1f8c4c86
6 changes: 5 additions & 1 deletion .github/workflows/test-integration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,11 @@ jobs:
uses: actions/checkout@v3

- name: Install Python package dependency
run: pip install cowsay
run: |
pip install \
--ignore-installed \
--user \
cowsay

- name: Run action with board that has external Python package dependency
# Use action from local path
Expand Down
24 changes: 24 additions & 0 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,13 @@ inputs:
runs:
using: composite
steps:
# User installations of external Python package platform dependencies will be located here.
- name: Get system Python "user site-packages" path
id: system-user-site-packages
shell: bash
run: |
echo "path=$(python -m site --user-site)" >> $GITHUB_OUTPUT

- name: Install Python
uses: actions/setup-python@v4.5.0
with:
Expand All @@ -67,6 +74,23 @@ runs:
poetry install \
--only main

- name: Make user-installed Python packages available to platforms
shell: bash
working-directory: ${{ github.action_path }}
run: |
readonly PYTHON_ENVIRONMENT_PATH="$(
poetry env info \
--path
)"
readonly VENV_SITE_PACKAGES_PATH="$(
poetry run \
python -c \
'import site; print(site.getsitepackages()[0])'
)"
echo \
"${{ steps.system-user-site-packages.outputs.path }}" > \
"${VENV_SITE_PACKAGES_PATH}/system-user-site-packages.pth"

- name: Run script
shell: bash
env:
Expand Down
17 changes: 17 additions & 0 deletions docs/FAQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,20 @@ The **arduino/compile-sketches** action runs in the same environment as the rest
```

---

#### Python Packages

The **arduino/compile-sketches** action uses a Python [virtual environment](https://docs.python.org/glossary.html#term-virtual-environment). In order to enable user installation of Python [package](https://docs.python.org/glossary.html#term-package) dependencies of boards platforms, the packages installed in the "[user site-packages](https://peps.python.org/pep-0370/)" folder are included in this virtual environment.

In order to be certain your installation of a package dependency will be available to the platform, add the [`--ignore-installed`](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-ignore-installed) and [`--user`](https://pip.pypa.io/en/stable/cli/pip_install/#install-user) flags to the [**pip**](https://pip.pypa.io/) command used to install the package.

---

**Example:**

```yaml
- run: pip install --ignore-installed --user pyserial
- uses: arduino/compile-sketches@v1
```

---