Support dpctl in virtual environment on Windows out of the box #1922

oleksandr-pavlyk · 2024-12-04T15:01:57Z

Since location of "Library\bin" in the virtual environment is not on the default search path, importing of dpctl fails due to unmet dependencies for native extensions of dpctl submodules.

This change introduces "_init_helper.py" file which implements the following logic using built-in os Python module:

If os.add_dll_directory attribute exists, and VIRTUAL_ENV environment variable is set, and os.path.join(os.environ["VIRTUAL_ENV"], "Library", "bin") folder exists, call os.add_dll_directory with that directory.

With this change the gh-1745 is fixed, and python -m dpctl -f works out of the box.

Only GPU devices are visible, and to enable CPU device two additional steps must be performed:

Edit %VIRUAL_ENV%\Library\bin\cl.cfg and set CL_CONFIG_TBB_DLL_PATH variable at the bottom of the configuration file to the expanded value of %VIRUAL_ENV%\Library\bin\tbb12.dll but use forward slashes, instead of native backward slashes.
Append %VIRUAL_ENV%\Library\bin to the PATH using set "PATH=%PATH%:%VIRTUAL_ENV%\Library\bin"

After these changes python -m dpctl -f should see CPU device.

@icfaust

Have you provided a meaningful PR description?
Have you added a test, reproducer or referred to an issue with a reproducer?
Have you tested your changes locally for CPU and GPU devices?
Have you made sure that new changes do not introduce compiler warnings?
Have you checked performance impact of proposed changes?
Have you added documentation for your changes, if necessary?
Have you added your changes to the changelog?
If this PR is a work in progress, are you opening the PR as a draft?

Since location of Library\bin in virtual environment is not on the default search path, importing of dpctl failes due to unmet dependencies for native extensions of dpctl submodules. This change introduces _init_helper.py which implements the following logic using built-in os Python module: 1. If os.add_dll_directory exists, and VIRTUAL_ENV environment variable is set, and os.path.join(os.environ["VIRTUAL_ENV"], "Library", "bin") exists, call os.add_dll_directory with that directory. With this change the gh-1745 is fixed, and "python -m dpctl -f" works out of the box. Only GPU devices are visible, and to enable CPU device two additional steps must be performed: 1. Edit %VIRUAL_ENV%\Library\bin\cl.cfg and set CL_CONFIG_TBB_DLL_PATH variable at the bottom of the configuration file to the expanded value of %VIRUAL_ENV%\Library\bin\tbb12.dll but use forward slashes, instead of native backward slashes. 2. Append %VIRUAL_ENV%\Library\bin to the PATH using `set "PATH=%PATH%:%VIRTUAL_ENV%\Library\bin"` After these changes `python -m dpctl -f` should see CPU device.

github-actions · 2024-12-04T15:38:47Z

Deleted rendered PR docs from intelpython.github.com/dpctl, latest should be updated shortly. 🤞

coveralls · 2024-12-04T15:47:06Z

coverage: 87.647% (-0.02%) from 87.671%
when pulling c6eea90 on support-venv-windows-gh-1745
into 5e35e0e on master.

github-actions · 2024-12-04T15:50:00Z

Array API standard conformance tests for dpctl=0.19.0dev0=py310hdf72452_294 ran successfully.
Passed: 895
Failed: 1
Skipped: 118

dpctl/_init_helper.py

icfaust · 2024-12-04T23:00:54Z

@ethanglaser PR fix for the issue we talked about.

icfaust · 2024-12-05T08:40:01Z

You should be able to steer and modify the cl.cfg if you use os.sep to generalize your pathing instead of hardcoding /

Co-authored-by: ndgrigorian <46709016+ndgrigorian@users.noreply.github.com>

oleksandr-pavlyk · 2024-12-05T14:30:32Z

@icfaust The cl.cfg configuration file is not a part of dpctl wheel, it is a part of intel-opencl-rt wheel. It does not seem right to automatically change it.

dpctl could provide a script that a user can run to modify cl.cfg, but it should be user's explicit action to invoke it.

An alternative might be to configure OpenCL driver for CPU device to read cl.cfg provided by either a user or dpctl which we could instruct user to modify as needed.

github-actions · 2024-12-05T14:45:13Z

Array API standard conformance tests for dpctl=0.19.0dev0=py310hdf72452_295 ran successfully.
Passed: 895
Failed: 1
Skipped: 118

ndgrigorian

Installed wheel on Windows machine and it worked out of the box, so LGTM!

oleksandr-pavlyk · 2024-12-05T22:24:17Z

@icfaust By the way, it turns out that augmenting PATH variable in item 2 rendered changes in item 1 superfluous. The CPU device is discoverable even without modifying "cl.cfg" file.

github-actions · 2024-12-06T20:23:28Z

Array API standard conformance tests for dpctl=0.19.0dev0=py310hdf72452_296 ran successfully.
Passed: 894
Failed: 2
Skipped: 118

dpctl/_init_helper.py

ndgrigorian · 2024-12-06T21:55:21Z

@oleksandr-pavlyk
In addition, with sys being imported, we can also do if sys.platform == "win32": instead of checking for add_dll_directory attribute, which seems a little clearer.

oleksandr-pavlyk · 2024-12-06T22:14:31Z

Thank you for the suggestion @ndgrigorian .

I have pushed the change to implement it, and the fix no longer relies on environment variables, alleviating security concerns.

@ndgrigorian

Use check suggested in PEP-0405 and pointed out by @ndgrigorian to compare sys.base_exec_prefix and sys.exec_prefix which would be different under virtual environment. The check that pyvenv.cfg exists is retained.

github-actions · 2024-12-06T22:55:46Z

Array API standard conformance tests for dpctl=0.19.0dev0=py310hdf72452_297 ran successfully.
Passed: 895
Failed: 1
Skipped: 118

github-actions · 2024-12-06T23:09:53Z

Array API standard conformance tests for dpctl=0.19.0dev0=py310hdf72452_297 ran successfully.
Passed: 895
Failed: 1
Skipped: 118

ndgrigorian

LGTM, tested in virtualenv on Windows and it worked.

Thanks @oleksandr-pavlyk !

Backport gh-1922

oleksandr-pavlyk requested a review from ndgrigorian as a code owner December 4, 2024 15:01

oleksandr-pavlyk requested a review from antonwolfy December 4, 2024 15:02

oleksandr-pavlyk changed the title ~~Support dpctl in vritual environment out of the box~~ Support dpctl in virtual environment on Windows out of the box Dec 4, 2024

ndgrigorian reviewed Dec 4, 2024

View reviewed changes

dpctl/_init_helper.py Outdated Show resolved Hide resolved

Update dpctl/_init_helper.py

d8bc19e

Co-authored-by: ndgrigorian <46709016+ndgrigorian@users.noreply.github.com>

oleksandr-pavlyk requested a review from ndgrigorian December 5, 2024 19:48

ndgrigorian previously approved these changes Dec 5, 2024

View reviewed changes

More robust check that VIRUAL_ENV is valid

eab51fd

oleksandr-pavlyk dismissed ndgrigorian’s stale review via eab51fd December 6, 2024 19:37

oleksandr-pavlyk requested a review from ndgrigorian December 6, 2024 20:49

ndgrigorian reviewed Dec 6, 2024

View reviewed changes

dpctl/_init_helper.py Outdated Show resolved Hide resolved

Avoid reliance of VIRTUAL_ENV environment variable

c6eea90

Use check suggested in PEP-0405 and pointed out by @ndgrigorian to compare sys.base_exec_prefix and sys.exec_prefix which would be different under virtual environment. The check that pyvenv.cfg exists is retained.

oleksandr-pavlyk force-pushed the support-venv-windows-gh-1745 branch from 16c3f9d to c6eea90 Compare December 6, 2024 22:29

oleksandr-pavlyk requested a review from ndgrigorian December 7, 2024 00:32

ndgrigorian approved these changes Dec 7, 2024

View reviewed changes

oleksandr-pavlyk merged commit 567ca24 into master Dec 7, 2024
49 of 51 checks passed

oleksandr-pavlyk deleted the support-venv-windows-gh-1745 branch December 7, 2024 00:43

oleksandr-pavlyk mentioned this pull request Dec 7, 2024

Backport gh-1922 #1924

Merged

8 tasks

oleksandr-pavlyk added a commit that referenced this pull request Dec 7, 2024

Merge pull request #1924 from IntelPython/backport-gh-1922

69be39d

Backport gh-1922

oleksandr-pavlyk mentioned this pull request Dec 10, 2024

dpctl pip install broken on windows for python3.11 #1745

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support dpctl in virtual environment on Windows out of the box #1922

Support dpctl in virtual environment on Windows out of the box #1922

oleksandr-pavlyk commented Dec 4, 2024

github-actions bot commented Dec 4, 2024 •

edited

Loading

coveralls commented Dec 4, 2024 •

edited

Loading

github-actions bot commented Dec 4, 2024

icfaust commented Dec 4, 2024

icfaust commented Dec 5, 2024

oleksandr-pavlyk commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

ndgrigorian left a comment

oleksandr-pavlyk commented Dec 5, 2024

github-actions bot commented Dec 6, 2024

ndgrigorian commented Dec 6, 2024

oleksandr-pavlyk commented Dec 6, 2024

github-actions bot commented Dec 6, 2024

github-actions bot commented Dec 6, 2024

ndgrigorian left a comment

Support dpctl in virtual environment on Windows out of the box #1922

Support dpctl in virtual environment on Windows out of the box #1922

Conversation

oleksandr-pavlyk commented Dec 4, 2024

github-actions bot commented Dec 4, 2024 • edited Loading

coveralls commented Dec 4, 2024 • edited Loading

github-actions bot commented Dec 4, 2024

icfaust commented Dec 4, 2024

icfaust commented Dec 5, 2024

oleksandr-pavlyk commented Dec 5, 2024

github-actions bot commented Dec 5, 2024

ndgrigorian left a comment

Choose a reason for hiding this comment

oleksandr-pavlyk commented Dec 5, 2024

github-actions bot commented Dec 6, 2024

ndgrigorian commented Dec 6, 2024

oleksandr-pavlyk commented Dec 6, 2024

github-actions bot commented Dec 6, 2024

github-actions bot commented Dec 6, 2024

ndgrigorian left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 4, 2024 •

edited

Loading

coveralls commented Dec 4, 2024 •

edited

Loading