Skip to content

Commit af7e43d

Browse files
brandon-leapyearZac-HD
authored andcommitted
Add support + docs for running individual tests
1 parent ea10729 commit af7e43d

File tree

3 files changed

+63
-119
lines changed

3 files changed

+63
-119
lines changed

CONTRIBUTING.rst

+51-1
Original file line numberDiff line numberDiff line change
@@ -165,7 +165,7 @@ Some notable commands:
165165
``./build.sh check-coverage`` will verify 100% code coverage by running a
166166
curated subset of the test suite.
167167

168-
``./build.sh check-py36`` (etc.) will run most of the test suite against a
168+
``./build.sh check-py37`` (etc.) will run most of the test suite against a
169169
particular python version.
170170

171171
``./build.sh format`` will reformat your code according to the Hypothesis coding style. You should use this before each
@@ -180,3 +180,53 @@ Run ``./build.sh tasks`` for a list of all supported build task names.
180180
Note: The build requires a lot of different versions of python, so rather than have you install them yourself,
181181
the build system will install them itself in a local directory. This means that the first time you run a task you
182182
may have to wait a while as the build downloads and installs the right version of python for you.
183+
184+
~~~~~~~~~~~~~
185+
Running Tests
186+
~~~~~~~~~~~~~
187+
188+
The tasks described above will run all of the tests (e.g. ``check-py37``). But
189+
the ``tox`` task will give finer-grained control over the test runner. At a
190+
high level, the task takes the form:
191+
192+
.. code-block::
193+
194+
./build.sh tox py37-custom 3.7.13 [tox args] -- [pytest args]
195+
196+
Namely, first provide the tox environment (see ``tox.ini``), then the python
197+
version to test with, then any ``tox`` or ``pytest`` args as needed. For
198+
example, to run all of the tests in the file
199+
``tests/nocover/test_conjecture_engine.py`` with python 3.8:
200+
201+
.. code-block::
202+
203+
./build.sh tox py38-custom 3.8.13 -- tests/nocover/test_conjecture_engine.py
204+
205+
See the ``tox`` docs and ``pytest`` docs for more information:
206+
* https://docs.pytest.org/en/latest/how-to/usage.html
207+
* https://tox.wiki/en/latest/config.html#cli
208+
209+
^^^^^^^^^^^
210+
Test Layout
211+
^^^^^^^^^^^
212+
213+
See ``hypothesis-python/tests/README.rst``
214+
215+
^^^^^^^^^^^^^^^^
216+
Useful Arguments
217+
^^^^^^^^^^^^^^^^
218+
219+
Some useful arguments to pytest include:
220+
221+
* You can pass ``-n 0`` to turn off ``pytest-xdist``'s parallel test execution.
222+
Sometimes for running just a small number of tests its startup time is longer
223+
than the time it saves (this will vary from system to system), so this can
224+
be helpful if you find yourself waiting on test runners to start a lot.
225+
* You can use ``-k`` to select a subset of tests to run. This matches on substrings
226+
of the test names. For example ``-kfoo`` will only run tests that have "foo" as
227+
a substring of their name. You can also use composite expressions here.
228+
e.g. ``-k'foo and not bar'`` will run anything containing foo that doesn't
229+
also contain bar. `More information on how to select tests to run can be found
230+
in the pytest documentation <https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`__.
231+
232+

guides/testing-hypothesis.rst

+2-116
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ a starting point, not the final goal.
3737
because when it was found and fixed, someone wrote a test to make sure it
3838
couldn't come back!
3939

40-
The ``tests/`` directory has some notes in the README file on where various
40+
The ``hypothesis-python/tests/`` directory has some notes in the README file on where various
4141
kinds of tests can be found or added. Go there for the practical stuff, or
4242
just ask one of the maintainers for help on a pull request!
4343

@@ -48,122 +48,8 @@ Further reading: How `SQLite is tested <https://sqlite.org/testing.html>`_,
4848
Dan Luu writes about `fuzz testing <https://danluu.com/testing/>`_ and
4949
`broken processes <https://danluu.com/wat/>`_, among other things.
5050

51-
---------------------------------------
52-
Setting up a virtualenv to run tests in
53-
---------------------------------------
54-
55-
If you want to run individual tests rather than relying on the make tasks
56-
(which you probably will), it's easiest to do this in a virtualenv.
57-
58-
The following will give you a working virtualenv for running tests in:
59-
60-
.. code-block:: bash
61-
62-
pip install virtualenv
63-
python -m virtualenv testing-venv
64-
65-
# On Windows: testing-venv\Scripts\activate
66-
source testing-venv/bin/activate
67-
68-
# Can also use pip install -e .[all] to get
69-
# all optional dependencies
70-
pip install -e .
71-
72-
# Test specific dependencies.
73-
pip install -r requirements/test.in
74-
75-
Now whenever you want to run tests you can just activate the virtualenv
76-
using ``source testing-venv/bin/activate`` or ``testing-venv\Scripts\activate``
77-
and all of the dependencies will be available to you and your local copy
78-
of Hypothesis will be on the path (so any edits will be picked up automatically
79-
and you don't need to reinstall it in the local virtualenv).
80-
8151
-------------
8252
Running Tests
8353
-------------
8454

85-
In order to run tests outside of the make/tox/etc set up, you'll need an
86-
environment where Hypothesis is on the path and all of the testing dependencies
87-
are installed.
88-
We recommend doing this inside a virtualenv as described in the previous section.
89-
90-
All testing is done using `pytest <https://docs.pytest.org/en/latest/>`_,
91-
with a couple of plugins installed. For advanced usage we recommend reading the
92-
pytest documentation, but this section will give you a primer in enough of the
93-
common commands and arguments to get started.
94-
95-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
96-
Selecting Which Files to Run
97-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
98-
99-
The following invocation runs all of the tests in the file
100-
`tests/cover/test_conjecture_engine.py`:
101-
102-
.. code-block::
103-
104-
python -m pytest tests/cover/test_conjecture_engine.py
105-
106-
If you want to run multiple files you can pass them all as arguments, and if
107-
you pass a directory then it will run all files in that directory.
108-
For example the following runs all the files in `test_conjecture_engine.py`
109-
and `test_slippage.py`
110-
111-
.. code-block::
112-
113-
python -m pytest tests/cover/test_conjecture_engine.py tests/cover/test_slippage.py
114-
115-
If you were running this in bash (if you're not sure: if you're not on Windows
116-
you probably are) you could also use the syntax:
117-
118-
.. code-block::
119-
120-
python -m pytest tests/cover/test_{conjecture_engine,slippage}.py
121-
122-
And the following would run all tests under `tests/cover`:
123-
124-
.. code-block::
125-
126-
python -m pytest tests/cover
127-
128-
129-
~~~~~~~~~~~
130-
Test Layout
131-
~~~~~~~~~~~
132-
133-
The top level structure of the tests in Hypothesis looks as follows:
134-
135-
* ``cover`` contains tests that we measure coverage for. This is intended to
136-
be a fairly minimal and fast set of tests that still gives pretty good
137-
confidence in the behaviour of the test suite. It is currently failing at
138-
both "minimal" and "fast", but we're trying to move it back in that
139-
direction. Try not to add tests to this unless they're actually to cover
140-
some specific target.
141-
* ``nocover`` is a general dumping ground for slower tests that aren't needed
142-
to achieve coverage.
143-
* ``quality`` is for expensive tests about the distribution or shrinking of
144-
examples. These will only be run on one Python version.
145-
* The remaining test directories are for testing specific extras modules and
146-
should have the same name.
147-
148-
As a rule of thumb when writing new tests, they should go in nocover unless
149-
they are for a specific extras module or to deliberately target a particular
150-
line for coverage. In the latter case, prefer fast unit tests over larger and
151-
slower integration tests (we are not currently very good at this).
152-
153-
154-
~~~~~~~~~~~~~~~~
155-
Useful Arguments
156-
~~~~~~~~~~~~~~~~
157-
158-
Some useful arguments to pytest include:
159-
160-
* You can pass ``-n 0`` to turn off ``pytest-xdist``'s parallel test execution.
161-
Sometimes for running just a small number of tests its startup time is longer
162-
than the time it saves (this will vary from system to system), so this can
163-
be helpful if you find yourself waiting on test runners to start a lot.
164-
* You can use ``-k`` to select a subset of tests to run. This matches on substrings
165-
of the test names. For example ``-kfoo`` will only run tests that have "foo" as
166-
a substring of their name. You can also use composite expressions here.
167-
e.g. ``-k'foo and not bar'`` will run anything containing foo that doesn't
168-
also contain bar. `More information on how to select tests to run can be found
169-
in the pytest documentation <https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`__.
55+
Tests are run via ``build.sh``. See ``CONTRIBUTING.rst`` for more details.

tooling/src/hypothesistooling/__main__.py

+10-2
Original file line numberDiff line numberDiff line change
@@ -356,7 +356,7 @@ def documentation():
356356
)
357357

358358

359-
def run_tox(task, version):
359+
def run_tox(task, version, *args):
360360
python = install.python_executable(version)
361361

362362
# Create a version of the name that tox will pick up for the correct
@@ -373,7 +373,7 @@ def run_tox(task, version):
373373
env["PATH"] = os.path.dirname(python) + ":" + env["PATH"]
374374
print(env["PATH"])
375375

376-
pip_tool("tox", "-e", task, env=env, cwd=hp.HYPOTHESIS_PYTHON)
376+
pip_tool("tox", "-e", task, *args, env=env, cwd=hp.HYPOTHESIS_PYTHON)
377377

378378

379379
# See update_python_versions() above
@@ -449,6 +449,14 @@ def check_pypy38():
449449
run_tox("pypy3-full", PYPY38)
450450

451451

452+
@task()
453+
def tox(*args):
454+
if len(args) == 0:
455+
print("Usage: ./build.sh tox TOX_ENV PY_VERSION [tox args]")
456+
sys.exit(1)
457+
run_tox(args[0], args[1], *args[2:])
458+
459+
452460
def standard_tox_task(name):
453461
TASKS["check-" + name] = python_tests(lambda: run_tox(name, PYMAIN))
454462

0 commit comments

Comments
 (0)