Add support + docs for running individual tests

Author: brandon-leapyear

CONTRIBUTING.rst

@@ -165,7 +165,7 @@ Some notable commands:
 ``./build.sh check-coverage`` will verify 100% code coverage by running a
 curated subset of the test suite.
 
-``./build.sh check-py36`` (etc.) will run most of the test suite against a
+``./build.sh check-py37`` (etc.) will run most of the test suite against a
 particular python version.
 
 ``./build.sh format`` will reformat your code according to the Hypothesis coding style. You should use this before each
@@ -180,3 +180,53 @@ Run ``./build.sh tasks`` for a list of all supported build task names.
 Note: The build requires a lot of different versions of python, so rather than have you install them yourself,
 the build system will install them itself in a local directory. This means that the first time you run a task you
 may have to wait a while as the build downloads and installs the right version of python for you.
+
+~~~~~~~~~~~~~
+Running Tests
+~~~~~~~~~~~~~
+
+The tasks described above will run all of the tests (e.g. ``check-py37``). But
+the ``tox`` task will give finer-grained control over the test runner. At a
+high level, the task takes the form:
+
+.. code-block::
+
+    ./build.sh tox py37-custom 3.7.13 [tox args] -- [pytest args]
+
+Namely, first provide the tox environment (see ``tox.ini``), then the python
+version to test with, then any ``tox`` or ``pytest`` args as needed. For
+example, to run all of the tests in the file
+``tests/nocover/test_conjecture_engine.py`` with python 3.8:
+
+.. code-block::
+
+    ./build.sh tox py38-custom 3.8.13 -- tests/nocover/test_conjecture_engine.py
+
+See the ``tox`` docs and ``pytest`` docs for more information:
+* https://docs.pytest.org/en/latest/how-to/usage.html
+* https://tox.wiki/en/latest/config.html#cli
+
+^^^^^^^^^^^
+Test Layout
+^^^^^^^^^^^
+
+See ``hypothesis-python/tests/README.rst``
+
+^^^^^^^^^^^^^^^^
+Useful Arguments
+^^^^^^^^^^^^^^^^
+
+Some useful arguments to pytest include:
+
+* You can pass ``-n 0`` to turn off ``pytest-xdist``'s parallel test execution.
+  Sometimes for running just a small number of tests its startup time is longer
+  than the time it saves (this will vary from system to system), so this can
+  be helpful if you find yourself waiting on test runners to start a lot.
+* You can use ``-k`` to select a subset of tests to run. This matches on substrings
+  of the test names. For example ``-kfoo`` will only run tests that have "foo" as
+  a substring of their name. You can also use composite expressions here.
+  e.g. ``-k'foo and not bar'`` will run anything containing foo that doesn't
+  also contain bar.  `More information on how to select tests to run can be found
+  in the pytest documentation <https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`__.
+
+

guides/testing-hypothesis.rst

@@ -37,7 +37,7 @@ a starting point, not the final goal.
   because when it was found and fixed, someone wrote a test to make sure it
   couldn't come back!
 
-The ``tests/`` directory has some notes in the README file on where various
+The ``hypothesis-python/tests/`` directory has some notes in the README file on where various
 kinds of tests can be found or added.  Go there for the practical stuff, or
 just ask one of the maintainers for help on a pull request!
 
@@ -48,122 +48,8 @@ Further reading: How `SQLite is tested <https://sqlite.org/testing.html>`_,
 Dan Luu writes about `fuzz testing <https://danluu.com/testing/>`_ and
 `broken processes <https://danluu.com/wat/>`_, among other things.
 
----------------------------------------
-Setting up a virtualenv to run tests in
----------------------------------------
-
-If you want to run individual tests rather than relying on the make tasks
-(which you probably will), it's easiest to do this in a virtualenv.
-
-The following will give you a working virtualenv for running tests in:
-
-.. code-block:: bash
-
-  pip install virtualenv
-  python -m virtualenv testing-venv
-
-  # On Windows: testing-venv\Scripts\activate
-  source testing-venv/bin/activate
-
-  # Can also use pip install -e .[all] to get
-  # all optional dependencies
-  pip install -e .
-
-  # Test specific dependencies.
-  pip install -r requirements/test.in
-
-Now whenever you want to run tests you can just activate the virtualenv
-using ``source testing-venv/bin/activate`` or ``testing-venv\Scripts\activate``
-and all of the dependencies will be available to you and your local copy
-of Hypothesis will be on the path (so any edits will be picked up automatically
-and you don't need to reinstall it in the local virtualenv).
-
 -------------
 Running Tests
 -------------
 
-In order to run tests outside of the make/tox/etc set up, you'll need an
-environment where Hypothesis is on the path and all of the testing dependencies
-are installed.
-We recommend doing this inside a virtualenv as described in the previous section.
-
-All testing is done using `pytest <https://docs.pytest.org/en/latest/>`_,
-with a couple of plugins installed. For advanced usage we recommend reading the
-pytest documentation, but this section will give you a primer in enough of the
-common commands and arguments to get started.
-
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Selecting Which Files to Run
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The following invocation runs all of the tests in the file
-`tests/cover/test_conjecture_engine.py`:
-
-.. code-block::
-
-    python -m pytest tests/cover/test_conjecture_engine.py
-
-If you want to run multiple files you can pass them all as arguments, and if
-you pass a directory then it will run all files in that directory.
-For example the following runs all the files in `test_conjecture_engine.py`
-and `test_slippage.py`
-
-.. code-block::
-
-    python -m pytest tests/cover/test_conjecture_engine.py tests/cover/test_slippage.py
-
-If you were running this in bash (if you're not sure: if you're not on Windows
-you probably are) you could also use the syntax:
-
-.. code-block::
-
-    python -m pytest tests/cover/test_{conjecture_engine,slippage}.py
-
-And the following would run all tests under `tests/cover`:
-
-.. code-block::
-
-    python -m pytest tests/cover
-
-
-~~~~~~~~~~~
-Test Layout
-~~~~~~~~~~~
-
-The top level structure of the tests in Hypothesis looks as follows:
-
-* ``cover`` contains tests that we measure coverage for. This is intended to
-  be a fairly minimal and fast set of tests that still gives pretty good
-  confidence in the behaviour of the test suite. It is currently failing at
-  both "minimal" and "fast", but we're trying to move it back in that
-  direction. Try not to add tests to this unless they're actually to cover
-  some specific target.
-* ``nocover`` is a general dumping ground for slower tests that aren't needed
-  to achieve coverage.
-* ``quality`` is for expensive tests about the distribution or shrinking of
-  examples. These will only be run on one Python version.
-* The remaining test directories are for testing specific extras modules and
-  should have the same name.
-
-As a rule of thumb when writing new tests, they should go in nocover unless
-they are for a specific extras module or to deliberately target a particular
-line for coverage. In the latter case, prefer fast unit tests over larger and
-slower integration tests (we are not currently very good at this).
-
-
-~~~~~~~~~~~~~~~~
-Useful Arguments
-~~~~~~~~~~~~~~~~
-
-Some useful arguments to pytest include:
-
-* You can pass ``-n 0`` to turn off ``pytest-xdist``'s parallel test execution.
-  Sometimes for running just a small number of tests its startup time is longer
-  than the time it saves (this will vary from system to system), so this can
-  be helpful if you find yourself waiting on test runners to start a lot.
-* You can use ``-k`` to select a subset of tests to run. This matches on substrings
-  of the test names. For example ``-kfoo`` will only run tests that have "foo" as
-  a substring of their name. You can also use composite expressions here.
-  e.g. ``-k'foo and not bar'`` will run anything containing foo that doesn't
-  also contain bar.  `More information on how to select tests to run can be found
-  in the pytest documentation <https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests>`__.
+Tests are run via ``build.sh``. See ``CONTRIBUTING.rst`` for more details.

tooling/src/hypothesistooling/__main__.py

@@ -356,7 +356,7 @@ def documentation():
         )
 
 
-def run_tox(task, version):
+def run_tox(task, version, *args):
     python = install.python_executable(version)
 
     # Create a version of the name that tox will pick up for the correct
@@ -373,7 +373,7 @@ def run_tox(task, version):
     env["PATH"] = os.path.dirname(python) + ":" + env["PATH"]
     print(env["PATH"])
 
-    pip_tool("tox", "-e", task, env=env, cwd=hp.HYPOTHESIS_PYTHON)
+    pip_tool("tox", "-e", task, *args, env=env, cwd=hp.HYPOTHESIS_PYTHON)
 
 
 # See update_python_versions() above
@@ -449,6 +449,14 @@ def check_pypy38():
     run_tox("pypy3-full", PYPY38)
 
 
+@task()
+def tox(*args):
+    if len(args) == 0:
+        print("Usage: ./build.sh tox TOX_ENV PY_VERSION [tox args]")
+        sys.exit(1)
+    run_tox(args[0], args[1], *args[2:])
+
+
 def standard_tox_task(name):
     TASKS["check-" + name] = python_tests(lambda: run_tox(name, PYMAIN))