Feature: quick CLIs (skorch-dev#390)

* Helper functions to for CLIs with almost boilerplate.

Add a helper function parse_args that makes it very simple to build
custom CLIs. Add an example for the usage of this and extend docs.

* Extend and adjust README and docs.

* Add CLI implementation and unit tests.

* Update dev requirements: pytest >= 3.4

* Add fire library to dev requirements.

* Remove fire from dev requirements, install in travis.

fire is not on the conda channels, so install would fail. Also, modify
cli tests to be skipped if fire is not installed.

* Correct typos.

* Add the option to have custom defaults.

E.g., if you would like to use batch_size=256 as a default instead of
128, you can now pass a dict `{'batch_size': 256}` to
`parse_args`. This will not only update your model to use those
defaults but also change the help to show your custom defaults.

To achieve the latter effect, it was necessary to parse the sklearn
docstrings for default values and replace them by the new
default. This turned out to be more difficult than expected because
the docstring defaults are not always written in the same fashion. I
tried to catch some variants that I found but there are certainly more
variants out there. It should, however, work fine with the way we
write docstrings in skorch.

* Fix typo in docs/user/helper.rst

Co-Authored-By: benjamin-work <benjamin.bossan@ottogroup.com>

* Update docstring, remove unnecessary try..except.

* Simplify function that matches span for docstring match.

Author: benjamin-work

.travis.yml

@@ -30,6 +30,7 @@ install:
   - source activate skorch-env
   - cat requirements.txt requirements-dev.txt > reqs.txt
   - conda install --file=reqs.txt
+  - pip install fire
   - pip install .
   - conda install -c pytorch pytorch-cpu==${PYTORCH_VERSION}
 script:

CHANGES.md

@@ -22,10 +22,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   a re-initialization of the optimizer (#369)
 - Support for scipy sparse CSR matrices as input (as, e.g., returned by sklearn's
   `CountVectorizer`); note that they are cast to dense matrices during batching
+- Helper functions to build command line interfaces with almost no
+  boilerplate, [example][1811191713] that shows usage
 
 [1810251445]: https://colab.research.google.com/github/dnouri/skorch/blob/master/notebooks/Basic_Usage.ipynb
 [1810261633]: https://colab.research.google.com/github/dnouri/skorch/blob/master/notebooks/Advanced_Usage.ipynb
 [1811011230]: https://colab.research.google.com/github/dnouri/skorch/blob/master/notebooks/MNIST.ipynb
+[1811191713]: https://github.com/dnouri/skorch/tree/master/examples/cli
 
 ### Changed
 

docs/user/helper.rst

@@ -5,6 +5,7 @@ Helper
 This module provides helper functions and classes for the user. They
 make working with skorch easier but are not used by skorch itself.
 
+
 SliceDict
 ---------
 
@@ -16,3 +17,161 @@ length of the arrays and not the number of keys, and you get a
 ``dict``, you would normally not be able to use sklearn
 :class:`~sklearn.model_selection.GridSearchCV` and similar things;
 with :class:`.SliceDict`, this works.
+
+
+Command line interface helpers
+------------------------------
+
+Often you want to wrap up your experiments by writing a small script
+that allows others to reproduce your work. With the help of skorch and
+the fire_ library, it becomes very easy to write command line
+interfaces without boilerplate. All arguments pertaining to skorch or
+its PyTorch module are immediately available as command line
+arguments, without the need to write a custom parser. If docstrings in
+the numpydoc_ specification are available, there is also an
+comprehensive help for the user. Overall, this allows you to make your
+work reproducible without the usual hassle.
+
+There is an example_ in the skorch repository that shows how to use
+the CLI tools. Below is a snippet that shows the output created by the
+help function without writing a single line of argument parsing:
+
+.. code:: bash
+
+    $ python examples/cli/train.py pipeline --help
+
+    <SelectKBest> options:
+       --select__score_func : callable
+         Function taking two arrays X and y, and returning a pair of arrays
+         (scores, pvalues) or a single array with scores.
+         Default is f_classif (see below "See also"). The default function only
+         works with classification tasks.
+       --select__k : int or "all", optional, default=10
+         Number of top features to select.
+         The "all" option bypasses selection, for use in a parameter search.
+
+    ...
+
+    <NeuralNetClassifier> options:
+       --net__module : torch module (class or instance)
+         A PyTorch :class:`~torch.nn.Module`. In general, the
+         uninstantiated class should be passed, although instantiated
+         modules will also work.
+       --net__criterion : torch criterion (class, default=torch.nn.NLLLoss)
+         Negative log likelihood loss. Note that the module should return
+         probabilities, the log is applied during ``get_loss``.
+       --net__optimizer : torch optim (class, default=torch.optim.SGD)
+         The uninitialized optimizer (update rule) used to optimize the
+         module
+       --net__lr : float (default=0.01)
+         Learning rate passed to the optimizer. You may use ``lr`` instead
+         of using ``optimizer__lr``, which would result in the same outcome.
+       --net__max_epochs : int (default=10)
+         The number of epochs to train for each ``fit`` call. Note that you
+         may keyboard-interrupt training at any time.
+       --net__batch_size : int (default=128)
+         ...
+       --net__verbose : int (default=1)
+         Control the verbosity level.
+       --net__device : str, torch.device (default='cpu')
+         The compute device to be used. If set to 'cuda', data in torch
+         tensors will be pushed to cuda tensors before being sent to the
+         module.
+
+    <MLPClassifier> options:
+       --net__module__hidden_units : int (default=10)
+         Number of units in hidden layers.
+       --net__module__num_hidden : int (default=1)
+         Number of hidden layers.
+       --net__module__nonlin : torch.nn.Module instance (default=torch.nn.ReLU())
+         Non-linearity to apply after hidden layers.
+       --net__module__dropout : float (default=0)
+         Dropout rate. Dropout is applied between layers.
+
+Installation
+^^^^^^^^^^^^
+
+To use this functionality, you need some further libraries that are not
+part of skorch, namely fire_ and numpydoc_. You can install them
+thusly:
+
+
+.. code:: bash
+
+    pip install fire numpydoc
+
+Usage
+^^^^^
+
+When you write your own script, only the following bits need to be
+added:
+
+.. code:: python
+
+    import fire
+    from skorch.helper import parse_args
+
+    # your model definition and data fetching code below
+    ...
+
+    def main(**kwargs):
+        X, y = get_data()
+        my_model = get_model()
+
+        # important: wrap the model with the parsed arguments
+        parsed = parse_args(kwargs)
+        my_model = parsed(my_model)
+
+        my_model.fit(X, y)
+
+
+    if __name__ == '__main__':
+        fire.Fire(main)
+
+
+This even works if your neural net is part of an sklearn pipeline, in
+which case the help extends to all other estimators of your pipeline.
+
+In case you would like to change some defaults for the net (e.g. using
+a ``batch_size`` of 256 instead of 128), this is also possible. You
+should have a dictionary containing your new defaults and pass it as
+an additional argument to ``parse_args``:
+
+.. code:: python
+
+    my_defaults = {'batch_size': 128, 'module__hidden_units': 30}
+
+    def main(**kwargs):
+        ...
+        parsed = parse_args(kwargs, defaults=my_defaults)
+        my_model = parsed(my_model)
+
+
+This will update the displayed help to your new defaults, as well as
+set the parameters on the net or pipeline for you. However, the
+arguments passed via the commandline have precedence. Thus, if you
+additionally pass ``--batch_size 512`` to the script, batch size will
+be 512.
+
+Restrictions
+^^^^^^^^^^^^
+
+Almost all arguments should work out of the box. Therefore, you get
+command line arguments for the number of epochs, learning rate, batch
+size, etc. for free. Moreover, you can access the module parameters
+with the double-underscore notation as usual with skorch
+(e.g. ``--module__num_units 100``). This should cover almost all
+common cases.
+
+Parsing command line arguments that are non-primitive Python objects
+is more difficult, though. skorch's custom parsing should support
+normal Python types and simple custom objects, e.g. this works:
+``--module__nonlin 'torch.nn.RReLU(0.1, upper=0.4)'``. More complex
+parsing might not work. E.g., it is currently not possible to add new
+callbacks through the command line (but you can modify existing ones
+as usual).
+
+
+.. _fire: https://github.com/google/python-fire
+.. _numpydoc: https://github.com/numpy/numpydoc
+.. _example: https://github.com/dnouri/skorch/tree/master/examples/cli

examples/cli/README.md

@@ -0,0 +1,144 @@
+# skorch helpers for command line interfaces (CLIs)
+
+Often you want to wrap up your experiments by writing a small script
+that allows others to reproduce your work. With the help of skorch and
+the fire library, it becomes very easy to write command line
+interfaces without boilerplate. All arguments pertaining to skorch or
+its PyTorch module are immediately available as command line
+arguments, without the need to write a custom parser. If docstrings in
+the numpydoc specification are available, there is also an
+comprehensive help for the user. Overall, this allows you to make your
+work reproducible without the usual hassle.
+
+This example is a showcase of how easy CLIs become with skorch.
+
+## Installation
+
+To use this functionaliy, you need some further libraries that are not
+part of skorch, namely fire and numpydoc. You can install them thusly:
+
+```bash
+pip install fire numpydoc
+```
+
+## Usage
+
+The `train.py` file contains an example of how to write your own CLI
+with the help of skorch. As you can see, this file almost exclusively
+consists of the proper logic, there is no argument parsing
+involved.
+
+When you write your own script, only the following bits need to be
+added:
+
+```python
+
+import fire
+from skorch.helper import parse_args
+
+# your model definition and data fetching code below
+...
+
+def main(**kwargs):
+    X, y = get_data()
+    my_model = get_model()
+
+    # important: wrap the model with the parsed arguments
+    parsed = parse_args(kwargs)
+    my_model = parsed(my_model)
+
+    my_model.fit(X, y)
+
+
+if __name__ == '__main__':
+    fire.Fire(main)
+
+```
+
+This even works if your neural net is part of an sklearn pipeline, in
+which case the help extends to all other estimators of your pipeline.
+
+In case you would like to change some defaults for the net (e.g. using
+a `batch_size` of 256 instead of 128), this is also possible. You
+should have a dictionary containing your new defaults and pass it as
+an additional argument to `parse_args`:
+
+```python
+
+my_defaults = {'batch_size': 128, 'module__hidden_units': 30}
+
+def main(**kwargs):
+    ...
+    parsed = parse_args(kwargs, defaults=my_defaults)
+    my_model = parsed(my_model)
+
+```
+
+This will update the displayed help to your new defaults, as well as
+set the parameters on the net or pipeline for you. However, the
+arguments passed via the commandline have precedence. Thus, if you
+additionally pass ``--batch_size 512`` to the script, batch size will
+be 512.
+
+For more information on how to use fire, follow [this
+link](https://github.com/google/python-fire).
+
+## Restrictions
+
+Almost all arguments should work out of the box. Therefore, you get
+command line arguments for the number of epochs, learning rate, batch
+size, etc. for free. Moreover, you can access the module parameters
+with the double-underscore notation as usual with skorch
+(e.g. `--module__num_units 100`). This should cover almost all common
+cases.
+
+Parsing command line arguments that are non-primitive Python objects
+is more difficult, though. skorch's custom parsing should support
+normal Python types and simple custom objects, e.g. this works:
+`--module__nonlin 'torch.nn.RReLU(0.1, upper=0.4)'`. More complex
+parsing might not work. E.g., it is currently not possible to add new
+callbacks through the command line (but you can modify existing ones
+as usual).
+
+## Running the script
+
+### Getting Help
+
+In this example, there are two variants, only the net ("net") and the
+net within an sklearn pipeline ("pipeline"). To get general help for
+each, run:
+
+```bash
+python train.py net -- --help
+python train.py pipeline -- --help
+```
+
+To get help for model-specific parameters, run:
+
+```bash
+python train.py net --help
+python train.py pipeline --help
+```
+
+### Training a Model
+
+Run
+
+```bash
+python train.py net  # only the net
+python train.py pipeline  # net with pipeline
+```
+
+with the defaults.
+
+Example with just the net and some non-defaults:
+
+```bash
+python train.py net --n_samples 1000 --output_file 'model.pkl' --lr 0.1 --max_epochs 5 --device 'cuda' --module__hidden_units 50 --module__nonlin 'torch.nn.RReLU(0.1, upper=0.4)' --callbacks__valid_acc__on_train --callbacks__valid_acc__name train_acc
+```
+
+Example with an sklearn pipeline:
+
+```bash
+python train.py pipeline --n_samples 1000 --net__lr 0.1 --net__module__nonlin 'torch.nn.LeakyReLU()' --scale__minmax__feature_range '(-2, 2)' --scale__normalize__norm l1
+```

examples/cli/train.py

@@ -0,0 +1,207 @@
+"""Simple training script for a MLP classifier.
+
+See accompanying README.md for more details.
+
+"""
+
+import pickle
+
+import fire
+import numpy as np
+from sklearn.datasets import make_classification
+from sklearn.feature_selection import SelectKBest
+from sklearn.pipeline import FeatureUnion
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import MinMaxScaler
+from sklearn.preprocessing import Normalizer
+from skorch import NeuralNetClassifier
+import torch
+from torch import nn
+
+from skorch.helper import parse_args
+
+
+np.random.seed(0)
+torch.manual_seed(0)
+torch.cuda.manual_seed(0)
+
+
+# number of input features
+N_FEATURES = 20
+
+# number of classes
+N_CLASSES = 2
+
+# custom defaults for net
+DEFAULTS_NET = {
+    'batch_size': 256,
+    'module__hidden_units': 30,
+}
+
+# custom defaults for pipeline
+DEFAULTS_PIPE = {
+    'scale__minmax__feature_range': (-1, 1),
+    'net__batch_size': 256,
+    'net__module__hidden_units': 30,
+}
+
+
+class MLPClassifier(nn.Module):
+    """A simple multi-layer perceptron module.
+
+    This can be adapted for usage in different contexts, e.g. binary
+    and multi-class classification, regression, etc.
+
+    Note: This docstring is used to create the help for the CLI.
+
+    Parameters
+    ----------
+    hidden_units : int (default=10)
+      Number of units in hidden layers.
+
+    num_hidden : int (default=1)
+      Number of hidden layers.
+
+    nonlin : torch.nn.Module instance (default=torch.nn.ReLU())
+      Non-linearity to apply after hidden layers.
+
+    dropout : float (default=0)
+      Dropout rate. Dropout is applied between layers.
+
+    """
+    def __init__(
+            self,
+            hidden_units=10,
+            num_hidden=1,
+            nonlin=nn.ReLU(),
+            dropout=0,
+    ):
+        super().__init__()
+        self.hidden_units = hidden_units
+        self.num_hidden = num_hidden
+        self.nonlin = nonlin
+        self.dropout = dropout
+
+        self.reset_params()
+
+    def reset_params(self):
+        """(Re)set all parameters."""
+        units = [N_FEATURES]
+        units += [self.hidden_units] * self.num_hidden
+        units += [N_CLASSES]
+
+        sequence = []
+        for u0, u1 in zip(units, units[1:]):
+            sequence.append(nn.Linear(u0, u1))
+            sequence.append(self.nonlin)
+            sequence.append(nn.Dropout(self.dropout))
+
+        sequence = sequence[:-2]
+        self.sequential = nn.Sequential(*sequence)
+
+    def forward(self, X):
+        return nn.Softmax(dim=-1)(self.sequential(X))
+
+
+def get_data(n_samples=100):
+    """Get synthetic classification data with n_samples samples."""
+    X, y = make_classification(
+        n_samples=n_samples,
+        n_features=N_FEATURES,
+        n_classes=N_CLASSES,
+        random_state=0,
+    )
+    X = X.astype(np.float32)
+    return X, y
+
+
+def get_model(with_pipeline=False):
+    """Get a multi-layer perceptron model.
+
+    Optionally, put it in a pipeline that scales the data.
+
+    """
+    model = NeuralNetClassifier(MLPClassifier)
+    if with_pipeline:
+        model = Pipeline([
+            ('scale', FeatureUnion([
+                ('minmax', MinMaxScaler()),
+                ('normalize', Normalizer()),
+            ])),
+            ('select', SelectKBest(k=N_FEATURES)),  # keep input size constant
+            ('net', model),
+        ])
+    return model
+
+
+def save_model(model, output_file):
+    """Save model to output_file, if given"""
+    if not output_file:
+        return
+
+    with open(output_file, 'wb') as f:
+        pickle.dump(model, f)
+    print("Saved model to file '{}'.".format(output_file))
+
+
+def net(n_samples=100, output_file=None, **kwargs):
+    """Train an MLP classifier on synthetic data.
+
+    n_samples : int (default=100)
+      Number of training samples
+
+    output_file : str (default=None)
+      If not None, file name used to save the model.
+
+    kwargs : dict
+      Additional model parameters.
+
+    """
+
+    model = get_model(with_pipeline=False)
+    # important: wrap the model with the parsed arguments
+    parsed = parse_args(kwargs, defaults=DEFAULTS_NET)
+    model = parsed(model)
+
+    X, y = get_data(n_samples=n_samples)
+    print("Training MLP classifier")
+    model.fit(X, y)
+
+    save_model(model, output_file)
+
+
+def pipeline(n_samples=100, output_file=None, **kwargs):
+    """Train an MLP classifier in a pipeline on synthetic data.
+
+    The pipeline scales the input data before passing it to the net.
+
+    Note: This docstring is used to create the help for the CLI.
+
+    Parameters
+    ----------
+    n_samples : int (default=100)
+      Number of training samples
+
+    output_file : str (default=None)
+      If not None, file name used to save the model.
+
+    kwargs : dict
+      Additional model parameters.
+
+    """
+
+    model = get_model(with_pipeline=True)
+    # important: wrap the model with the parsed arguments
+    parsed = parse_args(kwargs, defaults=DEFAULTS_PIPE)
+    model = parsed(model)
+
+    X, y = get_data(n_samples=n_samples)
+    print("Training MLP classifier in a pipeline")
+    model.fit(X, y)
+
+    save_model(model, output_file)
+
+
+if __name__ == '__main__':
+    # register 2 functions, "net" and "pipeline"
+    fire.Fire({'net': net, 'pipeline': pipeline})

requirements-dev.txt

@@ -1,11 +1,11 @@
+flaky
 jupyter
 matplotlib>=2.0.2
 numpydoc
 openpyxl
 pandas
 pylint
-pytest
+pytest>=3.4
 pytest-cov
 sphinx
 sphinx_rtd_theme
-flaky

skorch/cli.py

@@ -0,0 +1,336 @@
+"""Helper functions for quick command line interfaces with skorch and
+fire.
+
+"""
+
+from functools import partial
+from importlib import import_module
+from itertools import chain
+import re
+import shlex
+import sys
+
+from sklearn.base import BaseEstimator
+from sklearn.pipeline import FeatureUnion
+from sklearn.pipeline import Pipeline
+
+try:
+    from fire.parser import DefaultParseValue
+except ImportError:
+    raise ImportError("Using skorch cli helpers requires the fire library,"
+                      " you can install it with pip: pip install fire.")
+
+try:
+    from numpydoc.docscrape import ClassDoc
+except ImportError:
+    raise ImportError("Using skorch cli helpers requires the numpydoc library,"
+                      " you can install it with pip: pip install numpydoc.")
+
+
+__all__ = ['parse_args']
+
+
+# matches: bar(), foo.bar(), foo.bar(baz)
+P_PARAMS = re.compile(r"(?P<name>^[a-zA-Z][a-zA-Z0-9_\.]*)(?P<params>\(.*\)$)")
+
+P_DEFAULTS = re.compile(
+    # standard, matches: int (default=123)
+    r"(.+\s\(default\s?\=\s?(?P<default>.+)\)$)|"
+    # no parens, matches: int, default=123
+    r"(.+\sdefault\s?\=\s?(?P<default_np>.+)$)|"
+    # no equal, matches: int, default 123
+    r"(.+default\s(?P<default_ne>.+))|"
+    # 'by-default', matches: str (l2 by default)
+    r"[^\(]+\((?P<default_bd>[^\"\']+)(\sby\sdefault\)?)|"
+    # 'by-default-double-tick', matches: "l1" or "l2" ("l2" by default)
+    r"[^\(]+\(\"(?P<default_bd_dt>.+)\"\sby\sdefault\)?|"
+    # 'by-default-single-tick', matches: 'l1' or 'l2' ('l2' by default)
+    r"[^\(]+\(\'(?P<default_bd_st>.+)\'\sby\sdefault\)?"
+)
+
+
+def _param_split(params):
+    return (p.strip(' ,') for p in shlex.split(params))
+
+
+def _get_span(s, pattern):
+    """Return the span of the first group that matches the pattern."""
+    i, j = -1, -1
+
+    match = pattern.match(s)
+    if not match:
+        return i, j
+
+    for group_name in pattern.groupindex:
+        i, j = match.span(group_name)
+        if (i, j) != (-1, -1):
+            return i, j
+
+    return i, j
+
+
+def _substitute_default(s, new_value):
+    """Replaces the default value in a parameter docstring by a new value.
+
+    The docstring must conform to the numpydoc style and have the form
+    "something (keyname=<value-to-replace>)"
+
+    If no matching pattern is found or ``new_value`` is None, return
+    the input untouched.
+
+    Examples
+    --------
+    >>> _replace_default('int (default=128)', 256)
+    'int (default=256)'
+    >>> _replace_default('nonlin (default = ReLU())', nn.Hardtanh(1, 2))
+    'nonlin (default = Hardtanh(min_val=1, max_val=2))'
+
+    """
+    if new_value is None:
+        return s
+
+    # BB: ideally, I would like to replace the 'default*' group
+    # directly but I haven't found a way to do this
+    i, j = _get_span(s, pattern=P_DEFAULTS)
+    if (i, j) == (-1, -1):
+        return s
+    return '{}{}{}'.format(s[:i], new_value, s[j:])
+
+
+def _parse_args_kwargs(params):
+    args = ()
+    kwargs = {}
+    for param in _param_split(params):
+        if '=' not in param:
+            args += (DefaultParseValue(param),)
+        else:
+            k, v = param.split('=')
+            kwargs[k.strip()] = DefaultParseValue(v)
+    return args, kwargs
+
+
+def _resolve_dotted_name(dotted_name):
+    """Returns objects from strings
+
+    Deals e.g. with 'torch.nn.Softmax(dim=-1)'.
+
+    Modified from palladium:
+
+    https://github.com/ottogroup/palladium/blob/8a066a9a7690557d9b1b6ed54b7d1a1502ba59e3/palladium/util.py
+
+    with added support for instantiated objects.
+
+    """
+    if not isinstance(dotted_name, str):
+        return dotted_name
+
+    if '.' not in dotted_name:
+        return dotted_name
+
+    args = None
+    params = None
+    match = P_PARAMS.match(dotted_name)
+    if match:
+        dotted_name = match.group('name')
+        params = match.group('params')
+
+    module, name = dotted_name.rsplit('.', 1)
+    attr = import_module(module)
+    attr = getattr(attr, name)
+
+    if params:
+        args, kwargs = _parse_args_kwargs(params[1:-1])
+        attr = attr(*args, **kwargs)
+
+    return attr
+
+
+def parse_net_kwargs(kwargs):
+    """Parse arguments for the estimator.
+
+    Resolves dotted names and instantiated classes.
+
+    Examples
+    --------
+    >>> kwargs = {'lr': 0.1, 'module__nonlin': 'torch.nn.Hardtanh(-2, max_val=3)'}
+    >>> parse_net_kwargs(kwargs)
+    {'lr': 0.1, 'module__nonlin': Hardtanh(min_val=-2, max_val=3)}
+
+    """
+    if not kwargs:
+        return kwargs
+
+    resolved = {}
+    for k, v in kwargs.items():
+        resolved[k] = _resolve_dotted_name(v)
+
+    return resolved
+
+
+def _yield_preproc_steps(model):
+    if not isinstance(model, Pipeline):
+        return
+
+    for key, val in model.get_params().items():
+        if isinstance(val, BaseEstimator):
+            if not isinstance(val, (Pipeline, FeatureUnion)):
+                yield key, val
+
+
+def _yield_estimators(model):
+    """Yield estimator and its prefix from the model.
+
+    First, pipeline preprocessing steps are yielded (if there are
+    any). Next the neural net is yielded. Finally, the module is
+    yielded.
+
+    """
+    yield from _yield_preproc_steps(model)
+
+    net_prefixes = []
+    module_prefixes = []
+
+    if isinstance(model, Pipeline):
+        name = model.steps[-1][0]
+        net_prefixes.append(name)
+        module_prefixes.append(name)
+        net = model.steps[-1][1]
+    else:
+        net = model
+
+    yield '__'.join(net_prefixes), net
+
+    module = net.module
+    module_prefixes.append('module')
+    yield '__'.join(module_prefixes), module
+
+
+def _extract_estimator_cls(estimator):
+    if isinstance(estimator, partial):
+        # is partialled
+        return estimator.func
+    if not isinstance(estimator, type):
+        # is instance
+        return estimator.__class__
+    return estimator
+
+
+def _yield_printable_params(param, prefix, defaults):
+    name, default, descr = param
+    name = name if not prefix else '__'.join((prefix, name))
+    default = _substitute_default(default, defaults.get(name))
+
+    printable = '--{} : {}'.format(name, default)
+    yield printable
+
+    for line in descr:
+        yield line
+
+
+def _get_help_for_params(params, prefix='--', defaults=None, indent=2):
+    defaults = defaults or {}
+    for param in params:
+        first, *rest = tuple(_yield_printable_params(
+            param, prefix=prefix, defaults=defaults))
+        yield " " * indent + first
+        for line in rest:
+            yield " " * 2 * indent + line
+
+
+def _get_help_for_estimator(prefix, estimator, defaults=None):
+    """Yield help lines for the given estimator and prefix."""
+    defaults = defaults or {}
+    estimator = _extract_estimator_cls(estimator)
+    yield "<{}> options:".format(estimator.__name__)
+
+    doc = ClassDoc(estimator)
+    yield from _get_help_for_params(
+        doc['Parameters'],
+        prefix=prefix,
+        defaults=defaults,
+    )
+    yield ''  # add a newline line between estimators
+
+
+def print_help(model, defaults=None):
+    """Print help for the command line arguments of the given model.
+
+    Parameters
+    ----------
+    model : sklearn.base.BaseEstimator
+      The basic model, e.g. a ``NeuralNet`` or sklearn ``Pipeline``.
+
+    defautls : dict or None (default=None)
+      Optionally, change the default values to use custom
+      defaults. Commandline arguments have precedence over defaults.
+
+    """
+    defaults = defaults or {}
+
+    print("This is the help for the model-specific parameters.")
+    print("To invoke help for the remaining options, run:")
+    print("python {} -- --help".format(sys.argv[0]))
+    print()
+
+    lines = (_get_help_for_estimator(prefix, estimator, defaults=defaults) for
+             prefix, estimator in _yield_estimators(model))
+    print('\n'.join(chain(*lines)))
+
+
+def parse_args(kwargs, defaults=None):
+    """Apply command line arguments or show help.
+
+    Use this in conjunction with the fire library to quickly build
+    command line interfaces for your scripts.
+
+    This function returns another function that must be called with
+    the estimator (e.g. ``NeuralNet``) to apply the parsed command
+    line arguments. If the --help option is found, show the
+    estimator-specific help instead.
+
+    Examples
+    --------
+    Content of my_script.py:
+
+    >>> def main(**kwargs):
+    >>>     X, y = get_data()
+    >>>     my_model = get_model()
+    >>>     parsed = parse_args(kwargs)
+    >>>     my_model = parsed(my_model)
+    >>>     my_model.fit(X, y)
+    >>>
+    >>> if __name__ == '__main__':
+    >>>     fire.Fire(main)
+
+    Parameters
+    ----------
+    kwargs : dict
+      The arguments as parsed by fire.
+
+    defautls : dict or None (default=None)
+      Optionally, change the default values to use custom
+      defaults. Commandline arguments have precedence over defaults.
+
+    Returns
+    -------
+    print_help_and_exit : callable
+      If --help is in the arguments, print help and exit.
+
+    set_params : callable
+      If --help is not in the options, apply command line arguments to
+      the estimator and return it.
+
+    """
+    defaults = defaults or {}
+
+    def print_help_and_exit(estimator):
+        print_help(estimator, defaults=defaults)
+        sys.exit()
+
+    def set_params(estimator):
+        estimator.set_params(**defaults)
+        return estimator.set_params(**parse_net_kwargs(kwargs))
+
+    if kwargs.get('help'):
+        return print_help_and_exit
+    return set_params

skorch/helper.py

@@ -10,6 +10,7 @@
 
 from skorch.utils import _make_split
 from skorch.utils import _make_optimizer
+from skorch.cli import parse_args
 from skorch.utils import is_torch_data_type
 
 

skorch/tests/test_cli.py

@@ -0,0 +1,321 @@
+"""Test for cli.py"""
+
+from math import cos
+import os
+import subprocess
+from unittest.mock import Mock
+from unittest.mock import patch
+
+import numpy as np
+import pytest
+from sklearn.pipeline import FeatureUnion
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import MinMaxScaler
+from torch import nn
+from torch.nn import RReLU
+
+
+fire_installed = True
+try:
+    import fire
+except ImportError:
+    fire_installed = False
+
+
+@pytest.mark.skipif(not fire_installed, reason='fire libarary not installed')
+class TestCli:
+    @pytest.fixture
+    def resolve_dotted_name(self):
+        from skorch.cli import _resolve_dotted_name
+        return _resolve_dotted_name
+
+    @pytest.mark.parametrize('name, expected', [
+        (0, 0),
+        (1.23, 1.23),
+        ('foo', 'foo'),
+        ('math.cos', cos),
+        ('torch.nn', nn),
+        ('torch.nn.ReLU', nn.ReLU),
+    ])
+    def test_resolve_dotted_name(self, resolve_dotted_name, name, expected):
+        result = resolve_dotted_name(name)
+        assert result == expected
+
+    def test_resolve_dotted_name_instantiated(self, resolve_dotted_name):
+        result = resolve_dotted_name('torch.nn.RReLU(0.123, upper=0.456)')
+        assert isinstance(result, RReLU)
+        assert np.isclose(result.lower, 0.123)
+        assert np.isclose(result.upper, 0.456)
+
+    @pytest.fixture
+    def parse_net_kwargs(self):
+        from skorch.cli import parse_net_kwargs
+        return parse_net_kwargs
+
+    def test_parse_net_kwargs(self, parse_net_kwargs):
+        kwargs = {
+            'lr': 0.05,
+            'max_epochs': 5,
+            'module__num_units': 10,
+            'module__nonlin': 'torch.nn.RReLU(0.123, upper=0.456)',
+        }
+        parsed_kwargs = parse_net_kwargs(kwargs)
+
+        assert len(parsed_kwargs) == 4
+        assert np.isclose(parsed_kwargs['lr'], 0.05)
+        assert parsed_kwargs['max_epochs'] == 5
+        assert parsed_kwargs['module__num_units'] == 10
+        assert isinstance(parsed_kwargs['module__nonlin'], RReLU)
+        assert np.isclose(parsed_kwargs['module__nonlin'].lower, 0.123)
+        assert np.isclose(parsed_kwargs['module__nonlin'].upper, 0.456)
+
+    @pytest.fixture
+    def net_cls(self):
+        from skorch import NeuralNetClassifier
+        return NeuralNetClassifier
+
+    @pytest.fixture
+    def net(self, net_cls, classifier_module):
+        return net_cls(classifier_module)
+
+    @pytest.fixture
+    def pipe(self, net):
+        return Pipeline([
+            ('features', FeatureUnion([
+                ('scale', MinMaxScaler()),
+            ])),
+            ('net', net),
+        ])
+
+    @pytest.fixture
+    def yield_estimators(self):
+        from skorch.cli import _yield_estimators
+        return _yield_estimators
+
+    def test_yield_estimators_net(self, yield_estimators, net):
+        result = list(yield_estimators(net))
+
+        assert result[0][0] == ''
+        assert result[0][1] is net
+        assert result[1][0] == 'module'
+        assert result[1][1] is net.module
+
+    def test_yield_estimators_pipe(self, yield_estimators, pipe):
+        result = list(yield_estimators(pipe))
+        scaler = pipe.named_steps['features'].transformer_list[0][1]
+        net = pipe.named_steps['net']
+        module = net.module
+
+        assert result[0][0] == 'features__scale'
+        assert result[0][1] is scaler
+        assert result[1][0] == 'net'
+        assert result[1][1] is net
+        assert result[2][0] == 'net__module'
+        assert result[2][1] is module
+
+    @pytest.fixture
+    def substitute_default(self):
+        from skorch.cli import _substitute_default
+        return _substitute_default
+
+    @pytest.mark.parametrize('s, new_value, expected', [
+        ('', '', ''),
+        ('', 'foo', ''),
+        ('bar', 'foo', 'bar'),
+        ('int (default=128)', '', 'int (default=)'),
+        ('int (default=128)', None, 'int (default=128)'),
+        ('int (default=128)', '""', 'int (default="")'),
+        ('int (default=128)', '128', 'int (default=128)'),
+        ('int (default=128)', '256', 'int (default=256)'),
+        ('int (default=128)', 256, 'int (default=256)'),
+        ('with_parens (default=(1, 2))', (3, 4), 'with_parens (default=(3, 4))'),
+        ('int (default =128)', '256', 'int (default =256)'),
+        ('int (default= 128)', '256', 'int (default= 256)'),
+        ('int (default = 128)', '256', 'int (default = 256)'),
+        (
+            'nonlin (default = ReLU())',
+            nn.Hardtanh(1, 2),
+            'nonlin (default = {})'.format(nn.Hardtanh(1, 2))
+        ),
+        (
+            # from sklearn MinMaxScaler
+            'tuple (min, max), default=(0, 1)',
+            (-1, 1),
+            'tuple (min, max), default=(-1, 1)'
+        ),
+        (
+            # from sklearn MinMaxScaler
+            'boolean, optional, default True',
+            False,
+            'boolean, optional, default False'
+        ),
+        (
+            # from sklearn Normalizer
+            "'l1', 'l2', or 'max', optional ('l2' by default)",
+            'l1',
+            "'l1', 'l2', or 'max', optional ('l1' by default)"
+        ),
+        (
+            # same but double ticks
+            '"l1", "l2", or "max", optional ("l2" by default)',
+            'l1',
+            '"l1", "l2", or "max", optional ("l1" by default)'
+        ),
+        (
+            # same but no ticks
+            "l1, l2, or max, optional (l2 by default)",
+            'l1',
+            "l1, l2, or max, optional (l1 by default)"
+        ),
+        (
+            "tuple, optional ((1, 1) by default)",
+            (2, 2),
+            "tuple, optional ((2, 2) by default)"
+        ),
+        (
+            "nonlin (ReLU() by default)",
+            nn.Tanh(),
+            "nonlin (Tanh() by default)"
+        ),
+    ])
+    def test_replace_default(self, substitute_default, s, new_value, expected):
+        result = substitute_default(s, new_value)
+        assert result == expected
+
+    @pytest.fixture
+    def print_help(self):
+        from skorch.cli import print_help
+        return print_help
+
+    def test_print_help_net(self, print_help, net, capsys):
+        print_help(net)
+        out = capsys.readouterr()[0]
+
+        expected_snippets = [
+            '-- --help',
+            '<NeuralNetClassifier> options',
+            '--module : torch module (class or instance)',
+            '--batch_size : int (default=128)',
+            '<MLPModule> options',
+            '--module__hidden_units : int (default=10)'
+        ]
+        for snippet in expected_snippets:
+            assert snippet in out
+
+    def test_print_help_net_custom_defaults(self, print_help, net, capsys):
+        defaults = {'batch_size': 256, 'module__hidden_units': 55}
+        print_help(net, defaults)
+        out = capsys.readouterr()[0]
+
+        expected_snippets = [
+            '-- --help',
+            '<NeuralNetClassifier> options',
+            '--module : torch module (class or instance)',
+            '--batch_size : int (default=256)',
+            '<MLPModule> options',
+            '--module__hidden_units : int (default=55)'
+        ]
+        for snippet in expected_snippets:
+            assert snippet in out
+
+    def test_print_help_pipeline(self, print_help, pipe, capsys):
+        print_help(pipe)
+        out = capsys.readouterr()[0]
+
+        expected_snippets = [
+            '-- --help',
+            '<MinMaxScaler> options',
+            '--features__scale__feature_range',
+            '<NeuralNetClassifier> options',
+            '--net__module : torch module (class or instance)',
+            '--net__batch_size : int (default=128)',
+            '<MLPModule> options',
+            '--net__module__hidden_units : int (default=10)'
+        ]
+        for snippet in expected_snippets:
+            assert snippet in out
+
+    def test_print_help_pipeline_custom_defaults(
+            self, print_help, pipe, capsys):
+        defaults = {'net__batch_size': 256, 'net__module__hidden_units': 55}
+        print_help(pipe, defaults=defaults)
+        out = capsys.readouterr()[0]
+
+        expected_snippets = [
+            '-- --help',
+            '<MinMaxScaler> options',
+            '--features__scale__feature_range',
+            '<NeuralNetClassifier> options',
+            '--net__module : torch module (class or instance)',
+            '--net__batch_size : int (default=256)',
+            '<MLPModule> options',
+            '--net__module__hidden_units : int (default=55)'
+        ]
+        for snippet in expected_snippets:
+            assert snippet in out
+
+    @pytest.fixture
+    def parse_args(self):
+        from skorch.cli import parse_args
+        return parse_args
+
+    @pytest.fixture
+    def estimator(self, net_cls):
+        mock = Mock(net_cls)
+        return mock
+
+    def test_parse_args_help(self, parse_args, estimator):
+        with patch('skorch.cli.sys.exit') as exit:
+            with patch('skorch.cli.print_help') as help:
+                parsed = parse_args({'help': True, 'foo': 'bar'})
+                parsed(estimator)
+
+        assert estimator.set_params.call_count == 0  # kwargs and defaults
+        assert help.call_count == 1
+        assert exit.call_count == 1
+
+    def test_parse_args_run(self, parse_args, estimator):
+        kwargs = {'foo': 'bar', 'baz': 'math.cos'}
+        with patch('skorch.cli.sys.exit') as exit:
+            with patch('skorch.cli.print_help') as help:
+                parsed = parse_args(kwargs)
+                parsed(estimator)
+
+        assert estimator.set_params.call_count == 2  # defaults and kwargs
+
+        defaults_set_params = estimator.set_params.call_args_list[0][1]
+        assert not defaults_set_params  # no defaults specified
+
+        kwargs_set_params = estimator.set_params.call_args_list[1][1]
+        assert kwargs_set_params['foo'] == 'bar'
+        assert kwargs_set_params['baz'] == cos
+
+        assert help.call_count == 0
+        assert exit.call_count == 0
+
+    def test_parse_args_net_custom_defaults(self, parse_args, net):
+        defaults = {'batch_size': 256, 'module__hidden_units': 55}
+        kwargs = {'batch_size': 123, 'module__nonlin': nn.Hardtanh(1, 2)}
+        parsed = parse_args(kwargs, defaults)
+        net = parsed(net)
+
+        # cmd line args have precedence over defaults
+        assert net.batch_size == 123
+        assert net.module_.hidden_units == 55
+        assert isinstance(net.module_.nonlin, nn.Hardtanh)
+        assert net.module_.nonlin.min_val == 1
+        assert net.module_.nonlin.max_val == 2
+
+    def test_parse_args_pipe_custom_defaults(self, parse_args, pipe):
+        defaults = {'net__batch_size': 256, 'net__module__hidden_units': 55}
+        kwargs = {'net__batch_size': 123, 'net__module__nonlin': nn.Hardtanh(1, 2)}
+        parsed = parse_args(kwargs, defaults)
+        pipe = parsed(pipe)
+        net = pipe.steps[-1][1]
+
+        # cmd line args have precedence over defaults
+        assert net.batch_size == 123
+        assert net.module_.hidden_units == 55
+        assert isinstance(net.module_.nonlin, nn.Hardtanh)
+        assert net.module_.nonlin.min_val == 1
+        assert net.module_.nonlin.max_val == 2