Angular Devkit Benchmark

This tool provides benchmark information for processes. The tool will gathering metrics from a given command, then average them out over the total runs.

It currently shows only time, process, cpu, and memory used but might be extended in the future.

This tool was created to provide an objective and reproducible way of benchmarking process performance.

Given a process (or its source code), process inputs and environment, keeping two of these elements constant while varying the third should allow for meaningful benchmarks over time.

In the context of the DevKit, we publish many CLI tools and have access to their source code. By tracking tool resource usage we can catch performance regressions or improvements on our CI.

STABILITY AND SUPPORT DISCLAIMER

This package is not currently stable. Usage, output and API may change at any time. Support is not ensured.

Installation

You can install the benchmark tool via npm install -g benchmark for a global install, or without -g for a local install. Installing globally gives you access to the benchmark binary in your PATH.

CLI Usage

Call the benchmark binary, followed by options, then double dash, then the command to benchmark.

For more information on the available options, run benchmark --help:

$ benchmark --help
[benchmark] benchmark [options] -- [command to benchmark]

Collects process stats from running the command.

Options:
    --help                    Show this message.
    (... other available options)

Example:
    benchmark --iterations=3 -- node my-script.js

Example

Given the naive implementation of a fibonacci number calculator below:

// fibonacci.js
const fib = (n) => n > 1 ? fib(n - 1) + fib(n - 2) : n;
console.log(fib(parseInt(process.argv[2])));

Run benchmark -- node fibonacci.js 40 to benchmark calculating the 40th fibonacci number:

$ benchmark -- node fibonacci.js 40
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   node fibonacci.js 40 (at D:\sandbox\latest-project)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 2365.40 ms (2449.00, 2444.00, 2357.00, 2312.00, 2265.00)
[benchmark]   Average Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Peak Process usage: 1.00 process(es) (1.00, 1.00, 1.00, 1.00, 1.00)
[benchmark]   Average CPU usage: 4.72 % (5.03, 4.86, 4.50, 4.50, 4.69)
[benchmark]   Peak CPU usage: 23.40 % (25.00, 23.40, 21.80, 21.80, 25.00)
[benchmark]   Average Memory usage: 22.34 MB (22.32, 22.34, 22.34, 22.35, 22.35)
[benchmark]   Peak Memory usage: 22.34 MB (22.32, 22.34, 22.34, 22.35, 22.35)

Example in watch mode

watch-script accepts a node script when using the CLI. When using the API you can provide a Command instance.

benchmark --verbose --watch-timeout 10000 --watch-matcher "Compiled successfully" --watch-script watch-script.js -- ng serve
[benchmark] Benchmarking process over 5 iterations, with up to 5 retries.
[benchmark]   ng serve (at D:\sandbox\latest-project)
[benchmark] Process Stats
[benchmark]   Elapsed Time: 470.40 ms (820.00, 557.00, 231.00, 509.00, 235.00)
[benchmark]   Average Process usage: 2.00 process(es) (2.00, 2.00, 2.00, 2.00, 2.00)
[benchmark]   Peak Process usage: 2.00 process(es) (2.00, 2.00, 2.00, 2.00, 2.00)
[benchmark]   Average CPU usage: 33.77 % (31.27, 0.00, 101.70, 35.90, 0.00)
[benchmark]   Peak CPU usage: 59.72 % (125.10, 0.00, 101.70, 71.80, 0.00)
[benchmark]   Average Memory usage: 665.49 MB (619.84, 657.17, 669.47, 685.19, 695.76)
[benchmark]   Peak Memory usage: 672.44 MB (639.87, 661.04, 669.47, 689.66, 702.14)

API Usage

You can also use the benchmarking API directly:

import { Command, defaultStatsCapture, runBenchmark } from '@angular-devkit/benchmark';

const command = new Command('node', ['fibonacci.js', '40']);
const captures = [defaultStatsCapture];

runBenchmark({ command, command }).subscribe(results => {
  // results is:[{
  //   "name": "Process Stats",
  //   "metrics": [{
  //     "name": "Elapsed Time", "unit": "ms", "value": 1883.6,
  //     "componentValues": [1733, 1957, 1580, 1763, 2385]
  //   }, {
  //     "name": "Average Process usage", "unit": "process(es)", "value": 1,
  //     "componentValues": [1, 1, 1, 1, 1]
  //   }, {
  //     "name": "Peak Process usage", "unit": "process(es)", "value": 1,
  //     "componentValues": [1, 1, 1, 1, 1]
  //   }, {
  //     "name": "Average CPU usage", "unit": "%", "value": 3.0855555555555556,
  //     "componentValues": [1.9625, 1.9500000000000002, 1.9500000000000002, 4.887499999999999, 4.677777777777778]
  //   }, {
  //     "name": "Peak CPU usage", "unit": "%", "value": 19.380000000000003,
  //     "componentValues": [15.7, 15.6, 15.6, 25, 25]
  //   }, {
  //     "name": "Average Memory usage", "unit": "MB", "value": 22.364057600000002,
  //     "componentValues": [22.383104, 22.332416, 22.401024, 22.355968, 22.347776]
  //   }, {
  //     "name": "Peak Memory usage", "unit": "MB", "value": 22.3649792,
  //     "componentValues": [22.384639999999997, 22.335487999999998, 22.401024, 22.355968, 22.347776]
  //   }]
  // }]
});

A good example of API usage is the main binary itself, found in ./src/main.ts. We recommend using TypeScript to get full access to the interfaces included.