Testing rules¶

Testing your ick rules is a great way to make sure they work!

There are a few ways to test your rules:

Write automated tests alongside your rules. You can these with the ick test-rules command. This page explains how to set up and run these tests.
Run your rule manually on samples of code you want to modify. See the next section for details of manual testing.

Ick rules test directory structure¶

Where to place your tests¶

To test your ick rules, create a tests/ directory next to your rule file, and then add a directory that matches the rule name. In there, each test case should receive its own directory, with the directory name being the name of the test.

Subdirectories named `tests` are reserved for test fixtures and are never
scanned for rule definitions. This means you cannot place an `ick.toml` (or
`pyproject.toml`) inside a `tests/` directory and expect it to register rules —
any such file will be silently ignored by rule discovery.

Structure of a single test¶

Each test for an ick rule consists of two directories:

input/: Contains the initial state of files before the rule runs
output/: Contains the expected state of files after the rule runs

Test for an expected exception using `error.txt`¶

If you want an error/exception to occur during your test, put the exception verbatim in output/error.txt.

Test for an expected stdout using `output.txt`¶

If you expect your rule to exit with return code 99 and write to stdout (for example, “Work is still needed on file …”), put the expected stdout verbatim in output/output.txt.

Test structure visualized¶

For two given ick rules, which we’ve creatively named rule1 and rule2, the following file structure will add tests called test_rule1, test_work_needed, and test_error to rule1 and test_rule2 to rule2. These will be invoked when you run ick test-rules.

.
|-- ick.toml
|-- some_dir
|   |-- ick.toml
|   |-- rule1.py
|   |-- rule2.py
|   |-- tests
|   |   |-- rule1
|   |   |   |-- test_rule1
|   |   |   |   |-- input
|   |   |   |   |   |-- foo.bar
|   |   |   |   |-- output
|   |   |   |   |   |-- foo.bar
|   |   |   |-- test_work_needed
|   |   |   |   |-- input
|   |   |   |   |   |-- foo.bar
|   |   |   |   |-- output
|   |   |   |   |   |-- foo.bar
|   |   |   |   |   |-- output.txt
|   |   |   |-- test_error
|   |   |   |   |-- input
|   |   |   |   |   |-- foo.bar
|   |   |   |   |-- output
|   |   |   |   |   |-- foo.bar
|   |   |   |   |   |-- error.txt
|   |   |-- rule2
|   |   |   |-- test_rule2
|   |   |   |   |-- input
|   |   |   |   |   |-- foo.bar
|   |   |   |   |-- output
|   |   |   |   |   |-- foo.bar

Each directory in tests/rule1 is a different test for rule1. As long as tests/rule1 exists in the same directory as rule1.py, ick will find the tests with no extra configuration.

Running tests¶

Use the ick test-rules command to run all tests for your rules. The command will:

Find all rules in all of the configured rulesets
Look for test directories matching the rule names
Run each test and report results

For each test, ick will:

Copy the contents of input/ to a temporary directory
Run the rule on those files
Compare the results with the contents of output/, including any expected errors in error.txt and stdouts in output.txt.

If the files match exactly, the test passes. If there are any differences, the test fails and prints the diff.

Test output¶

When running tests, you’ll see output like:

$ ick test-rules
testing...
  rule1: .. PASS

The two dots before PASS each represent a successful test for a given rule. (So if rule2 in the example above passes, we’d only see one dot) If a test fails, you’ll see an F instead like:

$ ick test-rules
testing...
  rule1: F. FAIL

In the case of a fail, ick will also tell you the following for each failed test:

What differences were found between the expected and actual output
Any exceptions that occurred during the test

If a test is not provided, ick test-rules will note that the rule has no test and mark it as passed.

Coverage¶

For tests written in Python, you can measure test coverage by setting the ICK_COVERAGE_PY environment variable to “1” when running tests:

$ ICK_COVERAGE_PY=1 ick test-rules

This uses coverage.py to create a separate coverage data file for each test run. To combine the data files and report on them, install coverage into your Python environment and use the coverage command:

$ coverage combine -q
$ coverage report -m

We are interested in adding support for tests written in other languages. If you have ideas or want to contribute, please open an issue or pull request.