The Mars Test Framework (MTF) is a suite of tools for testing Mars program code, and the Mars compiler itself.
The framework tests three things:
A test suite is simply a collection of .mar files (Mars source code files) and .mtc files (Mars Test Case). MTC files are optional, and correspond to a .mar file of the same name.
A “test case” is a single .mar file and possibly a .mtc file of the same name. Test cases are always just a single .mar file (though they can import other modules, eg. to test importing).
MTC files are YAML text files which provide instructions about the test case, controlling the behaviour of the testing system.
At the top-level is a dict mapping section names to dicts.
Valid sections are: compile, run (both optional).
eg:
compile:
(compile section)
run:
(run section)
Inside each section is a dict specific to that section (see sections below).
Note that Unicode characters in MTC files are encoded with UTF-8 before being used, unless otherwise noted.
Compilation testing involves compiling a .mar file just to check whether compilation succeeds or fails, and check that appropriate warnings and errors are produced. This sort of test does not involve running the code at all.
The most basic test case is a single .mar file with nothing special about it. Such a case will be compiled and will pass if it compiles successfully without warnings, and fail if it does not compile, or there are warnings.
If the case has a .mtc file with a compile section, that section is consulted for further information (which can affect the runtime and unit tests as well). Its dictionary has the following attributes:
Hence, the default value for compile is as follows:
compile:
outcome: succeed
errors: ""
expect: pass
environ: {}
libs: []
The test framework will only report unexpected outcomes. This means that passing cases are not normally reported, but a passing case which is expected to fail will be reported, so you know you have fixed something. Use expect: fail or expect: compile_error if you have a test case which you know fails but intend to fix.
Runtime testing is executing functions and checking their output. It implies that compilation succeeds (if both compilation testing and runtime testing are specified, compilation must not expect failure).
Compilation is not performed with the -m option, so the program does not do anything by default.
Runtime testing requires an .mtc file. The run section is consulted for information on how to do the test.
It is a list of individual cases, which are tested separately. Each case has a dictionary.
The case’s dictionary has the following attributes:
Required. (Typically used to call a function to be tested).
stdin: Input passed to stdin (may be text or binary). If omitted, empty.
stdout: Expected stdout. If omitted, is not tested.
stderr: Expected stdout. If omitted, expect empty.
expect: pass | fail Will only report unexpected outcomes. (If expecting fail, then “pass” will be reported. Use this if you have a test case which you know fails but intend to fix) (pass is default).
Note
The test framework has no way to distinguish between when a program is computing and when it is waiting for input. Therefore, if a test case waits for input and none is supplied, MTF will wait forever. Be sure to supply a line of stdin for each line expected by the program.
The third kind of test is a “unit test”, which tests individual functions inside Mars files. These are usually written to verify the correctness of Mars code (eg. library code), rather than the compiler.
A module (unittest) is supplied in the Mars standard library, which can be imported by test suites and called. It doesn’t produce nice output - instead, it dumps machine-readable output to stdout, which is expected to be processed by a wrapper.
The Mars Test Framework is such a wrapper. It can be passed a Mars file and will automatically execute any function beginning with “test_”, and filter the output into nice human-readable output. This does not require an .mtc file. Unit test functions must have type () -> io Num (they are also allowed to have type Num -> io Num, but this is deprecated).
Note that like compile and runtime testing, you are able to specify that a case is expected to fail (but is intended to be fixed), using begin_case_pending.
See unittest for more information.
compile:
outcome: succeed
errors: |
Warning: Some warning message
environ: {"SOME_VAR": "some value"}
libs: ["m", "png"]
run:
- call: foo(2)
stdin: |
forty two
stdout: |
expected stdout text
perhaps across several lines
- call: foo(1)
stdin: "forty one\n"
stderr: "Incorrect\n"