.. Mars Documentation
   Copyright (C) 2008 Matt Giuca
   3/9/2008

.. This program is free software: you can redistribute it and/or modify
   it under the terms of the GNU General Public License as published by
   the Free Software Foundation, either version 3 of the License, or
   (at your option) any later version.

.. This program is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

.. You should have received a copy of the GNU General Public License
   along with this program.  If not, see <http://www.gnu.org/licenses/>.

.. _testing:
.. highlight:: yaml

Testing Framework
=================

.. sectionauthor:: Matt Giuca <matt.giuca@gmail.com>

The Mars Test Framework (MTF) is a suite of tools for testing Mars
program code, and the Mars compiler itself.

The framework tests three things:

* Compilation (correct and incorrect code, errors and warnings),
* Compiler-inferred properties, and
* Runtime correctness (system and unit tests on Mars programs)

Test suite
----------

A test suite is simply a collection of :file:`.mar` files (Mars source code
files) and :file:`.mtc` files (Mars Test Case). MTC files are optional, and
correspond to a :file:`.mar` file of the same name.

A "test case" is a single :file:`.mar` file and possibly a :file:`.mtc` file
of the same name.  Test cases are always just a single :file:`.mar` file
(though they can import other modules, eg. to test importing).

MTC file format
---------------

MTC files are `YAML <http://yaml.org/>`_ text files which provide instructions
about the test case, controlling the behaviour of the testing system.

At the top-level is a dict mapping section names to dicts.

Valid sections are: :data:`compile`, :data:`run` (both optional).

eg::

    compile:
        (compile section)
    run:
        (run section)

Inside each section is a dict specific to that section (see sections below).

Note that Unicode characters in MTC files are encoded with UTF-8 before being
used, unless otherwise noted.

Compilation testing
-------------------

Compilation testing involves compiling a :file:`.mar` file just to check
whether compilation succeeds or fails, and check that appropriate warnings and
errors are produced. This sort of test does not involve running the code at
all.

The most basic test case is a single :file:`.mar` file with nothing special
about it.  Such a case will be compiled and will **pass** if it compiles
successfully without warnings, and **fail** if it does not compile, or there
are warnings.

If the case has a .mtc file with a :data:`compile` section, that section is
consulted for further information (which can affect the runtime and unit tests
as well). Its dictionary has the following attributes:

* :data:`outcome`: :data:`succeed` | :data:`fail` (:data:`succeed` is
  default). This dictates what the outcome of compilation *should* be
  (assuming a correct implementation) -- :data:`succeed` means the program
  should be accepted, and :data:`fail` means the program should be rejected.
* :data:`errors`: string data -- the expected output of stderr from the
  compiler.  If omitted, expects stderr to be empty.
* :data:`expect`: :data:`pass` | :data:`fail` | :data:`compile_error`
  (:data:`pass` is default). This dictates whether the current implementation
  is correct -- :data:`pass` means it behaves correctly (compiles correctly or
  fails to compile as required, depending on :data:`outcome`), :data:`fail`
  means it doesn't behave correctly when the program should be rejected
  (compiles when it shouldn't, or gives the wrong compiler error message), and
  :data:`compile_error` means it doesn't compile when it should.
* :data:`environ`: Dict mapping environment strings to values. If supplied,
  will be appended to the environment (replacing existing values), but
  existing variables will not be removed. The environment will be used for all
  runtime and unit tests for the given module.
* :data:`libs`: List of strings containing library names. If supplied, will be
  passed to :command:`mars` with the :option:`-l` option. The libraries will
  be used for all runtime and unit tests for the given module.

Hence, the default value for :data:`compile` is as follows::

    compile:
        outcome: succeed
        errors: ""
        expect: pass
        environ: {}
        libs: []

The test framework will only report unexpected outcomes. This means that
passing cases are not normally reported, but a passing case which is expected
to fail *will* be reported, so you know you have fixed something. Use
``expect: fail`` or ``expect: compile_error`` if you have a test case which
you know fails but intend to fix.

Runtime testing
---------------

Runtime testing is executing functions and checking their output. It implies
that compilation succeeds (if both compilation testing and runtime testing are
specified, compilation must not expect failure).

Compilation is not performed with the :option:`-m <mars -m>` option, so the
program does not do anything by default.

Runtime testing requires an :file:`.mtc` file. The :data:`run` section is
consulted for information on how to do the test.

It is a list of individual cases, which are tested separately. Each case has a
dictionary.

The case's dictionary has the following attributes:

* :data:`call`: Mars expression to be evaluated after loading the module.
        Required. (Typically used to call a function to be tested).
* :data:`stdin`: Input passed to stdin (may be text or binary). If omitted,
  empty.
* :data:`stdout`: Expected stdout. If omitted, is not tested.
* :data:`stderr`: Expected stdout. If omitted, expect empty.
* :data:`expect`: :data:`pass` | :data:`fail`
  Will only report unexpected outcomes. (If expecting fail, then "**pass**"
  will be reported. Use this if you have a test case which you know fails but
  intend to fix) (:data:`pass` is default).

.. note::

   The test framework has no way to distinguish between when a program is
   *computing* and when it is *waiting for input*. Therefore, if a test case
   waits for input and none is supplied, MTF will wait forever. Be sure to
   supply a line of :data:`stdin` for each line expected by the program.

Unit testing
------------

The third kind of test is a "unit test", which tests individual functions
inside Mars files. These are usually written to verify the correctness of Mars
code (eg. library code), rather than the compiler.

A module (:mod:`unittest`) is supplied in the Mars standard library, which can
be imported by test suites and called. It doesn't produce nice output -
instead, it dumps machine-readable output to stdout, which is expected to be
processed by a wrapper.

The Mars Test Framework is such a wrapper. It can be passed a Mars file and
will automatically execute any function beginning with "``test_``", and filter
the output into nice human-readable output. This does not require an
:file:`.mtc` file. Unit test functions must have type :type:`() -> io Num`
(they are also allowed to have type :type:`Num -> io Num`, but this is
deprecated).

Note that like compile and runtime testing, you are able to specify that a
case is expected to fail (but is intended to be fixed), using
:func:`~unittest.begin_case_pending`.

See :mod:`unittest` for more information.

Example MTC file
----------------

::

    compile:
        outcome: succeed
        errors: |
            Warning: Some warning message
        environ: {"SOME_VAR": "some value"}
        libs: ["m", "png"]
    run:
        - call: foo(2)
          stdin: |
              forty two
          stdout: |
              expected stdout text
              perhaps across several lines
        - call: foo(1)
          stdin: "forty one\n"
          stderr: "Incorrect\n"