\section{Automated testing}

We use automated testing for the Max implementation.  This module
sets up the LISP environment so we can define and run the tests.

<<test>>=
<<Use the automated testing package>>
<<Create the main Max test suite instance>>
<<Create a function to run all Max tests>>
<<Create a test fixture used by many Max modules>>
@ 

We chose the XP Test Suite by onShore Development because it was the
only test suite for Common LISP we could find and get to work.  It
works adequately, but needs improvement.  Fortunately, it is public
domain so we can simply make the changes as we need them.  We'll need
to make some improvements soon, because I think I'm avoiding writing
more test cases to avoid having to deal with the existing situation.

<<Use the automated testing package>>=

(defpackage :max-test (:use :cl :max :xptest))
;        (:export max-test-suite))
(in-package :max-test)
;(use-package :xptest)
@ 
A test suite is a collection of other test suites and individual tests.
We make one test suite the ``main'' test suite, which will contain all
the tests for Max.  In each module, we'll add Max tests or test suites
to this one.

<<Create the main Max test suite instance>>=
; Create a master test suite
(defparameter *max-test-suite*
        (make-test-suite "main Max test suite"
                "All other max test suites should be added to this one"
        )
)
@
The standard interface provided by onShore's package requires
running tests to produce a ``test results'' object, and then calling a
reporting function on the results.  We create a function, TEST, to
combine both tasks: It runs all tests and displays their results.

The result reporting function needs improvements soon, since it
displays at minimum 3 lines per test, with no summary at the end, so
you have to scroll back quite a ways in order to determine which tests
succeeded or failed.

In order to see the reasons for any test failures, you have to call
[[(TEST :verbose t)]].  This adds even more lines of text to
successful tests, though, so we'll remain terse by default.

<<Create a function to run all Max tests>>=
; runs all max tests
(defun test (&key (verbose nil))
    (let ((results (run-test *max-test-suite*)))
        (report-result results)
	(test-summary results)
    )
)
@ 

A test fixture is an object containing some variables that store
intermediate results for tests.  In many Max modules, no test fixture
is needed, but the testing package requires one to be declared for
every test.  Here we define the empty test fixture and call it
``nullfix'' so we don't have to redefine it every time it
is needed.

<<Create a test fixture used by many Max modules>>=
; The empty fixture, used by a lot of tests since we test a lot of globals
(def-test-fixture nullfix () ())

@ 

\subsection{Combining Testing with Literate Programming}

Prior to the use of literate programming for documenting Max, the
tests were at the end of the file for the module they were testing.
Most other programs I have seen that use testing put the tests in a
separate file and directory, using a name related to the name of the
file they are testing.  Regardless of where the tests are stored,
we'll now present tests right along with the code they are testing.  
Just as literate programming allows documentation to be kept in sync
with code, it will allow the tests to be kept in sync as well.