12.3.1 Factorial example

As an example, consider the set of seven factorial functions developed in Chapters 4 and 9 (in source file Factorial.hs). All have the requirement to implement the mathematical function

fact$(n) = \prod_{i=1}^{i=n}\,i$

for any $n \geq 0$. The specification is ambiguous on what should be the result of calling the function with a negative argument.

12.3.2 Arrange

To carry out black-box testing, we must arrange our input values. The factorial function tests do not require any special testing “objects”.

We first partition the input domain. We identify two equivalence classes of inputs for the factorial function:

1. the set of nonnegative integers for which the mathematical function is defined and the Haskell function returns that value within the positive Int range

2. the set of nonnegative integers for which the mathematical function is defined but the Haskell function returns a value that overflows the Int range

The class 2 values result are errors, but integer overflow is typically not detected by the hardware.

We also note that the negative integers are outside the range of the specification.

Next, we select the following values inside the “lower” boundary of class 1 above:

• 0, empty case at the lower boundary
• 1, smallest nonempty case at the lower boundary

Then we choose representative values within class 1:

• 2, one larger than the smallest nonempty case
• 5, arbitrary value representative of values away from the boundary

Note: The choice of two representative values might be considered a violation of the “minimize test overlap” principle from Chapter 11. So it could be acceptable to drop the input of 2. Of course, we could argue that we should check 2 as a possible boundary value.

We also select the value -1, which is just outside the lower boundary implied by the $n \geq 0$ requirement.

All of the factorial functions have the type signature (where N is 1, 2, 3, 4, 4', 5, or 6):

    factN :: Int -> Int

Thus the factN functions also have an “upper” boundary that depends on the maximum value of the Int type on a particular machine. The author is testing these functions on a machine with 64-bit, two’s complement integers. Thus the largest integer whose factorial is less than $2^{63}$ is 20.

We thus select input the following input values:

• 20, which is just inside the upper boundary of class 1

• 21, which is just outside class 1 and inside class 2

12.3.3 Act

We can test a factorial function at a chosen input value by simply applying the function to the value such as the following:

    fact1 0

A Haskell function has no side effects, so we just need to examine the integer result returned by the function to determine whether it satisfies the function’s specification.

12.3.4 Assert

We can test the result of a function by stating a Boolean expression—an assertion—that the value satisfies some property that we want to check.

In simple cases like the factorial function, we can just compare the actual result for equality with the expected result. If the comparison yields True, then the test subject “passes” the test.

    fact1 0 == 1

12.3.5 Aggregating into test script

There are testing frameworks for Haskell (e.g., HUnit [3], QuickCheck [2], or Tasty [4]), but, in this section, we manually develop a simple test script.

We can state a Haskell IO program to print the test and whether or not it passes the test. (Simple input and output will eventually be discussed in a Chapter 10. For now, see the Haskell Wikibooks [7] page on “Simple input and output”.)

Below is a Haskell IO script that tests class 1 boundary values 0 and 1 and “happy path” representative values 2 and 5.

    pass :: Bool -> String
    pass True  = "PASS"
    pass False = "FAIL"

    main :: IO ()
    main = do
        putStrLn  "\nTesting fact1"
        putStrLn ("fact1 0 == 1:      " ++ pass (fact1 0 == 1)) 
        putStrLn ("fact1 1 == 1:      " ++ pass (fact1 1 == 1)) 
        putStrLn ("fact1 2 == 2:      " ++ pass (fact1 2 == 2)) 
        putStrLn ("fact1 5 == 120:    " ++ pass (fact1 5 == 120)) 

The do construct begins a sequence of IO commands. The IO command putStrLn outputs a string to the standard output followed by a newline character.

Testing a value below the lower boundary of class 1 is tricky. The specification does not require any particular behavior for -1. As we saw in Chapter 4, some of the function calls result in overflow of the runtime stack, some fail because all of the patterns fail, and some fail with an explicit error call. However, all these trigger a Haskell exception.

Our test script can catch these exceptions using the following code.

        putStrLn ("fact1 (-1)  == 1:  " 
                  ++ pass (fact1 (-1)  == 1))
            `catch` (\(StackOverflow)    
                      -> putStrLn ("[Stack Overflow] (EXPECTED)"))
            `catch` (\(PatternMatchFail msg)
                      -> putStrLn ("[Pattern Match Failure]\n...."
                                   ++ msg))
            `catch` (\(ErrorCall msg)
                      -> putStrLn ("[Error Call]\n...." ++ msg))

To catch the exceptions, the program needs to import the module Control.Exception from the Haskell library.

    import Prelude hiding (catch) 
    import Control.Exception 

By catching the exception, the test program prints an appropriate error message and then continues with the next test; otherwise the program would halt when the exception is thrown.

Testing an input value in class 2 (i.e., outside the boundary of class 1) is also tricky.

First, the values we need to test depend on the default integer (Int) size on the particular machine.

Second, because the actual value of the factorial is outside the Int range, we cannot express the test with Haskell Ints. Fortunately, by converting the values to the unbounded Integer type, the code can compare the result to the expected value.

The code below tests input values 20 and 21.

        putStrLn ("fact1 20 == 2432902008176640000:   "
                  ++ pass (toInteger (fact1 20) == 
                                      2432902008176640000))
        putStrLn ("fact1 21 == 51090942171709440000:  "
                  ++ pass (toInteger (fact1 21) == 
                                      51090942171709440000)
                  ++ " (EXPECT FAIL for 64-bit Int)" )

The above is a black-box unit test. It is not specific to any one of the seven factorial functions defined in Chapters 4 and 9. (These are defined in the source file Factorial.hs.) The series of tests can be applied any of the functions.

The test script for the entire set of functions from Chapters 4 and 9 (and others) are in the source file TestFactorial.hs.

12.4.1 Rational arithmetic modules example

For this section, we use the rational arithmetic example from Chapter 7.

In the rational arithmetic example, we define two abstract (information-hiding) modules: RationalRep and Rational.

Given that the Rational module depends on the RationalRep module, we first consider testing the latter.

12.4.2 Data representation modules

Chapter 7 defines the abstract module RationalRep and presents two distinct implementations, RationalCore and RationalDeferGCD. The two implementations differ in how the rational numbers are represented using data type Rat. (See source files RationalCore.hs and RationalDeferGCD.hs.)

Consider the public function signatures of RationalRep (from Chapter 7):

    makeRat :: Int -> Int -> Rat
    numer   :: Rat -> Int 
    denom   :: Rat -> Int 
    zeroRat :: Rat
    showRat :: Rat -> String

Because the results of makeRat and zeroRat and the inputs to numer, denom, and showRat are abstract, we cannot test them directly as we did the factorial functions Section 12.3. For example, we cannot just call makeRat with two integers and compare the result to some specific concrete value. Similarly, we cannot test numer and denom directly by providing them some specific input value.

However, we can test both through the abstract interface, taking advantages of the interface invariant.

RationalRep Interface Invariant (from Chapter 7):

: For any valid Haskell rational number r, all the following hold:

-   `r`{.haskell} $\in$ `Rat`{.haskell}
-   `denom r > 0`{.haskell}
-   if `numer r == 0`{.haskell}, then `denom r == 1`{.haskell}
-   `numer r`{.haskell} and `denom r`{.haskell} are relatively prime
-   the (mathematical) rational number value is
    $\frac{\texttt{numer r}}{\texttt{denom r}}$
    

The invariant allows us to check combinations of the functions to see if they give the expected results. For example, suppose we define x' and y' as follows:

    x' = numer (makeRat x y)
    y' = denom (makeRat x y)

Then the interface invariant and contracts for makeRat, numer, and denom allow us to infer that the (mathematical) rational number values $\frac{\texttt{x'}}{\texttt{y'}}$ and $\frac{\texttt{x}}{\texttt{y}}$ are equal.

This enables us to devise pairs of test assertions such as

    numer (makeRat 1 2) == 1
    denom (makeRat 1 2) == 2

and

    numer (makeRat 4 (-2)) == -2
    denom (makeRat 4 (-2)) == 1

to indirectly test the functions in terms of their interactions with each other. All the tests above should succeed if the module is designed and implemented according to its specification.

Similarly, we cannot directly test the private functions signum', abs', and gcd'. But we try to choose inputs the tests above to cover testing of these functions. (Private functions should be tested as the module is being developed to detect any more problems.)

    maxInt  = (maxBound :: Int)
    minInt  = (minBound :: Int)

    numer (makeRat 12 8) 

    denom (makeRat 12 8) 

    numer (makeRat 12 8) == 3

    denom (makeRat 12 8) == 2

    pass :: Bool -> String
    pass True  = "PASS"
    pass False = "FAIL"

    main :: IO ()
    main =
        do
            -- Test 3/2
            putStrLn ("numer (makeRat 3 2) == 3:               " ++
                      pass (numer (makeRat 3 2) == 3))
            putStrLn ("denom (makeRat 3 2) == 2:               " ++
                      pass (denom (makeRat 3 2) == 2))
            -- Test -3/-2
            putStrLn ("numer (makeRat (-3) (-2)) == 3:         " ++
                      pass (numer (makeRat (-3) (-2)) == 3))
            putStrLn ("denom (makeRat (-3) (-2)) == 2:         " ++
                      pass (denom (makeRat (-3) (-2)) == 2))
            -- Test 12/8
            putStrLn ("numer (makeRat 12 8) == 3:              " ++
                      pass (numer (makeRat 12 8) == 3))
            putStrLn ("denom (makeRat 12 8) == 2:              " ++
                      pass (denom (makeRat 12 8) == 2))
            -- Test -12/-8
            putStrLn ("numer (makeRat (-12) (-8)) == 3:        " ++
                      pass (numer (makeRat (-12) (-8)) == 3))
            putStrLn ("denom (makeRat (-12) (-8)) == 2:        " ++
                      pass (denom (makeRat (-12) (-8)) == 2))
            -- Test 0/0 
            putStrLn ("makeRat 0 0 is error:                   "
                      ++ show (makeRat 0 0)) 
                `catch` (\(ErrorCall msg) 
                             -> putStrLn ("[Error Call] (EXPECTED)\n"
                                      ++ msg)) 

12.4.3 Rational arithmetic modules

TODO: Write section

The interface to the module Rational consists of the functions negRat, addRat, subRat, mulRat, divRat, and eqRat, the RationalRep module’s interface. It does not add any new data types, constructors, or destructors.

The Rational abstract module’s functions preserve the interface invariant for the RationalRep abstract module, but it does not add any new components to the invariant.

12.4.4 Reflection on this example

TODO: Update after completing chapter

I designed and implemented the Rational and RationalCore modules using the approach described in the early sections of Chapter 7, doing somewhat ad hoc testing of the modules with the REPL. I later developed the RationalDeferGCD module, abstracting from the RationalCore module. After that, I wrote Chapter 7 to describe the example and the development process. Even later, I constructed the systematic test scripts and wrote Chapters 11 and 12 (this chapter).

As I am closing out the discussion of this example, I find it useful to reflect upon the process.

• The problem seemed quite simple, but I learned there are several subtle issues in the problem and the modules developed to solve it. As the saying goes, “the devil is in the details”.

• In my initial development and testing of these simple modules, I got the “happy paths” right and covered the primary error conditions. Although singer Bobby McFerrin’s song “Don’t Worry, Be Happy” may give good advice for many life circumstances, it should not be taken too literally for software development and testing.

• In writing both Chapter 7 and this chapter, I realized that my statements of the preconditions, postconditions, and interface invariants of RationalRep abstraction needed to be reconsidered and restated more carefully. Specifying a good abstract interface for a family of modules is challenging.

• In developing the systematic test scripts, I encountered other issues I had either not considered sufficiently or overlooked totally:

  • the full implications of using the finite data Int data type for the rational arithmetic modules

  • the impact of the underlying integer arithmetic representation (e.g., as two’s complement) on the Haskell code

  • the effects of calls of functions like numer, denom, and showRat with invalid input data

  • a subtle violation of the interface invariant in the RationalDeferGCD implementations of makeRat and showRat

  • the value of a systematic input domain partitioning for both developing good tests and understanding the problem

It took me much longer to develop the systematic tests and document them than it did to develop the modules initially. I clearly violated the Meszaros’s final principle, “ensure commensurate effort and responsibility” described in the previous chapter (also in Mesazaros [6, Ch. 5]).

For future programming, I learned I need to pay attention to other of Meszaros’s principles such as “design for testability”, “minimize untestable code”, “communicate intent”, and perhaps “write tests first” or at least to develop the tests hand-in-hand with the program.