Without test cases, we don’t know if our program works as expected. We need to make sure that when we modify our program, we are not introducing new bugs. Creating test cases to test the different inputs that our program accepts is important.
Novice programmers often write test cases, confirm that the program works and then modify the test cases to check for other types of input. The problem with this approach is that all the test cases are in the programmer’s head. When the program is modified, there is no good way to run all the previous test cases.
In test-driven development, the programmer starts with writing a test and then writes the necessary code to pass the test and regularly refactors the code.
While we will not adapt a fully test-driven development approach in this course, we will emphasize writing tests to check the correctness of our program.
There are multiple C++ testing frameworks. Most of these frameworks are appropriate for much larger project. We will instead rely on the much simpler assert statements to test our code.
Below is the main.cpp file from https://github.com/pisan342/hello-factorial.
/**
* test functions for factorial
**/
#include "factorial.h"
#include <cassert>
#include <iostream>
using namespace std;
void test01() {
assert(fact(1) == 1);
assert(fact(3) == 6);
}
void test02() {
assert(fact(-1) == 1);
assert(fact(5) == 120);
}
int main() {
test01();
test02();
memoryLeakFunction();
cout << "Done." << endl;
return 0;
}
The assert statement takes one parameter. If it is true, the program continues. If it is false, the program exits with a message. Multiple conditions can be combined in an assert statement as necessary. A common trick is to add a descriptive string to assert statements such as assert(fact(1) == 1 && "Checking factorial of 1 failed"); A string always evaluates to true, so only the first condition determines the value passed to assert.
assert statements are also useful for checking the expected value of parameters passed into a function. When compiling a release version of the software, the program can be compiled to disable all the assert statements, so there is no performance impact on the final program.
assert statements are a very basic form of unit testing. If the program successfully executes passing all the tests, we can conclude it is passing all the tests and the functions are working as intended.
We might still have functions that are not getting called or lines of code that are never executed. To check the code coverage of our programs, we will use llvm-profdata and llvm-cov. The overall process is as follows:
(We will never do these steps by hand, but it is included here, so you understand the overall process)
- Compile the program:
clang++-g -fprofile-instr-generate -fcoverage-mapping *.cpp -o a.out-code-coverage - Execute the program to create default.profraw file:
./a.out-code-coverage - Combine profiling data from muliple runs of the program:
llvm-profdata merge default.profraw -output=a.out-code-coverage .profdata - Create a visual from the profiling data:
llvm-cov show a.out-code-coverage-instr-profile=a.out-code-coverage.profdata - Create a report from the profiling data:
llvm-cov report a.out-code-coverage-instr-profile=a.out-code-coverage.profdata
The following steps would produce an output similar to what is seen below
factorial.cpp:
1| |/*
2| | * factorial definition
3| | *
...
18| 10| return N * fact(N - 1);
19| 10| } else {
20| 0| cout << "Too large: " << N << endl;
21| 0| return -1;
22| 0| }
23| 15|}
24| |
...
37| |
38| 0|void unusedFunction() {
39| 0| cout << "A cout statement ";
40| 0| cout << "that is never called " << endl;
41| 0|}
Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
factorial.cpp 12 2 83.33% 3 1 66.67% 27 7 74.07%
main.cpp 3 0 100.00% 3 0 100.00% 16 0 100.00%
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 15 2 86.67% 6 1 83.33% 43 7 83.72%
The lines 20-21 as well as lines 38-41 in main.cpp are never executed. The report shows that factorial.cpp has 83.33% coverage while main.cpp has 100% coverage.
Since we do not want to execute these commands by hand, we will use the https://github.com/pisan342/hello-factorial/blob/master/check-code-coverage.sh script to generate this output for us.