This post explains how to set up tests making use of OpenCV’s built in framework.

Introduction

Promoting Chilitags, our C++ library for detecting fiducial markers, from “research code” to published code means that we have no excuse any more for not having any test at all. After a short comparisons for testing frameworks, it seemed that Google Test would best fit our need.

Chilitags is heavily based on OpenCV. It turns out that OpenCV already integrates Google Test, and adds their own features, so you don’t have to write a function to assert that two matrices are equal for example. I’m more and more conviced that “OpenCV” means “we already did everything you may and will need” in some secret language.

However, I failed to find documentation on OpenCV’s ts module, even though they use it to test all the other modules. It is not listed in OpenCV’s documentation, and Google didn’t help either. I asked on StackOverflow how to use OpenCV’s test framework with CMake, but haven’t received an answer. This post aims at answering myself the question with what I found.

First, I’ll show how to configure CMake to compile an OpenCV’s executable, and then explain how to run several of them as part of CMake’s tests.

Compiling an OpenCV test executable

This section gives a minimal example of an OpenCV based library being tested by the OpenCV test framework. We give the code of a dummy library, short test codes, and the CMake set up to build them all.

The code to be tested

Let’s say we want to test a dummy library, whose interface is declared in mylib.hpp:

1
2
3
4
5
6
7
8
9
10

#ifndef mylib_hpp
#define mylib_hpp

#include <opencv2/core/core.hpp>

namespace mylib {
  void myfunction (cv::InputOutputArray myparam);
}

#endif

and implemented in mylib.cpp:

1
2
3
4
5

#include <mylib.hpp>

void mylib::myfunction (cv::InputOutputArray myparam) {
  myparam.getMat() = cv::Scalar::all(0);
}

Right… Obviously, this library is not the focus of this article.

The test code

We first write a simple test case in mytest.cpp:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

#include <opencv2/ts/ts.hpp>
#include <mylib.hpp>

class MyTestClass : public cvtest::BaseTest
{
public:
    MyTestClass(): mat(cv::Scalar::all(1)){}

protected:
    cv::Mat mat;

    void run(int) {
      ts->set_failed_test_info(cvtest::TS::OK);

      //check that all values are between 1 and 1 (and not Nan)
      if (0 != cvtest::check(mat, 1, 1, 0) )
          ts->set_failed_test_info(cvtest::TS::FAIL_INVALID_TEST_DATA);

      mylib::myfunction(mat);
      
      //check that all values are between 0 and 0 (and not Nan)
      if (0 != cvtest::check(mat, 0, 0, 0) )
          ts->set_failed_test_info(cvtest::TS::FAIL_INVALID_OUTPUT);
  }
};

TEST(MyTestSuite, ATestThatPasses) {
  MyTestClass myTestClass;
  myTestClass.safe_run();
}

TEST(MyTestSuite, ATestThatFails) {
  bool mybool = false;
  ASSERT_TRUE(mybool);
}

This test cases will be automagically run in a specified folder if they are compiled with such test_main.cpp:

1
2
3

#include <opencv2/ts/ts.hpp>

CV_TEST_MAIN(".")

That’s it!

Structure and CMake Files

All these files will be structured typically as follow:

1
2
3
4
5
6
7
8
9
10
11
12

mylib/
  CMakeLists.txt
  lib/
      CMakeLists.txt
      include/
          mylib.hpp
      src/
          mylib.cpp
  test/
      CMakeLists.txt
      mytest.cpp
      test_main.cpp

The general CMakeLists is standard…

1
2
3
4
5
6
7
8
9
10
11
12
13

cmake_minimum_required(VERSION 2.6)

project(mylib)

include_directories(lib/include)

add_subdirectory(lib)

option (WITH_TESTS "build tests" ON)

if (WITH_TESTS)
  add_subdirectory(test)
endif()

…and so is the one to compile the library (in src/CMakeLists.txt):

1
2
3
4
5

file(GLOB_RECURSE mylib_source src/*)
add_library(mylib SHARED ${mylib_source})

find_package(OpenCV REQUIRED core)
target_link_libraries(mylib ${OpenCV_LIBS})

Last but not least, it is not much more complicated to compile an OpenCV test:

1
2
3
4
5
6
7

file(GLOB mytest_source_files *.cpp)
add_executable(mytest ${mytest_source_files})

target_link_libraries(mytest mylib)

find_package( OpenCV REQUIRED ts)
target_link_libraries( mytest ${OpenCV_LIBS} )

And we’re done.

Running the tests

Finally we can compile averything and run the test in a standard CMake process. Here is the one if you use Makefile:

1
2
3
4

mkdir build
cd build
cmake ..
make

This creates a mytest executable in build/test which can be run to produce this output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from MyTestSuite
[ RUN      ] MyTestSuite.ATestThatPasses
[       OK ] MyTestSuite.ATestThatPasses (0 ms)
[ RUN      ] MyTestSuite.ATestThatFails
pathto/opencv/test/mytest.cpp:34: Failure
Value of: mybool
  Actual: false
Expected: true
[  FAILED  ] MyTestSuite.ATestThatFails (0 ms)
[----------] 2 tests from MyTestSuite (1 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (1 ms total)
[  PASSED  ] 1 test.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] MyTestSuite.ATestThatFails

 1 FAILED TEST

Yay!

Configuring CMake to run the tests

Now we may want to use several test executables for various reason. For example, we may want to split unrelated test, or split a test that takes too long to run, or make another kind of test (unit tests, performances tests, internationalisation tests or what not), or simply run non C++ executable.

It would be a bit cumbersome to launch them all by hand, especially when CMake integrates testing facilites.

Instead, we simply tell CMake which programs are test programs by adding an enable_testing() directive to the general CMakeLists:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

cmake_minimum_required(VERSION 2.6)

project(mylib)

include_directories(lib/include)

add_subdirectory(lib)

option (WITH_TESTS "build tests" ON)

if (WITH_TESTS)
  enable_testing()
  add_subdirectory(test)
endif()

We also specify an add_test(testname executablename) directive for each test executable:

1
2
3
4
5
6
7
8
9

file(GLOB mytest_source_files *.cpp)
add_executable(mytest ${mytest_source_files})

target_link_libraries(mytest mylib)

find_package( OpenCV REQUIRED ts)
target_link_libraries( mytest ${OpenCV_LIBS} )

add_test(mycmaketest mytest)

This way, CMake will create a test target which can be built by your IDE to run the tests. For example when I run make test, I get the following output:

1
2
3
4
5
6
7
8
9
10
11
12

Running tests...
Test project pathto/build
    Start 1: mycmaketest
1/1 Test #1: mycmaketest ......................***Failed    0.00 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.01 sec

The following tests FAILED:
    1 - mycmaketest (Failed)
Errors while running CTest

This output is a lot less detailed than the previous one, which makes sense if there are a lot of tests happing, There is also an option to display the output of each test program (as shown before). With make, it’s make test ARGS=-V.

Conclusion

Of course, there is much more to say about CMake’s test facilities or OpenCV’s test framework, not to mention the embedded Google Test framework. While the documentation for the former and the latter can be found easily, I have yet to find the documentation for OpenCV’s ts module. As an alternative, I guess I’ll stick to reading the source of the module, and look at the examples in the test folders of the other modules.

Do not hesitate to comment on the StackOverflow question if you know more !