Marshall's C++ Musings

Testing libc++ with Address Sanitizer


I’ve been running the libc++ tests off and on for a while. It’s a quite extensive test suite, but I wondered if there were any bugs that the test suite was not uncovering. In the upcoming clang 3.3, there is a new feature named Address Sanitizer which inserts a bunch of runtime checks into your executable to see if there are any “out of bounds” reads and writes to memory.

In the back of my head, I’ve always thought that it would be nice to be able to say that libc++ was “ASan clean” (i.e, passed all of the test suite when running with Address Sanitizer).

So I decided to do that. [ All of this work was done on Mac OS X 10.8.2/3, btw ]

How to run the tests:

There’s a script for running the tests. It’s called testit.

    $ cd $LLVM/libcxx/test ; ./testit

where $LLVM/libcxx is where libc++ is checked out.

This takes about 30 minutes to run.

Without Address Sanitizer, libc++ fails 12 out of the 4348 tests.

Running the tests with Address Sanitizer

    $ cd $LLVM/libcxx/test ; CC=/path/to/tot/clang++ OPTIONS= "-std=c++11 -stdlib=libc++ -fsanitize=address" ./testit

Note: the default options are “-std=c++11 -stdlib=libc++”, that’s what you get if you don’t specify anything

This takes about 92 minutes; just a bit more than three times as long.

With Address Sanitizer, libc++ fails 54 tests (again, out of 4348)

What are the failures?

Once Howard and I fixed the random tests and the bug in the iostreams code, I re-ran the tests using a recently build dylib.

    $ cd $LLVM/libcxx/test ; DYLD_LIBRARY_PATH=$LLVM/libcxx/lib CC=/path/to/tot/clang++ OPTIONS= "-std=c++11 -stdlib=libc++ -fsanitize=address" ./testit

This gave us 16 failures:


I’m glad to see that there were so few problems in the libc++ code. It’s a fundamental building block for applications on Mac OS X. And now it’s better than it was when we started this exercise.

However, we did find a couple bugs in the test suite, and one heap-smashing bug in libc++. We also found a limitation in Address Sanitizer, too – which I’m hoping the developers will address soon.