Thursday, October 31, 2013

Getting valgrind memcheck to work with uclibc

We recently built valgrind for uclibc through Angstrom and were dismayed to find that it would not find memory leaks, even ones we purposely leaked to test it with. Valgrind didn't complain of any errors, even with verbose debug output. Scratching our heads, we deep dived into the insanity that is valgrind and its virtual processor (which is actually incredibly fascinating and highly worth a look). After days of debugging, mainly through print statements, we found that the valgrind preload libraries (vgpreload_*-arm-linux.so) that actually contain the new malloc/free symbols were not being loaded.

It turns out that valgrind actually sets the LD_PRELOAD environment variable early in execution, before loading the executable that is to be analyzed. This should instruct the runtime link loader to load the libraries in LD_PRELOAD, which are the vgpreload ones. Unfortunately, if your uclibc is configured without LD_PRELOAD (like ours was), the loader will ignore LD_PRELOAD and valgrind won't know that the vgpreload libraries were never loaded and malloc was not replaced.

tl;dr uclibc must be configured with LD_PRELOAD in order for valgrind's memcheck to work.