Mirantis | The #1 Pure Play OpenStack Company

Edge of the Stack: Find Memleaks Without Process Restarts

Articles and tutorials in the “Edge of the Stack” series cover fundamental programming issues and concerns that might not come up when dealing with OpenStack directly, but are certainly relevant to the OpenStack ecosystem. For example, drivers are often written in C rather than Python, leading to concerns about memory leaks and other C-specific issues. The “Edge of the Stack” series is meant to provide information on these peripheral, but still important areas.

Memory leaks are one of most frequent problems that plague programs written in C. And, there are a lot of different ways to solve this problem. You can use profilers, for example, valgrind. But what do you do if you need to find memory leaks on the embedded system or server without special software? This article describes one of the solutions–using special glibc functions mtrace and muntrace.

To get some information about these functions, just type “man 3 mtrace” in the Unix console or see the Wikipedia definition.

The most common way to use them is shown below.

To make mtrace run, you should also export the MALLOC_TRACE variable before the start, as in the following example:

If you run this application, you will see /home/YourUserName/path/to/program/MallocTraceOutputFile.txt appear. This file will contain a log of alloc and free invocations. Now you can find all of the places where memory leaks happened using the mtrace script (man 1 mtrace):

It’s a pity, but the mtrace script doesn’t show the debug info (filename and line number). You have several ways to find this information:

  • If you’ve compiled your application with the -g flag and didn’t strip it, you could just open /home/YourUserName/path/to/program/MallocTraceOutputFile.txt and find the line with same  “address” and “caller” what are you interested in, for example:

  • If you run a stripped binary file (for example, there is no place on the hardware), this information will be absent from the log file.

  • But if you have not stripped the binary file you, can use addr2line (man 1 addr2line) to convert the address to source filename:linenumber format. Just use addr2line -e <exename> <addr>. This won’t work if the allocation is invoked in the library. The other way is to use the GNU Debugger (gdb). Also, gdb helps you check the compiled applications that don’t use mtrace.

  • If you have a daemon that starts on system startup you can attach gdb to it, call mtrace, and get the memory login runtime:

    Now the gdb has attached to the cfgmgr. We should set MALLOC_TRACE and call mtrace:

    If you have opened another console to reproduce leaks, you can just continue with the process:

To stop and call muntrace, just hit Ctrl+C and do the following:

Now you can analyze your /mnt/cf/MallocTrace.txt.

If you don’t have another console to reproduce leaks, you should detach the daemon and quit after the mtrace invocation. To stop the logs, you need to attach a daemon one more time and call muntrace. Do not forget to detach gdb later.

It’s very difficult to analyze logs. The main reason–there is no stacktrace. I’ve found only one way so far. To get the stacktrace, you should use gdb breakpoints, as in the following example:

So, there’s the stacktrace.

To round things out, I’ll explain some possible improvements for this method.

Large log file size are not allowed on some systems, for example, on embedded systems. To avoid their usage, we could create a FIFO object (named pipe in Unix-like systems) and attach the mtrace script to it.

  1. I’ve mentioned that the final mtrace result doesn’t contain the caller name. So, it’s better to improve the mtrace script to show this information. (See the fixed mtrace script.)

  2. The next improvement is to show trace any time you want (do not call muntrace). I’d like to improve mtrace script setting signal handler to show run-time information about leaks. It will allow us to trace memory for a long time. (See the  fixed mtrace script.)

  3. I’ve also modified the script to show links between the alloc and free invocations, which I think you will find helpful. (See the modified script.)

The last working script follows:

Now you’ll be able to find memory leaks in your programs using mtrace. Tracing memory allocations is very useful when you can’t use special software. Also it can save your time if you’ve detected a leak suspect. In that case, just put the debug output into the suspicious segment of code and execute the program.

I hope that you find this information helpful. You can take a look at the final code in appendixes A and B.

Google Plus Mirantis

Some HTML is OK

or, reply to this post via trackback.