Revision 7 - 2009-04-12 at 22:02:27

Detecting Leaky Objects

Overview

Recently dandv noticed that MojoMojo processes were possibly leaking memory. An semi-formal investigation was done in order to look into this further. This paper describes what was done to determine that there was indeed a leek, and that the Syntax::Highlight::Engine::Kate module was causing he majority of it.

Verifying a Leak.

The process I used to verify a leak was pretty simple:

• Start a window with top running. Monitor the RES column (memory used) for mojomojo_server.pl
• Start an instance of MojoMojo using the development server (script/mojomojo_server.pl).
• Make one request to http://localhost:3000/ (get MojoMojo fully loaded on the root page).
• Note the MB of usage by the perl process behind MojoMojo (often around 115MB is what it started at).
• Make 100 request of root page (http://localhost:3000/) using apache benchmark utility ab. ab -n 100 http://localhost:3000/
• Note the MB of usage by the MojoMojo perl process.

Tracking Objects Not Destroyed

A nifty module called Devel::Leak::Object allow one to see what objects have not been garbage collected. It will list the object class and how many objects of the class type are left upon exit of the program.

Create MojoMojo::Controller::Root::exit() action

In order to useDevel::Leak::Object on MojoMojo we need a way to exit from the process so that we can get the results. To exit from MojoMojo server weo setup an exit action1. This action will run based on an environment variable being set2. Here's the code:

sub exit : Local {
    my ($self, $c) = @_;
    if ($ENV{MOJOMOJO_EXIT_OK}) {
        exit(0);
    }
    else {
       # $c->stash( template => 'error.tt' );
        $c->res->status (403); # forbidden
        $c->res->body('EXIT NOT OK');
        $c->detach();
    }
}

Set Environment Variable to Enable exit()

Notice the exit() action only exits when the environment variable MOJOMOJO_EXIT_OK is set. This is done from bash as:

export MOJOMOJO_EXIT_OK = 1

Start MojoMojo with Devel::Leak::Object Enabled

perl -MDevel::Leak::Object=GLOBAL_bless script/mojomojo_server.pl -d

This tracks all objects3.

Generate Data

We will generate data for the object tracking report by making requests as we did before:

Make Requests

ab -n 100 http://localhost:3000/

Exit MojoMojo Process

Use the exit action (and MojoMojo's remapping /exit to /.exit) to terminate the MojoMojo process and get the object tracking report. Request:

http://localhost:3000/.exit

Object Report

What appears on STOUT4 is the report which looks like:

"leak_report.txt":http://mojomojo.org/development/Debug/Detecting_Leaky_Objects.attachment/32

* with my $kate = kate() in inside formatcentent(), the RAM usage increased by 15MB/100 requests * with my $kate scoped to package (outside format_content), the RAM usage was 2-3MB/100 request.

"leak_report.txt":http://mojomojo.org/development/Debug/Detecting_Leaky_Objects.attachment/32