Re: [Maposmatic-dev] File descriptor leak problem

[David MENTRE]

Hello Thomas,

2012/4/16 Thomas Petazzoni <address@hidden>:
> From the investigations Maxime and I conducted today, it seems like
> Mapnik keeps the .shp (and related) files opened once the rendering is
> terminated. We use such files for the grid and the administrative
> boundary shape, one per page in multi-page renderings. So for a 50
> pages multi-page rendering, there are about 104 (50*2 + 2 for the cover
> + 2 for the overview) shape files that remain open.

I am glad to see that your are making progress in locating this
annoying bug. Locating it is half fixing it. ;-)

> An option would be to move the rendering process into a separate Unix
> process for every rendering, so that all resources (memory, file
> descriptors, etc.) are cleaned up.

I think this option is the most tolerant to this error and other
similar errors that would certainly occur in the future. While it
might not be the cleanest from an idealistic point of view, it uses
the isolation features of our underlying OS and ensures that
everything is properly cleaned-up. In fact, I thought it was the way
the rendering pipeline was implemented. :-)

Regarding performance, I think (but not measured) that the overhead of
forking a process is negligible compared to other overheads we have,
like database queries.

> Another option would be to fix the problem itself, but it seems to be
> within Mapnik internals and/or the Python API that does not provide the
> way of telling Mapnik to release the resources it has taken.

But if you want to follow this path, have you tried to trigger the
garbage collector through gc.collect()? I don't know Python well so
there might be another method to call, to finalize certain objects for
example.
  http://docs.python.org/library/gc.html

Best regards,
david