Re: [Pan-devel] potential memory improvement

[Charles Kerr]

Ken Geis wrote:
> I've been loading a BIG group off newsfeeds.com and my ISP, and Pan
> (0.126) often gets up to 2G of memory usage when I visit this group.
> I'm starting to look into the usage of STL within the Quark class.
>
> I'm probably not going to find big areas for improvement on memory
> usage, but I'm going to try.

:)

> I think that __gnu_cxx::hashtable can be used instead of 
> __gnu_cxx::hash_map for Quark._lookup.  That could save the size of a 
> StringView for each entry in the hashtable.  hash_map is redundant 
> because in each node, it stores a key-value pair; the value that Quark 
> is storing (Impl) already contains the key.  I'm looking in to this.  If 
> anyone else wants to look into it, it'd be nice.  I haven't touched C++ 
> in several years.

This is a good idea, and I've spent a couple of minutes dinking with it.
hash_set looks like a very thin wrapper around hashtable, and still has
a non-gcc parallel (std::set) that we can fall back on.

Suprisingly, hash_set and hash_table had nearly identical memory use.
Also, sizeof(Impl) stubbornly stayed at 24 on my x64 box even after
I changed StringView::len and Impl::refcount from ulongs to uint32_ts.

By bypassing StringView altogether and having defining Impl to
hold a char* and two uint32_t's for len and refcount, I was able
to get sizeof(Impl) down to 16.  Top said that in a large group
this saved about 5 megs of memory and loaded articles about 15%
faster, so it's worth keeping as a first stab at the problem,
but it doesn't fix the 2GB kind of overhead you're seeing.

Incidentally, are you on an 32-bit or 64-bit architecture?

Charles