[pooma-dev] comments on Pool

[Allan Stokes]

I performed some more tests.  The same Krylov Pool assert is occuring on
FreeBSD/gcc as well as Linux/kcc and Linux/kcc --opt builds.

Along the way I also took a look inside the Pool module and I have a couple
of comments.

Comments within Pool state that Pool "maintains a set of page-sized chunks
of memory".  Currently, the page sized used by the implementation is 4096-8.

I noted that the grow() function makes no special effort to allocate
page-sized chunks on VM page boundaries.  It just calls new
char[alloc_this];  Perhaps the implementation is relying on certain C++
platforms to return near-page-size allocations on VM boundaries on best
effort?

In additional to its potential impact on VM behaviour, there is also the
issue of virtual memory TLB entries (translation lookaside buffers).  I've
seen code (primarily x86 platforms) where TLB thrash was a significant
performance factor.  Enforcing true VM page alignment can also benefit TLB
locality.

The forced scalar alignment in Pool is currently on 8 byte boundaries, which
Pool describes as "align on double words".  I believe there are now some
architectural scalar types where a 16-byte alignment is desirable, if not
mandatory (SSE on x86, extended floats on IA-64).  Ultimately, this should
be under the control of an allocator policy type.

Going one more level up, there are some situations where you are best served
by imposing alignment on cache-line boundaries.

For example, on an architecture with 64 byte cache lines, objects of size ~=
64-8 will average two full cache line fetches when accessed, and the average
cache line will contain portions of two different blocks (which can become
painful if the Pool is allowed to contain objects capable of being run
concurrently under different physical threads of execution).

Perhaps the best thing to do is note some of these issues in Pool.h for
attention the next time the surgeon cuts.

Allan