IIRC from last time I tested this (please don't trust my memory 100%), it was a substantial per-thread overhead issue.
I have not combed through the IlmImf code to verify this, but what it smelled like to me is if each thread in the pool has a retained buffer that it uses as scratch space for the decoding/uncompresion, data format conversions, and so on. (You know, you have to read the compressed data into some buffer, and then perform the decompression into another buffer... that sort of thing.) I get the feeling that each thread in the pool has its own area for this, and it's probably quite large so that it can do these decode/conversions on a big batch of pixels at once.
Soren, I think it would be worth repeating your experiment with the renderer and OpenEXR set to use just one thread (or maybe 1, 2, 4, ...). Obviously, it's not quite apples-to-apples because the access pattern will be totally different as you change thread counts. But if you see this "unaccounted overhead" in the process size growing steadily with the number of threads (while the number of textures you access and the maximum open at once stays the same), then that tells you something.
Do you know how many threads you were using?
Remember, also, that Soren is not comparing ImageCache to *nothing*, he's comparing ImageCache with exr files to the same ImageCache with tiff files, rendering the same frame with the same texture access patterns. This really is a direct test of speed and overhead of the two file format libraries.
Thanks, Larry! My EXR textures are already mipmapped and tiled so I don't think we're using any extra memory on auto-tile, but it certainly was true that we were using OIIO with the default 100 file handle cache size. We'll give it a go with a much higher value.
I'm still surprised that libIlmImf requires that much more memory. If our handle cache size was limited to 100 files at a time, that's an extra 38.75 MB per texture. Even if it's caching all 1957 texture handles at once that's still an extra 1.98 MB per texture.
_______________________________________________ Openexr-devel mailing list address@hiddenhttps://lists.nongnu.org/mailman/listinfo/openexr-devel
|