Re: [Openexr-devel] EXR texture memory overhead

[Søren Ragsdale]

Thanks, Larry! My EXR textures are already mipmapped and tiled so I don't think we're using any extra memory on auto-tile, but it certainly was true that we were using OIIO with the default 100 file handle cache size. We'll give it a go with a much higher value.

I'm still surprised that libIlmImf requires that much more memory. If our handle cache size was limited to 100 files at a time, that's an extra 38.75 MB per texture. Even if it's caching all 1957 texture handles at once that's still an extra 1.98 MB per texture.

On Fri, Sep 16, 2016 at 5:56 PM, Larry Gritz <address@hidden> wrote:

See how there were 1957 unique textures, but you created 144168 ImageInputs? That means your file (handle) cache was too small, you were accessing the files in an incoherent order and the "working set" was much larger than the file cache size (100), so you ended up closing and reopening files a massive number of times (73 times each, on average).

The solution to this is to raise the TextureSystem's (that is, the underlying ImageCache's) "max_open_files" significantly. The default of 100 is extremely conservative -- trying to keep you out of trouble on operating systems with a low handle count per process, or with ImageCache's file operations being potentially only a small portion of your process's file needs.

On a modern Linux system, you should be able to have several thousand files open simultaneously. I would do that and see what it does to all your figures. At the very least, it should substantially lower the "file open time" and maybe the file I/O time overall.

But, it may make the memory issues worse -- if libIlmImf is internally holding more memory per open file than libtiff does (my tests indicate this as well), then having more files open at a time may exacerbate the problem. I haven't tested this for a while, though, but I think that Karl is correct that part of this is that each of the threads in OpenEXR's pool is holding a bit of working memory, and that adds up, so the higher your thread count, the more memory use overhead there will be for libIlmImf, and it's very hard for the app to account for that or control it.

It is simply a fact that on a per-thread, per-file basis, libIlmImf uses more overhead memory per open file than libtiff does. I have not looked into it deeply enough to know if that's a good design tradeoff or not (for example, maybe that memory is put to good use, and helps speed up some operations).

This is the kind of thing you would never notice reading OpenEXR files sequentially, but in something like a texture system where you may have thousands of files open simultaneously, an extra 256KB of overhead per file adds up fast.

> On Sep 16, 2016, at 3:34 AM, Søren Ragsdale <address@hidden> wrote:
>
> Hello, OpenEXR devs. I've been doing some comparative rendering tests I've found something a bit surprising.
>
> TIFF and EXR texture access *times* seems more or less the same, which is fine because the underlying data is equivalent. (Same data type, compression, tile size, etc.) But the RAM overhead seems much higher for EXRs. We've got a 9GB render using TIFFs and a 13GB render using EXRs.
>
> Does anyone have some theories why EXR texture access is requiring 4GB more memory?
>
>
> Prman-20.11, OSL shaders, OIIO/TIFF textures:
> real 00:21:46
> VmRSS 9,063.45 MB
> OpenImageIO ImageCache statistics (shared) ver 1.7.3dev
>   Options:  max_memory_MB=4000.0 max_open_files=100 autotile=64
>             autoscanline=0 automip=1 forcefloat=0 accept_untiled=1
>             accept_unmipped=1 read_before_insert=0 deduplicate=1
>             unassociatedalpha=0 failure_retries=0
>   Images : 1957 unique
>     ImageInputs : 136432 created, 100 current, 796 peak
>     Total size of all images referenced : 166.0 GB
>     Read from disk : 55.5 GB
>     File I/O time : 7h 2m 33.9s (16m 54.2s average per thread)
>     File open time only : 27m 44.0s
>
>
> Prman-20.11, OSL shaders, OIIO/EXR textures:
> real 00:21:14
> VmRSS 12,938.83 MB
> OpenImageIO ImageCache statistics (shared) ver 1.7.3dev
>   Options:  max_memory_MB=4000.0 max_open_files=100 autotile=64
>             autoscanline=0 automip=1 forcefloat=0 accept_untiled=1
>             accept_unmipped=1 read_before_insert=0 deduplicate=1
>             unassociatedalpha=0 failure_retries=0
>   Images : 1957 unique
>     ImageInputs : 133168 created, 100 current, 771 peak
>     Total size of all images referenced : 166.0 GB
>     Read from disk : 55.5 GB
>     File I/O time : 6h 15m 42.1s (15m 1.7s average per thread)
>     File open time only : 1m 22.5s
>

> _______________________________________________
> Openexr-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/openexr-devel

--
Larry Gritz
address@hidden