[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Openexr-devel] EXR texture memory overhead
From: |
Alexandre |
Subject: |
Re: [Openexr-devel] EXR texture memory overhead |
Date: |
Fri, 16 Sep 2016 15:13:23 +0200 |
I think the bottleneck is in OpenImageIO's ImageCache rather than OpenEXR by
itself.
I’ve spent quite some time debugging OpenImageIO in this regard. The worst case
scenario you can give to OpenImageIO is when trying to read untiled
multi-layered EXR files.
Most of people seem to only be working with zip scanlines because this suits
Nuke scan-line architecture perfectly but it is a nightmare in reality for all
other applications that don’t work with scan-lines.
The OpenImageIO cache can be set in auto-tile mode, in which case it will
open/close the file multiple times to decode (so it is slower) but can use less
memory because it doesn’t require to allocate as much big chunks of memory.
When not set to auto-tile it will just decode the full image, meaning that
OpenEXR will allocate a big chunk of memory to decompress, OpenImageIO will
allocate a big chunk of memory to convert to the user requested data format.
And here is the worst part, OpenImageIO will leave the file opened in the cache
on the thread local storage of the calling thread.
And you might even go worse than that if you’ve got multiple threads trying to
decode different untiled EXR files concurrently, then OpenImageIO will just
blow up your computer’s RAM + swap and the kernel should kill your app very
quickly.
There are a couple of workarounds:
- Make all your files go through an initial pass of converting them to tiled
EXR files (with exrmaketiled)
- Don’t use OpenImageIO cache at all
The Foundry has come up with an extension (in a pull request) to let a chance
to the application calling OpenEXR to pass its own buffers (instead of the
ones used internally) so that the decompression of the EXR files could happen
outside of OpenEXR itself in the calling application controlled memory and
threads.
This is very important if your application is going to do other stuff
concurrently than just reading your 1 EXR file.
On our side we are going to try and implement that in OpenImageIO so that in
the same way you could pass your own buffers and threads to OpenImageIO which
would in turn pass them to OpenEXR.
> On 16 Sep 2016, at 12:34, Søren Ragsdale <address@hidden> wrote:
>
> Hello, OpenEXR devs. I've been doing some comparative rendering tests I've
> found something a bit surprising.
>
> TIFF and EXR texture access *times* seems more or less the same, which is
> fine because the underlying data is equivalent. (Same data type, compression,
> tile size, etc.) But the RAM overhead seems much higher for EXRs. We've got a
> 9GB render using TIFFs and a 13GB render using EXRs.
>
> Does anyone have some theories why EXR texture access is requiring 4GB more
> memory?
>
>
> Prman-20.11, OSL shaders, OIIO/TIFF textures:
> real 00:21:46
> VmRSS 9,063.45 MB
> OpenImageIO ImageCache statistics (shared) ver 1.7.3dev
> Options: max_memory_MB=4000.0 max_open_files=100 autotile=64
> autoscanline=0 automip=1 forcefloat=0 accept_untiled=1
> accept_unmipped=1 read_before_insert=0 deduplicate=1
> unassociatedalpha=0 failure_retries=0
> Images : 1957 unique
> ImageInputs : 136432 created, 100 current, 796 peak
> Total size of all images referenced : 166.0 GB
> Read from disk : 55.5 GB
> File I/O time : 7h 2m 33.9s (16m 54.2s average per thread)
> File open time only : 27m 44.0s
>
>
> Prman-20.11, OSL shaders, OIIO/EXR textures:
> real 00:21:14
> VmRSS 12,938.83 MB
> OpenImageIO ImageCache statistics (shared) ver 1.7.3dev
> Options: max_memory_MB=4000.0 max_open_files=100 autotile=64
> autoscanline=0 automip=1 forcefloat=0 accept_untiled=1
> accept_unmipped=1 read_before_insert=0 deduplicate=1
> unassociatedalpha=0 failure_retries=0
> Images : 1957 unique
> ImageInputs : 133168 created, 100 current, 771 peak
> Total size of all images referenced : 166.0 GB
> Read from disk : 55.5 GB
> File I/O time : 6h 15m 42.1s (15m 1.7s average per thread)
> File open time only : 1m 22.5s
>
> _______________________________________________
> Openexr-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/openexr-devel
- [Openexr-devel] EXR texture memory overhead, Søren Ragsdale, 2016/09/16
- Re: [Openexr-devel] EXR texture memory overhead, Karl Rasche, 2016/09/16
- Re: [Openexr-devel] EXR texture memory overhead,
Alexandre <=
- Re: [Openexr-devel] EXR texture memory overhead, Larry Gritz, 2016/09/16
- Re: [Openexr-devel] EXR texture memory overhead, Karl Rasche, 2016/09/16
- Re: [Openexr-devel] EXR texture memory overhead, Larry Gritz, 2016/09/16
- Re: [Openexr-devel] EXR texture memory overhead, Karl Rasche, 2016/09/16
- Re: [Openexr-devel] EXR texture memory overhead, Larry Gritz, 2016/09/16
- Re: [Openexr-devel] EXR texture memory overhead, Ed Hanway, 2016/09/16
- Re: [Openexr-devel] EXR texture memory overhead, Alexandre, 2016/09/16
- Re: [Openexr-devel] EXR texture memory overhead, Larry Gritz, 2016/09/16
- Re: [Openexr-devel] EXR texture memory overhead, Alexandre, 2016/09/16
- Re: [Openexr-devel] EXR texture memory overhead, Larry Gritz, 2016/09/16