Isotropix Forums

We are running some testing for EXRs rendered from Clarisse 4.0SP1

Here are some stats, rendering @3424x2202 and using 23 AOVs (PBR, Cryptomatte and 2 Utilities):

EXR (ZIPS compression) - 300mb per frame // single AOV using split AOV preference
EXR (ZIPS compression) - 750mb per frame // Multi-channel exr
EXR (ZIPS compression) - 160mb per frame // Multi-channel exr // Half res

*ZIPS compression I've found unusable in Nuke, it will crash or be extremely slow to use

EXR (ZIP compression) - 32mb per frame // single AOV using split AOV preference
EXR (ZIP compression) - 370mb per frame // Multi-channel exr

*Going above roughly 8 AOV's becomes sluggish in Nuke, as it does with any large EXRs. As there are so many AOVs needed for cryptomatte we have split our renders into two, one containing crypto and utilities, the other containing the beauty and PBR AOVs

I am not sure if you have a specific question but to comment on your statement about large EXRs containing lots of AOVs:

The reason why reading these is slow and unstable is because Clarisse (like most renderers, to be fair) writes multi-channel EXRs without splitting the AOVs into subimages, which is a feature which was introduced with OpenEXR 2.0, often referred to as multipart-exr. When using subimages, only the specific AOVs you want to view or use at a time has to be read, instead of the whole file.

I opened a feature request on this topic [8723] quite a while ago. A workaround for the time being would be to set up a post-render process to split the channels into subimages, e.g. with OpenImageIO. After doing that, reading these EXRs is just as fast as reading a single-AOV sequence.

Thanks for the extra information, good to know about the OIIO trick, I'll have a look into that today.

I was just testing yesterday and thought I'd post my findings on here.

Another thing I found is that the cryptomatte AOVs alone (with rgba being solid black) come to a huge 500mb a frame, compared to the beauty and it's AOVs coming in at around 130mb a frame.

Maybe it's just due to our huge amount of assets in one scene driving the amount of data up so much.

Anyway, thanks again mate

This might be caused by a scatterer with lots of instances in conjunction with the Object Cryptomatte, since every single instance will get its own entry there: viewtopic.php?f=5&t=5084

Yea that's exactly what it is then, is the metadata needed for the cryptomatte do you know? Maybe when doing the OIIO rebuild of the EXR we could strip out the metadata that's not needed, to further lighten them?

Metadata are needed if you want to make masks by rules (script). If you have regular use of cryptomatte and create your masks by picking colors you don't need those meta's.

Cheers

I think the linked thread gave the wrong impression that the metadata alone is responsible for these huge file sizes while in fact the actual image data (in the Cryptomatte Object AOV) which has to encode every single object as well increases even more in size. I did some tests of a scenario with lots of instances in the past where I compared the size of the metadata with the image data and the latter would always be more relevant. I can't recall the precise numbers but of an roughly 250MB Cryptomatte Object EXR, only 30MB are metadata.

If we are talking performance, the metadata is indeed a huge problem regardless of their size, since although it is possible to attach metadata only to specific subimage, the whole metadata will always be read/decoded regardless, which leads to a huge drop in performance in the described scenario.

Mmmm, just made the test here and I have this : (Exr, no compression, removing metadata in nuke and rewrite the image.)



In this example, meta's are quite big...

Interesting, could you elaborate a bit on what the test consists of, like the number of visible instances in the render and its resolution or maybe even share the scene file? How does it compare with compression turned on?

Did you make sure to write out 32bit images when cutting the metadata in Nuke?

I didn't make the test with compression ON. But here is the image I rendered and the settings :



3.9 Millions geometries, perhaps half in screen.

Cheers