Isotropix Forums

EXR Performance

General Discussion about Isotropix and CG related topics

EXR Performance

Unread postby dane » Mon Apr 15, 2019 5:09 pm

We are running some testing for EXRs rendered from Clarisse 4.0SP1

Here are some stats, rendering @3424x2202 and using 23 AOVs (PBR, Cryptomatte and 2 Utilities):

EXR (ZIPS compression) - 300mb per frame // single AOV using split AOV preference
EXR (ZIPS compression) - 750mb per frame // Multi-channel exr
EXR (ZIPS compression) - 160mb per frame // Multi-channel exr // Half res


*ZIPS compression I've found unusable in Nuke, it will crash or be extremely slow to use

EXR (ZIP compression) - 32mb per frame // single AOV using split AOV preference
EXR (ZIP compression) - 370mb per frame // Multi-channel exr


*Going above roughly 8 AOV's becomes sluggish in Nuke, as it does with any large EXRs. As there are so many AOVs needed for cryptomatte we have split our renders into two, one containing crypto and utilities, the other containing the beauty and PBR AOVs
dane
 
Posts: 36
Joined: Wed Aug 29, 2018 11:50 am

Re: EXR Performance

Unread postby t_b_s » Mon Apr 15, 2019 5:57 pm

I am not sure if you have a specific question but to comment on your statement about large EXRs containing lots of AOVs:

The reason why reading these is slow and unstable is because Clarisse (like most renderers, to be fair) writes multi-channel EXRs without splitting the AOVs into subimages, which is a feature which was introduced with OpenEXR 2.0, often referred to as multipart-exr. When using subimages, only the specific AOVs you want to view or use at a time has to be read, instead of the whole file.

I opened a feature request on this topic [8723] quite a while ago. A workaround for the time being would be to set up a post-render process to split the channels into subimages, e.g. with OpenImageIO. After doing that, reading these EXRs is just as fast as reading a single-AOV sequence.
t_b_s
 
Posts: 18
Joined: Sat Dec 17, 2016 1:03 pm

Re: EXR Performance

Unread postby dane » Tue Apr 16, 2019 8:24 am

Thanks for the extra information, good to know about the OIIO trick, I'll have a look into that today.

I was just testing yesterday and thought I'd post my findings on here.

Another thing I found is that the cryptomatte AOVs alone (with rgba being solid black) come to a huge 500mb a frame, compared to the beauty and it's AOVs coming in at around 130mb a frame.

Maybe it's just due to our huge amount of assets in one scene driving the amount of data up so much.

Anyway, thanks again mate
dane
 
Posts: 36
Joined: Wed Aug 29, 2018 11:50 am

Re: EXR Performance

Unread postby t_b_s » Tue Apr 16, 2019 8:44 am

This might be caused by a scatterer with lots of instances in conjunction with the Object Cryptomatte, since every single instance will get its own entry there: viewtopic.php?f=5&t=5084
t_b_s
 
Posts: 18
Joined: Sat Dec 17, 2016 1:03 pm

Re: EXR Performance

Unread postby dane » Tue Apr 16, 2019 9:19 am

Yea that's exactly what it is then, is the metadata needed for the cryptomatte do you know? Maybe when doing the OIIO rebuild of the EXR we could strip out the metadata that's not needed, to further lighten them?
dane
 
Posts: 36
Joined: Wed Aug 29, 2018 11:50 am

Re: EXR Performance

Unread postby dboude » Tue Apr 16, 2019 9:41 am

Metadata are needed if you want to make masks by rules (script). If you have regular use of cryptomatte and create your masks by picking colors you don't need those meta's.

Cheers
Démian
Isotropix
Technical Artist - Clarisse Specialist
User avatar
dboude
 
Posts: 813
Joined: Mon Jul 03, 2017 10:51 am

Re: EXR Performance

Unread postby t_b_s » Tue Apr 16, 2019 12:17 pm

I think the linked thread gave the wrong impression that the metadata alone is responsible for these huge file sizes while in fact the actual image data (in the Cryptomatte Object AOV) which has to encode every single object as well increases even more in size. I did some tests of a scenario with lots of instances in the past where I compared the size of the metadata with the image data and the latter would always be more relevant. I can't recall the precise numbers but of an roughly 250MB Cryptomatte Object EXR, only 30MB are metadata.

If we are talking performance, the metadata is indeed a huge problem regardless of their size, since although it is possible to attach metadata only to specific subimage, the whole metadata will always be read/decoded regardless, which leads to a huge drop in performance in the described scenario.
t_b_s
 
Posts: 18
Joined: Sat Dec 17, 2016 1:03 pm

Re: EXR Performance

Unread postby dboude » Tue Apr 16, 2019 2:23 pm

Mmmm, just made the test here and I have this : (Exr, no compression, removing metadata in nuke and rewrite the image.)

cryptomatte_meta.png
cryptomatte_meta.png (5.25 KiB) Viewed 167 times


In this example, meta's are quite big...
Démian
Isotropix
Technical Artist - Clarisse Specialist
User avatar
dboude
 
Posts: 813
Joined: Mon Jul 03, 2017 10:51 am

Re: EXR Performance

Unread postby t_b_s » Tue Apr 16, 2019 3:39 pm

Interesting, could you elaborate a bit on what the test consists of, like the number of visible instances in the render and its resolution or maybe even share the scene file? How does it compare with compression turned on?

Did you make sure to write out 32bit images when cutting the metadata in Nuke?
t_b_s
 
Posts: 18
Joined: Sat Dec 17, 2016 1:03 pm

Re: EXR Performance

Unread postby dboude » Tue Apr 16, 2019 5:00 pm

I didn't make the test with compression ON. But here is the image I rendered and the settings :

cryptomatte_beauty.png


3.9 Millions geometries, perhaps half in screen.

Cheers ;)
Démian
Isotropix
Technical Artist - Clarisse Specialist
User avatar
dboude
 
Posts: 813
Joined: Mon Jul 03, 2017 10:51 am

Next

Return to General Discussion
cron