Isotropix Forums

Inconsistent Missing Textures in Renders

General Discussion about Isotropix and CG related topics

Inconsistent Missing Textures in Renders

Unread postby nareineru » Fri Jan 11, 2019 8:54 am

Hello.
We are currently experiencing an issue with our renders: some assets in rendered frames are missing textures which definitely should not happen.
We have an ongoing project which has hundreds of textures per scene. When tested locally, the problem never happens, but when we render through our farm, about a frame out of ten frames suffer from this issue. The appearance of this issue is inconsistent and is not just occurring from certain render farm nodes. We could observe the issue from the beginning of the project and the frequency of occurrence is getting bigger and bigger. The issue is quite critical since the final render of the project could take about 20 hours per frames and each shot could have more than ten thousands of frames to render, so I really wish we could solve the problem. Any ideas and suggestions would be appreciated.
Thanks.
nareineru
 
Posts: 16
Joined: Tue Feb 06, 2018 10:48 am

Re: Inconsistent Missing Textures in Renders

Unread postby sam » Fri Jan 11, 2019 9:38 am

Hi there,

This is indeed very odd. Could you please give share more details with us? Does this happens with Map files or Stream Map Files or both? Does it happens on all type of texture maps or just on specific channels such as displacements for example? Do you specify a config file to your render nodes through the command line? What is the size the texture cache size specified in the config file? Have you monitored the network IO? Have you checked the render logs on bad frames to spot any error messages?

Thanks!
Sam Assadian
Isotropix
CEO/Founder
User avatar
sam
 
Posts: 1379
Joined: Fri Jan 25, 2013 11:33 pm

Re: Inconsistent Missing Textures in Renders

Unread postby mdkai » Fri Jan 11, 2019 9:56 am

Hi
What Sam is pointing out already .
Verify if there is any pattern like memory overhead or an issue with the farm or server.
Are you using deadline royalrender or any other package ?
Verify if blades would render the same frame twice with the same issue or if it is sporadic.
This does sound very much like an IO issue suffering from server slowdown.
Can you give some specifics about memory usage per frame ?
Are the jobs running one per blade or do they run 2 or more at the same time , are they virtualised/split ?

Cheers
Kai
mdkai
 
Posts: 260
Joined: Tue Oct 07, 2014 7:24 pm

Re: Inconsistent Missing Textures in Renders

Unread postby nareineru » Mon Jan 14, 2019 10:43 am

Thank you for the quick response!

1. Does this happens with Map files or Stream Map Files or both?
- We are using the Stream Map File for the project.

2. Does it happens on all type of texture maps or just on specific channels such as displacements for example?
- It usually gives pink colored objects in a render, and we also have seen wrong displacement map render.
An interesting thing is: for example when we scatter about a few hundreds of asset A, only a few asset A objects in scatterer drops texture, not a whole scattered asset A in some frames.


3. Do you specify a config file to your render nodes through the command line? What is the size the texture cache size specified in the config file?
- We did not specify a config file through the command line, but we did add texture cache option, which can be changed freely based on a scale of the project.


4. Have you monitored the network IO? Have you checked the render logs on bad frames to spot any error messages?
- We did not monitor the network IO, but when we tried rendering a simple scene, the render did not suffer from the issue. As the scene gets bigger, the more the issue occurs, it seems.

Here is some log.


Images : 7399 unique
ImageInputs : 116079 created, 100 current, 141 peak
Total pixel data size of all images referenced : 346.7 GB
Total actual file size of all images referenced : 70.4 GB
Pixel data read : 6.4 GB
File I/O time : 15m 47.8s (16.9s average per thread)
File open time only : 1m 10.8s
Tiles: 488943 created, 488924 current, 488924 peak
total tile requests : 39405608569
micro-cache misses : 1880265275 (4.77157%)
main cache misses : 475503 (0.00120669%)
redundant reads: 6 tiles, 76 KB Peak
cache memory : 6.5 GB
1708 were exact duplicates of other images
56 not tiled, 56 not MIP-mapped
45 were constant-valued in all pixels
Top files by bytes read:
1 46.4 MB ( 0.7%) 4096x4096x4.u8 .../tx/4k/v01/chinahouseHighA_bump.1001.tx MIP-COUNT[1705,918,25 6,64,16,4,1,1,1,1,1,1,1]3
...

Top files by I/O time:
1 11.8s ( 1.2%) 1024x1024x3.u8 .../tx/1k/v01/chinaHouseDPillar_diff.1011.tif UNTILED UNMIPPED MIP- COUNT[16,64,16,4,1,1,1,1,1,1,1]
...

Files with slowest I/O rates:
1 0.00 MB/s (0.00MB/0.69s) 8x 8x4.u8 .../tx/4k/v01/ground/bjmGround_maskA.1193.tx DUPLICATES .../tx/4k/v01/ground/bjmGround_maskA.1197.tx
...

Top files by redundant I/O:
1 0.0 MB (31.6%) 1024x1024x3.u8 .../tx/1k/v01/chinaHouseMCement_spec.1012.tx MIP-COUNT[3,4,5,4,1,1,3 ,1,1,1,1]
...

Broken or invalid files: 1
1 .../tx/4k/v01/cacCrowdACloth_nor_v01.1001.tx



5. Verify if there is any pattern like memory overhead or an issue with the farm or server.
- We are currently working with system team to investigate if there were any sign of memory overhead on farm or storage server.



6. Are you using deadline royalrender or any other package ?
- We are currently running Tractor on Linux based servers.


7. Verify if blades would render the same frame twice with the same issue or if it is sporadic.
- Each time when we give the exact frame render to the exact blade, it gives a frame with different textures missing or a correctly rendered frame.

8. This does sound very much like an IO issue suffering from server slowdown.
Can you give some specifics about memory usage per frame ?
- We did not encounter any performance slowdown of servers. The issue persisted even when the farms were mostly in idle.
We have blades with 128GB, 256GB rams allocated. The artists mostly utilize 256GB blades.

Here is a log that shows memory usage.


Memory
------
application = 19.1 MiB
scene = 1.4 GiB
resources = 36.1 GiB
embedded resources = 4.3 GiB
---------------------
total = 40.6 GiB
peak = 40.6 GiB
---------------------
heap free = 1.5 GiB
unaccounted = 1.6 GiB
---------------------
available = 206.4 GiB

Application
-----------
total memory size = 19.1 MiB
core = 19.1 MiB
image history = 0.0 B
widgets = 0.0 B

Scene
----- memory size = 1.4 GiB
objects = 39.6 MiB
attributes = 593.7 MiB
modules = 813.6 MiB
factory = 6.8 MiB
---------------------
object count = 46558
attribute count = 3776353
local attribute count = 842711
attribute value count = 1800434

Resources
---------
memory size = 36.1 GiB
geometries = 16.7 GiB
accel structures = 5.3 GiB
images = 2.9 GiB
streamed texture cache = 6.5 GiB
texture cache = 41.0 B
misc = 4.7 GiB

9. Are the jobs running one per blade or do they run 2 or more at the same time , are they virtualised/split ?
- We are currently running one job per blade.

Thanks.
nareineru
 
Posts: 16
Joined: Tue Feb 06, 2018 10:48 am

Re: Inconsistent Missing Textures in Renders

Unread postby desmond » Mon Jan 14, 2019 2:06 pm

If i may chime in between the pros:

Let your devops check network and disks/shares mounting. If mounts are not mounted all the time you can have this kind of issue.

Disk IO also could be culprit but it seems you have hefty network there(from reading your posts - i can only deduce that blades with 256gb ram should have at least 10gbit network card). How many blades are you running / networkbandwith = bandwith per machine. you could also run them in a cascade system, so delay the job start on every other machine by the loading time of 1 machine (this way you speed up loading but renderes longer.).
Also specify a config file if youre rendering with cnode. Good luck!
User avatar
desmond
 
Posts: 47
Joined: Mon Feb 05, 2018 8:46 am

Re: Inconsistent Missing Textures in Renders

Unread postby mdkai » Tue Jan 15, 2019 9:53 am

Another thing to consider is, if the department's share parts of the server ..for example, if there is a sim department and caches are generated or read in, this could also cause high I/O stress to the server.

A desperate move to see if this is indeed related to clarisse rather then network/IO, is to copy all dependencies and source files to a SSD drive and mount it as local drive to one or multiple blades for testing
Then have the blade(s) start a job using a localised version.
mdkai
 
Posts: 260
Joined: Tue Oct 07, 2014 7:24 pm

Re: Inconsistent Missing Textures in Renders

Unread postby sam » Tue Jan 15, 2019 10:19 am

When a stream map file returns pink (the color can be changed BTW) it means that it failed to find the texture on disk. So that's clearly I/O related as Clarisse fails to pull the map from the server from some reason.

Are you sure there isn't any other errors in CNode render logs (you just sent the OIIO logs)? By the way when the log says "56 not tiled, 56 not MIP-mapped" it's usually not really efficient.

Thanks
Sam Assadian
Isotropix
CEO/Founder
User avatar
sam
 
Posts: 1379
Joined: Fri Jan 25, 2013 11:33 pm


Return to General Discussion