Isotropix Forums

Exporting property values as double

Clarisse Scripting related topics

Re: Exporting property values as double

Unread postby jboissinot » Wed Apr 10, 2019 11:41 pm

Hi,

Yes we did find out together about Katana might not be supporting float64 matrix4x4. I remember doing some tests with the ResourceProperty API, which was importing values with exponential format if exported in float64 while it seems to be working well with the IOHelpers API, so that's good.

Now, to go back to my question, I actually convert the matrix values into a string following your recommendation with IOHelpers. However, I was wondering why those values were interpreted as matrix if stored as strings in a FLOAT_64 property type? I thought they would be read like a custom array of values as doubles.

As such, I was wondering if it'd be possible to store 16 double values in a property of size 1 per point index, like we can do for strings for instance.


What I meant by this is actually to store a custom array of 16 values per point index, and was expecting to be able to store this array of 16 values per point in a property of size 1, while it automatically increments the point index while storing each value of the array.

Thanks,
Jeremy
jboissinot
 
Posts: 52
Joined: Tue Jan 29, 2019 10:36 pm

Re: Exporting property values as double

Unread postby anemoff » Thu Apr 11, 2019 6:06 pm

If you created a property with dimension 16 and type TYPE_FLOAT_64, then it's totally normal that the property is interpreted as a m44d.

About the strings: IOHelpers::edit_particles_property uses strings as input type to set the property values.
Under the hood, it converts the string values to the actual type of the property, in this case, doubles.
The strings are never stored in the property, it's just a convenient way to have a single entry function to set values, regardless of the underlying property type.

If understood correctly, what you want to do is this: for each point on the point cloud, store an array of 16 doubles (ie. its matrix).
The simplified steps to do this is:
- create the property, with type TYPE_FLOAT_64, and with dimension 16 (ie. the value of the property on each point will have 16 values, therefore there will be 16 x N values in the property)
- get the matrix array for each point, convert it to string
- iterate over the number of points in your point cloud: for each point set the stringified matrix value for the corresponding point index

Here is an extract of an example (see the attached zip file for the whole project):

python code

# get the particle container and its point count
container = ix.get_item("project://scene/context/container")
point_count = int(container.get_module().get_point_count())

# create the point property values: as many matrices as points in the particle container, and put the values in a string array
values = []
for i in range(point_count):
m44d = create_dummy_m44d(True)
values.append(m44d_to_string(m44d))

# create the point indices: [0, 1, 2, ...]
indices = range(point_count)

# create the m44d property (dim 16, float64)
prop_name = 'm44_float64'
create_point_property(container, prop_name, ix.api.ResourceProperty.TYPE_FLOAT_64, 16)

# set the values on the associated point indices
set_point_property_indexed_values(container, prop_name, values, indices)
print ''

# print them for debug
print_point_property_indices(container, prop_name)
print ''
print_property_values(container, prop_name, ix.api.ResourceProperty.TYPE_CHAR)
print ''


I've also changed the way it converts the matrix to a string, to get more decimals (by default Python keeps few decimals when converting a floating number to string)

python code

def m44_to_string(m44, separator = ' '):
"""
Get the GMathMatrix4x4d flattened values as a string separated with the separator.
"""
s = ''
for i in range(4):
for j in range(4):
s += '{:.16f}'.format(m44d.get_item(i, j)) # 16 decimals
if j < 3: s += separator
if i < 3: s += separator
return s


Does this help?

I'm working on seeing what improvements should be done to make the APIs more simpler and friendlier
Attachments
export_matrix_property_v4.zip
(9.49 KiB) Downloaded 41 times
Anthony Nemoff
Isotropix
R&D Engineer
User avatar
anemoff
 
Posts: 191
Joined: Wed Jan 13, 2016 10:10 am

Re: Exporting property values as double

Unread postby jboissinot » Thu Apr 11, 2019 9:07 pm

Hi Anthony,

Yes that's right, if I refer to the standard Alembic formats you sent in this topic, a property with type TYPE_FLOAT_64 and dimension 16 will be exported as a matrix4x4, which is not supported in Katana.

As a workaround, we exported all the matrix values in a TYPE_FLOAT_64 property with dimension 1. This method works for Katana but not for Houdini as the values are not stored by 16 per point.

The goal is to find a way that will work for both. What you sent is actually what I did already, which doesn't work with Katana. The idea was to use the same method but with a property with dimension 1, as we do for a string property with dimension 1 in which we can store strings of different lengths per point. This is actually what we do in Houdini, we store the 16 matrix values per point in a float64 property with dimension 1, and this works for Katana. So, I was trying to find a way to match what we do in Houdini, but when I set the property with dimension 1, I'm getting a set values error, like it's not able to store 16 values per point in a property with dimension 1.

Do you think this is something that we could do? Let me know your thoughts on this.

Thanks,
Jeremy
jboissinot
 
Posts: 52
Joined: Tue Jan 29, 2019 10:36 pm

Re: Exporting property values as double

Unread postby sam » Fri Apr 12, 2019 12:38 am

Hi Jeremy,

Have you tried to contact Foundry support to actually implement the Alembic standard in Katana? This seems to me to be the ideal solution as it already works on Houdini and Clarisse.
Sam Assadian
Isotropix
CEO/Founder
User avatar
sam
 
Posts: 1418
Joined: Fri Jan 25, 2013 11:33 pm

Re: Exporting property values as double

Unread postby anemoff » Fri Apr 12, 2019 11:26 am

Hi,

By definition, you can store only 1 value per point if the property dimension is 1.
So no, you can't store all the 16 values of a matrix4x4, per point, in a property with dimension 1.
I suppose this explains the set values errors you get.

A first idea is to use 4 properties of dimension 4, to store 1 matrix row per property.
I hope that Katana supports vec4d. If so, then you would be able to read the 4 properties and rebuild your matrix.
I have successfully imported such properties in Houdini. They appear as Quaternions (vec4d).
I have attached a new project and scripts, and the resulting abc file.

A second idea is to create a property of dimension 17 but only use the first 16 values. This way the property won't be interpreted as a matrix in Katana, and maybe it will work. It's up to you then to read the property in Katana and keep only the first 16 values and put them in a matrix.
However, I have tested to read such property in Houdini and, as expected, it can't read it because of its non-standard size...

If this still doesn't help I think the best solution would be to schedule a live chat or video chat so that we can discuss.
Please contact us by email.

Cheers,
Attachments
export_matrix_property_v5.zip
(12.31 KiB) Downloaded 43 times
Anthony Nemoff
Isotropix
R&D Engineer
User avatar
anemoff
 
Posts: 191
Joined: Wed Jan 13, 2016 10:10 am

Re: Exporting property values as double

Unread postby jboissinot » Fri Apr 19, 2019 9:59 pm

Hi guys,

Sam,

I haven't contacted Foundry support yet as I was still investigating on the ability of storing 16 values per point in a property of dimension 1, and also because the float64 matrix support is not such a big issue as we don't really need to get those values as a matrix object for instancing in Katana. That being said, this could be an option to see if they can implement it.

Anthony,

I agree with what you say about storing multiple values in a property of dimension 1. We already tried storing the matrix values in a property of dimension 4 but this was not working either in Katana. From our testing, it was only working with a dimension from 1 to 3. Anyway, storing the matrix values in one single property instead of combining 4 properties was a better option for us performance-wise.
I understand the hack with using a property of dimension 17 but this will be quite odd and not really clean data in the workflow and may break the data read as you say.

To go back to the error I was getting, when I used the set_double() method of a ResourceProperty with a given item index and array of doubles, I was expecting it to actually store the array of values in the given item index of a property of dimension 1, while it automatically increments the item index when storing the values of the array.

In Houdini, we are able to store 16 values in a float64 type property of dimension 1, and this works in Clarisse and Katana.
When I check the attributes data after unpacking the Alembic node, it confirms that it's not a matrix object but a custom array of 16 values.
I will send you an email with this so you can have a better look at it.

Thanks again for your support and help, hoping that I sent you more useful info that may help us on this and let us know if that'd be possible to store the matrix values with the same approach that we use in Houdini.

Jeremy
jboissinot
 
Posts: 52
Joined: Tue Jan 29, 2019 10:36 pm

Re: Exporting property values as double

Unread postby anemoff » Wed Apr 24, 2019 5:46 pm

jboissinot wrote:To go back to the error I was getting, when I used the set_double() method of a ResourceProperty with a given item index and array of doubles, I was expecting it to actually store the array of values in the given item index of a property of dimension 1, while it automatically increments the item index when storing the values of the array.

Actually, there's an undocumented safeguard when using the function set_double that takes an array: it will take at most "item_value_count" (aka size or dimension) values from the input array. This is the expected behavior: the safeguard is there to avoid assigning to many values on one item.
Therefore if you created a ResourceProperty with item_value_count = 1, it will only take the 1st value of the array and ignore all others because there's no room for them. It doesn't increment the size of each value.

Here is a small example that demonstrates the usage and that I hope clarifies the API:

python code

# create a ResourceProperty of size 1 and count 3 (e.g. 1 float value per point, for 3 points)
rp = ix.api.ResourceProperty('test')
rp.init(ix.api.ResourceProperty.TYPE_FLOAT_64, 1, 3)

# print property info
print 'item count (count): {}'.format(rp.get_item_count())
print 'item value count (size): {}'.format(rp.get_item_value_count(0))
print 'value count (= item count x item value count): {}'.format(rp.get_value_count())

# dummy value to be assigned, bigger than the property size, intentionally
fake_matrix = ix.api.DoubleArray(16)
for i in range(16):
fake_matrix[i] = i
print 'fake_matrix[{}] = {}'.format(i, fake_matrix[i])

# set the values on each item:
# only the matrix value at index 0 is assigned, the rest is ignored because they don't fit
for i in range(rp.get_item_count()):
rp.set_double(i, fake_matrix)

# print each item value
for i in range(rp.get_item_count()):
value = rp.get_double(i) # in this case, "value" is an array of 1 float because size is 1, and not 16 as expected after assigning the fake matrix
str = ''
for j in range(rp.get_item_value_count(i)):
str += '{} '.format(value[j])

print 'actual_values[{}]: {}'.format(i, str)

print ''


jboissinot wrote:In Houdini, we are able to store 16 values in a float64 type property of dimension 1, and this works in Clarisse and Katana.

OK, that answers one of my questions from the email. I was too fast to reply to the email before re-reading this post.
I'm still interested in the abcecho output from the property created by Houdini, in order to see if we can make Clarisse export its properties with the same structure.

To sum up, can you confirm that is what you have?
- Export m44d from Houdini: OK in Clarisse, OK in Katana
- Export m44d from Clarisse: OK in Houdini, NOT OK in Katana

For now, I don't have a solution to make the Alembic exported by Clarisse work in Katana.
But the abcecho of the Houdini export might help understand why.
Anthony Nemoff
Isotropix
R&D Engineer
User avatar
anemoff
 
Posts: 191
Joined: Wed Jan 13, 2016 10:10 am

Re: Exporting property values as double

Unread postby jboissinot » Thu Apr 25, 2019 11:41 pm

Hi Anthony,

So we did clarify this confusion from the email I sent you and do need to store the matrix values in a property of size 16 after all.

To sum up, can you confirm that is what you have?
- Export m44d from Houdini: OK in Clarisse, OK in Katana
- Export m44d from Clarisse: OK in Houdini, NOT OK in Katana


To clarify, this should be more like this actually:
- Export m44d from Houdini: OK in Clarisse, NOT OK in Katana
- Export m44d from Clarisse: OK in Houdini, NOT OK in Katana

As discussed, Katana doesn't seem to be supporting float64 m44d, so exporting them from Houdini doesn't work either.
As explained in my email, in Houdini, we actually export the matrix values in a float64 property but not as a matrix object.

So, the idea behind this was to be able to export float64 properties with a size of 16 as custom array of 16 doubles values and not matrix objects.
I think this is actually something we might have discussed in our emails:

Clarisse properties don't have "interpretations" (vec2, vec3, m33, m44, ...) attached to them. They are just flat arrays of values with information (name, type, number of values, etc).
However when exporting Clarisse properties to Alembic, if we see properties with information that matches what Alembic calls "geometry parameters" (.arbGeomParams), then we will export them as such.
Properties that don't match geometry parameters patterns are exported as "user properties", but many applications don't support these. Clarisse does.
For increased compatibility with other apps, we try to export properties as geometry parameters as much as possible. But currently, you can't choose which mode to use for each property.
This means that any property with 16 float32 or float64 values will be exported as matrices, and applications will read them as such,

For example:
- a property of type float 32 with size 3 it will be exported as a geometry parameter "vec3f"
- a property of type float 32 with size 9 it will be exported as a geometry parameter "m33f"
- a property of type float 64 with size 16 it will be exported as a geometry parameter "m44d"
- but an exotic property of type int16 with size 10 will be exported as a user property "flat array of 10 x int16"


I think this is it indeed when we talked about exporting properties with a custom mode.

Thanks,
Jeremy
jboissinot
 
Posts: 52
Joined: Tue Jan 29, 2019 10:36 pm

Re: Exporting property values as double

Unread postby anemoff » Fri Apr 26, 2019 4:51 pm

jboissinot wrote:To clarify, this should be more like this actually:
- Export m44d from Houdini: OK in Clarisse, NOT OK in Katana
- Export m44d from Clarisse: OK in Houdini, NOT OK in Katana
As discussed, Katana doesn't seem to be supporting float64 m44d, so exporting them from Houdini doesn't work either.
As explained in my email, in Houdini, we actually export the matrix values in a float64 property but not as a matrix object.

That's good to know.
So Houdini to Katana works if the property is a simple array of 16 values instead of a matrix, right?
And Katana reads it correctly read as float64 too? (i.e. float64 is indeed supported by Katana but with limitations?)

I don't think we can provide a workaround without doing some development in Clarisse in order to provide more export options and be compatible with Katana at the cost of losing the property interpretation (which doesn't seem essential).

I'll keep you posted.
Anthony Nemoff
Isotropix
R&D Engineer
User avatar
anemoff
 
Posts: 191
Joined: Wed Jan 13, 2016 10:10 am

Re: Exporting property values as double

Unread postby jboissinot » Mon Apr 29, 2019 2:50 pm

Hi,

So Houdini to Katana works if the property is a simple array of 16 values instead of a matrix, right?

Yes, that's the workaround we use to get values properly readable as doubles in Katana.
Katana does read correctly the float64 values of properties of size 1 or vec3, while they are not interpreted as doubles with quaternion, and simply not supported with matrix4x4 from out testing.

That's why we came up with using a simple array of 16 values instead of a matrix, also because we don't really need these values to be exported as matrix to be working in Katana.

Thanks,
Jeremy
jboissinot
 
Posts: 52
Joined: Tue Jan 29, 2019 10:36 pm

Previous

Return to Scripting
cron