Isotropix Forums

Hi,

Unlike other property types that we can define with a constant size, we were wondering if it's possible to set the values in a char type property with strings of different lengths?

In fact, we noticed that the string values were not properly set while trying to store them with an object scene path or asset name for instance, which can have different lengths.
To fix this, we ended up filling the strings with spaces so that they fit in the char property size as a workaround but it's not the best.

The way the set_string() is built - from what I could read in the documentation - is that it stores every char of a string but doesn't work if the string length doesn't match the char property size.

Thanks,
Jeremy

Hi Jeremy,

jboissinot wrote:Unlike other property types that we can define with a constant size, we were wondering if it's possible to set the values in a char type property with strings of different lengths?

Yes, you can write multiple values in a single string property. But you will need to separate them with a character separator (space or ';' for example) to be able to split them back when reading them in the other application.
For example: "0.1234 1.23 0.000000003 5.45143 ..."
If you have lots of points and lots of values per point it will result in a huge string. If so, I would recommend converting the string to binary form (hexadecimal) when writing it, to save some memory.
And, there's also the problem that float or double values might be truncated when converted to string, losing precision.

jboissinot wrote:In fact, we noticed that the string values were not properly set while trying to store them with an object scene path or asset name for instance, which can have different lengths.
To fix this, we ended up filling the strings with spaces so that they fit in the char property size as a workaround but it's not the best.

The way the set_string() is built - from what I could read in the documentation - is that it stores every char of a string but doesn't work if the string length doesn't match the char property size.

- Are you creating the string property by script or using the Property Editor?
- What size are you giving to your string property when initializing it?
- ResourceProperty.set_string will truncate the given string to the size (item value count, aka dimension or extent) that was initially given when the property was created with "init". Resizing a value afterward through the ResourceProperty API is not straightforward.
- If possible I suggest using the API in IOHelpers.edit_particles_property, to set/edit values as I dit in my example. It will automatically resize each value to the size of the new value size, otherwise it's pretty heavy to do by yourself, especially by script.

Hi Anthony,

I'm creating the string property by script and set a size of 100 - but could be shorter, maybe 40 or 50 would be long enough - when initializing it as you can see in the following code snippet as an example:

Code: Select all

prop = createProperty('name')
prop.init(ix.api.ResourceProperty.TYPE_CHAR, 100, nbPoints)

Then, I simply set the property name value while parsing the points with the set_string() - with the name variable having a different length for each point for instance:

Code: Select all

prop.set_string(index, name)

But, I'm getting the following error when exporting the Alembic file:

Code: Select all

Alembic export: an error occured during the export process: OArrayProperty::set()
ERROR: EXCEPTION:
Illegal NULL character found in string data

I ended up filling the name with empty spaces to fix the issue for the moment with something like this:

Code: Select all

prop.set_string(index, '{:100}'.format(name))

So, if I understand correctly, you'd recommend setting all the string values at once as one single string and split with a character separator?

Hi,

thanks for the details.

Actually, I misunderstood your question on your first post. So you can forget my previous answer. Sorry about that!
I thought you wanted to store multiple values in a single string property value.

About your original question, yes, that's that possible using IOHelpers.
Check the example in my other post, the one at the end where I create a string property.

python code

# create the string values for each point
string_values= []
for i in range(point_count):
    # each string has an increasing size
    string_values.append('text_%s' % str(i).zfill(i + 1))

# this method assumes the property already exists
set_point_property_values(particle_container, 'my_string_prop', string_values)

Let me know if you need more help with this.

In fact, the ResourceProperty C++ API has different "init" method that takes an array of sizes along with an array of values, so that you can specify the size of each value instead of having to set the same size for all of them.

cpp code

//! \brief Init the items of the property with different numbers of values
//! \param[in] type the type of values stored in each item
//! \param[in] item_value_count array containing the number of values for each item
//! \param[in] item_count the number of items
void init(const Type& type, const unsigned int *item_value_count, const unsigned int& item_count);

But sadly, this method isn't available in the Clarisse Python API because SWIG (the tool that creates the Python bindings) doesn't support C++ method overloads (SWIG: Ambiguity in Overloading). The problem is that both methods have the same name but different parameter types, and only the 1st one becomes available in Python because of SWIG's limitation.

I really recommend you to use the IOHelpers API, which does a lot of stuff under the hood that isn't always possible to do in Python (like resizing or replacing existing property values).
Let us know if you chose to do so and if you need help rewriting some of your scripts with IOHelpers.

-----

About the Alembic error:

jboissinot wrote:But, I'm getting the following error when exporting the Alembic file:

Code: Select all

Alembic export: an error occured during the export process: OArrayProperty::set()
ERROR: EXCEPTION:
Illegal NULL character found in string data

This is an error in the Alembic library caused by the Clarise string property which is partially uninitialized when you don't fill the extra space.
I haven't managed to reproduce the same error, but I'm pretty sure the reason is that when you declare a ResourceProperty of type TYPE_CHAR, where each item has a size of N, if you don't fill all the N characters, then the remaining characters will contain random uninitialized values among which, maybe, the null character '0' which indicates the end of the string.

For example, if you set "hello" in a string of capacity 100, and don't fill the remaining characters, it could look like this in memory:

Code: Select all

[ h , e, l, l, o, 3, z, g, T, 0, ... ]

Clarisse doesn't know the string actually ends after "hello", it knows it has 100 characters so it will export all of them.
Then Alembic expects a string of exactly 100 characters but finds an unexpected null character before the end, and throws an error.
That's why filling the values with spaces fixes the error.

-----

Cheers,

Hi,

Indeed I did try the init method with the array of sizes and did notice that it was not supported with the Python API so thanks for confirming this.

I tried the IOHelpers API method and we do get clean string property values of different sizes per point. Using this method works better for this particular need but we noticed that the values were not properly indexed. In fact, it seems that the property values are not really indexed as they are all stored per point. I think this is something you're aware of when I read the comments you put in the code.

Code: Select all

# NOTE: this function would require some reworking to use indexed values in order to reduce the number of stored values, to share
# values between points. For example you can have N values, and assign those N values to M points, where M = point_count and N <= M.
# - point 0     -> assign value at index 0
# - point 1     -> assign value at index 0
# - point 2     -> assign value at index 2
# ...
# - point M-1   -> assign value at index 1

It's not really a big deal for the moment as we can use a different method for exporting to Houdini or Katana, but I was wondering if this is something that could possibly be fixed? I mean fixing the indexed values and optimize the data as well.

About the Alembic error, we did find out that it was because values were missing in the char property array.

Thanks,
Jeremy

Hi,

Indeed, the first script that I sent you doesn't use indexing, but it is possible.

I have updated and improved the script that I gave you before in order to be able to set indexed values.
I have also made it simpler to enable/disable the various examples at the end of the script. Check the comments in the code.

I added 2 new ones to edit indexed values. All functions are documented, but here is a quick summary:

1. set_point_property_values(particle_container_object, prop_name, values)
This is the function from the first script. It sets values on the points without indexing.

python code

values = ... # list of N values, one for each point
set_point_property_values(particle_container_object, prop_name, values)

2. set_point_property_value_at_indices(container, prop_name, value, point_indices)
Use this function to set the same value on a list of points (a subset of all points, or all of them).

python code

point_indices = [3, 5, 7, 9] # indices of points to edit
value = 'foo' # value to set on all points listed above
set_point_property_value_at_indices(container, prop_name, value, point_indices)

3. set_point_property_indexed_values(container, prop_name, indexed_values, point_value_indices)
Use this one to set indexed values on all points.

python code

# create the list of values
indexed_values = [
    'AAA',  # value index = 0
    'BBB',  # value index = 1
    'CCC',  # value index = 2
    'DDD'   # value index = 3
]
# create the list of value indices for each point (for 10 points)
point_value_indices = [
    0,      # point 0 -> value AAA
    0,      # point 1 -> value AAA
    1,      # point 2 -> value BBB
    2,      # point 3 -> value CCC
    2,      # point 4 -> value CCC
    0,      # point 5 -> value AAA
    3,      # point 6 -> value DDD
    3,      # point 7 -> value DDD
    2,      # point 8 -> value CCC
    1       # point 9 -> value BBB
]
set_point_property_indexed_values(container, prop_name, indexed_values, point_value_indices)

Regarding point indices (or IDs):
- Given a Particle Container with N points, point indices go from 0 to N-1.
- If you erase points or values: points indices and value indices will be reindexed and start from 0.
- If you add points or values: new point indices and values indices will have indices that follow the last point index or value index.
- Points without value have a special value index equal to 4294967295 (unsigned int max). In my script a have a global var INVALID_INDEX for that.

Note: the Property Editor widget is not refreshed when properties are edited by script, you'll need to close it and open a new one to see the changes.

I hope this helps. Don't hesitate if you have more questions.

Cheers,

Hi,

This helped us a lot and does work as the string values are now properly indexed using the new revised functions that you sent us.
The way to get the indexed values was a bit odd as it was returning something like:

Code: Select all

index 0 -> value AAABBBCCCDDD
index 1 -> value AAABBBCCC
index 2 -> value AAABBB
index 3 -> value AAA

But we truncate it with the value size to get the proper value as we could see how you were doing it in your script.

The only thing that we noticed though is that indexed values set on a property of a point cloud crashes Clarisse v4 when re-opening the project, while it seems to be working when opening it with v3.6.

Is it something that you could investigate to see what causes this issue?

Thanks,
Jeremy

Hi,

jboissinot wrote:The way to get the indexed values was a bit odd as it was returning something like:

Code: Select all

index 0 -> value AAABBBCCCDDD
index 1 -> value AAABBBCCC
index 2 -> value AAABBB
index 3 -> value AAA

But we truncate it with the value size to get the proper value as we could see how you were doing it in your script.

Indeed, that's odd but "normal". We should improve the API so that get_string returns the truncated string.

jboissinot wrote:The only thing that we noticed though is that indexed values set on a property of a point cloud crashes Clarisse v4 when re-opening the project, while it seems to be working when opening it with v3.6.

Is it something that you could investigate to see what causes this issue?

We will investigate that and keep you posted.

Cheers,

Hi Jeremy,

Sorry for the late response.

I haven't been able to reproduce the crash that you mentioned.
When it crashes, did you export it from 4.0 and re-import it on 4.0? Or is it an Alembic exported from 3.6 and imported on 4.0?
Can you detail the steps that lead to the crash, and ideally send us a minimal project to reproduce it?
Feel free to submit a bug report and set it private if needed.

Thanks,

Hi Anthony,

I did further tests to help troubleshoot what may cause the issue.
From what I found out, I think it's because we have a point cloud with properties that were created with the ResourceProperty API and others with the IOHelpers API, and therefore might create some kind of conflict.
And from my testing, the issue doesn't seem to come from the use of indexed values after all.

Do you think creating properties with both APIs could be this issue?

Thanks,
Jeremy