ImageJ

Table.applyMacro fails with string columns

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

4 messages Options

Stein Rørvik

Table.applyMacro fails with string columns

When Analyze Particles is run on a stack, the Label column includes a string with <ImageName>:<SliceLabel>.
I want to split these fields, so that only <ImageName> is in one column and only <SliceLabel> in another.

I tried to do this using Table.applyMacro, but am unable to get it to work.

See the below macro.
Table.applyMacro is able to remove ImageName: from ImageName:SliceLabel in the Label column,
but not :SliceLabel from ImageName:SliceLabel in the Image column.

So it is this that does not work:
Table.applyMacro("Image=Label;");
Table.applyMacro("Image=replace(Image,':.+$','');");

Combining the two lines neither works:
Table.applyMacro("Image=replace(Label,':.+$','');");

I added some test lines to see the behaviour in a column with mixed syntax.
The string replacement does not work here either, and copying the column results in NaN.

It seems that Table.applyMacro does not work handle string values?
It only works in the Label column, not the custom columns made by copying data.

Or is this an intentional limitation, that custom column are always numeric to maintain decimal precision?
If the Results table has different types for string (Label) and numeric columns,
is there any way I can add a new column to the table, specifying that it should be string?

Demo macro:
------------------------------
requires("1.52a");
run("Close All");

run("Bat Cochlea Volume (19K)");
for (s=1; s < nSlices; s++) {
setSlice(s);
setMetadata("Label", "slice " + IJ.pad(s, 3));
}
setThreshold(1,255);
run("Set Measurements...", "area mean stack display redirect=None decimal=4");
run("Analyze Particles...", "size=700-Infinity show=Overlay display clear in_situ stack");
selectWindow("Results");

waitForUser("Label column is now ImageName:SliceLabel");

//split the Label column
Table.applyMacro("Image=Label;"); //does not work, is all NaN
Table.applyMacro("Image=replace(Image,':.+$','');"); //does not work
Table.applyMacro("Label=replace(Label,'^.+:','');"); //ok
Table.update;
//apparently, applyMacro does not work with string data except within the Label column

waitForUser("Label column now has the Label part, but Image column does not have the ImageName part");

//some further test lines
Table.set("Test", 0, 12345);
Table.set("Test", 1, 1234/56);
Table.set("Test", 2, "test:123");
Table.set("Test", 3, "test");
Table.set("Test", 4, "12345");
Table.applyMacro("Test=replace(Test,'^.+:','');"); //does not work
Table.applyMacro("Test2=Test"); //does not work, string values are NaN
Table.update;

waitForUser("The copied string values from the Test column are now NaN");
------------------------------

I am using daily build ImageJ 1.52k with Java 1.6 on Windows 7/64-bit.

Stein

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Michael Schmid

Re: Table.applyMacro fails with string columns

Hi Stein,

yes, applying macros to tables is restricted to numeric values, except
for the "Label" column, which is always a String. This is also in the
documentation of Table.applyMacro:
"Currently only supports numeric values except for row labels."

Concerning the interior workings of ImageJ tables (ResultsTable class):
If there is a String value, there is still a numeric value in the
background, which is Double.NaN (not a number). If a String value exists
in the ArrayList of Strings for that cell *and* the numeric value is
NaN, the value is read as String.

When suggesting that feature to Wayne a while ago, I did not care about
String variables in "normal" columns because it is difficult to write a
macro that can cope with a variable being either a number or a String.
If the user is not aware that a variable can be a String, and there is a
String in a column, the macro will either terminate with an error or
(even worse), the result will be wrong (e.g. "10"+1 = 101, not 11).

Note that e.g. Tables created by listing plot values use blank String
values where it is not obvious: If there are data series with different
length, the shorter columns are padded with blank Strings to reach the
full length. If a user now applies a macro to such an underfull column,
the blank values will be taken as NaN, and almost any calculation will
result in NaN again, which I consider reasonable behavior. If we supply
a String value to the variable the macro will fail.

Also note that there is currently no variableType(variable) macro
function (at least I am not aware of any), so one would have to do
something like the following:
if (MyColumn+0 == MyColumn || MyColumn+0 == NaN)
{do the numeric calculation with MyColumn}
I think no one wants to write Table.applyMacro macros like this.

So, the bottom line:
If this is desired, one could extend Table.applyMacro to *write* String
values.
I see no good way to make Table.applyMacro *read* String or numeric
values from mixed columns. The only solution I could imagine would be
reading *everything* in a column as String if the first item in a column
is a String.

In a macro, of course you can still use the "classical" way, with a loop
and Table.get, Table.getString, Table.set.

Michael
________________________________________________________________

On 2019-01-12 13:56, Stein Rørvik wrote:

> When Analyze Particles is run on a stack, the Label column includes a
> string with <ImageName>:<SliceLabel>.
> I want to split these fields, so that only <ImageName> is in one
> column and only <SliceLabel> in another.
>
> I tried to do this using Table.applyMacro, but am unable to get it to
> work.
>
> See the below macro.
> Table.applyMacro is able to remove ImageName: from
> ImageName:SliceLabel in the Label column,
> but not :SliceLabel from ImageName:SliceLabel in the Image column.
>
> So it is this that does not work:
> Table.applyMacro("Image=Label;");
> Table.applyMacro("Image=replace(Image,':.+$','');");
>
> Combining the two lines neither works:
> Table.applyMacro("Image=replace(Label,':.+$','');");
>
> I added some test lines to see the behaviour in a column with mixed
> syntax.
> The string replacement does not work here either, and copying the
> column results in NaN.
>
> It seems that Table.applyMacro does not work handle string values?
> It only works in the Label column, not the custom columns made by
> copying data.
>
> Or is this an intentional limitation, that custom column are always
> numeric to maintain decimal precision?
> If the Results table has different types for string (Label) and numeric
> columns,
> is there any way I can add a new column to the table, specifying that
> it should be string?
>
> Demo macro:
> ------------------------------
> requires("1.52a");
> run("Close All");
>
> run("Bat Cochlea Volume (19K)");
> for (s=1; s < nSlices; s++) {
> setSlice(s);
> setMetadata("Label", "slice " + IJ.pad(s, 3));
> }
> setThreshold(1,255);
> run("Set Measurements...", "area mean stack display redirect=None
> decimal=4");
> run("Analyze Particles...", "size=700-Infinity show=Overlay display
> clear in_situ stack");
> selectWindow("Results");
>
> waitForUser("Label column is now ImageName:SliceLabel");
>
> //split the Label column
> Table.applyMacro("Image=Label;"); //does not work, is all NaN
> Table.applyMacro("Image=replace(Image,':.+$','');"); //does not
> work
> Table.applyMacro("Label=replace(Label,'^.+:','');"); //ok
> Table.update;
> //apparently, applyMacro does not work with string data except within
> the Label column
>
> waitForUser("Label column now has the Label part, but Image column
> does not have the ImageName part");
>
> //some further test lines
> Table.set("Test", 0, 12345);
> Table.set("Test", 1, 1234/56);
> Table.set("Test", 2, "test:123");
> Table.set("Test", 3, "test");
> Table.set("Test", 4, "12345");
> Table.applyMacro("Test=replace(Test,'^.+:','');"); //does
> not work
> Table.applyMacro("Test2=Test"); //does not work, string
> values are NaN
> Table.update;
>
> waitForUser("The copied string values from the Test column are now
> NaN");
> ------------------------------
>
> I am using daily build ImageJ 1.52k with Java 1.6 on Windows 7/64-bit.
>
>
> Stein
>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Stein Rørvik

Re: Table.applyMacro fails with string columns

Thanks for this thorough explanation of how the internals work and why my macro did not work.
I had overlooked the mention of strings not being supported in the documentation of Table.applyMacro.

I agree that it is then safest to then let Table macros not support numbers,
as a situation of letting "10"+1 be 101 instead of 11 can lead to dangerous and hard to detect bugs.

I also agree that reading blank values in padded cells as NaN is the most reasonable behaviour.

The "classical" way of doing what I want in my example is anyway not very hard:

for (i=0; i<Table.size; i++) {
str = Table.getString("Label", i);
Table.set("Image", i, replace(str,':.+$',''));
Table.set("Label", i, replace(str,'^.+:',''));
}
Table.update;

Stein

-----Original Message-----
From: ImageJ Interest Group <[hidden email]> On Behalf Of Michael Schmid
Sent: 12. januar 2019 16:56
To: [hidden email]
Subject: Re: Table.applyMacro fails with string columns

Hi Stein,

yes, applying macros to tables is restricted to numeric values, except for the "Label" column, which is always a String. This is also in the documentation of Table.applyMacro:
"Currently only supports numeric values except for row labels."

Concerning the interior workings of ImageJ tables (ResultsTable class):
If there is a String value, there is still a numeric value in the background, which is Double.NaN (not a number). If a String value exists in the ArrayList of Strings for that cell *and* the numeric value is NaN, the value is read as String.

When suggesting that feature to Wayne a while ago, I did not care about String variables in "normal" columns because it is difficult to write a macro that can cope with a variable being either a number or a String.
If the user is not aware that a variable can be a String, and there is a String in a column, the macro will either terminate with an error or (even worse), the result will be wrong (e.g. "10"+1 = 101, not 11).

Note that e.g. Tables created by listing plot values use blank String values where it is not obvious: If there are data series with different length, the shorter columns are padded with blank Strings to reach the full length. If a user now applies a macro to such an underfull column, the blank values will be taken as NaN, and almost any calculation will result in NaN again, which I consider reasonable behavior. If we supply a String value to the variable the macro will fail.

Also note that there is currently no variableType(variable) macro function (at least I am not aware of any), so one would have to do something like the following:
if (MyColumn+0 == MyColumn || MyColumn+0 == NaN)
{do the numeric calculation with MyColumn} I think no one wants to write Table.applyMacro macros like this.

So, the bottom line:
If this is desired, one could extend Table.applyMacro to *write* String values.
I see no good way to make Table.applyMacro *read* String or numeric values from mixed columns. The only solution I could imagine would be reading *everything* in a column as String if the first item in a column is a String.

In a macro, of course you can still use the "classical" way, with a loop and Table.get, Table.getString, Table.set.

Michael
________________________________________________________________

On 2019-01-12 13:56, Stein Rørvik wrote:

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Stein Rørvik

Re: Table.applyMacro fails with string columns

Typo, I meant " safest to then let Table macros only support numbers"

-----Original Message-----
From: ImageJ Interest Group <[hidden email]> On Behalf Of Stein Rørvik
Sent: 12. januar 2019 18:38
To: [hidden email]
Subject: Re: Table.applyMacro fails with string columns

Thanks for this thorough explanation of how the internals work and why my macro did not work.
I had overlooked the mention of strings not being supported in the documentation of Table.applyMacro.

I agree that it is then safest to then let Table macros not support numbers, as a situation of letting "10"+1 be 101 instead of 11 can lead to dangerous and hard to detect bugs.

I also agree that reading blank values in padded cells as NaN is the most reasonable behaviour.

The "classical" way of doing what I want in my example is anyway not very hard:

for (i=0; i<Table.size; i++) {
str = Table.getString("Label", i);
Table.set("Image", i, replace(str,':.+$',''));
Table.set("Label", i, replace(str,'^.+:','')); } Table.update;

Stein

-----Original Message-----
From: ImageJ Interest Group <[hidden email]> On Behalf Of Michael Schmid
Sent: 12. januar 2019 16:56
To: [hidden email]
Subject: Re: Table.applyMacro fails with string columns

Hi Stein,

yes, applying macros to tables is restricted to numeric values, except for the "Label" column, which is always a String. This is also in the documentation of Table.applyMacro:
"Currently only supports numeric values except for row labels."

Concerning the interior workings of ImageJ tables (ResultsTable class):
If there is a String value, there is still a numeric value in the background, which is Double.NaN (not a number). If a String value exists in the ArrayList of Strings for that cell *and* the numeric value is NaN, the value is read as String.

When suggesting that feature to Wayne a while ago, I did not care about String variables in "normal" columns because it is difficult to write a macro that can cope with a variable being either a number or a String.
If the user is not aware that a variable can be a String, and there is a String in a column, the macro will either terminate with an error or (even worse), the result will be wrong (e.g. "10"+1 = 101, not 11).

Note that e.g. Tables created by listing plot values use blank String values where it is not obvious: If there are data series with different length, the shorter columns are padded with blank Strings to reach the full length. If a user now applies a macro to such an underfull column, the blank values will be taken as NaN, and almost any calculation will result in NaN again, which I consider reasonable behavior. If we supply a String value to the variable the macro will fail.

Also note that there is currently no variableType(variable) macro function (at least I am not aware of any), so one would have to do something like the following:
if (MyColumn+0 == MyColumn || MyColumn+0 == NaN)
{do the numeric calculation with MyColumn} I think no one wants to write Table.applyMacro macros like this.

So, the bottom line:
If this is desired, one could extend Table.applyMacro to *write* String values.
I see no good way to make Table.applyMacro *read* String or numeric values from mixed columns. The only solution I could imagine would be reading *everything* in a column as String if the first item in a column is a String.

In a macro, of course you can still use the "classical" way, with a loop and Table.get, Table.getString, Table.set.

Michael
________________________________________________________________

On 2019-01-12 13:56, Stein Rørvik wrote:

> When Analyze Particles is run on a stack, the Label column includes a
> string with <ImageName>:<SliceLabel>.
> I want to split these fields, so that only <ImageName> is in one
> column and only <SliceLabel> in another.
>
> I tried to do this using Table.applyMacro, but am unable to get it to
> work.
>
> See the below macro.
> Table.applyMacro is able to remove ImageName: from
> ImageName:SliceLabel in the Label column, but not :SliceLabel from
> ImageName:SliceLabel in the Image column.
>
> So it is this that does not work:
> Table.applyMacro("Image=Label;");
> Table.applyMacro("Image=replace(Image,':.+$','');");
>
> Combining the two lines neither works:
> Table.applyMacro("Image=replace(Label,':.+$','');");
>
> I added some test lines to see the behaviour in a column with mixed
> syntax.
> The string replacement does not work here either, and copying the
> column results in NaN.
>
> It seems that Table.applyMacro does not work handle string values?
> It only works in the Label column, not the custom columns made by
> copying data.
>
> Or is this an intentional limitation, that custom column are always
> numeric to maintain decimal precision?
> If the Results table has different types for string (Label) and
> numeric columns, is there any way I can add a new column to the table,
> specifying that it should be string?
>
> Demo macro:
> ------------------------------
> requires("1.52a");
> run("Close All");
>
> run("Bat Cochlea Volume (19K)");
> for (s=1; s < nSlices; s++) {
> setSlice(s);
> setMetadata("Label", "slice " + IJ.pad(s, 3)); }
> setThreshold(1,255); run("Set Measurements...", "area mean stack
> display redirect=None decimal=4"); run("Analyze Particles...",
> "size=700-Infinity show=Overlay display clear in_situ stack");
> selectWindow("Results");
>
> waitForUser("Label column is now ImageName:SliceLabel");
>
> //split the Label column
> Table.applyMacro("Image=Label;"); //does not work, is all NaN
> Table.applyMacro("Image=replace(Image,':.+$','');"); //does not
> work
> Table.applyMacro("Label=replace(Label,'^.+:','');"); //ok
> Table.update;
> //apparently, applyMacro does not work with string data except within
> the Label column
>
> waitForUser("Label column now has the Label part, but Image column
> does not have the ImageName part");
>
> //some further test lines
> Table.set("Test", 0, 12345);
> Table.set("Test", 1, 1234/56);
> Table.set("Test", 2, "test:123");
> Table.set("Test", 3, "test");
> Table.set("Test", 4, "12345");
> Table.applyMacro("Test=replace(Test,'^.+:','');"); //does
> not work
> Table.applyMacro("Test2=Test"); //does not work, string
> values are NaN
> Table.update;
>
> waitForUser("The copied string values from the Test column are now
> NaN");
> ------------------------------
>
> I am using daily build ImageJ 1.52k with Java 1.6 on Windows 7/64-bit.
>
>
> Stein
>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html