Glitch with Batch Processing

[Marcus Jones]

I'm batch processing pictures using a macro to determine percentage of green plant tissue from small plot research trials.  Every so often a picture doesn't appear to process correctly and the coverage estimates are greatly underestimated.

Examples of the original image and the subsequent image that doesn’t process correctly after running through the macro are attached.

When I manually set the color threshold levels, convert to binary, and run the Analyze Particles function, the vast majority of green tissue is properly selected.  But for some reason, the plant tissue in the bottom 2/3rds of the picture isn’t correctly selected when run through the macro.  This issue is rather sporadic and doesn’t always occur.  I’ve ran entire sets of images (50-100) without this occurring.

Here is the code for the macro I’m using:

run("Set Scale...", "distance=3494.5 known=100 unit=cm");
run("Color Threshold...");
// Color Thresholder 1.49m
// Autogenerated macro, single images only!
min=newArray(3);
max=newArray(3);
filter=newArray(3);
a=getTitle();
run("HSB Stack");
run("Convert Stack to Images");
selectWindow("Hue");
rename("0");
selectWindow("Saturation");
rename("1");
selectWindow("Brightness");
rename("2");
min[0]=35;
max[0]=125;
filter[0]="pass";
min[1]=45;
max[1]=255;
filter[1]="pass";
min[2]=0;
max[2]=255;
filter[2]="pass";
for (i=0;i<3;i++){
selectWindow(""+i);
setThreshold(min[i], max[i]);
run("Convert to Mask");
if (filter[i]=="stop")  run("Invert");
}
imageCalculator("AND create", "0","1");
imageCalculator("AND create", "Result of 0","2");
for (i=0;i<3;i++){
selectWindow(""+i);
close();
}
selectWindow("Result of 0");
close();
selectWindow("Result of Result of 0");
rename(a);
// Colour Thresholding-------------
run("8-bit");
setAutoThreshold("Default");
//run("Threshold...");
setOption("BlackBackground", false);
run("Convert to Mask"); run("Remove Outliers...", "radius=2 threshold=50 which=Dark");

run("Analyze Particles...", "  show=Masks display summarize");


I’ve updated to the most recent version of ImageJ to no avail.

>>Does anyone know why this is happening and is there anything I can do to avoid this?


--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Brandon Hurr]

I've not been able to replicate this error on my machine using the
following macro and your image. At the end, do Window > Tile and they will
splay out like this: http://i.imgur.com/QrH3pLh.jpg

I ran this in FIJI (2.0.0-rc-43/1.51b6) and ImageJ (1.50i) Java 1.8.0_91 64
bit Mac OSX 10.11.5.

This took almost 4GB of RAM on my machine. Adjust z loop numbers down if
you don't have at least 6 GB of RAM.

// begin macro
// reset all windows and measurements
run("Options...", "iterations=1 count=1 edm=Overwrite");
run("Line Width...", "line=1");
run("Colors...", "foreground=black background=white selection=yellow");
run("Clear Results");
run("Close All");
print("\\Clear");
run("ROI Manager...");

open(File.openDialog("Find grass file"));

setBatchMode(true);

orig = getTitle();
selectWindow(orig);


for(z=0; z<27; z++) {
selectWindow(orig);
run("Duplicate...", "title=Test");
run("Set Scale...", "distance=3494.5 known=100 unit=cm");
run("Color Threshold...");
// Color Thresholder 1.49m
// Autogenerated macro, single images only!
min=newArray(3);
max=newArray(3);
filter=newArray(3);
a=getTitle();
run("HSB Stack");
run("Convert Stack to Images");
selectWindow("Hue");
rename("0");
selectWindow("Saturation");
rename("1");
selectWindow("Brightness");
rename("2");
min[0]=35;
max[0]=125;
filter[0]="pass";
min[1]=45;
max[1]=255;
filter[1]="pass";
min[2]=0;
max[2]=255;
filter[2]="pass";
for (i=0;i<3;i++){
selectWindow(""+i);
setThreshold(min[i], max[i]);
run("Convert to Mask");
if (filter[i]=="stop")  run("Invert");
}
imageCalculator("AND create", "0","1");
imageCalculator("AND create", "Result of 0","2");
for (i=0;i<3;i++){
selectWindow(""+i);
close();
}
selectWindow("Result of 0");
close();
selectWindow("Result of Result of 0");
rename(a);
// Colour Thresholding-------------
run("8-bit");
setAutoThreshold("Default");
//run("Threshold...");
setOption("BlackBackground", false);
run("Convert to Mask");
run("Remove Outliers...", "radius=2 threshold=50 which=Dark");
run("Analyze Particles...", "  show=Masks display summarize");
selectWindow("Test");
run("Close");
}

setBatchMode("exit and display");

exit("Finished!");
//end macro

On Tue, May 24, 2016 at 8:46 AM, Marcus Jones <[hidden email]>
wrote:

> I'm batch processing pictures using a macro to determine percentage of
> green plant tissue from small plot research trials.  Every so often a
> picture doesn't appear to process correctly and the coverage estimates are
> greatly underestimated.
>
> Examples of the original image and the subsequent image that doesn’t
> process correctly after running through the macro are attached.
>
> When I manually set the color threshold levels, convert to binary, and run
> the Analyze Particles function, the vast majority of green tissue is
> properly selected.  But for some reason, the plant tissue in the bottom
> 2/3rds of the picture isn’t correctly selected when run through the macro.
> This issue is rather sporadic and doesn’t always occur.  I’ve ran entire
> sets of images (50-100) without this occurring.
>
> Here is the code for the macro I’m using:
>
> run("Set Scale...", "distance=3494.5 known=100 unit=cm");
> run("Color Threshold...");
> // Color Thresholder 1.49m
> // Autogenerated macro, single images only!
> min=newArray(3);
> max=newArray(3);
> filter=newArray(3);
> a=getTitle();
> run("HSB Stack");
> run("Convert Stack to Images");
> selectWindow("Hue");
> rename("0");
> selectWindow("Saturation");
> rename("1");
> selectWindow("Brightness");
> rename("2");
> min[0]=35;
> max[0]=125;
> filter[0]="pass";
> min[1]=45;
> max[1]=255;
> filter[1]="pass";
> min[2]=0;
> max[2]=255;
> filter[2]="pass";
> for (i=0;i<3;i++){
> selectWindow(""+i);
> setThreshold(min[i], max[i]);
> run("Convert to Mask");
> if (filter[i]=="stop")  run("Invert");
> }
> imageCalculator("AND create", "0","1");
> imageCalculator("AND create", "Result of 0","2");
> for (i=0;i<3;i++){
> selectWindow(""+i);
> close();
> }
> selectWindow("Result of 0");
> close();
> selectWindow("Result of Result of 0");
> rename(a);
> // Colour Thresholding-------------
> run("8-bit");
> setAutoThreshold("Default");
> //run("Threshold...");
> setOption("BlackBackground", false);
> run("Convert to Mask"); run("Remove Outliers...", "radius=2 threshold=50
> which=Dark");
>
> run("Analyze Particles...", "  show=Masks display summarize");
>
>
> I’ve updated to the most recent version of ImageJ to no avail.
>
> >>Does anyone know why this is happening and is there anything I can do to
> avoid this?
>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Dan McDonald]

Hi Marcus,

I ran your code on my machine and everything worked as expected. (I did interpret the position of some comment delimiters in a specific manner that might be different than actually in your code.)

Your example images suggest to me some kind of hiccup occurs after the first run("Convert to Mask") triad.  Many years ago I started adding wait statements after long-processing-time commands to prevent one command from starting before another begins.  I don't know if that problem of not synchronizing properly has been fixed or not in recent time.  Your image looks like the top 1/3 was processed one way and then the bottom 2/3 another.

Did the macro execute to completion when it gave the output image? It does not appear to have been processed with your run('Remove Outliers...") command.

If it is a synchronization problem adding strategic wait statements or putting your macro into setBatchMode(true) mode might help.

Best regards,

Dan

-----Original Message-----
From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of Marcus Jones
Sent: Tuesday, May 24, 2016 11:47 AM
To: [hidden email]
Subject: Glitch with Batch Processing

I'm batch processing pictures using a macro to determine percentage of green plant tissue from small plot research trials.  Every so often a picture doesn't appear to process correctly and the coverage estimates are greatly underestimated.

Examples of the original image and the subsequent image that doesn’t process correctly after running through the macro are attached.

When I manually set the color threshold levels, convert to binary, and run the Analyze Particles function, the vast majority of green tissue is properly selected.  But for some reason, the plant tissue in the bottom 2/3rds of the picture isn’t correctly selected when run through the macro.  This issue is rather sporadic and doesn’t always occur.  I’ve ran entire sets of images (50-100) without this occurring.

Here is the code for the macro I’m using:

run("Set Scale...", "distance=3494.5 known=100 unit=cm"); run("Color Threshold..."); // Color Thresholder 1.49m // Autogenerated macro, single images only!
min=newArray(3);
max=newArray(3);
filter=newArray(3);
a=getTitle();
run("HSB Stack");
run("Convert Stack to Images");
selectWindow("Hue");
rename("0");
selectWindow("Saturation");
rename("1");
selectWindow("Brightness");
rename("2");
min[0]=35;
max[0]=125;
filter[0]="pass";
min[1]=45;
max[1]=255;
filter[1]="pass";
min[2]=0;
max[2]=255;
filter[2]="pass";
for (i=0;i<3;i++){
selectWindow(""+i);
setThreshold(min[i], max[i]);
run("Convert to Mask");
if (filter[i]=="stop")  run("Invert");
}
imageCalculator("AND create", "0","1");
imageCalculator("AND create", "Result of 0","2"); for (i=0;i<3;i++){ selectWindow(""+i); close(); } selectWindow("Result of 0"); close(); selectWindow("Result of Result of 0"); rename(a); // Colour Thresholding------------- run("8-bit"); setAutoThreshold("Default"); //run("Threshold..."); setOption("BlackBackground", false); run("Convert to Mask"); run("Remove Outliers...", "radius=2 threshold=50 which=Dark");

run("Analyze Particles...", "  show=Masks display summarize");


I’ve updated to the most recent version of ImageJ to no avail.

>>Does anyone know why this is happening and is there anything I can do to avoid this?


--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[BenTupper]

Hi,

> On May 24, 2016, at 2:24 PM, Dan McDonald <[hidden email]> wrote:
>
> Hi Marcus,
>
> I ran your code on my machine and everything worked as expected. (I did interpret the position of some comment delimiters in a specific manner that might be different than actually in your code.)
>
> Your example images suggest to me some kind of hiccup occurs after the first run("Convert to Mask") triad.  Many years ago I started adding wait statements after long-processing-time commands to prevent one command from starting before another begins.  I don't know if that problem of not synchronizing properly has been fixed or not in recent time.  Your image looks like the top 1/3 was processed one way and then the bottom 2/3 another.
>

That's a good idea.  Another way to help manage which image is active is to use getImageID(), selectImage(id) and friends.  That way you can explicitly manage the images by unique ID rather than by name.  

https://rsb.info.nih.gov/ij/developer/macro/functions.html#getImageID

Ben

> Did the macro execute to completion when it gave the output image? It does not appear to have been processed with your run('Remove Outliers...") command.
>
> If it is a synchronization problem adding strategic wait statements or putting your macro into setBatchMode(true) mode might help.
>
> Best regards,
>
> Dan
>
> -----Original Message-----
> From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of Marcus Jones
> Sent: Tuesday, May 24, 2016 11:47 AM
> To: [hidden email]
> Subject: Glitch with Batch Processing
>
> I'm batch processing pictures using a macro to determine percentage of green plant tissue from small plot research trials.  Every so often a picture doesn't appear to process correctly and the coverage estimates are greatly underestimated.
>
> Examples of the original image and the subsequent image that doesn’t process correctly after running through the macro are attached.
>
> When I manually set the color threshold levels, convert to binary, and run the Analyze Particles function, the vast majority of green tissue is properly selected.  But for some reason, the plant tissue in the bottom 2/3rds of the picture isn’t correctly selected when run through the macro.  This issue is rather sporadic and doesn’t always occur.  I’ve ran entire sets of images (50-100) without this occurring.
>
> Here is the code for the macro I’m using:
>
> run("Set Scale...", "distance=3494.5 known=100 unit=cm"); run("Color Threshold..."); // Color Thresholder 1.49m // Autogenerated macro, single images only!
> min=newArray(3);
> max=newArray(3);
> filter=newArray(3);
> a=getTitle();
> run("HSB Stack");
> run("Convert Stack to Images");
> selectWindow("Hue");
> rename("0");
> selectWindow("Saturation");
> rename("1");
> selectWindow("Brightness");
> rename("2");
> min[0]=35;
> max[0]=125;
> filter[0]="pass";
> min[1]=45;
> max[1]=255;
> filter[1]="pass";
> min[2]=0;
> max[2]=255;
> filter[2]="pass";
> for (i=0;i<3;i++){
> selectWindow(""+i);
> setThreshold(min[i], max[i]);
> run("Convert to Mask");
> if (filter[i]=="stop")  run("Invert");
> }
> imageCalculator("AND create", "0","1");
> imageCalculator("AND create", "Result of 0","2"); for (i=0;i<3;i++){ selectWindow(""+i); close(); } selectWindow("Result of 0"); close(); selectWindow("Result of Result of 0"); rename(a); // Colour Thresholding------------- run("8-bit"); setAutoThreshold("Default"); //run("Threshold..."); setOption("BlackBackground", false); run("Convert to Mask"); run("Remove Outliers...", "radius=2 threshold=50 which=Dark");
>
> run("Analyze Particles...", "  show=Masks display summarize");
>
>
> I’ve updated to the most recent version of ImageJ to no avail.
>
>>> Does anyone know why this is happening and is there anything I can do to avoid this?
>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[ctrueden]

I just want to mention that Marcus also posted this question on the ImageJ
forum, where additional discussion is happening:

http://forum.imagej.net/t/glitch-with-batch-processing-images/1691

The joys of cross-posting!

--
Curtis Rueden
LOCI software architect - http://loci.wisc.edu/software
ImageJ2 lead, Fiji maintainer - http://imagej.net/User:Rueden
Did you know ImageJ has a forum? http://forum.imagej.net/


On Tue, May 24, 2016 at 1:43 PM, Ben Tupper <[hidden email]> wrote:

> Hi,
>
> > On May 24, 2016, at 2:24 PM, Dan McDonald <
> [hidden email]> wrote:
> >
> > Hi Marcus,
> >
> > I ran your code on my machine and everything worked as expected. (I did
> interpret the position of some comment delimiters in a specific manner that
> might be different than actually in your code.)
> >
> > Your example images suggest to me some kind of hiccup occurs after the
> first run("Convert to Mask") triad.  Many years ago I started adding wait
> statements after long-processing-time commands to prevent one command from
> starting before another begins.  I don't know if that problem of not
> synchronizing properly has been fixed or not in recent time.  Your image
> looks like the top 1/3 was processed one way and then the bottom 2/3
> another.
> >
>
> That's a good idea.  Another way to help manage which image is active is
> to use getImageID(), selectImage(id) and friends.  That way you can
> explicitly manage the images by unique ID rather than by name.
>
> https://rsb.info.nih.gov/ij/developer/macro/functions.html#getImageID
>
> Ben
>
> > Did the macro execute to completion when it gave the output image? It
> does not appear to have been processed with your run('Remove Outliers...")
> command.
> >
> > If it is a synchronization problem adding strategic wait statements or
> putting your macro into setBatchMode(true) mode might help.
> >
> > Best regards,
> >
> > Dan
> >
> > -----Original Message-----
> > From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of
> Marcus Jones
> > Sent: Tuesday, May 24, 2016 11:47 AM
> > To: [hidden email]
> > Subject: Glitch with Batch Processing
> >
> > I'm batch processing pictures using a macro to determine percentage of
> green plant tissue from small plot research trials.  Every so often a
> picture doesn't appear to process correctly and the coverage estimates are
> greatly underestimated.
> >
> > Examples of the original image and the subsequent image that doesn’t
> process correctly after running through the macro are attached.
> >
> > When I manually set the color threshold levels, convert to binary, and
> run the Analyze Particles function, the vast majority of green tissue is
> properly selected.  But for some reason, the plant tissue in the bottom
> 2/3rds of the picture isn’t correctly selected when run through the macro.
> This issue is rather sporadic and doesn’t always occur.  I’ve ran entire
> sets of images (50-100) without this occurring.
> >
> > Here is the code for the macro I’m using:
> >
> > run("Set Scale...", "distance=3494.5 known=100 unit=cm"); run("Color
> Threshold..."); // Color Thresholder 1.49m // Autogenerated macro, single
> images only!
> > min=newArray(3);
> > max=newArray(3);
> > filter=newArray(3);
> > a=getTitle();
> > run("HSB Stack");
> > run("Convert Stack to Images");
> > selectWindow("Hue");
> > rename("0");
> > selectWindow("Saturation");
> > rename("1");
> > selectWindow("Brightness");
> > rename("2");
> > min[0]=35;
> > max[0]=125;
> > filter[0]="pass";
> > min[1]=45;
> > max[1]=255;
> > filter[1]="pass";
> > min[2]=0;
> > max[2]=255;
> > filter[2]="pass";
> > for (i=0;i<3;i++){
> > selectWindow(""+i);
> > setThreshold(min[i], max[i]);
> > run("Convert to Mask");
> > if (filter[i]=="stop")  run("Invert");
> > }
> > imageCalculator("AND create", "0","1");
> > imageCalculator("AND create", "Result of 0","2"); for (i=0;i<3;i++){
> selectWindow(""+i); close(); } selectWindow("Result of 0"); close();
> selectWindow("Result of Result of 0"); rename(a); // Colour
> Thresholding------------- run("8-bit"); setAutoThreshold("Default");
> //run("Threshold..."); setOption("BlackBackground", false); run("Convert to
> Mask"); run("Remove Outliers...", "radius=2 threshold=50 which=Dark");
> >
> > run("Analyze Particles...", "  show=Masks display summarize");
> >
> >
> > I’ve updated to the most recent version of ImageJ to no avail.
> >
> >>> Does anyone know why this is happening and is there anything I can do
> to avoid this?
> >
> >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html