Results Table in Batch Processing

[Kidd, Mary]

Hello ImageJ Users,

I hope I have not missed something obvious, but here is my issue.  I have been using ImageJ to batch process approximately 5000 files at at time using Analyze Particles.  Even though the setBatchMode flag is set to true, the Results table displays to screen.  The code runs fairly fast at the beginning, but bogs down as more and more lines of results are added, which is understandable.  If I close the Results table while the script is running, it reopens and speeds up again.

I want to speed up the processing of these images, preferably so that I could submit the jobs to a cluster.  I tried it with Fiji yesterday in headless mode for the first time on my mac, and it worked for a while before it threw an exception.  I am not sure if it was a memory issue or something else.  I increased the allowed memory usage and it did make it further before dying.  Previously, I had been using IJ1 where the script worked fine (no memory issues), albeit slow as described.

It seems like my options are to either create a results file for each image file, which is tedious to post-process, or that I am missing something in the Analyze Particles call that would take care of this issue.  I know I have it set to display, but it seems like I cannot save the results table when I tell it not to display.

I am grateful for any advice.

For reference, here is the relevant part of the script, which is within a for-loop cycling over the files:
*****

getStatistics(area, mean, min, max);

setThreshold(10, max);

run("Set Measurements...", "area mean min centroid integrated limit redirect=None decimal=3");

run("Analyze Particles...", "size=0-Infinity circularity=0.00-1.00 show=Nothing display");


saveAs("Results", dir1+"Results.txt");

     close();

*****

Thank you,
Mary

--
Mary F. Kidd
Assistant Professor, Physics Department

Tennessee Technological University
PO Box 5051
Cookeville, TN 38501

Office:  [phone]

Confidentiality Notice:  This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed.  This communication may contain confidential work product or other legally privileged, confidential or proprietary information.  If you are not the intended recipient, please be advised that you have received this message in error and that any use, dissemination, printing or copying of this message is strictly prohibited.  If you have received this message in error, please permanently delete it from your computer system and contact the sender at the above address and/or telephone number.  Thank you.

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Gabriel Landini]

On Wednesday 13 Aug 2014 16:37:38 Kidd, Mary wrote:
> I hope I have not missed something obvious, but here is my issue.  I have
> been using ImageJ to batch process approximately 5000 files at at time
> using Analyze Particles.  Even though the setBatchMode flag is set to true,
> the Results table displays to screen.  The code runs fairly fast at the
> beginning, but bogs down as more and more lines of results are added, which
> is understandable.  If I close the Results table while the script is
> running, it reopens and speeds up again.

Can you please check that your IJ install is up to date. I reported a similar
slow-down of the Results Table using Particles8 and Wayne promptly issued a
fix.

Regards

Gabriel

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Rasband, Wayne (NIH/NIMH) [E]]

On Aug 13, 2014, at 12:37 PM, Kidd, Mary wrote:

> Hello ImageJ Users,
>
> I hope I have not missed something obvious, but here is my issue.  I have been using ImageJ to batch process approximately 5000 files at at time using Analyze Particles.  Even though the setBatchMode flag is set to true, the Results table displays to screen.  The code runs fairly fast at the beginning, but bogs down as more and more lines of results are added, which is understandable.  If I close the Results table while the script is running, it reopens and speeds up again.
>
> I want to speed up the processing of these images, preferably so that I could submit the jobs to a cluster.  I tried it with Fiji yesterday in headless mode for the first time on my mac, and it worked for a while before it threw an exception.  I am not sure if it was a memory issue or something else.  I increased the allowed memory usage and it did make it further before dying.  Previously, I had been using IJ1 where the script worked fine (no memory issues), albeit slow as described.
>
> It seems like my options are to either create a results file for each image file, which is tedious to post-process, or that I am missing something in the Analyze Particles call that would take care of this issue.  I know I have it set to display, but it seems like I cannot save the results table when I tell it not to display.

Move saveAs("Results",path) outside the loop and the macro should run much faster. The following test macro, which runs the particle analyzer 5000 times on the blobs sample image, does not bog down as more lines are added to the Results table. The "Results.txt" file created by this macro is 17.4 MB, so it is obviously better to only save it one time.

-wayne

  setBatchMode(true);
  run("Clear Results");
  run("Blobs (25K)");
  setAutoThreshold("Default");
  run("Set Measurements...", "area mean min centroid integrated decimal=3");
  for (i=0; i<5000; i++)
     run("Analyze Particles...", "display");
  saveAs("Results", getDirectory("home")+"Downloads/Results.txt");

> I am grateful for any advice.
>
> For reference, here is the relevant part of the script, which is within a for-loop cycling over the files:
> *****
>
> getStatistics(area, mean, min, max);
>
> setThreshold(10, max);
>
> run("Set Measurements...", "area mean min centroid integrated limit redirect=None decimal=3");
>
> run("Analyze Particles...", "size=0-Infinity circularity=0.00-1.00 show=Nothing display");
>
>
> saveAs("Results", dir1+"Results.txt");
>
>     close();
>
> *****
>
> Thank you,
> Mary
>
> --
> Mary F. Kidd
> Assistant Professor, Physics Department
>
> Tennessee Technological University
> PO Box 5051
> Cookeville, TN 38501
>
> Office:  [phone]
>
> Confidentiality Notice:  This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed.  This communication may contain confidential work product or other legally privileged, confidential or proprietary information.  If you are not the intended recipient, please be advised that you have received this message in error and that any use, dissemination, printing or copying of this message is strictly prohibited.  If you have received this message in error, please permanently delete it from your computer system and contact the sender at the above address and/or telephone number.  Thank you.

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html