Running scripts with an open Virtual Hyperstack

[Greg]

Hi,

today I witnessed a rather odd behavior of Fiji when trying to run my jython scripts on a opened virtual hyperstack. So basically it took ages to run my script so I stripped it down to the point, where I realized that you just have to try to run an empty (!) script from the script editor to see all your RAM eaten up. I tested that for jython and javascript.

So what I did:

opened a 7gb ome.tiff with Bio Formats Importer in virtual hyperstack mode
opened the script editor via file -> new -> script
choosed python/java script as language
pressed run on the empty(!) script

What I see:

When monitoring the RAM, I see it gets filled rather quickly by Fiji up to the size of 9.5Gb. For bigger input files more RAM gets 'used' accordingly. So for files exceeding my physical amount of Memory Fiji becomes unresponsive rather quickly. I experienced the same behavior for nd2 files.

Is there a way to cirmumvent this ? I guess Fiji tries to load the whole image stack despite having chosen 'virtual stack' in the first place ?

Best,
Greg

[ctrueden]

Hi Greg,

> you just have to try to run an empty (!) script from the script editor
> to see all your RAM eaten up.

Firstly, I suggest checking:

  Edit > Options > ImageJ2

And ensure that "Enable ImageJ2 data structures" is _unchecked_.

There is a known issue with IJ1<->IJ2 image syncing where virtual stacks
can indeed be erroneously expanded to non-virtual images.

This option being enabled would also explain some of the sluggishness you
previously reported.

If that doesn't work, then I suggest you follow the directions at:
  http://imagej.net/Debugging#Debugging_JVM_hangs

To diagnose what exactly is happening to fill up all the RAM. In short, try
pressing Shift+\ (Plugins > Debug > Dump Stack) to open up a window with
information about what ImageJ is doing during the problematic operation.

You can then copy/paste the Stack Dump text (using e.g. pastebin.com or
gist.github.com) and send a link to the mailing list, so that developers
can investigate the issue.

Regards,
Curtis

On Wed, Mar 25, 2015 at 8:51 AM, Greg <[hidden email]> wrote:

> Hi,
>
> today I witnessed a rather odd behavior of Fiji when trying to run my
> jython
> scripts on a opened virtual hyperstack. So basically it took ages to run my
> script so I stripped it down to the point, where I realized that you just
> have to try to run an empty (!) script from the script editor to see all
> your RAM eaten up. I tested that for jython and javascript.
>
> So what I did:
>
> opened a 7gb ome.tiff with Bio Formats Importer in virtual hyperstack mode
> opened the script editor via file -> new -> script
> choosed python/java script as language
> pressed run on the empty(!) script
>
> What I see:
>
> When monitoring the RAM, I see it gets filled rather quickly by Fiji up to
> the size of 9.5Gb. For bigger input files more RAM gets 'used' accordingly.
> So for files exceeding my physical amount of Memory Fiji becomes
> unresponsive rather quickly. I experienced the same behavior for nd2 files.
>
> Is there a way to cirmumvent this ? I guess Fiji tries to load the whole
> image stack despite having chosen 'virtual stack' in the first place ?
>
> Best,
> Greg
>
>
>
>
> --
> View this message in context:
> http://imagej.1557.x6.nabble.com/Running-scripts-with-an-open-Virtual-Hyperstack-tp5012163.html
> Sent from the ImageJ mailing list archive at Nabble.com.
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Greg]

Hi Curtis,

ya you are right, it might actually not related to scripts/plugins at all I think, this was just the first action I undertook after opening the virtual stack. That totally explains that slugginess ! The ImageJ2 datastructure option is and was disabled.

So this is the stack dump right after opening the virtual stack, ca. 1.2Gb are used by Fiji at this moment:

http://pastebin.com/ex1yG6JM

The next dump is (hopefully) right in the moment when the RAM is filled rather quickly (around 8Gb at that very moment), this time to initiate the RAM flood I just started the script manager (without trying to run an empty script):

http://pastebin.com/LucCkLcA

After that, every [next slice] fills up the RAM a little more right now it is at 14.2Gb (the input file is 7Gb):

http://pastebin.com/BmiEketC

In the end Fiji is now almost unresponsive and filled 18Gb of my physical RAM (I got 32). Even when I close the virtual hyperstack, there are still 18Gb used as indicated by the activity Monitor from Mac OS 10.10:

http://pastebin.com/jApGDHs2

Please tell me if you need anything else, I am running ImageJ 149p as Fiji.
Thanks for your help !

Best,
Greg

[ctrueden]

Hi Greg,

> this is the stack dump right after opening the virtual stack

Thank you very much for that vital information. It identified a serious bug
in ImageJ2, which I believe is now fixed.

Would you mind testing it? You can find the relevant JAR files here:

  http://curtis.imagej.net/2015-03-25-bad-syncing/

Place the two JAR files in your ImageJ installation's jars folder, and
delete the previous versions.

Two major changes of note:

1) The new imagej-common hopefully fixes the crazy RAM flooding.

2) The new imagej-legacy fully eliminates the "Enable ImageJ2 data
structures" option so that users are not bitten by problems relating to it
in the future. Developers who need this feature can instead now set the
"imagej.legacy.sync" system property, if they want to test it.

Please let us know how it goes; I am optimistic that these changes will
greatly improve matters.

We plan to upload these fixes to the ImageJ update site by end of day
tomorrow, unless you report back with continued (or worse) problems.

Regards,
Curtis

On Wed, Mar 25, 2015 at 11:15 AM, Greg <[hidden email]> wrote:

> Hi Curtis,
>
> ya you are right, it might actually not related to scripts/plugins at all I
> think, this was just the first action I undertook after opening the virtual
> stack. That totally explains that slugginess ! The ImageJ2 datastructure
> option is and was disabled.
>
> So this is the stack dump right after opening the virtual stack, ca. 1.2Gb
> are used by Fiji at this moment:
>
> http://pastebin.com/ex1yG6JM
>
> The next dump is (hopefully) right in the moment when the RAM is filled
> rather quickly (around 8Gb at that very moment), this time to initiate the
> RAM flood I just started the script manager (without trying to run an empty
> script):
>
> http://pastebin.com/LucCkLcA
>
> After that, every [next slice] fills up the RAM a little more right now it
> is at 14.2Gb (the input file is 7Gb):
>
> http://pastebin.com/BmiEketC
>
> In the end Fiji is now almost unresponsive and filled 18Gb of my physical
> RAM (I got 32). Even when I close the virtual hyperstack, there are still
> 18Gb used as indicated by the activity Monitor from Mac OS 10.10:
>
> http://pastebin.com/jApGDHs2
>
> Please tell me if you need anything else, I am running ImageJ 149p as Fiji.
> Thanks for your help !
>
> Best,
> Greg
>
>
>
>
> --
> View this message in context:
> http://imagej.1557.x6.nabble.com/Running-scripts-with-an-open-Virtual-Hyperstack-tp5012163p5012169.html
> Sent from the ImageJ mailing list archive at Nabble.com.
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Greg]

Hi Curtis,

thank you very much for that quick fixing response ! Unfortunately for me its quite late already, so that I can only do the testing when I am back at the institute tomorrow morning. It will be one of  the first things I do tomorrow and I report the results right back here !

Thanks again and greetings from Europe,
Greg

Am 25.03.2015 21:33 schrieb "ctrueden [via ImageJ]" <[hidden email]>:

Hi Greg,

> this is the stack dump right after opening the virtual stack

Thank you very much for that vital information. It identified a serious bug
in ImageJ2, which I believe is now fixed.

Would you mind testing it? You can find the relevant JAR files here:

  http://curtis.imagej.net/2015-03-25-bad-syncing/

Place the two JAR files in your ImageJ installation's jars folder, and
delete the previous versions.

Two major changes of note:

1) The new imagej-common hopefully fixes the crazy RAM flooding.

2) The new imagej-legacy fully eliminates the "Enable ImageJ2 data
structures" option so that users are not bitten by problems relating to it
in the future. Developers who need this feature can instead now set the
"imagej.legacy.sync" system property, if they want to test it.

Please let us know how it goes; I am optimistic that these changes will
greatly improve matters.

We plan to upload these fixes to the ImageJ update site by end of day
tomorrow, unless you report back with continued (or worse) problems.

Regards,
Curtis

On Wed, Mar 25, 2015 at 11:15 AM, Greg <[hidden email]> wrote:

> Hi Curtis,
>
> ya you are right, it might actually not related to scripts/plugins at all I
> think, this was just the first action I undertook after opening the virtual
> stack. That totally explains that slugginess ! The ImageJ2 datastructure
> option is and was disabled.
>
> So this is the stack dump right after opening the virtual stack, ca. 1.2Gb
> are used by Fiji at this moment:
>
> http://pastebin.com/ex1yG6JM
>
> The next dump is (hopefully) right in the moment when the RAM is filled
> rather quickly (around 8Gb at that very moment), this time to initiate the
> RAM flood I just started the script manager (without trying to run an empty
> script):
>
> http://pastebin.com/LucCkLcA
>
> After that, every [next slice] fills up the RAM a little more right now it
> is at 14.2Gb (the input file is 7Gb):
>
> http://pastebin.com/BmiEketC
>
> In the end Fiji is now almost unresponsive and filled 18Gb of my physical
> RAM (I got 32). Even when I close the virtual hyperstack, there are still
> 18Gb used as indicated by the activity Monitor from Mac OS 10.10:
>
> http://pastebin.com/jApGDHs2
>
> Please tell me if you need anything else, I am running ImageJ 149p as Fiji.
> Thanks for your help !
>
> Best,
> Greg
>
>
>
>
> --
> View this message in context:
> http://imagej.1557.x6.nabble.com/Running-scripts-with-an-open-Virtual-Hyperstack-tp5012163p5012169.html
> Sent from the ImageJ mailing list archive at Nabble.com.
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

---

If you reply to this email, your message will be added to the discussion below:

http://imagej.1557.x6.nabble.com/Running-scripts-with-an-open-Virtual-Hyperstack-tp5012163p5012175.html

To unsubscribe from Running scripts with an open Virtual Hyperstack, click here.
NAML

[Rasband, Wayne (NIH/NIMH) [E]]

> On Mar 25, 2015, at 9:51 AM, Greg <[hidden email]> wrote:
>
> Hi,
>
> today I witnessed a rather odd behavior of Fiji when trying to run my jython
> scripts on a opened virtual hyperstack. So basically it took ages to run my
> script so I stripped it down to the point, where I realized that you just
> have to try to run an empty (!) script from the script editor to see all
> your RAM eaten up. I tested that for jython and javascript.

You should be able to avoid these problems by saving the hyperstack in TIFF format and opening it using IJ.openVirtual(). As an example, the following JavaScript code, which opens a 3500x3500x3x10 hyperstack, switches between slices in 1/3 second and does not use an excessive amount of RAM.

  IJ.doCommand("Monitor Memory...");
  imp = IJ.openVirtual("");
  imp.show();
  IJ.run(imp, "Set... ", "zoom=100 x=0 y=0");
  times = new Array();
  slices = imp.getNSlices();
  for (i=2; i<=slices; i++) {
     t0 = System.currentTimeMillis();
     imp.setZ(i);
     times[i-1] = System.currentTimeMillis()-t0;
     print(i+"/"+slices+" "+times[i-1]);
  }
  sum = 0;
  for (i in times)
     sum += times[i];
  print("Average time: "+(sum/slices)/1000);

Use the following code to create the 10 slice test hyperstack. Each slice is 70MB (3500*3500*3*2).

  imp = IJ.createImage("HyperStack", "16-bit color-mode label", 3500, 3500, 3, 10, 1);
  IJ.saveAs(imp, "Tiff", "”);

TIFF files created by ImageJ larger than 4GB do not conform to the TIFF standard so other applications have to open them as raw files.

-wayne

> opened a 7gb ome.tiff with Bio Formats Importer in virtual hyperstack mode
> opened the script editor via file -> new -> script
> choosed python/java script as language
> pressed run on the empty(!) script
>
> What I see:
>
> When monitoring the RAM, I see it gets filled rather quickly by Fiji up to
> the size of 9.5Gb. For bigger input files more RAM gets 'used' accordingly.
> So for files exceeding my physical amount of Memory Fiji becomes
> unresponsive rather quickly. I experienced the same behavior for nd2 files.
>
> Is there a way to cirmumvent this ? I guess Fiji tries to load the whole
> image stack despite having chosen 'virtual stack' in the first place ?
>
> Best,
> Greg
>
> --
> View this message in context: http://imagej.1557.x6.nabble.com/Running-scripts-with-an-open-Virtual-Hyperstack-tp5012163.html
> Sent from the ImageJ mailing list archive at Nabble.com.

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Greg]

Hi,

so Curtis the fixes improved things for me ! I tried a few scripts and switched some frames, no real RAM flood in sight. When I open a 7Gb virtual stack Fiji stays stable at around 2.4Gb of RAM usage. When I open a 55Gb virtual stack, RAM usage increases to around 5.0Gb, but Fiji is still responsive. Albeit, when I close the virtual stack the RAM gets not freed. But when I reopen a virtual stack, no more RAM gets used. So for me I can totally work/live with that. Reading nd2 files this way is now much more sluggy than reading in the bfconverted ome.tiff files. Switching takes about 1secwith ome.tiff, with nd2 files it sometimes gets stuck and takes up to 3sec.

And @Wayne, as discussed with Curtis in this thread http://imagej.1557.x6.nabble.com/bfconverter-nd2-to-tif-td5012133.html , we have really huge (around 55Gb) Hyperstacks here, so converting them to ImageJ Tiff format does not really work for us, because we do not have enough physical memory to do the conversion!

Greets,
Greg

[ctrueden]

Hi Greg,

> the fixes improved things for me

Glad to hear it.

> when I close the virtual stack the RAM gets not freed.
> But when I reopen a virtual stack, no more RAM gets used.

Then I expect that ImageJ itself reports a reduction in RAM usage. See:
http://imagej.net/FAQ#Why_does_ImageJ_not_release_any_memory_back_to_the_system.3F

> Reading nd2 files this way is now much more sluggy than reading in the
> bfconverted ome.tiff files. Switching takes about 1secwith ome.tiff,
> with nd2 files it sometimes gets stuck and takes up to 3sec.

Not surprising; OME-TIFFs are written to disk in an
easy-to-read-plane-by-plane way.

Regards,
Curtis

On Thu, Mar 26, 2015 at 4:08 AM, Greg <[hidden email]> wrote:

> Hi,
>
> so Curtis the fixes improved things for me ! I tried a few scripts and
> switched some frames, no real RAM flood in sight. When I open a 7Gb virtual
> stack Fiji stays stable at around 2.4Gb of RAM usage. When I open a 55Gb
> virtual stack, RAM usage increases to around 5.0Gb, but Fiji is still
> responsive. Albeit, when I close the virtual stack the RAM gets not freed.
> But when I reopen a virtual stack, no more RAM gets used. So for me I can
> totally work/live with that. Reading nd2 files this way is now much more
> sluggy than reading in the bfconverted ome.tiff files. Switching takes
> about
> 1secwith ome.tiff, with nd2 files it sometimes gets stuck and takes up to
> 3sec.
>
> And @Wayne, as discussed with Curtis in this thread
> http://imagej.1557.x6.nabble.com/bfconverter-nd2-to-tif-td5012133.html ,
> we
> have really huge (around 55Gb) Hyperstacks here, so converting them to
> ImageJ Tiff format does not really work for us, because we do not have
> enough physical memory to do the conversion!
>
> Greets,
> Greg
>
>
>
> --
> View this message in context:
> http://imagej.1557.x6.nabble.com/Running-scripts-with-an-open-Virtual-Hyperstack-tp5012163p5012183.html
> Sent from the ImageJ mailing list archive at Nabble.com.
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html