ImageJ

Out of Memory Problems

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

18 messages Options

Olivier Burri

Out of Memory Problems

Hi all,

We're running a simple script to resize extremely large images to a more manageable size for imageJ.

The original image is in the vicinity of 40'000x40'000 pixels

What the plugin does (In macro language)
1. Split the image in 4096x4096 chunks
2. Load a single chunk, resize it
3. Place this chunk in a stack
4. Close the loaded chunk and its scaled counterpart, leaving only the stack.
5. Repeat for all chunks
6. When done, make a montage of the stack.

This used to work just fine, but on a recent trial, imageJ ran out of memory (We have 32 Gb on the system, most of which is allocated to the JVM and ImageJ sees and uses it).

Normally we used no more than a few Gb per resize, but as we load the image, the memory usage goes way up (>10Gb) but still succeeds.

However, when we run the script again, instead of reusing the previously allocated memory, it keeps rising until an out of memory error.

We've tried just opening and closing the same image over and over and memory usage keeps climbing up to the Out of memory error as well, suggesting this does not come from the macro code itself.

Has something changed in the way ImageJ/Fiji handles garbage cleanup?

Thanks for any info!

Oli

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

ctrueden

Re: Out of Memory Problems

Hi Oli,

Thanks for the report.

> Has something changed in the way ImageJ/Fiji handles garbage cleanup?

Can you please test whether the problem occurs with the "Life-Line" version
of Fiji available for download from:
http://fiji.sc/Downloads

This is the last version of Fiji before a recent update which began to
integrate some substantial changes. (Nothing specific to memory management,
but things that could conceivably affect it in some cases...)

Since it sounds like your macro does not use any Fiji-specific plugins, you
could also test with "vanilla" ImageJ 1.47 available from:
http://imagej.net/download.html

If the problem happens only with the latest Fiji, and not the Life-Line
version, it is probably due to those recent changes I mentioned, and we can
investigate further. Otherwise, I am not sure what is going on...

Regards,
Curtis

On Fri, Jul 19, 2013 at 8:08 AM, Burri Olivier <[hidden email]>wrote:

> Hi all,
>
> We're running a simple script to resize extremely large images to a more
> manageable size for imageJ.
>
> The original image is in the vicinity of 40'000x40'000 pixels
>
> What the plugin does (In macro language)
> 1. Split the image in 4096x4096 chunks
> 2. Load a single chunk, resize it
> 3. Place this chunk in a stack
> 4. Close the loaded chunk and its scaled counterpart, leaving only the
> stack.
> 5. Repeat for all chunks
> 6. When done, make a montage of the stack.
>
> This used to work just fine, but on a recent trial, imageJ ran out of
> memory (We have 32 Gb on the system, most of which is allocated to the JVM
> and ImageJ sees and uses it).
>
> Normally we used no more than a few Gb per resize, but as we load the
> image, the memory usage goes way up (>10Gb) but still succeeds.
>
> However, when we run the script again, instead of reusing the previously
> allocated memory, it keeps rising until an out of memory error.
>
> We've tried just opening and closing the same image over and over and
> memory usage keeps climbing up to the Out of memory error as well,
> suggesting this does not come from the macro code itself.
>
>
> Has something changed in the way ImageJ/Fiji handles garbage cleanup?
>
> Thanks for any info!
>
> Oli
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Olivier Burri

Re: Out of Memory Problems

Hi Curtis,

Thanks for the info, I downloaded the "Life Line" version
In terms of loading images (Just .tif files) as well as for our macro, I see the problem disappears so it does seem like it might come from the latest installement of Fiji.

If you want more detailed info, please let me know!

Best

Oli

Olivier Burri
Engineer - Image Processing
& Software Development
EPFL - SV - PTECH - PTBIOP

-----Original Message-----
From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of Curtis Rueden
Sent: vendredi 19 juillet 2013 19:29
To: [hidden email]
Subject: Re: Out of Memory Problems

Hi Oli,

Thanks for the report.

> Has something changed in the way ImageJ/Fiji handles garbage cleanup?

Can you please test whether the problem occurs with the "Life-Line" version of Fiji available for download from:
http://fiji.sc/Downloads

This is the last version of Fiji before a recent update which began to integrate some substantial changes. (Nothing specific to memory management, but things that could conceivably affect it in some cases...)

Since it sounds like your macro does not use any Fiji-specific plugins, you could also test with "vanilla" ImageJ 1.47 available from:
http://imagej.net/download.html

If the problem happens only with the latest Fiji, and not the Life-Line version, it is probably due to those recent changes I mentioned, and we can investigate further. Otherwise, I am not sure what is going on...

Regards,
Curtis

On Fri, Jul 19, 2013 at 8:08 AM, Burri Olivier <[hidden email]>wrote:

> Hi all,
>
> We're running a simple script to resize extremely large images to a
> more manageable size for imageJ.
>
> The original image is in the vicinity of 40'000x40'000 pixels
>
> What the plugin does (In macro language) 1. Split the image in
> 4096x4096 chunks 2. Load a single chunk, resize it 3. Place this chunk
> in a stack 4. Close the loaded chunk and its scaled counterpart,
> leaving only the stack.
> 5. Repeat for all chunks
> 6. When done, make a montage of the stack.
>
> This used to work just fine, but on a recent trial, imageJ ran out of
> memory (We have 32 Gb on the system, most of which is allocated to the
> JVM and ImageJ sees and uses it).
>
> Normally we used no more than a few Gb per resize, but as we load the
> image, the memory usage goes way up (>10Gb) but still succeeds.
>
> However, when we run the script again, instead of reusing the
> previously allocated memory, it keeps rising until an out of memory error.
>
> We've tried just opening and closing the same image over and over and
> memory usage keeps climbing up to the Out of memory error as well,
> suggesting this does not come from the macro code itself.
>
>
> Has something changed in the way ImageJ/Fiji handles garbage cleanup?
>
> Thanks for any info!
>
> Oli
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

dscho

Re: Out of Memory Problems

Hi Olivier,

On Mon, 22 Jul 2013, Burri Olivier wrote:

> Thanks for the info, I downloaded the "Life Line" version

It was Curtis' brilliant idea to force me to provide that version. I
cannot thank him enough!

> In terms of loading images (Just .tif files) as well as for our macro, I
> see the problem disappears so it does seem like it might come from the
> latest installement of Fiji.

I investigated and am almost certain to have solved the issue with this:

https://github.com/imagej/imagej/commit/849c2b9a

I also uploaded a new ij-legacy.jar from the fixed version.

May I ask you to update Fiji and test again?

Thanks,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Cameron Nowell-2

Re: Out of Memory Problems

In reply to this post by Olivier Burri

Hi Guys,

I would like to add that the life line versions fixed similar memory overflow problems when running scripts on a couple of Macs here as well.

Cheers

Cam

Cameron J. Nowell
Research Facilities Manager

Monash Institute of Pharmaceutical Sciences
Monash University
399 Royal Parade
(Mail address: 381 Royal Parade)
Parkville, VIC, 3052
Australia

Email: [hidden email]
Mobile: +61 422882700
Office: +61 9903 9587

LinkedIn: Profile
Research Gate: Profile

-----Original Message-----
From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of Burri Olivier
Sent: Tuesday, 23 July 2013 12:43 AM
To: [hidden email]
Subject: Re: Out of Memory Problems

Hi Curtis,

Thanks for the info, I downloaded the "Life Line" version In terms of loading images (Just .tif files) as well as for our macro, I see the problem disappears so it does seem like it might come from the latest installement of Fiji.

If you want more detailed info, please let me know!

Best

Oli

Olivier Burri
Engineer - Image Processing
& Software Development
EPFL - SV - PTECH - PTBIOP

-----Original Message-----
From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of Curtis Rueden
Sent: vendredi 19 juillet 2013 19:29
To: [hidden email]
Subject: Re: Out of Memory Problems

Hi Oli,

Thanks for the report.

> Has something changed in the way ImageJ/Fiji handles garbage cleanup?

Can you please test whether the problem occurs with the "Life-Line" version of Fiji available for download from:
http://fiji.sc/Downloads

This is the last version of Fiji before a recent update which began to integrate some substantial changes. (Nothing specific to memory management, but things that could conceivably affect it in some cases...)

Since it sounds like your macro does not use any Fiji-specific plugins, you could also test with "vanilla" ImageJ 1.47 available from:
http://imagej.net/download.html

If the problem happens only with the latest Fiji, and not the Life-Line version, it is probably due to those recent changes I mentioned, and we can investigate further. Otherwise, I am not sure what is going on...

Regards,
Curtis

On Fri, Jul 19, 2013 at 8:08 AM, Burri Olivier <[hidden email]>wrote:

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Olivier Burri

Re: Out of Memory Problems

In reply to this post by dscho

Dear Johannes,

Thanks for the email and the info.

I've tried loading a large dataset over and over through a macro (Load, close loop) after updating Fiji
And I still get the out of memory error.

Here it is, let me know if there's anything else you need.

Best

Oli

(Fiji Is Just) ImageJ 1.48a; Java 1.6.0_24 [64-bit]; Windows 7 6.1; 18404MB of 18411MB (99%)

java.lang.OutOfMemoryError: Java heap space
at loci.common.DataTools.makeDataArray(DataTools.java:574)
at loci.plugins.util.ImageProcessorReader.openProcessors(ImageProcessorReader.java:215)
at loci.plugins.in.ImagePlusReader.readProcessors(ImagePlusReader.java:416)
at loci.plugins.in.ImagePlusReader.readPlanes(ImagePlusReader.java:380)
at loci.plugins.in.ImagePlusReader.readImage(ImagePlusReader.java:277)
at loci.plugins.in.ImagePlusReader.readImages(ImagePlusReader.java:238)
at loci.plugins.in.ImagePlusReader.readImages(ImagePlusReader.java:216)
at loci.plugins.in.ImagePlusReader.openImagePlus(ImagePlusReader.java:114)
at loci.plugins.in.Importer.readPixels(Importer.java:150)
at loci.plugins.in.Importer.run(Importer.java:89)
at loci.plugins.LociImporter.run(LociImporter.java:81)
at ij.IJ.runUserPlugIn(IJ.java:196)
at ij.IJ.runPlugIn(IJ.java:160)
at ij.IJ.runPlugIn(IJ.java:149)
at HandleExtraFileTypes.openImage(HandleExtraFileTypes.java:421)
at HandleExtraFileTypes.run(HandleExtraFileTypes.java:57)
at ij.IJ.runUserPlugIn(IJ.java:196)
at ij.IJ.runPlugIn(IJ.java:160)
at ij.IJ.runPlugIn(IJ.java:149)
at ij.io.Opener.openWithHandleExtraFileTypes(Opener.java:438)
at ij.io.Opener.openImage(Opener.java:310)
at ij.io.Opener.openImage(Opener.java:333)
at ij.io.Opener.open(Opener.java:143)
at ij.io.Opener.openAndAddToRecent(Opener.java:246)
at ij.plugin.DragAndDrop.openFile(DragAndDrop.java:176)
at ij.plugin.DragAndDrop.run(DragAndDrop.java:152)
at java.lang.Thread.run(Thread.java:662)

Olivier Burri
Engineer - Image Processing
& Software Development
EPFL - SV - PTECH - PTBIOP

-----Original Message-----
From: Johannes Schindelin [mailto:[hidden email]]
Sent: lundi 22 juillet 2013 18:15
To: Burri Olivier
Cc: [hidden email]
Subject: Re: Out of Memory Problems

Hi Olivier,

On Mon, 22 Jul 2013, Burri Olivier wrote:

> Thanks for the info, I downloaded the "Life Line" version

It was Curtis' brilliant idea to force me to provide that version. I cannot thank him enough!

> In terms of loading images (Just .tif files) as well as for our macro,
> I see the problem disappears so it does seem like it might come from
> the latest installement of Fiji.

I investigated and am almost certain to have solved the issue with this:

https://github.com/imagej/imagej/commit/849c2b9a

I also uploaded a new ij-legacy.jar from the fixed version.

May I ask you to update Fiji and test again?

Thanks,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

dscho

Re: Out of Memory Problems

Hi Olivier,

On Tue, 23 Jul 2013, Burri Olivier wrote:

> I've tried loading a large dataset over and over through a macro (Load,
> close loop) after updating Fiji And I still get the out of memory error.

Well, this is my fault. For some reason, I failed to upload a new version
of ij-legacy.jar when I said that I had. But now it worked, and this
beautiful macro demonstrates that my fix works at least with the Clown
sample (I was unable to run Plugins>Utilities>Monitor Memory... from the
macro without blocking the rest of the commands, so please call that by
hand before running it):

path = getDirectory("imagej") + "samples/clown.jpg";
useBF = true;
setBatchMode(true);
for (i = 0; i < 100; i++) {
if (useBF) {
run("Bio-Formats", "open=[" + path + "] "
+ "autoscale color_mode=Default "
+ "view=Hyperstack stack_order=XYCZT");
} else {
open(path);
}
close();
}

For easy debugging and extensive testing, I also tried with "!true" for
both useBF and setBatchMode(), and the issues that I could indeed
reproduce with the Fiji version as of half an hour ago are now gone.

Ciao,
Dscho

P.S.: The macro assumes that you have unpacked Fiji into your home
directory and that you ran File>Open Samples>Cache Sample Images. This is
so far a Fiji-only function (Wayne, if you read this and re-implement it
for ImageJ 1.x, please let me know so that I can prevent breakages in
Fiji).

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Rasband, Wayne (NIH/NIMH) [E]

Re: Out of Memory Problems

On Jul 23, 2013, at 1:34 PM, Johannes Schindelin wrote:

> Hi Olivier,
>
> On Tue, 23 Jul 2013, Burri Olivier wrote:
>
>> I've tried loading a large dataset over and over through a macro (Load,
>> close loop) after updating Fiji And I still get the out of memory error.
>
> Well, this is my fault. For some reason, I failed to upload a new version
> of ij-legacy.jar when I said that I had. But now it worked, and this
> beautiful macro demonstrates that my fix works at least with the Clown
> sample (I was unable to run Plugins>Utilities>Monitor Memory... from the
> macro without blocking the rest of the commands, so please call that by
> hand before running it):

A macro can run Plugins>Utilities>Monitor Memory without blocking by using the doCommand() macro function, which runs a menu command in a separate thread. For example, this macro starts the Memory Monitor and then opens and closes 5000 1MB images.

setBatchMode(true);
doCommand("Monitor Memory...");
n = 5000;
for (i = 0; i < n; i++) {
showStatus((i+1)+"/"+n);
newImage("Untitled", "8-bit ramp", 1024, 1024, 1);
close();
}

-wayne

> path = getDirectory("imagej") + "samples/clown.jpg";
> useBF = true;
> setBatchMode(true);
> for (i = 0; i < 100; i++) {
> if (useBF) {
> run("Bio-Formats", "open=[" + path + "] "
> + "autoscale color_mode=Default "
> + "view=Hyperstack stack_order=XYCZT");
> } else {
> open(path);
> }
> close();
> }
>
> For easy debugging and extensive testing, I also tried with "!true" for
> both useBF and setBatchMode(), and the issues that I could indeed
> reproduce with the Fiji version as of half an hour ago are now gone.
>
> Ciao,
> Dscho
>
> P.S.: The macro assumes that you have unpacked Fiji into your home
> directory and that you ran File>Open Samples>Cache Sample Images. This is
> so far a Fiji-only function (Wayne, if you read this and re-implement it
> for ImageJ 1.x, please let me know so that I can prevent breakages in
> Fiji).
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Hans Wurscht

Re: Out of Memory Problems

In reply to this post by Olivier Burri

Dear List,

I experience the same problem when using the doCommand in a macro:

openImage();

macro "open image in new thread"{
newImage("Untitled", "8-bit white", 400, 400, 1);

}

function openImage(){
setBatchMode(true);
for(i=0;i<1000;i++){
doCommand("open image in new thread");
wait(100);
}
wait(3000);
run("Close All");
setBatchMode(false);

}

Once finished, you can see that the threads are terminated, but the
memory is not freed up.

Running the garbage collector from the menu does not help. I use the
doComand option to run some live analyis during timelapse recordings,
and this bug leads to an out of memory error and failue of the
analysis.

I tried IJ 1.47u, 1.48a with openJDK 6 / 7 on Ubuntu 12.04 32bit
without success.

Do you have any suggestions for me?

Thanks a lot,

Gabriel

On Jul 23, 2013, at 1:34 PM, Johannes Schindelin wrote:

A macro can run Plugins>Utilities>Monitor Memory without blocking by
using the doCommand() macro function, which runs a menu command in a
separate thread. For example, this macro starts the Memory Monitor and
then opens and closes 5000 1MB images.

setBatchMode(true);
doCommand("Monitor Memory...");
n = 5000;
for (i = 0; i < n; i++) {
showStatus((i+1)+"/"+n);
newImage("Untitled", "8-bit ramp", 1024, 1024, 1);
close();
}

-wayne

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Top of Message<https://list.nih.gov/cgi-bin/wa.exe?A2=ind1307&L=IMAGEJ&P=R56424&1=IMAGEJ&9=A&J=on&d=No+Match%3BMatch%3BMatches&z=4#TOP>|
Previous
Page<https://list.nih.gov/cgi-bin/wa.exe?A1=ind1307&L=IMAGEJ&D=0&1=IMAGEJ&9=A&J=on&d=No+Match%3BMatch%3BMatches&z=4>|
Permalink <https://list.nih.gov/cgi-bin/wa.exe?A2=IMAGEJ;b6ecca0f.1307>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

dscho

Re: Out of Memory Problems

Hi Hans Wurscht (your real name? If so: awesome!),

On Thu, 25 Jul 2013, Hans Wurscht wrote:

> I experience the same problem when using the doCommand in a macro:
>
> openImage();
>
> macro "open image in new thread"{
> newImage("Untitled", "8-bit white", 400, 400, 1);
> }
>
> function openImage(){
> setBatchMode(true);
> for(i=0;i<1000;i++){
> doCommand("open image in new thread");
> wait(100);
> }
> wait(3000);
> run("Close All");
> setBatchMode(false);
> }
>
> Once finished, you can see that the threads are terminated, but the
> memory is not freed up.

Most likely, the doCommand()-started threads terminate *after*
setBatchMode(false).

> Running the garbage collector from the menu does not help. I use the
> doComand option to run some live analyis during timelapse recordings,
> and this bug leads to an out of memory error and failue of the
> analysis.
>
> I tried IJ 1.47u, 1.48a with openJDK 6 / 7 on Ubuntu 12.04 32bit
> without success.

Are you using Fiji? Have you tried upgrading to IJ 1.48a (the current
version)?

Ciao,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Michael Schmid

Re: Out of Memory Problems

In reply to this post by Hans Wurscht

Hi Hans & everyone,

It is the same memory leak with (Sun/Oracle/Apple) Java 1.6 on Mac OS X 10.6.8, so it is not a problem of the Java virtual machine.

Strange enough, "Close All" seems to nicely clean the list of BatchMode images: print(nImages) gives the full number before, but 0 thereafter.

Nevertheless, these images seem to be not garbage-collectable.

It's the same problem if:
- I open a .jpg instead of creating a new 8-bit image in the separate thread, so 'newImage' does not cause the problem.
- I replace doCommand by run("open image in new thread");

I have looked into the sources but found nothing where a reference to these images would remain. The only remote possibility is that java keeps references to the old threads somewhere, and the thread keeps a reference to the Interpreter, where the ImagePlus is referred to as batchMacroImage.

In any case, maybe ImagePlus.close() should call flush() if there is no image window, to make sure everything is cleaned up?
(For images that have an associated ImageWindow, closing the ImageWindow calls flush of the ImagePlus.)
I don't understand why it should be a problem in the present case, but there might be cases with ROIs associate to an ImagePlus, there the roi holds a reference to the ImagePlus and vice versa, and it won't be garbage collectable in BatchMode.

Michael
________________________________________________________________
On Jul 25, 2013, at 15:14, Hans Wurscht wrote:

> Dear List,
>
> I experience the same problem when using the doCommand in a macro:
>
> openImage();
>
> macro "open image in new thread"{
> newImage("Untitled", "8-bit white", 400, 400, 1);
>
> }
>
> function openImage(){
> setBatchMode(true);
> for(i=0;i<1000;i++){
> doCommand("open image in new thread");
> wait(100);
> }
> wait(3000);
> run("Close All");
> setBatchMode(false);
>
> }
>
> Once finished, you can see that the threads are terminated, but the
> memory is not freed up.
>
> Running the garbage collector from the menu does not help. I use the
> doComand option to run some live analyis during timelapse recordings,
> and this bug leads to an out of memory error and failue of the
> analysis.
>
> I tried IJ 1.47u, 1.48a with openJDK 6 / 7 on Ubuntu 12.04 32bit
> without success.
>
> Do you have any suggestions for me?
>
> Thanks a lot,
>
> Gabriel
>
>
> On Jul 23, 2013, at 1:34 PM, Johannes Schindelin wrote:
>
>> Hi Olivier,
>>
>> On Tue, 23 Jul 2013, Burri Olivier wrote:
>>
>>> I've tried loading a large dataset over and over through a macro (Load,
>>> close loop) after updating Fiji And I still get the out of memory error.
>>
>> Well, this is my fault. For some reason, I failed to upload a new version
>> of ij-legacy.jar when I said that I had. But now it worked, and this
>> beautiful macro demonstrates that my fix works at least with the Clown
>> sample (I was unable to run Plugins>Utilities>Monitor Memory... from the
>> macro without blocking the rest of the commands, so please call that by
>> hand before running it):
>
> A macro can run Plugins>Utilities>Monitor Memory without blocking by
> using the doCommand() macro function, which runs a menu command in a
> separate thread. For example, this macro starts the Memory Monitor and
> then opens and closes 5000 1MB images.
>
> setBatchMode(true);
> doCommand("Monitor Memory...");
> n = 5000;
> for (i = 0; i < n; i++) {
> showStatus((i+1)+"/"+n);
> newImage("Untitled", "8-bit ramp", 1024, 1024, 1);
> close();
> }
>
> -wayne
>
>> path = getDirectory("imagej") + "samples/clown.jpg";
>> useBF = true;
>> setBatchMode(true);
>> for (i = 0; i < 100; i++) {
>> if (useBF) {
>> run("Bio-Formats", "open=[" + path + "] "
>> + "autoscale color_mode=Default "
>> + "view=Hyperstack stack_order=XYCZT");
>> } else {
>> open(path);
>> }
>> close();
>> }
>>
>> For easy debugging and extensive testing, I also tried with "!true" for
>> both useBF and setBatchMode(), and the issues that I could indeed
>> reproduce with the Fiji version as of half an hour ago are now gone.
>>
>> Ciao,
>> Dscho
>>
>> P.S.: The macro assumes that you have unpacked Fiji into your home
>> directory and that you ran File>Open Samples>Cache Sample Images. This is
>> so far a Fiji-only function (Wayne, if you read this and re-implement it
>> for ImageJ 1.x, please let me know so that I can prevent breakages in
>> Fiji).
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> Top of Message<https://list.nih.gov/cgi-bin/wa.exe?A2=ind1307&L=IMAGEJ&P=R56424&1=IMAGEJ&9=A&J=on&d=No+Match%3BMatch%3BMatches&z=4#TOP>|
> Previous
> Page<https://list.nih.gov/cgi-bin/wa.exe?A1=ind1307&L=IMAGEJ&D=0&1=IMAGEJ&9=A&J=on&d=No+Match%3BMatch%3BMatches&z=4>|
> Permalink <https://list.nih.gov/cgi-bin/wa.exe?A2=IMAGEJ;b6ecca0f.1307>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

dscho

Re: Out of Memory Problems

Hi Michael & Hans,

On Thu, 25 Jul 2013, Michael Schmid wrote:

> It is the same memory leak with (Sun/Oracle/Apple) Java 1.6 on Mac OS X
> 10.6.8, so it is not a problem of the Java virtual machine.
>
> Strange enough, "Close All" seems to nicely clean the list of BatchMode
> images: print(nImages) gives the full number before, but 0 thereafter.
>
> Nevertheless, these images seem to be not garbage-collectable.

Your mail got me curious because I know that you test with bare-bones
ImageJ 1.48a. So I did something very similar and after the memory was not
released, called JVisualVM (part of the JDK since Java 1.6), attached to
the appropriate Java instance, clicked on the "Monitor" tab's "Heap Dump"
table, and in the "Classes" tab of the newly opened tab, sorted by size
(yes, this is a little bit involved, but it does a good job of letting you
debug non-GC'able objects) and sure enough, there were tons of byte[]
which accumulated to a very large size.

Turns out that the culprit is WindowManager's tempImageTable which
associates a temporary image with each thread (which is a little fragile
binding but works most of the time).

So I guess that Executer's run() method needs a try { ... } finally {
WindowManager.setTempCurrentImage(null); }.

Wayne, I opened a pull request with a fix:
https://github.com/fiji/imagej1/pull/2

Ciao,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Rasband, Wayne (NIH/NIMH) [E]

Re: Out of Memory Problems

In reply to this post by Hans Wurscht

On Jul 25, 2013, at 9:14 AM, Hans Wurscht wrote:

> Dear List,
>
> I experience the same problem when using the doCommand in a macro:

Thanks to Johannes Schindelin, this bug is fixed in the ImageJ 1.48b daily build. Use the following macro to demonstrate both the problem and the fix. Run it using ImageJ 1.48b1 and the memory used to open the 100 images is reclaimed when you run the garbage collector by clicking in the "Memory" window. With ImageJ 1.48a and earlier, the memory is not reclaimed when you click in the "Memory" window.

-wayne

doCommand("Monitor Memory...");
setBatchMode(true);
for(i=0;i<100;i++) {
run("Open Image");
wait(20);
}
run("Close All");
exit;

macro "Open Image" {
newImage("Untitled", "8-bit ramp", 1024, 1024, 1);
}

> openImage();
>
> macro "open image in new thread"{
> newImage("Untitled", "8-bit white", 400, 400, 1);
>
> }
>
> function openImage(){
> setBatchMode(true);
> for(i=0;i<1000;i++){
> doCommand("open image in new thread");
> wait(100);
> }
> wait(3000);
> run("Close All");
> setBatchMode(false);
>
> }
>
> Once finished, you can see that the threads are terminated, but the
> memory is not freed up.
>
> Running the garbage collector from the menu does not help. I use the
> doComand option to run some live analyis during timelapse recordings,
> and this bug leads to an out of memory error and failue of the
> analysis.
>
> I tried IJ 1.47u, 1.48a with openJDK 6 / 7 on Ubuntu 12.04 32bit
> without success.
>
> Do you have any suggestions for me?
>
> Thanks a lot,
>
> Gabriel
>
>
> On Jul 23, 2013, at 1:34 PM, Johannes Schindelin wrote:
>
>> Hi Olivier,
>>
>> On Tue, 23 Jul 2013, Burri Olivier wrote:
>>
>>> I've tried loading a large dataset over and over through a macro (Load,
>>> close loop) after updating Fiji And I still get the out of memory error.
>>
>> Well, this is my fault. For some reason, I failed to upload a new version
>> of ij-legacy.jar when I said that I had. But now it worked, and this
>> beautiful macro demonstrates that my fix works at least with the Clown
>> sample (I was unable to run Plugins>Utilities>Monitor Memory... from the
>> macro without blocking the rest of the commands, so please call that by
>> hand before running it):
>
> A macro can run Plugins>Utilities>Monitor Memory without blocking by
> using the doCommand() macro function, which runs a menu command in a
> separate thread. For example, this macro starts the Memory Monitor and
> then opens and closes 5000 1MB images.
>
> setBatchMode(true);
> doCommand("Monitor Memory...");
> n = 5000;
> for (i = 0; i < n; i++) {
> showStatus((i+1)+"/"+n);
> newImage("Untitled", "8-bit ramp", 1024, 1024, 1);
> close();
> }
>
> -wayne
>
>> path = getDirectory("imagej") + "samples/clown.jpg";
>> useBF = true;
>> setBatchMode(true);
>> for (i = 0; i < 100; i++) {
>> if (useBF) {
>> run("Bio-Formats", "open=[" + path + "] "
>> + "autoscale color_mode=Default "
>> + "view=Hyperstack stack_order=XYCZT");
>> } else {
>> open(path);
>> }
>> close();
>> }
>>
>> For easy debugging and extensive testing, I also tried with "!true" for
>> both useBF and setBatchMode(), and the issues that I could indeed
>> reproduce with the Fiji version as of half an hour ago are now gone.
>>
>> Ciao,
>> Dscho
>>
>> P.S.: The macro assumes that you have unpacked Fiji into your home
>> directory and that you ran File>Open Samples>Cache Sample Images. This is
>> so far a Fiji-only function (Wayne, if you read this and re-implement it
>> for ImageJ 1.x, please let me know so that I can prevent breakages in
>> Fiji).
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> Top of Message<https://list.nih.gov/cgi-bin/wa.exe?A2=ind1307&L=IMAGEJ&P=R56424&1=IMAGEJ&9=A&J=on&d=No+Match%3BMatch%3BMatches&z=4#TOP>|
> Previous
> Page<https://list.nih.gov/cgi-bin/wa.exe?A1=ind1307&L=IMAGEJ&D=0&1=IMAGEJ&9=A&J=on&d=No+Match%3BMatch%3BMatches&z=4>|
> Permalink <https://list.nih.gov/cgi-bin/wa.exe?A2=IMAGEJ;b6ecca0f.1307>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Hans Wurscht

Re: Out of Memory Problems

Dear Johannes, Michael and Wayne

Thanks a a lot for your efforts!

Cheers,
Hans

2013/7/26 Rasband, Wayne (NIH/NIMH) [E] <[hidden email]>

> On Jul 25, 2013, at 9:14 AM, Hans Wurscht wrote:
>
> > Dear List,
> >
> > I experience the same problem when using the doCommand in a macro:
>
> Thanks to Johannes Schindelin, this bug is fixed in the ImageJ 1.48b daily
> build. Use the following macro to demonstrate both the problem and the fix.
> Run it using ImageJ 1.48b1 and the memory used to open the 100 images is
> reclaimed when you run the garbage collector by clicking in the "Memory"
> window. With ImageJ 1.48a and earlier, the memory is not reclaimed when you
> click in the "Memory" window.
>
> -wayne
>
> doCommand("Monitor Memory...");
> setBatchMode(true);
> for(i=0;i<100;i++) {
> run("Open Image");
> wait(20);
> }
> run("Close All");
> exit;
>
> macro "Open Image" {
> newImage("Untitled", "8-bit ramp", 1024, 1024, 1);
> }
>
>
> > openImage();
> >
> > macro "open image in new thread"{
> > newImage("Untitled", "8-bit white", 400, 400, 1);
> >
> > }
> >
> > function openImage(){
> > setBatchMode(true);
> > for(i=0;i<1000;i++){
> > doCommand("open image in new thread");
> > wait(100);
> > }
> > wait(3000);
> > run("Close All");
> > setBatchMode(false);
> >
> > }
> >
> > Once finished, you can see that the threads are terminated, but the
> > memory is not freed up.
> >
> > Running the garbage collector from the menu does not help. I use the
> > doComand option to run some live analyis during timelapse recordings,
> > and this bug leads to an out of memory error and failue of the
> > analysis.
> >
> > I tried IJ 1.47u, 1.48a with openJDK 6 / 7 on Ubuntu 12.04 32bit
> > without success.
> >
> > Do you have any suggestions for me?
> >
> > Thanks a lot,
> >
> > Gabriel
> >
> >
> > On Jul 23, 2013, at 1:34 PM, Johannes Schindelin wrote:
> >
> >> Hi Olivier,
> >>
> >> On Tue, 23 Jul 2013, Burri Olivier wrote:
> >>
> >>> I've tried loading a large dataset over and over through a macro (Load,
> >>> close loop) after updating Fiji And I still get the out of memory
> error.
> >>
> >> Well, this is my fault. For some reason, I failed to upload a new
> version
> >> of ij-legacy.jar when I said that I had. But now it worked, and this
> >> beautiful macro demonstrates that my fix works at least with the Clown
> >> sample (I was unable to run Plugins>Utilities>Monitor Memory... from the
> >> macro without blocking the rest of the commands, so please call that by
> >> hand before running it):
> >
> > A macro can run Plugins>Utilities>Monitor Memory without blocking by
> > using the doCommand() macro function, which runs a menu command in a
> > separate thread. For example, this macro starts the Memory Monitor and
> > then opens and closes 5000 1MB images.
> >
> > setBatchMode(true);
> > doCommand("Monitor Memory...");
> > n = 5000;
> > for (i = 0; i < n; i++) {
> > showStatus((i+1)+"/"+n);
> > newImage("Untitled", "8-bit ramp", 1024, 1024, 1);
> > close();
> > }
> >
> > -wayne
> >
> >> path = getDirectory("imagej") + "samples/clown.jpg";
> >> useBF = true;
> >> setBatchMode(true);
> >> for (i = 0; i < 100; i++) {
> >> if (useBF) {
> >> run("Bio-Formats", "open=[" + path + "] "
> >> + "autoscale color_mode=Default "
> >> + "view=Hyperstack stack_order=XYCZT");
> >> } else {
> >> open(path);
> >> }
> >> close();
> >> }
> >>
> >> For easy debugging and extensive testing, I also tried with "!true" for
> >> both useBF and setBatchMode(), and the issues that I could indeed
> >> reproduce with the Fiji version as of half an hour ago are now gone.
> >>
> >> Ciao,
> >> Dscho
> >>
> >> P.S.: The macro assumes that you have unpacked Fiji into your home
> >> directory and that you ran File>Open Samples>Cache Sample Images. This
> is
> >> so far a Fiji-only function (Wayne, if you read this and re-implement it
> >> for ImageJ 1.x, please let me know so that I can prevent breakages in
> >> Fiji).
> >>
> >> --
> >> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >
> > Top of Message<
> https://list.nih.gov/cgi-bin/wa.exe?A2=ind1307&L=IMAGEJ&P=R56424&1=IMAGEJ&9=A&J=on&d=No+Match%3BMatch%3BMatches&z=4#TOP
> >|
> > Previous
> > Page<
> https://list.nih.gov/cgi-bin/wa.exe?A1=ind1307&L=IMAGEJ&D=0&1=IMAGEJ&9=A&J=on&d=No+Match%3BMatch%3BMatches&z=4
> >|
> > Permalink <https://list.nih.gov/cgi-bin/wa.exe?A2=IMAGEJ;b6ecca0f.1307>
> >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Gabriel Landini

Re: Out of Memory Problems

In reply to this post by Rasband, Wayne (NIH/NIMH) [E]

On Friday 26 Jul 2013 05:22:31 Rasband, Wayne [E] wrote:
> With ImageJ 1.48a and earlier, the memory is not reclaimed when you
> click in the "Memory" window.

Is this code supposed to work if pasted all in a macro window (it does not
here) or does it have to be split into 2 macros?

> doCommand("Monitor Memory...");
> setBatchMode(true);
> for(i=0;i<100;i++) {
> run("Open Image");
> wait(20);
> }
> run("Close All");
> exit;
>
> macro "Open Image" {
> newImage("Untitled", "8-bit ramp", 1024, 1024, 1);
> }
>

If I replace run("Open Image"); with the line newImage(... etc
(i.e. avoid having 2 macros) the memory consumption increases and it is never
released by the garbage collector. IJ1.48b1 Java 1.6.0_35 (64bit linux).
Sorry if I misunderstood

Cheers
Gabriel

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Rasband, Wayne (NIH/NIMH) [E]

Re: Out of Memory Problems

On Jul 26, 2013, at 7:59 AM, Gabriel Landini wrote:

> On Friday 26 Jul 2013 05:22:31 Rasband, Wayne [E] wrote:
>> With ImageJ 1.48a and earlier, the memory is not reclaimed when you
>> click in the "Memory" window.
>
> Is this code supposed to work if pasted all in a macro window (it does not
> here) or does it have to be split into 2 macros?

You have to first install it using the Macros>Install Macros command, which creates a Plugins>Macros>Open Image command.

>> doCommand("Monitor Memory...");
>> setBatchMode(true);
>> for(i=0;i<100;i++) {
>> run("Open Image");
>> wait(20);
>> }
>> run("Close All");
>> exit;
>>
>> macro "Open Image" {
>> newImage("Untitled", "8-bit ramp", 1024, 1024, 1);
>> }
>>
>
> If I replace run("Open Image"); with the line newImage(... etc
> (i.e. avoid having 2 macros) the memory consumption increases and it is never
> released by the garbage collector. IJ1.48b1 Java 1.6.0_35 (64bit linux).
> Sorry if I misunderstood

I am unable to reproduce this problem with either v1.48a or v1.48b1 using Java 1.6.0_24 and 32-bit Linux. Did you try forcing the garbage collector to run by clicking in the "Memory" window?

-wayne
--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Gabriel Landini

Re: Out of Memory Problems

On Friday 26 Jul 2013 15:43:11 Rasband, Wayne [E] wrote:

> On Jul 26, 2013, at 7:59 AM, Gabriel Landini wrote:
> > On Friday 26 Jul 2013 05:22:31 Rasband, Wayne [E] wrote:
> >> With ImageJ 1.48a and earlier, the memory is not reclaimed when you
> >> click in the "Memory" window.
> >
> > Is this code supposed to work if pasted all in a macro window (it does not
> > here) or does it have to be split into 2 macros?
>
> You have to first install it using the Macros>Install Macros command, which
> creates a Plugins>Macros>Open Image command.

Oh, thanks. OK, the Open Image command is shown now.

I just noted that I had in my run file these switches:

-Xincgc -XX:+DisableExplicitGC

without them the memory window regains the space when clicker.
I do not remember when I added those. Any suggestions if they should be there
or not?

Many thanks

Gabriel

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Seth Pappas

Simple column shift plugin?

In reply to this post by Rasband, Wayne (NIH/NIMH) [E]

Hello,
Is there a simple way to shift a column of pixels to the left or right?
Technically it would be a wrap-around ....for instance in a 640 x 480 array, I
would like column 640 to become column 1, and then all the other columns shifted over by 1.

Same with row shifting....

Any help would be appreciated.

Thanks,
Seth Pappas

-----Original Message-----
From: Rasband, Wayne (NIH/NIMH) [E] (NIH/NIMH) [E] <[hidden email]>
To: IMAGEJ <[hidden email]>
Sent: Fri, Jul 26, 2013 10:45 am
Subject: Re: Out of Memory Problems

On Jul 26, 2013, at 7:59 AM, Gabriel Landini wrote:

> On Friday 26 Jul 2013 05:22:31 Rasband, Wayne [E] wrote:
>> With ImageJ 1.48a and earlier, the memory is not reclaimed when you
>> click in the "Memory" window.
>
> Is this code supposed to work if pasted all in a macro window (it does not
> here) or does it have to be split into 2 macros?

You have to first install it using the Macros>Install Macros command, which
creates a Plugins>Macros>Open Image command.

I am unable to reproduce this problem with either v1.48a or v1.48b1 using Java
1.6.0_24 and 32-bit Linux. Did you try forcing the garbage collector to run by
clicking in the "Memory" window?

-wayne
--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html