memory usage buildup

[JaapK]

Hello,

I have a plugin that processes a large number of 10MP image files. It
does a particle analysis, draws some additional things in the output
image, saves it, does some more processing, saves some data, enfin: a
lot.
It runs alright, but when I monitor the amount of memory that imagej
occupies, I see it increasing rapidly. I have 8GB of internal memory
(on a 64 bit Windows 7 machine) which is completely occupied after
processing about 90 images.

Could it be a problem of garbage collection? Is there any way to force
ImageJ to do that? I should also note that the plugin is written in
Jython. It is executed from within a wrapper plugin that instantiates
a JythonInterpreter object.

Can anyone help?

Kind regards,
Jaap Kokorian

[Michael Doube]

Hi Jaap,

> I have a plugin that processes a large number of 10MP image files. It
> does a particle analysis, draws some additional things in the output
> image, saves it, does some more processing, saves some data, enfin: a
> lot.
> It runs alright, but when I monitor the amount of memory that imagej
> occupies, I see it increasing rapidly. I have 8GB of internal memory
> (on a 64 bit Windows 7 machine) which is completely occupied after
> processing about 90 images.
>
> Could it be a problem of garbage collection? Is there any way to force
> ImageJ to do that? I should also note that the plugin is written in
> Jython. It is executed from within a wrapper plugin that instantiates
> a JythonInterpreter object.

It sounds like a memory leak.  In my experience the garbage collector
has not been to blame in this situation, instead bad programming can
leave objects in a state in which they are unable to be garbage collected.

The Sun JDK comes with VisualVM with which you can run a set of profiles
on ImageJ (and the offending plugin); from these you can hopefully find
the leak.  I found this guide very helpful:
http://rejeev.blogspot.com/2009/04/analyzing-memory-leak-in-java.html

Michael

[Cammer, Michael]

For years I've been having problems on Windows Vista 32 bit systems with memory filling up.  Fortunately, there are two ways to deal with this.  One is that now when you click on the status bar, ImageJ does garbage collection which works great.  In long macros with batch processing I have the macro call the garbage collection plugin which you can get on the ImageJ website.  You need to call it a few times.  
Sincerely,
Michael C.

-----Original Message-----


It sounds like a memory leak.  


------------------------------------------------------------
This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email.
=================================

[Michael Doube]

> when you click on the status bar, ImageJ does
> garbage collection which works great.  In long macros with batch
> processing I have the macro call the garbage collection plugin which
> you can get on the ImageJ website.  You need to call it a few times.

For sure, if that works then ignore my suggestion, which is much more
complicated. If you find that your plugin is resistant to manual garbage
collection, you'll need to probe deeper.

Michael

[JaapK]

I have run the garbage collector at the start of every iteration and I
set all imageprocessors, imageplus's, particleanalyzers etc to "None"
manually (the jython equivalent of 'null'). Also, I used the visualVM
tool to check a bit what was going on.
I managed to decrease the buildup of memory, but I haven't been able
to stabilize it completely. It might be a 'non-proper' programming
thing after all. The script is becoming a bit cluttered anyway. I am
going to convert from jython to pure java and tidy it up in the
process. Hopefully that will solve it.

Thanks for your support!

Regards,
Jaap Kokorian

2011/1/3 Michael Doube <[hidden email]>:

>> when you click on the status bar, ImageJ does
>> garbage collection which works great.  In long macros with batch
>> processing I have the macro call the garbage collection plugin which
>> you can get on the ImageJ website.  You need to call it a few times.
>
> For sure, if that works then ignore my suggestion, which is much more
> complicated. If you find that your plugin is resistant to manual garbage
> collection, you'll need to probe deeper.
>
> Michael
>

[JaapK]

Hello again,

After some futile attempts involving a lot of redundant calls to
System.gc(), I solved the memory leak thanks to a day of diagnostics
with visualVM.

The leak occurred in a static function I wrote. It took an ImagePlus
with a Roi that had been set to it (using imp.setRoi(r) ) as an
argument. The function then got the ImageProcessor of the ImagePlus by
invoking imp.getProcessor(), and did some drawing operations on the
image, within the Roi. The function returned an integer.

This is what went wrong: in Java you are not supposed to pass object
by reference as I did with the ImagePlus object. Apparently some kind
of a ghost ImageProcessor or ImagePlus was created in the function
that could not be garbage collected.

I rewrote the function to accept an ImageProcessor and a Roi
separately and made it return a new ImageProcessor. Problem solved.

Thank you very much for all the advice.

Kind regards,
Jaap Kokorian

Op 13 januari 2011 09:09:05 UTC+1 heeft Jaap Kokorian
<[hidden email]> het volgende geschreven:

> Hello again,
>
> After some futile attempts involving a lot of redundant calls to System.gc(), I solved the memory leak thanks to a day of diagnostics with visualVM.
>
> The leak occurred in a static function I wrote that took an ImagePlus with a roi that had been set to it (using imp.setRoi(r) ) as an argument. The function the got the ImageProcessor of the ImagePlus by invoking imp.getProcessor(), and did some drawing operations on the image within the roi. The function returned an integer.
>
> This is what went wrong: in Java you are not supposed to pass object by reference as I did with the ImagePlus object. Apparently some kind of a ghost ImageProcessor or ImagePlus was created in the function that could not be garbage collected.
>
> I rewrote the function to accept an ImageProcessor and a Roi separately and made it return a new ImageProcessor. Problem solved.
>
> Thank you very much for all the advice.
>
> Kind regards,
> Jaap Kokorian
>
>
> 2011/1/5 Jaap Kokorian <[hidden email]>:
>> I have run the garbage collector at the start of every iteration and I
>> set all imageprocessors, imageplus's, particleanalyzers etc to "None"
>> manually (the jython equivalent of 'null'). Also, I used the visualVM
>> tool to check a bit what was going on.
>> I managed to decrease the buildup of memory, but I haven't been able
>> to stabilize it completely. It might be a 'non-proper' programming
>> thing after all. The script is becoming a bit cluttered anyway. I am
>> going to convert from jython to pure java and tidy it up in the
>> process. Hopefully that will solve it.
>>
>> Thanks for your support!
>>
>> Regards,
>> Jaap Kokorian
>>
>> 2011/1/3 Michael Doube <[hidden email]>:
>>>> when you click on the status bar, ImageJ does
>>>> garbage collection which works great.  In long macros with batch
>>>> processing I have the macro call the garbage collection plugin which
>>>> you can get on the ImageJ website.  You need to call it a few times.
>>>
>>> For sure, if that works then ignore my suggestion, which is much more
>>> complicated. If you find that your plugin is resistant to manual garbage
>>> collection, you'll need to probe deeper.
>>>
>>> Michael
>>>
>>
>
>