huge memory leaks in Vista 64 with LOCI Bioformats

[John Alexander-7]

Hi,

I'm running the latest ImageJ with the latest LOCI Bioformats - and I've
noticed huge memory leaks.

If I open up a GB of images (slidebook format, imported with the LOCI
Bio-formats reader), process some, then close most, imageJ will report
only 100 MB of memory usage, but the Windows Task Manager reports the
JAVA machine is using over 2 GB!

I'm not totally sure this is related to the LOCI Bioformats plugin
either.  I believe that if I open a lot of TIFF images, process them,
then close most of them, the JAVA Machine again reports high memory
usage even though ImageJ reports very little.

Can somebody assist me in figuring out how to rectify this?  The problem
is that after a bit of image processing, my machine begins to bog down
due to increased memory swapping


John

[ctrueden]

Hi John,

This problem was mentioned a few months ago, too:

http://n2.nabble.com/Java-memory-not-release-when-image-stacks-closed-td637015.html

So it is not specific to Bio-Formats. It is not specific to ImageJ either,
but actually a property of Java itself.

For example, here is a thread on Experts Exchange describing someone who had
this problem with Tomcat server:

http://www.experts-exchange.com/Web/Application_Servers/Q_20899033.html

And someone replied in one of the answers: "THe JVM does not release
memorythe OS allocated to it - it handles its own
memory and the assumption is that getting memory from the OS is time
consuming, so it won't release it based on the GC behavior (this would
dramatically decrease performance)."

However, I am not sure whether this statement is accurate -- there seems to
be a lot of misinformation about Java memory behavior online, and it is hard
to filter out the bogus information.

I wrote a tiny Java application for testing, and the behavior does seem to
depend on the operating system:

On Mac OS X 10.4, the "Real" memory entry for a Java application never
decreases, despite the resources being freed from within Java and garbage
collected.

On Windows XP, the "Mem Usage" column in Task Manager does decrease when
memory is freed. There is also some weirdness with minimized applications
reportedly consuming far fewer resources -- maybe a bug in XP's Task
Manager, or maybe some property of Windows XP's memory management.

On Ubuntu Linux 10.4, the Memory column in System Monitor decreases properly
when memory is freed.

I do not have a Windows Vista machine on hand to test, but if you are
feeling adventurous you can try yourself using my program:

https://skyking.microscopy.wisc.edu/svn/java/trunk/utils/TestMem.java

Anyone else know more about this phenomenon?

-Curtis

On Mon, Dec 8, 2008 at 12:12 PM, John Alexander <[hidden email]>wrote:

> Hi,
>
> I'm running the latest ImageJ with the latest LOCI Bioformats - and I've
> noticed huge memory leaks.
>
> If I open up a GB of images (slidebook format, imported with the LOCI
> Bio-formats reader), process some, then close most, imageJ will report
> only 100 MB of memory usage, but the Windows Task Manager reports the
> JAVA machine is using over 2 GB!
>
> I'm not totally sure this is related to the LOCI Bioformats plugin
> either.  I believe that if I open a lot of TIFF images, process them,
> then close most of them, the JAVA Machine again reports high memory
> usage even though ImageJ reports very little.
>
> Can somebody assist me in figuring out how to rectify this?  The problem
> is that after a bit of image processing, my machine begins to bog down
> due to increased memory swapping
>
>
> John
>

[ctrueden]

Hi John,

I'm CCing the list as this stuff may be of interest to others.

I should have been more clear about the "malloc" function: it allocates 32
MB of memory, but only once. It won't stack up. There is a block of data the
program has, that is either null, or 32 MB in size. If you click malloc a
second time, it will re-allocate the 32MB, and the Java garbage collector
will eat the old 32 MB piece of data since it has been overwritten.
Nonetheless, you will still see a temporary spike in memory usage before the
garbage collector kicks in -- and if Java is set to allow only a 64 MB
maximum heap, which it is by default on many systems, it will actually throw
an OutOfMemoryException because it allocates the new block before releasing
the old block.

Clicking "free" does two things: 1) nulls out the data block; and 2) calls
the garbage collector. Sometimes calling the garbage collector once is not
enough to fully free everything in Java, but you can get maximal results by
mashing the free button a few times.

So, in summary, the reason you are seeing the behavior you describe is that
when you mash "malloc" really fast, it allocates a bunch of 32 MB blocks
before the garbage collector gets around to releasing the old ones. But if
you mash "free" a few times it should drop (nearly) all the way back down.
And when Java starts getting close to its max, it forces a garbage
collection, since it needs the memory at that point -- which is why you get
the "flip over" you noticed.

The question is, what does the Windows Task Manager say about all this? Does
the Mem Usage go back down after clicking "free" a few times? Or does it
stay at its peak?

Lastly, here is an interesting article about the differences between Vista's
approach to memory, versus XP and other older Windows versions:

http://www.codinghorror.com/blog/archives/000688.html

I am not sure how accurate or up to date it is, but it does suggest that
perhaps Vista not releasing memory is not as bad as it may seem -- perhaps
Vista *would* release the memory once you have other applications actively
demanding it. Hard to say without doing some more testing.

-Curtis

On Mon, Dec 8, 2008 at 2:08 PM, John Alexander <[hidden email]>wrote:

> Thanks,
>
> Can you help me interpret some of this?
>
> Running your program blindly ... (so trusting, I know) ...
>
> At start up, I get:
> Total: 62848 KB
> Max: 932096 KB
> Free: 56721 KB
> Used: 6470 KB
>
> I then click malloc 10 times ...
> Total: 358336 KB
> Max: 932096 KB
> Free: 21730 KB
> Used: 336950 KB
>
> I click free once:
> Total: 358336 KB
> Max: 932096 KB
> Free: 355780 KB
> Used: 2560 KB
>
> I then click malloc 10 times, and get the same as above.
>
> Then 10 MORE times and I get
> Total: 686656 KB
> Max: 932096 KB
> Free: 27030 KB
> Used: 659924 KB
>
> Followed by free:
> Total: 686656 KB
> Max: 932096 KB
> Free: 684424 KB
> Used: 2513 KB
>
> interesting ... when I keep clicking malloc ... it never
> reaches the Max ... when Total gets close to Max ... it flips
> over.
>
> For example, closing then starting the program all over ...
> and clicking malloc 21 times gets me:
> Total: 718208 KB
> Max: 932096 KB
> Free: 25734 KB
> Used: 692167 KB
>
> and on the 22nd click of malloc, I get:
> Total: 106112 KB
> Max: 932096 KB
> Free: 38170 KB
> Used: 68280 KB
>
> My machine specs are vista 64-bit running Java 1.6 64-bit with
> 4 GB system RAM.
>
> John
>
>
> John Alexander, Ph.D.
> Post-doctoral Fellow
> William Green Lab
> University of Chicago
> 947 E. 58th St.
> Abott Hall 402
> Chicago, IL 60637
> off: [phone]
> fax: [phone]
> [hidden email]
>