Using more (>4GB) memory is possible with linux AMD64

[Jan de Sonneville]

Hi People,

Once in a while I read about memory issues using java applications. I
tested the following runtime conditions on an Ubuntu AMD64 installation on
both (dual) opteron and athlon processors:

java -jar -XmxNg -XX:+AggressiveHeap ij.jar
where you can replace N with the amount of gigabytes of ram you'd like to use

If you have multiple processors you can also add:
-XX:ParallelThreads=N --XX:+UseParallelOldGC
where N has to be replaced by the number of processors cores you'd like to
use (see dmesg)

I've tested this using Sun jre version 6 (installable through package
manager for ubuntu 7.04, be sure that you use the sun version of java and
not the standard GPL one)

The sun website shows us that there are limits on the amount of memory you
can use, and that the memory limit of every operating system (except
solaris) is 2,3,4 GB. I've no problems increasing the limit over 4 GB
using ubuntu AMD64. I've tested up to 8 GB without problems (except
swapping time as I don't have 8 GB of ram..)
So if you have more ram and you'd like to use it...use linux AMD64!

All the best, and Wayne thanks, keep up the good work!
Jan de Sonneville
([hidden email])

[Michael Weber-4]

Hi Jan,

using more memory also works fine with Windows x64. So far I was using
7,x GB out of 8 GB system memory. I can check what's possible on our 16
GB system, but it's probably working as well.

However, the multiple processor tweak sounds interesting. As far as I
know, Java doesn't support multiple cores, or does it already? Is it
supported by ImageJ plugins as well? So one could have things eight
times faster.

cheers,
Michael


Jan de Sonneville wrote:

> Hi People,
>
> Once in a while I read about memory issues using java applications. I
> tested the following runtime conditions on an Ubuntu AMD64 installation on
> both (dual) opteron and athlon processors:
>
> java -jar -XmxNg -XX:+AggressiveHeap ij.jar
> where you can replace N with the amount of gigabytes of ram you'd like to use
>
> If you have multiple processors you can also add:
> -XX:ParallelThreads=N --XX:+UseParallelOldGC
> where N has to be replaced by the number of processors cores you'd like to
> use (see dmesg)
>
> I've tested this using Sun jre version 6 (installable through package
> manager for ubuntu 7.04, be sure that you use the sun version of java and
> not the standard GPL one)
>
> The sun website shows us that there are limits on the amount of memory you
> can use, and that the memory limit of every operating system (except
> solaris) is 2,3,4 GB. I've no problems increasing the limit over 4 GB
> using ubuntu AMD64. I've tested up to 8 GB without problems (except
> swapping time as I don't have 8 GB of ram..)
> So if you have more ram and you'd like to use it...use linux AMD64!
>
> All the best, and Wayne thanks, keep up the good work!
> Jan de Sonneville
> ([hidden email])

[Jeff Brandenburg]

On Aug 7, 2007, at 11:31 AM, Michael Weber wrote:

> Hi Jan,
>
> using more memory also works fine with Windows x64. So far I was  
> using 7,x GB out of 8 GB system memory. I can check what's possible  
> on our 16 GB system, but it's probably working as well.
>
> However, the multiple processor tweak sounds interesting. As far as  
> I know, Java doesn't support multiple cores, or does it already? Is  
> it supported by ImageJ plugins as well? So one could have things  
> eight times faster.

Java does support multiple cores, at least on the OS X systems I  
use.  I've written code using Java's standard thread model that keeps  
four cores 100% busy.

To get a speedup in ImageJ, you would need code that explicitly  
divides its work among multiple threads.  I and others have done some  
work in that direction -- I hacked up a version of Slicer that ran on  
multiple threads, and I think someone else made similar modifications  
to StackProcessor -- but I don't think anything is available for  
general release yet.  (My code made invasive changes to ImageStack,  
and was based on ImageJ 1.29, so it's not especially useful today.)
--
        -jeffB (Jeff Brandenburg, Duke Center for In-Vivo Microscopy)

[Michael Schmid]

Hi ImageJers,

Java does support multiple cores, I agree. In my experience, with
Windows XP and CoreDuo processors, load sharing between cores
works well without any additional action such as a modification
of the Java settings.

Since ImageJ 1.38u, plugin-filters can take advantage of multiple
cores by specifying the flag PARALLELIZE_STACKS. With this flag,
when processing stacks, processing will be split into separate
threads, each of them doing part of the slices. You can specify
the number of threads in Edit>Options>Memory & Threads. This field
defaults to the number of cores.

Some internal filters such as Gaussian Blur and the Rank Filters
(minimum, maximum, mean, median...) handle stacks in parallel
already.

Restrictions:

- Parallel processing needs more memory, so you may have to set
the number of threads to a lower value than the number of cores
when processing large images.

- Of course, you must not use PARALLELIZE_STACKS for plugin-filters
where the sequence of the slices is important for processing
(if the result of slice number i is needed for slice i+1)

- If you need to know the slice number, the old method of simply
incrementing the slice number upon each call of the filter won't
work any more. Instead, you can get the slice number by a trick
if you have an ExtendedPlugInFilter - see the interface at
   http://rsb.info.nih.gov/ij/developer/source/ij/plugin/filter/ 
ExtendedPlugInFilter.java.html
There you get a reference to the PlugInFilterRunner running the
filter in the showDialog method, then you can use the
   public int getSliceNumber()
method of the PlugInFilterRunner.


Michael
________________________________________________________________


On 8 Aug 2007, at 14:44, Jeff Brandenburg wrote:

> On Aug 7, 2007, at 11:31 AM, Michael Weber wrote:
>
>> Hi Jan,
>>
>> using more memory also works fine with Windows x64. So far I was  
>> using 7,x GB out of 8 GB system memory. I can check what's  
>> possible on our 16 GB system, but it's probably working as well.
>>
>> However, the multiple processor tweak sounds interesting. As far  
>> as I know, Java doesn't support multiple cores, or does it  
>> already? Is it supported by ImageJ plugins as well? So one could  
>> have things eight times faster.
>
> Java does support multiple cores, at least on the OS X systems I  
> use.  I've written code using Java's standard thread model that  
> keeps four cores 100% busy.
>
> To get a speedup in ImageJ, you would need code that explicitly  
> divides its work among multiple threads.  I and others have done  
> some work in that direction -- I hacked up a version of Slicer that  
> ran on multiple threads, and I think someone else made similar  
> modifications to StackProcessor -- but I don't think anything is  
> available for general release yet.  (My code made invasive changes  
> to ImageStack, and was based on ImageJ 1.29, so it's not especially  
> useful today.)
> --
> -jeffB (Jeff Brandenburg, Duke Center for In-Vivo Microscopy)