trakem2: Mipmap generation slowing down over time

[Moritz]

Dear all,

I have a problem generating mipmaps: The generation is fast (about an image in 3 seconds) when I freshly open the project and then gets much slower (orders of magnitudes) within minutes.

I am working on Win7 16 cores (+16 hyperthreading cores) Xeon system with 128GB of RAM with a dataset of about 450 GB (16-bit dm3 files of 4x4k 12,000+ images) on spinning HD.

Stitching was done with deactivated mipmaps and worked like a charm.

I followed all recommendations which I found in the old thread of the similar topic:
http://imagej.1557.x6.nabble.com/TrakEM-Mipmap-regeneration-slow-td5003921.html

My cfg file is contained of:
"./fiji -Xms70g -Xmx70g -Xincgc -XX:MaxPermSize=256m -XX:PermSize=256m -XX:NewRatio=5 -XX:CMSTriggerRatio=50 --"
I tried the different threading modes. However the problem persisted.

I also tried using the bsh script mentioned in the old thread. However it was putting out the mipmap of a single image in about  half an hour...

Help would be greatly appreciated!

Best regards, Moritz


Moritz A. Kirschmann
Friedrich Miescher Institute
for Biomedical Research
Maulbeerstrasse 66
P.O. Box 3775
4002 Basel
Switzerland


--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Saalfeld, Stephan]

Hi Moritz,

> I followed all recommendations which I found in the old thread of the
> similar topic:
> http://imagej.1557.x6.nabble.com/TrakEM-Mipmap-regeneration-slow-td5003921.html

the end of the old thread mentions write-back cache being disabled as
the cause of the problem.  Have you verified this?

> My cfg file is contained of:
> "./fiji -Xms70g -Xmx70g -Xincgc -XX:MaxPermSize=256m -XX:PermSize=256m
> -XX:NewRatio=5 -XX:CMSTriggerRatio=50 --"
> I tried the different threading modes. However the problem persisted.
>
> I also tried using the bsh script mentioned in the old thread. However
> it was putting out the mipmap of a single image in about  half an
> hour...

Do you have any filters preprocessing the images?  Can you please try
toi limit the number of parallel threads generating mipmaps?  In line 44

https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/updatemipmaps.bsh#L44

write

for ( int i = 0; i < 1; ++i )

and please report back the speed for generating one image.  I fear that
the generation of a single mipmap may be already highly multi-threaded
for some reasons and that by processing many mipmaps in parallel, you
generate an excessive number of threads.

I am sorry that TrakEM2 has no proper mechanism to limit the number of
threads used for particular operations.  It's due to its historically
grown design and to the fact that it uses all kinds of independent tools
that all do not care about this issue.

Let me know how it goes.

Best,
Stephan

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Moritz]

Dear Stephan,

thank you very much for your fast reply!

1) Yes, write-back cache is activated.

2) I have no filters (preprocessors) active but I set the 'Set min and max layerwise to fixed values' after alignment.

3) I let the script run with only one thread. (Does the number of threads in FIJIs preferences ('options>memory & threads') have an influence on the script? If yes this is a problem:
I have a problem when I change the number of threads in 'options>memory & threads'. I can change it, but upon restart of FIJI the default of 32 is set.
Do I need to restart it or does this change have effect immediately?
To fix this issue I tried the following
A) I redownloaded Fiji.
B) I copied FIJI to a different path (before c:\utilities\FIJI now c:\users\myname\FIJI )
C) I disactivated write protection of the FIJI folder and subfolders
D) I ran FIJI as admin.
:( Still the changes are not saved and the default of 32 reappears.
What can I do to fix this?)

Without restart of FIJI after changing the number of threads the script calculated one mipmap in 10s. However after a few hundred it got much slower.

What else can I do to pinpoint the bottleneck?

Best regards and again many thanks for your help!
 Moritz

-----Original Message-----
From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of Stephan Saalfeld
Sent: Thursday, March 20, 2014 3:33 PM
To: [hidden email]
Subject: Re: trakem2: Mipmap generation slowing down over time

Hi Moritz,

> I followed all recommendations which I found in the old thread of the
> similar topic:
> http://imagej.1557.x6.nabble.com/TrakEM-Mipmap-regeneration-slow-td5003921.html

1) the end of the old thread mentions write-back cache being disabled as
the cause of the problem.  Have you verified this?


> My cfg file is contained of:
> "./fiji -Xms70g -Xmx70g -Xincgc -XX:MaxPermSize=256m -XX:PermSize=256m
> -XX:NewRatio=5 -XX:CMSTriggerRatio=50 --"
> I tried the different threading modes. However the problem persisted.
>
> I also tried using the bsh script mentioned in the old thread. However
> it was putting out the mipmap of a single image in about  half an
> hour...

2) Do you have any filters preprocessing the images?

3) Can you please try to limit the number of parallel threads generating mipmaps?  In line 44

https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/updatemipmaps.bsh#L44

write

for ( int i = 0; i < 1; ++i )

and please report back the speed for generating one image.  I fear that
the generation of a single mipmap may be already highly multi-threaded
for some reasons and that by processing many mipmaps in parallel, you
generate an excessive number of threads.

I am sorry that TrakEM2 has no proper mechanism to limit the number of
threads used for particular operations.  It's due to its historically
grown design and to the fact that it uses all kinds of independent tools
that all do not care about this issue.

Let me know how it goes.

Best,
Stephan

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Saalfeld, Stephan]

Hi Moritz,

unfortunately, it can still be many things.  Could you please repeat the
experiment with slowing down mipmap generation with jvisualvm open and
connected to the Fiji JVM?  Do you see an excessive number of threads
generated or do you see memory on the edge and garbage collection
wasting a lot of time blocking other threads?  Jvisualvm should be
located somewhere in the JRE distributed with Fiji or you download it
with another JDK distribution.

What is your file system?  NTFS?  Have you tried having your TrakEM2
project directory on a FAT32 partition?  My information is probably
outdated but NTFS supports extended permissions to files and therefore
has to run through an ACL per access which may slow down access in
directories with many small files such as in the TrakEM2 project
directory.  This is only wild guessing though.  Can you mount a Linux
file system (ext4) and try it there?

Hope we're identifying the issue soon.

Best,
Stephan

On Fri, 2014-03-21 at 14:24 +0000, Kirschmann, Moritz wrote:

> Dear Stephan,
>
> thank you very much for your fast reply!
>
> 1) Yes, write-back cache is activated.
>
> 2) I have no filters (preprocessors) active but I set the 'Set min and max layerwise to fixed values' after alignment.
>
> 3) I let the script run with only one thread. (Does the number of threads in FIJIs preferences ('options>memory & threads') have an influence on the script? If yes this is a problem:
> I have a problem when I change the number of threads in 'options>memory & threads'. I can change it, but upon restart of FIJI the default of 32 is set.
> Do I need to restart it or does this change have effect immediately?
> To fix this issue I tried the following
> A) I redownloaded Fiji.
> B) I copied FIJI to a different path (before c:\utilities\FIJI now c:\users\myname\FIJI )
> C) I disactivated write protection of the FIJI folder and subfolders
> D) I ran FIJI as admin.
> :( Still the changes are not saved and the default of 32 reappears.
> What can I do to fix this?)
>
> Without restart of FIJI after changing the number of threads the script calculated one mipmap in 10s. However after a few hundred it got much slower.
>
> What else can I do to pinpoint the bottleneck?
>
> Best regards and again many thanks for your help!
>  Moritz
>
> -----Original Message-----
> From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of Stephan Saalfeld
> Sent: Thursday, March 20, 2014 3:33 PM
> To: [hidden email]
> Subject: Re: trakem2: Mipmap generation slowing down over time
>
> Hi Moritz,
>
> > I followed all recommendations which I found in the old thread of the
> > similar topic:
> > http://imagej.1557.x6.nabble.com/TrakEM-Mipmap-regeneration-slow-td5003921.html
>
> 1) the end of the old thread mentions write-back cache being disabled as
> the cause of the problem.  Have you verified this?
>
>
> > My cfg file is contained of:
> > "./fiji -Xms70g -Xmx70g -Xincgc -XX:MaxPermSize=256m -XX:PermSize=256m
> > -XX:NewRatio=5 -XX:CMSTriggerRatio=50 --"
> > I tried the different threading modes. However the problem persisted.
> >
> > I also tried using the bsh script mentioned in the old thread. However
> > it was putting out the mipmap of a single image in about  half an
> > hour...
>
> 2) Do you have any filters preprocessing the images?
>
> 3) Can you please try to limit the number of parallel threads generating mipmaps?  In line 44
>
> https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/updatemipmaps.bsh#L44
>
> write
>
> for ( int i = 0; i < 1; ++i )
>
> and please report back the speed for generating one image.  I fear that
> the generation of a single mipmap may be already highly multi-threaded
> for some reasons and that by processing many mipmaps in parallel, you
> generate an excessive number of threads.
>
> I am sorry that TrakEM2 has no proper mechanism to limit the number of
> threads used for particular operations.  It's due to its historically
> grown design and to the fact that it uses all kinds of independent tools
> that all do not care about this issue.
>
> Let me know how it goes.
>
> Best,
> Stephan
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--

Stephan Saalfeld, Ph.D.
Group Leader
Janelia Farm Research Campus
19700 Helix Drive | Ashburn, VA 20147
Phone: [phone] | Fax: [phone]
[hidden email]

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html