TrakEM Mipmap regeneration slow

[Ashwin]

Hi,

I am attempting to regenerate mipmaps for a large EM data-set and seem to be running into problems with the mipmap generation.

Once the images are imported into TrakEM2 (with enable mipmap turned off), I begin aligning the sections (with enable mipmap turned on). In the log window, I notice that TrakEM2 has submitted to generate mipmaps for all images that its trying to extract features form.

Here I can follow the mipmap regeneration, the problem that I notice is that the mipmap creation and saving times increase progressively, to the point that it takes ~20 min to regenerate mipmaps from 1 tile.

I use the following strategy to regenerate mipmaps;

Strategy A: your data consists of large images (over 4000x4000). Right-click on the TrakEM2 display and choose "Project - Properties...", and set the mipmap threads to 1. Now, mipmaps will be regenerated for one single image at a time, using 12 threads (given 12 cores) for computing the Gaussians.

my comp specs; 24 processors, 140G memory.
here is how I call Fiji;  ./ImageJ-linux64 -Xms55g -Xmx55g -Xincgc -XX:MaxPermSize=256m -XX:PermSize=256m
-XX:NewRatio=5 -XX:CMSTriggerRatio=50 -XX:+UseCompressedOops --

I am not sure if I am doing this right or not, but I would like to be able to do this faster.

Any help or suggestions are welcome.

Thanks,
Ashwin

[Albert Cardona-2]

2013/7/11 Ashwin <[hidden email]>

> Hi,
>
> I am attempting to regenerate mipmaps for a large EM data-set and seem to
> be
> running into problems with the mipmap generation.
>
> Once the images are imported into TrakEM2 (with enable mipmap turned off),
> I
> begin aligning the sections (with enable mipmap turned on). In the log
> window, I notice that TrakEM2 has submitted to generate mipmaps for all
> images that its trying to extract features form.
>
> Here I can follow the mipmap regeneration, the problem that I notice is
> that
> the mipmap creation and saving times increase progressively, to the point
> that it takes ~20 min to regenerate mipmaps from 1 tile.
>
> I use the following strategy to regenerate mipmaps;
>
> Strategy A: your data consists of large images (over 4000x4000).
> Right-click
> on the TrakEM2 display and choose "Project - Properties...", and set the
> mipmap threads to 1. Now, mipmaps will be regenerated for one single image
> at a time, using 12 threads (given 12 cores) for computing the Gaussians.
>
> my comp specs; 24 processors, 140G memory.
> here is how I call Fiji;  ./ImageJ-linux64 -Xms55g -Xmx55g -Xincgc
> -XX:MaxPermSize=256m -XX:PermSize=256m
> -XX:NewRatio=5 -XX:CMSTriggerRatio=50 -XX:+UseCompressedOops --
>
> I am not sure if I am doing this right or not, but I would like to be able
> to do this faster.
>
> Any help or suggestions are welcome.
>
> Thanks,
> Ashwin
>
>

... [show rest of quote]

Ashwin,

1. what are the type and dimensions of your images?

2. CompressedOops won't work when allocating over 31 GB of heap.

3. What operating system?

Albert


--
http://albert.rierol.net
http://www.ini.uzh.ch/~acardona/

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Ashwin]

Ashwin,

1. what are the type and dimensions of your images?

The images are all 8bit .tiff and they are 8kx8K.  Each layer had 16x 8K images and there are 1300 layers in the stack.

2. CompressedOops won't work when allocating over 31 GB of heap.

OK. So how do I call Fiji if I want to assign more heap space. Also is it correct to assume larger heap space equals faster processing?

3. What operating system?

running in ubuntu 12.10

Albert

[Albert Cardona-2]

2013/7/11 Ashwin <[hidden email]>

> Ashwin,
>
> 1. what are the type and dimensions of your images?
>
> The images are all 8bit .tiff and they are 8kx8K.  Each layer had 16x 8K
> images and there are 1300 layers in the stack.
>


Considering the capabilities of your computer, consider processing multiple
tiles at a time. Every tile needs about 10x its size in RAM, e.g. ~700 MB
for your tiles. With a heap of 55 GB, you should be able to process a
couple dozen easily. Use then a single thread for the Gaussian downsampling.



>
> 2. CompressedOops won't work when allocating over 31 GB of heap.
>
> OK. So how do I call Fiji if I want to assign more heap space. Also is it
> correct to assume larger heap space equals faster processing?
>


Larger heap results in larger garbage collection in general.


Overall, I have no idea why the process would stall to the point that it
takes 20 minutes to generate mipmaps for a single tile. If garbage
collection is the culprit, you will see it in the terminal if you add the
-verbose:gc parameter to the launcher command.

Albert

>
> 3. What operating system?
>
> running in ubuntu 12.10
>
> Albert
>
>
>
> --
> View this message in context:
> http://imagej.1557.x6.nabble.com/TrakEM-Mipmap-regeneration-slow-tp5003921p5003923.html
> Sent from the ImageJ mailing list archive at Nabble.com.
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

... [show rest of quote]

--
http://albert.rierol.net
http://www.ini.uzh.ch/~acardona/

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Ashwin]

OK.  If I understand correctly, I call the launcher as so;

./ImageJ-linux64 -Xms55g -Xmx55g -Xincgc -XX:MaxPermSize=256m -XX:PermSize=256m
-XX:NewRatio=5 -XX:CMSTriggerRatio=50 -verbose:gc --

And in project>properties> set mipmap threads to 1

in Fiji>Edit>Options>Memory&Threads; threads =24

correct? I shall give this a try.


Thanks for the help.

Ashwin

[Stephan Saalfeld]

Hi Ashwin,

please change, under Project>Project Properties, Image resizing mode to

Area downsampling

this avoids Gaussian downsampling, saves memory and is a lot faster
while still appropriate in quality.  If you can spare some diskspace,
change mipmap format to

rag

Mipmaps will now generate ~100x faster, including disk access this can
still make up to 10x speedup, memory demands are generally lower.

The other behavior I don't get but I have noticed similar hickups
before.  Try this Beanshell script to regenerate the mipmaps:

https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/updatemipmaps.bsh

Change line 39 to 0 to start from first layer.  Consider to update some
ranges of layers and save in between when dealing with large projects.

Best,
Stephan



On Thu, 2013-07-11 at 09:35 -0700, Ashwin wrote:

> Hi,
>
> I am attempting to regenerate mipmaps for a large EM data-set and seem to be
> running into problems with the mipmap generation.
>
> Once the images are imported into TrakEM2 (with enable mipmap turned off), I
> begin aligning the sections (with enable mipmap turned on). In the log
> window, I notice that TrakEM2 has submitted to generate mipmaps for all
> images that its trying to extract features form.
>
> Here I can follow the mipmap regeneration, the problem that I notice is that
> the mipmap creation and saving times increase progressively, to the point
> that it takes ~20 min to regenerate mipmaps from 1 tile.
>
> I use the following strategy to regenerate mipmaps;
>
> Strategy A: your data consists of large images (over 4000x4000). Right-click
> on the TrakEM2 display and choose "Project - Properties...", and set the
> mipmap threads to 1. Now, mipmaps will be regenerated for one single image
> at a time, using 12 threads (given 12 cores) for computing the Gaussians.
>
> my comp specs; 24 processors, 140G memory.
> here is how I call Fiji;  ./ImageJ-linux64 -Xms55g -Xmx55g -Xincgc
> -XX:MaxPermSize=256m -XX:PermSize=256m
> -XX:NewRatio=5 -XX:CMSTriggerRatio=50 -XX:+UseCompressedOops --
>
> I am not sure if I am doing this right or not, but I would like to be able
> to do this faster.
>
> Any help or suggestions are welcome.
>
> Thanks,
> Ashwin
>
>
>
>
>
> --
> View this message in context: http://imagej.1557.x6.nabble.com/TrakEM-Mipmap-regeneration-slow-tp5003921.html
> Sent from the ImageJ mailing list archive at Nabble.com.
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

... [show rest of quote]

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Albert Cardona-2]

2013/7/11 Ashwin <[hidden email]>

> OK.  If I understand correctly, I call the launcher as so;
>
> ./ImageJ-linux64 -Xms55g -Xmx55g -Xincgc -XX:MaxPermSize=256m
> -XX:PermSize=256m
> -XX:NewRatio=5 -XX:CMSTriggerRatio=50 -verbose:gc --
>
> And in project>properties> set mipmap threads to 1
>
> in Fiji>Edit>Options>Memory&Threads; threads =24
>
> correct? I shall give this a try.
>

Threads: the opposite. 24 mipmap threads, and 1 for
Fiji>Edit>Options>Memory&Threads.

I second Saalfeld's comment: use Area Downsampling and .rag format for
mipmaps.

Albert

--
http://albert.rierol.net
http://www.ini.uzh.ch/~acardona/

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Ashwin]

Hi Stephan,

please change, under Project>Project Properties, Image resizing mode to

Area downsampling

This is the default. I have not changed this. Also, I save the mipmaps as .rag (also the default)

I shall give the script a try.

Thanks,
Ashwin

[Stephan Saalfeld]

Ok.  I was misdirected by that I thought you saw Gaussian convolution
happening during mipmap generation.  That would happen only of the
resizing mode was Gaussian downsampling.

To clarify:  With are downsampling, Gaussian convolution happens not
during mipmap generation but only during feature extraction.  That is,
you should limit the number of Threads for feature extraction to a low
number (e.g. 4) to avoid excessive multi-threading beyond your
CPU-count.  One of the most expensive operations during feature
extraction is Gaussian convolution with large kernels and we have
all-CPUs multi-threaded this some time ago in ImageJ's core.

Please let me know how it goes.

Best,
Stephan



On Thu, 2013-07-11 at 13:14 -0700, Ashwin wrote:

> Hi Stephan,
>
> please change, under Project>Project Properties, Image resizing mode to
>
> Area downsampling
>
> This is the default. I have not changed this. Also, I save the mipmaps as
> .rag (also the default)
>
> I shall give the script a try.
>
> Thanks,
> Ashwin
>
>
>
>
>
> --
> View this message in context: http://imagej.1557.x6.nabble.com/TrakEM-Mipmap-regeneration-slow-tp5003921p5003930.html
> Sent from the ImageJ mailing list archive at Nabble.com.
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

... [show rest of quote]

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Ashwin]

Thanks for the inputs Albert and Stephan. The problem it seem was at my end and not TrakEM2. The  write-back cache was disabled, this made random writes very slow and that was what was being reflected while regenerating mipmaps. After enabling the speed up is certainly noticeable.

Thanks,
Ashwin