ImageJ loading of BigTiff files VERY slow

[Bill N. Whitney - whitnewn]

I am running ImageJ version 1.49t (from Fiji) on Windows 7.

I am working with a microscope that creates image stacks and stores them as TIFF files.
Sometimes, the image stack files can get bigger than 4GB, so we can also create BigTiff files.

I can load a REGULAR Tiff file into ImageJ in at most 30 seconds. A file right at the 4GB limit takes 20-30 seconds to load. Smaller files are faster.

Even a small BigTiff format file ( 64 MB) takes much longer than this, roughly 120 - 180 seconds.

It takes a long time for ImageJ to START processing the individual images (2-3 minutes), AND it takes a long time for ImageJ to process each individual image (~30 sec).

For example, loading a 1.8GB BigTiff file takes about 4-5 hours to load. The same file, stored as regular Tiff, takes about 12 seconds.
I have a 5GB BigTiff file, and after letting it run for 3 hours, it had only processed 85 of 500 images, so I killed the process.

Any insights into what is going on here? I can provide sample files if that would be helpful .....

Thanks,
Bill

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Laurentius Henricus Holtzer]

Hi,

I have a similar problem, which might be related. I am using the BioFormats plugin to open large data sets. Those data sets consist of multiple regular tiff-files in which each tiff file contains an image stack for one time point. Each individual file has a size in the range of 50-300 MB.

When I use ‘Group files with similar names’ to open all files at once, this is very slow if the total size is more than a few GB. Like Bill I had to kill FIJI since the process was running for hours and FIJI was not responsive anymore. The thread killer didn’t work either.

The only solution I found is to downgrade to version 1.49s.

Cheers,
Laurent

> On Jun 5, 2015, at 11:37 PM, Bill N. Whitney - whitnewn <[hidden email]> wrote:
>
> I am running ImageJ version 1.49t (from Fiji) on Windows 7.
>
> I am working with a microscope that creates image stacks and stores them as TIFF files.
> Sometimes, the image stack files can get bigger than 4GB, so we can also create BigTiff files.
>
> I can load a REGULAR Tiff file into ImageJ in at most 30 seconds. A file right at the 4GB limit takes 20-30 seconds to load. Smaller files are faster.
>
> Even a small BigTiff format file ( 64 MB) takes much longer than this, roughly 120 - 180 seconds.
>
> It takes a long time for ImageJ to START processing the individual images (2-3 minutes), AND it takes a long time for ImageJ to process each individual image (~30 sec).
>
> For example, loading a 1.8GB BigTiff file takes about 4-5 hours to load. The same file, stored as regular Tiff, takes about 12 seconds.
> I have a 5GB BigTiff file, and after letting it run for 3 hours, it had only processed 85 of 500 images, so I killed the process.
>
> Any insights into what is going on here? I can provide sample files if that would be helpful .....
>
> Thanks,
> Bill
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[ctrueden]

Hi Bill,

Using File > Open will use ImageJ 1.x's built-in TIFF reader when possible,
and fall back to other TIFF readers including the Bio-Formats plugins in
some cases.

Are you testing with File > Import > Bio-Formats in all scenarios? That
will compare apples to apples.

Are your files on a network drive? Or local storage? Currently, performance
can be much worse on a remote mounted file system.

You can also try opening with SCIFIO, using the File > Import > Image...
command, although that code may suffer from performance problems similar to
Bio-Formats at the moment. We have plans to make it much faster [1].

Lastly, see this page for ideas on how to debug why things are taking so
long:
   http://imagej.net/Debugging#Debugging_JVM_hangs

Regards,
Curtis

[1] https://github.com/scifio/scifio/issues/7

On Fri, Jun 5, 2015 at 11:37 PM, Bill N. Whitney - whitnewn <
[hidden email]> wrote:

> I am running ImageJ version 1.49t (from Fiji) on Windows 7.
>
> I am working with a microscope that creates image stacks and stores them
> as TIFF files.
> Sometimes, the image stack files can get bigger than 4GB, so we can also
> create BigTiff files.
>
> I can load a REGULAR Tiff file into ImageJ in at most 30 seconds. A file
> right at the 4GB limit takes 20-30 seconds to load. Smaller files are
> faster.
>
> Even a small BigTiff format file ( 64 MB) takes much longer than this,
> roughly 120 - 180 seconds.
>
> It takes a long time for ImageJ to START processing the individual images
> (2-3 minutes), AND it takes a long time for ImageJ to process each
> individual image (~30 sec).
>
> For example, loading a 1.8GB BigTiff file takes about 4-5 hours to load.
> The same file, stored as regular Tiff, takes about 12 seconds.
> I have a 5GB BigTiff file, and after letting it run for 3 hours, it had
> only processed 85 of 500 images, so I killed the process.
>
> Any insights into what is going on here? I can provide sample files if
> that would be helpful .....
>
> Thanks,
> Bill
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Rasband, Wayne (NIH/NIMH) [E]]

> On Jun 8, 2015, at 5:15 AM, Laurentius Henricus Holtzer <[hidden email]> wrote:
>
> Hi,
>
> I have a similar problem, which might be related. I am using the BioFormats plugin to open large data sets. Those data sets consist of multiple regular tiff-files in which each tiff file contains an image stack for one time point. Each individual file has a size in the range of 50-300 MB.
>
> When I use ‘Group files with similar names’ to open all files at once, this is very slow if the total size is more than a few GB. Like Bill I had to kill FIJI since the process was running for hours and FIJI was not responsive anymore. The thread killer didn’t work either.
>
> The only solution I found is to downgrade to version 1.49s.

This sounds like a Bio-Formats issue so downgrading to ImageJ 1.49s should not make a difference. What version of Bio-Formats are you using? The current version is 5.1.2. Use the Help>About Plugins>Bio-Formats Plugins command to get the version. Bio-Formats issues should be brought up on the ome-users mailing list at

     http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users/

Have you tried the File>Import>Image Sequence command? It should be much faster. Make sure “Use SCIFIO when opening files” is disabled in Edit>Options>ImageJ2.

-wayne

>
> Cheers,
> Laurent
>
>
>
>
>> On Jun 5, 2015, at 11:37 PM, Bill N. Whitney - whitnewn <[hidden email]> wrote:
>>
>> I am running ImageJ version 1.49t (from Fiji) on Windows 7.
>>
>> I am working with a microscope that creates image stacks and stores them as TIFF files.
>> Sometimes, the image stack files can get bigger than 4GB, so we can also create BigTiff files.
>>
>> I can load a REGULAR Tiff file into ImageJ in at most 30 seconds. A file right at the 4GB limit takes 20-30 seconds to load. Smaller files are faster.
>>
>> Even a small BigTiff format file ( 64 MB) takes much longer than this, roughly 120 - 180 seconds.
>>
>> It takes a long time for ImageJ to START processing the individual images (2-3 minutes), AND it takes a long time for ImageJ to process each individual image (~30 sec).
>>
>> For example, loading a 1.8GB BigTiff file takes about 4-5 hours to load. The same file, stored as regular Tiff, takes about 12 seconds.
>> I have a 5GB BigTiff file, and after letting it run for 3 hours, it had only processed 85 of 500 images, so I killed the process.
>>
>> Any insights into what is going on here? I can provide sample files if that would be helpful .....
>>
>> Thanks,
>> Bill

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[ctrueden]

Hi Laurent,

> When I use ‘Group files with similar names’ to open all files at once,
> this is very slow if the total size is more than a few GB.

You can troubleshoot why things are so slow using the instructions here:
    http://imagej.net/Debugging#Debugging_JVM_hangs

> The only solution I found is to downgrade to version 1.49s.

If you see an I/O performance difference between 1.49s and 1.49t, that is
vital (and rather surprising) information. Are you using the "Help > Update
ImageJ..." command to switch between them when testing?

Regards,
Curtis

On Mon, Jun 8, 2015 at 11:15 AM, Laurentius Henricus Holtzer <
[hidden email]> wrote:

> Hi,
>
> I have a similar problem, which might be related. I am using the
> BioFormats plugin to open large data sets. Those data sets consist of
> multiple regular tiff-files in which each tiff file contains an image stack
> for one time point. Each individual file has a size in the range of 50-300
> MB.
>
> When I use ‘Group files with similar names’ to open all files at once,
> this is very slow if the total size is more than a few GB. Like Bill I had
> to kill FIJI since the process was running for hours and FIJI was not
> responsive anymore. The thread killer didn’t work either.
>
> The only solution I found is to downgrade to version 1.49s.
>
> Cheers,
> Laurent

> > On Jun 5, 2015, at 11:37 PM, Bill N. Whitney - whitnewn <
> [hidden email]> wrote:
> >
> > I am running ImageJ version 1.49t (from Fiji) on Windows 7.
> >
> > I am working with a microscope that creates image stacks and stores them
> as TIFF files.
> > Sometimes, the image stack files can get bigger than 4GB, so we can also
> create BigTiff files.
> >
> > I can load a REGULAR Tiff file into ImageJ in at most 30 seconds. A file
> right at the 4GB limit takes 20-30 seconds to load. Smaller files are
> faster.
> >
> > Even a small BigTiff format file ( 64 MB) takes much longer than this,
> roughly 120 - 180 seconds.
> >
> > It takes a long time for ImageJ to START processing the individual
> images (2-3 minutes), AND it takes a long time for ImageJ to process each
> individual image (~30 sec).
> >
> > For example, loading a 1.8GB BigTiff file takes about 4-5 hours to load.
> The same file, stored as regular Tiff, takes about 12 seconds.
> > I have a 5GB BigTiff file, and after letting it run for 3 hours, it had
> only processed 85 of 500 images, so I killed the process.
> >
> > Any insights into what is going on here? I can provide sample files if
> that would be helpful .....
> >
> > Thanks,
> > Bill
> >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Laurentius Henricus Holtzer]

On Jun 8, 2015, at 6:55 PM, Rasband, Wayne (NIH/NIMH) [E] <[hidden email]<mailto:[hidden email]>> wrote:


On Jun 8, 2015, at 5:15 AM, Laurentius Henricus Holtzer <[hidden email]<mailto:[hidden email]>> wrote:

Hi,

I have a similar problem, which might be related. I am using the BioFormats plugin to open large data sets. Those data sets consist of multiple regular tiff-files in which each tiff file contains an image stack for one time point. Each individual file has a size in the range of 50-300 MB.

When I use ‘Group files with similar names’ to open all files at once, this is very slow if the total size is more than a few GB. Like Bill I had to kill FIJI since the process was running for hours and FIJI was not responsive anymore. The thread killer didn’t work either.

The only solution I found is to downgrade to version 1.49s.

This sounds like a Bio-Formats issue so downgrading to ImageJ 1.49s should not make a difference. What version of Bio-Formats are you using? The current version is 5.1.2. Use the Help>About Plugins>Bio-Formats Plugins command to get the version. Bio-Formats issues should be brought up on the ome-users mailing list at

    http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users/

Thank you for your reply. I did some testing and indeed it seems to be a BioFormats issue. I got the FIJI with Imagej 1.49s from the ImageJ Jenkins server, and that version also included Bio-Formats 5.1.1. I upgraded to Bio-Formats 5.1.2, while keeping ImageJ 1.49s and I got the same problem.

After some more testing I also realized that in the end all the images are loaded using Bio-Formats 5.1.2, but that FIJI becomes very slow and unresponsive while it is loading. For example the status bar which counts the opened images freezes and it takes a long time before you can see anything in the window that is supposed to show the data.


Have you tried the File>Import>Image Sequence command? It should be much faster. Make sure “Use SCIFIO when opening files” is disabled in Edit>Options>ImageJ2.


That works indeed much faster, thanks!


Laurent

-wayne



Cheers,
Laurent




On Jun 5, 2015, at 11:37 PM, Bill N. Whitney - whitnewn <[hidden email]<mailto:[hidden email]>> wrote:

I am running ImageJ version 1.49t (from Fiji) on Windows 7.

I am working with a microscope that creates image stacks and stores them as TIFF files.
Sometimes, the image stack files can get bigger than 4GB, so we can also create BigTiff files.

I can load a REGULAR Tiff file into ImageJ in at most 30 seconds. A file right at the 4GB limit takes 20-30 seconds to load. Smaller files are faster.

Even a small BigTiff format file ( 64 MB) takes much longer than this, roughly 120 - 180 seconds.

It takes a long time for ImageJ to START processing the individual images (2-3 minutes), AND it takes a long time for ImageJ to process each individual image (~30 sec).

For example, loading a 1.8GB BigTiff file takes about 4-5 hours to load. The same file, stored as regular Tiff, takes about 12 seconds.
I have a 5GB BigTiff file, and after letting it run for 3 hours, it had only processed 85 of 500 images, so I killed the process.

Any insights into what is going on here? I can provide sample files if that would be helpful .....

Thanks,
Bill



--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
Dr. ir. Laurent Holtzer
Gonzalez-Gaitan lab
Department of Biochemistry
University of Geneva, Sciences II
30, quai Ernest-Ansermet
1211 Geneva 4, Switzerland
email: [hidden email]<mailto:[hidden email]>
tel: [phone]


--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Laurentius Henricus Holtzer]

For some reason the formatting of my previous message disappeared. Therefore I am sending it again. Sorry for the inconvenience.

> On Jun 8, 2015, at 6:55 PM, Rasband, Wayne (NIH/NIMH) [E] <[hidden email]> wrote:
>
>
>> On Jun 8, 2015, at 5:15 AM, Laurentius Henricus Holtzer <[hidden email]> wrote:
>>
>> Hi,
>>
>> I have a similar problem, which might be related. I am using the BioFormats plugin to open large data sets. Those data sets consist of multiple regular tiff-files in which each tiff file contains an image stack for one time point. Each individual file has a size in the range of 50-300 MB.
>>
>> When I use ‘Group files with similar names’ to open all files at once, this is very slow if the total size is more than a few GB. Like Bill I had to kill FIJI since the process was running for hours and FIJI was not responsive anymore. The thread killer didn’t work either.
>>
>> The only solution I found is to downgrade to version 1.49s.
>
> This sounds like a Bio-Formats issue so downgrading to ImageJ 1.49s should not make a difference. What version of Bio-Formats are you using? The current version is 5.1.2. Use the Help>About Plugins>Bio-Formats Plugins command to get the version. Bio-Formats issues should be brought up on the ome-users mailing list at
>
>     http://lists.openmicroscopy.org.uk/mailman/listinfo/ome-users/
>

Thank you for your reply. I did some testing and indeed it seems to be a BioFormats issue. I got the FIJI with Imagej 1.49s from the ImageJ Jenkins server, and that version also included Bio-Formats 5.1.1. I upgraded to Bio-Formats 5.1.2, while keeping ImageJ 1.49s and I got the same problem.

After some more testing I also realized that in the end all the images are loaded using Bio-Formats 5.1.2, but that FIJI becomes very slow and unresponsive while it is loading. For example the status bar which counts the opened images freezes and it takes a long time before you can see anything in the window that is supposed to show the data.


> Have you tried the File>Import>Image Sequence command? It should be much faster. Make sure “Use SCIFIO when opening files” is disabled in Edit>Options>ImageJ2.
>

That works indeed much faster, thanks!

Laurent

> -wayne
>
>
>>
>> Cheers,
>> Laurent
>>
>>
>>
>>
>>> On Jun 5, 2015, at 11:37 PM, Bill N. Whitney - whitnewn <[hidden email]> wrote:
>>>
>>> I am running ImageJ version 1.49t (from Fiji) on Windows 7.
>>>
>>> I am working with a microscope that creates image stacks and stores them as TIFF files.
>>> Sometimes, the image stack files can get bigger than 4GB, so we can also create BigTiff files.
>>>
>>> I can load a REGULAR Tiff file into ImageJ in at most 30 seconds. A file right at the 4GB limit takes 20-30 seconds to load. Smaller files are faster.
>>>
>>> Even a small BigTiff format file ( 64 MB) takes much longer than this, roughly 120 - 180 seconds.
>>>
>>> It takes a long time for ImageJ to START processing the individual images (2-3 minutes), AND it takes a long time for ImageJ to process each individual image (~30 sec).
>>>
>>> For example, loading a 1.8GB BigTiff file takes about 4-5 hours to load. The same file, stored as regular Tiff, takes about 12 seconds.
>>> I have a 5GB BigTiff file, and after letting it run for 3 hours, it had only processed 85 of 500 images, so I killed the process.
>>>
>>> Any insights into what is going on here? I can provide sample files if that would be helpful .....
>>>
>>> Thanks,
>>> Bill
>
>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html