Alternative to Tif?

[Jacob Keller-2]

Dear Imagers,

is there an alternative to tiff which allows larger files without needing
to resort to raw format? Whenever I go above ~3 GB I get a message that I
will need certain info, given in a text box, to re-open that file, but it's
a bit of a pain, an seems unnecessary. I usually input .czi files, then
need to output processed files.

Jacob Keller

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Peterbauer Thomas]

On 2017-11-01 14:54, Jacob Keller wrote:
> is there an alternative to tiff which allows larger files without needing
> to resort to raw format?

Use the Bio-Formats plugin. It can save as BigTIFF (.btf), essentially
without size limit (2^64 bit)!

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Jacob Keller-2]

I tried this, but it multiplied the size, and also made it very slow to
open. Is that inherent, or something I did wrong?

JPK


On Wed, Nov 1, 2017 at 11:23 AM, Peterbauer Thomas <
[hidden email]> wrote:

>
>
> On 2017-11-01 14:54, Jacob Keller wrote:
> > is there an alternative to tiff which allows larger files without needing
> > to resort to raw format?
>
> Use the Bio-Formats plugin. It can save as BigTIFF (.btf), essentially
> without size limit (2^64 bit)!
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Peterbauer Thomas]

On 2017-11-01 16:31, Jacob Keller wrote:
> I tried this, but it multiplied the size, and also made it very slow to
> open.

Well, Bio-Formats is generally rather slow in reading & writing.
However, I cannot confirm your size issue. I just created an 8-bit image
with 2048 x 2048 pixels and 1000 slices (3.9 GB, just below the 4
GB-limit), and saved as plain ImageJ-TIFF and BigTIFF. Both ended up on
hard drive with the same size (3.9 GB). Note that a small increase of
the size of the file header is expected, since the offsets (marking
where one frame starts/ends) are stored in 64 bits rather than 32 bit as
in standard-TIFFs. I should also mention that you *can* go above 4 GB
with ImageJ-TIFF. The text box popping up just warns you that
third-party TIFF-readers may fail. ImageJ itself can read its own
king-size TIFFs beyond 4 GB (the largest I successfully tried to write &
read afterwards was 2048x2048 pixels, 16-bit, 680 frames, a
5.3-GB-stack). So the choice how to save depends on what you want to do
with these images after processing in ImageJ/Fiji.

(The machine I used for these tests is a Linux laptop, 64-bit-OS, with
16 GB RAM)

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

Hi, did you consider HDF5 as used by the BigDataViewer?

All the best,
Stephan

Sent from my iPhone

On Nov 1, 2017, at 14:28, Peterbauer Thomas <[hidden email]<mailto:[hidden email]>> wrote:

On 2017-11-01 16:31, Jacob Keller wrote:
I tried this, but it multiplied the size, and also made it very slow to
open.

Well, Bio-Formats is generally rather slow in reading & writing.
However, I cannot confirm your size issue. I just created an 8-bit image
with 2048 x 2048 pixels and 1000 slices (3.9 GB, just below the 4
GB-limit), and saved as plain ImageJ-TIFF and BigTIFF. Both ended up on
hard drive with the same size (3.9 GB). Note that a small increase of
the size of the file header is expected, since the offsets (marking
where one frame starts/ends) are stored in 64 bits rather than 32 bit as
in standard-TIFFs. I should also mention that you *can* go above 4 GB
with ImageJ-TIFF. The text box popping up just warns you that
third-party TIFF-readers may fail. ImageJ itself can read its own
king-size TIFFs beyond 4 GB (the largest I successfully tried to write &
read afterwards was 2048x2048 pixels, 16-bit, 680 frames, a
5.3-GB-stack). So the choice how to save depends on what you want to do
with these images after processing in ImageJ/Fiji.

(The machine I used for these tests is a Linux laptop, 64-bit-OS, with
16 GB RAM)

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Jacob Keller-2]

Yes, but afaik there is no converter in IJ. Is there?

JPK

On Fri, Nov 10, 2017 at 1:30 PM, [hidden email] <
[hidden email]> wrote:

> Hi, did you consider HDF5 as used by the BigDataViewer?
>
> All the best,
> Stephan
>
> Sent from my iPhone
>
> On Nov 1, 2017, at 14:28, Peterbauer Thomas <[hidden email].
> AT<mailto:[hidden email]>> wrote:
>
> On 2017-11-01 16:31, Jacob Keller wrote:
> I tried this, but it multiplied the size, and also made it very slow to
> open.
>
> Well, Bio-Formats is generally rather slow in reading & writing.
> However, I cannot confirm your size issue. I just created an 8-bit image
> with 2048 x 2048 pixels and 1000 slices (3.9 GB, just below the 4
> GB-limit), and saved as plain ImageJ-TIFF and BigTIFF. Both ended up on
> hard drive with the same size (3.9 GB). Note that a small increase of
> the size of the file header is expected, since the offsets (marking
> where one frame starts/ends) are stored in 64 bits rather than 32 bit as
> in standard-TIFFs. I should also mention that you *can* go above 4 GB
> with ImageJ-TIFF. The text box popping up just warns you that
> third-party TIFF-readers may fail. ImageJ itself can read its own
> king-size TIFFs beyond 4 GB (the largest I successfully tried to write &
> read afterwards was 2048x2048 pixels, 16-bit, 680 frames, a
> 5.3-GB-stack). So the choice how to save depends on what you want to do
> with these images after processing in ImageJ/Fiji.
>
> (The machine I used for these tests is a Linux laptop, 64-bit-OS, with
> 16 GB RAM)
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Stein Rørvik]

I routinely work with normal TIFF µCT stacks of 16bit 2000x2000x2000 voxels, which are 16GB filesize and have never had any problem, as long as I stay in ImageJ. The warning is just a warning that third party programs might not be able to read the file. If you continue to work in ImageJ you can therefore just ignore this message. Just keep in mind that if you use FAT32 formatted media such as USB keys or similar, the media will then not support storage of >4GB files, and you need at least twice the stack size available as physical memory on your computer to open them.

When dealing with stacks of these sizes, I prefer to work with virtual stacks. Since they are not fully loaded in memory, you will not have any memory issues. Also, since the entire file is not read, it loads in just a fraction of a second even when the file is 16GB or larger. You can load both a folder of single slices, or one big file as a virtual stack. Even a raw file can be opened as a virtual stack if you know its data layout. The main disadvantage with virtual stacks is that vertical slicing is slow since that requires the entire file to be read, sequentially.

Stein

-----Original Message-----
From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of Peterbauer Thomas
Sent: 01. november 2017 17:17
To: [hidden email]
Subject: Re: Alternative to Tif?

On 2017-11-01 16:31, Jacob Keller wrote:
> I tried this, but it multiplied the size, and also made it very slow
> to open.

Well, Bio-Formats is generally rather slow in reading & writing.
However, I cannot confirm your size issue. I just created an 8-bit image with 2048 x 2048 pixels and 1000 slices (3.9 GB, just below the 4 GB-limit), and saved as plain ImageJ-TIFF and BigTIFF. Both ended up on hard drive with the same size (3.9 GB). Note that a small increase of the size of the file header is expected, since the offsets (marking where one frame starts/ends) are stored in 64 bits rather than 32 bit as in standard-TIFFs. I should also mention that you *can* go above 4 GB with ImageJ-TIFF. The text box popping up just warns you that third-party TIFF-readers may fail. ImageJ itself can read its own king-size TIFFs beyond 4 GB (the largest I successfully tried to write & read afterwards was 2048x2048 pixels, 16-bit, 680 frames, a 5.3-GB-stack). So the choice how to save depends on what you want to do with these images after processing in ImageJ/Fiji.

(The machine I used for these tests is a Linux laptop, 64-bit-OS, with
16 GB RAM)

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Adrian Daerr-2]

> When dealing with stacks of these sizes, I prefer to work with virtual stacks.
> [..] Even a raw file can be opened as a virtual stack if you know its data layout.

I concur on the convenience of virtual stacks. A small but annoying
shortcoming of raw virtual stacks however is that when you do not know
the exact number of images, and enter a higher number in the dialog,
then your stack display will irreversibly freeze once you attempt to
access a slice beyond the end of file instead of merely updating the
size of the stack. Of course you can calculate the right size from the
byte size of your file divided by the camera resolution, but this
operation partly negates the convenience of being able to
quasi-instantaneously view and browse through recordings on the disk via
virtual stacks.

> The main disadvantage with virtual stacks is that vertical slicing
> is slow since that requires the entire file to be read, sequentially.

It would indeed be great to have a version of the reslice function
optimized for the slow random access on mechanical hard disks.

cheers,
Adrian

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Wayne Rasband-2]

> On Nov 13, 2017, at 8:08 AM, Adrian Daerr <[hidden email]> wrote:
>
>> When dealing with stacks of these sizes, I prefer to work with virtual stacks. [..] Even a raw file can be opened as a virtual stack if you know its data layout.
>
> I concur on the convenience of virtual stacks. A small but annoying shortcoming of raw virtual stacks however is that when you do not know the exact number of images, and enter a higher number in the dialog, then your stack display will irreversibly freeze once you attempt to access a slice beyond the end of file instead of merely updating the size of the stack. Of course you can calculate the right size from the byte size of your file divided by the camera resolution, but this operation partly negates the convenience of being able to quasi-instantaneously view and browse through recordings on the disk via virtual stacks.

This bug is fixed in the latest ImageJ daily build (1.51s24). Now when you enter a higher number of images in the dialog ImageJ will calculate the correct number based on the offset, image size, gap between images and the file size.

-wayne

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Adrian Daerr-2]

>> A small but annoying shortcoming of raw virtual stacks however is that when you do not know the exact number of images, and enter a higher number in the dialog, then your stack display will irreversibly freeze once you attempt to access a slice beyond the end of file instead of merely updating the size of the stack.

On 11/14/2017 10:46 PM, Wayne Rasband wrote:
> This bug is fixed in the latest ImageJ daily build (1.51s24). Now when you enter a higher number of images in the dialog ImageJ will calculate the correct number based on the offset, image size, gap between images and the file size.

Great, thank you very much Wayne!

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html