Problems with Saving Tiff greater than 4 GB & bigTiff exporter

[T.J. Wardill]

 

Dear ImageJ list,

Can someone tell me exactly which format is a multipage-tif file that
ImageJ/Fiji saves when the total size is over 4GB? I have a problem that
Matlab cannot load subframes from these large files. I also tried using
the bioformats exporter to get a bigTiff formatted file but found this
would not work. I tried the tif and OME-Tif extensions but got the log
error that stated loci.formats.FormatException: File is too large; call
setBigTiff(true), and the saving up to 4gb when the log error occurred
was quite slow. There does not seem to be anywhere easy to set this
parameter within Fiji. This is probably a simple error but any help
would be most appreciated.

Best, Trevor
--

DR TREVOR WARDILL (BBSRC David Phillips Fellow)
Department of Physiology, Development and Neuroscience
Physiology Building, University of Cambridge
Downing Street, Cambridge, CB2 3EG
(M): 07914 945 555 (W): 01223 746 919
[hidden email]

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
 

[dscho]

Hi Trevor,

On Thu, 16 Oct 2014, T.J. Wardill wrote:

> Can someone tell me exactly which format is a multipage-tif file that
> ImageJ/Fiji saves when the total size is over 4GB?

There are unfortunately multiple ways to save .tiff files. In your case,
you might want to use File>Save As>SCIFIO... or File>Save As>Bio-Formats...

Ciao,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[dscho]

Hi Trevor,

please do not cut out the community of the conversation. I re-Cc:ed the
mailing list. The question you asked could have been easily answered by
other users than myself.

On Thu, 16 Oct 2014, T.J. Wardill wrote:

> Unfortunately in my version of Fiji & ImageJ, I do not have these save
> as options. Are these a plugin?

Yes. The Bio-Formats and SCIFIO plugins, respectively. If you want to use
plain ImageJ 1.x, you will have to get them from somewhere and install
them manually. In Fiji, Bio-Formats is included since 2008 and SCIFIO is
included since sometime last year.

Ciao,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[T.J. Wardill]

 

Thanks Johannes,

So my original post asked about the bioformats plugin, noting it will
not save tif files larger than 4 GB. I cannot use ImageJ2 as it has too
many bugs for my purposes. I do not want to use single files for each
frame as I have many 10,000's of frames from dual high speed cameras. I
can currently use 2GB chunks as tifs but wanted to manage my data in
single takes which are about 15-20 GB.

SCIFIO seems to be integrated with the save function in ImageJ, but the
file format for large tiffs, is not readable by Matlab, so at the moment
there is no solution that I can find in standard ImageJ/Fiji that will
save a multipage tiff that is about 15-20GB and can be read by Matlab.
Now it may just be that Matlab is useless with bigtif file formats but I
am not sure this is the case.

If anyone can please tell the file format for the large tifs saved with
ImageJ standard I would be most grateful. This may help me diagnose why
Matlab cannot read them.

Best regards, Trevor

On 2014-10-16 15:01, Johannes Schindelin wrote:

> Hi Trevor,
>
> please do not cut out the community of the conversation. I re-Cc:ed the
> mailing list. The question you asked could have been easily answered by
> other users than myself.
>
> On Thu, 16 Oct 2014, T.J. Wardill wrote:
>
>> Unfortunately in my version of Fiji & ImageJ, I do not have these save as options. Are these a plugin?
>
> Yes. The Bio-Formats and SCIFIO plugins, respectively. If you want to use
> plain ImageJ 1.x, you will have to get them from somewhere and install
> them manually. In Fiji, Bio-Formats is included since 2008 and SCIFIO is
> included since sometime last year.
>
> Ciao,
> Johannes
>
> Thanks Johannes, look like good ideas, are these in ImageJ?
>
> Unfortunately in my version of Fiji & ImageJ, I do not have these save as options. Are these a plugin?
>
> Regards, Trevor
>
> On 2014-10-16 14:04, Johannes Schindelin wrote:
>
> Hi Trevor,
>
> On Thu, 16 Oct 2014, T.J. Wardill wrote:
> Can someone tell me exactly which format is a multipage-tif file that ImageJ/Fiji saves when the total size is over 4GB?
>
> There are unfortunately multiple ways to save .tiff files. In your case,
> you might want to use File>Save As>SCIFIO... or File>Save As>Bio-Formats...
>
> Ciao,
> Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
 

[Mark Hiner]

>File>Save As>SCIFIO... or File>Save As>Bio-Formats...

I'm not aware of these specific plugin paths. To save using Bio-Formats,
you can use the Bio-Formats Exporter (under Plugins > Bio-Formats).

You can write out using SCIFIO via File > Export > Image... but note that
this requires data that was either a) read in via File > Import > Image...
or b) opened with Edit > Options > ImageJ2 > Enable ImageJ2 data structures
enabled.

Regards,
Mark

On Thu, Oct 16, 2014 at 9:01 AM, Johannes Schindelin <
[hidden email]> wrote:

> Hi Trevor,
>
> please do not cut out the community of the conversation. I re-Cc:ed the
> mailing list. The question you asked could have been easily answered by
> other users than myself.
>
> On Thu, 16 Oct 2014, T.J. Wardill wrote:
>
> > Unfortunately in my version of Fiji & ImageJ, I do not have these save
> > as options. Are these a plugin?
>
> Yes. The Bio-Formats and SCIFIO plugins, respectively. If you want to use
> plain ImageJ 1.x, you will have to get them from somewhere and install
> them manually. In Fiji, Bio-Formats is included since 2008 and SCIFIO is
> included since sometime last year.
>
> Ciao,
> Johannes
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[dscho]

Hi Trevor,

On Thu, 16 Oct 2014, T.J. Wardill wrote:

> SCIFIO seems to be integrated with the save function in ImageJ, but the
> file format for large tiffs, is not readable by Matlab,

Why not use Miji to use Fiji functionality (including SCIFIO) inside
Matlab? http://fiji.sc/Miji

Ciao,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Emanuele Martini]

but are we sure that is a problem of ImageJ/Fiji saving and not in matlab loading?
I think that as Johannes says Miji could be a nice solution.

Ciao,
Emanuele Martini @ IFOM-Ieo Research Campus

[Rasband, Wayne (NIH/NIMH) [E]]

On Oct 16, 2014, at 4:33 AM, T.J. Wardill wrote:

> Dear ImageJ list,
>
> Can someone tell me exactly which format is a multipage-tif file that
> ImageJ/Fiji saves when the total size is over 4GB?

Stacks larger than 4GB are saved in "raw" format, as if you had used File>Save As>Raw except the raw data is preceded by a small header. You can find the header size (offset) by opening the file in ImageJ with debugging enabled in Edit>Options>Misc. Matlab should be able to open the files if it has the ability to open raw files. When saving stacks larger than 4GB, the latest ImageJ daily build (1.49j3) displays the information needed to open the stack as raw data. Here is an example of the information displayed:

  Stack is larger than 4GB. Most TIFF readers will only open the first image. Use this information to open as raw:
  name=Untitled, dir=, width=1024, height=1024, nImages=2050, offset=203, type=ushort, byteOrder=little, format=0, url=, whiteIsZero=f, lutSize=0, comp=1, ranges=null, samples=1

-wayne

> I have a problem that
> Matlab cannot load subframes from these large files. I also tried using
> the bioformats exporter to get a bigTiff formatted file but found this
> would not work. I tried the tif and OME-Tif extensions but got the log
> error that stated loci.formats.FormatException: File is too large; call
> setBigTiff(true), and the saving up to 4gb when the log error occurred
> was quite slow. There does not seem to be anywhere easy to set this
> parameter within Fiji. This is probably a simple error but any help
> would be most appreciated.
>
> Best, Trevor
> --
>
> DR TREVOR WARDILL (BBSRC David Phillips Fellow)
> Department of Physiology, Development and Neuroscience
> Physiology Building, University of Cambridge
> Downing Street, Cambridge, CB2 3EG
> (M): 07914 945 555 (W): 01223 746 919
> [hidden email]
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Djoere Gaublomme]

Hi,

I was wondering if there is an update to this.
I am trying to save a  >4GB file as OME-TIFF via the BioFormats  
exporter in a macro, but I get this error as well.  I want to import  
the file into Imaris with timing metadata still attached (which works  
with ome-tiff).  I have tried saving as raw, but Imaris doesn't accept  
this format.

I read that SCIFIO should already automatically handle the  
bigTiff=true setting, yet I still get the error.
My IT knowledge currently doesn't go much further than writing a  
simple macro, so I have to admit feeling somewhat confused when  
reading explanations about this bug.

The macro command I tried is:
run("Bio-Formats Exporter", "save="+ outputfolder + name +".ome.tif  
compression=Uncompressed");

I tried with an updated version of Fiji (ImageJ  1.49p)

Is there a simple solution to this problem yet?

Thanks

Djoere




Quoting "Rasband, Wayne (NIH/NIMH) [E]" <[hidden email]>:

> On Oct 16, 2014, at 4:33 AM, T.J. Wardill wrote:
>
>> Dear ImageJ list,
>>
>> Can someone tell me exactly which format is a multipage-tif file that
>> ImageJ/Fiji saves when the total size is over 4GB?
>
> Stacks larger than 4GB are saved in "raw" format, as if you had used  
> File>Save As>Raw except the raw data is preceded by a small header.  
> You can find the header size (offset) by opening the file in ImageJ  
> with debugging enabled in Edit>Options>Misc. Matlab should be able  
> to open the files if it has the ability to open raw files. When  
> saving stacks larger than 4GB, the latest ImageJ daily build  
> (1.49j3) displays the information needed to open the stack as raw  
> data. Here is an example of the information displayed:
>
>   Stack is larger than 4GB. Most TIFF readers will only open the  
> first image. Use this information to open as raw:
>   name=Untitled, dir=, width=1024, height=1024, nImages=2050,  
> offset=203, type=ushort, byteOrder=little, format=0, url=,  
> whiteIsZero=f, lutSize=0, comp=1, ranges=null, samples=1
>
> -wayne
>
>> I have a problem that
>> Matlab cannot load subframes from these large files. I also tried using
>> the bioformats exporter to get a bigTiff formatted file but found this
>> would not work. I tried the tif and OME-Tif extensions but got the log
>> error that stated loci.formats.FormatException: File is too large; call
>> setBigTiff(true), and the saving up to 4gb when the log error occurred
>> was quite slow. There does not seem to be anywhere easy to set this
>> parameter within Fiji. This is probably a simple error but any help
>> would be most appreciated.
>>
>> Best, Trevor
>> --
>>
>> DR TREVOR WARDILL (BBSRC David Phillips Fellow)
>> Department of Physiology, Development and Neuroscience
>> Physiology Building, University of Cambridge
>> Downing Street, Cambridge, CB2 3EG
>> (M): 07914 945 555 (W): 01223 746 919
>> [hidden email]
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Niko Ehrenfeuchter]

Hi,

I'd be curious as well as we are basically having the same issue here...

Unfortunately, I can't really offer any useful hints either. From my
experience, the combination with Imaris is even more tricky as they
can't read BigTIFFs - at least none that I tried.

Best
~Niko

On 17.03.2015 17:22, Djoere Gaublomme wrote:

> Hi,
>
> I was wondering if there is an update to this.
> I am trying to save a  >4GB file as OME-TIFF via the BioFormats exporter
> in a macro, but I get this error as well.  I want to import the file
> into Imaris with timing metadata still attached (which works with
> ome-tiff).  I have tried saving as raw, but Imaris doesn't accept this
> format.
>
> I read that SCIFIO should already automatically handle the bigTiff=true
> setting, yet I still get the error.
> My IT knowledge currently doesn't go much further than writing a simple
> macro, so I have to admit feeling somewhat confused when reading
> explanations about this bug.
>
> The macro command I tried is:
> run("Bio-Formats Exporter", "save="+ outputfolder + name +".ome.tif
> compression=Uncompressed");
>
> I tried with an updated version of Fiji (ImageJ  1.49p)
>
> Is there a simple solution to this problem yet?
>
> Thanks
>
> Djoere
>
>
>
>
> Quoting "Rasband, Wayne (NIH/NIMH) [E]" <[hidden email]>:
>
>> On Oct 16, 2014, at 4:33 AM, T.J. Wardill wrote:
>>
>>> Dear ImageJ list,
>>>
>>> Can someone tell me exactly which format is a multipage-tif file that
>>> ImageJ/Fiji saves when the total size is over 4GB?
>>
>> Stacks larger than 4GB are saved in "raw" format, as if you had used
>> File>Save As>Raw except the raw data is preceded by a small header.
>> You can find the header size (offset) by opening the file in ImageJ
>> with debugging enabled in Edit>Options>Misc. Matlab should be able to
>> open the files if it has the ability to open raw files. When saving
>> stacks larger than 4GB, the latest ImageJ daily build (1.49j3)
>> displays the information needed to open the stack as raw data. Here is
>> an example of the information displayed:
>>
>>   Stack is larger than 4GB. Most TIFF readers will only open the first
>> image. Use this information to open as raw:
>>   name=Untitled, dir=, width=1024, height=1024, nImages=2050,
>> offset=203, type=ushort, byteOrder=little, format=0, url=,
>> whiteIsZero=f, lutSize=0, comp=1, ranges=null, samples=1
>>
>> -wayne
>>
>>> I have a problem that
>>> Matlab cannot load subframes from these large files. I also tried using
>>> the bioformats exporter to get a bigTiff formatted file but found this
>>> would not work. I tried the tif and OME-Tif extensions but got the log
>>> error that stated loci.formats.FormatException: File is too large; call
>>> setBigTiff(true), and the saving up to 4gb when the log error occurred
>>> was quite slow. There does not seem to be anywhere easy to set this
>>> parameter within Fiji. This is probably a simple error but any help
>>> would be most appreciated.
>>>
>>> Best, Trevor
>>> --
>>>
>>> DR TREVOR WARDILL (BBSRC David Phillips Fellow)
>>> Department of Physiology, Development and Neuroscience
>>> Physiology Building, University of Cambridge
>>> Downing Street, Cambridge, CB2 3EG
>>> (M): 07914 945 555 (W): 01223 746 919
>>> [hidden email]
>>>
>>> --
>>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>>
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
Niko Ehrenfeuchter | Image Analysis Specialist | Biozentrum, University
of Basel | Klingelbergstr. 50/70 | CH-4056 Basel
Phone: [phone] | [hidden email] |
www.biozentrum.unibas.ch | www.microscopynetwork.unibas.ch

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[ctrueden]

Hi Djoere, Niko and everyone,

> Is there a simple solution to this problem yet?

I filed an issue to track the various "better support for big TIFF files"
scenarios people have raised:
    https://github.com/imagej/imagej/issues/117

Regards,
Curtis

On Wed, Mar 18, 2015 at 7:18 AM, Niko Ehrenfeuchter <
[hidden email]> wrote:

> Hi,
>
> I'd be curious as well as we are basically having the same issue here...
>
> Unfortunately, I can't really offer any useful hints either. From my
> experience, the combination with Imaris is even more tricky as they can't
> read BigTIFFs - at least none that I tried.
>
> Best
> ~Niko
>
>
> On 17.03.2015 17:22, Djoere Gaublomme wrote:
>
>> Hi,
>>
>> I was wondering if there is an update to this.
>> I am trying to save a  >4GB file as OME-TIFF via the BioFormats exporter
>> in a macro, but I get this error as well.  I want to import the file
>> into Imaris with timing metadata still attached (which works with
>> ome-tiff).  I have tried saving as raw, but Imaris doesn't accept this
>> format.
>>
>> I read that SCIFIO should already automatically handle the bigTiff=true
>> setting, yet I still get the error.
>> My IT knowledge currently doesn't go much further than writing a simple
>> macro, so I have to admit feeling somewhat confused when reading
>> explanations about this bug.
>>
>> The macro command I tried is:
>> run("Bio-Formats Exporter", "save="+ outputfolder + name +".ome.tif
>> compression=Uncompressed");
>>
>> I tried with an updated version of Fiji (ImageJ  1.49p)
>>
>> Is there a simple solution to this problem yet?
>>
>> Thanks
>>
>> Djoere
>>
>>
>>
>>
>> Quoting "Rasband, Wayne (NIH/NIMH) [E]" <[hidden email]>:
>>
>>  On Oct 16, 2014, at 4:33 AM, T.J. Wardill wrote:
>>>
>>>  Dear ImageJ list,
>>>>
>>>> Can someone tell me exactly which format is a multipage-tif file that
>>>> ImageJ/Fiji saves when the total size is over 4GB?
>>>>
>>>
>>> Stacks larger than 4GB are saved in "raw" format, as if you had used
>>> File>Save As>Raw except the raw data is preceded by a small header.
>>> You can find the header size (offset) by opening the file in ImageJ
>>> with debugging enabled in Edit>Options>Misc. Matlab should be able to
>>> open the files if it has the ability to open raw files. When saving
>>> stacks larger than 4GB, the latest ImageJ daily build (1.49j3)
>>> displays the information needed to open the stack as raw data. Here is
>>> an example of the information displayed:
>>>
>>>   Stack is larger than 4GB. Most TIFF readers will only open the first
>>> image. Use this information to open as raw:
>>>   name=Untitled, dir=, width=1024, height=1024, nImages=2050,
>>> offset=203, type=ushort, byteOrder=little, format=0, url=,
>>> whiteIsZero=f, lutSize=0, comp=1, ranges=null, samples=1
>>>
>>> -wayne
>>>
>>>  I have a problem that
>>>> Matlab cannot load subframes from these large files. I also tried using
>>>> the bioformats exporter to get a bigTiff formatted file but found this
>>>> would not work. I tried the tif and OME-Tif extensions but got the log
>>>> error that stated loci.formats.FormatException: File is too large; call
>>>> setBigTiff(true), and the saving up to 4gb when the log error occurred
>>>> was quite slow. There does not seem to be anywhere easy to set this
>>>> parameter within Fiji. This is probably a simple error but any help
>>>> would be most appreciated.
>>>>
>>>> Best, Trevor
>>>> --
>>>>
>>>> DR TREVOR WARDILL (BBSRC David Phillips Fellow)
>>>> Department of Physiology, Development and Neuroscience
>>>> Physiology Building, University of Cambridge
>>>> Downing Street, Cambridge, CB2 3EG
>>>> (M): 07914 945 555 (W): 01223 746 919
>>>> [hidden email]
>>>>
>>>> --
>>>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>>>
>>>>
>>> --
>>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>>
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>
>
> --
> Niko Ehrenfeuchter | Image Analysis Specialist | Biozentrum, University of
> Basel | Klingelbergstr. 50/70 | CH-4056 Basel
> Phone: [phone] | [hidden email] |
> www.biozentrum.unibas.ch | www.microscopynetwork.unibas.ch
>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Christian Tischer]

Hi All,

[Plugins>Bio-Formats>Bio-Formats Exporter] "Image Cytometry Standard"

does the trick: it can save >4GB files with 8 or 16 bit and XYZCT and Imaris loads them both fast and correct. I tried also several times using tif or ome.tif (e.g. one tif stack per C and T) but got inconsistent results.

I happen to know that also the Huygens Deconvolution Software likes ICS and Arivis in principle also should be able to load it. I don't know about Ilastik yet..

Maybe ICS is a format that could be pushed more?

https://en.wikipedia.org/wiki/Image_Cytometry_Standard

Tischi