Saving large files in ome.tif or ids/ics

[Jason Miller]

Hi all-

This may be a Bioformats issue as well, so I will separately e-mail the OME
folks, but similar to recent past posts on this list, I'm having trouble
saving files that are > 4GB into ome.tif or IDS/ICS formats. I have a 5GB
file, open as a virtual file, and when I go to export via Bioformats
Exporter to IDS/ICS format, I get the following error:

(Fiji Is Just) ImageJ 2.0.0-rc-15/1.49p; Java 1.6.0_24 [64-bit]; Windows 7
6.1; 98MB of 12209MB (<1%)

java.lang.IllegalArgumentException: Negative position
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:600)
at
loci.common.NIOByteBufferProvider.allocateDirect(NIOByteBufferProvider.java:131)
at
loci.common.NIOByteBufferProvider.allocate(NIOByteBufferProvider.java:116)
at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:551)
at loci.common.NIOFileHandle.seek(NIOFileHandle.java:273)
at
loci.common.RandomAccessOutputStream.seek(RandomAccessOutputStream.java:79)
at loci.formats.out.ICSWriter.saveBytes(ICSWriter.java:139)
at loci.formats.FormatWriter.saveBytes(FormatWriter.java:126)
at loci.plugins.out.Exporter.run(Exporter.java:629)
at loci.plugins.LociExporter.run(LociExporter.java:77)
at
ij.plugin.filter.PlugInFilterRunner.processOneImage(PlugInFilterRunner.java:263)
at ij.plugin.filter.PlugInFilterRunner.<init>(PlugInFilterRunner.java:112)
at ij.IJ.runUserPlugIn(IJ.java:201)
at ij.IJ.runPlugIn(IJ.java:163)
at ij.Executer.runCommand(Executer.java:131)
at ij.Executer.run(Executer.java:64)
at java.lang.Thread.run(Thread.java:662)


When I go to export to ome.tif format, I get the same error previously
reported on this list:
loci.formats.FormatException: File is too large; call setBigTiff(true)

Is it possible to retain the ome.tif format for >4GB? In my case, this is
for importing into Huygens for deconvolution.

Thanks so very much
-Jason
--

Jason Miller, MD, PhD

University of Michigan Kellogg Eye Center

--

Home address:

117 Worden Ave

Ann Arbor, MI 48103

Cell: [phone]

E-mail: *[hidden email] <[hidden email]>*


"When I was 5 years old, my mother always told me that happiness was the
key to life. When I went to school, they asked me what I wanted to be when
I grew up. I wrote down, 'Happy.' They told me I didn't understand the
assignment. I told them they didn't understand life." - John Lennon

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Jason Miller]

Hi all-

I received an answer from the OME folks about my question and figured I'd
post here to keep this in the same thread. This was helpful info for me.

-------------------------------
Dear Jason,

> Similar to recent past posts on this list, I'm having trouble saving
> files that are > 4GB into ome.tif or IDS/ICS formats. I have a 5GB file,
> open as a virtual file, and when I go to export via Bioformats Exporter
> to IDS/ICS format, I get the following error:
[stacktrace]

For IDS/ICS, this was fixed in this PR last week:
   https://github.com/openmicroscopy/bioformats/pull/1738
I have tested this with >5GB images and it works correctly.

> When I go to export to ome.tif format, I get the same error previously
> reported on this list:
> loci.formats.FormatException: File is too large; call setBigTiff(true)
>
> Is it possible to retain the ome.tif format for >4GB? In my case, this
> is for importing into Huygens for deconvolution.

For plain (not OME) TIFF:
   https://github.com/openmicroscopy/bioformats/pull/1744

Both of the above fixes are in the newly-released Bio-Formats 5.1.1
release.  If using Fiji, you should automatically get this update; for
ImageJ you can download the updated plugin from
   http://downloads.openmicroscopy.org/bio-formats/5.1.1/

For OME-TIFF, it's not currently possible to enable BigTIFF when
exporting from ImageJ using the exporter dialogue.  For the plain TIFF
case, rather than relying on an API call to enable BigTIFF, we now
enable it if the image data is >4GiB or a "big" TIFF file extension such
as .btf, .tf8 or .tf2 is used.  We would like to do the same for
OME-TIFF, but this first requires updating the OME-TIFF specification
which currently only allows a .ome.tiff or .ome.tif file extension.
Once that's done, you would be able to use e.g. ".ome.btf" and get a
"big" OME-TIFF file.  However, this does require some consideration
since (for example) these files would not be readable with older
versions of Bio-Formats unless you renamed them to ".ome.tiff" after export.

To summarise, it isn't currently possible to export >4GB ome.tiff files
using the Bio-Formats Exporter.  It is possible via the Java API by
calling writer.setBigTiff(true) if this is an option for you.  This is a
known limitation which we are working on; see the discussion in the PR
#1744 above.

There is a ticket open here for this:
   https://trac.openmicroscopy.org/ome/ticket/12858
I can add you as a CC on it if you like, so you'll be kept up to date
with its progress?


Kind regards,
Roger

--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
College of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK   Tel: [phone]

The University of Dundee is a registered Scottish Charity, No: SC015096

On Thu, Apr 30, 2015 at 5:28 PM, Jason Miller <[hidden email]>
wrote:

> Hi all-
>
> This may be a Bioformats issue as well, so I will separately e-mail the
> OME folks, but similar to recent past posts on this list, I'm having
> trouble saving files that are > 4GB into ome.tif or IDS/ICS formats. I have
> a 5GB file, open as a virtual file, and when I go to export via Bioformats
> Exporter to IDS/ICS format, I get the following error:
>
> (Fiji Is Just) ImageJ 2.0.0-rc-15/1.49p; Java 1.6.0_24 [64-bit]; Windows 7
> 6.1; 98MB of 12209MB (<1%)
>
> java.lang.IllegalArgumentException: Negative position
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:600)
> at
> loci.common.NIOByteBufferProvider.allocateDirect(NIOByteBufferProvider.java:131)
> at
> loci.common.NIOByteBufferProvider.allocate(NIOByteBufferProvider.java:116)
> at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:551)
> at loci.common.NIOFileHandle.seek(NIOFileHandle.java:273)
> at
> loci.common.RandomAccessOutputStream.seek(RandomAccessOutputStream.java:79)
> at loci.formats.out.ICSWriter.saveBytes(ICSWriter.java:139)
> at loci.formats.FormatWriter.saveBytes(FormatWriter.java:126)
> at loci.plugins.out.Exporter.run(Exporter.java:629)
> at loci.plugins.LociExporter.run(LociExporter.java:77)
> at
> ij.plugin.filter.PlugInFilterRunner.processOneImage(PlugInFilterRunner.java:263)
> at ij.plugin.filter.PlugInFilterRunner.<init>(PlugInFilterRunner.java:112)
> at ij.IJ.runUserPlugIn(IJ.java:201)
> at ij.IJ.runPlugIn(IJ.java:163)
> at ij.Executer.runCommand(Executer.java:131)
> at ij.Executer.run(Executer.java:64)
> at java.lang.Thread.run(Thread.java:662)
>
>
> When I go to export to ome.tif format, I get the same error previously
> reported on this list:
> loci.formats.FormatException: File is too large; call setBigTiff(true)
>
> Is it possible to retain the ome.tif format for >4GB? In my case, this is
> for importing into Huygens for deconvolution.
>
> Thanks so very much
> -Jason
> --
>
> Jason Miller, MD, PhD
>
> University of Michigan Kellogg Eye Center
>
> --
>
> Home address:
>
> 117 Worden Ave
>
> Ann Arbor, MI 48103
>
> Cell: [phone]
>
> E-mail: *[hidden email]
> <[hidden email]>*
>
>
> "When I was 5 years old, my mother always told me that happiness was the
> key to life. When I went to school, they asked me what I wanted to be when
> I grew up. I wrote down, 'Happy.' They told me I didn't understand the
> assignment. I told them they didn't understand life." - John Lennon
>

--

Jason Miller, MD, PhD

University of Michigan Kellogg Eye Center

--

Home address:

117 Worden Ave

Ann Arbor, MI 48103

Cell: [phone]

E-mail: *[hidden email] <[hidden email]>*


"When I was 5 years old, my mother always told me that happiness was the
key to life. When I went to school, they asked me what I wanted to be when
I grew up. I wrote down, 'Happy.' They told me I didn't understand the
assignment. I told them they didn't understand life." - John Lennon

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html