I just used ImageJ to output a 36GB AVI file: 8000X8000 pixels X 191
frames, uncompressed. If I use the File>Import>AVI... command to open it, the first dialog correctly offers the range 1-191 slices. However, when I open that range as a virtual stack, I get only the first 9 frames displayed in the resulting stack window (conspicuously ~2.2GB of data). They're at full resolution, but the slider won't take me past #9, and the subtitle says 1/9. Any notions as to why this is? I thought that recent changes to the AVI reader had it opening larger AVIs. Thanks, Bill |
Hi Bill,
Were these AVIs written by ImageJ? ImageJ currently does not support writing AVIs larger than 2 GB (I fear that this is undocumented and there is no error message). If the AVIs came from some other program and you can open them in other AVI viewers, but not in ImageJ 1.46f or later, please turn Edit>Options>Mics>Debug Mode on and post the log that you get when opening the AVI (or send it to me off-list). Michael ________________________________________________________________ On Feb 22, 2012, at 20:31, Bill Mohler wrote: > I just used ImageJ to output a 36GB AVI file: 8000X8000 pixels X 191 frames, uncompressed. > > If I use the File>Import>AVI... command to open it, the first dialog correctly offers the range 1-191 slices. However, when I open that range as a virtual stack, I get only the first 9 frames displayed in the resulting stack window (conspicuously ~2.2GB of data). They're at full resolution, but the slider won't take me past #9, and the subtitle says 1/9. > > Any notions as to why this is? I thought that recent changes to the AVI reader had it opening larger AVIs. > > Thanks, > Bill |
In reply to this post by Bill Mohler
Hi Bill,
the AVI format (a.k.a. AVI 1.0) was originally designed for files smaller than 2 GB, with 32-bit integer offsets in the index to the frames. Later on, so-called AVI-2.0 (OpenDML AVI) was devised for longer files. It consists of an initial AVI 1.0 file, up to 1GB in size, and some additional structures for pointing to extra data blocks at the end of the AVI 1.0. The AVI writer of ImageJ does not implement any of these of AVI 2.0 structures. When reading data into memory (no virtual stack), starting from the first frame, ImageJ ignores the index, so it should have no problem reading its own AVI files even if they are arbitrarily large, as long as it has enough memory. But I fear that this is not an option for a 36 GB file. As a workaround, with uncompressed frames, for the moment you may use File>Import Raw. First open the AVI (first two frames are enough) in debug mode to get the offsets and ends of the frames; and convert them from hex to decimal. Then, in the 'Import Raw' dialog, enter the start of the first frame as 'offset' (typically 4116) and the difference between the end of the first and start of the 2nd frame minus 1 as 'gap between frames' (usually this will be 8). File>Import Raw also supports virtual stacks. Michael ________________________________________________________________ On Feb 22, 2012, at 20:31, Bill Mohler wrote: > I just used ImageJ to output a 36GB AVI file: 8000X8000 pixels X 191 frames, uncompressed. > > If I use the File>Import>AVI... command to open it, the first dialog correctly offers the range 1-191 slices. However, when I open that range as a virtual stack, I get only the first 9 frames displayed in the resulting stack window (conspicuously ~2.2GB of data). They're at full resolution, but the slider won't take me past #9, and the subtitle says 1/9. > > Any notions as to why this is? I thought that recent changes to the AVI reader had it opening larger AVIs. > > Thanks, > Bill |
Hi,
On Thu, 23 Feb 2012, Michael Schmid wrote: > the AVI format (a.k.a. AVI 1.0) was originally designed for files > smaller than 2 GB, with 32-bit integer offsets in the index to the > frames. Later on, so-called AVI-2.0 (OpenDML AVI) was devised for > longer files. It consists of an initial AVI 1.0 file, up to 1GB in > size, and some additional structures for pointing to extra data blocks > at the end of the AVI 1.0. > > The AVI writer of ImageJ does not implement any of these of AVI 2.0 > structures. My understanding of ImageJ was that it was meant to be neat and small without overboarding "featuritis". As such, more complicated functionality was meant to be implemented by external plugins. Plugins such as Bio-Formats. As it happens, reading http://loci.wisc.edu/bio-formats-format/avi-audio-video-interleave I cannot help but note the comment: We'd like to have: More AVI datasets, including files with audio tracks and/or multiple video tracks files compressed with a common, unsupported codec 2+ GB files Rather than trying to push in changes to the AVI writer and AVI reader in ImageJ (which would likely result in side effects breaking something else in the process), would it not make way more sense to work on getting Bio-Formats to do what you want to do? Chances are that it already does, anyway. There are several very dedicated people working on it, people whose expertise are file formats. They are really good at it, I speak from experience. As a bonus, if Bio-Formats does the job for you, a wide range of applications including CellProfiler, Matlab, XuvTools, Va3D (TAFKA V3D), Imaris and many more support exactly what you need to read and write your data. I guess what I want to say is: let the tool that is suited best for the task perform it. To a hammer, everything looks like a nail. (In this case, ImageJ1 being the hammer and your >2GB data the egg.) Ciao, Johannes |
Hi Johannes,
well, I don't want to start a religious war, but as file sizes are increasing, I think that it becomes increasingly important to support larger files. The overhead for writing AVI 2.0 when the file size exceeds some limit (1 GB as in the standard, or 2 GB as VirtualDub does it) would be rather small. People outside biology (like me; I am a physicist) may not need Bio Formats or most of the plugins in Fiji, but one should expect that a file type supported by the ImageJ is written correctly. I don't see why writing AVI 2 for large files would break anything. Writing an AVI 1 format beyond 2 GB (with inconsistent 32-bit index) obviously *does* cause problems. Anyhow, don't expect an AVI Writer with AVI 2.0 from me soon; as I said, I have currently no time for this. Michael ________________________________________________________________ On Feb 23, 2012, at 10:48, Johannes Schindelin wrote: > Hi, > > On Thu, 23 Feb 2012, Michael Schmid wrote: > >> the AVI format (a.k.a. AVI 1.0) was originally designed for files >> smaller than 2 GB, with 32-bit integer offsets in the index to the >> frames. Later on, so-called AVI-2.0 (OpenDML AVI) was devised for >> longer files. It consists of an initial AVI 1.0 file, up to 1GB in >> size, and some additional structures for pointing to extra data blocks >> at the end of the AVI 1.0. >> >> The AVI writer of ImageJ does not implement any of these of AVI 2.0 >> structures. > > My understanding of ImageJ was that it was meant to be neat and small > without overboarding "featuritis". As such, more complicated functionality > was meant to be implemented by external plugins. > > Plugins such as Bio-Formats. > > As it happens, reading > http://loci.wisc.edu/bio-formats-format/avi-audio-video-interleave I > cannot help but note the comment: > > We'd like to have: > More AVI datasets, including > > files with audio tracks and/or multiple video tracks > files compressed with a common, unsupported codec > 2+ GB files > > Rather than trying to push in changes to the AVI writer and AVI reader in > ImageJ (which would likely result in side effects breaking something else > in the process), would it not make way more sense to work on getting > Bio-Formats to do what you want to do? > > Chances are that it already does, anyway. There are several very dedicated > people working on it, people whose expertise are file formats. They are > really good at it, I speak from experience. > > As a bonus, if Bio-Formats does the job for you, a wide range of > applications including CellProfiler, Matlab, XuvTools, Va3D (TAFKA V3D), > Imaris and many more support exactly what you need to read and write your > data. > > I guess what I want to say is: let the tool that is suited best for the > task perform it. To a hammer, everything looks like a nail. (In this case, > ImageJ1 being the hammer and your >2GB data the egg.) > > Ciao, > Johannes |
In reply to this post by Bill Mohler
Hi,
[it seems that the mailing list recently acquired the bad habit of defaulting to reply-to-author-only. This is even worse than the defaulting to reply-to-list-and-leave-out-everybody-who-might-not-be-subscribed setting. Hopefully somebody fixes that in the near future, I almost sent this mail to Michael only by mistake.] On Thu, 23 Feb 2012, Michael Schmid wrote: > well, I don't want to start a religious war, I do not know what is religious about trying to determine the best tools for a job at hand. > but as file sizes are increasing, I think that it becomes increasingly > important to support larger files. The overhead for writing AVI 2.0 when > the file size exceeds some limit (1 GB as in the standard, or 2 GB as > VirtualDub does it) would be rather small. I agree that it is important to support larger files. I just do not agree to cram in some super-special functionality into ImageJ1 when more often then not, such changes have introduced bugs affecting more general workflows, while other pretty generic functionality does not make it into ImageJ1 with the argument that ImageJ1 is supposed to be small ("you can write a plugin if you want that functionality"). > People outside biology (like me; I am a physicist) may not need Bio > Formats [...] Color me confused. On the one hand you want support for large files, but on the other hand you refuse to use the tool that does support them right now, and supports them well (I say that because I have tried it). > or most of the plugins in Fiji, Was I talking about Fiji? Sorry, my mistake. I wanted to talk about Bio-Formats because it is Bio-Formats that has first-rate support for tons of file formats. > but one should expect that a file type supported by the ImageJ is > written correctly. When asked, I would agree that ImageJ should not have support for all kinds of obscure or rarely used file formats. After all, certain file formats -- such as .tif or .avi -- are mere containers that can contain data encoded using any number of both free and proprietary codecs and would therefore be very hard to support completely. Support for .avi files, for example, would never be complete without an implementation of the DIVX codec. (And no, I do not think that Bio-Formats supports that codec yet, but if we were ever to implement that codec, Bio-Formats would be the correct place to add it, not ImageJ.) > I don't see why writing AVI 2 for large files would break anything. It seems that the old code (in which I did not see anything that could break) managed not to issue an error when writing invalid data. Is it so unrealistic to expect that the code writing AVI 2 could introduce similar bugs? > Writing an AVI 1 format beyond 2 GB (with inconsistent 32-bit index) > obviously *does* cause problems. > > Anyhow, don't expect an AVI Writer with AVI 2.0 from me soon; as I said, > I have currently no time for this. I am sorry if I left the impression that I expected that from you. To the contrary, I wanted to suggest that this is a perfect example of the concept of "separation of concerns": just like Bio-Formats should not try to implement an image processing user interface (ImageJ does that already), ImageJ should not try to implement every corner-case in AVI writing or reading. And it seems that you contribute exclusively to ImageJ1. I agree, though, that the bug should be fixed where the AVI writer happily writes invalid files without telling the user that it did. I hope I managed to express myself clearer this time, Johannes |
Hi Michael, Johannes et. al,
> People outside biology (like me; I am a physicist) may not need Bio > > Formats [...] > > Color me confused. On the one hand you want support for large files, but > on the other hand you refuse to use the tool that does support them right > now, and supports them well (I say that because I have tried it). > The name "Bio-Formats" is increasingly a misnomer. Folks like Michael who, as he points out, do not work in biology, may think that Bio-Formats is inapplicable to their work simply because of the name. This is unfortunate, because the library has grown beyond only microscopy, biology and life sciences. If you look at the list of supported file formats, there are many general-purpose ones, as well as some formats from astronomy, earth sciences, medical imaging and other disciplines. As such, we are undertaking a refactoring of Bio-Formats, to split it into two parts. "SCIFIO" (the SCientific Image Format Input/Output library) is BSD-licensed and houses the core infrastructure of the library as well as support for common graphics formats, while "Bio-Formats" is a GPL-licensed collection of microscopy format readers that function as plugins to SCIFIO. This restructuring has the added benefit of allowing us to distribute SCIFIO with ImageJ2 for its core I/O capabilities. We hope this reorganization and rebranding will help people in other scientific disciplines to understand that SCIFIO is a tool for more than just life sciences. I agree, though, that the bug should be fixed where the AVI writer happily > writes invalid files without telling the user that it did. > Yes, the fact is that ImageJ1 currently does offer out-of-the-box support for writing AVI files, so I think some change should be made to avoid writing out malformed large AVIs. My vote would be for it to simply issue an error message saying the image is too large to write as AVI, and suggest using another plugin such as Bio-Formats instead. This would be an easy change to the code, without taking much of anyone's time, and would avoid people getting burned by the problem in the future. Regards, Curtis On Thu, Feb 23, 2012 at 5:48 AM, Johannes Schindelin < [hidden email]> wrote: > Hi, > > [it seems that the mailing list recently acquired the bad habit of > defaulting to reply-to-author-only. This is even worse than the defaulting > to reply-to-list-and-leave-out-everybody-who-might-not-be-subscribed > setting. Hopefully somebody fixes that in the near future, I almost sent > this mail to Michael only by mistake.] > > On Thu, 23 Feb 2012, Michael Schmid wrote: > > > well, I don't want to start a religious war, > > I do not know what is religious about trying to determine the best tools > for a job at hand. > > > but as file sizes are increasing, I think that it becomes increasingly > > important to support larger files. The overhead for writing AVI 2.0 when > > the file size exceeds some limit (1 GB as in the standard, or 2 GB as > > VirtualDub does it) would be rather small. > > I agree that it is important to support larger files. I just do not agree > to cram in some super-special functionality into ImageJ1 when more often > then not, such changes have introduced bugs affecting more general > workflows, while other pretty generic functionality does not make it into > ImageJ1 with the argument that ImageJ1 is supposed to be small ("you can > write a plugin if you want that functionality"). > > > People outside biology (like me; I am a physicist) may not need Bio > > Formats [...] > > Color me confused. On the one hand you want support for large files, but > on the other hand you refuse to use the tool that does support them right > now, and supports them well (I say that because I have tried it). > > > or most of the plugins in Fiji, > > Was I talking about Fiji? Sorry, my mistake. I wanted to talk about > Bio-Formats because it is Bio-Formats that has first-rate support for tons > of file formats. > > > but one should expect that a file type supported by the ImageJ is > > written correctly. > > When asked, I would agree that ImageJ should not have support for all > kinds of obscure or rarely used file formats. After all, certain file > formats -- such as .tif or .avi -- are mere containers that can contain > data encoded using any number of both free and proprietary codecs and > would therefore be very hard to support completely. Support for .avi > files, for example, would never be complete without an implementation of > the DIVX codec. (And no, I do not think that Bio-Formats supports that > codec yet, but if we were ever to implement that codec, Bio-Formats would > be the correct place to add it, not ImageJ.) > > > I don't see why writing AVI 2 for large files would break anything. > > It seems that the old code (in which I did not see anything that could > break) managed not to issue an error when writing invalid data. Is it so > unrealistic to expect that the code writing AVI 2 could introduce similar > bugs? > > > Writing an AVI 1 format beyond 2 GB (with inconsistent 32-bit index) > > obviously *does* cause problems. > > > > Anyhow, don't expect an AVI Writer with AVI 2.0 from me soon; as I said, > > I have currently no time for this. > > I am sorry if I left the impression that I expected that from you. > > To the contrary, I wanted to suggest that this is a perfect example of the > concept of "separation of concerns": just like Bio-Formats should not try > to implement an image processing user interface (ImageJ does that > already), ImageJ should not try to implement every corner-case in AVI > writing or reading. And it seems that you contribute exclusively to > ImageJ1. > > I agree, though, that the bug should be fixed where the AVI writer happily > writes invalid files without telling the user that it did. > > I hope I managed to express myself clearer this time, > Johannes > |
Free forum by Nabble | Edit this page |