AVI, Virtual Stack, >1GB boundery?

[JesperSP]

Hi everyone,

I'm trying to analyze some very large AVI movies in ImageJ. I thought I had everything covered when I found the "open as virtual stack" function. However, when I'm trying to open a 1.3GB AVI containing 15495 frames of 348x256 it stops exactly at frame 11934 (348x256*11934 = 1063176192 bytes corresponding to just below 1GB). I have tried to tell the plugin that I only want to open frames 11900-15495, both as virtual and non-virtual stack and in this case it only opens frames 11900-11934. I managed to save all the frames as TIFs in a folder and then I was able to open all a virtual stack with all frames using the "Import"->"TIFF virtual stack..." function. I then saved the stack as AVI with PNG compression which resulted in a file of 552 MB. I was able to load the resulting AVI file without any problems in ImageJ, both as virtual and non-virtual stack.

My conclusion is that ImageJ (or Java in general?) cannot access positions beyond 1GB of files (under windows XP 32Bit).

I'm going to be analyzing a lot of these very large files (possibly up to 14GB) in the future, so I would very much like to find a different workaround than saving all images as individual tifs or compressing movies. Is there a way to read AVI's larger than 1GB in ImageJ? If it's imposible under windows XP 32bit, is it then possible if I install a 64bit operating system (e.g. UBUNTU 64bit?).

Suggestions are very welcome ....

Cheers,
Jesper

[Michael Schmid]

Hi Jesper,

ImageJ only supports AVI 1.0, which is limited to 2 GB.
AVI 2.0 is not supported, but it is possible that the first RIFF list  
of an AVI 2 is read - it has the AVI 1 format.

The AVI 2 specifications says that the first (AVI 1 compatible) RIFF  
list of an AVI 2 should be no more than 1 GB in size. Some AVI  
writers ignore this and write up to 2 GB (which is not problem for  
AVI 1 readers), others adhere to the specs.

So I suspect that you have an AVI 2.0 movie, and only the beginning  
is read.

Java does support long (64-bit) pointers in files, and if I remember  
it correctly the AVI reader uses long pointers, so it could address  
essentially arbitrary files sizes. The pointers and field lengths in  
the AVI 1 format have 32 bits, however. That's what causes the 2 GB  
limit of AVI 1.

Michael
________________________________________________________________

On 26 Nov 2008, at 16:00, JesperSP wrote:

> Hi everyone,
>
> I'm trying to analyze some very large AVI movies in ImageJ. I  
> thought I had
> everything covered when I found the "open as virtual stack" function.
> However, when I'm trying to open a 1.3GB AVI containing 15495  
> frames of
> 348x256 it stops exactly at frame 11934 (348x256*11934 = 1063176192  
> bytes
> corresponding to just below 1GB). I have tried to tell the plugin  
> that I
> only want to open frames 11900-15495, both as virtual and non-
> virtual stack
> and in this case it only opens frames 11900-11934. I managed to  
> save all the
> frames as TIFs in a folder and then I was able to open all a  
> virtual stack
> with all frames using the "Import"->"TIFF virtual stack..."  
> function. I then
> saved the stack as AVI with PNG compression which resulted in a  
> file of 552
> MB. I was able to load the resulting AVI file without any problems in
> ImageJ, both as virtual and non-virtual stack.
>
> My conclusion is that ImageJ (or Java in general?) cannot access  
> positions
> beyond 1GB of files (under windows XP 32Bit).
>
> I'm going to be analyzing a lot of these very large files (possibly  
> up to
> 14GB) in the future, so I would very much like to find a different
> workaround than saving all images as individual tifs or compressing  
> movies.
> Is there a way to read AVI's larger than 1GB in ImageJ? If it's  
> imposible
> under windows XP 32bit, is it then possible if I install a 64bit  
> operating
> system (e.g. UBUNTU 64bit?).
>
> Suggestions are very welcome ....
>
> Cheers,
> Jesper
>

[Jeffrey B. Woodward]

Not sure if it will help, but I wrote a Virtual Quicktime Stack plugin
for ImageJ that I should be able to share. In theory, if you can view
the AVI in question with Quicktime, then the plugin should also work
within ImageJ; however, I have not tested the plugin with AVI files (I
wrote it in order to deal with large mpeg files). I found the
documentation for the Java Quicktime bindings to be a bit frustrating,
but I was eventually able to work through those issues. The most
important note was on getting the Quicktime Java bindings installed in
the first place....the key to that was making sure that the Java JDK is
installed *prior* to installing Quicktime (uninstall and reinstall
Quicktime if necessary...the newer versions of the Quicktime installer
don't prompt to install the Java bindings...it just seems to do it
automatically if it can find a JDK).

In summary, contact me if you want to give a Virtual Quicktime Stack
plugin a shot.

-Woody


Michael Schmid wrote:

> Hi Jesper,
>
> ImageJ only supports AVI 1.0, which is limited to 2 GB.
> AVI 2.0 is not supported, but it is possible that the first RIFF list
> of an AVI 2 is read - it has the AVI 1 format.
>
> The AVI 2 specifications says that the first (AVI 1 compatible) RIFF
> list of an AVI 2 should be no more than 1 GB in size. Some AVI writers
> ignore this and write up to 2 GB (which is not problem for AVI 1
> readers), others adhere to the specs.
>
> So I suspect that you have an AVI 2.0 movie, and only the beginning is
> read.
>
> Java does support long (64-bit) pointers in files, and if I remember
> it correctly the AVI reader uses long pointers, so it could address
> essentially arbitrary files sizes. The pointers and field lengths in
> the AVI 1 format have 32 bits, however. That's what causes the 2 GB
> limit of AVI 1.
>
> Michael
> ________________________________________________________________
>
> On 26 Nov 2008, at 16:00, JesperSP wrote:
>
>> Hi everyone,
>>
>> I'm trying to analyze some very large AVI movies in ImageJ. I thought
>> I had
>> everything covered when I found the "open as virtual stack" function.
>> However, when I'm trying to open a 1.3GB AVI containing 15495 frames of
>> 348x256 it stops exactly at frame 11934 (348x256*11934 = 1063176192
>> bytes
>> corresponding to just below 1GB). I have tried to tell the plugin that I
>> only want to open frames 11900-15495, both as virtual and non-virtual
>> stack
>> and in this case it only opens frames 11900-11934. I managed to save
>> all the
>> frames as TIFs in a folder and then I was able to open all a virtual
>> stack
>> with all frames using the "Import"->"TIFF virtual stack..." function.
>> I then
>> saved the stack as AVI with PNG compression which resulted in a file
>> of 552
>> MB. I was able to load the resulting AVI file without any problems in
>> ImageJ, both as virtual and non-virtual stack.
>>
>> My conclusion is that ImageJ (or Java in general?) cannot access
>> positions
>> beyond 1GB of files (under windows XP 32Bit).
>>
>> I'm going to be analyzing a lot of these very large files (possibly
>> up to
>> 14GB) in the future, so I would very much like to find a different
>> workaround than saving all images as individual tifs or compressing
>> movies.
>> Is there a way to read AVI's larger than 1GB in ImageJ? If it's
>> imposible
>> under windows XP 32bit, is it then possible if I install a 64bit
>> operating
>> system (e.g. UBUNTU 64bit?).
>>
>> Suggestions are very welcome ....
>>
>> Cheers,
>> Jesper
>>
>
>

[Burger Wilhelm]

Jesper,

if you can afford to process the video frame by frame (i.e., without
loading the entire video at once), I recommend to look at my MovieIO
package, to be found here:

http://staff.fh-hagenberg.at/burger/imagej/MovieIO.html

While processing the video, it only maintains a single ImageProcessor
and thus has a very small memory footprint. Examples demonstrate its
simple API, it works EITHER with JMF OR Quicktime. I have used it to
process long and large (even HD) videos. Hope this helps.

Regards,
Wilhelm

> -----Original Message-----
> From: ImageJ Interest Group [mailto:[hidden email]] On
> Behalf Of JesperSP
> Sent: Wednesday, November 26, 2008 4:01 PM
> To: [hidden email]
> Subject: AVI, Virtual Stack, >1GB boundery?
>
> Hi everyone,
>
> I'm trying to analyze some very large AVI movies in ImageJ. I
> thought I had
> everything covered when I found the "open as virtual stack" function.
> However, when I'm trying to open a 1.3GB AVI containing 15495
> frames of
> 348x256 it stops exactly at frame 11934 (348x256*11934 =
> 1063176192 bytes
> corresponding to just below 1GB). I have tried to tell the
> plugin that I
> only want to open frames 11900-15495, both as virtual and
> non-virtual stack
> and in this case it only opens frames 11900-11934. I managed
> to save all the
> frames as TIFs in a folder and then I was able to open all a
> virtual stack
> with all frames using the "Import"->"TIFF virtual stack..."
> function. I then
> saved the stack as AVI with PNG compression which resulted in
> a file of 552
> MB. I was able to load the resulting AVI file without any problems in
> ImageJ, both as virtual and non-virtual stack.
>
> My conclusion is that ImageJ (or Java in general?) cannot
> access positions
> beyond 1GB of files (under windows XP 32Bit).
>
> I'm going to be analyzing a lot of these very large files
> (possibly up to
> 14GB) in the future, so I would very much like to find a different
> workaround than saving all images as individual tifs or
> compressing movies.
> Is there a way to read AVI's larger than 1GB in ImageJ? If
> it's imposible
> under windows XP 32bit, is it then possible if I install a
> 64bit operating
> system (e.g. UBUNTU 64bit?).
>
> Suggestions are very welcome ....
>
> Cheers,
> Jesper
> --
> View this message in context:
> http://n2.nabble.com/AVI%2C-Virtual-Stack%2C-%3E1GB-boundery--
> tp1581053p1581053.html
> Sent from the ImageJ mailing list archive at Nabble.com.
>

[JesperSP]

The file I'm analyzing is actually an LVI file (I just renamed it to AVI, so windows media player would play it). It is generated by a Lumenera infinity capture camera. This is probably why my attempts to find an AVI splitter that could split it into smaller files failed. But ImageJ appears to read the LVI->AVI renamed file just fine up till the 1GB file limit.

If I don't rename the file to AVI, ImageJ will try to open the LVI file with the LOCI plugins that I've installed. Then it only gets to 4096 frames!

Maybe I should try and write a special reader for the LVI files myself.

Cheers,
Jesper

Burger Wilhelm wrote

Jesper,

if you can afford to process the video frame by frame (i.e., without
loading the entire video at once), I recommend to look at my MovieIO
package, to be found here:

http://staff.fh-hagenberg.at/burger/imagej/MovieIO.html

While processing the video, it only maintains a single ImageProcessor
and thus has a very small memory footprint. Examples demonstrate its
simple API, it works EITHER with JMF OR Quicktime. I have used it to
process long and large (even HD) videos. Hope this helps.

Regards,
Wilhelm



> -----Original Message-----
> From: ImageJ Interest Group [mailto:[e-mail]] On
> Behalf Of JesperSP
> Sent: Wednesday, November 26, 2008 4:01 PM
> To: [e-mail]
> Subject: AVI, Virtual Stack, >1GB boundery?
>
> Hi everyone,
>
> I'm trying to analyze some very large AVI movies in ImageJ. I
> thought I had
> everything covered when I found the "open as virtual stack" function.
> However, when I'm trying to open a 1.3GB AVI containing 15495
> frames of
> 348x256 it stops exactly at frame 11934 (348x256*11934 =
> 1063176192 bytes
> corresponding to just below 1GB). I have tried to tell the
> plugin that I
> only want to open frames 11900-15495, both as virtual and
> non-virtual stack
> and in this case it only opens frames 11900-11934. I managed
> to save all the
> frames as TIFs in a folder and then I was able to open all a
> virtual stack
> with all frames using the "Import"->"TIFF virtual stack..."
> function. I then
> saved the stack as AVI with PNG compression which resulted in
> a file of 552
> MB. I was able to load the resulting AVI file without any problems in
> ImageJ, both as virtual and non-virtual stack.
>
> My conclusion is that ImageJ (or Java in general?) cannot
> access positions
> beyond 1GB of files (under windows XP 32Bit).
>
> I'm going to be analyzing a lot of these very large files
> (possibly up to
> 14GB) in the future, so I would very much like to find a different
> workaround than saving all images as individual tifs or
> compressing movies.
> Is there a way to read AVI's larger than 1GB in ImageJ? If
> it's imposible
> under windows XP 32bit, is it then possible if I install a
> 64bit operating
> system (e.g. UBUNTU 64bit?).
>
> Suggestions are very welcome ....
>
> Cheers,
> Jesper
> --
> View this message in context:
> http://n2.nabble.com/AVI%2C-Virtual-Stack%2C-%3E1GB-boundery--
> tp1581053p1581053.html
> Sent from the ImageJ mailing list archive at Nabble.com.
>

[Jeffrey B. Woodward]

Wayne Rasband wrote:

> On Nov 26, 2008, at 10:50 AM, Jeffrey B. Woodward wrote:
>
>> Not sure if it will help, but I wrote a Virtual Quicktime Stack
>> plugin for ImageJ that I should be able to share. In theory, if you
>> can view the AVI in question with Quicktime, then the plugin should
>> also work within ImageJ; however, I have not tested the plugin with
>> AVI files (I wrote it in order to deal with large mpeg files). I
>> found the documentation for the Java Quicktime bindings to be a bit
>> frustrating, but I was eventually able to work through those issues.
>> The most important note was on getting the Quicktime Java bindings
>> installed in the first place....the key to that was making sure that
>> the Java JDK is installed *prior* to installing Quicktime (uninstall
>> and reinstall Quicktime if necessary...the newer versions of the
>> Quicktime installer don't prompt to install the Java bindings...it
>> just seems to do it automatically if it can find a JDK).
>>
>> In summary, contact me if you want to give a Virtual Quicktime Stack
>> plugin a shot.
>
> Hi Woody,
>
> I would like to check out your Virtual Quicktime Stack plugin. I
> sounds like it could very useful. I can test it with AVI files.
>
> Thanks,
>
> -wayne
>
>

If I remember correctly, the maillist doesn't allow attachments. I have
extracted the QTVirtualStack components from my project (and changed the
name of the java package), and I put together a quick ant build script.
I will host it at the following URL "for a while":
http://www.cs.dartmouth.edu/~jnw/QTVirtualStack.zip

Wayne, if you'd like to clean it up and incorporate it into an official
build of ImageJ that would be cool; otherwise, feedback/comments are
equally useful.

Thanks,

-Woody

[Bill Mohler]

Woody-

Thanks for sharing this.
For us non-developers, how would we install this as an executable
plugin?  Do we need to do a build to create the .class file that we
can run?  Is this in a README in the package?

Thanks,
Bill

>Wayne Rasband wrote:
>>On Nov 26, 2008, at 10:50 AM, Jeffrey B. Woodward wrote:
>>
>>>Not sure if it will help, but I wrote a Virtual Quicktime Stack
>>>plugin for ImageJ that I should be able to share. In theory, if
>>>you can view the AVI in question with Quicktime, then the plugin
>>>should also work within ImageJ; however, I have not tested the
>>>plugin with AVI files (I wrote it in order to deal with large mpeg
>>>files). I found the documentation for the Java Quicktime bindings
>>>to be a bit frustrating, but I was eventually able to work through
>>>those issues. The most important note was on getting the Quicktime
>>>Java bindings installed in the first place....the key to that was
>>>making sure that the Java JDK is installed *prior* to installing
>>>Quicktime (uninstall and reinstall Quicktime if necessary...the
>>>newer versions of the Quicktime installer don't prompt to install
>>>the Java bindings...it just seems to do it automatically if it can
>>>find a JDK).
>>>
>>>In summary, contact me if you want to give a Virtual Quicktime
>>>Stack plugin a shot.
>>
>>Hi Woody,
>>
>>I would like to check out your Virtual Quicktime Stack plugin. I
>>sounds like it could very useful. I can test it with AVI files.
>>
>>Thanks,
>>
>>-wayne
>>
>
>If I remember correctly, the maillist doesn't allow attachments. I have
>extracted the QTVirtualStack components from my project (and changed the
>name of the java package), and I put together a quick ant build script.
>I will host it at the following URL "for a while":
>http://www.cs.dartmouth.edu/~jnw/QTVirtualStack.zip
>
>Wayne, if you'd like to clean it up and incorporate it into an official
>build of ImageJ that would be cool; otherwise, feedback/comments are
>equally useful.
>
>Thanks,
>
>-Woody

[ctrueden]

Hi Jesper,

If you would be willing to send us a sample dataset, we can investigate
improving the support in Bio-Formats, as our time allows. Let me know if
interested and I'll send you our FTP server information off-list.

-Curtis

On Wed, Nov 26, 2008 at 12:01 PM, JesperSP <[hidden email]> wrote:

> The file I'm analyzing is actually an LVI file (I just renamed it to AVI,
> so
> windows media player would play it). It is generated by a Lumenera infinity
> capture camera. This is probably why my attempts to find an AVI splitter
> that could split it into smaller files failed. But ImageJ appears to read
> the LVI->AVI renamed file just fine up till the 1GB file limit.
>
> If I don't rename the file to AVI, ImageJ will try to open the LVI file
> with
> the LOCI plugins that I've installed. Then it only gets to 4096 frames!
>
> Maybe I should try and write a special reader for the LVI files myself.
>
> Cheers,
> Jesper
>
>
>
> Burger Wilhelm wrote:
> >
> > Jesper,
> >
> > if you can afford to process the video frame by frame (i.e., without
> > loading the entire video at once), I recommend to look at my MovieIO
> > package, to be found here:
> >
> > http://staff.fh-hagenberg.at/burger/imagej/MovieIO.html
> >
> > While processing the video, it only maintains a single ImageProcessor
> > and thus has a very small memory footprint. Examples demonstrate its
> > simple API, it works EITHER with JMF OR Quicktime. I have used it to
> > process long and large (even HD) videos. Hope this helps.
> >
> > Regards,
> > Wilhelm
> >
> >
> >
> >> -----Original Message-----
> >> From: ImageJ Interest Group [mailto:[hidden email]] On
> >> Behalf Of JesperSP
> >> Sent: Wednesday, November 26, 2008 4:01 PM
> >> To: [hidden email]
> >> Subject: AVI, Virtual Stack, >1GB boundery?
> >>
> >> Hi everyone,
> >>
> >> I'm trying to analyze some very large AVI movies in ImageJ. I
> >> thought I had
> >> everything covered when I found the "open as virtual stack" function.
> >> However, when I'm trying to open a 1.3GB AVI containing 15495
> >> frames of
> >> 348x256 it stops exactly at frame 11934 (348x256*11934 =
> >> 1063176192 bytes
> >> corresponding to just below 1GB). I have tried to tell the
> >> plugin that I
> >> only want to open frames 11900-15495, both as virtual and
> >> non-virtual stack
> >> and in this case it only opens frames 11900-11934. I managed
> >> to save all the
> >> frames as TIFs in a folder and then I was able to open all a
> >> virtual stack
> >> with all frames using the "Import"->"TIFF virtual stack..."
> >> function. I then
> >> saved the stack as AVI with PNG compression which resulted in
> >> a file of 552
> >> MB. I was able to load the resulting AVI file without any problems in
> >> ImageJ, both as virtual and non-virtual stack.
> >>
> >> My conclusion is that ImageJ (or Java in general?) cannot
> >> access positions
> >> beyond 1GB of files (under windows XP 32Bit).
> >>
> >> I'm going to be analyzing a lot of these very large files
> >> (possibly up to
> >> 14GB) in the future, so I would very much like to find a different
> >> workaround than saving all images as individual tifs or
> >> compressing movies.
> >> Is there a way to read AVI's larger than 1GB in ImageJ? If
> >> it's imposible
> >> under windows XP 32bit, is it then possible if I install a
> >> 64bit operating
> >> system (e.g. UBUNTU 64bit?).
> >>
> >> Suggestions are very welcome ....
> >>
> >> Cheers,
> >> Jesper
> >> --
> >> View this message in context:
> >> http://n2.nabble.com/AVI%2C-Virtual-Stack%2C-%3E1GB-boundery--
> >> tp1581053p1581053.html
> >> Sent from the ImageJ mailing list archive at Nabble.com.
> >>
> >
> >
>
> --
> View this message in context:
> http://n2.nabble.com/AVI%2C-Virtual-Stack%2C-%3E1GB-boundery--tp1581053p1581915.html
> Sent from the ImageJ mailing list archive at Nabble.com.
>

[Jeffrey B. Woodward]

Bill,

Sorry for the lack of "non-developer" instructions. Try the following:

1) Unzip the QTVirtualStack.zip file to some temporary location (or copy
out just the single file that is needed: QTVirtualStack_.jar)
2) Copy/Move QTVirtualStack_.jar to {ImageJ_Root}/plugins/jars.
3) Start Up ImageJ...
4) There should be a new option under File->Import called "QT Virtual
Stack..." -- select it...
5) A file explorer should have opened...navigator to your movie file and
open it.

In short, the source code and build script is there; however, I did
include the "binary" version as well for what should be a drop-in and go
experience.

Developer instructions for rebuilding it from source are pretty
straightforward: "ant jar" should do the trick.

If things go wrong, it is likely because the Quicktime for Java bindings
aren't installed on your system...see my notes in the original post for
that.

Sorry that this isn't bundled up as a "real" ImageJ plugin. If someone
wants to take on that challenge, by all means...I am not one of those
developers that gets offended by other developers cleaning-up/enhancing
my code where I was too lazy to do so myself ;-)  If there is demand for
it being packaged as an ImageJ plugin, and nobody else will take that
on, then I could probably be talked out of my laziness...

-Woody


Bill Mohler wrote:

> Woody-
>
> Thanks for sharing this.
> For us non-developers, how would we install this as an executable
> plugin?  Do we need to do a build to create the .class file that we
> can run?  Is this in a README in the package?
>
> Thanks,
> Bill
>
>> Wayne Rasband wrote:
>>> On Nov 26, 2008, at 10:50 AM, Jeffrey B. Woodward wrote:
>>>
>>>> Not sure if it will help, but I wrote a Virtual Quicktime Stack
>>>> plugin for ImageJ that I should be able to share. In theory, if you
>>>> can view the AVI in question with Quicktime, then the plugin should
>>>> also work within ImageJ; however, I have not tested the plugin with
>>>> AVI files (I wrote it in order to deal with large mpeg files). I
>>>> found the documentation for the Java Quicktime bindings to be a bit
>>>> frustrating, but I was eventually able to work through those
>>>> issues. The most important note was on getting the Quicktime Java
>>>> bindings installed in the first place....the key to that was making
>>>> sure that the Java JDK is installed *prior* to installing Quicktime
>>>> (uninstall and reinstall Quicktime if necessary...the newer
>>>> versions of the Quicktime installer don't prompt to install the
>>>> Java bindings...it just seems to do it automatically if it can find
>>>> a JDK).
>>>>
>>>> In summary, contact me if you want to give a Virtual Quicktime
>>>> Stack plugin a shot.
>>>
>>> Hi Woody,
>>>
>>> I would like to check out your Virtual Quicktime Stack plugin. I
>>> sounds like it could very useful. I can test it with AVI files.
>>>
>>> Thanks,
>>>
>>> -wayne
>>>
>>
>> If I remember correctly, the maillist doesn't allow attachments. I have
>> extracted the QTVirtualStack components from my project (and changed the
>> name of the java package), and I put together a quick ant build script.
>> I will host it at the following URL "for a while":
>> http://www.cs.dartmouth.edu/~jnw/QTVirtualStack.zip
>>
>> Wayne, if you'd like to clean it up and incorporate it into an official
>> build of ImageJ that would be cool; otherwise, feedback/comments are
>> equally useful.
>>
>> Thanks,
>>
>> -Woody
>
>

[Bill Mohler]

Thanks!  Easy enough.  It works great for my large .mov files!

With regard to QTVR object (2-dimensionally navigable .mov files),
which I like, it opens them as linear movies.

To the list:  How doable would it be to treat such a Virtual stack
from QuickTime as a HyperStack (with T and Z navigation) in an imageJ
window?

Bill

>Bill,
>
>Sorry for the lack of "non-developer" instructions. Try the following:
>
>1) Unzip the QTVirtualStack.zip file to some temporary location (or
>copy out just the single file that is needed: QTVirtualStack_.jar)
>2) Copy/Move QTVirtualStack_.jar to {ImageJ_Root}/plugins/jars.
>3) Start Up ImageJ...
>4) There should be a new option under File->Import called "QT
>Virtual Stack..." -- select it...
>5) A file explorer should have opened...navigator to your movie file
>and open it.
>
>In short, the source code and build script is there; however, I did
>include the "binary" version as well for what should be a drop-in
>and go experience.
>
>Developer instructions for rebuilding it from source are pretty
>straightforward: "ant jar" should do the trick.
>
>If things go wrong, it is likely because the Quicktime for Java
>bindings aren't installed on your system...see my notes in the
>original post for that.
>
>Sorry that this isn't bundled up as a "real" ImageJ plugin. If
>someone wants to take on that challenge, by all means...I am not one
>of those developers that gets offended by other developers
>cleaning-up/enhancing my code where I was too lazy to do so myself
>;-)  If there is demand for it being packaged as an ImageJ plugin,
>and nobody else will take that on, then I could probably be talked
>out of my laziness...
>
>-Woody
>
>
>Bill Mohler wrote:
>>Woody-
>>
>>Thanks for sharing this.
>>For us non-developers, how would we install this as an executable
>>plugin?  Do we need to do a build to create the .class file that we
>>can run?  Is this in a README in the package?
>>
>>Thanks,
>>Bill
>>
>>>Wayne Rasband wrote:
>>>>On Nov 26, 2008, at 10:50 AM, Jeffrey B. Woodward wrote:
>>>>
>>>>>Not sure if it will help, but I wrote a Virtual Quicktime Stack
>>>>>plugin for ImageJ that I should be able to share. In theory, if
>>>>>you can view the AVI in question with Quicktime, then the plugin
>>>>>should also work within ImageJ; however, I have not tested the
>>>>>plugin with AVI files (I wrote it in order to deal with large
>>>>>mpeg files). I found the documentation for the Java Quicktime
>>>>>bindings to be a bit frustrating, but I was eventually able to
>>>>>work through those issues. The most important note was on
>>>>>getting the Quicktime Java bindings installed in the first
>>>>>place....the key to that was making sure that the Java JDK is
>>>>>installed *prior* to installing Quicktime (uninstall and
>>>>>reinstall Quicktime if necessary...the newer versions of the
>>>>>Quicktime installer don't prompt to install the Java
>>>>>bindings...it just seems to do it automatically if it can find a
>>>>>JDK).
>>>>>
>>>>>In summary, contact me if you want to give a Virtual Quicktime
>>>>>Stack plugin a shot.
>>>>
>>>>Hi Woody,
>>>>
>>>>I would like to check out your Virtual Quicktime Stack plugin. I
>>>>sounds like it could very useful. I can test it with AVI files.
>>>>
>>>>Thanks,
>>>>
>>>>-wayne
>>>>
>>>
>>>If I remember correctly, the maillist doesn't allow attachments. I have
>>>extracted the QTVirtualStack components from my project (and changed the
>>>name of the java package), and I put together a quick ant build script.
>>>I will host it at the following URL "for a while":
>>>http://www.cs.dartmouth.edu/~jnw/QTVirtualStack.zip
>>>
>>>Wayne, if you'd like to clean it up and incorporate it into an official
>>>build of ImageJ that would be cool; otherwise, feedback/comments are
>>>equally useful.
>>>
>>>Thanks,
>>>
>>>-Woody

[Wayne Rasband]

Thanks to Jeffrey Woodward's amazing QTVirtualStack class, there's a
new version of the QuickTime Opener plugin at

     http://rsb.info.nih.gov/ij/plugins/movie-opener.html

that adds a "Use Virtual Stack" option. Copy the updated
QuickTime_Plugins.jar file to the plugins folder, restart ImageJ, and
you will be able to open QuickTime and AVI files as virtual stacks
using File>Import>Using QuickTime. Files with a ".mov" extension can be
opened using drag and drop. I haven't tried opening an AVI file larger
than 2GB but that is likely to work.

-wayne

On Nov 26, 2008, at 1:31 PM, Jeffrey B. Woodward wrote:

> Wayne Rasband wrote:
>> On Nov 26, 2008, at 10:50 AM, Jeffrey B. Woodward wrote:
>>
>>> Not sure if it will help, but I wrote a Virtual Quicktime Stack
>>> plugin for ImageJ that I should be able to share. In theory, if you
>>> can view the AVI in question with Quicktime, then the plugin should
>>> also work within ImageJ; however, I have not tested the plugin with
>>> AVI files (I wrote it in order to deal with large mpeg files). I
>>> found the documentation for the Java Quicktime bindings to be a bit
>>> frustrating, but I was eventually able to work through those issues.
>>> The most important note was on getting the Quicktime Java bindings
>>> installed in the first place....the key to that was making sure that
>>> the Java JDK is installed *prior* to installing Quicktime (uninstall
>>> and reinstall Quicktime if necessary...the newer versions of the
>>> Quicktime installer don't prompt to install the Java bindings...it
>>> just seems to do it automatically if it can find a JDK).
>>>
>>> In summary, contact me if you want to give a Virtual Quicktime Stack
>>> plugin a shot.
>>
>> Hi Woody,
>>
>> I would like to check out your Virtual Quicktime Stack plugin. I
>> sounds like it could very useful. I can test it with AVI files.
>>
>> Thanks,
>>
>> -wayne
>>
>>
>
> If I remember correctly, the maillist doesn't allow attachments. I have
> extracted the QTVirtualStack components from my project (and changed
> the
> name of the java package), and I put together a quick ant build script.
> I will host it at the following URL "for a while":
> http://www.cs.dartmouth.edu/~jnw/QTVirtualStack.zip
>
> Wayne, if you'd like to clean it up and incorporate it into an official
> build of ImageJ that would be cool; otherwise, feedback/comments are
> equally useful.
>
> Thanks,
>
> -Woody
>

[dscho]

Hi,

On Wed, 26 Nov 2008, Wayne Rasband wrote:

> Thanks to Jeffrey Woodward's amazing QTVirtualStack class, there's a new
> version of the QuickTime Opener plugin at
>
>     http://rsb.info.nih.gov/ij/plugins/movie-opener.html

What is the recommended way to make this work on 64-bit MacOSX (which by
now should be the standard for MacOSX)?

Ciao,
Dscho

[Jeffrey B. Woodward]

Johannes Schindelin wrote:

> Hi,
>
> On Wed, 26 Nov 2008, Wayne Rasband wrote:
>
>  
>> Thanks to Jeffrey Woodward's amazing QTVirtualStack class, there's a new
>> version of the QuickTime Opener plugin at
>>
>>     http://rsb.info.nih.gov/ij/plugins/movie-opener.html
>>    
>
> What is the recommended way to make this work on 64-bit MacOSX (which by
> now should be the standard for MacOSX)?
>
> Ciao,
> Dscho
>
>  

Dscho,

I am no expert on the topic, but I believe the issue is more to do with
Apple's support/commitment (or lack thereof) to QTJ (Quicktime for Java
bindings) than it is with anything in the plugin or ImageJ. Perhaps
there is a more recent reference, but from the looks of the following
(amongst other sites and postings to the same effect that I have seen)
it doesn't look promising:
   
http://developer.apple.com/documentation/Java/Conceptual/Java14Development/01-JavaOverview/JavaOverview.html

In particular, read the "32-bit and 64-bit Java" section of the
aforementioned web page.

If anyone else has better/more-up-to-date information, please share it
with the list.

Best,

-Woody

[JesperSP]

Woah, that was fast... I went on thanksgiving vacation and you've already fixed my problem as I come back. Thanks Wayne and Woody. I found out that the problem with my LVI files was related to the program that writes these files: According to the FourCC tag (RIFF) of the file, the size is <1GB, which is incorrect. Using Bink'n'Smacker i "converted" the file to an uncompressed avi, which the old AVI loader can load without any problems. (SO I WAS WRONG ABOUT THE >1GB LIMITATION ON IMAGEJ /JAVA!)

However, the QT plugin that you have just made can even open the old (corrupted) file with no problems. (Thanks again). I only have one, small bug-report, which is that the "convert to 8bit" doesn't work together with the "virtual stack" option. However, this is not at big issue as it is very fast to convert to 8-bit before analyzing each frame.

Cheers,
Jesper

Wayne Rasband wrote

Thanks to Jeffrey Woodward's amazing QTVirtualStack class, there's a
new version of the QuickTime Opener plugin at

     http://rsb.info.nih.gov/ij/plugins/movie-opener.html

that adds a "Use Virtual Stack" option. Copy the updated
QuickTime_Plugins.jar file to the plugins folder, restart ImageJ, and
you will be able to open QuickTime and AVI files as virtual stacks
using File>Import>Using QuickTime. Files with a ".mov" extension can be
opened using drag and drop. I haven't tried opening an AVI file larger
than 2GB but that is likely to work.

-wayne

On Nov 26, 2008, at 1:31 PM, Jeffrey B. Woodward wrote:

> Wayne Rasband wrote:
>> On Nov 26, 2008, at 10:50 AM, Jeffrey B. Woodward wrote:
>>
>>> Not sure if it will help, but I wrote a Virtual Quicktime Stack
>>> plugin for ImageJ that I should be able to share. In theory, if you
>>> can view the AVI in question with Quicktime, then the plugin should
>>> also work within ImageJ; however, I have not tested the plugin with
>>> AVI files (I wrote it in order to deal with large mpeg files). I
>>> found the documentation for the Java Quicktime bindings to be a bit
>>> frustrating, but I was eventually able to work through those issues.
>>> The most important note was on getting the Quicktime Java bindings
>>> installed in the first place....the key to that was making sure that
>>> the Java JDK is installed *prior* to installing Quicktime (uninstall
>>> and reinstall Quicktime if necessary...the newer versions of the
>>> Quicktime installer don't prompt to install the Java bindings...it
>>> just seems to do it automatically if it can find a JDK).
>>>
>>> In summary, contact me if you want to give a Virtual Quicktime Stack
>>> plugin a shot.
>>
>> Hi Woody,
>>
>> I would like to check out your Virtual Quicktime Stack plugin. I
>> sounds like it could very useful. I can test it with AVI files.
>>
>> Thanks,
>>
>> -wayne
>>
>>
>
> If I remember correctly, the maillist doesn't allow attachments. I have
> extracted the QTVirtualStack components from my project (and changed
> the
> name of the java package), and I put together a quick ant build script.
> I will host it at the following URL "for a while":
> http://www.cs.dartmouth.edu/~jnw/QTVirtualStack.zip
>
> Wayne, if you'd like to clean it up and incorporate it into an official
> build of ImageJ that would be cool; otherwise, feedback/comments are
> equally useful.
>
> Thanks,
>
> -Woody
>

[JesperSP]

Hi Michael,

You're right and I was wrong again . The LVI file that I have appear to follow the specs for a large AVI files (http://www.morgan-multimedia.com/download/odmlff2.pdf), which is perhaps why Windows Media Player has no problem playing it. I have now recorded LVI movies that are above 20GB in size, that I need to analyze. Here I tried Quicktime importing method, both MovieIO (thanks Wilhelm) and the QTVirtualStackOpener/movie-opener(thanks Woody/Wayne). (I haven't been able to get the JMF MovieIO to work.) Unfortunately, quicktime doesn't seem to open my files when they're bigger than 4GB  - even if i try loading them outside ImageJ in the Apple QuickTime Player. So I have read up on the AVI file format, and "hacked the AVI_reader.java so it recognizes the extra RIFF->AVIX segments beyond 1GB - it works great, but it does take a very long time for ImageJ to scan a 20GB file for the pointers to all the frames. If anyone want it I can upload it on my server, but I cannot guarantee that it'll work for all large AVI files. For the simple analysis I want to make, I'm considering making a low-level analysis program in a faster language like Borland Delphi.

/Jesper    

Michael Schmid-3 wrote

Hi Jesper,

ImageJ only supports AVI 1.0, which is limited to 2 GB.
AVI 2.0 is not supported, but it is possible that the first RIFF list  
of an AVI 2 is read - it has the AVI 1 format.

The AVI 2 specifications says that the first (AVI 1 compatible) RIFF  
list of an AVI 2 should be no more than 1 GB in size. Some AVI  
writers ignore this and write up to 2 GB (which is not problem for  
AVI 1 readers), others adhere to the specs.

So I suspect that you have an AVI 2.0 movie, and only the beginning  
is read.

Java does support long (64-bit) pointers in files, and if I remember  
it correctly the AVI reader uses long pointers, so it could address  
essentially arbitrary files sizes. The pointers and field lengths in  
the AVI 1 format have 32 bits, however. That's what causes the 2 GB  
limit of AVI 1.

Michael
________________________________________________________________

On 26 Nov 2008, at 16:00, JesperSP wrote:

> Hi everyone,
>
> I'm trying to analyze some very large AVI movies in ImageJ. I  
> thought I had
> everything covered when I found the "open as virtual stack" function.
> However, when I'm trying to open a 1.3GB AVI containing 15495  
> frames of
> 348x256 it stops exactly at frame 11934 (348x256*11934 = 1063176192  
> bytes
> corresponding to just below 1GB). I have tried to tell the plugin  
> that I
> only want to open frames 11900-15495, both as virtual and non-
> virtual stack
> and in this case it only opens frames 11900-11934. I managed to  
> save all the
> frames as TIFs in a folder and then I was able to open all a  
> virtual stack
> with all frames using the "Import"->"TIFF virtual stack..."  
> function. I then
> saved the stack as AVI with PNG compression which resulted in a  
> file of 552
> MB. I was able to load the resulting AVI file without any problems in
> ImageJ, both as virtual and non-virtual stack.
>
> My conclusion is that ImageJ (or Java in general?) cannot access  
> positions
> beyond 1GB of files (under windows XP 32Bit).
>
> I'm going to be analyzing a lot of these very large files (possibly  
> up to
> 14GB) in the future, so I would very much like to find a different
> workaround than saving all images as individual tifs or compressing  
> movies.
> Is there a way to read AVI's larger than 1GB in ImageJ? If it's  
> imposible
> under windows XP 32bit, is it then possible if I install a 64bit  
> operating
> system (e.g. UBUNTU 64bit?).
>
> Suggestions are very welcome ....
>
> Cheers,
> Jesper
>

[dscho]

Hi,

On Mon, 8 Dec 2008, JesperSP wrote:

> I have read up on the AVI file format, and "hacked the AVI_reader.java
> so it recognizes the extra RIFF->AVIX segments beyond 1GB - it works
> great, but it does take a very long time for ImageJ to scan a 20GB file
> for the pointers to all the frames. If anyone want it I can upload it on
> my server, but I cannot guarantee that it'll work for all large AVI
> files. For the simple analysis I want to make, I'm considering making a
> low-level analysis program in a faster language like Borland Delphi.

I doubt that the speed has anything to do with the speed of the language,
as the problem here is seeking on the disk.

Regardless, if you use Java 1.5 or newer, I sincerely doubt that the speed
difference between Java and Delphi is substantial.  Although I do hear the
myth that Java is slow all the time.

In any case, making your changes public would help a few people, I guess.

Ciao,
Dscho

[JesperSP]

Hi Johannes,

I think bashing my own ability rapidly develop applications in Java, as much as my (apparently) faulty belief that Java is slower than native Win32 code . The plugin that I modified seeks through the 20GB file to locate all frames before it loads the first frame for the virtual stack. I would be better if it simply loaded the first frame, and waited to scan for further frames until these were needed (same as what MovieIO does). I could try to program this in java, but I'm much more familiar with Delphi, so I guess I could make faster progress trying to make the same in Delphi (I was thinking out loud I guess).. Maybe I should take the time familiarize myself with Eclipse (I guess Delphi is being depreciated), so that I can build java application/plugins faster..

Cheers,
Jesper

Johannes Schindelin wrote

Hi,

On Mon, 8 Dec 2008, JesperSP wrote:

> I have read up on the AVI file format, and "hacked the AVI_reader.java
> so it recognizes the extra RIFF->AVIX segments beyond 1GB - it works
> great, but it does take a very long time for ImageJ to scan a 20GB file
> for the pointers to all the frames. If anyone want it I can upload it on
> my server, but I cannot guarantee that it'll work for all large AVI
> files. For the simple analysis I want to make, I'm considering making a
> low-level analysis program in a faster language like Borland Delphi.

I doubt that the speed has anything to do with the speed of the language,
as the problem here is seeking on the disk.

Regardless, if you use Java 1.5 or newer, I sincerely doubt that the speed
difference between Java and Delphi is substantial.  Although I do hear the
myth that Java is slow all the time.

In any case, making your changes public would help a few people, I guess.

Ciao,
Dscho

[Michael Schmid]

Hi Jesper,

as far as I know AVI 2 requires use of index tables, though they seem  
to have a different (two-level) format than AVI 1 (and different  
fourCCs, 'indx' and 'ix##', not 'idx1').
   http://www.the-labs.com/Video/odmlff2-avidef.pdf
So one should be able to get the pointers to the individual frames  
from these tables, without scanning the full file. Reading the tables  
may require some programming effort, but it should make opening a  
large file as virtual stack much faster.

There seems to be some confusion in AVI 1 how to interpret the  
offsets given in the index tables (what they are relative to); and  
programs should probably check the first frame to get the way how the  
offset is calculated (hummm - one of these great format  
specifications by Microsoft that Microsoft does not adhere to; some  
sources say that the reference to the offset was changed once more in  
avifil32.dll of Vista). At least in theory, I think that this problem  
should not exist for the AVI 2 index.

Best wishes,

Michael
________________________________________________________________

On 9 Dec 2008, at 18:59, JesperSP wrote:

> Hi Johannes,
>
> I think bashing my own ability rapidly develop applications in  
> Java, as much
> as my (apparently) faulty belief that Java is slower than native  
> Win32 code
> ;-). The plugin that I modified seeks through the 20GB file to  
> locate all
> frames before it loads the first frame for the virtual stack. I  
> would be
> better if it simply loaded the first frame, and waited to scan for  
> further
> frames until these were needed (same as what MovieIO does). I could  
> try to
> program this in java, but I'm much more familiar with Delphi, so I  
> guess I
> could make faster progress trying to make the same in Delphi (I was  
> thinking
> out loud I guess).. Maybe I should take the time familiarize myself  
> with
> Eclipse (I guess Delphi is being depreciated), so that I can build  
> java
> application/plugins faster..
>
> Cheers,
> Jesper
>
>
> Johannes Schindelin wrote:
>>
>> Hi,
>>
>> On Mon, 8 Dec 2008, JesperSP wrote:
>>
>>> I have read up on the AVI file format, and "hacked the  
>>> AVI_reader.java
>>> so it recognizes the extra RIFF->AVIX segments beyond 1GB - it works
>>> great, but it does take a very long time for ImageJ to scan a  
>>> 20GB file
>>> for the pointers to all the frames. If anyone want it I can  
>>> upload it on
>>> my server, but I cannot guarantee that it'll work for all large AVI
>>> files. For the simple analysis I want to make, I'm considering  
>>> making a
>>> low-level analysis program in a faster language like Borland Delphi.
>>
>> I doubt that the speed has anything to do with the speed of the  
>> language,
>> as the problem here is seeking on the disk.
>>
>> Regardless, if you use Java 1.5 or newer, I sincerely doubt that  
>> the speed
>> difference between Java and Delphi is substantial.  Although I do  
>> hear the
>> myth that Java is slow all the time.
>>
>> In any case, making your changes public would help a few people, I  
>> guess.
>>
>> Ciao,
>> Dscho
>>
>>
>
> --
> View this message in context: http://n2.nabble.com/AVI%2C-Virtual- 
> Stack%2C-%3E1GB-boundery--tp1581053p1635055.html
> Sent from the ImageJ mailing list archive at Nabble.com.

[Franklin Shaffer-2]

I am taking high speed videos of 3D object (particle) flow fields with a 2D high speed camera.  I have the classic problem in that particles are in-focus in a plane and particles outside the plane are out-of-focus.  

I am testing FFT bandpass and other filters to remove the out-of-focus particles.

If anyone has experience with filters or ImageJ plugins that work will for removing out-of-focus objects, please advise.

Thank you,
Frank Shaffer

DOE National Energy Technology Laboratory

[Stephan Preibisch]

Dear Frank,

I wrote an Autofocus plugin for our brightfield microscope using
FFT-bandpass filter which did a good job. It performs better than comparing
the standard deviation of the whole image (concerning quality and time). But
I am so not sure it does a good job for finding particles which are out of
plane.

As you have the advantage that you do not have to do it real time (I think),
you could try Mr. Saalfelds feature detection plugin
(http://fly.mpi-cbg.de/~saalfeld/javasift.html). It detectes features of
different sizes in the image. You could try to analyze the occurrence and
reason from the detected object sizes whether they are in focus or not.

All the best,
Stephan


-----Original Message-----
From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of
Franklin Shaffer
Sent: Monday, February 02, 2009 8:42 PM
To: [hidden email]
Subject: removing out-of-focus objects

I am taking high speed videos of 3D object (particle) flow fields with a 2D
high speed camera.  I have the classic problem in that particles are
in-focus in a plane and particles outside the plane are out-of-focus.  

I am testing FFT bandpass and other filters to remove the out-of-focus
particles.

If anyone has experience with filters or ImageJ plugins that work will for
removing out-of-focus objects, please advise.

Thank you,
Frank Shaffer

DOE National Energy Technology Laboratory