AVI file maximum size?

[Glen MacDonald-2]

Hi,
Is there a size limitation for JPEG compressed AVI files?  A user brought in AVIs of 10 to 15 Gb that could not be opened as virtual stacks.  


Regards,
Glen MacDonald
Digital Microscopy Center
Box 357923
University of Washington
Seattle, WA 98195-7923  USA
[phone]
[hidden email]

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Herbie]

Good day Glen,

there were several discussions about this size limit on the list.

The latest so far dates back to 24. Sept 2015 and ends with an action
(07. Oct. 2015) of Wayne:

"Thanks to Michael Schmid, this bug is fixed in the latest ImageJ daily
build (1.50d10). The AVI Reader now opens most oversized (>4GB) ImageJ
AVI 1 files and the AVI Writer uses the AVI 2 format, which is AVI 1
compatible for files up to about 0.9 GB in size."

Perhaps you can retrieve the whole exchange

Best

Herbie

::::::::::::::::::::::::::::::::::::::::::::
Am 16.04.16 um 00:14 schrieb Glen MacDonald:

> Hi, Is there a size limitation for JPEG compressed AVI files?  A user
> brought in AVIs of 10 to 15 Gb that could not be opened as virtual
> stacks.
>
>
> Regards, Glen MacDonald Digital Microscopy Center Box 357923
> University of Washington Seattle, WA 98195-7923  USA [phone]
> [hidden email]
>
> -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Michael Schmid]

Hi Glen,

in principle, ImageJ should open AVIs of almost essentially any size as
virtual stacks. Please check that
(1) the codec is supported (check for error messages),
(2) the table of frames (roughly 16 bytes per frame) and one single frame
fit into memory (single frames must be less than 2 GPixels),
(3) you have ImageJ 1.50g or later.
(4) Please also check that the file is not corrupted, e.g. is the file
readable by VLC?

If you have an AVI where these conditions are fulfilled, and ImageJ cannot
read it, please let me know
- any error messages,
- output in Debug mode (Edit>Options>Misc) when reading it.
- does ImageJ open a shorter file created with the same software?
- I also paste a simple AVI analyzer below; it would be also helpful to
have its output to analyze the problem.

If I can't determine the problem this way, maybe you can temporarily
upload the file  somewhere so I can get it and try to analyze the problem?

Best,

Michael
______________________________________________________________________


On Sat, April 16, 2016 00:14, Glen MacDonald wrote:

> Hi,
> Is there a size limitation for JPEG compressed AVI files?  A user brought
> in AVIs of 10 to 15 Gb that could not be opened as virtual stacks.
>
>
> Regards,
> Glen MacDonald
> Digital Microscopy Center
> Box 357923
> University of Washington
> Seattle, WA 98195-7923  USA
> [phone]
> [hidden email]
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

______________________________________________________________________

import ij.plugin.*;
import ij.*;
import ij.process.*;
import ij.gui.*;
import ij.io.*;
import ij.util.Tools;
import java.awt.*;
import java.awt.image.*;
import java.io.*;
import java.util.*;

/**
 * ImageJ Plugin for analyzing the basic structure of RIFF files (AVI)
 *
 * Michael Schmid, 2015-10-18, based on the ImageJ AVI_Reader
 *
 * <pre>
 * The AVI format looks like this:
 * RIFF AVI RIFF HEADER, AVI CHUNK
 * | LIST hdrl MAIN AVI HEADER
 * | | avih AVI HEADER
 * | | LIST strl STREAM LIST(s) (One per stream)
 * | | | strh STREAM HEADER (Required after above; fourcc type is
'vids' for video stream)
 * | | | strf STREAM FORMAT (for video: BitMapInfo; may also contain
palette)
 * | | | strd OPTIONAL -- STREAM DATA (ignored in this plugin)
 * | | | strn OPTIONAL -- STREAM NAME (ignored in this plugin)
 * | | | indx OPTIONAL -- MAIN 'AVI 2.0' INDEX
 * | LIST movi MOVIE DATA
 * | | ix00 partial video index of 'AVI 2.0', usually missing in AVI 1
(ix01 would be for audio)
 * | | [rec] RECORD DATA (one record per frame for interleaved video;
optional, unsupported in this plugin)
 * | | |-dataSubchunks RAW DATA: '??wb' for audio, '??db' and '??dc' for
uncompressed and
 * | | compressed video, respectively. "??" denotes stream number,
usually "00" or "01"
 * | idx1 AVI 1 INDEX ('old-style'); may be missing in very old formats
 * RIFF AVIX 'AVI 2.0' only: further chunks
 * | LIST movi more movie data, as above, usually with ix00 index
 * Any number of further chunks (RIFF tags) may follow
 *
 * Items ('chunks') with one fourcc (four-character code such as 'strh')
start with two 4-byte words:
 * the fourcc and the size of the data area.
 * Items with two fourcc (e.g. 'LIST hdrl') have three 4-byte words: the
first fourcc, the size and the
 * second fourcc. Note that the size includes the 4 bytes needed for the
second fourcc.
 *
 * Chunks with fourcc 'JUNK' can appear anywhere and should be ignored.
 *
 * </pre>
 */

public class AVI_Analyzer implements PlugIn {
        private final static long   SIZE_MASK = 0xffffffffL;//for conversion of
sizes from unsigned int to long
        private final static int    paddingGranularity = 2; //tags start at even
address
   //the input file
        private RandomAccessFile   raFile;
        private long                fileSize = -1;
        //
        private int                 level = 0;              //nesting level, 0 is
outermost RIFF
        private final boolean       verbose = IJ.debugMode;
        private int                 printRepeatCount = 0;   //counts repetitions
of the same type
        private String              lastPrintType;          //last type for
printing (chunk type or 'entry')
        private String              printBuffer;            //buffer with the
last of many repetitive lines


        /** The plugin is invoked by ImageJ using this method.
         * @param arg String 'arg' may be used to select the path. If it is an
empty string,
         * a file open dialog is shown, and the resulting ImagePlus is displayed
thereafter.
         * The ImagePlus is not displayed only if 'arg' is a non-empty String; it
can be
         * retrieved with getImagePlus().
         */
        public void run (String arg) {
                OpenDialog od = new OpenDialog("Open AVI or BMP File to Analyze", arg);
                String fileName = od.getFileName();
                if (fileName == null) return;
                String fileDir = od.getDirectory();
                String path = fileDir + fileName;
                try {
                        openAndRead(path);          //open and read
                } catch (Exception e) {
                        IJ.handleException(e);
                        if (raFile != null) try {
                            raFile.close();
                        } catch (Exception e2) {}
                        return;
                }
    }

        /** Open the file with given path and read its header */
        private void openAndRead(String path) throws IOException {
                File file = new File(path); // o p e n
                raFile = new RandomAccessFile(file, "r");
                fileSize = raFile.length();
                if (fileSize < 12)
                    throw new RuntimeException("Error: File size only "+fileSize+" bytes");
                String fileType = readFourccString(); // f i l e h e a d e r
                if (!fileType.equals("RIFF"))
                    throw new RuntimeException("Error: Invalid file type (should be
'RIFF'): '"+fileType+"'");

                printText(path);
                printText("File Size = 0x"+Long.toHexString(fileSize));
                raFile.seek(0);
                while (raFile.getFilePointer() < fileSize-12) {   // loop over top-level
(RIFF) elements
                    long pos = raFile.getFilePointer();
                    int type = readInt();
                    if (type == 0) break;                           // looks like end of
file
                    raFile.seek(pos);
                    analyzeChunk(fileSize);
        }
                raFile.close();
        }

    /* Analyzes a chunk and writes information on it. The chunk must not
go beyond position 'endPos' **/
    private void analyzeChunk(long endPos) throws IOException {
        level++;
        long startPos = raFile.getFilePointer();
        String type = readFourccString();               //first 4 bytes:
four-character code describing chunk type
        long size = readInt() & SIZE_MASK;              //next 4 bytes:
size of what follows (chunk length is this +8)
        long nextPos = raFile.getFilePointer() + size;
        boolean isList = type.equals("RIFF") || type.equals("LIST");
        String listType = isList ? readFourccString() : null;
        printChunk(level, type, listType, startPos, nextPos);
        if (startPos+8+size > endPos)
            printText("ERROR: size (0x"+Long.toHexString(size)+") too
long, exceeds permitted length");
        if (isList)
            while (raFile.getFilePointer() < Math.min(nextPos, endPos))
                analyzeChunk(Math.min(nextPos, endPos));

        if (nextPos%paddingGranularity!=0)
                    nextPos = (nextPos/paddingGranularity+1)*paddingGranularity;
                raFile.seek(nextPos);
        level--;
    }

    /** prints info on a chunk, but only for a limited number of
successive equal non-list chunks of the same type */
    private void printChunk(int level, String type, String listType, long
startPos, long nextPos) {
        StringBuffer sb = new StringBuffer();
        for (int i=1; i<level; i++) sb.append("  | ");    //indent levels
        sb.append("'");
        sb.append(type);
        sb.append("' ");
        if (listType != null) {
            sb.append("'");
            sb.append(listType);
            sb.append("' ");
        }
        sb.append("@0x");
        sb.append(Long.toHexString(startPos));
        sb.append("-0x");
        sb.append(Long.toHexString(nextPos));
        String s = sb.toString();
        if (type.equals(lastPrintType) && listType == null)
            printRepeatCount++;
        else
            printRepeatCount = 0;
        if (printRepeatCount < 3) {
            printBuffer();
            IJ.log(s);
        } else
            printBuffer = s;
        lastPrintType = type;
    }

    /** prints a text irrespective of repetitions that might occur */
    private void printText(String text) {
        printBuffer();
        IJ.log(text);
    }

    /** prints the buffer holding the last of many repetitions */
    private void printBuffer() {
        if (printBuffer != null) {
            IJ.log("           ... ");
            IJ.log(printBuffer);
        }
        printBuffer = null;
        lastPrintType = null;
    }

        /** Write an intData RGB value converted from YUV,
         * The y range between 16 and 235 becomes 0...255
         * u, v should be between -112 and +112
         */
        final void writeRGBfromYUV(int y, int u, int v, int[]pixels, int
intArrayIndex) {
                //int r = (int)(1.164*(y-16)+1.596*v+0.5);
                //int g = (int)(1.164*(y-16)-0.391*u-0.813*v+0.5);
                //int b = (int)(1.164*(y-16)+2.018*u+0.5);
                int r = (9535*y + 13074*v -148464) >> 13;
                int g = (9535*y - 6660*v - 3203*u -148464) >> 13;
                int b = (9535*y + 16531*u -148464) >> 13;
                if (r>255) r=255; if (r<0) r=0;
                if (g>255) g=255; if (g<0) g=0;
                if (b>255) b=255; if (b<0) b=0;
                pixels[intArrayIndex] = 0xff000000 | (r<<16) | (g<<8) | b;
        }

        /** Read 8-byte int with Intel (little-endian) byte order
         * (note: RandomAccessFile.readLong has other byte order than AVI) */

        final long readLong() throws IOException {
                long low = readInt() & 0x00000000FFFFFFFFL;
                long high = readInt() & 0x00000000FFFFFFFFL;
                long result = high <<32 | low;
                return (long) result; //(high << 32 | low);
        }
        /** Read 4-byte int with Intel (little-endian) byte order
         * (note: RandomAccessFile.readInt has other byte order than AVI) */

        final int readInt() throws IOException {
                int result = 0;
                for (int shiftBy = 0; shiftBy < 32; shiftBy += 8)
                        result |= (raFile.readByte() & 0xff) << shiftBy;
                return result;
        }

        /** Read 2-byte short with Intel (little-endian) byte order
         * (note: RandomAccessFile.readShort has other byte order than AVI) */
        final int readShort() throws IOException {
                int low   = raFile.readByte() & 0xff;
                int high  = raFile.readByte() & 0xff;
                return high << 8 | low;
        }

    /** Reads a four-character code and returns it as String */
    final String readFourccString() throws IOException {
        String s = "";
        for (int i=0; i<4; i++) {
            int c = raFile.readByte();
            if (c>=32 && c<127)     //printable 7-bit ascii (as it should be)
                s += Character.toString((char)c);
            else
                s += "<"+Tools.int2hex(c,2)+">";
        }
        return s;
    }

        /** returns a string of a four-cc code corresponding to an int (Intel
byte order) */
        private String fourccString(int fourcc) {
                String s = "";
                for (int i=0; i<4; i++) {
                        int c = fourcc&0xff;
                        s += Character.toString((char)c);
                        fourcc >>= 8;
                }
                return s;
        }

}

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html