Montage java.lang.NegativeArraySizeException

[Mark J. Chopping]

I have my -batch mosaicing macro working... almost. Can anyone tell me what is causing the java "java.lang.NegativeArraySizeException" error below? The routines are called with:
 
xvfb-run-safe ~/ImageJ/jre/bin/java -Xmx12000m -jar ~/ImageJ/ij.jar -ijpath ~/ImageJ -batch mymacro.ijm args > log

The job is going to need a bit of memory but seems to have completed ok when the macro is run in GUI mode (only another 578 to go!). That is, when run from the GUI there were no errors and the large (1.4 Gb) output TIF was generated. However, when running with -batch the log reports:

Mosaicing the 2120 (53 x 40) images...  [message from the macro, then]
Image Sequence run; doing Make Montage...
java.lang.NegativeArraySizeException
        at ij.process.ByteProcessor.createProcessor(ByteProcessor.java:149)
        at ij.plugin.MontageMaker.makeMontage2(MontageMaker.java:143)
        at ij.plugin.MontageMaker.makeMontage(MontageMaker.java:122)
        at ij.plugin.MontageMaker.run(MontageMaker.java:51)
        at ij.IJ.runPlugIn(IJ.java:169)
        at ij.Executer.runCommand(Executer.java:132)
        at ij.Executer.run(Executer.java:65)
        at ij.IJ.run(IJ.java:279)
        at ij.macro.Functions.doRun(Functions.java:600)
        at ij.macro.Functions.doFunction(Functions.java:95)
        at ij.macro.Interpreter.doStatement(Interpreter.java:227)
        at ij.macro.Interpreter.doBlock(Interpreter.java:605)
        at ij.macro.Interpreter.runFirstMacro(Interpreter.java:704)
        at ij.macro.Interpreter.doStatement(Interpreter.java:260)
        at ij.macro.Interpreter.doStatements(Interpreter.java:215)
        at ij.macro.Interpreter.run(Interpreter.java:112)
        at ij.macro.Interpreter.run(Interpreter.java:82)
        at ij.macro.Interpreter.run(Interpreter.java:93)
        at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:153)
        at ij.plugin.Macro_Runner.runMacroFile(Macro_Runner.java:137)
        at ij.IJ.runMacroFile(IJ.java:130)
        at ij.ImageJ.main(ImageJ.java:695)

Does anyone know what is causing this?

 Mark

 Mark Chopping
 Montclair State University

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[ctrueden]

Hi Mark,

Your montage result image is probably too large; there is a 2 gigapixel
limit on plane size.

I added a FAQ entry:

http://imagej.net/FAQ#I_got_a_NegativeArraySizeException._What_caused_it.3F

Regards,
Curtis

On Mon, Aug 31, 2015 at 3:51 PM, Mark Chopping <
[hidden email]> wrote:

> I have my -batch mosaicing macro working... almost. Can anyone tell me
> what is causing the java "java.lang.NegativeArraySizeException" error
> below? The routines are called with:
>
> xvfb-run-safe ~/ImageJ/jre/bin/java -Xmx12000m -jar ~/ImageJ/ij.jar
> -ijpath ~/ImageJ -batch mymacro.ijm args > log
>
> The job is going to need a bit of memory but seems to have completed ok
> when the macro is run in GUI mode (only another 578 to go!). That is, when
> run from the GUI there were no errors and the large (1.4 Gb) output TIF was
> generated. However, when running with -batch the log reports:
>
> Mosaicing the 2120 (53 x 40) images...  [message from the macro, then]
> Image Sequence run; doing Make Montage...
> java.lang.NegativeArraySizeException
>         at ij.process.ByteProcessor.createProcessor(ByteProcessor.java:149)
>         at ij.plugin.MontageMaker.makeMontage2(MontageMaker.java:143)
>         at ij.plugin.MontageMaker.makeMontage(MontageMaker.java:122)
>         at ij.plugin.MontageMaker.run(MontageMaker.java:51)
>         at ij.IJ.runPlugIn(IJ.java:169)
>         at ij.Executer.runCommand(Executer.java:132)
>         at ij.Executer.run(Executer.java:65)
>         at ij.IJ.run(IJ.java:279)
>         at ij.macro.Functions.doRun(Functions.java:600)
>         at ij.macro.Functions.doFunction(Functions.java:95)
>         at ij.macro.Interpreter.doStatement(Interpreter.java:227)
>         at ij.macro.Interpreter.doBlock(Interpreter.java:605)
>         at ij.macro.Interpreter.runFirstMacro(Interpreter.java:704)
>         at ij.macro.Interpreter.doStatement(Interpreter.java:260)
>         at ij.macro.Interpreter.doStatements(Interpreter.java:215)
>         at ij.macro.Interpreter.run(Interpreter.java:112)
>         at ij.macro.Interpreter.run(Interpreter.java:82)
>         at ij.macro.Interpreter.run(Interpreter.java:93)
>         at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:153)
>         at ij.plugin.Macro_Runner.runMacroFile(Macro_Runner.java:137)
>         at ij.IJ.runMacroFile(IJ.java:130)
>         at ij.ImageJ.main(ImageJ.java:695)
>
> Does anyone know what is causing this?
>
>  Mark
>
>  Mark Chopping
>  Montclair State University
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[John Hayes]

Hi Mark,

Can you check that the values for the 'rows' and 'columns' from your macro are going into the 'Make Montage' call correctly? It seems like the call to createProcessor here is being passed one or more negative values (http://rsb.info.nih.gov/ij/developer/source/ij/plugin/MontageMaker.java.html):
> ImageProcessor montage = ip.createProcessor(montageWidth+borderWidth/2, montageHeight+borderWidth/2);

Are those values ones you get from the "Image Sequence" or are they hardcoded by you in the macro?

John

Le 31 août 2015 à 16:51, Mark Chopping a écrit :

> I have my -batch mosaicing macro working... almost. Can anyone tell me what is causing the java "java.lang.NegativeArraySizeException" error below? The routines are called with:
>
> xvfb-run-safe ~/ImageJ/jre/bin/java -Xmx12000m -jar ~/ImageJ/ij.jar -ijpath ~/ImageJ -batch mymacro.ijm args > log
>
> The job is going to need a bit of memory but seems to have completed ok when the macro is run in GUI mode (only another 578 to go!). That is, when run from the GUI there were no errors and the large (1.4 Gb) output TIF was generated. However, when running with -batch the log reports:
>
> Mosaicing the 2120 (53 x 40) images...  [message from the macro, then]
> Image Sequence run; doing Make Montage...
> java.lang.NegativeArraySizeException
>       at ij.process.ByteProcessor.createProcessor(ByteProcessor.java:149)
>       at ij.plugin.MontageMaker.makeMontage2(MontageMaker.java:143)
>       at ij.plugin.MontageMaker.makeMontage(MontageMaker.java:122)
>       at ij.plugin.MontageMaker.run(MontageMaker.java:51)
>       at ij.IJ.runPlugIn(IJ.java:169)
>       at ij.Executer.runCommand(Executer.java:132)
>       at ij.Executer.run(Executer.java:65)
>       at ij.IJ.run(IJ.java:279)
>       at ij.macro.Functions.doRun(Functions.java:600)
>       at ij.macro.Functions.doFunction(Functions.java:95)
>       at ij.macro.Interpreter.doStatement(Interpreter.java:227)
>       at ij.macro.Interpreter.doBlock(Interpreter.java:605)
>       at ij.macro.Interpreter.runFirstMacro(Interpreter.java:704)
>       at ij.macro.Interpreter.doStatement(Interpreter.java:260)
>       at ij.macro.Interpreter.doStatements(Interpreter.java:215)
>       at ij.macro.Interpreter.run(Interpreter.java:112)
>       at ij.macro.Interpreter.run(Interpreter.java:82)
>       at ij.macro.Interpreter.run(Interpreter.java:93)
>       at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:153)
>       at ij.plugin.Macro_Runner.runMacroFile(Macro_Runner.java:137)
>       at ij.IJ.runMacroFile(IJ.java:130)
>       at ij.ImageJ.main(ImageJ.java:695)
>
> Does anyone know what is causing this?
>
> Mark
>
> Mark Chopping
> Montclair State University
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Mark J. Chopping]

Hi Curtis,  

Thanks. The 8-bit output image is somewhat smaller than:

xLocs  = 42,400 (53  800x800-pixel image chips)
yLocs  = 32,000 (40  800x800-pixel image chips)

...so it is reasonably large, about 1.3 Gb and 2,120 chips to mosaic. However, montage seems to work fine when I run it in GUI mode -- that's why I was surprised when it stopped when ImageJ was running from -batch. There are Large Montage and Custom Montage Plugins mentioned in the archives but I didn't think size/memory would be a problem if the vanilla montage worked from the GUI ok.

 Mark

Mark Chopping
Montclair State University

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Mark J. Chopping]

Hi John,

Thanks. The values are hardcoded but since I'm increasingly good at making blunders, I get the macro to print stuff to stdout for debugging, so with

xvfb-run-safe ~/ImageJ/jre/bin/java -Xmx4096m -jar ~/ImageJ/ij.jar -ijpath ~/ImageJ -batch MosaicEm.ijm path:base > MLOG &

I can look at MLOG (names have been changed to protect the innocent):

$ more MLOG
path      =/correct/path/to/directories/
base      =g_WV01_12AUG080532155-P1BS-102001001CE43800_orth
log        =/correct/place/with/logs/g_WV01_12AUG080532155-P1BS-102001001CE43800_orth.log
tdi         =/correct/place/with/tdi/g_WV01_12AUG080532155-P1BS-102001001CE43800_orth.tdi
outDir    =/correct/path/to/the/TIFFDIR/
sampleTIFF=/correct/path/to/the/TIFFDIR/_Crowns/Crown_Filled_0000.tif
WSize:  800
srcCols  =43176
srcRows  =32636
xLocs     :  53
yLocs     :  40
NumSlices :  40
numfiles  :  2120
Mosaicing the 2120 (53 x 40) images...
Image Sequence run; doing Make Montage...
java.lang.NegativeArraySizeException

The macro code is:

  print("Mosaicing the "+numfiles+" ("+xLocations+" x "+ NumSlices +") images...");
  run("Image Sequence...", "open="+sampleTIFF+" number="+numfiles+" starting=1 increment=1 scale=100 file=.tif sort use");
  print("Image Sequence run; doing Make Montage...");
  run("Make Montage...", "columns="+xLocations+" rows="+ NumSlices +" scale=1 first=1 last="+numfiles+" increment=1 border=0");
  saveAs("Tiff", outDir+"CAN_"+base+".tif");

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[John Hayes]

Hi Mark,

I guess what I'm getting at is that Image Sequence may not be completing
correctly before starting the Montage Maker. This may be similar to the
recent problem Avital was having. Check out Wayne's response on using the
IJ.run command.

HTH,

John
On Aug 31, 2015 6:19 PM, "Mark Chopping" <[hidden email]>
wrote:

> Hi John,
>
> Thanks. The values are hardcoded but since I'm increasingly good at making
> blunders, I get the macro to print stuff to stdout for debugging, so with
>
> xvfb-run-safe ~/ImageJ/jre/bin/java -Xmx4096m -jar ~/ImageJ/ij.jar -ijpath
> ~/ImageJ -batch MosaicEm.ijm path:base > MLOG &
>
> I can look at MLOG (names have been changed to protect the innocent):
>
> $ more MLOG
> path      =/correct/path/to/directories/
> base      =g_WV01_12AUG080532155-P1BS-102001001CE43800_orth
> log
> =/correct/place/with/logs/g_WV01_12AUG080532155-P1BS-102001001CE43800_orth.log
> tdi
>  =/correct/place/with/tdi/g_WV01_12AUG080532155-P1BS-102001001CE43800_orth.tdi
> outDir    =/correct/path/to/the/TIFFDIR/
> sampleTIFF=/correct/path/to/the/TIFFDIR/_Crowns/Crown_Filled_0000.tif
> WSize:  800
> srcCols  =43176
> srcRows  =32636
> xLocs     :  53
> yLocs     :  40
> NumSlices :  40
> numfiles  :  2120
> Mosaicing the 2120 (53 x 40) images...
> Image Sequence run; doing Make Montage...
> java.lang.NegativeArraySizeException
>
> The macro code is:
>
>   print("Mosaicing the "+numfiles+" ("+xLocations+" x "+ NumSlices +")
> images...");
>   run("Image Sequence...", "open="+sampleTIFF+" number="+numfiles+"
> starting=1 increment=1 scale=100 file=.tif sort use");
>   print("Image Sequence run; doing Make Montage...");
>   run("Make Montage...", "columns="+xLocations+" rows="+ NumSlices +"
> scale=1 first=1 last="+numfiles+" increment=1 border=0");
>   saveAs("Tiff", outDir+"CAN_"+base+".tif");
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[ctrueden]

Hi Mark,

I am still betting that the math of width*height in the new ByteProcessor
is overflowing to go negative. You could use remote debugging to debug it
[1], or else hack the ImageJ1 source [2] to add some print statements.

Regards,
Curtis

[1] http://imagej.net/Debugging#Attaching_to_ImageJ_instances
[2] https://github.com/imagej/ImageJA

On Mon, Aug 31, 2015 at 5:11 PM, Mark Chopping <
[hidden email]> wrote:

> Hi John,
>
> Thanks. The values are hardcoded but since I'm increasingly good at making
> blunders, I get the macro to print stuff to stdout for debugging, so with
>
> xvfb-run-safe ~/ImageJ/jre/bin/java -Xmx4096m -jar ~/ImageJ/ij.jar -ijpath
> ~/ImageJ -batch MosaicEm.ijm path:base > MLOG &
>
> I can look at MLOG (names have been changed to protect the innocent):
>
> $ more MLOG
> path      =/correct/path/to/directories/
> base      =g_WV01_12AUG080532155-P1BS-102001001CE43800_orth
> log
> =/correct/place/with/logs/g_WV01_12AUG080532155-P1BS-102001001CE43800_orth.log
> tdi
>  =/correct/place/with/tdi/g_WV01_12AUG080532155-P1BS-102001001CE43800_orth.tdi
> outDir    =/correct/path/to/the/TIFFDIR/
> sampleTIFF=/correct/path/to/the/TIFFDIR/_Crowns/Crown_Filled_0000.tif
> WSize:  800
> srcCols  =43176
> srcRows  =32636
> xLocs     :  53
> yLocs     :  40
> NumSlices :  40
> numfiles  :  2120
> Mosaicing the 2120 (53 x 40) images...
> Image Sequence run; doing Make Montage...
> java.lang.NegativeArraySizeException
>
> The macro code is:
>
>   print("Mosaicing the "+numfiles+" ("+xLocations+" x "+ NumSlices +")
> images...");
>   run("Image Sequence...", "open="+sampleTIFF+" number="+numfiles+"
> starting=1 increment=1 scale=100 file=.tif sort use");
>   print("Image Sequence run; doing Make Montage...");
>   run("Make Montage...", "columns="+xLocations+" rows="+ NumSlices +"
> scale=1 first=1 last="+numfiles+" increment=1 border=0");
>   saveAs("Tiff", outDir+"CAN_"+base+".tif");
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Mark J. Chopping]

The problem seems to have been a conflict between Montage and xvfb, so I have a partial solution: run the batch job in X11 without xvfb, like this:

~/ImageJ/jre/bin/java -Xmx4096m -jar ~/ImageJ/ij.jar -ijpath ~/ImageJ -batch mosaci_macro.ijm path/base > MLOG2 &

Result:

$ more MLOG2
...
Window Size:  800 x 800
SrcCols  =43176
SrcRows  =32636
xLocs     :  53
yLocs     :  40
numfiles  :  2120
Mosaicing the 2120 (53 x 40) images...
Image Sequence run; doing Make Montage...
Mosaic completed.

$ ls -l
 1356800240 Aug 31 22:54 Mosaiced_image.tif

This is great: although it means that I have to keep an X session alive rather than completely backgrounding all the jobs, at least I can run many jobs without intervention, if only sequentially (on each VM).

Many thanks to Curtis Rueden and John Hayes for excellent suggestions.

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html