handling very large images

[Aryeh Weiss]

I have a very large image (25x30 images, each is 1600x1200 pixels) that
I want to load into imagej.

This produces the java.lang.NegativeArraySizeException that was
discussed in the following thread:
imagej.1557.x6.nabble.com/Montage-java-lang-NegativeArraySizeException-td5014201.html

In this thread, Curtis refers to a FAQ entry
http://imagej.net/FAQ#I_got_a_NegativeArraySizeException._What_caused_it.3F
but I did not find it in the FAQ after following the link.

This image is not a stack, so I cannot use virtual stack.
I tried to use the crop on import option, and asked for a 12000x12000
section.
It started off ok , and reported that it is verifying the cropping.
Memory monitor reported that about 6.7GB were being used.
It ran for about 45 minutes, and then finally threw the
NegativeArraySizeException.

Now I am trying it with a 5000x5000 crop, which should be no problem to
load. memory monitor claims
6.4 GB are in use. Again, no image has appeared even after about 20 minutes.

Is there a work around to load it?
Alternatively, is there a way to subsample it?

--aryeh

--
Aryeh Weiss
Faculty of Engineering
Bar Ilan University
Ramat Gan 52900 Israel

Ph:  972-3-5317638
FAX: [phone]


--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Mark Hiner]

Hi Aryeh,

> but I did not find it in the FAQ after following the link.

Looks like it moved to the "Troubleshooting" section of the help pages:
 http://imagej.net/Troubleshooting#NegativeArraySizeException

> Again, no image has appeared even after about 20 minutes.

For situations like this you can try manually taking stack traces[1] to get
an idea of where it's getting stuck. If you check where the program is at 5
minutes and compare at 20 minutes you can get an idea of whether you have
hit an actual deadlock or are still processing. If you're up for it, this
is also a good use case for attaching a debugger to ImageJ[2] - then you
can truly see what's going on.

[1] http://imagej.net/Troubleshooting#If_ImageJ_freezes_or_hangs
[2] http://imagej.net/Debugging#Attaching_to_ImageJ_instances

On Thu, Dec 3, 2015 at 6:34 AM, Aryeh Weiss <[hidden email]> wrote:

> I have a very large image (25x30 images, each is 1600x1200 pixels) that I
> want to load into imagej.
>
> This produces the java.lang.NegativeArraySizeException that was discussed
> in the following thread:
>
> imagej.1557.x6.nabble.com/Montage-java-lang-NegativeArraySizeException-td5014201.html
>
> In this thread, Curtis refers to a FAQ entry
> http://imagej.net/FAQ#I_got_a_NegativeArraySizeException._What_caused_it.3F
> but I did not find it in the FAQ after following the link.
>
> This image is not a stack, so I cannot use virtual stack.
> I tried to use the crop on import option, and asked for a 12000x12000
> section.
> It started off ok , and reported that it is verifying the cropping.
> Memory monitor reported that about 6.7GB were being used.
> It ran for about 45 minutes, and then finally threw the
> NegativeArraySizeException.
>
> Now I am trying it with a 5000x5000 crop, which should be no problem to
> load. memory monitor claims
> 6.4 GB are in use. Again, no image has appeared even after about 20
> minutes.
>
> Is there a work around to load it?
> Alternatively, is there a way to subsample it?
>
> --aryeh
>
> --
> Aryeh Weiss
> Faculty of Engineering
> Bar Ilan University
> Ramat Gan 52900 Israel
>
> Ph:  972-3-5317638
> FAX: [phone]
>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Mark J. Chopping]

Hi Aryeh,

I have put some pretty large (>4GB) mosaics together using Image Sequence
with the "use" (virtual memory) option, followed by Make Montage, by
running ImageJ from the commandline: change directory to (for example,)
/Applications/ImageJ_148 (or wherever your ImageJ binary is), with:

java -jar -Xmx5120m ImageJ64_148.app/Contents/Resources/Java/ij.jar

-- where the number after Xmx is the desired memory limit in MB.

Macro code:

print("Mosaicing the "+numfiles+" ("+xLocations+" x "+ NumSlices +") images...");

run("Image Sequence...", "open="+sampleTIFF+" number="+numfiles+" starting=1 increment=1 scale=100 file=.tif sort use");

print("Image Sequence run; doing Make Montage...");

run("Make Montage...", "columns="+xLocations+" rows="+ NumSlices +" scale=1 first=1 last="+numfiles+" increment=1 border=0");

saveAs("Tiff", outDir+"CAN_"+base+".tif");

-- where xLocations and NumSlices are the number of images in X and Y,
respectively; and sampleTIFF is one of a series of TIFFs that are tiles,
numbered sequentially (_0000.tif, _0001.tif, _0002.tif and so on)

This is the only way I could avoid the NegativeArraySizeException erorr.

HTH.

   Mark

  Mark Chopping, Ph.D.
  Montclair State University, Montclair, NJ 07043
  http://www.montclair.edu/csam/remote-sensing-lab/

----

> Date: Thu, 3 Dec 2015 14:34:39 +0200
> From: Aryeh Weiss <[hidden email]>
> To: [hidden email]
> Subject: handling very large images
>
> I have a very large image (25x30 images, each is 1600x1200 pixels) that I
> want to load into imagej.
>
> This produces the java.lang.NegativeArraySizeException that was discussed in
> the following thread:
> imagej.1557.x6.nabble.com/Montage-java-lang-NegativeArraySizeException-td5014201.html
>
> In this thread, Curtis refers to a FAQ entry
> http://imagej.net/FAQ#I_got_a_NegativeArraySizeException._What_caused_it.3F
> but I did not find it in the FAQ after following the link.
>
> This image is not a stack, so I cannot use virtual stack.
> I tried to use the crop on import option, and asked for a 12000x12000
> section.
> It started off ok , and reported that it is verifying the cropping.
> Memory monitor reported that about 6.7GB were being used.
> It ran for about 45 minutes, and then finally threw the
> NegativeArraySizeException.
>
> Now I am trying it with a 5000x5000 crop, which should be no problem to load.
> memory monitor claims
> 6.4 GB are in use. Again, no image has appeared even after about 20 minutes.
>
> Is there a work around to load it?
> Alternatively, is there a way to subsample it?
>
> --aryeh
>
> --
> Aryeh Weiss
> Faculty of Engineering
> Bar Ilan University
> Ramat Gan 52900 Israel
>
> Ph:  972-3-5317638
> FAX: [phone]
>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Aryeh Weiss]

Hi Mark

Thank yo for your reply.

On 03/12/2015 3:44 PM, Mark Hiner wrote:
> Hi Aryeh,
>
> > but I did not find it in the FAQ after following the link.
>
> Looks like it moved to the "Troubleshooting" section of the help pages:
> http://imagej.net/Troubleshooting#NegativeArraySizeException
>
Found it -- thanks.

> > Again, no image has appeared even after about 20 minutes.
>
> For situations like this you can try manually taking stack traces[1]
> to get an idea of where it's getting stuck. If you check where the
> program is at 5 minutes and compare at 20 minutes you can get an idea
> of whether you have hit an actual deadlock or are still processing. If
> you're up for it, this is also a good use case for attaching a
> debugger to ImageJ[2] - then you can truly see what's going on.
>
> [1] http://imagej.net/Troubleshooting#If_ImageJ_freezes_or_hangs
> [2] http://imagej.net/Debugging#Attaching_to_ImageJ_instances
>

The stack trace does not appear to change, so I suppose that it is stuck.
I do not know how to interpret it, but here is the bottom part of it:

"Signal Dispatcher" daemon prio=9 id=5 group=system
    java.lang.Thread.State: RUNNABLE

"Timer-0" daemon prio=5 id=20 group=main
    java.lang.Thread.State: WAITING
     at java.lang.Object.wait(Native Method)
     at java.lang.Object.wait(Object.java:502)
     at java.util.TimerThread.mainLoop(Timer.java:526)
     at java.util.TimerThread.run(Timer.java:505)

"TimerQueue" daemon prio=5 id=30 group=system
    java.lang.Thread.State: WAITING
     at sun.misc.Unsafe.park(Native Method)
     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
     at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
     at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
     at javax.swing.TimerQueue.run(TimerQueue.java:171)
     at java.lang.Thread.run(Thread.java:745)

Running in a debugger is currently beyond my skill set.

I tried it in Java 6, where I run it from the shell prompt. The last few
lines in the output to the shell are:
Calculating image offsets
Finding XML metadata
Parsing XML
Finished assembling XML string
Populating metadata

And then it stops. Memory usage is around 6.4GB . I find it odd that it
would use so much memory, when I asked it to read 2000X2000 pixels.

Fiji is not dead -- I can exit cleanly, but apparently the thread that
handles this is stuck.

Best regards,
--aryeh

> On Thu, Dec 3, 2015 at 6:34 AM, Aryeh Weiss <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     I have a very large image (25x30 images, each is 1600x1200 pixels)
>     that I want to load into imagej.
>
>     This produces the java.lang.NegativeArraySizeException that was
>     discussed in the following thread:
>     imagej.1557.x6.nabble.com/Montage-java-lang-NegativeArraySizeException-td5014201.html
>     <http://imagej.1557.x6.nabble.com/Montage-java-lang-NegativeArraySizeException-td5014201.html>
>
>     In this thread, Curtis refers to a FAQ entry
>     http://imagej.net/FAQ#I_got_a_NegativeArraySizeException._What_caused_it.3F
>     but I did not find it in the FAQ after following the link.
>
>     This image is not a stack, so I cannot use virtual stack.
>     I tried to use the crop on import option, and asked for a
>     12000x12000 section.
>     It started off ok , and reported that it is verifying the cropping.
>     Memory monitor reported that about 6.7GB were being used.
>     It ran for about 45 minutes, and then finally threw the
>     NegativeArraySizeException.
>
>     Now I am trying it with a 5000x5000 crop, which should be no
>     problem to load. memory monitor claims
>     6.4 GB are in use. Again, no image has appeared even after about
>     20 minutes.
>
>     Is there a work around to load it?
>     Alternatively, is there a way to subsample it?
>
>     --aryeh
>
>     --
>     Aryeh Weiss
>     Faculty of Engineering
>     Bar Ilan University
>     Ramat Gan 52900 Israel
>
>     Ph:  972-3-5317638
>     FAX: [phone]
>
>
>     --
>     ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
>

--
Aryeh Weiss
Faculty of Engineering
Bar Ilan University
Ramat Gan 52900 Israel

Ph:  972-3-5317638
FAX: [phone]


--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Aryeh Weiss]

Hi Mark

Thank you for your reply. I indeed have the individual images and can
read a subset of them, and then mosaic them. However, I really need to
stitch them (which I can do), and my first choice is  to read in the
stitched image that  I already have.

I run with a 12GB memory limit, so that should be sufficient.

Best regards
--aryeh


On 03/12/2015 4:13 PM, Mark J. Chopping wrote:

> Hi Aryeh,
>
> I have put some pretty large (>4GB) mosaics together using Image
> Sequence with the "use" (virtual memory) option, followed by Make
> Montage, by running ImageJ from the commandline: change directory to
> (for example,) /Applications/ImageJ_148 (or wherever your ImageJ
> binary is), with:
>
> java -jar -Xmx5120m ImageJ64_148.app/Contents/Resources/Java/ij.jar
>
> -- where the number after Xmx is the desired memory limit in MB.
>
> Macro code:
>
> print("Mosaicing the "+numfiles+" ("+xLocations+" x "+ NumSlices +")
> images...");
>
> run("Image Sequence...", "open="+sampleTIFF+" number="+numfiles+"
> starting=1 increment=1 scale=100 file=.tif sort use");
>
> print("Image Sequence run; doing Make Montage...");
>
> run("Make Montage...", "columns="+xLocations+" rows="+ NumSlices +"
> scale=1 first=1 last="+numfiles+" increment=1 border=0");
>
> saveAs("Tiff", outDir+"CAN_"+base+".tif");
>
> -- where xLocations and NumSlices are the number of images in X and Y,
> respectively; and sampleTIFF is one of a series of TIFFs that are
> tiles, numbered sequentially (_0000.tif, _0001.tif, _0002.tif and so on)
>
> This is the only way I could avoid the NegativeArraySizeException erorr.
>
> HTH.
>
>   Mark
>
>  Mark Chopping, Ph.D.
>  Montclair State University, Montclair, NJ 07043
>  http://www.montclair.edu/csam/remote-sensing-lab/
>
> ----
>> Date: Thu, 3 Dec 2015 14:34:39 +0200
>> From: Aryeh Weiss <[hidden email]>
>> To: [hidden email]
>> Subject: handling very large images
>>
>> I have a very large image (25x30 images, each is 1600x1200 pixels)
>> that I want to load into imagej.
>>
>> This produces the java.lang.NegativeArraySizeException that was
>> discussed in the following thread:
>> imagej.1557.x6.nabble.com/Montage-java-lang-NegativeArraySizeException-td5014201.html
>>
>>
>> In this thread, Curtis refers to a FAQ entry
>> http://imagej.net/FAQ#I_got_a_NegativeArraySizeException._What_caused_it.3F 
>>
>> but I did not find it in the FAQ after following the link.
>>
>> This image is not a stack, so I cannot use virtual stack.
>> I tried to use the crop on import option, and asked for a 12000x12000
>> section.
>> It started off ok , and reported that it is verifying the cropping.
>> Memory monitor reported that about 6.7GB were being used.
>> It ran for about 45 minutes, and then finally threw the
>> NegativeArraySizeException.
>>
>> Now I am trying it with a 5000x5000 crop, which should be no problem
>> to load. memory monitor claims
>> 6.4 GB are in use. Again, no image has appeared even after about 20
>> minutes.
>>
>> Is there a work around to load it?
>> Alternatively, is there a way to subsample it?
>>
>> --aryeh
>>
>> --
>> Aryeh Weiss
>> Faculty of Engineering
>> Bar Ilan University
>> Ramat Gan 52900 Israel
>>
>> Ph:  972-3-5317638
>> FAX: [phone]
>>
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>
>

--
Aryeh Weiss
Faculty of Engineering
Bar Ilan University
Ramat Gan 52900 Israel

Ph:  972-3-5317638
FAX: [phone]

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html