imagej and Java 1.8
Locked
 
|
I have noticed that when using Fiji with Java 1.8, the response
gradually gets slower, until eventually I need to restart Fiji. The issue seems to be connected to screen updates, but I cannot pin it down to a single item, although the ROI Manager seems to often be involved. For example, toggling "Show All" in the ROI manager tends to become really slow. Now, the memory monitor shows 6.8GB in use, although I have very little open. Double clicking on the Fiji window does not appear to have forced GC. I am just adding this update in the hope that these observations may prove useful to the people who know more about Fiji and Java 1.8 than I. If there are addition things that I can do to provide useful data for this, please let me know. --aryeh -- Aryeh Weiss Faculty of Engineering Bar Ilan University Ramat Gan 52900 Israel Ph: 972-3-5317638 FAX: 972-3-7384051 -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
What version of ImageJ are you using? What OS? I worked with Chris recently
on the ImageJ forum to track down a slowness related to window refreshing with Java 8 and OSX. I believe this was fixed in 1.50e. If you do the same things in ImageJ (not FIJI) itself does it happen? On Sun, Dec 6, 2015 at 17:41 Aryeh Weiss <[hidden email]> wrote: > I have noticed that when using Fiji with Java 1.8, the response > gradually gets slower, until eventually > I need to restart Fiji. The issue seems to be connected to screen > updates, but I cannot pin it down to a single item, although the ROI > Manager seems to often be involved. > For example, toggling "Show All" in the ROI manager tends to become > really slow. > > Now, the memory monitor shows 6.8GB in use, although I have very little > open. Double clicking on the Fiji window does not appear to have forced GC. > > I am just adding this update in the hope that these observations may > prove useful to the people who know more about Fiji and Java 1.8 than I. > If there are addition things that I can do to provide useful data for > this, please let me know. > > --aryeh > > -- > Aryeh Weiss > Faculty of Engineering > Bar Ilan University > Ramat Gan 52900 Israel > > Ph: 972-3-5317638 > FAX: 972-3-7384051 > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Thank you for your reply.
On 06/12/2015 12:54 PM, Brandon Hurr wrote: > What version of ImageJ are you using? What OS? I worked with Chris > recently on the ImageJ forum to track down a slowness related to > window refreshing with Java 8 and OSX. I believe this was fixed in 1.50e. > The OS is OSX 10.11.1 ImageJ version is 1.50f13 Java is 1.8.0_66 (64 bit) > If you do the same things in ImageJ (not FIJI) itself does it happen? I have not tried this because it would require finding all of the plugins/components that I use in Fiji and moving them to ImageJ. --aryeh > On Sun, Dec 6, 2015 at 17:41 Aryeh Weiss <[hidden email] > <mailto:[hidden email]>> wrote: > > I have noticed that when using Fiji with Java 1.8, the response > gradually gets slower, until eventually > I need to restart Fiji. The issue seems to be connected to screen > updates, but I cannot pin it down to a single item, although the ROI > Manager seems to often be involved. > For example, toggling "Show All" in the ROI manager tends to become > really slow. > > Now, the memory monitor shows 6.8GB in use, although I have very > little > open. Double clicking on the Fiji window does not appear to have > forced GC. > > I am just adding this update in the hope that these observations may > prove useful to the people who know more about Fiji and Java 1.8 > than I. > If there are addition things that I can do to provide useful data for > this, please let me know. > > --aryeh > > -- > Aryeh Weiss > Faculty of Engineering > Bar Ilan University > Ramat Gan 52900 Israel > > Ph: 972-3-5317638 > FAX: 972-3-7384051 > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > -- Aryeh Weiss Faculty of Engineering Bar Ilan University Ramat Gan 52900 Israel Ph: 972-3-5317638 FAX: 972-3-7384051 -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
Re: imagej and Java 1.8
|
|
In reply to this post by Aryeh Weiss
On Dec 6, 2015, at 5:24 AM, Aryeh Weiss <[hidden email]> wrote:
> > I have noticed that when using Fiji with Java 1.8, the response gradually gets slower, until eventually > I need to restart Fiji. The issue seems to be connected to screen updates, but I cannot pin it down to a single item, although the ROI Manager seems to often be involved. > For example, toggling "Show All" in the ROI manager tends to become really slow. The gradual slowdown of ImageJ with Java 1.8 on OS X appears to be related to setting the menu bar at the top of the screen when image windows are activated. With the ImageJ 1.50e update, the Stack to Images, Window>Tile and Window>Cascade commands, and non-batch mode macros that process multiple images, attempt to minimize setting the menu bar. There is a test macro at http://imagej.nih.gov/ij/macros/SetMenuBarBenchmark.txt which runs much faster on ImageJ 1.50e and later. The latest ImageJ daily build (1.50f16) also reduces the number of times the menu bar is set when switching between windows that have their own menu bar (e.g. “Results" and “Log”) and the “ImageJ" window. The daily build displays a count of the number of times the menu bar has been set when you press the “i” key (Image>Show Info). Note that the "Show Info" command now works even if no image windows are open. > Now, the memory monitor shows 6.8GB in use, although I have very little open. Double clicking on the Fiji window does not appear to have forced GC. This looks like a memory leak. Clicking in the ImageJ status bar, or in the Memory Monitor window, runs the Java garbage collector, which reclaims unused memory unless there is a memory leak. Sometimes you need to click several times to reclaim all unused memory. Memory leaks can also cause ImageJ to slowdown. Do you see the memory leak when using Java 1.6? It would be very helpful if you could figure out which plugin is causing the memory leak. > I am just adding this update in the hope that these observations may prove useful to the people who know more about Fiji and Java 1.8 than I. If there are addition things that I can do to provide useful data for this, please let me know. -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Hi Aryeh,
>It would be very helpful if you could figure out which plugin is causing the memory leak. When the memory leak occurs, you can acquire a heap dump to take a snapshot of object allocations in Java. At that point the memory leak culprit often becomes obvious. On the wiki we have instructions[1] and a practice exercise[2] for acquiring and analyzing heap dumps. If you can acquire the heap dump but aren't sure how to interpret the results, you can also save it as a .hprof file and share it here so we can help analyze it. Best, Mark [1] http://imagej.net/Debugging#Debugging_JVM_hangs [2] http://imagej.net/Debugging_Exercises#Exercise_7:_Out_of_memory On Sun, Dec 6, 2015 at 2:23 PM, Rasband, Wayne (NIH/NIMH) [E] < [hidden email]> wrote: > On Dec 6, 2015, at 5:24 AM, Aryeh Weiss <[hidden email]> wrote: > > > > I have noticed that when using Fiji with Java 1.8, the response > gradually gets slower, until eventually > > I need to restart Fiji. The issue seems to be connected to screen > updates, but I cannot pin it down to a single item, although the ROI > Manager seems to often be involved. > > For example, toggling "Show All" in the ROI manager tends to become > really slow. > > The gradual slowdown of ImageJ with Java 1.8 on OS X appears to be related > to setting the menu bar at the top of the screen when image windows are > activated. With the ImageJ 1.50e update, the Stack to Images, Window>Tile > and Window>Cascade commands, and non-batch mode macros that process > multiple images, attempt to minimize setting the menu bar. There is a test > macro at > > http://imagej.nih.gov/ij/macros/SetMenuBarBenchmark.txt > > which runs much faster on ImageJ 1.50e and later. The latest ImageJ daily > build (1.50f16) also reduces the number of times the menu bar is set when > switching between windows that have their own menu bar (e.g. “Results" and > “Log”) and the “ImageJ" window. The daily build displays a count of the > number of times the menu bar has been set when you press the “i” key > (Image>Show Info). Note that the "Show Info" command now works even if no > image windows are open. > > > Now, the memory monitor shows 6.8GB in use, although I have very little > open. Double clicking on the Fiji window does not appear to have forced GC. > > This looks like a memory leak. Clicking in the ImageJ status bar, or in > the Memory Monitor window, runs the Java garbage collector, which reclaims > unused memory unless there is a memory leak. Sometimes you need to click > several times to reclaim all unused memory. Memory leaks can also cause > ImageJ to slowdown. Do you see the memory leak when using Java 1.6? It > would be very helpful if you could figure out which plugin is causing the > memory leak. > > > I am just adding this update in the hope that these observations may > prove useful to the people who know more about Fiji and Java 1.8 than I. If > there are addition things that I can do to provide useful data for this, > please let me know. > > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Thanks to Wayne and Mark for their replies.
On 07/12/2015 5:02 PM, Mark Hiner wrote: > Hi Aryeh, > >> It would be very helpful if you could figure out which plugin is causing >> the memory leak. >> >> When the memory leak occurs, you can acquire a heap dump to take a snapshot >> of object allocations in Java. At that point the memory leak culprit often >> becomes obvious. >> >> On the wiki we have instructions[1] and a practice exercise[2] for >> acquiring and analyzing heap dumps. If you can acquire the heap dump but >> aren't sure how to interpret the results, you can also save it as a .hprof >> file and share it here so we can help analyze it. running a script that opens a few 1.3Gpixel images, then run it after, then close the images and get a heap dump when the images have been closed but the memory is not freed (which is reproducible). At this point I learned that generating 6GB+ heap dumps is a great way to run out of disk space in a hurry :-). So I will do this again, this time writing to an external disk and deleting the dumps (but that will have to wait until tomorrow). Compressed, the dumps are about 1 GB . In order to send it to you, I will probably need to send it to an FTP server (like I have done with bio-formats. DO you have a server to which I can send the dumps? > Best, > Mark > > [1] http://imagej.net/Debugging#Debugging_JVM_hangs > [2] http://imagej.net/Debugging_Exercises#Exercise_7:_Out_of_memory > > On Sun, Dec 6, 2015 at 2:23 PM, Rasband, Wayne (NIH/NIMH) [E] < > [hidden email]> wrote: > >> On Dec 6, 2015, at 5:24 AM, Aryeh Weiss <[hidden email]> wrote: >>> I have noticed that when using Fiji with Java 1.8, the response >>> gradually gets slower, until eventually >>> I need to restart Fiji. The issue seems to be connected to screen >>> updates, but I cannot pin it down to a single item, although the ROI >>> Manager seems to often be involved. >>> For example, toggling "Show All" in the ROI manager tends to become really slow. >>> >> >> The gradual slowdown of ImageJ with Java 1.8 on OS X appears to be related >> to setting the menu bar at the top of the screen when image windows are >> activated. With the ImageJ 1.50e update, the Stack to Images, Window>Tile >> and Window>Cascade commands, and non-batch mode macros that process >> multiple images, attempt to minimize setting the menu bar. I think that this problem is exacerbated on a dual monitor OSX system, because whenever windows are transfered between monitors the menu bar is updated. >>> Now, the memory monitor shows 6.8GB in use, although I have very little >> open. Double clicking on the Fiji window does not appear to have forced GC. >> >> This looks like a memory leak. Clicking in the ImageJ status bar, or in >> the Memory Monitor window, runs the Java garbage collector, which reclaims >> unused memory unless there is a memory leak. Sometimes you need to click >> several times to reclaim all unused memory. Memory leaks can also cause >> ImageJ to slowdown. Do you see the memory leak when using Java 1.6? It >> would be very helpful if you could figure out which plugin is causing the >> memory leak. >> >> which leaves 3 1.3Gpixel images open, and when that is finished, the memory usage appears appropriate. But when I close the images, and attempt to garbage-collect, them memory is not freed up. I have not been able to test it under Java 1.6 because when I run my macro, I get the following error when loading the large image: Invalid memory access of location 0x1aa84b080 rip=0x7fff8a310f1a fijiJ6.sh: line 3: 92360 Segmentation fault: 11 $J6/bin/$($IJ_HOME/Contents/MacOS/ImageJ-macosx --dry-run | sed 's/ -Djava.ext.dirs=[^ ]*//') The code that is run at this point is: for w in WindowManager.getNonImageTitles(): if (w[:-3] != '.py' and w[:-4]!='.ijm' and w[:] != 'Recorder') and w[:] != "Command Finder" and w[:] != "Memory": print w WindowManager.getWindow(w).close() for w in WindowManager.getImageTitles(): WindowManager.getWindow(w).close() op = OpenDialog("Choose input image...", "") path = op.getDirectory()+ op.getFileName() inputName = op.getFileName() inputDir = op.getDirectory() inputPath = inputDir + inputName if inputName[-4] == ".": inputPrefix = inputName[:-4] # assumes that a suffix exists else: inputPrefix = inputName inputImp = ImagePlus(inputPath) inputImp.show() ImageConverter(inputImp).convertToGray8() inputIp = inputImp.getProcessor() The program fails before the inputImp.show() command paints the window. If I load this image outside of the macro (just by opening with the menu), then it will take a long time to open, but it will open properly. This happens in Java 1.6 In Java 1.8 it opens as expected, my script runs and all is well for a while, until things get slow. Best regards, --aryeh -- Aryeh Weiss Faculty of Engineering Bar Ilan University Ramat Gan 52900 Israel Ph: 972-3-5317638 FAX: 972-3-7384051 -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Hi Aryeh,
On Mon, Dec 7, 2015 at 11:46 AM, Aryeh Weiss <[hidden email]> wrote: > Thanks for this information. I decided to run IJ, do a heap dump before > running a script that opens a few 1.3Gpixel images, then run it after, then > close the images and get a heap dump when the images have been closed but > the memory is not freed (which is reproducible). This sounds like the perfect set of information! > In order to send it to you, I will probably need to send it to an FTP server (like I have done with bio-formats. DO you have a server to which I can send the dumps? Actually you should be able to use the Upload Sample Image[1] command to send the zip. If that doesn't work for you let me know and I can send FTP credentails. Thanks, - Mark [1] http://imagej.net/Upload_Sample_Image -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Hi Mark
On 07/12/2015 11:04 PM, Mark Hiner wrote: > Hi Aryeh, > > On Mon, Dec 7, 2015 at 11:46 AM, Aryeh Weiss <[hidden email] > <mailto:[hidden email]>> wrote: > > Thanks for this information. I decided to run IJ, do a heap dump > before running a script that opens a few 1.3Gpixel images, then > run it after, then close the images and get a heap dump when the > images have been closed but the memory is not freed (which is > reproducible). > > > This sounds like the perfect set of information! > > > In order to send it to you, I will probably need to send it to an > FTP server (like I have done with bio-formats. DO you have a server to > which I can send the dumps? > > Actually you should be able to use the Upload Sample Image[1] command > to send the zip. If that doesn't work for you let me know and I can > send FTP credentails. > I uploaded a set of heap dumps using Upload Sample Image as you suggested. They are: heapdump-1449551945772 -- made immediately after launching Fiji. Small, as expected. heapdump-1449552198991 -- made immediately after running my python script. Three large image are open. heapdump-1449552667004 -- made after closing those three large images and double clicking in the Fiji window a few times to run the garbage collector heapdump-1449553044705 -- made after running the script a second time, during which Fiji ran out of memory. Two large images are open. heapdump-1449553286368 -- made after closing those two windows and garbage collecting. As expected, memory was not freed. The large images are a bit under 1Gpixel, 8-bit images. I hope that this is useful. Thanks for your efforts. --aryeh -- Aryeh Weiss Faculty of Engineering Bar Ilan University Ramat Gan 52900 Israel Ph: 972-3-5317638 FAX: 972-3-7384051 -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Hi Aryeh!
>I hope that this is useful. Yep this is exactly what I needed and your thoroughness is greatly appreciated. :) I started with heapdump-144955328636 and it was immediately clear that something on the python side is keeping hard references to your images. It looks like they are being stored in a ConcurrentHashMap which is not cleaned up when your script ends. So it looks like a bug in the Python script you're using. I put up some screen captures[1] of the reference path that's keeping your objects alive, along with a brief interpretation. The variable names are distinct so I'm hoping it will be obvious where in your script they are being mapped. Ideally I would suggest caching weak references on the python side if that's an option, or possibly clearing the cache at the end of the script. If it's not clear how to fix the script and you'd like more eyes on it, share it here - happy to have a look! :) Also happy to give more info on the heap dump analysis if anything's unclear there. If anyone else is interested in looking at the heap dumps I personally recommend the Eclipse Memory Analyzer plugin[2]. Best, Mark [1] https://gist.github.com/hinerm/727453aab0837246367e [2] https://eclipse.org/mat/downloads.php On Tue, Dec 8, 2015 at 2:25 AM, Aryeh Weiss <[hidden email]> wrote: > Hi Mark > > On 07/12/2015 11:04 PM, Mark Hiner wrote: > > Hi Aryeh, > > On Mon, Dec 7, 2015 at 11:46 AM, Aryeh Weiss <[hidden email]> wrote: > >> Thanks for this information. I decided to run IJ, do a heap dump before >> running a script that opens a few 1.3Gpixel images, then run it after, then >> close the images and get a heap dump when the images have been closed but >> the memory is not freed (which is reproducible). > > > This sounds like the perfect set of information! > > > In order to send it to you, I will probably need to send it to an FTP > server (like I have done with bio-formats. DO you have a server to which I > can send the dumps? > > Actually you should be able to use the Upload Sample Image[1] command to > send the zip. If that doesn't work for you let me know and I can send FTP > credentails. > > > I uploaded a set of heap dumps using Upload Sample Image as you > suggested. > They are: > heapdump-1449551945772 -- made immediately after launching Fiji. Small, as > expected. > heapdump-1449552198991 -- made immediately after running my python > script. Three large image are open. > heapdump-1449552667004 -- made after closing those three large images and > double clicking in the Fiji window > a few times to run the > garbage collector > heapdump-1449553044705 -- made after running the script a second time, > during which Fiji ran out of memory. > Two large images are open. > heapdump-1449553286368 -- made after closing those two windows and garbage > collecting. > As expected, memory was not > freed. > > The large images are a bit under 1Gpixel, 8-bit images. > > I hope that this is useful. > Thanks for your efforts. > --aryeh > > -- > Aryeh Weiss > Faculty of Engineering > Bar Ilan University > Ramat Gan 52900 Israel > > Ph: 972-3-5317638 > FAX: 972-3-7384051 > > > -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Hi Mark
Thanks again for your quick reply. This reply is copied to the list because It may be informative. I have a minimal script that does something similar to what I reported. from ij import IJ, Prefs, ImagePlus from ij.io import DirectoryChooser, OpenDialog from ij.process import ImageConverter, ByteProcessor, BinaryProcessor op = OpenDialog("Choose input image...", "") path = op.getDirectory()+ op.getFileName() inputName = op.getFileName() inputDir = op.getDirectory() inputPath = inputDir + inputName if inputName[-4] == ".": inputPrefix = inputName[:-4] # assumes that a suffix exists else: inputPrefix = inputName inputImp = ImagePlus(inputPath) inputImp.show() ImageConverter(inputImp).convertToGray8() # The following line makes all the difference. # inputIp = inputImp.getProcessor() The line inputIp = inputImp.getProcessor() makes a reproducible difference. If it exists, then after running this program, the image memory is not freed. If I run this script multiple times, it seems to retain one copy (maybe it overwrites the reference). If I comment out this line, then gc will free all of the memory. Moreover, if I ran it with this line uncommented, and the memory got "stuck", and then I run it without this lline (ie commented out), the next gc will free *all* of the memory. I hope this is a useful clue. --aryeh On 08/12/2015 7:09 PM, Mark Hiner wrote: > Thanks Aryeh! > > >How are hard references created? > > So first let me apologize for dragging you into the world of > references. It is not a happy place and certainly not something a user > or biologist should have to worry about, as it gets into the inner > workings of Java itself. > > There's a nice succinct description of the different references here[1]. > > Basically when an object is created it gets a "strong" by default, and > you have to use special Java classes to get the other reference types. > You get memory leaks when those strong references persist longer than > expected (for example because they were created by something > fundamental to the program). > > The way to fix the problem varies based on the actual cause - in your > case it could be a bug in Jython itself. > > I'll look at your script and report back! > > Thanks again, > Mark > > [1] > https://www.rallydev.com/blog/engineering/java-references-strong-soft-weak-phantom > > > On Tue, Dec 8, 2015 at 10:33 AM, Aryeh Weiss <[hidden email] > <mailto:[hidden email]>> wrote: > > Hi Mark > > Thank you for this quick reply. > I am replying off-list so that I can attach the script. > > On 08/12/2015 5:07 PM, Mark Hiner wrote: >> Hi Aryeh! >> >> >I hope that this is useful. >> >> Yep this is exactly what I needed and your thoroughness is >> greatly appreciated. :) >> >> I started with heapdump-144955328636 and it was immediately clear >> that something on the python side is keeping hard references to >> your images. It looks like they are being stored in a >> ConcurrentHashMap which is not cleaned up when your script ends. >> >> So it looks like a bug in the Python script you're using. I put >> up some screen captures[1] of the reference path that's keeping >> your objects alive, along with a brief interpretation. The >> variable names are distinct so I'm hoping it will be obvious >> where in your script they are being mapped. Ideally I would >> suggest caching weak references on the python side if that's an >> option, or possibly clearing the cache at the end of the script. >> > I do not have any variable named snapshotPixels (or rLUT1, rLUT2, > etc), so I assume that these are variables internal to some plugin > that is used (or perhaps internal to IJ. > > Some very basic questions: > How are hard references created? > Is there a way at the end of a script to tell it to clear all such > reference? > >> If it's not clear how to fix the script and you'd like more eyes >> on it, share it here - happy to have a look! :) Also happy to >> give more info on the heap dump analysis if anything's unclear there. >> > > I attached the script -- it is not well written (to say the very > least), as I am new at muddling my way through the API. The most > recent addition to it is the mouse listener which I added to allow > the user to select three points near the end of the script. > > Best regards > --aryeh > >> If anyone else is interested in looking at the heap dumps I >> personally recommend the Eclipse Memory Analyzer plugin[2]. >> >> Best, >> Mark >> >> [1] https://gist.github.com/hinerm/727453aab0837246367e >> [2] https://eclipse.org/mat/downloads.php >> >> On Tue, Dec 8, 2015 at 2:25 AM, Aryeh Weiss <[hidden email] >> <mailto:[hidden email]>> wrote: >> >> >> Hi Mark >> >> On 07/12/2015 11:04 PM, Mark Hiner wrote: >>> Hi Aryeh, >>> >>> On Mon, Dec 7, 2015 at 11:46 AM, Aryeh Weiss >>> <[hidden email] <mailto:[hidden email]>> wrote: >>> >>> Thanks for this information. I decided to run IJ, do a >>> heap dump before running a script that opens a few >>> 1.3Gpixel images, then run it after, then close the >>> images and get a heap dump when the images have been >>> closed but the memory is not freed (which is reproducible). >>> >>> >>> This sounds like the perfect set of information! >>> >>> > In order to send it to you, I will probably need to send >>> it to an FTP server (like I have done with bio-formats. DO >>> you have a server to which I can send the dumps? >>> >>> Actually you should be able to use the Upload Sample >>> Image[1] command to send the zip. If that doesn't work for >>> you let me know and I can send FTP credentails. >>> >> >> I uploaded a set of heap dumps using Upload Sample Image as >> you suggested. >> They are: >> heapdump-1449551945772 -- made immediately after launching >> Fiji. Small, as expected. >> heapdump-1449552198991 -- made immediately after running my >> python script. Three large image are open. >> heapdump-1449552667004 -- made after closing those three >> large images and double clicking in the Fiji window >> a few times to run the garbage collector >> heapdump-1449553044705 -- made after running the script a >> second time, during which Fiji ran out of memory. >> Two large images are open. >> heapdump-1449553286368 -- made after closing those two >> windows and garbage collecting. >> As expected, memory was not freed. >> >> The large images are a bit under 1Gpixel, 8-bit images. >> >> I hope that this is useful. >> Thanks for your efforts. >> --aryeh >> -- Aryeh Weiss Faculty of Engineering Bar Ilan University Ramat Gan 52900 Israel Ph: 972-3-5317638 FAX: 972-3-7384051 -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Hi Aryeh,
>I hope this is a useful clue. Indeed, this script is very helpful; it's always great to have a minimal script to reproduce a problem. Jython has some infrastructure to maintain a Python "sys" environment inside Java. My naive guess is that this sticks around after running, but is destroyed and recreated when a new script runs... so it is quite probably an issue in the core script engine.. That said, I'm having problems actually seeing the memory leak with the script. :( Do you have any local modifications besides the BioVoxxel and 3d image suite update sites? If so, would you mind running "Help > Report a Bug" and pasting back the system information, so I can figure out what I'm missing... Best, Mark On Tue, Dec 8, 2015 at 11:29 AM, Aryeh Weiss <[hidden email]> wrote: > Hi Mark > > Thanks again for your quick reply. > This reply is copied to the list because It may be informative. > > I have a minimal script that does something similar to what I reported. > > from ij import IJ, Prefs, ImagePlus > from ij.io import DirectoryChooser, OpenDialog > from ij.process import ImageConverter, ByteProcessor, BinaryProcessor > > op = OpenDialog("Choose input image...", "") > path = op.getDirectory()+ op.getFileName() > inputName = op.getFileName() > inputDir = op.getDirectory() > inputPath = inputDir + inputName > > if inputName[-4] == ".": > inputPrefix = inputName[:-4] # assumes that a suffix exists > else: > inputPrefix = inputName > > > inputImp = ImagePlus(inputPath) > inputImp.show() > ImageConverter(inputImp).convertToGray8() > > # The following line makes all the difference. > # inputIp = inputImp.getProcessor() > > The line > inputIp = inputImp.getProcessor() > > makes a reproducible difference. If it exists, then after running this > program, the image memory is not freed. > If I run this script multiple times, it seems to retain one copy (maybe it > overwrites the reference). > > If I comment out this line, then gc will free all of the memory. > Moreover, if I ran it with this line uncommented, and the memory got > "stuck", and then I run it without > this lline (ie commented out), the next gc will free *all* of the memory. > > I hope this is a useful clue. > --aryeh > > > > > On 08/12/2015 7:09 PM, Mark Hiner wrote: > > Thanks Aryeh! > > >How are hard references created? > > So first let me apologize for dragging you into the world of references. > It is not a happy place and certainly not something a user or biologist > should have to worry about, as it gets into the inner workings of Java > itself. > > There's a nice succinct description of the different references here[1]. > > Basically when an object is created it gets a "strong" by default, and you > have to use special Java classes to get the other reference types. You get > memory leaks when those strong references persist longer than expected (for > example because they were created by something fundamental to the program). > > The way to fix the problem varies based on the actual cause - in your case > it could be a bug in Jython itself. > > I'll look at your script and report back! > > Thanks again, > Mark > > [1] > https://www.rallydev.com/blog/engineering/java-references-strong-soft-weak-phantom > > > On Tue, Dec 8, 2015 at 10:33 AM, Aryeh Weiss <[hidden email]> wrote: > >> Hi Mark >> >> Thank you for this quick reply. >> I am replying off-list so that I can attach the script. >> >> On 08/12/2015 5:07 PM, Mark Hiner wrote: >> >> Hi Aryeh! >> >> >I hope that this is useful. >> >> Yep this is exactly what I needed and your thoroughness is greatly >> appreciated. :) >> >> I started with heapdump-144955328636 and it was immediately clear that >> something on the python side is keeping hard references to your images. It >> looks like they are being stored in a ConcurrentHashMap which is not >> cleaned up when your script ends. >> >> So it looks like a bug in the Python script you're using. I put up some >> screen captures[1] of the reference path that's keeping your objects alive, >> along with a brief interpretation. The variable names are distinct so I'm >> hoping it will be obvious where in your script they are being mapped. >> Ideally I would suggest caching weak references on the python side if >> that's an option, or possibly clearing the cache at the end of the script. >> >> I do not have any variable named snapshotPixels (or rLUT1, rLUT2, etc), >> so I assume that these are variables internal to some plugin that is used >> (or perhaps internal to IJ. >> >> Some very basic questions: >> How are hard references created? >> Is there a way at the end of a script to tell it to clear all such >> reference? >> >> If it's not clear how to fix the script and you'd like more eyes on it, >> share it here - happy to have a look! :) Also happy to give more info on >> the heap dump analysis if anything's unclear there. >> >> >> I attached the script -- it is not well written (to say the very least), >> as I am new at muddling my way through the API. The most recent addition >> to it is the mouse listener which I added to allow the user to select three >> points near the end of the script. >> >> Best regards >> --aryeh >> >> If anyone else is interested in looking at the heap dumps I personally >> recommend the Eclipse Memory Analyzer plugin[2]. >> >> Best, >> Mark >> >> [1] https://gist.github.com/hinerm/727453aab0837246367e >> [2] https://eclipse.org/mat/downloads.php >> >> On Tue, Dec 8, 2015 at 2:25 AM, Aryeh Weiss < <[hidden email]> >> [hidden email]> wrote: >> >>> >>> Hi Mark >>> >>> On 07/12/2015 11:04 PM, Mark Hiner wrote: >>> >>> Hi Aryeh, >>> >>> On Mon, Dec 7, 2015 at 11:46 AM, Aryeh Weiss < <[hidden email]> >>> [hidden email]> wrote: >>> >>>> Thanks for this information. I decided to run IJ, do a heap dump before >>>> running a script that opens a few 1.3Gpixel images, then run it after, then >>>> close the images and get a heap dump when the images have been closed but >>>> the memory is not freed (which is reproducible). >>> >>> >>> This sounds like the perfect set of information! >>> >>> > In order to send it to you, I will probably need to send it to an FTP >>> server (like I have done with bio-formats. DO you have a server to which I >>> can send the dumps? >>> >>> Actually you should be able to use the Upload Sample Image[1] command to >>> send the zip. If that doesn't work for you let me know and I can send FTP >>> credentails. >>> >>> >>> I uploaded a set of heap dumps using Upload Sample Image as you >>> suggested. >>> They are: >>> heapdump-1449551945772 -- made immediately after launching Fiji. Small, >>> as expected. >>> heapdump-1449552198991 -- made immediately after running my python >>> script. Three large image are open. >>> heapdump-1449552667004 -- made after closing those three large images >>> and double clicking in the Fiji window >>> a few times to run the >>> garbage collector >>> heapdump-1449553044705 -- made after running the script a second time, >>> during which Fiji ran out of memory. >>> Two large images are open. >>> heapdump-1449553286368 -- made after closing those two windows and >>> garbage collecting. >>> As expected, memory was >>> not freed. >>> >>> The large images are a bit under 1Gpixel, 8-bit images. >>> >>> I hope that this is useful. >>> Thanks for your efforts. >>> --aryeh >>> >> > > -- > Aryeh Weiss > Faculty of Engineering > Bar Ilan University > Ramat Gan 52900 Israel > > Ph: 972-3-5317638 > FAX: 972-3-7384051 > > > -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
Re: Imagej and Java 1.8
|
|
In reply to this post by Aryeh Weiss
> On Dec 06, 2015; 5:24AM, Aryeh Weiss wrote:
> > I have noticed that when using Fiji with Java 1.8, the response > gradually gets slower, until eventually > I need to restart Fiji. The issue seems to be connected to screen > updates, but I cannot pin it down to a single item, although the ROI > Manager seems to often be involved. > For example, toggling "Show All" in the ROI manager tends to become > really slow. > > Now, the memory monitor shows 6.8GB in use, although I have very little > open. Double clicking on the Fiji window does not appear to have forced GC. I can reproduce this memory leak using the following Python script. The memory used by the 1GB image created by the script is not reclaimed when you click in the “Memory” window. Jython appears to be holding a reference to 'ip' after the script ends. Adding "ip=None" to the script does not help. One workaround is to use JavaScript. -wayne IJ.doCommand("Monitor Memory..."); img = IJ.createImage("Untitled","8-bit",32768,32768,1); ip=img.getProcessor() img.close() > I am just adding this update in the hope that these observations may > prove useful to the people who know more about Fiji and Java 1.8 than I. > If there are addition things that I can do to provide useful data for > this, please let me know. > > --aryeh -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
«
Return to ImageJ
|
1 view|%1 views
| Free forum by Nabble | Edit this page |