problem with loading approx. 2GB large data set into FIJI on a 64bit WIN7, 24GB RAM machine

[Standa Vinopal]

Hi all,

I acquire a lot of images on Olympus ScanR, which I need to adjust in ImageJ/Fiji before further analysis. I have a problem with loading a set of 800-900 separate 12-bit tif images, each 2.6 MB big, to FIJI or ImageJ. Loading of images starts normally, but after approx. 500-700 images it is slower and slower and finally it stops and FIJI freezes. When FIJI process is not killed in time, the whole system freezes and needs hard restart. The amount of used RAM by FIJI seen in Windows Task Manager reaches approx. 5,2 GB, Processor load is around 13% and fluctuates a little. There is plenty of system resources unused. It always stops at this value of FIJI-used memory with different data sets excluding the possibility that a wrong image causes the problem. Another interesting feature of this state is that the opened windows start blinking and appearing on the other display (we have two). IT guys do not know what could cause this, video card works normally in other applications and benchmarks. I switched off all the fancy visuals like Aero in Win7, but without an effect.

Could you help me to solve that? I know that I could load my images sequentially, but it is not why we bought high-performance computer. ;)

Our IT guys made some tests and concluded that computer hardware and software is OK. However, they were not sure about Java settings. Neither do I, because I do not understand it at all.

The problem appears both in FIJI 1.46p; Java 1.6.0_24 [64bit] and in ImageJ 1.46m; 1.6.0_20 [64bit]. Memory limit in ImageJ/Fiji is set to 18 GB.
Java installed in the computer is 1.7.0_04.

Computer:
QuadCore Intel core i7 960, 3200 MHz
Chipset: Intel Tylersburg x58, Intel Nehalem
24567 MB DDR3-1333 (6x4 GB)
Nvidia GeForce GTX 550Ti
180 GB SSD OCZ-VERTEX2 ATA
2TB WDC WD2003FYYs-02W0B0
OS: MS Win7 Enterprise, 64bit, version 6.1.7601, SP1; Aero switched off, only Basic Visual mode

Thanks in advance for your help!

Best wishes,

Standa

---------------------------------------------------------------
Stanislav Vinopal, Ph.D.
Laboratory of Biology of Cytoskeleton
Institute of Molecular Genetics ASCR, v.v.i.
Videnska 1083, 142 20 Prague 4
Czech Republic
phone: [phone]
e-mail: [hidden email]
www.img.cas.cz/dbc

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Michael Schmid]

Hi Standa,

looks like a problem of either windows or Java.

Having many hundreds of open windows is probably too much for the Windows operating system. Can you open the images as stack, or in BatchMode (when processing them in a macro), to avoid showing all of them in separate windows?  Anyhow, it won't be of much value to have 800 open windows on the screen at the same time.
Maybe the WindowTabs might also help (I have never tried it).


In case I misunderstood it and you don't have that many open windows, some suggestions to better understand the problem:

- What does Plugins>Utilities>Monitor Memory say? Also clicking on the ImageJ status bar might show you the amount of memory used (as long as it responds).

- What is the size of one of these images in ImageJ (I presume, they are loaded as 16-bit images)? Does the number of open images multiplied by the image size roughly correspond to the memory used by ImageJ and the memory shown in the Task Manager?
[you would need another byte per pixel for each displayed image]

Maybe you load a smaller dataset that does not freeze the computer for the tests.


Michael
________________________________________________________________
On Jun 7, 2012, at 14:22, Standa Vinopal wrote:

> Hi all,
>
> I acquire a lot of images on Olympus ScanR, which I need to adjust in ImageJ/Fiji before further analysis. I have a problem with loading a set of 800-900 separate 12-bit tif images, each 2.6 MB big, to FIJI or ImageJ. Loading of images starts normally, but after approx. 500-700 images it is slower and slower and finally it stops and FIJI freezes. When FIJI process is not killed in time, the whole system freezes and needs hard restart. The amount of used RAM by FIJI seen in Windows Task Manager reaches approx. 5,2 GB, Processor load is around 13% and fluctuates a little. There is plenty of system resources unused. It always stops at this value of FIJI-used memory with different data sets excluding the possibility that a wrong image causes the problem. Another interesting feature of this state is that the opened windows start blinking and appearing on the other display (we have two). IT guys do not know what could cause this, video card works normally in other applications and benchmarks. I switched off all the fancy visuals like Aero in Win7, but without an effect.
>
> Could you help me to solve that? I know that I could load my images sequentially, but it is not why we bought high-performance computer. ;)
>
> Our IT guys made some tests and concluded that computer hardware and software is OK. However, they were not sure about Java settings. Neither do I, because I do not understand it at all.
>
> The problem appears both in FIJI 1.46p; Java 1.6.0_24 [64bit] and in ImageJ 1.46m; 1.6.0_20 [64bit]. Memory limit in ImageJ/Fiji is set to 18 GB.
> Java installed in the computer is 1.7.0_04.
>
> Computer:
> QuadCore Intel core i7 960, 3200 MHz
> Chipset: Intel Tylersburg x58, Intel Nehalem
> 24567 MB DDR3-1333 (6x4 GB)
> Nvidia GeForce GTX 550Ti
> 180 GB SSD OCZ-VERTEX2 ATA
> 2TB WDC WD2003FYYs-02W0B0
> OS: MS Win7 Enterprise, 64bit, version 6.1.7601, SP1; Aero switched off, only Basic Visual mode
>
> Thanks in advance for your help!
>
> Best wishes,
>
> Standa
>
> ---------------------------------------------------------------
> Stanislav Vinopal, Ph.D.
> Laboratory of Biology of Cytoskeleton
> Institute of Molecular Genetics ASCR, v.v.i.
> Videnska 1083, 142 20 Prague 4
> Czech Republic
> phone: [phone]
> e-mail: [hidden email]
> www.img.cas.cz/dbc
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Standa Vinopal]

Hi Michael,

you are right, the problem probably is that I have many hundreds of open windows. I would like to avoid it, however, I do not know, how I could open many separate images as a stack.
I tried Import --> Open image Sequence,  or Import ––>Selected as a Stack and nothing works. You are also right that I could bypass this problem by writing a macro processing each image separately (I only need to adjust grey levels and then convert to 8bit), but I wonder whether there is some easier way to load these large data sets into ImageJ or FIJI.

BTW I could not find Monitor memory utility in both my installed ImageJ or Fiji...?

Cheers,

standa

______________________________________________________________

> Od: "Michael Schmid" <[hidden email]>
> Komu: <[hidden email]>
> Datum: 11.06.2012 13:50
> Předmět: Re: problem with loading approx. 2GB large data set into FIJI on a 64bit WIN7, 24GB RAM machine
>
>Hi Standa,
>
>looks like a problem of either windows or Java.
>
>Having many hundreds of open windows is probably too much for the Windows operating system. Can you open the images as stack, or in BatchMode (when processing them in a macro), to avoid showing all of them in separate windows?  Anyhow, it won't be of much value to have 800 open windows on the screen at the same time.
>Maybe the WindowTabs might also help (I have never tried it).
>
>
>In case I misunderstood it and you don't have that many open windows, some suggestions to better understand the problem:
>
>- What does Plugins>Utilities>Monitor Memory say? Also clicking on the ImageJ status bar might show you the amount of memory used (as long as it responds).
>
>- What is the size of one of these images in ImageJ (I presume, they are loaded as 16-bit images)? Does the number of open images multiplied by the image size roughly correspond to the memory used by ImageJ and the memory shown in the Task Manager?
>[you would need another byte per pixel for each displayed image]
>
>Maybe you load a smaller dataset that does not freeze the computer for the tests.
>
>
>Michael
>________________________________________________________________
>On Jun 7, 2012, at 14:22, Standa Vinopal wrote:
>
>> Hi all,
>>
>> I acquire a lot of images on Olympus ScanR, which I need to adjust in ImageJ/Fiji before further analysis. I have a problem with loading a set of 800-900 separate 12-bit tif images, each 2.6 MB big, to FIJI or ImageJ. Loading of images starts normally, but after approx. 500-700 images it is slower and slower and finally it stops and FIJI freezes. When FIJI process is not killed in time, the whole system freezes and needs hard restart. The amount of used RAM by FIJI seen in Windows Task Manager reaches approx. 5,2 GB, Processor load is around 13% and fluctuates a little. There is plenty of system resources unused. It always stops at this value of FIJI-used memory with different data sets excluding the possibility that a wrong image causes the problem. Another interesting feature of this state is that the opened windows start blinking and appearing on the other display (we have two). IT guys do not know what could cause this, video card works normally in other applications a

 nd benchmarks. I switched off all the fancy visuals like Aero in Win7, but without an effect.

>>
>> Could you help me to solve that? I know that I could load my images sequentially, but it is not why we bought high-performance computer. ;)
>>
>> Our IT guys made some tests and concluded that computer hardware and software is OK. However, they were not sure about Java settings. Neither do I, because I do not understand it at all.
>>
>> The problem appears both in FIJI 1.46p; Java 1.6.0_24 [64bit] and in ImageJ 1.46m; 1.6.0_20 [64bit]. Memory limit in ImageJ/Fiji is set to 18 GB.
>> Java installed in the computer is 1.7.0_04.
>>
>> Computer:
>> QuadCore Intel core i7 960, 3200 MHz
>> Chipset: Intel Tylersburg x58, Intel Nehalem
>> 24567 MB DDR3-1333 (6x4 GB)
>> Nvidia GeForce GTX 550Ti
>> 180 GB SSD OCZ-VERTEX2 ATA
>> 2TB WDC WD2003FYYs-02W0B0
>> OS: MS Win7 Enterprise, 64bit, version 6.1.7601, SP1; Aero switched off, only Basic Visual mode
>>
>> Thanks in advance for your help!
>>
>> Best wishes,
>>
>> Standa
>>
>> ---------------------------------------------------------------
>> Stanislav Vinopal, Ph.D.
>> Laboratory of Biology of Cytoskeleton
>> Institute of Molecular Genetics ASCR, v.v.i.
>> Videnska 1083, 142 20 Prague 4
>> Czech Republic
>> phone: [phone]
>> e-mail: [hidden email]
>> www.img.cas.cz/dbc
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
>--
>ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Michael Schmid]

Hi Standa,

as far as I know, Import>Open Image Sequence should work with all image types that are supported by the 'open' command of ImageJ (or drag and drop onto ImageJ).

If it's an external plugin that reads your files, you can create a macro that opens the files one by one and adds them to a stack (assuming they are all the same size).  It should be reasonably fast in BatchMode.  Or maybe you can modify the plugin to read several files into a stack (assuming you have the source code)?

Plugins>Utilities>Monitor Memory - no idea why you don't see it.
  http://rsb.info.nih.gov/ij/docs/guide/146-31.html#toc-Subsubsection-31.3.5

Michael
________________________________________________________________
On Jun 14, 2012, at 13:32, Standa Vinopal wrote:

> Hi Michael,
>
> you are right, the problem probably is that I have many hundreds of open windows. I would like to avoid it, however, I do not know, how I could open many separate images as a stack.
> I tried Import --> Open image Sequence,  or Import ––>Selected as a Stack and nothing works. You are also right that I could bypass this problem by writing a macro processing each image separately (I only need to adjust grey levels and then convert to 8bit), but I wonder whether there is some easier way to load these large data sets into ImageJ or FIJI.
>
> BTW I could not find Monitor memory utility in both my installed ImageJ or Fiji...?
>
> Cheers,
>
> standa
>
> ______________________________________________________________
>> Od: "Michael Schmid" <[hidden email]>
>> Komu: <[hidden email]>
>> Datum: 11.06.2012 13:50
>> Předmět: Re: problem with loading approx. 2GB large data set into FIJI on a 64bit WIN7, 24GB RAM machine
>>
>> Hi Standa,
>>
>> looks like a problem of either windows or Java.
>>
>> Having many hundreds of open windows is probably too much for the Windows operating system. Can you open the images as stack, or in BatchMode (when processing them in a macro), to avoid showing all of them in separate windows?  Anyhow, it won't be of much value to have 800 open windows on the screen at the same time.
>> Maybe the WindowTabs might also help (I have never tried it).
>>
>>
>> In case I misunderstood it and you don't have that many open windows, some suggestions to better understand the problem:
>>
>> - What does Plugins>Utilities>Monitor Memory say? Also clicking on the ImageJ status bar might show you the amount of memory used (as long as it responds).
>>
>> - What is the size of one of these images in ImageJ (I presume, they are loaded as 16-bit images)? Does the number of open images multiplied by the image size roughly correspond to the memory used by ImageJ and the memory shown in the Task Manager?
>> [you would need another byte per pixel for each displayed image]
>>
>> Maybe you load a smaller dataset that does not freeze the computer for the tests.
>>
>>
>> Michael
>> ________________________________________________________________
>> On Jun 7, 2012, at 14:22, Standa Vinopal wrote:
>>
>>> Hi all,
>>>
>>> I acquire a lot of images on Olympus ScanR, which I need to adjust in ImageJ/Fiji before further analysis. I have a problem with loading a set of 800-900 separate 12-bit tif images, each 2.6 MB big, to FIJI or ImageJ. Loading of images starts normally, but after approx. 500-700 images it is slower and slower and finally it stops and FIJI freezes. When FIJI process is not killed in time, the whole system freezes and needs hard restart. The amount of used RAM by FIJI seen in Windows Task Manager reaches approx. 5,2 GB, Processor load is around 13% and fluctuates a little. There is plenty of system resources unused. It always stops at this value of FIJI-used memory with different data sets excluding the possibility that a wrong image causes the problem. Another interesting feature of this state is that the opened windows start blinking and appearing on the other display (we have two). IT guys do not know what could cause this, video card works normally in other applications a
> nd benchmarks. I switched off all the fancy visuals like Aero in Win7, but without an effect.
>>>
>>> Could you help me to solve that? I know that I could load my images sequentially, but it is not why we bought high-performance computer. ;)
>>>
>>> Our IT guys made some tests and concluded that computer hardware and software is OK. However, they were not sure about Java settings. Neither do I, because I do not understand it at all.
>>>
>>> The problem appears both in FIJI 1.46p; Java 1.6.0_24 [64bit] and in ImageJ 1.46m; 1.6.0_20 [64bit]. Memory limit in ImageJ/Fiji is set to 18 GB.
>>> Java installed in the computer is 1.7.0_04.
>>>
>>> Computer:
>>> QuadCore Intel core i7 960, 3200 MHz
>>> Chipset: Intel Tylersburg x58, Intel Nehalem
>>> 24567 MB DDR3-1333 (6x4 GB)
>>> Nvidia GeForce GTX 550Ti
>>> 180 GB SSD OCZ-VERTEX2 ATA
>>> 2TB WDC WD2003FYYs-02W0B0
>>> OS: MS Win7 Enterprise, 64bit, version 6.1.7601, SP1; Aero switched off, only Basic Visual mode
>>>
>>> Thanks in advance for your help!
>>>
>>> Best wishes,
>>>
>>> Standa
>>>
>>> ---------------------------------------------------------------
>>> Stanislav Vinopal, Ph.D.
>>> Laboratory of Biology of Cytoskeleton
>>> Institute of Molecular Genetics ASCR, v.v.i.
>>> Videnska 1083, 142 20 Prague 4
>>> Czech Republic
>>> phone: [phone]
>>> e-mail: [hidden email]
>>> www.img.cas.cz/dbc
>>>
>>> --
>>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[ctrueden]

Hi Standa,

> I tried Import --> Open image Sequence,  or Import ––>Selected as a Stack
and nothing works.

Did you try using Bio-Formats to open your Olympus ScanR data? It should
handle your datasets more intelligently than Import Image Sequence does.

Bio-Formats is bundled with Fiji (http://fiji.sc/), our can be installed
into vanilla ImageJ (http://loci.wisc.edu/bio-formats/imagej).

Regards,
Curtis


On Thu, Jun 14, 2012 at 11:22 AM, Michael Schmid <[hidden email]>wrote:

> Hi Standa,
>
> as far as I know, Import>Open Image Sequence should work with all image
> types that are supported by the 'open' command of ImageJ (or drag and drop
> onto ImageJ).
>
> If it's an external plugin that reads your files, you can create a macro
> that opens the files one by one and adds them to a stack (assuming they are
> all the same size).  It should be reasonably fast in BatchMode.  Or maybe
> you can modify the plugin to read several files into a stack (assuming you
> have the source code)?
>
> Plugins>Utilities>Monitor Memory - no idea why you don't see it.
>
> http://rsb.info.nih.gov/ij/docs/guide/146-31.html#toc-Subsubsection-31.3.5
>
> Michael
> ________________________________________________________________
> On Jun 14, 2012, at 13:32, Standa Vinopal wrote:
>
> > Hi Michael,
> >
> > you are right, the problem probably is that I have many hundreds of open
> windows. I would like to avoid it, however, I do not know, how I could open
> many separate images as a stack.
> > I tried Import --> Open image Sequence,  or Import ––>Selected as a
> Stack and nothing works. You are also right that I could bypass this
> problem by writing a macro processing each image separately (I only need to
> adjust grey levels and then convert to 8bit), but I wonder whether there is
> some easier way to load these large data sets into ImageJ or FIJI.
> >
> > BTW I could not find Monitor memory utility in both my installed ImageJ
> or Fiji...?
> >
> > Cheers,
> >
> > standa
> >
> > ______________________________________________________________
> >> Od: "Michael Schmid" <[hidden email]>
> >> Komu: <[hidden email]>
> >> Datum: 11.06.2012 13:50
> >> Předmět: Re: problem with loading approx. 2GB large data set into FIJI
> on a 64bit WIN7, 24GB RAM machine
> >>
> >> Hi Standa,
> >>
> >> looks like a problem of either windows or Java.
> >>
> >> Having many hundreds of open windows is probably too much for the
> Windows operating system. Can you open the images as stack, or in BatchMode
> (when processing them in a macro), to avoid showing all of them in separate
> windows?  Anyhow, it won't be of much value to have 800 open windows on the
> screen at the same time.
> >> Maybe the WindowTabs might also help (I have never tried it).
> >>
> >>
> >> In case I misunderstood it and you don't have that many open windows,
> some suggestions to better understand the problem:
> >>
> >> - What does Plugins>Utilities>Monitor Memory say? Also clicking on the
> ImageJ status bar might show you the amount of memory used (as long as it
> responds).
> >>
> >> - What is the size of one of these images in ImageJ (I presume, they
> are loaded as 16-bit images)? Does the number of open images multiplied by
> the image size roughly correspond to the memory used by ImageJ and the
> memory shown in the Task Manager?
> >> [you would need another byte per pixel for each displayed image]
> >>
> >> Maybe you load a smaller dataset that does not freeze the computer for
> the tests.
> >>
> >>
> >> Michael
> >> ________________________________________________________________
> >> On Jun 7, 2012, at 14:22, Standa Vinopal wrote:
> >>
> >>> Hi all,
> >>>
> >>> I acquire a lot of images on Olympus ScanR, which I need to adjust in
> ImageJ/Fiji before further analysis. I have a problem with loading a set of
> 800-900 separate 12-bit tif images, each 2.6 MB big, to FIJI or ImageJ.
> Loading of images starts normally, but after approx. 500-700 images it is
> slower and slower and finally it stops and FIJI freezes. When FIJI process
> is not killed in time, the whole system freezes and needs hard restart. The
> amount of used RAM by FIJI seen in Windows Task Manager reaches approx. 5,2
> GB, Processor load is around 13% and fluctuates a little. There is plenty
> of system resources unused. It always stops at this value of FIJI-used
> memory with different data sets excluding the possibility that a wrong
> image causes the problem. Another interesting feature of this state is that
> the opened windows start blinking and appearing on the other display (we
> have two). IT guys do not know what could cause this, video card works
> normally in other applications a
> > nd benchmarks. I switched off all the fancy visuals like Aero in Win7,
> but without an effect.
> >>>
> >>> Could you help me to solve that? I know that I could load my images
> sequentially, but it is not why we bought high-performance computer. ;)
> >>>
> >>> Our IT guys made some tests and concluded that computer hardware and
> software is OK. However, they were not sure about Java settings. Neither do
> I, because I do not understand it at all.
> >>>
> >>> The problem appears both in FIJI 1.46p; Java 1.6.0_24 [64bit] and in
> ImageJ 1.46m; 1.6.0_20 [64bit]. Memory limit in ImageJ/Fiji is set to 18 GB.
> >>> Java installed in the computer is 1.7.0_04.
> >>>
> >>> Computer:
> >>> QuadCore Intel core i7 960, 3200 MHz
> >>> Chipset: Intel Tylersburg x58, Intel Nehalem
> >>> 24567 MB DDR3-1333 (6x4 GB)
> >>> Nvidia GeForce GTX 550Ti
> >>> 180 GB SSD OCZ-VERTEX2 ATA
> >>> 2TB WDC WD2003FYYs-02W0B0
> >>> OS: MS Win7 Enterprise, 64bit, version 6.1.7601, SP1; Aero switched
> off, only Basic Visual mode
> >>>
> >>> Thanks in advance for your help!
> >>>
> >>> Best wishes,
> >>>
> >>> Standa
> >>>
> >>> ---------------------------------------------------------------
> >>> Stanislav Vinopal, Ph.D.
> >>> Laboratory of Biology of Cytoskeleton
> >>> Institute of Molecular Genetics ASCR, v.v.i.
> >>> Videnska 1083, 142 20 Prague 4
> >>> Czech Republic
> >>> phone: [phone]
> >>> e-mail: [hidden email]
> >>> www.img.cas.cz/dbc
> >>>
> >>> --
> >>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >>
> >> --
> >> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >>
> >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html