Re: Issues and annoyances with trakem2 working on a gigapixel montage
Posted by Saalfeld, Stephan on Feb 09, 2017; 4:33pm
URL: http://imagej.273.s1.nabble.com/Issues-and-annoyances-with-trakem2-working-on-a-gigapixel-montage-tp5018056p5018072.html
No need to feel embarrassed. The script is pretty messy and not too
well documented.
Best,
Stephan
On Thu, 2017-02-09 at 17:46 +0200, Stoyan Pavlov wrote:
> Thanks! I feel so embarrassed right now.
>
> На 9.02.2017 г. 5:04 сл.об. "Saalfeld, Stephan" <[hidden email]
> mi.org>
> написа:
>
> >
> > Yes,
> >
> > by changing this line
> >
> > https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/catm
> > aid-
> > export2.bsh#L96
> >
> > to
> >
> > ImagePlus.COLOR_RGB,
> >
> >
> >
> > (this is where the fork is made
> >
> > https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/main/ja
> > va/i
> > ni/trakem2/persistence/Loader.java#L2815-L2820
> >
> > )
> >
> > Best,
> > Stephan
> >
> >
> > On Thu, 2017-02-09 at 13:22 +0200, Stoyan Pavlov wrote:
> > >
> > > Dear Stephan,
> > > I tried the bsh script and it works very well and fast, but it
> > > exports the
> > > images in gray scale. Is there an easy way to make it export
> > > them in
> > > RGB?
> > > Best regards
> > > Stoyan
> > >
> > > На 8.02.2017 г. 18:30 "Stoyan Pavlov" <[hidden email]>
> > > написа:
> > >
> > > >
> > > >
> > > > Hi Stephan,
> > > > Thank you very much! I will try the scripts, first thing in the
> > > > morning. I
> > > > figured as much. After all exactly this 2G java limitation for
> > > > arrays is
> > > > what brought me to trakem2. ;-)
> > > >
> > > > >> 1. Is your project directory on a file share?
> > > >
> > > > No. Everything is saved locally.
> > > >
> > > >
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > 2. If you do not want to use mipmaps, switch mipmaps >>off
> > > > > > *before*
> > > > > > importing tiles. Switching them off during the madness
> > > > > > >>has
> > > > > > no effects
> > > > Indeed I observed this behaviour: even if one cancels the
> > > > mipmap
> > > > generation job, it continues in Fiji until finished. Only
> > > > workaround is to
> > > > quit Fiji . But when the project is loaded it tries to
> > > > regenerate
> > > > the
> > > > mipmaps...
> > > >
> > > > I guess the java loader alternative is used the sameway any
> > > > other
> > > > script?
> > > > (New>Script etc...)
> > > >
> > > > I'll let you know whether it all worked.
> > > >
> > > > Best regards,
> > > > Stoyan
> > > >
> > > > На 8.02.2017 г. 5:22 сл.об. "Saalfeld, Stephan" <
> > > > [hidden email]> написа:
> > > >
> > > > Hi,
> > > >
> > > > TrakEM2's internal CATMAID export mechanisms goes through
> > > > single
> > > > image
> > > > exports. This is limited to 2GPx planes (sad---
> > > > right?). Please
> > > > use
> > > > this script instead:
> > > >
> > > > https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/
> > > > catm
> > > > aid-
> > > > export2.bsh
> > > >
> > > > Regarding the mipmap-madness...:
> > > >
> > > > 1. Is your project directory on a file share? If yes, please
> > > > move
> > > > it
> > > > to a local (external?) harddrive, you are working with 2D
> > > > images,
> > > > that
> > > > should fit. TrakEM2 is somehow sensitive to how network drives
> > > > frequently fail or rather delay what they are supposed to
> > > > do. This
> > > > is
> > > > an annoyance in TrakEM2 and could/ should be fixed. No time
> > > > though...
> > > >
> > > > 2. If you do not want to use mipmaps, switch mipmaps off
> > > > *before*
> > > > importing tiles. Switching them off during the madness has no
> > > > effects,
> > > > too many threads running already and they will not get properly
> > > > interrupted.
> > > >
> > > > 3. JPG import works faster because JPGs read faster, this
> > > > sounds
> > > > like
> > > > you are I/O limited, which makes me believe in 1. more ;). Oh
> > > > yes,
> > > > are
> > > > all your images the same size and have intensities 0
> > > > --255? Then
> > > > you
> > > > can use the extended import file syntax
> > > >
> > > > https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/
> > > > main/java/ini/trakem2/persistence/Loader.java#L2013-L2027
> > > >
> > > > if you specify the columns 5--9 and switch off mipmaps, none of
> > > > your
> > > > images will be touched during import, i.e. import will be
> > > > almost
> > > > instantaneous. Save the project, do the rest.
> > > >
> > > > Let me know if that helps or where it breaks.
> > > >
> > > > Cheers,
> > > > Stephan
> > > >
> > > > On Wed, 2017-02-08 at 16:23 +0200, Stoyan Pavlov wrote:
> > > > >
> > > > >
> > > > > Hello Curtis,
> > > > > Thank you very much. I will wait a while for an answer. Last
> > > > > time
> > > > > they
> > > > > answered pretty quickly. If however there's no movement on
> > > > > this I
> > > > > will
> > > > > probably accept your offer to file the bug report as
> > > > > proposed,
> > > > > because I
> > > > > am not familiar with github and coding.
> > > > >
> > > > > Best regards!
> > > > > Stoyan
> > > > >
> > > > > P.S. Sorry about the outbirst from few minutes ago!
> > > > > На 8.02.2017 г. 1:40 сл.об. "Curtis Rueden" <[hidden email]
> > > > > u>
> > > > > написа:
> > > > >
> > > > > Hi Stoyan, Herbie, everyone,
> > > > >
> > > > > Stoyan wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > First I want to emphasize, how astonishing and useful I
> > > > > > find
> > > > > > the
> > > > > > trakem2 package/plugin! It is really great and I am really
> > > > > > thankful
> > > > > > to
> > > > > > the developers for puting all this effort into creating
> > > > > > this
> > > > > > suite
> > > > > > for
> > > > > > working with large image datasets.
> > > > > Glad to hear it is useful for you.
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Recently I stumbled upon several issues
> > > > > Thank you for the detailed report. Hopefully Albert or
> > > > > Stephan
> > > > > will
> > > > > be able
> > > > > to comment. At minimum, we should file these as issues in the
> > > > > GitHub
> > > > > repository, so they are not forgotten. If you do not hear
> > > > > back
> > > > > further from
> > > > > anyone, you can file them at https://github.com/trakem2/TrakE
> > > > > M2/i
> > > > > ssue
> > > > > s, or
> > > > > ping me and I will take care of it.
> > > > >
> > > > > Herbie wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > I have no idea about "trakem2"
> > > > > See this page for details:
> > > > > http://imagej.net/TrakEM2
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > JEPG compression in 99% of all cases is a lossy method and
> > > > > > more
> > > > > > or
> > > > > > less changes the image content, i.e. it introduces
> > > > > > artifacts.
> > > > > Indeed—there is a discussion about that here:
> > > > >
> > > > > http://imagej.net/Principles#Why_.28lossy.29_JPEGs_should_
> > > > > not_be_used_in_imaging
> > > > >
> > > > > That said, there are cases where it is acceptable to use
> > > > > JPEG.
> > > > > Essentially,
> > > > > if you have massively large data, and you do not plan to
> > > > > analyze
> > > > > quantitatively for the most part, it might be OK for
> > > > > pragmatic
> > > > > reasons, as
> > > > > long as you understand what you are doing.
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > With respect to "trakem2"-issues I'd contact the authors
> > > > > > directly.
> > > > > I suggest keeping the discussion public. From the Help page (
> > > > > http://imagej.net/Help):
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > The ImageJ community firmly believes in public discussion
> > > > > > of
> > > > > > the
> > > > > > software. In this way, each question benefits not only the
> > > > > > one
> > > > > > asking,
> > > > > > but everyone in the community, including everyone who
> > > > > > subsequently
> > > > > > does a web search on the same topic.
> > > > > Regards,
> > > > > Curtis
> > > > >
> > > > > --
> > > > > Curtis Rueden
> > > > > LOCI software architect - https://loci.wisc.edu/software
> > > > > ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Ruede
> > > > > n
> > > > > Did you know ImageJ has a forum? http://forum.imagej.net/
> > > > >
> > > > >
> > > > > On Wed, Feb 8, 2017 at 4:21 AM, Herbie <[hidden email]>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Good day Stoyan,
> > > > > >
> > > > > > I have no idea about "trakem2" but please let me remark
> > > > > > that
> > > > > > using
> > > > > > JEPG-compressed images for scientific purposes is an
> > > > > > absolute
> > > > > > no-
> > > > > > no. JEPG
> > > > > > compression in 99% of all cases is a lossy method and more
> > > > > > or
> > > > > > less
> > > > > > changes
> > > > > > the image content, i.e. it introduces artifacts.
> > > > > >
> > > > > > With respect to "trakem2"-issues I'd contact the authors
> > > > > > directly.
> > > > > >
> > > > > > Best
> > > > > >
> > > > > > Herbie
> > > > > >
> > > > > > :::::::::::::::::::::::::::::::::::::::::::
> > > > > > Am 08.02.17 um 10:40 schrieb Stoyan Pavlov:
> > > > > >
> > > > > > Dear friends and colleagues,
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > First I want to emphasize, how astonishing and useful I
> > > > > > > find
> > > > > > > the
> > > > > > > trakem2
> > > > > > > package/plugin! It is really great and I am really
> > > > > > > thankful
> > > > > > > to
> > > > > > > the
> > > > > > > developers for puting all this effort into creating this
> > > > > > > suite
> > > > > > > for
> > > > > working
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > with large image datasets.
> > > > > > > I use it mainly for creating/stitching/blending ,
> > > > > > > processing
> > > > > > > and
> > > > > exporting
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > of gigapixel montages of microscopic images into a
> > > > > > > readable
> > > > > > > software
> > > > > > > independent image pyramid.
> > > > > > > Recently I stumbled upon several issues (I don't know
> > > > > > > whether
> > > > > > > they are
> > > > > > > bugs
> > > > > > > or just due to limitation in the software) for which I
> > > > > > > couldn't
> > > > > > > find any
> > > > > > > mention and/or solution in the documentation and the
> > > > > > > forum.
> > > > > > >
> > > > > > > First I will describe what I am doing and next I will
> > > > > > > list
> > > > > > > the
> > > > > > > issues
> > > > > that
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > bother me.
> > > > > > > I am creating a single layer montage with dimensions
> > > > > > > 63698 x
> > > > > > > 57218 px
> > > > > from
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > a set of 3111 images (each 1388x1040). The images are
> > > > > > > acquired
> > > > > > > via
> > > > > > > axiovision and exported as single tiles (24bit RGB in
> > > > > > > either
> > > > > > > jpg
> > > > > > > or tif
> > > > > > > format). Before importing the images I align them using
> > > > > > > the
> > > > > > > MIST
> > > > > > > plugin
> > > > > > > and
> > > > > > > use the global positions generated from it to import them
> > > > > > > into
> > > > > > > trakem2
> > > > > via
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > text file. The resulting montage is exactly as expected
> > > > > > > and
> > > > > > > have
> > > > > > > no
> > > > > > > remarks
> > > > > > > about it.
> > > > > > > 1) I don't know if its true but it seems to me that if I
> > > > > > > use
> > > > > > > jpg
> > > > > > > as
> > > > > format
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > for the source images they are imported faster into
> > > > > > > trakem2.
> > > > > > > However once
> > > > > > > the import is finished the software returns errors about
> > > > > > > generating
> > > > > > > mipmaps
> > > > > > > (see attached log file) and it continuously tries to
> > > > > > > regenerate
> > > > > > > them
> > > > > > > (which
> > > > > > > I guess keeps CPU quite busy). This happens regardless
> > > > > > > mipmaps
> > > > > > > are
> > > > > enabled
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > or disabled in Display>Properties, and is independent
> > > > > > > from
> > > > > > > the
> > > > > > > number of
> > > > > > > threads assigned to mipmap generation. If I close the
> > > > > > > project
> > > > > > > and
> > > > > > > Fiji
> > > > > > > (even after system restart) and then I open them again
> > > > > > > the
> > > > > > > issue
> > > > > > > returns.
> > > > > > > This behaviour is not oberved if my source images are in
> > > > > > > tif
> > > > > > > format.
> > > > > > > The other issues are related to the export of a flat
> > > > > > > image
> > > > > > > (here
> > > > > > > the file
> > > > > > > format of the source images does not matter):
> > > > > > >
> > > > > > > 2) When I am trying to export to a tiled format (Export>
> > > > > > > Make
> > > > > > > flat
> > > > > > > image...>Save for web (CATMAID)) regardless of the image
> > > > > > > format I
> > > > > > > choose
> > > > > > > for the export I cannot change the size of the exported
> > > > > > > tiles
> > > > > > > to
> > > > > > > other
> > > > > > > than
> > > > > > > 256x256. The dialogue allows me to change it, but when
> > > > > > > the
> > > > > > > process
> > > > > > > finishes
> > > > > > > in the folder there are only 256x256 tile images. While
> > > > > > > this
> > > > > > > is
> > > > > > > not a
> > > > > huge
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > problem, still there is difference between managing some
> > > > > > > 80000
> > > > > > > images or
> > > > > > > 40000.
> > > > > > >
> > > > > > > 3) Also when I am trying to export tiled format via
> > > > > > > Export>
> > > > > > > Make
> > > > > > > flat
> > > > > > > image...>Save for web (CATMAID), if the patches are
> > > > > > > deselected
> > > > > > > only a
> > > > > > > portion of the image is exported . Regardless of the
> > > > > > > format
> > > > > > > of
> > > > > > > the source
> > > > > > > images and of the exported images there are only 63 rows
> > > > > > > by
> > > > > > > 63
> > > > > > > columns of
> > > > > > > 256x256 tiles at zoom level 0 in the folder with the
> > > > > > > exported
> > > > > > > images. All
> > > > > > > metadata in the json file is correct and there are no
> > > > > > > errors
> > > > > > > reported by
> > > > > > > the System log or the Fiji Log. If the patches are
> > > > > > > selected
> > > > > > > with
> > > > > > > Select
> > > > > > > All
> > > > > > > (Ctrl + A) the image is exported in its entirety, however
> > > > > > > here
> > > > > > > comes the
> > > > > > > next not as much as an issue but slight discomfort.
> > > > > > >
> > > > > > > 4) The image is always exported to a quadrangular array
> > > > > > > consisting (for
> > > > > > > the
> > > > > > > reasons already described) of 256 rows and 256 columns of
> > > > > > > 256x256
> > > > > > > tiles
> > > > > at
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > zoom level 0. That is the image dimensions (63698 x 57218
> > > > > > > px)
> > > > > > > are
> > > > > expanded
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > to 65536x65536 and filled with black tiles. This of
> > > > > > > course
> > > > > > > take
> > > > > > > up
> > > > > > > additional space and need to be cropped away before
> > > > > > > uploading
> > > > > > > the
> > > > > > > image
> > > > > to
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > our servers. I understand that this is probably because
> > > > > > > of
> > > > > > > limitations to
> > > > > > > the algorithm but still if there is some workaround it
> > > > > > > would
> > > > > > > have
> > > > > > > been
> > > > > > > better.
> > > > > > >
> > > > > > > 5) I thought that this problems (2-4) might be due to the
> > > > > > > unequal
> > > > > > > dimensions of the original source images and I tried to
> > > > > > > work
> > > > > > > around them
> > > > > > > by
> > > > > > > creating a sibling project with retiled images. Here I
> > > > > > > had
> > > > > > > another
> > > > > > > unexpected result - the image was exported but the larger
> > > > > > > dimension was
> > > > > > > cut
> > > > > > > by about 6000 pxs from the right side and than restored
> > > > > > > to
> > > > > > > the
> > > > > > > original
> > > > > > > size by adding black pixels , which basically cuts away
> > > > > > > the
> > > > > > > right
> > > > > > > portion
> > > > > > > of my image (see attached screenshot) .
> > > > > > >
> > > > > > > The workstation that I use has the following parameters:
> > > > > > > CPU: Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core
> > > > > > > processors)
> > > > > > > RAM: 64 GB
> > > > > > > OS: Windows 7 64bit
> > > > > > > In an attachment I added the ImageJ-properties output,
> > > > > > > portion of
> > > > > > > the
> > > > > > > trakem2 log with the errors generating the mipmaps, and a
> > > > > > > screenshot of
> > > > > > > the
> > > > > > > output of the sibling project with retiled images.
> > > > > > >
> > > > > > > I hope someone will find a way to correct these problems,
> > > > > > > because
> > > > > > > surely
> > > > > > > they are not fatal and there are workarounds that we use,
> > > > > > > but
> > > > > > > it
> > > > > > > is
> > > > > > > certainly annoying.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Stoyan
> > > > > > >
> > > > > > > ---
> > > > > > > Dr. Stoyan P. Pavlov, MD, PhD
> > > > > > > Departament of Anatomy, Histology and Embryology
> > > > > > > Medical University "Prof. Dr. Paraskev Stoyanov", Varna
> > > > > > > Prof. Marin Drinov Str.55
> > > > > > > 9002 Varna
> > > > > > > Bulgaria
> > > > > > > Tel: +359 (0) 52 - 677 - 052 <052%20677%20052>
> > > > > > > e-mail: [hidden email]
> > > > > > > [hidden email]
> > > > > > >
> > > > > > > --
> > > > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > > > >
> > > > > > >
> > > > > > --
> > > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > > >
> > > > > --
> > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > >
> > > > > --
> > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > --
> > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > >
> > > >
> > > >
> > > --
> > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
URL: http://imagej.273.s1.nabble.com/Issues-and-annoyances-with-trakem2-working-on-a-gigapixel-montage-tp5018056p5018072.html
No need to feel embarrassed. The script is pretty messy and not too
well documented.
Best,
Stephan
On Thu, 2017-02-09 at 17:46 +0200, Stoyan Pavlov wrote:
> Thanks! I feel so embarrassed right now.
>
> На 9.02.2017 г. 5:04 сл.об. "Saalfeld, Stephan" <[hidden email]
> mi.org>
> написа:
>
> >
> > Yes,
> >
> > by changing this line
> >
> > https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/catm
> > aid-
> > export2.bsh#L96
> >
> > to
> >
> > ImagePlus.COLOR_RGB,
> >
> >
> >
> > (this is where the fork is made
> >
> > https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/main/ja
> > va/i
> > ni/trakem2/persistence/Loader.java#L2815-L2820
> >
> > )
> >
> > Best,
> > Stephan
> >
> >
> > On Thu, 2017-02-09 at 13:22 +0200, Stoyan Pavlov wrote:
> > >
> > > Dear Stephan,
> > > I tried the bsh script and it works very well and fast, but it
> > > exports the
> > > images in gray scale. Is there an easy way to make it export
> > > them in
> > > RGB?
> > > Best regards
> > > Stoyan
> > >
> > > На 8.02.2017 г. 18:30 "Stoyan Pavlov" <[hidden email]>
> > > написа:
> > >
> > > >
> > > >
> > > > Hi Stephan,
> > > > Thank you very much! I will try the scripts, first thing in the
> > > > morning. I
> > > > figured as much. After all exactly this 2G java limitation for
> > > > arrays is
> > > > what brought me to trakem2. ;-)
> > > >
> > > > >> 1. Is your project directory on a file share?
> > > >
> > > > No. Everything is saved locally.
> > > >
> > > >
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > 2. If you do not want to use mipmaps, switch mipmaps >>off
> > > > > > *before*
> > > > > > importing tiles. Switching them off during the madness
> > > > > > >>has
> > > > > > no effects
> > > > Indeed I observed this behaviour: even if one cancels the
> > > > mipmap
> > > > generation job, it continues in Fiji until finished. Only
> > > > workaround is to
> > > > quit Fiji . But when the project is loaded it tries to
> > > > regenerate
> > > > the
> > > > mipmaps...
> > > >
> > > > I guess the java loader alternative is used the sameway any
> > > > other
> > > > script?
> > > > (New>Script etc...)
> > > >
> > > > I'll let you know whether it all worked.
> > > >
> > > > Best regards,
> > > > Stoyan
> > > >
> > > > На 8.02.2017 г. 5:22 сл.об. "Saalfeld, Stephan" <
> > > > [hidden email]> написа:
> > > >
> > > > Hi,
> > > >
> > > > TrakEM2's internal CATMAID export mechanisms goes through
> > > > single
> > > > image
> > > > exports. This is limited to 2GPx planes (sad---
> > > > right?). Please
> > > > use
> > > > this script instead:
> > > >
> > > > https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/
> > > > catm
> > > > aid-
> > > > export2.bsh
> > > >
> > > > Regarding the mipmap-madness...:
> > > >
> > > > 1. Is your project directory on a file share? If yes, please
> > > > move
> > > > it
> > > > to a local (external?) harddrive, you are working with 2D
> > > > images,
> > > > that
> > > > should fit. TrakEM2 is somehow sensitive to how network drives
> > > > frequently fail or rather delay what they are supposed to
> > > > do. This
> > > > is
> > > > an annoyance in TrakEM2 and could/ should be fixed. No time
> > > > though...
> > > >
> > > > 2. If you do not want to use mipmaps, switch mipmaps off
> > > > *before*
> > > > importing tiles. Switching them off during the madness has no
> > > > effects,
> > > > too many threads running already and they will not get properly
> > > > interrupted.
> > > >
> > > > 3. JPG import works faster because JPGs read faster, this
> > > > sounds
> > > > like
> > > > you are I/O limited, which makes me believe in 1. more ;). Oh
> > > > yes,
> > > > are
> > > > all your images the same size and have intensities 0
> > > > --255? Then
> > > > you
> > > > can use the extended import file syntax
> > > >
> > > > https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/
> > > > main/java/ini/trakem2/persistence/Loader.java#L2013-L2027
> > > >
> > > > if you specify the columns 5--9 and switch off mipmaps, none of
> > > > your
> > > > images will be touched during import, i.e. import will be
> > > > almost
> > > > instantaneous. Save the project, do the rest.
> > > >
> > > > Let me know if that helps or where it breaks.
> > > >
> > > > Cheers,
> > > > Stephan
> > > >
> > > > On Wed, 2017-02-08 at 16:23 +0200, Stoyan Pavlov wrote:
> > > > >
> > > > >
> > > > > Hello Curtis,
> > > > > Thank you very much. I will wait a while for an answer. Last
> > > > > time
> > > > > they
> > > > > answered pretty quickly. If however there's no movement on
> > > > > this I
> > > > > will
> > > > > probably accept your offer to file the bug report as
> > > > > proposed,
> > > > > because I
> > > > > am not familiar with github and coding.
> > > > >
> > > > > Best regards!
> > > > > Stoyan
> > > > >
> > > > > P.S. Sorry about the outbirst from few minutes ago!
> > > > > На 8.02.2017 г. 1:40 сл.об. "Curtis Rueden" <[hidden email]
> > > > > u>
> > > > > написа:
> > > > >
> > > > > Hi Stoyan, Herbie, everyone,
> > > > >
> > > > > Stoyan wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > First I want to emphasize, how astonishing and useful I
> > > > > > find
> > > > > > the
> > > > > > trakem2 package/plugin! It is really great and I am really
> > > > > > thankful
> > > > > > to
> > > > > > the developers for puting all this effort into creating
> > > > > > this
> > > > > > suite
> > > > > > for
> > > > > > working with large image datasets.
> > > > > Glad to hear it is useful for you.
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Recently I stumbled upon several issues
> > > > > Thank you for the detailed report. Hopefully Albert or
> > > > > Stephan
> > > > > will
> > > > > be able
> > > > > to comment. At minimum, we should file these as issues in the
> > > > > GitHub
> > > > > repository, so they are not forgotten. If you do not hear
> > > > > back
> > > > > further from
> > > > > anyone, you can file them at https://github.com/trakem2/TrakE
> > > > > M2/i
> > > > > ssue
> > > > > s, or
> > > > > ping me and I will take care of it.
> > > > >
> > > > > Herbie wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > I have no idea about "trakem2"
> > > > > See this page for details:
> > > > > http://imagej.net/TrakEM2
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > JEPG compression in 99% of all cases is a lossy method and
> > > > > > more
> > > > > > or
> > > > > > less changes the image content, i.e. it introduces
> > > > > > artifacts.
> > > > > Indeed—there is a discussion about that here:
> > > > >
> > > > > http://imagej.net/Principles#Why_.28lossy.29_JPEGs_should_
> > > > > not_be_used_in_imaging
> > > > >
> > > > > That said, there are cases where it is acceptable to use
> > > > > JPEG.
> > > > > Essentially,
> > > > > if you have massively large data, and you do not plan to
> > > > > analyze
> > > > > quantitatively for the most part, it might be OK for
> > > > > pragmatic
> > > > > reasons, as
> > > > > long as you understand what you are doing.
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > With respect to "trakem2"-issues I'd contact the authors
> > > > > > directly.
> > > > > I suggest keeping the discussion public. From the Help page (
> > > > > http://imagej.net/Help):
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > The ImageJ community firmly believes in public discussion
> > > > > > of
> > > > > > the
> > > > > > software. In this way, each question benefits not only the
> > > > > > one
> > > > > > asking,
> > > > > > but everyone in the community, including everyone who
> > > > > > subsequently
> > > > > > does a web search on the same topic.
> > > > > Regards,
> > > > > Curtis
> > > > >
> > > > > --
> > > > > Curtis Rueden
> > > > > LOCI software architect - https://loci.wisc.edu/software
> > > > > ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Ruede
> > > > > n
> > > > > Did you know ImageJ has a forum? http://forum.imagej.net/
> > > > >
> > > > >
> > > > > On Wed, Feb 8, 2017 at 4:21 AM, Herbie <[hidden email]>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Good day Stoyan,
> > > > > >
> > > > > > I have no idea about "trakem2" but please let me remark
> > > > > > that
> > > > > > using
> > > > > > JEPG-compressed images for scientific purposes is an
> > > > > > absolute
> > > > > > no-
> > > > > > no. JEPG
> > > > > > compression in 99% of all cases is a lossy method and more
> > > > > > or
> > > > > > less
> > > > > > changes
> > > > > > the image content, i.e. it introduces artifacts.
> > > > > >
> > > > > > With respect to "trakem2"-issues I'd contact the authors
> > > > > > directly.
> > > > > >
> > > > > > Best
> > > > > >
> > > > > > Herbie
> > > > > >
> > > > > > :::::::::::::::::::::::::::::::::::::::::::
> > > > > > Am 08.02.17 um 10:40 schrieb Stoyan Pavlov:
> > > > > >
> > > > > > Dear friends and colleagues,
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > First I want to emphasize, how astonishing and useful I
> > > > > > > find
> > > > > > > the
> > > > > > > trakem2
> > > > > > > package/plugin! It is really great and I am really
> > > > > > > thankful
> > > > > > > to
> > > > > > > the
> > > > > > > developers for puting all this effort into creating this
> > > > > > > suite
> > > > > > > for
> > > > > working
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > with large image datasets.
> > > > > > > I use it mainly for creating/stitching/blending ,
> > > > > > > processing
> > > > > > > and
> > > > > exporting
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > of gigapixel montages of microscopic images into a
> > > > > > > readable
> > > > > > > software
> > > > > > > independent image pyramid.
> > > > > > > Recently I stumbled upon several issues (I don't know
> > > > > > > whether
> > > > > > > they are
> > > > > > > bugs
> > > > > > > or just due to limitation in the software) for which I
> > > > > > > couldn't
> > > > > > > find any
> > > > > > > mention and/or solution in the documentation and the
> > > > > > > forum.
> > > > > > >
> > > > > > > First I will describe what I am doing and next I will
> > > > > > > list
> > > > > > > the
> > > > > > > issues
> > > > > that
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > bother me.
> > > > > > > I am creating a single layer montage with dimensions
> > > > > > > 63698 x
> > > > > > > 57218 px
> > > > > from
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > a set of 3111 images (each 1388x1040). The images are
> > > > > > > acquired
> > > > > > > via
> > > > > > > axiovision and exported as single tiles (24bit RGB in
> > > > > > > either
> > > > > > > jpg
> > > > > > > or tif
> > > > > > > format). Before importing the images I align them using
> > > > > > > the
> > > > > > > MIST
> > > > > > > plugin
> > > > > > > and
> > > > > > > use the global positions generated from it to import them
> > > > > > > into
> > > > > > > trakem2
> > > > > via
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > text file. The resulting montage is exactly as expected
> > > > > > > and
> > > > > > > have
> > > > > > > no
> > > > > > > remarks
> > > > > > > about it.
> > > > > > > 1) I don't know if its true but it seems to me that if I
> > > > > > > use
> > > > > > > jpg
> > > > > > > as
> > > > > format
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > for the source images they are imported faster into
> > > > > > > trakem2.
> > > > > > > However once
> > > > > > > the import is finished the software returns errors about
> > > > > > > generating
> > > > > > > mipmaps
> > > > > > > (see attached log file) and it continuously tries to
> > > > > > > regenerate
> > > > > > > them
> > > > > > > (which
> > > > > > > I guess keeps CPU quite busy). This happens regardless
> > > > > > > mipmaps
> > > > > > > are
> > > > > enabled
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > or disabled in Display>Properties, and is independent
> > > > > > > from
> > > > > > > the
> > > > > > > number of
> > > > > > > threads assigned to mipmap generation. If I close the
> > > > > > > project
> > > > > > > and
> > > > > > > Fiji
> > > > > > > (even after system restart) and then I open them again
> > > > > > > the
> > > > > > > issue
> > > > > > > returns.
> > > > > > > This behaviour is not oberved if my source images are in
> > > > > > > tif
> > > > > > > format.
> > > > > > > The other issues are related to the export of a flat
> > > > > > > image
> > > > > > > (here
> > > > > > > the file
> > > > > > > format of the source images does not matter):
> > > > > > >
> > > > > > > 2) When I am trying to export to a tiled format (Export>
> > > > > > > Make
> > > > > > > flat
> > > > > > > image...>Save for web (CATMAID)) regardless of the image
> > > > > > > format I
> > > > > > > choose
> > > > > > > for the export I cannot change the size of the exported
> > > > > > > tiles
> > > > > > > to
> > > > > > > other
> > > > > > > than
> > > > > > > 256x256. The dialogue allows me to change it, but when
> > > > > > > the
> > > > > > > process
> > > > > > > finishes
> > > > > > > in the folder there are only 256x256 tile images. While
> > > > > > > this
> > > > > > > is
> > > > > > > not a
> > > > > huge
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > problem, still there is difference between managing some
> > > > > > > 80000
> > > > > > > images or
> > > > > > > 40000.
> > > > > > >
> > > > > > > 3) Also when I am trying to export tiled format via
> > > > > > > Export>
> > > > > > > Make
> > > > > > > flat
> > > > > > > image...>Save for web (CATMAID), if the patches are
> > > > > > > deselected
> > > > > > > only a
> > > > > > > portion of the image is exported . Regardless of the
> > > > > > > format
> > > > > > > of
> > > > > > > the source
> > > > > > > images and of the exported images there are only 63 rows
> > > > > > > by
> > > > > > > 63
> > > > > > > columns of
> > > > > > > 256x256 tiles at zoom level 0 in the folder with the
> > > > > > > exported
> > > > > > > images. All
> > > > > > > metadata in the json file is correct and there are no
> > > > > > > errors
> > > > > > > reported by
> > > > > > > the System log or the Fiji Log. If the patches are
> > > > > > > selected
> > > > > > > with
> > > > > > > Select
> > > > > > > All
> > > > > > > (Ctrl + A) the image is exported in its entirety, however
> > > > > > > here
> > > > > > > comes the
> > > > > > > next not as much as an issue but slight discomfort.
> > > > > > >
> > > > > > > 4) The image is always exported to a quadrangular array
> > > > > > > consisting (for
> > > > > > > the
> > > > > > > reasons already described) of 256 rows and 256 columns of
> > > > > > > 256x256
> > > > > > > tiles
> > > > > at
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > zoom level 0. That is the image dimensions (63698 x 57218
> > > > > > > px)
> > > > > > > are
> > > > > expanded
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > to 65536x65536 and filled with black tiles. This of
> > > > > > > course
> > > > > > > take
> > > > > > > up
> > > > > > > additional space and need to be cropped away before
> > > > > > > uploading
> > > > > > > the
> > > > > > > image
> > > > > to
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > our servers. I understand that this is probably because
> > > > > > > of
> > > > > > > limitations to
> > > > > > > the algorithm but still if there is some workaround it
> > > > > > > would
> > > > > > > have
> > > > > > > been
> > > > > > > better.
> > > > > > >
> > > > > > > 5) I thought that this problems (2-4) might be due to the
> > > > > > > unequal
> > > > > > > dimensions of the original source images and I tried to
> > > > > > > work
> > > > > > > around them
> > > > > > > by
> > > > > > > creating a sibling project with retiled images. Here I
> > > > > > > had
> > > > > > > another
> > > > > > > unexpected result - the image was exported but the larger
> > > > > > > dimension was
> > > > > > > cut
> > > > > > > by about 6000 pxs from the right side and than restored
> > > > > > > to
> > > > > > > the
> > > > > > > original
> > > > > > > size by adding black pixels , which basically cuts away
> > > > > > > the
> > > > > > > right
> > > > > > > portion
> > > > > > > of my image (see attached screenshot) .
> > > > > > >
> > > > > > > The workstation that I use has the following parameters:
> > > > > > > CPU: Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core
> > > > > > > processors)
> > > > > > > RAM: 64 GB
> > > > > > > OS: Windows 7 64bit
> > > > > > > In an attachment I added the ImageJ-properties output,
> > > > > > > portion of
> > > > > > > the
> > > > > > > trakem2 log with the errors generating the mipmaps, and a
> > > > > > > screenshot of
> > > > > > > the
> > > > > > > output of the sibling project with retiled images.
> > > > > > >
> > > > > > > I hope someone will find a way to correct these problems,
> > > > > > > because
> > > > > > > surely
> > > > > > > they are not fatal and there are workarounds that we use,
> > > > > > > but
> > > > > > > it
> > > > > > > is
> > > > > > > certainly annoying.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Stoyan
> > > > > > >
> > > > > > > ---
> > > > > > > Dr. Stoyan P. Pavlov, MD, PhD
> > > > > > > Departament of Anatomy, Histology and Embryology
> > > > > > > Medical University "Prof. Dr. Paraskev Stoyanov", Varna
> > > > > > > Prof. Marin Drinov Str.55
> > > > > > > 9002 Varna
> > > > > > > Bulgaria
> > > > > > > Tel: +359 (0) 52 - 677 - 052 <052%20677%20052>
> > > > > > > e-mail: [hidden email]
> > > > > > > [hidden email]
> > > > > > >
> > > > > > > --
> > > > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > > > >
> > > > > > >
> > > > > > --
> > > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > > >
> > > > > --
> > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > >
> > > > > --
> > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > --
> > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > >
> > > >
> > > >
> > > --
> > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
signature.asc (484 bytes) | Free forum by Nabble | Disable Popup Ads | Edit this page |