Re: Issues and annoyances with trakem2 working on a gigapixel montage
Posted by Saalfeld, Stephan on
URL: http://imagej.273.s1.nabble.com/Issues-and-annoyances-with-trakem2-working-on-a-gigapixel-montage-tp5018056p5018062.html
Hi,
TrakEM2's internal CATMAID export mechanisms goes through single image
exports. This is limited to 2GPx planes (sad---right?). Please use
this script instead:
https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/catmaid-
export2.bsh
Regarding the mipmap-madness...:
1. Is your project directory on a file share? If yes, please move it
to a local (external?) harddrive, you are working with 2D images, that
should fit. TrakEM2 is somehow sensitive to how network drives
frequently fail or rather delay what they are supposed to do. This is
an annoyance in TrakEM2 and could/ should be fixed. No time though...
2. If you do not want to use mipmaps, switch mipmaps off *before*
importing tiles. Switching them off during the madness has no effects,
too many threads running already and they will not get properly
interrupted.
3. JPG import works faster because JPGs read faster, this sounds like
you are I/O limited, which makes me believe in 1. more ;). Oh yes, are
all your images the same size and have intensities 0--255? Then you
can use the extended import file syntax
https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/main/java/ini/trakem2/persistence/Loader.java#L2013-L2027
if you specify the columns 5--9 and switch off mipmaps, none of your
images will be touched during import, i.e. import will be almost
instantaneous. Save the project, do the rest.
Let me know if that helps or where it breaks.
Cheers,
Stephan
On Wed, 2017-02-08 at 16:23 +0200, Stoyan Pavlov wrote:
> Hello Curtis,
> Thank you very much. I will wait a while for an answer. Last time
> they
> answered pretty quickly. If however there's no movement on this I
> will
> probably accept your offer to file the bug report as proposed,
> because I
> am not familiar with github and coding.
>
> Best regards!
> Stoyan
>
> P.S. Sorry about the outbirst from few minutes ago!
> На 8.02.2017 г. 1:40 сл.об. "Curtis Rueden" <[hidden email]>
> написа:
>
> Hi Stoyan, Herbie, everyone,
>
> Stoyan wrote:
> >
> > First I want to emphasize, how astonishing and useful I find the
> > trakem2 package/plugin! It is really great and I am really thankful
> > to
> > the developers for puting all this effort into creating this suite
> > for
> > working with large image datasets.
> Glad to hear it is useful for you.
>
> >
> > Recently I stumbled upon several issues
> Thank you for the detailed report. Hopefully Albert or Stephan will
> be able
> to comment. At minimum, we should file these as issues in the GitHub
> repository, so they are not forgotten. If you do not hear back
> further from
> anyone, you can file them at https://github.com/trakem2/TrakEM2/issue
> s, or
> ping me and I will take care of it.
>
> Herbie wrote:
> >
> > I have no idea about "trakem2"
> See this page for details:
> http://imagej.net/TrakEM2
>
> >
> > JEPG compression in 99% of all cases is a lossy method and more or
> > less changes the image content, i.e. it introduces artifacts.
> Indeed—there is a discussion about that here:
>
> http://imagej.net/Principles#Why_.28lossy.29_JPEGs_should_
> not_be_used_in_imaging
>
> That said, there are cases where it is acceptable to use JPEG.
> Essentially,
> if you have massively large data, and you do not plan to analyze
> quantitatively for the most part, it might be OK for pragmatic
> reasons, as
> long as you understand what you are doing.
>
> >
> > With respect to "trakem2"-issues I'd contact the authors directly.
> I suggest keeping the discussion public. From the Help page (
> http://imagej.net/Help):
>
> >
> > The ImageJ community firmly believes in public discussion of the
> > software. In this way, each question benefits not only the one
> > asking,
> > but everyone in the community, including everyone who subsequently
> > does a web search on the same topic.
> Regards,
> Curtis
>
> --
> Curtis Rueden
> LOCI software architect - https://loci.wisc.edu/software
> ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden
> Did you know ImageJ has a forum? http://forum.imagej.net/
>
>
> On Wed, Feb 8, 2017 at 4:21 AM, Herbie <[hidden email]> wrote:
>
> >
> > Good day Stoyan,
> >
> > I have no idea about "trakem2" but please let me remark that using
> > JEPG-compressed images for scientific purposes is an absolute no-
> > no. JEPG
> > compression in 99% of all cases is a lossy method and more or less
> > changes
> > the image content, i.e. it introduces artifacts.
> >
> > With respect to "trakem2"-issues I'd contact the authors directly.
> >
> > Best
> >
> > Herbie
> >
> > :::::::::::::::::::::::::::::::::::::::::::
> > Am 08.02.17 um 10:40 schrieb Stoyan Pavlov:
> >
> > Dear friends and colleagues,
> > >
> > > First I want to emphasize, how astonishing and useful I find the
> > > trakem2
> > > package/plugin! It is really great and I am really thankful to
> > > the
> > > developers for puting all this effort into creating this suite
> > > for
> working
> >
> > >
> > > with large image datasets.
> > > I use it mainly for creating/stitching/blending , processing and
> exporting
> >
> > >
> > > of gigapixel montages of microscopic images into a readable
> > > software
> > > independent image pyramid.
> > > Recently I stumbled upon several issues (I don't know whether
> > > they are
> > > bugs
> > > or just due to limitation in the software) for which I couldn't
> > > find any
> > > mention and/or solution in the documentation and the forum.
> > >
> > > First I will describe what I am doing and next I will list the
> > > issues
> that
> >
> > >
> > > bother me.
> > > I am creating a single layer montage with dimensions 63698 x
> > > 57218 px
> from
> >
> > >
> > > a set of 3111 images (each 1388x1040). The images are acquired
> > > via
> > > axiovision and exported as single tiles (24bit RGB in either jpg
> > > or tif
> > > format). Before importing the images I align them using the MIST
> > > plugin
> > > and
> > > use the global positions generated from it to import them into
> > > trakem2
> via
> >
> > >
> > > text file. The resulting montage is exactly as expected and have
> > > no
> > > remarks
> > > about it.
> > > 1) I don't know if its true but it seems to me that if I use jpg
> > > as
> format
> >
> > >
> > > for the source images they are imported faster into trakem2.
> > > However once
> > > the import is finished the software returns errors about
> > > generating
> > > mipmaps
> > > (see attached log file) and it continuously tries to regenerate
> > > them
> > > (which
> > > I guess keeps CPU quite busy). This happens regardless mipmaps
> > > are
> enabled
> >
> > >
> > > or disabled in Display>Properties, and is independent from the
> > > number of
> > > threads assigned to mipmap generation. If I close the project and
> > > Fiji
> > > (even after system restart) and then I open them again the issue
> > > returns.
> > > This behaviour is not oberved if my source images are in tif
> > > format.
> > > The other issues are related to the export of a flat image (here
> > > the file
> > > format of the source images does not matter):
> > >
> > > 2) When I am trying to export to a tiled format (Export> Make
> > > flat
> > > image...>Save for web (CATMAID)) regardless of the image format I
> > > choose
> > > for the export I cannot change the size of the exported tiles to
> > > other
> > > than
> > > 256x256. The dialogue allows me to change it, but when the
> > > process
> > > finishes
> > > in the folder there are only 256x256 tile images. While this is
> > > not a
> huge
> >
> > >
> > > problem, still there is difference between managing some 80000
> > > images or
> > > 40000.
> > >
> > > 3) Also when I am trying to export tiled format via Export> Make
> > > flat
> > > image...>Save for web (CATMAID), if the patches are deselected
> > > only a
> > > portion of the image is exported . Regardless of the format of
> > > the source
> > > images and of the exported images there are only 63 rows by 63
> > > columns of
> > > 256x256 tiles at zoom level 0 in the folder with the exported
> > > images. All
> > > metadata in the json file is correct and there are no errors
> > > reported by
> > > the System log or the Fiji Log. If the patches are selected with
> > > Select
> > > All
> > > (Ctrl + A) the image is exported in its entirety, however here
> > > comes the
> > > next not as much as an issue but slight discomfort.
> > >
> > > 4) The image is always exported to a quadrangular array
> > > consisting (for
> > > the
> > > reasons already described) of 256 rows and 256 columns of 256x256
> > > tiles
> at
> >
> > >
> > > zoom level 0. That is the image dimensions (63698 x 57218 px) are
> expanded
> >
> > >
> > > to 65536x65536 and filled with black tiles. This of course take
> > > up
> > > additional space and need to be cropped away before uploading the
> > > image
> to
> >
> > >
> > > our servers. I understand that this is probably because of
> > > limitations to
> > > the algorithm but still if there is some workaround it would have
> > > been
> > > better.
> > >
> > > 5) I thought that this problems (2-4) might be due to the unequal
> > > dimensions of the original source images and I tried to work
> > > around them
> > > by
> > > creating a sibling project with retiled images. Here I had
> > > another
> > > unexpected result - the image was exported but the larger
> > > dimension was
> > > cut
> > > by about 6000 pxs from the right side and than restored to the
> > > original
> > > size by adding black pixels , which basically cuts away the right
> > > portion
> > > of my image (see attached screenshot) .
> > >
> > > The workstation that I use has the following parameters:
> > > CPU: Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core processors)
> > > RAM: 64 GB
> > > OS: Windows 7 64bit
> > > In an attachment I added the ImageJ-properties output, portion of
> > > the
> > > trakem2 log with the errors generating the mipmaps, and a
> > > screenshot of
> > > the
> > > output of the sibling project with retiled images.
> > >
> > > I hope someone will find a way to correct these problems, because
> > > surely
> > > they are not fatal and there are workarounds that we use, but it
> > > is
> > > certainly annoying.
> > >
> > > Best regards,
> > > Stoyan
> > >
> > > ---
> > > Dr. Stoyan P. Pavlov, MD, PhD
> > > Departament of Anatomy, Histology and Embryology
> > > Medical University "Prof. Dr. Paraskev Stoyanov", Varna
> > > Prof. Marin Drinov Str.55
> > > 9002 Varna
> > > Bulgaria
> > > Tel: +359 (0) 52 - 677 - 052
> > > e-mail: [hidden email]
> > > [hidden email]
> > >
> > > --
> > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > >
> > >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
URL: http://imagej.273.s1.nabble.com/Issues-and-annoyances-with-trakem2-working-on-a-gigapixel-montage-tp5018056p5018062.html
Hi,
TrakEM2's internal CATMAID export mechanisms goes through single image
exports. This is limited to 2GPx planes (sad---right?). Please use
this script instead:
https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/catmaid-
export2.bsh
Regarding the mipmap-madness...:
1. Is your project directory on a file share? If yes, please move it
to a local (external?) harddrive, you are working with 2D images, that
should fit. TrakEM2 is somehow sensitive to how network drives
frequently fail or rather delay what they are supposed to do. This is
an annoyance in TrakEM2 and could/ should be fixed. No time though...
2. If you do not want to use mipmaps, switch mipmaps off *before*
importing tiles. Switching them off during the madness has no effects,
too many threads running already and they will not get properly
interrupted.
3. JPG import works faster because JPGs read faster, this sounds like
you are I/O limited, which makes me believe in 1. more ;). Oh yes, are
all your images the same size and have intensities 0--255? Then you
can use the extended import file syntax
https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/main/java/ini/trakem2/persistence/Loader.java#L2013-L2027
if you specify the columns 5--9 and switch off mipmaps, none of your
images will be touched during import, i.e. import will be almost
instantaneous. Save the project, do the rest.
Let me know if that helps or where it breaks.
Cheers,
Stephan
On Wed, 2017-02-08 at 16:23 +0200, Stoyan Pavlov wrote:
> Hello Curtis,
> Thank you very much. I will wait a while for an answer. Last time
> they
> answered pretty quickly. If however there's no movement on this I
> will
> probably accept your offer to file the bug report as proposed,
> because I
> am not familiar with github and coding.
>
> Best regards!
> Stoyan
>
> P.S. Sorry about the outbirst from few minutes ago!
> На 8.02.2017 г. 1:40 сл.об. "Curtis Rueden" <[hidden email]>
> написа:
>
> Hi Stoyan, Herbie, everyone,
>
> Stoyan wrote:
> >
> > First I want to emphasize, how astonishing and useful I find the
> > trakem2 package/plugin! It is really great and I am really thankful
> > to
> > the developers for puting all this effort into creating this suite
> > for
> > working with large image datasets.
> Glad to hear it is useful for you.
>
> >
> > Recently I stumbled upon several issues
> Thank you for the detailed report. Hopefully Albert or Stephan will
> be able
> to comment. At minimum, we should file these as issues in the GitHub
> repository, so they are not forgotten. If you do not hear back
> further from
> anyone, you can file them at https://github.com/trakem2/TrakEM2/issue
> s, or
> ping me and I will take care of it.
>
> Herbie wrote:
> >
> > I have no idea about "trakem2"
> See this page for details:
> http://imagej.net/TrakEM2
>
> >
> > JEPG compression in 99% of all cases is a lossy method and more or
> > less changes the image content, i.e. it introduces artifacts.
> Indeed—there is a discussion about that here:
>
> http://imagej.net/Principles#Why_.28lossy.29_JPEGs_should_
> not_be_used_in_imaging
>
> That said, there are cases where it is acceptable to use JPEG.
> Essentially,
> if you have massively large data, and you do not plan to analyze
> quantitatively for the most part, it might be OK for pragmatic
> reasons, as
> long as you understand what you are doing.
>
> >
> > With respect to "trakem2"-issues I'd contact the authors directly.
> I suggest keeping the discussion public. From the Help page (
> http://imagej.net/Help):
>
> >
> > The ImageJ community firmly believes in public discussion of the
> > software. In this way, each question benefits not only the one
> > asking,
> > but everyone in the community, including everyone who subsequently
> > does a web search on the same topic.
> Regards,
> Curtis
>
> --
> Curtis Rueden
> LOCI software architect - https://loci.wisc.edu/software
> ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden
> Did you know ImageJ has a forum? http://forum.imagej.net/
>
>
> On Wed, Feb 8, 2017 at 4:21 AM, Herbie <[hidden email]> wrote:
>
> >
> > Good day Stoyan,
> >
> > I have no idea about "trakem2" but please let me remark that using
> > JEPG-compressed images for scientific purposes is an absolute no-
> > no. JEPG
> > compression in 99% of all cases is a lossy method and more or less
> > changes
> > the image content, i.e. it introduces artifacts.
> >
> > With respect to "trakem2"-issues I'd contact the authors directly.
> >
> > Best
> >
> > Herbie
> >
> > :::::::::::::::::::::::::::::::::::::::::::
> > Am 08.02.17 um 10:40 schrieb Stoyan Pavlov:
> >
> > Dear friends and colleagues,
> > >
> > > First I want to emphasize, how astonishing and useful I find the
> > > trakem2
> > > package/plugin! It is really great and I am really thankful to
> > > the
> > > developers for puting all this effort into creating this suite
> > > for
> working
> >
> > >
> > > with large image datasets.
> > > I use it mainly for creating/stitching/blending , processing and
> exporting
> >
> > >
> > > of gigapixel montages of microscopic images into a readable
> > > software
> > > independent image pyramid.
> > > Recently I stumbled upon several issues (I don't know whether
> > > they are
> > > bugs
> > > or just due to limitation in the software) for which I couldn't
> > > find any
> > > mention and/or solution in the documentation and the forum.
> > >
> > > First I will describe what I am doing and next I will list the
> > > issues
> that
> >
> > >
> > > bother me.
> > > I am creating a single layer montage with dimensions 63698 x
> > > 57218 px
> from
> >
> > >
> > > a set of 3111 images (each 1388x1040). The images are acquired
> > > via
> > > axiovision and exported as single tiles (24bit RGB in either jpg
> > > or tif
> > > format). Before importing the images I align them using the MIST
> > > plugin
> > > and
> > > use the global positions generated from it to import them into
> > > trakem2
> via
> >
> > >
> > > text file. The resulting montage is exactly as expected and have
> > > no
> > > remarks
> > > about it.
> > > 1) I don't know if its true but it seems to me that if I use jpg
> > > as
> format
> >
> > >
> > > for the source images they are imported faster into trakem2.
> > > However once
> > > the import is finished the software returns errors about
> > > generating
> > > mipmaps
> > > (see attached log file) and it continuously tries to regenerate
> > > them
> > > (which
> > > I guess keeps CPU quite busy). This happens regardless mipmaps
> > > are
> enabled
> >
> > >
> > > or disabled in Display>Properties, and is independent from the
> > > number of
> > > threads assigned to mipmap generation. If I close the project and
> > > Fiji
> > > (even after system restart) and then I open them again the issue
> > > returns.
> > > This behaviour is not oberved if my source images are in tif
> > > format.
> > > The other issues are related to the export of a flat image (here
> > > the file
> > > format of the source images does not matter):
> > >
> > > 2) When I am trying to export to a tiled format (Export> Make
> > > flat
> > > image...>Save for web (CATMAID)) regardless of the image format I
> > > choose
> > > for the export I cannot change the size of the exported tiles to
> > > other
> > > than
> > > 256x256. The dialogue allows me to change it, but when the
> > > process
> > > finishes
> > > in the folder there are only 256x256 tile images. While this is
> > > not a
> huge
> >
> > >
> > > problem, still there is difference between managing some 80000
> > > images or
> > > 40000.
> > >
> > > 3) Also when I am trying to export tiled format via Export> Make
> > > flat
> > > image...>Save for web (CATMAID), if the patches are deselected
> > > only a
> > > portion of the image is exported . Regardless of the format of
> > > the source
> > > images and of the exported images there are only 63 rows by 63
> > > columns of
> > > 256x256 tiles at zoom level 0 in the folder with the exported
> > > images. All
> > > metadata in the json file is correct and there are no errors
> > > reported by
> > > the System log or the Fiji Log. If the patches are selected with
> > > Select
> > > All
> > > (Ctrl + A) the image is exported in its entirety, however here
> > > comes the
> > > next not as much as an issue but slight discomfort.
> > >
> > > 4) The image is always exported to a quadrangular array
> > > consisting (for
> > > the
> > > reasons already described) of 256 rows and 256 columns of 256x256
> > > tiles
> at
> >
> > >
> > > zoom level 0. That is the image dimensions (63698 x 57218 px) are
> expanded
> >
> > >
> > > to 65536x65536 and filled with black tiles. This of course take
> > > up
> > > additional space and need to be cropped away before uploading the
> > > image
> to
> >
> > >
> > > our servers. I understand that this is probably because of
> > > limitations to
> > > the algorithm but still if there is some workaround it would have
> > > been
> > > better.
> > >
> > > 5) I thought that this problems (2-4) might be due to the unequal
> > > dimensions of the original source images and I tried to work
> > > around them
> > > by
> > > creating a sibling project with retiled images. Here I had
> > > another
> > > unexpected result - the image was exported but the larger
> > > dimension was
> > > cut
> > > by about 6000 pxs from the right side and than restored to the
> > > original
> > > size by adding black pixels , which basically cuts away the right
> > > portion
> > > of my image (see attached screenshot) .
> > >
> > > The workstation that I use has the following parameters:
> > > CPU: Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core processors)
> > > RAM: 64 GB
> > > OS: Windows 7 64bit
> > > In an attachment I added the ImageJ-properties output, portion of
> > > the
> > > trakem2 log with the errors generating the mipmaps, and a
> > > screenshot of
> > > the
> > > output of the sibling project with retiled images.
> > >
> > > I hope someone will find a way to correct these problems, because
> > > surely
> > > they are not fatal and there are workarounds that we use, but it
> > > is
> > > certainly annoying.
> > >
> > > Best regards,
> > > Stoyan
> > >
> > > ---
> > > Dr. Stoyan P. Pavlov, MD, PhD
> > > Departament of Anatomy, Histology and Embryology
> > > Medical University "Prof. Dr. Paraskev Stoyanov", Varna
> > > Prof. Marin Drinov Str.55
> > > 9002 Varna
> > > Bulgaria
> > > Tel: +359 (0) 52 - 677 - 052
> > > e-mail: [hidden email]
> > > [hidden email]
> > >
> > > --
> > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > >
> > >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
ImageJ mailing list: http://imagej.nih.gov/ij/list.html
signature.asc (484 bytes) | Free forum by Nabble | Edit this page |