Issues and annoyances with trakem2 working on a gigapixel montage

[Stoyan Pavlov]

Dear friends and colleagues,
First I want to emphasize, how astonishing and useful I find the trakem2
package/plugin! It is really great and I am really thankful to the
developers for puting all this effort into creating this suite for working
with large image datasets.
I use it mainly for creating/stitching/blending , processing and exporting
of gigapixel montages of microscopic images into a readable software
independent image pyramid.
Recently I stumbled upon several issues (I don't know whether they are bugs
or just due to limitation in the software) for which I couldn't find any
mention and/or solution in the documentation and the forum.

First I will describe what I am doing and next I will list the issues that
bother me.
I am creating a single layer montage with dimensions 63698 x 57218 px from
a set of 3111 images (each 1388x1040). The images are acquired via
axiovision and exported as single tiles (24bit RGB in either jpg or tif
format). Before importing the images I align them using the MIST plugin and
use the global positions generated from it to import them into trakem2 via
text file. The resulting montage is exactly as expected and have no remarks
about it.
1) I don't know if its true but it seems to me that if I use jpg as format
for the source images they are imported faster into trakem2. However once
the import is finished the software returns errors about generating mipmaps
(see attached log file) and it continuously tries to regenerate them (which
I guess keeps CPU quite busy). This happens regardless mipmaps are enabled
or disabled in Display>Properties, and is independent from the number of
threads assigned to mipmap generation. If I close the project and Fiji
(even after system restart) and then I open them again the issue returns.
This behaviour is not oberved if my source images are in tif format.
The other issues are related to the export of a flat image (here the file
format of the source images does not matter):

2) When I am trying to export to a tiled format (Export> Make flat
image...>Save for web (CATMAID)) regardless of the image format I choose
for the export I cannot change the size of the exported tiles to other than
256x256. The dialogue allows me to change it, but when the process finishes
in the folder there are only 256x256 tile images. While this is not a huge
problem, still there is difference between managing some 80000 images or
40000.

3) Also when I am trying to export tiled format  via Export> Make flat
image...>Save for web (CATMAID), if the patches are deselected only a
portion of the image is exported . Regardless of the format of the source
images and of the exported images there are only 63 rows by 63 columns of
256x256 tiles at zoom level 0 in the folder with the exported images. All
metadata in the json file is correct and there are no errors reported by
the System log or the Fiji Log. If the patches are selected with Select All
(Ctrl + A) the image is exported in its entirety, however here comes the
next not as much as an issue but slight discomfort.

4) The image is always exported to a quadrangular array consisting (for the
reasons already described) of 256 rows and 256 columns of 256x256 tiles at
zoom level 0. That is the image dimensions (63698 x 57218 px) are expanded
to 65536x65536 and filled with black tiles. This of course take up
additional space and need to be cropped away before uploading the image to
our servers. I understand that this is probably because of limitations to
the algorithm but still if there is some workaround it would have been
better.

5) I thought that this problems (2-4) might be due to the unequal
dimensions of the original source images and I tried to work around them by
creating a sibling project with retiled images. Here I had another
unexpected result - the image was exported but the larger dimension was cut
by about 6000 pxs from the right side  and than restored to the original
size by adding black pixels , which basically cuts away the right portion
of my image (see attached screenshot) .

The workstation that I use has the following parameters:
 CPU:  Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core processors)
RAM: 64 GB
OS: Windows 7 64bit
In an attachment I added the ImageJ-properties output, portion of the
trakem2 log with the errors generating the mipmaps, and a screenshot of the
output of the sibling project with retiled images.

I hope someone will find a way to correct these problems, because surely
they are not fatal and there are workarounds that we use, but it is
certainly annoying.

Best regards,
Stoyan

---
Dr. Stoyan P. Pavlov, MD, PhD
Departament of Anatomy, Histology and Embryology
Medical University "Prof. Dr. Paraskev Stoyanov", Varna
Prof. Marin Drinov Str.55
9002 Varna
 Bulgaria
Tel: +359 (0) 52 - 677 - 052
e-mail: [hidden email]
           [hidden email]

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Herbie]

Good day Stoyan,

I have no idea about "trakem2" but please let me remark that using
JEPG-compressed images for scientific purposes is an absolute no-no.
JEPG compression in 99% of all cases is a lossy method and more or less
changes the image content, i.e. it introduces artifacts.

With respect to "trakem2"-issues I'd contact the authors directly.

Best

Herbie

:::::::::::::::::::::::::::::::::::::::::::
Am 08.02.17 um 10:40 schrieb Stoyan Pavlov:

> Dear friends and colleagues,
> First I want to emphasize, how astonishing and useful I find the trakem2
> package/plugin! It is really great and I am really thankful to the
> developers for puting all this effort into creating this suite for working
> with large image datasets.
> I use it mainly for creating/stitching/blending , processing and exporting
> of gigapixel montages of microscopic images into a readable software
> independent image pyramid.
> Recently I stumbled upon several issues (I don't know whether they are bugs
> or just due to limitation in the software) for which I couldn't find any
> mention and/or solution in the documentation and the forum.
>
> First I will describe what I am doing and next I will list the issues that
> bother me.
> I am creating a single layer montage with dimensions 63698 x 57218 px from
> a set of 3111 images (each 1388x1040). The images are acquired via
> axiovision and exported as single tiles (24bit RGB in either jpg or tif
> format). Before importing the images I align them using the MIST plugin and
> use the global positions generated from it to import them into trakem2 via
> text file. The resulting montage is exactly as expected and have no remarks
> about it.
> 1) I don't know if its true but it seems to me that if I use jpg as format
> for the source images they are imported faster into trakem2. However once
> the import is finished the software returns errors about generating mipmaps
> (see attached log file) and it continuously tries to regenerate them (which
> I guess keeps CPU quite busy). This happens regardless mipmaps are enabled
> or disabled in Display>Properties, and is independent from the number of
> threads assigned to mipmap generation. If I close the project and Fiji
> (even after system restart) and then I open them again the issue returns.
> This behaviour is not oberved if my source images are in tif format.
> The other issues are related to the export of a flat image (here the file
> format of the source images does not matter):
>
> 2) When I am trying to export to a tiled format (Export> Make flat
> image...>Save for web (CATMAID)) regardless of the image format I choose
> for the export I cannot change the size of the exported tiles to other than
> 256x256. The dialogue allows me to change it, but when the process finishes
> in the folder there are only 256x256 tile images. While this is not a huge
> problem, still there is difference between managing some 80000 images or
> 40000.
>
> 3) Also when I am trying to export tiled format  via Export> Make flat
> image...>Save for web (CATMAID), if the patches are deselected only a
> portion of the image is exported . Regardless of the format of the source
> images and of the exported images there are only 63 rows by 63 columns of
> 256x256 tiles at zoom level 0 in the folder with the exported images. All
> metadata in the json file is correct and there are no errors reported by
> the System log or the Fiji Log. If the patches are selected with Select All
> (Ctrl + A) the image is exported in its entirety, however here comes the
> next not as much as an issue but slight discomfort.
>
> 4) The image is always exported to a quadrangular array consisting (for the
> reasons already described) of 256 rows and 256 columns of 256x256 tiles at
> zoom level 0. That is the image dimensions (63698 x 57218 px) are expanded
> to 65536x65536 and filled with black tiles. This of course take up
> additional space and need to be cropped away before uploading the image to
> our servers. I understand that this is probably because of limitations to
> the algorithm but still if there is some workaround it would have been
> better.
>
> 5) I thought that this problems (2-4) might be due to the unequal
> dimensions of the original source images and I tried to work around them by
> creating a sibling project with retiled images. Here I had another
> unexpected result - the image was exported but the larger dimension was cut
> by about 6000 pxs from the right side  and than restored to the original
> size by adding black pixels , which basically cuts away the right portion
> of my image (see attached screenshot) .
>
> The workstation that I use has the following parameters:
>  CPU:  Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core processors)
> RAM: 64 GB
> OS: Windows 7 64bit
> In an attachment I added the ImageJ-properties output, portion of the
> trakem2 log with the errors generating the mipmaps, and a screenshot of the
> output of the sibling project with retiled images.
>
> I hope someone will find a way to correct these problems, because surely
> they are not fatal and there are workarounds that we use, but it is
> certainly annoying.
>
> Best regards,
> Stoyan
>
> ---
> Dr. Stoyan P. Pavlov, MD, PhD
> Departament of Anatomy, Histology and Embryology
> Medical University "Prof. Dr. Paraskev Stoyanov", Varna
> Prof. Marin Drinov Str.55
> 9002 Varna
>  Bulgaria
> Tel: +359 (0) 52 - 677 - 052
> e-mail: [hidden email]
>            [hidden email]
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[ctrueden]

Hi Stoyan, Herbie, everyone,

Stoyan wrote:
> First I want to emphasize, how astonishing and useful I find the
> trakem2 package/plugin! It is really great and I am really thankful to
> the developers for puting all this effort into creating this suite for
> working with large image datasets.

Glad to hear it is useful for you.

> Recently I stumbled upon several issues

Thank you for the detailed report. Hopefully Albert or Stephan will be able
to comment. At minimum, we should file these as issues in the GitHub
repository, so they are not forgotten. If you do not hear back further from
anyone, you can file them at https://github.com/trakem2/TrakEM2/issues, or
ping me and I will take care of it.

Herbie wrote:
> I have no idea about "trakem2"

See this page for details:
  http://imagej.net/TrakEM2

> JEPG compression in 99% of all cases is a lossy method and more or
> less changes the image content, i.e. it introduces artifacts.

Indeed—there is a discussion about that here:

http://imagej.net/Principles#Why_.28lossy.29_JPEGs_should_not_be_used_in_imaging

That said, there are cases where it is acceptable to use JPEG. Essentially,
if you have massively large data, and you do not plan to analyze
quantitatively for the most part, it might be OK for pragmatic reasons, as
long as you understand what you are doing.

> With respect to "trakem2"-issues I'd contact the authors directly.

I suggest keeping the discussion public. From the Help page (
http://imagej.net/Help):

> The ImageJ community firmly believes in public discussion of the
> software. In this way, each question benefits not only the one asking,
> but everyone in the community, including everyone who subsequently
> does a web search on the same topic.

Regards,
Curtis

--
Curtis Rueden
LOCI software architect - https://loci.wisc.edu/software
ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden
Did you know ImageJ has a forum? http://forum.imagej.net/


On Wed, Feb 8, 2017 at 4:21 AM, Herbie <[hidden email]> wrote:

> Good day Stoyan,
>
> I have no idea about "trakem2" but please let me remark that using
> JEPG-compressed images for scientific purposes is an absolute no-no. JEPG
> compression in 99% of all cases is a lossy method and more or less changes
> the image content, i.e. it introduces artifacts.
>
> With respect to "trakem2"-issues I'd contact the authors directly.
>
> Best
>
> Herbie
>
> :::::::::::::::::::::::::::::::::::::::::::
> Am 08.02.17 um 10:40 schrieb Stoyan Pavlov:
>
> Dear friends and colleagues,
>> First I want to emphasize, how astonishing and useful I find the trakem2
>> package/plugin! It is really great and I am really thankful to the
>> developers for puting all this effort into creating this suite for working
>> with large image datasets.
>> I use it mainly for creating/stitching/blending , processing and exporting
>> of gigapixel montages of microscopic images into a readable software
>> independent image pyramid.
>> Recently I stumbled upon several issues (I don't know whether they are
>> bugs
>> or just due to limitation in the software) for which I couldn't find any
>> mention and/or solution in the documentation and the forum.
>>
>> First I will describe what I am doing and next I will list the issues that
>> bother me.
>> I am creating a single layer montage with dimensions 63698 x 57218 px from
>> a set of 3111 images (each 1388x1040). The images are acquired via
>> axiovision and exported as single tiles (24bit RGB in either jpg or tif
>> format). Before importing the images I align them using the MIST plugin
>> and
>> use the global positions generated from it to import them into trakem2 via
>> text file. The resulting montage is exactly as expected and have no
>> remarks
>> about it.
>> 1) I don't know if its true but it seems to me that if I use jpg as format
>> for the source images they are imported faster into trakem2. However once
>> the import is finished the software returns errors about generating
>> mipmaps
>> (see attached log file) and it continuously tries to regenerate them
>> (which
>> I guess keeps CPU quite busy). This happens regardless mipmaps are enabled
>> or disabled in Display>Properties, and is independent from the number of
>> threads assigned to mipmap generation. If I close the project and Fiji
>> (even after system restart) and then I open them again the issue returns.
>> This behaviour is not oberved if my source images are in tif format.
>> The other issues are related to the export of a flat image (here the file
>> format of the source images does not matter):
>>
>> 2) When I am trying to export to a tiled format (Export> Make flat
>> image...>Save for web (CATMAID)) regardless of the image format I choose
>> for the export I cannot change the size of the exported tiles to other
>> than
>> 256x256. The dialogue allows me to change it, but when the process
>> finishes
>> in the folder there are only 256x256 tile images. While this is not a huge
>> problem, still there is difference between managing some 80000 images or
>> 40000.
>>
>> 3) Also when I am trying to export tiled format  via Export> Make flat
>> image...>Save for web (CATMAID), if the patches are deselected only a
>> portion of the image is exported . Regardless of the format of the source
>> images and of the exported images there are only 63 rows by 63 columns of
>> 256x256 tiles at zoom level 0 in the folder with the exported images. All
>> metadata in the json file is correct and there are no errors reported by
>> the System log or the Fiji Log. If the patches are selected with Select
>> All
>> (Ctrl + A) the image is exported in its entirety, however here comes the
>> next not as much as an issue but slight discomfort.
>>
>> 4) The image is always exported to a quadrangular array consisting (for
>> the
>> reasons already described) of 256 rows and 256 columns of 256x256 tiles at
>> zoom level 0. That is the image dimensions (63698 x 57218 px) are expanded
>> to 65536x65536 and filled with black tiles. This of course take up
>> additional space and need to be cropped away before uploading the image to
>> our servers. I understand that this is probably because of limitations to
>> the algorithm but still if there is some workaround it would have been
>> better.
>>
>> 5) I thought that this problems (2-4) might be due to the unequal
>> dimensions of the original source images and I tried to work around them
>> by
>> creating a sibling project with retiled images. Here I had another
>> unexpected result - the image was exported but the larger dimension was
>> cut
>> by about 6000 pxs from the right side  and than restored to the original
>> size by adding black pixels , which basically cuts away the right portion
>> of my image (see attached screenshot) .
>>
>> The workstation that I use has the following parameters:
>>  CPU:  Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core processors)
>> RAM: 64 GB
>> OS: Windows 7 64bit
>> In an attachment I added the ImageJ-properties output, portion of the
>> trakem2 log with the errors generating the mipmaps, and a screenshot of
>> the
>> output of the sibling project with retiled images.
>>
>> I hope someone will find a way to correct these problems, because surely
>> they are not fatal and there are workarounds that we use, but it is
>> certainly annoying.
>>
>> Best regards,
>> Stoyan
>>
>> ---
>> Dr. Stoyan P. Pavlov, MD, PhD
>> Departament of Anatomy, Histology and Embryology
>> Medical University "Prof. Dr. Paraskev Stoyanov", Varna
>> Prof. Marin Drinov Str.55
>> 9002 Varna
>>  Bulgaria
>> Tel: +359 (0) 52 - 677 - 052
>> e-mail: [hidden email]
>>            [hidden email]
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>
>>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Stoyan Pavlov]

Good day Herbie,
Since you have " no idea about "trakem2" " why are you answering? Please if
you don't have anything to say about the issue at hand don't flood the
channel.
I am perfectly aware about the scientific use of images and how and which
filetypes to use in analysis. Even more I am one of the people, who
constantly are advocating and helping younger researchers to use lossless
methods for image acquisition and analysis. Nowhere I ever mentioned in my
email about densitometry or colorimetry , neither have I mentioned about
analysis. I had  very specific questions regarding a popular tool developed
by the community and for the 8 years since I started using ImageJ this was
always the ideal place to start researching a problem.
Thanks for nothing,
Stoyan


На 8.02.2017 г. 12:53 сл.об. "Herbie" <[hidden email]> написа:

Good day Stoyan,

I have no idea about "trakem2" but please let me remark that using
JEPG-compressed images for scientific purposes is an absolute no-no. JEPG
compression in 99% of all cases is a lossy method and more or less changes
the image content, i.e. it introduces artifacts.

With respect to "trakem2"-issues I'd contact the authors directly.

Best

Herbie

:::::::::::::::::::::::::::::::::::::::::::
Am 08.02.17 um 10:40 schrieb Stoyan Pavlov:

> Dear friends and colleagues,
> First I want to emphasize, how astonishing and useful I find the trakem2
> package/plugin! It is really great and I am really thankful to the
> developers for puting all this effort into creating this suite for working
> with large image datasets.
> I use it mainly for creating/stitching/blending , processing and exporting
> of gigapixel montages of microscopic images into a readable software
> independent image pyramid.
> Recently I stumbled upon several issues (I don't know whether they are bugs
> or just due to limitation in the software) for which I couldn't find any
> mention and/or solution in the documentation and the forum.
>
> First I will describe what I am doing and next I will list the issues that
> bother me.
> I am creating a single layer montage with dimensions 63698 x 57218 px from
> a set of 3111 images (each 1388x1040). The images are acquired via
> axiovision and exported as single tiles (24bit RGB in either jpg or tif
> format). Before importing the images I align them using the MIST plugin and
> use the global positions generated from it to import them into trakem2 via
> text file. The resulting montage is exactly as expected and have no remarks
> about it.
> 1) I don't know if its true but it seems to me that if I use jpg as format
> for the source images they are imported faster into trakem2. However once
> the import is finished the software returns errors about generating mipmaps
> (see attached log file) and it continuously tries to regenerate them (which
> I guess keeps CPU quite busy). This happens regardless mipmaps are enabled
> or disabled in Display>Properties, and is independent from the number of
> threads assigned to mipmap generation. If I close the project and Fiji
> (even after system restart) and then I open them again the issue returns.
> This behaviour is not oberved if my source images are in tif format.
> The other issues are related to the export of a flat image (here the file
> format of the source images does not matter):
>
> 2) When I am trying to export to a tiled format (Export> Make flat
> image...>Save for web (CATMAID)) regardless of the image format I choose
> for the export I cannot change the size of the exported tiles to other than
> 256x256. The dialogue allows me to change it, but when the process finishes
> in the folder there are only 256x256 tile images. While this is not a huge
> problem, still there is difference between managing some 80000 images or
> 40000.
>
> 3) Also when I am trying to export tiled format  via Export> Make flat
> image...>Save for web (CATMAID), if the patches are deselected only a
> portion of the image is exported . Regardless of the format of the source
> images and of the exported images there are only 63 rows by 63 columns of
> 256x256 tiles at zoom level 0 in the folder with the exported images. All
> metadata in the json file is correct and there are no errors reported by
> the System log or the Fiji Log. If the patches are selected with Select All
> (Ctrl + A) the image is exported in its entirety, however here comes the
> next not as much as an issue but slight discomfort.
>
> 4) The image is always exported to a quadrangular array consisting (for the
> reasons already described) of 256 rows and 256 columns of 256x256 tiles at
> zoom level 0. That is the image dimensions (63698 x 57218 px) are expanded
> to 65536x65536 and filled with black tiles. This of course take up
> additional space and need to be cropped away before uploading the image to
> our servers. I understand that this is probably because of limitations to
> the algorithm but still if there is some workaround it would have been
> better.
>
> 5) I thought that this problems (2-4) might be due to the unequal
> dimensions of the original source images and I tried to work around them by
> creating a sibling project with retiled images. Here I had another
> unexpected result - the image was exported but the larger dimension was cut
> by about 6000 pxs from the right side  and than restored to the original
> size by adding black pixels , which basically cuts away the right portion
> of my image (see attached screenshot) .
>
> The workstation that I use has the following parameters:
>  CPU:  Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core processors)
> RAM: 64 GB
> OS: Windows 7 64bit
> In an attachment I added the ImageJ-properties output, portion of the
> trakem2 log with the errors generating the mipmaps, and a screenshot of the
> output of the sibling project with retiled images.
>
> I hope someone will find a way to correct these problems, because surely
> they are not fatal and there are workarounds that we use, but it is
> certainly annoying.
>
> Best regards,
> Stoyan
>
> ---
> Dr. Stoyan P. Pavlov, MD, PhD
> Departament of Anatomy, Histology and Embryology
> Medical University "Prof. Dr. Paraskev Stoyanov", Varna
> Prof. Marin Drinov Str.55
> 9002 Varna
>  Bulgaria
> Tel: +359 (0) 52 - 677 - 052
> e-mail: [hidden email]
>            [hidden email]
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Stoyan Pavlov]

Hello Curtis,
Thank you very much. I will wait a while for an answer. Last time they
answered pretty quickly. If however there's no movement on this I will
probably  accept your offer to file the bug report as proposed, because I
am not familiar with github and coding.

Best regards!
Stoyan

P.S. Sorry about the outbirst from  few minutes ago!
На 8.02.2017 г. 1:40 сл.об. "Curtis Rueden" <[hidden email]> написа:

Hi Stoyan, Herbie, everyone,

Stoyan wrote:
> First I want to emphasize, how astonishing and useful I find the
> trakem2 package/plugin! It is really great and I am really thankful to
> the developers for puting all this effort into creating this suite for
> working with large image datasets.

Glad to hear it is useful for you.

> Recently I stumbled upon several issues

Thank you for the detailed report. Hopefully Albert or Stephan will be able
to comment. At minimum, we should file these as issues in the GitHub
repository, so they are not forgotten. If you do not hear back further from
anyone, you can file them at https://github.com/trakem2/TrakEM2/issues, or
ping me and I will take care of it.

Herbie wrote:
> I have no idea about "trakem2"

See this page for details:
  http://imagej.net/TrakEM2

> JEPG compression in 99% of all cases is a lossy method and more or
> less changes the image content, i.e. it introduces artifacts.

Indeed—there is a discussion about that here:

http://imagej.net/Principles#Why_.28lossy.29_JPEGs_should_
not_be_used_in_imaging

That said, there are cases where it is acceptable to use JPEG. Essentially,
if you have massively large data, and you do not plan to analyze
quantitatively for the most part, it might be OK for pragmatic reasons, as
long as you understand what you are doing.

> With respect to "trakem2"-issues I'd contact the authors directly.

I suggest keeping the discussion public. From the Help page (
http://imagej.net/Help):

> The ImageJ community firmly believes in public discussion of the
> software. In this way, each question benefits not only the one asking,
> but everyone in the community, including everyone who subsequently
> does a web search on the same topic.

Regards,
Curtis

--
Curtis Rueden
LOCI software architect - https://loci.wisc.edu/software
ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden
Did you know ImageJ has a forum? http://forum.imagej.net/


On Wed, Feb 8, 2017 at 4:21 AM, Herbie <[hidden email]> wrote:

> Good day Stoyan,
>
> I have no idea about "trakem2" but please let me remark that using
> JEPG-compressed images for scientific purposes is an absolute no-no. JEPG
> compression in 99% of all cases is a lossy method and more or less changes
> the image content, i.e. it introduces artifacts.
>
> With respect to "trakem2"-issues I'd contact the authors directly.
>
> Best
>
> Herbie
>
> :::::::::::::::::::::::::::::::::::::::::::
> Am 08.02.17 um 10:40 schrieb Stoyan Pavlov:
>
> Dear friends and colleagues,
>> First I want to emphasize, how astonishing and useful I find the trakem2
>> package/plugin! It is really great and I am really thankful to the
>> developers for puting all this effort into creating this suite for

working
>> with large image datasets.
>> I use it mainly for creating/stitching/blending , processing and
exporting
>> of gigapixel montages of microscopic images into a readable software
>> independent image pyramid.
>> Recently I stumbled upon several issues (I don't know whether they are
>> bugs
>> or just due to limitation in the software) for which I couldn't find any
>> mention and/or solution in the documentation and the forum.
>>
>> First I will describe what I am doing and next I will list the issues
that
>> bother me.
>> I am creating a single layer montage with dimensions 63698 x 57218 px
from
>> a set of 3111 images (each 1388x1040). The images are acquired via
>> axiovision and exported as single tiles (24bit RGB in either jpg or tif
>> format). Before importing the images I align them using the MIST plugin
>> and
>> use the global positions generated from it to import them into trakem2
via
>> text file. The resulting montage is exactly as expected and have no
>> remarks
>> about it.
>> 1) I don't know if its true but it seems to me that if I use jpg as
format
>> for the source images they are imported faster into trakem2. However once
>> the import is finished the software returns errors about generating
>> mipmaps
>> (see attached log file) and it continuously tries to regenerate them
>> (which
>> I guess keeps CPU quite busy). This happens regardless mipmaps are
enabled

>> or disabled in Display>Properties, and is independent from the number of
>> threads assigned to mipmap generation. If I close the project and Fiji
>> (even after system restart) and then I open them again the issue returns.
>> This behaviour is not oberved if my source images are in tif format.
>> The other issues are related to the export of a flat image (here the file
>> format of the source images does not matter):
>>
>> 2) When I am trying to export to a tiled format (Export> Make flat
>> image...>Save for web (CATMAID)) regardless of the image format I choose
>> for the export I cannot change the size of the exported tiles to other
>> than
>> 256x256. The dialogue allows me to change it, but when the process
>> finishes
>> in the folder there are only 256x256 tile images. While this is not a

huge

>> problem, still there is difference between managing some 80000 images or
>> 40000.
>>
>> 3) Also when I am trying to export tiled format  via Export> Make flat
>> image...>Save for web (CATMAID), if the patches are deselected only a
>> portion of the image is exported . Regardless of the format of the source
>> images and of the exported images there are only 63 rows by 63 columns of
>> 256x256 tiles at zoom level 0 in the folder with the exported images. All
>> metadata in the json file is correct and there are no errors reported by
>> the System log or the Fiji Log. If the patches are selected with Select
>> All
>> (Ctrl + A) the image is exported in its entirety, however here comes the
>> next not as much as an issue but slight discomfort.
>>
>> 4) The image is always exported to a quadrangular array consisting (for
>> the
>> reasons already described) of 256 rows and 256 columns of 256x256 tiles

at
>> zoom level 0. That is the image dimensions (63698 x 57218 px) are
expanded
>> to 65536x65536 and filled with black tiles. This of course take up
>> additional space and need to be cropped away before uploading the image
to

>> our servers. I understand that this is probably because of limitations to
>> the algorithm but still if there is some workaround it would have been
>> better.
>>
>> 5) I thought that this problems (2-4) might be due to the unequal
>> dimensions of the original source images and I tried to work around them
>> by
>> creating a sibling project with retiled images. Here I had another
>> unexpected result - the image was exported but the larger dimension was
>> cut
>> by about 6000 pxs from the right side  and than restored to the original
>> size by adding black pixels , which basically cuts away the right portion
>> of my image (see attached screenshot) .
>>
>> The workstation that I use has the following parameters:
>>  CPU:  Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core processors)
>> RAM: 64 GB
>> OS: Windows 7 64bit
>> In an attachment I added the ImageJ-properties output, portion of the
>> trakem2 log with the errors generating the mipmaps, and a screenshot of
>> the
>> output of the sibling project with retiled images.
>>
>> I hope someone will find a way to correct these problems, because surely
>> they are not fatal and there are workarounds that we use, but it is
>> certainly annoying.
>>
>> Best regards,
>> Stoyan
>>
>> ---
>> Dr. Stoyan P. Pavlov, MD, PhD
>> Departament of Anatomy, Histology and Embryology
>> Medical University "Prof. Dr. Paraskev Stoyanov", Varna
>> Prof. Marin Drinov Str.55
>> 9002 Varna
>>  Bulgaria
>> Tel: +359 (0) 52 - 677 - 052
>> e-mail: [hidden email]
>>            [hidden email]
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>>
>>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Saalfeld, Stephan]

Hi,

TrakEM2's internal CATMAID export mechanisms goes through single image
exports.  This is limited to 2GPx planes (sad---right?).  Please use
this script instead:

https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/catmaid-
export2.bsh

Regarding the mipmap-madness...:

1. Is your project directory on a file share?  If yes, please move it
to a local (external?) harddrive, you are working with 2D images, that
should fit.  TrakEM2 is somehow sensitive to how network drives
frequently fail or rather delay what they are supposed to do.  This is
an annoyance in TrakEM2 and could/ should be fixed.  No time though...

2. If you do not want to use mipmaps, switch mipmaps off *before*
importing tiles.  Switching them off during the madness has no effects,
too many threads running already and they will not get properly
interrupted.

3. JPG import works faster because JPGs read faster, this sounds like
you are I/O limited, which makes me believe in 1. more ;).  Oh yes, are
all your images the same size and have intensities 0--255?  Then you
can use the extended import file syntax

https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/main/java/ini/trakem2/persistence/Loader.java#L2013-L2027

if you specify the columns 5--9 and switch off mipmaps, none of your
images will be touched during import, i.e. import will be almost
instantaneous.  Save the project, do the rest.

Let me know if that helps or where it breaks.

Cheers,
Stephan

On Wed, 2017-02-08 at 16:23 +0200, Stoyan Pavlov wrote:

> Hello Curtis,
> Thank you very much. I will wait a while for an answer. Last time
> they
> answered pretty quickly. If however there's no movement on this I
> will
> probably  accept your offer to file the bug report as proposed,
> because I
> am not familiar with github and coding.
>
> Best regards!
> Stoyan
>
> P.S. Sorry about the outbirst from  few minutes ago!
> На 8.02.2017 г. 1:40 сл.об. "Curtis Rueden" <[hidden email]>
> написа:
>
> Hi Stoyan, Herbie, everyone,
>
> Stoyan wrote:
> >
> > First I want to emphasize, how astonishing and useful I find the
> > trakem2 package/plugin! It is really great and I am really thankful
> > to
> > the developers for puting all this effort into creating this suite
> > for
> > working with large image datasets.
> Glad to hear it is useful for you.
>
> >
> > Recently I stumbled upon several issues
> Thank you for the detailed report. Hopefully Albert or Stephan will
> be able
> to comment. At minimum, we should file these as issues in the GitHub
> repository, so they are not forgotten. If you do not hear back
> further from
> anyone, you can file them at https://github.com/trakem2/TrakEM2/issue
> s, or
> ping me and I will take care of it.
>
> Herbie wrote:
> >
> > I have no idea about "trakem2"
> See this page for details:
>   http://imagej.net/TrakEM2
>
> >
> > JEPG compression in 99% of all cases is a lossy method and more or
> > less changes the image content, i.e. it introduces artifacts.
> Indeed—there is a discussion about that here:
>
> http://imagej.net/Principles#Why_.28lossy.29_JPEGs_should_
> not_be_used_in_imaging
>
> That said, there are cases where it is acceptable to use JPEG.
> Essentially,
> if you have massively large data, and you do not plan to analyze
> quantitatively for the most part, it might be OK for pragmatic
> reasons, as
> long as you understand what you are doing.
>
> >
> > With respect to "trakem2"-issues I'd contact the authors directly.
> I suggest keeping the discussion public. From the Help page (
> http://imagej.net/Help):
>
> >
> > The ImageJ community firmly believes in public discussion of the
> > software. In this way, each question benefits not only the one
> > asking,
> > but everyone in the community, including everyone who subsequently
> > does a web search on the same topic.
> Regards,
> Curtis
>
> --
> Curtis Rueden
> LOCI software architect - https://loci.wisc.edu/software
> ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden
> Did you know ImageJ has a forum? http://forum.imagej.net/
>
>
> On Wed, Feb 8, 2017 at 4:21 AM, Herbie <[hidden email]> wrote:
>
> >
> > Good day Stoyan,
> >
> > I have no idea about "trakem2" but please let me remark that using
> > JEPG-compressed images for scientific purposes is an absolute no-
> > no. JEPG
> > compression in 99% of all cases is a lossy method and more or less
> > changes
> > the image content, i.e. it introduces artifacts.
> >
> > With respect to "trakem2"-issues I'd contact the authors directly.
> >
> > Best
> >
> > Herbie
> >
> > :::::::::::::::::::::::::::::::::::::::::::
> > Am 08.02.17 um 10:40 schrieb Stoyan Pavlov:
> >
> > Dear friends and colleagues,
> > >
> > > First I want to emphasize, how astonishing and useful I find the
> > > trakem2
> > > package/plugin! It is really great and I am really thankful to
> > > the
> > > developers for puting all this effort into creating this suite
> > > for
> working
> >
> > >
> > > with large image datasets.
> > > I use it mainly for creating/stitching/blending , processing and
> exporting
> >
> > >
> > > of gigapixel montages of microscopic images into a readable
> > > software
> > > independent image pyramid.
> > > Recently I stumbled upon several issues (I don't know whether
> > > they are
> > > bugs
> > > or just due to limitation in the software) for which I couldn't
> > > find any
> > > mention and/or solution in the documentation and the forum.
> > >
> > > First I will describe what I am doing and next I will list the
> > > issues
> that
> >
> > >
> > > bother me.
> > > I am creating a single layer montage with dimensions 63698 x
> > > 57218 px
> from
> >
> > >
> > > a set of 3111 images (each 1388x1040). The images are acquired
> > > via
> > > axiovision and exported as single tiles (24bit RGB in either jpg
> > > or tif
> > > format). Before importing the images I align them using the MIST
> > > plugin
> > > and
> > > use the global positions generated from it to import them into
> > > trakem2
> via
> >
> > >
> > > text file. The resulting montage is exactly as expected and have
> > > no
> > > remarks
> > > about it.
> > > 1) I don't know if its true but it seems to me that if I use jpg
> > > as
> format
> >
> > >
> > > for the source images they are imported faster into trakem2.
> > > However once
> > > the import is finished the software returns errors about
> > > generating
> > > mipmaps
> > > (see attached log file) and it continuously tries to regenerate
> > > them
> > > (which
> > > I guess keeps CPU quite busy). This happens regardless mipmaps
> > > are
> enabled
> >
> > >
> > > or disabled in Display>Properties, and is independent from the
> > > number of
> > > threads assigned to mipmap generation. If I close the project and
> > > Fiji
> > > (even after system restart) and then I open them again the issue
> > > returns.
> > > This behaviour is not oberved if my source images are in tif
> > > format.
> > > The other issues are related to the export of a flat image (here
> > > the file
> > > format of the source images does not matter):
> > >
> > > 2) When I am trying to export to a tiled format (Export> Make
> > > flat
> > > image...>Save for web (CATMAID)) regardless of the image format I
> > > choose
> > > for the export I cannot change the size of the exported tiles to
> > > other
> > > than
> > > 256x256. The dialogue allows me to change it, but when the
> > > process
> > > finishes
> > > in the folder there are only 256x256 tile images. While this is
> > > not a
> huge
> >
> > >
> > > problem, still there is difference between managing some 80000
> > > images or
> > > 40000.
> > >
> > > 3) Also when I am trying to export tiled format  via Export> Make
> > > flat
> > > image...>Save for web (CATMAID), if the patches are deselected
> > > only a
> > > portion of the image is exported . Regardless of the format of
> > > the source
> > > images and of the exported images there are only 63 rows by 63
> > > columns of
> > > 256x256 tiles at zoom level 0 in the folder with the exported
> > > images. All
> > > metadata in the json file is correct and there are no errors
> > > reported by
> > > the System log or the Fiji Log. If the patches are selected with
> > > Select
> > > All
> > > (Ctrl + A) the image is exported in its entirety, however here
> > > comes the
> > > next not as much as an issue but slight discomfort.
> > >
> > > 4) The image is always exported to a quadrangular array
> > > consisting (for
> > > the
> > > reasons already described) of 256 rows and 256 columns of 256x256
> > > tiles
> at
> >
> > >
> > > zoom level 0. That is the image dimensions (63698 x 57218 px) are
> expanded
> >
> > >
> > > to 65536x65536 and filled with black tiles. This of course take
> > > up
> > > additional space and need to be cropped away before uploading the
> > > image
> to
> >
> > >
> > > our servers. I understand that this is probably because of
> > > limitations to
> > > the algorithm but still if there is some workaround it would have
> > > been
> > > better.
> > >
> > > 5) I thought that this problems (2-4) might be due to the unequal
> > > dimensions of the original source images and I tried to work
> > > around them
> > > by
> > > creating a sibling project with retiled images. Here I had
> > > another
> > > unexpected result - the image was exported but the larger
> > > dimension was
> > > cut
> > > by about 6000 pxs from the right side  and than restored to the
> > > original
> > > size by adding black pixels , which basically cuts away the right
> > > portion
> > > of my image (see attached screenshot) .
> > >
> > > The workstation that I use has the following parameters:
> > >  CPU:  Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core processors)
> > > RAM: 64 GB
> > > OS: Windows 7 64bit
> > > In an attachment I added the ImageJ-properties output, portion of
> > > the
> > > trakem2 log with the errors generating the mipmaps, and a
> > > screenshot of
> > > the
> > > output of the sibling project with retiled images.
> > >
> > > I hope someone will find a way to correct these problems, because
> > > surely
> > > they are not fatal and there are workarounds that we use, but it
> > > is
> > > certainly annoying.
> > >
> > > Best regards,
> > > Stoyan
> > >
> > > ---
> > > Dr. Stoyan P. Pavlov, MD, PhD
> > > Departament of Anatomy, Histology and Embryology
> > > Medical University "Prof. Dr. Paraskev Stoyanov", Varna
> > > Prof. Marin Drinov Str.55
> > > 9002 Varna
> > >  Bulgaria
> > > Tel: +359 (0) 52 - 677 - 052
> > > e-mail: [hidden email]
> > >            [hidden email]
> > >
> > > --
> > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > >
> > >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Stoyan Pavlov]

Hi Stephan,
Thank you very much! I will try the scripts, first thing in the morning. I
figured as much. After all exactly this 2G java limitation for arrays is
what brought me to trakem2. ;-)

 >>  1. Is your project directory on a file share?

No. Everything is saved locally.


>>  2. If you do not want to use mipmaps, switch mipmaps >>off *before*
>>importing tiles.  Switching them off during the madness >>has no effects

Indeed I observed this behaviour: even if one cancels the mipmap generation
job, it continues in Fiji until finished. Only workaround is to quit Fiji .
But when the project is loaded it tries to regenerate the mipmaps...

I guess the java loader alternative is used the sameway any other script?
(New>Script etc...)

I'll let you know whether it all worked.

Best regards,
Stoyan

На 8.02.2017 г. 5:22 сл.об. "Saalfeld, Stephan" <[hidden email]>
написа:

Hi,

TrakEM2's internal CATMAID export mechanisms goes through single image
exports.  This is limited to 2GPx planes (sad---right?).  Please use
this script instead:

https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/catmaid-
export2.bsh

Regarding the mipmap-madness...:

1. Is your project directory on a file share?  If yes, please move it
to a local (external?) harddrive, you are working with 2D images, that
should fit.  TrakEM2 is somehow sensitive to how network drives
frequently fail or rather delay what they are supposed to do.  This is
an annoyance in TrakEM2 and could/ should be fixed.  No time though...

2. If you do not want to use mipmaps, switch mipmaps off *before*
importing tiles.  Switching them off during the madness has no effects,
too many threads running already and they will not get properly
interrupted.

3. JPG import works faster because JPGs read faster, this sounds like
you are I/O limited, which makes me believe in 1. more ;).  Oh yes, are
all your images the same size and have intensities 0--255?  Then you
can use the extended import file syntax

https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/
src/main/java/ini/trakem2/persistence/Loader.java#L2013-L2027

if you specify the columns 5--9 and switch off mipmaps, none of your
images will be touched during import, i.e. import will be almost
instantaneous.  Save the project, do the rest.

Let me know if that helps or where it breaks.

Cheers,
Stephan

On Wed, 2017-02-08 at 16:23 +0200, Stoyan Pavlov wrote:

> Hello Curtis,
> Thank you very much. I will wait a while for an answer. Last time
> they
> answered pretty quickly. If however there's no movement on this I
> will
> probably  accept your offer to file the bug report as proposed,
> because I
> am not familiar with github and coding.
>
> Best regards!
> Stoyan
>
> P.S. Sorry about the outbirst from  few minutes ago!
> На 8.02.2017 г. 1:40 сл.об. "Curtis Rueden" <[hidden email]>
> написа:
>
> Hi Stoyan, Herbie, everyone,
>
> Stoyan wrote:
> >
> > First I want to emphasize, how astonishing and useful I find the
> > trakem2 package/plugin! It is really great and I am really thankful
> > to
> > the developers for puting all this effort into creating this suite
> > for
> > working with large image datasets.
> Glad to hear it is useful for you.
>
> >
> > Recently I stumbled upon several issues
> Thank you for the detailed report. Hopefully Albert or Stephan will
> be able
> to comment. At minimum, we should file these as issues in the GitHub
> repository, so they are not forgotten. If you do not hear back
> further from
> anyone, you can file them at https://github.com/trakem2/TrakEM2/issue
> s, or
> ping me and I will take care of it.
>
> Herbie wrote:
> >
> > I have no idea about "trakem2"
> See this page for details:
>   http://imagej.net/TrakEM2
>
> >
> > JEPG compression in 99% of all cases is a lossy method and more or
> > less changes the image content, i.e. it introduces artifacts.
> Indeed—there is a discussion about that here:
>
> http://imagej.net/Principles#Why_.28lossy.29_JPEGs_should_
> not_be_used_in_imaging
>
> That said, there are cases where it is acceptable to use JPEG.
> Essentially,
> if you have massively large data, and you do not plan to analyze
> quantitatively for the most part, it might be OK for pragmatic
> reasons, as
> long as you understand what you are doing.
>
> >
> > With respect to "trakem2"-issues I'd contact the authors directly.
> I suggest keeping the discussion public. From the Help page (
> http://imagej.net/Help):
>
> >
> > The ImageJ community firmly believes in public discussion of the
> > software. In this way, each question benefits not only the one
> > asking,
> > but everyone in the community, including everyone who subsequently
> > does a web search on the same topic.
> Regards,
> Curtis
>
> --
> Curtis Rueden
> LOCI software architect - https://loci.wisc.edu/software
> ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden
> Did you know ImageJ has a forum? http://forum.imagej.net/
>
>
> On Wed, Feb 8, 2017 at 4:21 AM, Herbie <[hidden email]> wrote:
>
> >
> > Good day Stoyan,
> >
> > I have no idea about "trakem2" but please let me remark that using
> > JEPG-compressed images for scientific purposes is an absolute no-
> > no. JEPG
> > compression in 99% of all cases is a lossy method and more or less
> > changes
> > the image content, i.e. it introduces artifacts.
> >
> > With respect to "trakem2"-issues I'd contact the authors directly.
> >
> > Best
> >
> > Herbie
> >
> > :::::::::::::::::::::::::::::::::::::::::::
> > Am 08.02.17 um 10:40 schrieb Stoyan Pavlov:
> >
> > Dear friends and colleagues,
> > >
> > > First I want to emphasize, how astonishing and useful I find the
> > > trakem2
> > > package/plugin! It is really great and I am really thankful to
> > > the
> > > developers for puting all this effort into creating this suite
> > > for
> working
> >
> > >
> > > with large image datasets.
> > > I use it mainly for creating/stitching/blending , processing and
> exporting
> >
> > >
> > > of gigapixel montages of microscopic images into a readable
> > > software
> > > independent image pyramid.
> > > Recently I stumbled upon several issues (I don't know whether
> > > they are
> > > bugs
> > > or just due to limitation in the software) for which I couldn't
> > > find any
> > > mention and/or solution in the documentation and the forum.
> > >
> > > First I will describe what I am doing and next I will list the
> > > issues
> that
> >
> > >
> > > bother me.
> > > I am creating a single layer montage with dimensions 63698 x
> > > 57218 px
> from
> >
> > >
> > > a set of 3111 images (each 1388x1040). The images are acquired
> > > via
> > > axiovision and exported as single tiles (24bit RGB in either jpg
> > > or tif
> > > format). Before importing the images I align them using the MIST
> > > plugin
> > > and
> > > use the global positions generated from it to import them into
> > > trakem2
> via
> >
> > >
> > > text file. The resulting montage is exactly as expected and have
> > > no
> > > remarks
> > > about it.
> > > 1) I don't know if its true but it seems to me that if I use jpg
> > > as
> format
> >
> > >
> > > for the source images they are imported faster into trakem2.
> > > However once
> > > the import is finished the software returns errors about
> > > generating
> > > mipmaps
> > > (see attached log file) and it continuously tries to regenerate
> > > them
> > > (which
> > > I guess keeps CPU quite busy). This happens regardless mipmaps
> > > are
> enabled
> >
> > >
> > > or disabled in Display>Properties, and is independent from the
> > > number of
> > > threads assigned to mipmap generation. If I close the project and
> > > Fiji
> > > (even after system restart) and then I open them again the issue
> > > returns.
> > > This behaviour is not oberved if my source images are in tif
> > > format.
> > > The other issues are related to the export of a flat image (here
> > > the file
> > > format of the source images does not matter):
> > >
> > > 2) When I am trying to export to a tiled format (Export> Make
> > > flat
> > > image...>Save for web (CATMAID)) regardless of the image format I
> > > choose
> > > for the export I cannot change the size of the exported tiles to
> > > other
> > > than
> > > 256x256. The dialogue allows me to change it, but when the
> > > process
> > > finishes
> > > in the folder there are only 256x256 tile images. While this is
> > > not a
> huge
> >
> > >
> > > problem, still there is difference between managing some 80000
> > > images or
> > > 40000.
> > >
> > > 3) Also when I am trying to export tiled format  via Export> Make
> > > flat
> > > image...>Save for web (CATMAID), if the patches are deselected
> > > only a
> > > portion of the image is exported . Regardless of the format of
> > > the source
> > > images and of the exported images there are only 63 rows by 63
> > > columns of
> > > 256x256 tiles at zoom level 0 in the folder with the exported
> > > images. All
> > > metadata in the json file is correct and there are no errors
> > > reported by
> > > the System log or the Fiji Log. If the patches are selected with
> > > Select
> > > All
> > > (Ctrl + A) the image is exported in its entirety, however here
> > > comes the
> > > next not as much as an issue but slight discomfort.
> > >
> > > 4) The image is always exported to a quadrangular array
> > > consisting (for
> > > the
> > > reasons already described) of 256 rows and 256 columns of 256x256
> > > tiles
> at
> >
> > >
> > > zoom level 0. That is the image dimensions (63698 x 57218 px) are
> expanded
> >
> > >
> > > to 65536x65536 and filled with black tiles. This of course take
> > > up
> > > additional space and need to be cropped away before uploading the
> > > image
> to
> >
> > >
> > > our servers. I understand that this is probably because of
> > > limitations to
> > > the algorithm but still if there is some workaround it would have
> > > been
> > > better.
> > >
> > > 5) I thought that this problems (2-4) might be due to the unequal
> > > dimensions of the original source images and I tried to work
> > > around them
> > > by
> > > creating a sibling project with retiled images. Here I had
> > > another
> > > unexpected result - the image was exported but the larger
> > > dimension was
> > > cut
> > > by about 6000 pxs from the right side  and than restored to the
> > > original
> > > size by adding black pixels , which basically cuts away the right
> > > portion
> > > of my image (see attached screenshot) .
> > >
> > > The workstation that I use has the following parameters:
> > >  CPU:  Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core processors)
> > > RAM: 64 GB
> > > OS: Windows 7 64bit
> > > In an attachment I added the ImageJ-properties output, portion of
> > > the
> > > trakem2 log with the errors generating the mipmaps, and a
> > > screenshot of
> > > the
> > > output of the sibling project with retiled images.
> > >
> > > I hope someone will find a way to correct these problems, because
> > > surely
> > > they are not fatal and there are workarounds that we use, but it
> > > is
> > > certainly annoying.
> > >
> > > Best regards,
> > > Stoyan
> > >
> > > ---
> > > Dr. Stoyan P. Pavlov, MD, PhD
> > > Departament of Anatomy, Histology and Embryology
> > > Medical University "Prof. Dr. Paraskev Stoyanov", Varna
> > > Prof. Marin Drinov Str.55
> > > 9002 Varna
> > >  Bulgaria
> > > Tel: +359 (0) 52 - 677 - 052
> > > e-mail: [hidden email]
> > >            [hidden email]
> > >
> > > --
> > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > >
> > >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Stoyan Pavlov]

Dear Stephan,
I tried the bsh script and it works very well and fast,  but it exports the
 images in gray scale. Is there an easy way to make it export them in RGB?
Best regards
Stoyan

На 8.02.2017 г. 18:30 "Stoyan Pavlov" <[hidden email]> написа:

> Hi Stephan,
> Thank you very much! I will try the scripts, first thing in the morning. I
> figured as much. After all exactly this 2G java limitation for arrays is
> what brought me to trakem2. ;-)
>
>  >>  1. Is your project directory on a file share?
>
> No. Everything is saved locally.
>
>
> >>  2. If you do not want to use mipmaps, switch mipmaps >>off *before*
> >>importing tiles.  Switching them off during the madness >>has no effects
>
> Indeed I observed this behaviour: even if one cancels the mipmap
> generation job, it continues in Fiji until finished. Only workaround is to
> quit Fiji . But when the project is loaded it tries to regenerate the
> mipmaps...
>
> I guess the java loader alternative is used the sameway any other script?
> (New>Script etc...)
>
> I'll let you know whether it all worked.
>
> Best regards,
> Stoyan
>
> На 8.02.2017 г. 5:22 сл.об. "Saalfeld, Stephan" <
> [hidden email]> написа:
>
> Hi,
>
> TrakEM2's internal CATMAID export mechanisms goes through single image
> exports.  This is limited to 2GPx planes (sad---right?).  Please use
> this script instead:
>
> https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/catmaid-
> export2.bsh
>
> Regarding the mipmap-madness...:
>
> 1. Is your project directory on a file share?  If yes, please move it
> to a local (external?) harddrive, you are working with 2D images, that
> should fit.  TrakEM2 is somehow sensitive to how network drives
> frequently fail or rather delay what they are supposed to do.  This is
> an annoyance in TrakEM2 and could/ should be fixed.  No time though...
>
> 2. If you do not want to use mipmaps, switch mipmaps off *before*
> importing tiles.  Switching them off during the madness has no effects,
> too many threads running already and they will not get properly
> interrupted.
>
> 3. JPG import works faster because JPGs read faster, this sounds like
> you are I/O limited, which makes me believe in 1. more ;).  Oh yes, are
> all your images the same size and have intensities 0--255?  Then you
> can use the extended import file syntax
>
> https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/
> main/java/ini/trakem2/persistence/Loader.java#L2013-L2027
>
> if you specify the columns 5--9 and switch off mipmaps, none of your
> images will be touched during import, i.e. import will be almost
> instantaneous.  Save the project, do the rest.
>
> Let me know if that helps or where it breaks.
>
> Cheers,
> Stephan
>
> On Wed, 2017-02-08 at 16:23 +0200, Stoyan Pavlov wrote:
> > Hello Curtis,
> > Thank you very much. I will wait a while for an answer. Last time
> > they
> > answered pretty quickly. If however there's no movement on this I
> > will
> > probably  accept your offer to file the bug report as proposed,
> > because I
> > am not familiar with github and coding.
> >
> > Best regards!
> > Stoyan
> >
> > P.S. Sorry about the outbirst from  few minutes ago!
> > На 8.02.2017 г. 1:40 сл.об. "Curtis Rueden" <[hidden email]>
> > написа:
> >
> > Hi Stoyan, Herbie, everyone,
> >
> > Stoyan wrote:
> > >
> > > First I want to emphasize, how astonishing and useful I find the
> > > trakem2 package/plugin! It is really great and I am really thankful
> > > to
> > > the developers for puting all this effort into creating this suite
> > > for
> > > working with large image datasets.
> > Glad to hear it is useful for you.
> >
> > >
> > > Recently I stumbled upon several issues
> > Thank you for the detailed report. Hopefully Albert or Stephan will
> > be able
> > to comment. At minimum, we should file these as issues in the GitHub
> > repository, so they are not forgotten. If you do not hear back
> > further from
> > anyone, you can file them at https://github.com/trakem2/TrakEM2/issue
> > s, or
> > ping me and I will take care of it.
> >
> > Herbie wrote:
> > >
> > > I have no idea about "trakem2"
> > See this page for details:
> >   http://imagej.net/TrakEM2
> >
> > >
> > > JEPG compression in 99% of all cases is a lossy method and more or
> > > less changes the image content, i.e. it introduces artifacts.
> > Indeed—there is a discussion about that here:
> >
> > http://imagej.net/Principles#Why_.28lossy.29_JPEGs_should_
> > not_be_used_in_imaging
> >
> > That said, there are cases where it is acceptable to use JPEG.
> > Essentially,
> > if you have massively large data, and you do not plan to analyze
> > quantitatively for the most part, it might be OK for pragmatic
> > reasons, as
> > long as you understand what you are doing.
> >
> > >
> > > With respect to "trakem2"-issues I'd contact the authors directly.
> > I suggest keeping the discussion public. From the Help page (
> > http://imagej.net/Help):
> >
> > >
> > > The ImageJ community firmly believes in public discussion of the
> > > software. In this way, each question benefits not only the one
> > > asking,
> > > but everyone in the community, including everyone who subsequently
> > > does a web search on the same topic.
> > Regards,
> > Curtis
> >
> > --
> > Curtis Rueden
> > LOCI software architect - https://loci.wisc.edu/software
> > ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden
> > Did you know ImageJ has a forum? http://forum.imagej.net/
> >
> >
> > On Wed, Feb 8, 2017 at 4:21 AM, Herbie <[hidden email]> wrote:
> >
> > >
> > > Good day Stoyan,
> > >
> > > I have no idea about "trakem2" but please let me remark that using
> > > JEPG-compressed images for scientific purposes is an absolute no-
> > > no. JEPG
> > > compression in 99% of all cases is a lossy method and more or less
> > > changes
> > > the image content, i.e. it introduces artifacts.
> > >
> > > With respect to "trakem2"-issues I'd contact the authors directly.
> > >
> > > Best
> > >
> > > Herbie
> > >
> > > :::::::::::::::::::::::::::::::::::::::::::
> > > Am 08.02.17 um 10:40 schrieb Stoyan Pavlov:
> > >
> > > Dear friends and colleagues,
> > > >
> > > > First I want to emphasize, how astonishing and useful I find the
> > > > trakem2
> > > > package/plugin! It is really great and I am really thankful to
> > > > the
> > > > developers for puting all this effort into creating this suite
> > > > for
> > working
> > >
> > > >
> > > > with large image datasets.
> > > > I use it mainly for creating/stitching/blending , processing and
> > exporting
> > >
> > > >
> > > > of gigapixel montages of microscopic images into a readable
> > > > software
> > > > independent image pyramid.
> > > > Recently I stumbled upon several issues (I don't know whether
> > > > they are
> > > > bugs
> > > > or just due to limitation in the software) for which I couldn't
> > > > find any
> > > > mention and/or solution in the documentation and the forum.
> > > >
> > > > First I will describe what I am doing and next I will list the
> > > > issues
> > that
> > >
> > > >
> > > > bother me.
> > > > I am creating a single layer montage with dimensions 63698 x
> > > > 57218 px
> > from
> > >
> > > >
> > > > a set of 3111 images (each 1388x1040). The images are acquired
> > > > via
> > > > axiovision and exported as single tiles (24bit RGB in either jpg
> > > > or tif
> > > > format). Before importing the images I align them using the MIST
> > > > plugin
> > > > and
> > > > use the global positions generated from it to import them into
> > > > trakem2
> > via
> > >
> > > >
> > > > text file. The resulting montage is exactly as expected and have
> > > > no
> > > > remarks
> > > > about it.
> > > > 1) I don't know if its true but it seems to me that if I use jpg
> > > > as
> > format
> > >
> > > >
> > > > for the source images they are imported faster into trakem2.
> > > > However once
> > > > the import is finished the software returns errors about
> > > > generating
> > > > mipmaps
> > > > (see attached log file) and it continuously tries to regenerate
> > > > them
> > > > (which
> > > > I guess keeps CPU quite busy). This happens regardless mipmaps
> > > > are
> > enabled
> > >
> > > >
> > > > or disabled in Display>Properties, and is independent from the
> > > > number of
> > > > threads assigned to mipmap generation. If I close the project and
> > > > Fiji
> > > > (even after system restart) and then I open them again the issue
> > > > returns.
> > > > This behaviour is not oberved if my source images are in tif
> > > > format.
> > > > The other issues are related to the export of a flat image (here
> > > > the file
> > > > format of the source images does not matter):
> > > >
> > > > 2) When I am trying to export to a tiled format (Export> Make
> > > > flat
> > > > image...>Save for web (CATMAID)) regardless of the image format I
> > > > choose
> > > > for the export I cannot change the size of the exported tiles to
> > > > other
> > > > than
> > > > 256x256. The dialogue allows me to change it, but when the
> > > > process
> > > > finishes
> > > > in the folder there are only 256x256 tile images. While this is
> > > > not a
> > huge
> > >
> > > >
> > > > problem, still there is difference between managing some 80000
> > > > images or
> > > > 40000.
> > > >
> > > > 3) Also when I am trying to export tiled format  via Export> Make
> > > > flat
> > > > image...>Save for web (CATMAID), if the patches are deselected
> > > > only a
> > > > portion of the image is exported . Regardless of the format of
> > > > the source
> > > > images and of the exported images there are only 63 rows by 63
> > > > columns of
> > > > 256x256 tiles at zoom level 0 in the folder with the exported
> > > > images. All
> > > > metadata in the json file is correct and there are no errors
> > > > reported by
> > > > the System log or the Fiji Log. If the patches are selected with
> > > > Select
> > > > All
> > > > (Ctrl + A) the image is exported in its entirety, however here
> > > > comes the
> > > > next not as much as an issue but slight discomfort.
> > > >
> > > > 4) The image is always exported to a quadrangular array
> > > > consisting (for
> > > > the
> > > > reasons already described) of 256 rows and 256 columns of 256x256
> > > > tiles
> > at
> > >
> > > >
> > > > zoom level 0. That is the image dimensions (63698 x 57218 px) are
> > expanded
> > >
> > > >
> > > > to 65536x65536 and filled with black tiles. This of course take
> > > > up
> > > > additional space and need to be cropped away before uploading the
> > > > image
> > to
> > >
> > > >
> > > > our servers. I understand that this is probably because of
> > > > limitations to
> > > > the algorithm but still if there is some workaround it would have
> > > > been
> > > > better.
> > > >
> > > > 5) I thought that this problems (2-4) might be due to the unequal
> > > > dimensions of the original source images and I tried to work
> > > > around them
> > > > by
> > > > creating a sibling project with retiled images. Here I had
> > > > another
> > > > unexpected result - the image was exported but the larger
> > > > dimension was
> > > > cut
> > > > by about 6000 pxs from the right side  and than restored to the
> > > > original
> > > > size by adding black pixels , which basically cuts away the right
> > > > portion
> > > > of my image (see attached screenshot) .
> > > >
> > > > The workstation that I use has the following parameters:
> > > >  CPU:  Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core processors)
> > > > RAM: 64 GB
> > > > OS: Windows 7 64bit
> > > > In an attachment I added the ImageJ-properties output, portion of
> > > > the
> > > > trakem2 log with the errors generating the mipmaps, and a
> > > > screenshot of
> > > > the
> > > > output of the sibling project with retiled images.
> > > >
> > > > I hope someone will find a way to correct these problems, because
> > > > surely
> > > > they are not fatal and there are workarounds that we use, but it
> > > > is
> > > > certainly annoying.
> > > >
> > > > Best regards,
> > > > Stoyan
> > > >
> > > > ---
> > > > Dr. Stoyan P. Pavlov, MD, PhD
> > > > Departament of Anatomy, Histology and Embryology
> > > > Medical University "Prof. Dr. Paraskev Stoyanov", Varna
> > > > Prof. Marin Drinov Str.55
> > > > 9002 Varna
> > > >  Bulgaria
> > > > Tel: +359 (0) 52 - 677 - 052 <052%20677%20052>
> > > > e-mail: [hidden email]
> > > >            [hidden email]
> > > >
> > > > --
> > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > >
> > > >
> > > --
> > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
>
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Saalfeld, Stephan]

Yes,

by changing this line

https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/catmaid-
export2.bsh#L96

to

ImagePlus.COLOR_RGB,



(this is where the fork is made

https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/main/java/i
ni/trakem2/persistence/Loader.java#L2815-L2820

)

Best,
Stephan


On Thu, 2017-02-09 at 13:22 +0200, Stoyan Pavlov wrote:

> Dear Stephan,
> I tried the bsh script and it works very well and fast,  but it
> exports the
>  images in gray scale. Is there an easy way to make it export them in
> RGB?
> Best regards
> Stoyan
>
> На 8.02.2017 г. 18:30 "Stoyan Pavlov" <[hidden email]>
> написа:
>
> >
> > Hi Stephan,
> > Thank you very much! I will try the scripts, first thing in the
> > morning. I
> > figured as much. After all exactly this 2G java limitation for
> > arrays is
> > what brought me to trakem2. ;-)
> >
> >  >>  1. Is your project directory on a file share?
> >
> > No. Everything is saved locally.
> >
> >
> > >
> > > >
> > > >  2. If you do not want to use mipmaps, switch mipmaps >>off
> > > > *before*
> > > > importing tiles.  Switching them off during the madness >>has
> > > > no effects
> > Indeed I observed this behaviour: even if one cancels the mipmap
> > generation job, it continues in Fiji until finished. Only
> > workaround is to
> > quit Fiji . But when the project is loaded it tries to regenerate
> > the
> > mipmaps...
> >
> > I guess the java loader alternative is used the sameway any other
> > script?
> > (New>Script etc...)
> >
> > I'll let you know whether it all worked.
> >
> > Best regards,
> > Stoyan
> >
> > На 8.02.2017 г. 5:22 сл.об. "Saalfeld, Stephan" <
> > [hidden email]> написа:
> >
> > Hi,
> >
> > TrakEM2's internal CATMAID export mechanisms goes through single
> > image
> > exports.  This is limited to 2GPx planes (sad---right?).  Please
> > use
> > this script instead:
> >
> > https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/catm
> > aid-
> > export2.bsh
> >
> > Regarding the mipmap-madness...:
> >
> > 1. Is your project directory on a file share?  If yes, please move
> > it
> > to a local (external?) harddrive, you are working with 2D images,
> > that
> > should fit.  TrakEM2 is somehow sensitive to how network drives
> > frequently fail or rather delay what they are supposed to do.  This
> > is
> > an annoyance in TrakEM2 and could/ should be fixed.  No time
> > though...
> >
> > 2. If you do not want to use mipmaps, switch mipmaps off *before*
> > importing tiles.  Switching them off during the madness has no
> > effects,
> > too many threads running already and they will not get properly
> > interrupted.
> >
> > 3. JPG import works faster because JPGs read faster, this sounds
> > like
> > you are I/O limited, which makes me believe in 1. more ;).  Oh yes,
> > are
> > all your images the same size and have intensities 0--255?  Then
> > you
> > can use the extended import file syntax
> >
> > https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/
> > main/java/ini/trakem2/persistence/Loader.java#L2013-L2027
> >
> > if you specify the columns 5--9 and switch off mipmaps, none of
> > your
> > images will be touched during import, i.e. import will be almost
> > instantaneous.  Save the project, do the rest.
> >
> > Let me know if that helps or where it breaks.
> >
> > Cheers,
> > Stephan
> >
> > On Wed, 2017-02-08 at 16:23 +0200, Stoyan Pavlov wrote:
> > >
> > > Hello Curtis,
> > > Thank you very much. I will wait a while for an answer. Last time
> > > they
> > > answered pretty quickly. If however there's no movement on this I
> > > will
> > > probably  accept your offer to file the bug report as proposed,
> > > because I
> > > am not familiar with github and coding.
> > >
> > > Best regards!
> > > Stoyan
> > >
> > > P.S. Sorry about the outbirst from  few minutes ago!
> > > На 8.02.2017 г. 1:40 сл.об. "Curtis Rueden" <[hidden email]>
> > > написа:
> > >
> > > Hi Stoyan, Herbie, everyone,
> > >
> > > Stoyan wrote:
> > > >
> > > >
> > > > First I want to emphasize, how astonishing and useful I find
> > > > the
> > > > trakem2 package/plugin! It is really great and I am really
> > > > thankful
> > > > to
> > > > the developers for puting all this effort into creating this
> > > > suite
> > > > for
> > > > working with large image datasets.
> > > Glad to hear it is useful for you.
> > >
> > > >
> > > >
> > > > Recently I stumbled upon several issues
> > > Thank you for the detailed report. Hopefully Albert or Stephan
> > > will
> > > be able
> > > to comment. At minimum, we should file these as issues in the
> > > GitHub
> > > repository, so they are not forgotten. If you do not hear back
> > > further from
> > > anyone, you can file them at https://github.com/trakem2/TrakEM2/i
> > > ssue
> > > s, or
> > > ping me and I will take care of it.
> > >
> > > Herbie wrote:
> > > >
> > > >
> > > > I have no idea about "trakem2"
> > > See this page for details:
> > >   http://imagej.net/TrakEM2
> > >
> > > >
> > > >
> > > > JEPG compression in 99% of all cases is a lossy method and more
> > > > or
> > > > less changes the image content, i.e. it introduces artifacts.
> > > Indeed—there is a discussion about that here:
> > >
> > > http://imagej.net/Principles#Why_.28lossy.29_JPEGs_should_
> > > not_be_used_in_imaging
> > >
> > > That said, there are cases where it is acceptable to use JPEG.
> > > Essentially,
> > > if you have massively large data, and you do not plan to analyze
> > > quantitatively for the most part, it might be OK for pragmatic
> > > reasons, as
> > > long as you understand what you are doing.
> > >
> > > >
> > > >
> > > > With respect to "trakem2"-issues I'd contact the authors
> > > > directly.
> > > I suggest keeping the discussion public. From the Help page (
> > > http://imagej.net/Help):
> > >
> > > >
> > > >
> > > > The ImageJ community firmly believes in public discussion of
> > > > the
> > > > software. In this way, each question benefits not only the one
> > > > asking,
> > > > but everyone in the community, including everyone who
> > > > subsequently
> > > > does a web search on the same topic.
> > > Regards,
> > > Curtis
> > >
> > > --
> > > Curtis Rueden
> > > LOCI software architect - https://loci.wisc.edu/software
> > > ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden
> > > Did you know ImageJ has a forum? http://forum.imagej.net/
> > >
> > >
> > > On Wed, Feb 8, 2017 at 4:21 AM, Herbie <[hidden email]> wrote:
> > >
> > > >
> > > >
> > > > Good day Stoyan,
> > > >
> > > > I have no idea about "trakem2" but please let me remark that
> > > > using
> > > > JEPG-compressed images for scientific purposes is an absolute
> > > > no-
> > > > no. JEPG
> > > > compression in 99% of all cases is a lossy method and more or
> > > > less
> > > > changes
> > > > the image content, i.e. it introduces artifacts.
> > > >
> > > > With respect to "trakem2"-issues I'd contact the authors
> > > > directly.
> > > >
> > > > Best
> > > >
> > > > Herbie
> > > >
> > > > :::::::::::::::::::::::::::::::::::::::::::
> > > > Am 08.02.17 um 10:40 schrieb Stoyan Pavlov:
> > > >
> > > > Dear friends and colleagues,
> > > > >
> > > > >
> > > > > First I want to emphasize, how astonishing and useful I find
> > > > > the
> > > > > trakem2
> > > > > package/plugin! It is really great and I am really thankful
> > > > > to
> > > > > the
> > > > > developers for puting all this effort into creating this
> > > > > suite
> > > > > for
> > > working
> > > >
> > > >
> > > > >
> > > > >
> > > > > with large image datasets.
> > > > > I use it mainly for creating/stitching/blending , processing
> > > > > and
> > > exporting
> > > >
> > > >
> > > > >
> > > > >
> > > > > of gigapixel montages of microscopic images into a readable
> > > > > software
> > > > > independent image pyramid.
> > > > > Recently I stumbled upon several issues (I don't know whether
> > > > > they are
> > > > > bugs
> > > > > or just due to limitation in the software) for which I
> > > > > couldn't
> > > > > find any
> > > > > mention and/or solution in the documentation and the forum.
> > > > >
> > > > > First I will describe what I am doing and next I will list
> > > > > the
> > > > > issues
> > > that
> > > >
> > > >
> > > > >
> > > > >
> > > > > bother me.
> > > > > I am creating a single layer montage with dimensions 63698 x
> > > > > 57218 px
> > > from
> > > >
> > > >
> > > > >
> > > > >
> > > > > a set of 3111 images (each 1388x1040). The images are
> > > > > acquired
> > > > > via
> > > > > axiovision and exported as single tiles (24bit RGB in either
> > > > > jpg
> > > > > or tif
> > > > > format). Before importing the images I align them using the
> > > > > MIST
> > > > > plugin
> > > > > and
> > > > > use the global positions generated from it to import them
> > > > > into
> > > > > trakem2
> > > via
> > > >
> > > >
> > > > >
> > > > >
> > > > > text file. The resulting montage is exactly as expected and
> > > > > have
> > > > > no
> > > > > remarks
> > > > > about it.
> > > > > 1) I don't know if its true but it seems to me that if I use
> > > > > jpg
> > > > > as
> > > format
> > > >
> > > >
> > > > >
> > > > >
> > > > > for the source images they are imported faster into trakem2.
> > > > > However once
> > > > > the import is finished the software returns errors about
> > > > > generating
> > > > > mipmaps
> > > > > (see attached log file) and it continuously tries to
> > > > > regenerate
> > > > > them
> > > > > (which
> > > > > I guess keeps CPU quite busy). This happens regardless
> > > > > mipmaps
> > > > > are
> > > enabled
> > > >
> > > >
> > > > >
> > > > >
> > > > > or disabled in Display>Properties, and is independent from
> > > > > the
> > > > > number of
> > > > > threads assigned to mipmap generation. If I close the project
> > > > > and
> > > > > Fiji
> > > > > (even after system restart) and then I open them again the
> > > > > issue
> > > > > returns.
> > > > > This behaviour is not oberved if my source images are in tif
> > > > > format.
> > > > > The other issues are related to the export of a flat image
> > > > > (here
> > > > > the file
> > > > > format of the source images does not matter):
> > > > >
> > > > > 2) When I am trying to export to a tiled format (Export> Make
> > > > > flat
> > > > > image...>Save for web (CATMAID)) regardless of the image
> > > > > format I
> > > > > choose
> > > > > for the export I cannot change the size of the exported tiles
> > > > > to
> > > > > other
> > > > > than
> > > > > 256x256. The dialogue allows me to change it, but when the
> > > > > process
> > > > > finishes
> > > > > in the folder there are only 256x256 tile images. While this
> > > > > is
> > > > > not a
> > > huge
> > > >
> > > >
> > > > >
> > > > >
> > > > > problem, still there is difference between managing some
> > > > > 80000
> > > > > images or
> > > > > 40000.
> > > > >
> > > > > 3) Also when I am trying to export tiled format  via Export>
> > > > > Make
> > > > > flat
> > > > > image...>Save for web (CATMAID), if the patches are
> > > > > deselected
> > > > > only a
> > > > > portion of the image is exported . Regardless of the format
> > > > > of
> > > > > the source
> > > > > images and of the exported images there are only 63 rows by
> > > > > 63
> > > > > columns of
> > > > > 256x256 tiles at zoom level 0 in the folder with the exported
> > > > > images. All
> > > > > metadata in the json file is correct and there are no errors
> > > > > reported by
> > > > > the System log or the Fiji Log. If the patches are selected
> > > > > with
> > > > > Select
> > > > > All
> > > > > (Ctrl + A) the image is exported in its entirety, however
> > > > > here
> > > > > comes the
> > > > > next not as much as an issue but slight discomfort.
> > > > >
> > > > > 4) The image is always exported to a quadrangular array
> > > > > consisting (for
> > > > > the
> > > > > reasons already described) of 256 rows and 256 columns of
> > > > > 256x256
> > > > > tiles
> > > at
> > > >
> > > >
> > > > >
> > > > >
> > > > > zoom level 0. That is the image dimensions (63698 x 57218 px)
> > > > > are
> > > expanded
> > > >
> > > >
> > > > >
> > > > >
> > > > > to 65536x65536 and filled with black tiles. This of course
> > > > > take
> > > > > up
> > > > > additional space and need to be cropped away before uploading
> > > > > the
> > > > > image
> > > to
> > > >
> > > >
> > > > >
> > > > >
> > > > > our servers. I understand that this is probably because of
> > > > > limitations to
> > > > > the algorithm but still if there is some workaround it would
> > > > > have
> > > > > been
> > > > > better.
> > > > >
> > > > > 5) I thought that this problems (2-4) might be due to the
> > > > > unequal
> > > > > dimensions of the original source images and I tried to work
> > > > > around them
> > > > > by
> > > > > creating a sibling project with retiled images. Here I had
> > > > > another
> > > > > unexpected result - the image was exported but the larger
> > > > > dimension was
> > > > > cut
> > > > > by about 6000 pxs from the right side  and than restored to
> > > > > the
> > > > > original
> > > > > size by adding black pixels , which basically cuts away the
> > > > > right
> > > > > portion
> > > > > of my image (see attached screenshot) .
> > > > >
> > > > > The workstation that I use has the following parameters:
> > > > >  CPU:  Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core
> > > > > processors)
> > > > > RAM: 64 GB
> > > > > OS: Windows 7 64bit
> > > > > In an attachment I added the ImageJ-properties output,
> > > > > portion of
> > > > > the
> > > > > trakem2 log with the errors generating the mipmaps, and a
> > > > > screenshot of
> > > > > the
> > > > > output of the sibling project with retiled images.
> > > > >
> > > > > I hope someone will find a way to correct these problems,
> > > > > because
> > > > > surely
> > > > > they are not fatal and there are workarounds that we use, but
> > > > > it
> > > > > is
> > > > > certainly annoying.
> > > > >
> > > > > Best regards,
> > > > > Stoyan
> > > > >
> > > > > ---
> > > > > Dr. Stoyan P. Pavlov, MD, PhD
> > > > > Departament of Anatomy, Histology and Embryology
> > > > > Medical University "Prof. Dr. Paraskev Stoyanov", Varna
> > > > > Prof. Marin Drinov Str.55
> > > > > 9002 Varna
> > > > >  Bulgaria
> > > > > Tel: +359 (0) 52 - 677 - 052 <052%20677%20052>
> > > > > e-mail: [hidden email]
> > > > >            [hidden email]
> > > > >
> > > > > --
> > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > >
> > > > >
> > > > --
> > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > >
> > > --
> > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > >
> > > --
> > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> >
> >
> >
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Stoyan Pavlov]

Thanks! I feel so embarrassed right now.

На 9.02.2017 г. 5:04 сл.об. "Saalfeld, Stephan" <[hidden email]>
написа:

> Yes,
>
> by changing this line
>
> https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/catmaid-
> export2.bsh#L96
>
> to
>
> ImagePlus.COLOR_RGB,
>
>
>
> (this is where the fork is made
>
> https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/main/java/i
> ni/trakem2/persistence/Loader.java#L2815-L2820
>
> )
>
> Best,
> Stephan
>
>
> On Thu, 2017-02-09 at 13:22 +0200, Stoyan Pavlov wrote:
> > Dear Stephan,
> > I tried the bsh script and it works very well and fast,  but it
> > exports the
> >  images in gray scale. Is there an easy way to make it export them in
> > RGB?
> > Best regards
> > Stoyan
> >
> > На 8.02.2017 г. 18:30 "Stoyan Pavlov" <[hidden email]>
> > написа:
> >
> > >
> > > Hi Stephan,
> > > Thank you very much! I will try the scripts, first thing in the
> > > morning. I
> > > figured as much. After all exactly this 2G java limitation for
> > > arrays is
> > > what brought me to trakem2. ;-)
> > >
> > >  >>  1. Is your project directory on a file share?
> > >
> > > No. Everything is saved locally.
> > >
> > >
> > > >
> > > > >
> > > > >  2. If you do not want to use mipmaps, switch mipmaps >>off
> > > > > *before*
> > > > > importing tiles.  Switching them off during the madness >>has
> > > > > no effects
> > > Indeed I observed this behaviour: even if one cancels the mipmap
> > > generation job, it continues in Fiji until finished. Only
> > > workaround is to
> > > quit Fiji . But when the project is loaded it tries to regenerate
> > > the
> > > mipmaps...
> > >
> > > I guess the java loader alternative is used the sameway any other
> > > script?
> > > (New>Script etc...)
> > >
> > > I'll let you know whether it all worked.
> > >
> > > Best regards,
> > > Stoyan
> > >
> > > На 8.02.2017 г. 5:22 сл.об. "Saalfeld, Stephan" <
> > > [hidden email]> написа:
> > >
> > > Hi,
> > >
> > > TrakEM2's internal CATMAID export mechanisms goes through single
> > > image
> > > exports.  This is limited to 2GPx planes (sad---right?).  Please
> > > use
> > > this script instead:
> > >
> > > https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/catm
> > > aid-
> > > export2.bsh
> > >
> > > Regarding the mipmap-madness...:
> > >
> > > 1. Is your project directory on a file share?  If yes, please move
> > > it
> > > to a local (external?) harddrive, you are working with 2D images,
> > > that
> > > should fit.  TrakEM2 is somehow sensitive to how network drives
> > > frequently fail or rather delay what they are supposed to do.  This
> > > is
> > > an annoyance in TrakEM2 and could/ should be fixed.  No time
> > > though...
> > >
> > > 2. If you do not want to use mipmaps, switch mipmaps off *before*
> > > importing tiles.  Switching them off during the madness has no
> > > effects,
> > > too many threads running already and they will not get properly
> > > interrupted.
> > >
> > > 3. JPG import works faster because JPGs read faster, this sounds
> > > like
> > > you are I/O limited, which makes me believe in 1. more ;).  Oh yes,
> > > are
> > > all your images the same size and have intensities 0--255?  Then
> > > you
> > > can use the extended import file syntax
> > >
> > > https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/
> > > main/java/ini/trakem2/persistence/Loader.java#L2013-L2027
> > >
> > > if you specify the columns 5--9 and switch off mipmaps, none of
> > > your
> > > images will be touched during import, i.e. import will be almost
> > > instantaneous.  Save the project, do the rest.
> > >
> > > Let me know if that helps or where it breaks.
> > >
> > > Cheers,
> > > Stephan
> > >
> > > On Wed, 2017-02-08 at 16:23 +0200, Stoyan Pavlov wrote:
> > > >
> > > > Hello Curtis,
> > > > Thank you very much. I will wait a while for an answer. Last time
> > > > they
> > > > answered pretty quickly. If however there's no movement on this I
> > > > will
> > > > probably  accept your offer to file the bug report as proposed,
> > > > because I
> > > > am not familiar with github and coding.
> > > >
> > > > Best regards!
> > > > Stoyan
> > > >
> > > > P.S. Sorry about the outbirst from  few minutes ago!
> > > > На 8.02.2017 г. 1:40 сл.об. "Curtis Rueden" <[hidden email]>
> > > > написа:
> > > >
> > > > Hi Stoyan, Herbie, everyone,
> > > >
> > > > Stoyan wrote:
> > > > >
> > > > >
> > > > > First I want to emphasize, how astonishing and useful I find
> > > > > the
> > > > > trakem2 package/plugin! It is really great and I am really
> > > > > thankful
> > > > > to
> > > > > the developers for puting all this effort into creating this
> > > > > suite
> > > > > for
> > > > > working with large image datasets.
> > > > Glad to hear it is useful for you.
> > > >
> > > > >
> > > > >
> > > > > Recently I stumbled upon several issues
> > > > Thank you for the detailed report. Hopefully Albert or Stephan
> > > > will
> > > > be able
> > > > to comment. At minimum, we should file these as issues in the
> > > > GitHub
> > > > repository, so they are not forgotten. If you do not hear back
> > > > further from
> > > > anyone, you can file them at https://github.com/trakem2/TrakEM2/i
> > > > ssue
> > > > s, or
> > > > ping me and I will take care of it.
> > > >
> > > > Herbie wrote:
> > > > >
> > > > >
> > > > > I have no idea about "trakem2"
> > > > See this page for details:
> > > >   http://imagej.net/TrakEM2
> > > >
> > > > >
> > > > >
> > > > > JEPG compression in 99% of all cases is a lossy method and more
> > > > > or
> > > > > less changes the image content, i.e. it introduces artifacts.
> > > > Indeed—there is a discussion about that here:
> > > >
> > > > http://imagej.net/Principles#Why_.28lossy.29_JPEGs_should_
> > > > not_be_used_in_imaging
> > > >
> > > > That said, there are cases where it is acceptable to use JPEG.
> > > > Essentially,
> > > > if you have massively large data, and you do not plan to analyze
> > > > quantitatively for the most part, it might be OK for pragmatic
> > > > reasons, as
> > > > long as you understand what you are doing.
> > > >
> > > > >
> > > > >
> > > > > With respect to "trakem2"-issues I'd contact the authors
> > > > > directly.
> > > > I suggest keeping the discussion public. From the Help page (
> > > > http://imagej.net/Help):
> > > >
> > > > >
> > > > >
> > > > > The ImageJ community firmly believes in public discussion of
> > > > > the
> > > > > software. In this way, each question benefits not only the one
> > > > > asking,
> > > > > but everyone in the community, including everyone who
> > > > > subsequently
> > > > > does a web search on the same topic.
> > > > Regards,
> > > > Curtis
> > > >
> > > > --
> > > > Curtis Rueden
> > > > LOCI software architect - https://loci.wisc.edu/software
> > > > ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden
> > > > Did you know ImageJ has a forum? http://forum.imagej.net/
> > > >
> > > >
> > > > On Wed, Feb 8, 2017 at 4:21 AM, Herbie <[hidden email]> wrote:
> > > >
> > > > >
> > > > >
> > > > > Good day Stoyan,
> > > > >
> > > > > I have no idea about "trakem2" but please let me remark that
> > > > > using
> > > > > JEPG-compressed images for scientific purposes is an absolute
> > > > > no-
> > > > > no. JEPG
> > > > > compression in 99% of all cases is a lossy method and more or
> > > > > less
> > > > > changes
> > > > > the image content, i.e. it introduces artifacts.
> > > > >
> > > > > With respect to "trakem2"-issues I'd contact the authors
> > > > > directly.
> > > > >
> > > > > Best
> > > > >
> > > > > Herbie
> > > > >
> > > > > :::::::::::::::::::::::::::::::::::::::::::
> > > > > Am 08.02.17 um 10:40 schrieb Stoyan Pavlov:
> > > > >
> > > > > Dear friends and colleagues,
> > > > > >
> > > > > >
> > > > > > First I want to emphasize, how astonishing and useful I find
> > > > > > the
> > > > > > trakem2
> > > > > > package/plugin! It is really great and I am really thankful
> > > > > > to
> > > > > > the
> > > > > > developers for puting all this effort into creating this
> > > > > > suite
> > > > > > for
> > > > working
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > with large image datasets.
> > > > > > I use it mainly for creating/stitching/blending , processing
> > > > > > and
> > > > exporting
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > of gigapixel montages of microscopic images into a readable
> > > > > > software
> > > > > > independent image pyramid.
> > > > > > Recently I stumbled upon several issues (I don't know whether
> > > > > > they are
> > > > > > bugs
> > > > > > or just due to limitation in the software) for which I
> > > > > > couldn't
> > > > > > find any
> > > > > > mention and/or solution in the documentation and the forum.
> > > > > >
> > > > > > First I will describe what I am doing and next I will list
> > > > > > the
> > > > > > issues
> > > > that
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > bother me.
> > > > > > I am creating a single layer montage with dimensions 63698 x
> > > > > > 57218 px
> > > > from
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > a set of 3111 images (each 1388x1040). The images are
> > > > > > acquired
> > > > > > via
> > > > > > axiovision and exported as single tiles (24bit RGB in either
> > > > > > jpg
> > > > > > or tif
> > > > > > format). Before importing the images I align them using the
> > > > > > MIST
> > > > > > plugin
> > > > > > and
> > > > > > use the global positions generated from it to import them
> > > > > > into
> > > > > > trakem2
> > > > via
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > text file. The resulting montage is exactly as expected and
> > > > > > have
> > > > > > no
> > > > > > remarks
> > > > > > about it.
> > > > > > 1) I don't know if its true but it seems to me that if I use
> > > > > > jpg
> > > > > > as
> > > > format
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > for the source images they are imported faster into trakem2.
> > > > > > However once
> > > > > > the import is finished the software returns errors about
> > > > > > generating
> > > > > > mipmaps
> > > > > > (see attached log file) and it continuously tries to
> > > > > > regenerate
> > > > > > them
> > > > > > (which
> > > > > > I guess keeps CPU quite busy). This happens regardless
> > > > > > mipmaps
> > > > > > are
> > > > enabled
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > or disabled in Display>Properties, and is independent from
> > > > > > the
> > > > > > number of
> > > > > > threads assigned to mipmap generation. If I close the project
> > > > > > and
> > > > > > Fiji
> > > > > > (even after system restart) and then I open them again the
> > > > > > issue
> > > > > > returns.
> > > > > > This behaviour is not oberved if my source images are in tif
> > > > > > format.
> > > > > > The other issues are related to the export of a flat image
> > > > > > (here
> > > > > > the file
> > > > > > format of the source images does not matter):
> > > > > >
> > > > > > 2) When I am trying to export to a tiled format (Export> Make
> > > > > > flat
> > > > > > image...>Save for web (CATMAID)) regardless of the image
> > > > > > format I
> > > > > > choose
> > > > > > for the export I cannot change the size of the exported tiles
> > > > > > to
> > > > > > other
> > > > > > than
> > > > > > 256x256. The dialogue allows me to change it, but when the
> > > > > > process
> > > > > > finishes
> > > > > > in the folder there are only 256x256 tile images. While this
> > > > > > is
> > > > > > not a
> > > > huge
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > problem, still there is difference between managing some
> > > > > > 80000
> > > > > > images or
> > > > > > 40000.
> > > > > >
> > > > > > 3) Also when I am trying to export tiled format  via Export>
> > > > > > Make
> > > > > > flat
> > > > > > image...>Save for web (CATMAID), if the patches are
> > > > > > deselected
> > > > > > only a
> > > > > > portion of the image is exported . Regardless of the format
> > > > > > of
> > > > > > the source
> > > > > > images and of the exported images there are only 63 rows by
> > > > > > 63
> > > > > > columns of
> > > > > > 256x256 tiles at zoom level 0 in the folder with the exported
> > > > > > images. All
> > > > > > metadata in the json file is correct and there are no errors
> > > > > > reported by
> > > > > > the System log or the Fiji Log. If the patches are selected
> > > > > > with
> > > > > > Select
> > > > > > All
> > > > > > (Ctrl + A) the image is exported in its entirety, however
> > > > > > here
> > > > > > comes the
> > > > > > next not as much as an issue but slight discomfort.
> > > > > >
> > > > > > 4) The image is always exported to a quadrangular array
> > > > > > consisting (for
> > > > > > the
> > > > > > reasons already described) of 256 rows and 256 columns of
> > > > > > 256x256
> > > > > > tiles
> > > > at
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > zoom level 0. That is the image dimensions (63698 x 57218 px)
> > > > > > are
> > > > expanded
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > to 65536x65536 and filled with black tiles. This of course
> > > > > > take
> > > > > > up
> > > > > > additional space and need to be cropped away before uploading
> > > > > > the
> > > > > > image
> > > > to
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > > our servers. I understand that this is probably because of
> > > > > > limitations to
> > > > > > the algorithm but still if there is some workaround it would
> > > > > > have
> > > > > > been
> > > > > > better.
> > > > > >
> > > > > > 5) I thought that this problems (2-4) might be due to the
> > > > > > unequal
> > > > > > dimensions of the original source images and I tried to work
> > > > > > around them
> > > > > > by
> > > > > > creating a sibling project with retiled images. Here I had
> > > > > > another
> > > > > > unexpected result - the image was exported but the larger
> > > > > > dimension was
> > > > > > cut
> > > > > > by about 6000 pxs from the right side  and than restored to
> > > > > > the
> > > > > > original
> > > > > > size by adding black pixels , which basically cuts away the
> > > > > > right
> > > > > > portion
> > > > > > of my image (see attached screenshot) .
> > > > > >
> > > > > > The workstation that I use has the following parameters:
> > > > > >  CPU:  Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core
> > > > > > processors)
> > > > > > RAM: 64 GB
> > > > > > OS: Windows 7 64bit
> > > > > > In an attachment I added the ImageJ-properties output,
> > > > > > portion of
> > > > > > the
> > > > > > trakem2 log with the errors generating the mipmaps, and a
> > > > > > screenshot of
> > > > > > the
> > > > > > output of the sibling project with retiled images.
> > > > > >
> > > > > > I hope someone will find a way to correct these problems,
> > > > > > because
> > > > > > surely
> > > > > > they are not fatal and there are workarounds that we use, but
> > > > > > it
> > > > > > is
> > > > > > certainly annoying.
> > > > > >
> > > > > > Best regards,
> > > > > > Stoyan
> > > > > >
> > > > > > ---
> > > > > > Dr. Stoyan P. Pavlov, MD, PhD
> > > > > > Departament of Anatomy, Histology and Embryology
> > > > > > Medical University "Prof. Dr. Paraskev Stoyanov", Varna
> > > > > > Prof. Marin Drinov Str.55
> > > > > > 9002 Varna
> > > > > >  Bulgaria
> > > > > > Tel: +359 (0) 52 - 677 - 052 <052%20677%20052>
> > > > > > e-mail: [hidden email]
> > > > > >            [hidden email]
> > > > > >
> > > > > > --
> > > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > > >
> > > > > >
> > > > > --
> > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > >
> > > > --
> > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > >
> > > > --
> > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > --
> > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > >
> > >
> > >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Saalfeld, Stephan]

No need to feel embarrassed.  The script is pretty messy and not too
well documented.

Best,
Stephan


On Thu, 2017-02-09 at 17:46 +0200, Stoyan Pavlov wrote:

> Thanks! I feel so embarrassed right now.
>
> На 9.02.2017 г. 5:04 сл.об. "Saalfeld, Stephan" <[hidden email]
> mi.org>
> написа:
>
> >
> > Yes,
> >
> > by changing this line
> >
> > https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/catm
> > aid-
> > export2.bsh#L96
> >
> > to
> >
> > ImagePlus.COLOR_RGB,
> >
> >
> >
> > (this is where the fork is made
> >
> > https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/main/ja
> > va/i
> > ni/trakem2/persistence/Loader.java#L2815-L2820
> >
> > )
> >
> > Best,
> > Stephan
> >
> >
> > On Thu, 2017-02-09 at 13:22 +0200, Stoyan Pavlov wrote:
> > >
> > > Dear Stephan,
> > > I tried the bsh script and it works very well and fast,  but it
> > > exports the
> > >  images in gray scale. Is there an easy way to make it export
> > > them in
> > > RGB?
> > > Best regards
> > > Stoyan
> > >
> > > На 8.02.2017 г. 18:30 "Stoyan Pavlov" <[hidden email]>
> > > написа:
> > >
> > > >
> > > >
> > > > Hi Stephan,
> > > > Thank you very much! I will try the scripts, first thing in the
> > > > morning. I
> > > > figured as much. After all exactly this 2G java limitation for
> > > > arrays is
> > > > what brought me to trakem2. ;-)
> > > >
> > > >  >>  1. Is your project directory on a file share?
> > > >
> > > > No. Everything is saved locally.
> > > >
> > > >
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > >  2. If you do not want to use mipmaps, switch mipmaps >>off
> > > > > > *before*
> > > > > > importing tiles.  Switching them off during the madness
> > > > > > >>has
> > > > > > no effects
> > > > Indeed I observed this behaviour: even if one cancels the
> > > > mipmap
> > > > generation job, it continues in Fiji until finished. Only
> > > > workaround is to
> > > > quit Fiji . But when the project is loaded it tries to
> > > > regenerate
> > > > the
> > > > mipmaps...
> > > >
> > > > I guess the java loader alternative is used the sameway any
> > > > other
> > > > script?
> > > > (New>Script etc...)
> > > >
> > > > I'll let you know whether it all worked.
> > > >
> > > > Best regards,
> > > > Stoyan
> > > >
> > > > На 8.02.2017 г. 5:22 сл.об. "Saalfeld, Stephan" <
> > > > [hidden email]> написа:
> > > >
> > > > Hi,
> > > >
> > > > TrakEM2's internal CATMAID export mechanisms goes through
> > > > single
> > > > image
> > > > exports.  This is limited to 2GPx planes (sad---
> > > > right?).  Please
> > > > use
> > > > this script instead:
> > > >
> > > > https://github.com/axtimwalde/fiji-scripts/blob/master/TrakEM2/
> > > > catm
> > > > aid-
> > > > export2.bsh
> > > >
> > > > Regarding the mipmap-madness...:
> > > >
> > > > 1. Is your project directory on a file share?  If yes, please
> > > > move
> > > > it
> > > > to a local (external?) harddrive, you are working with 2D
> > > > images,
> > > > that
> > > > should fit.  TrakEM2 is somehow sensitive to how network drives
> > > > frequently fail or rather delay what they are supposed to
> > > > do.  This
> > > > is
> > > > an annoyance in TrakEM2 and could/ should be fixed.  No time
> > > > though...
> > > >
> > > > 2. If you do not want to use mipmaps, switch mipmaps off
> > > > *before*
> > > > importing tiles.  Switching them off during the madness has no
> > > > effects,
> > > > too many threads running already and they will not get properly
> > > > interrupted.
> > > >
> > > > 3. JPG import works faster because JPGs read faster, this
> > > > sounds
> > > > like
> > > > you are I/O limited, which makes me believe in 1. more ;).  Oh
> > > > yes,
> > > > are
> > > > all your images the same size and have intensities 0
> > > > --255?  Then
> > > > you
> > > > can use the extended import file syntax
> > > >
> > > > https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/
> > > > main/java/ini/trakem2/persistence/Loader.java#L2013-L2027
> > > >
> > > > if you specify the columns 5--9 and switch off mipmaps, none of
> > > > your
> > > > images will be touched during import, i.e. import will be
> > > > almost
> > > > instantaneous.  Save the project, do the rest.
> > > >
> > > > Let me know if that helps or where it breaks.
> > > >
> > > > Cheers,
> > > > Stephan
> > > >
> > > > On Wed, 2017-02-08 at 16:23 +0200, Stoyan Pavlov wrote:
> > > > >
> > > > >
> > > > > Hello Curtis,
> > > > > Thank you very much. I will wait a while for an answer. Last
> > > > > time
> > > > > they
> > > > > answered pretty quickly. If however there's no movement on
> > > > > this I
> > > > > will
> > > > > probably  accept your offer to file the bug report as
> > > > > proposed,
> > > > > because I
> > > > > am not familiar with github and coding.
> > > > >
> > > > > Best regards!
> > > > > Stoyan
> > > > >
> > > > > P.S. Sorry about the outbirst from  few minutes ago!
> > > > > На 8.02.2017 г. 1:40 сл.об. "Curtis Rueden" <[hidden email]
> > > > > u>
> > > > > написа:
> > > > >
> > > > > Hi Stoyan, Herbie, everyone,
> > > > >
> > > > > Stoyan wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > First I want to emphasize, how astonishing and useful I
> > > > > > find
> > > > > > the
> > > > > > trakem2 package/plugin! It is really great and I am really
> > > > > > thankful
> > > > > > to
> > > > > > the developers for puting all this effort into creating
> > > > > > this
> > > > > > suite
> > > > > > for
> > > > > > working with large image datasets.
> > > > > Glad to hear it is useful for you.
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Recently I stumbled upon several issues
> > > > > Thank you for the detailed report. Hopefully Albert or
> > > > > Stephan
> > > > > will
> > > > > be able
> > > > > to comment. At minimum, we should file these as issues in the
> > > > > GitHub
> > > > > repository, so they are not forgotten. If you do not hear
> > > > > back
> > > > > further from
> > > > > anyone, you can file them at https://github.com/trakem2/TrakE
> > > > > M2/i
> > > > > ssue
> > > > > s, or
> > > > > ping me and I will take care of it.
> > > > >
> > > > > Herbie wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > I have no idea about "trakem2"
> > > > > See this page for details:
> > > > >   http://imagej.net/TrakEM2
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > JEPG compression in 99% of all cases is a lossy method and
> > > > > > more
> > > > > > or
> > > > > > less changes the image content, i.e. it introduces
> > > > > > artifacts.
> > > > > Indeed—there is a discussion about that here:
> > > > >
> > > > > http://imagej.net/Principles#Why_.28lossy.29_JPEGs_should_
> > > > > not_be_used_in_imaging
> > > > >
> > > > > That said, there are cases where it is acceptable to use
> > > > > JPEG.
> > > > > Essentially,
> > > > > if you have massively large data, and you do not plan to
> > > > > analyze
> > > > > quantitatively for the most part, it might be OK for
> > > > > pragmatic
> > > > > reasons, as
> > > > > long as you understand what you are doing.
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > With respect to "trakem2"-issues I'd contact the authors
> > > > > > directly.
> > > > > I suggest keeping the discussion public. From the Help page (
> > > > > http://imagej.net/Help):
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > The ImageJ community firmly believes in public discussion
> > > > > > of
> > > > > > the
> > > > > > software. In this way, each question benefits not only the
> > > > > > one
> > > > > > asking,
> > > > > > but everyone in the community, including everyone who
> > > > > > subsequently
> > > > > > does a web search on the same topic.
> > > > > Regards,
> > > > > Curtis
> > > > >
> > > > > --
> > > > > Curtis Rueden
> > > > > LOCI software architect - https://loci.wisc.edu/software
> > > > > ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Ruede
> > > > > n
> > > > > Did you know ImageJ has a forum? http://forum.imagej.net/
> > > > >
> > > > >
> > > > > On Wed, Feb 8, 2017 at 4:21 AM, Herbie <[hidden email]>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Good day Stoyan,
> > > > > >
> > > > > > I have no idea about "trakem2" but please let me remark
> > > > > > that
> > > > > > using
> > > > > > JEPG-compressed images for scientific purposes is an
> > > > > > absolute
> > > > > > no-
> > > > > > no. JEPG
> > > > > > compression in 99% of all cases is a lossy method and more
> > > > > > or
> > > > > > less
> > > > > > changes
> > > > > > the image content, i.e. it introduces artifacts.
> > > > > >
> > > > > > With respect to "trakem2"-issues I'd contact the authors
> > > > > > directly.
> > > > > >
> > > > > > Best
> > > > > >
> > > > > > Herbie
> > > > > >
> > > > > > :::::::::::::::::::::::::::::::::::::::::::
> > > > > > Am 08.02.17 um 10:40 schrieb Stoyan Pavlov:
> > > > > >
> > > > > > Dear friends and colleagues,
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > First I want to emphasize, how astonishing and useful I
> > > > > > > find
> > > > > > > the
> > > > > > > trakem2
> > > > > > > package/plugin! It is really great and I am really
> > > > > > > thankful
> > > > > > > to
> > > > > > > the
> > > > > > > developers for puting all this effort into creating this
> > > > > > > suite
> > > > > > > for
> > > > > working
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > with large image datasets.
> > > > > > > I use it mainly for creating/stitching/blending ,
> > > > > > > processing
> > > > > > > and
> > > > > exporting
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > of gigapixel montages of microscopic images into a
> > > > > > > readable
> > > > > > > software
> > > > > > > independent image pyramid.
> > > > > > > Recently I stumbled upon several issues (I don't know
> > > > > > > whether
> > > > > > > they are
> > > > > > > bugs
> > > > > > > or just due to limitation in the software) for which I
> > > > > > > couldn't
> > > > > > > find any
> > > > > > > mention and/or solution in the documentation and the
> > > > > > > forum.
> > > > > > >
> > > > > > > First I will describe what I am doing and next I will
> > > > > > > list
> > > > > > > the
> > > > > > > issues
> > > > > that
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > bother me.
> > > > > > > I am creating a single layer montage with dimensions
> > > > > > > 63698 x
> > > > > > > 57218 px
> > > > > from
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > a set of 3111 images (each 1388x1040). The images are
> > > > > > > acquired
> > > > > > > via
> > > > > > > axiovision and exported as single tiles (24bit RGB in
> > > > > > > either
> > > > > > > jpg
> > > > > > > or tif
> > > > > > > format). Before importing the images I align them using
> > > > > > > the
> > > > > > > MIST
> > > > > > > plugin
> > > > > > > and
> > > > > > > use the global positions generated from it to import them
> > > > > > > into
> > > > > > > trakem2
> > > > > via
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > text file. The resulting montage is exactly as expected
> > > > > > > and
> > > > > > > have
> > > > > > > no
> > > > > > > remarks
> > > > > > > about it.
> > > > > > > 1) I don't know if its true but it seems to me that if I
> > > > > > > use
> > > > > > > jpg
> > > > > > > as
> > > > > format
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > for the source images they are imported faster into
> > > > > > > trakem2.
> > > > > > > However once
> > > > > > > the import is finished the software returns errors about
> > > > > > > generating
> > > > > > > mipmaps
> > > > > > > (see attached log file) and it continuously tries to
> > > > > > > regenerate
> > > > > > > them
> > > > > > > (which
> > > > > > > I guess keeps CPU quite busy). This happens regardless
> > > > > > > mipmaps
> > > > > > > are
> > > > > enabled
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > or disabled in Display>Properties, and is independent
> > > > > > > from
> > > > > > > the
> > > > > > > number of
> > > > > > > threads assigned to mipmap generation. If I close the
> > > > > > > project
> > > > > > > and
> > > > > > > Fiji
> > > > > > > (even after system restart) and then I open them again
> > > > > > > the
> > > > > > > issue
> > > > > > > returns.
> > > > > > > This behaviour is not oberved if my source images are in
> > > > > > > tif
> > > > > > > format.
> > > > > > > The other issues are related to the export of a flat
> > > > > > > image
> > > > > > > (here
> > > > > > > the file
> > > > > > > format of the source images does not matter):
> > > > > > >
> > > > > > > 2) When I am trying to export to a tiled format (Export>
> > > > > > > Make
> > > > > > > flat
> > > > > > > image...>Save for web (CATMAID)) regardless of the image
> > > > > > > format I
> > > > > > > choose
> > > > > > > for the export I cannot change the size of the exported
> > > > > > > tiles
> > > > > > > to
> > > > > > > other
> > > > > > > than
> > > > > > > 256x256. The dialogue allows me to change it, but when
> > > > > > > the
> > > > > > > process
> > > > > > > finishes
> > > > > > > in the folder there are only 256x256 tile images. While
> > > > > > > this
> > > > > > > is
> > > > > > > not a
> > > > > huge
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > problem, still there is difference between managing some
> > > > > > > 80000
> > > > > > > images or
> > > > > > > 40000.
> > > > > > >
> > > > > > > 3) Also when I am trying to export tiled format  via
> > > > > > > Export>
> > > > > > > Make
> > > > > > > flat
> > > > > > > image...>Save for web (CATMAID), if the patches are
> > > > > > > deselected
> > > > > > > only a
> > > > > > > portion of the image is exported . Regardless of the
> > > > > > > format
> > > > > > > of
> > > > > > > the source
> > > > > > > images and of the exported images there are only 63 rows
> > > > > > > by
> > > > > > > 63
> > > > > > > columns of
> > > > > > > 256x256 tiles at zoom level 0 in the folder with the
> > > > > > > exported
> > > > > > > images. All
> > > > > > > metadata in the json file is correct and there are no
> > > > > > > errors
> > > > > > > reported by
> > > > > > > the System log or the Fiji Log. If the patches are
> > > > > > > selected
> > > > > > > with
> > > > > > > Select
> > > > > > > All
> > > > > > > (Ctrl + A) the image is exported in its entirety, however
> > > > > > > here
> > > > > > > comes the
> > > > > > > next not as much as an issue but slight discomfort.
> > > > > > >
> > > > > > > 4) The image is always exported to a quadrangular array
> > > > > > > consisting (for
> > > > > > > the
> > > > > > > reasons already described) of 256 rows and 256 columns of
> > > > > > > 256x256
> > > > > > > tiles
> > > > > at
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > zoom level 0. That is the image dimensions (63698 x 57218
> > > > > > > px)
> > > > > > > are
> > > > > expanded
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > to 65536x65536 and filled with black tiles. This of
> > > > > > > course
> > > > > > > take
> > > > > > > up
> > > > > > > additional space and need to be cropped away before
> > > > > > > uploading
> > > > > > > the
> > > > > > > image
> > > > > to
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > our servers. I understand that this is probably because
> > > > > > > of
> > > > > > > limitations to
> > > > > > > the algorithm but still if there is some workaround it
> > > > > > > would
> > > > > > > have
> > > > > > > been
> > > > > > > better.
> > > > > > >
> > > > > > > 5) I thought that this problems (2-4) might be due to the
> > > > > > > unequal
> > > > > > > dimensions of the original source images and I tried to
> > > > > > > work
> > > > > > > around them
> > > > > > > by
> > > > > > > creating a sibling project with retiled images. Here I
> > > > > > > had
> > > > > > > another
> > > > > > > unexpected result - the image was exported but the larger
> > > > > > > dimension was
> > > > > > > cut
> > > > > > > by about 6000 pxs from the right side  and than restored
> > > > > > > to
> > > > > > > the
> > > > > > > original
> > > > > > > size by adding black pixels , which basically cuts away
> > > > > > > the
> > > > > > > right
> > > > > > > portion
> > > > > > > of my image (see attached screenshot) .
> > > > > > >
> > > > > > > The workstation that I use has the following parameters:
> > > > > > >  CPU:  Intel Xeon CPU E5-2643 @3.3 GHz (2 x 8 core
> > > > > > > processors)
> > > > > > > RAM: 64 GB
> > > > > > > OS: Windows 7 64bit
> > > > > > > In an attachment I added the ImageJ-properties output,
> > > > > > > portion of
> > > > > > > the
> > > > > > > trakem2 log with the errors generating the mipmaps, and a
> > > > > > > screenshot of
> > > > > > > the
> > > > > > > output of the sibling project with retiled images.
> > > > > > >
> > > > > > > I hope someone will find a way to correct these problems,
> > > > > > > because
> > > > > > > surely
> > > > > > > they are not fatal and there are workarounds that we use,
> > > > > > > but
> > > > > > > it
> > > > > > > is
> > > > > > > certainly annoying.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Stoyan
> > > > > > >
> > > > > > > ---
> > > > > > > Dr. Stoyan P. Pavlov, MD, PhD
> > > > > > > Departament of Anatomy, Histology and Embryology
> > > > > > > Medical University "Prof. Dr. Paraskev Stoyanov", Varna
> > > > > > > Prof. Marin Drinov Str.55
> > > > > > > 9002 Varna
> > > > > > >  Bulgaria
> > > > > > > Tel: +359 (0) 52 - 677 - 052 <052%20677%20052>
> > > > > > > e-mail: [hidden email]
> > > > > > >            [hidden email]
> > > > > > >
> > > > > > > --
> > > > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > > > >
> > > > > > >
> > > > > > --
> > > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > > >
> > > > > --
> > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > >
> > > > > --
> > > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > > --
> > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > > >
> > > >
> > > >
> > > --
> > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Albert Cardona-2]

The extended csv format works for all 4 ImageJ types, 8-bit, 16-bit, 32-bit and COLOR_RGB, with ranges as adecuate to the bit ranges.

Albert

> On Feb 8, 2017, at 10:03 AM, Saalfeld, Stephan <[hidden email]> wrote:
>
> Oh yes, are
> all your images the same size and have intensities 0--255?  Then you
> can use the extended import file syntax
>
> https://github.com/trakem2/TrakEM2/blob/master/TrakEM2_/src/main/java/ini/trakem2/persistence/Loader.java#L2013-L2027
>
> if you specify the columns 5--9 and switch off mipmaps, none of your
> images will be touched during import, i.e. import will be almost
> instantaneous.  Save the project, do the rest.

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html