Headless ImageJ2-rc2 - Analyze Particles returns no result

[Paul van Schayck]

Dear all,

I'm running into several problems running ImageJ2-rc2 headless. I'm using this command:

./ImageJ-linux64 --headless -macro scripts/test.ijm

The first one is that the new "Open images using scifio" breaks image opening through open("image.png"):
java.lang.NullPointerException
        at net.imagej.legacy.IJ1Helper.updateRecentMenu(IJ1Helper.java:307)
        at net.imagej.legacy.plugin.DefaultLegacyOpener.open(DefaultLegacyOpener.java:144)
        at net.imagej.legacy.DefaultLegacyHooks.interceptFileOpen(DefaultLegacyHooks.java:427)
        at ij.IJ.open(IJ.java)

This can be fixed by disabling the new SCIFIO option.

The second one, is that this macro:

open(imagePath);
run("8-bit");
setThreshold(1, 255);
run("Analyze Particles...", "size=300-Infinity circularity=0.40-1.00");
print(nResults);

Will return 0. No particles seem to be found. Running the same macro not headless:
./ImageJ-linux64 -macro scripts/test.ijm
Will return a number of particles in a log window

Thirdly, the option "-macro" seems to be undocumented. Is this the preferred way to start macros? How about new scripts (.js or beanshell).

That were quite a few questions. Hope you can answer some of them.

Thank you,

Paul

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[dscho]

Hi Paul,

On Wed, 11 Jun 2014, Paul van Schayck wrote:

> I'm running into several problems running ImageJ2-rc2 headless. I'm
> using this command:
>
> ./ImageJ-linux64 --headless -macro scripts/test.ijm
>
> The first one is that the new "Open images using scifio" breaks image opening through open("image.png"):
> java.lang.NullPointerException
>         at net.imagej.legacy.IJ1Helper.updateRecentMenu(IJ1Helper.java:307)

Could I ask you to test my fix? If you replaced your
jars/imagej-legacy-*.jar with the version from

        http://jenkins.imagej.net/job/ImageJ-legacy/lastSuccessfulBuild/artifact/target/imagej-legacy-0.5.10-SNAPSHOT.jar

it should address that problem.

> This can be fixed by disabling the new SCIFIO option.

Note that we are currently stabilizing this code, with the enormous and
outstanding help by the community, such as you!

The idea of using SCIFIO is to make it possible to just File>Open just
about anything and get the correct images. The built-in TIFF reader of
ImageJ 1.x cuts a few corners, which is fast, but fails to open a couple
of valid TIFF files correctly, and it also does not heed extensions such
as OME-TIFF. SCIFIO addresses that.

I have no doubt that with the help of the community, we can make the
support code in ImageJ2 robust so that everybody can gain the benefits of
the SCIFIO library.

> The second one, is that this macro:
>
> open(imagePath);
> run("8-bit");
> setThreshold(1, 255);
> run("Analyze Particles...", "size=300-Infinity circularity=0.40-1.00");
> print(nResults);
>
> Will return 0. No particles seem to be found. Running the same macro not headless:
> ./ImageJ-linux64 -macro scripts/test.ijm
> Will return a number of particles in a log window

Have you tried with

        ./ImageJ-linux64 -batch scripts/test.ijm

? This is a way to call macros in batch mode that is admittedly not
documented very well... At least that will get you going again if all you
need is to call the script from the command line and let it obtain the
results via stdout.

I just tried to reproduce that headless mode problem and get a different
problem instead. Will keep you posted.

> Thirdly, the option "-macro" seems to be undocumented. Is this the
> preferred way to start macros? How about new scripts (.js or beanshell).

The -macro option is a standard IamgeJ 1.x option, documented
unfortunately only in the developer part of the documentation, far from
regular users' hunting grounds:

        http://jenkins.imagej.net/job/ImageJ1-javadoc/javadoc/ij/ImageJ.html

Note that it is also not *quite* correct: you can pass -batch as last
argument *without* a path, which comes in handy if you want to *evaluate*
an expression -- as opposed to a file containing macro code -- in batch
mode.

So hopefully this addresses two of your questions; I will keep you updated
on the remaining one.

Ciao,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[ctrueden]

Hi Paul,

> Thirdly, the option "-macro" seems to be undocumented. Is this the
> preferred way to start macros? How about new scripts (.js or
> beanshell).

In addition to the links Johannes sent out, see also:
http://wiki.imagej.net/Headless#Running_macros_in_headless_mode
http://wiki.imagej.net/Headless#Running_scripts_in_headless_mode

Regards,
Curtis


On Wed, Jun 11, 2014 at 9:23 AM, Johannes Schindelin <
[hidden email]> wrote:

> Hi Paul,
>
> On Wed, 11 Jun 2014, Paul van Schayck wrote:
>
> > I'm running into several problems running ImageJ2-rc2 headless. I'm
> > using this command:
> >
> > ./ImageJ-linux64 --headless -macro scripts/test.ijm
> >
> > The first one is that the new "Open images using scifio" breaks image
> opening through open("image.png"):
> > java.lang.NullPointerException
> >         at
> net.imagej.legacy.IJ1Helper.updateRecentMenu(IJ1Helper.java:307)
>
> Could I ask you to test my fix? If you replaced your
> jars/imagej-legacy-*.jar with the version from
>
>
> http://jenkins.imagej.net/job/ImageJ-legacy/lastSuccessfulBuild/artifact/target/imagej-legacy-0.5.10-SNAPSHOT.jar
>
> it should address that problem.
>
> > This can be fixed by disabling the new SCIFIO option.
>
> Note that we are currently stabilizing this code, with the enormous and
> outstanding help by the community, such as you!
>
> The idea of using SCIFIO is to make it possible to just File>Open just
> about anything and get the correct images. The built-in TIFF reader of
> ImageJ 1.x cuts a few corners, which is fast, but fails to open a couple
> of valid TIFF files correctly, and it also does not heed extensions such
> as OME-TIFF. SCIFIO addresses that.
>
> I have no doubt that with the help of the community, we can make the
> support code in ImageJ2 robust so that everybody can gain the benefits of
> the SCIFIO library.
>
> > The second one, is that this macro:
> >
> > open(imagePath);
> > run("8-bit");
> > setThreshold(1, 255);
> > run("Analyze Particles...", "size=300-Infinity circularity=0.40-1.00");
> > print(nResults);
> >
> > Will return 0. No particles seem to be found. Running the same macro not
> headless:
> > ./ImageJ-linux64 -macro scripts/test.ijm
> > Will return a number of particles in a log window
>
> Have you tried with
>
>         ./ImageJ-linux64 -batch scripts/test.ijm
>
> ? This is a way to call macros in batch mode that is admittedly not
> documented very well... At least that will get you going again if all you
> need is to call the script from the command line and let it obtain the
> results via stdout.
>
> I just tried to reproduce that headless mode problem and get a different
> problem instead. Will keep you posted.
>
> > Thirdly, the option "-macro" seems to be undocumented. Is this the
> > preferred way to start macros? How about new scripts (.js or beanshell).
>
> The -macro option is a standard IamgeJ 1.x option, documented
> unfortunately only in the developer part of the documentation, far from
> regular users' hunting grounds:
>
>
> http://jenkins.imagej.net/job/ImageJ1-javadoc/javadoc/ij/ImageJ.html
>
> Note that it is also not *quite* correct: you can pass -batch as last
> argument *without* a path, which comes in handy if you want to *evaluate*
> an expression -- as opposed to a file containing macro code -- in batch
> mode.
>
> So hopefully this addresses two of your questions; I will keep you updated
> on the remaining one.
>
> Ciao,
> Johannes
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[dscho]

Hi Paul,

On Wed, 11 Jun 2014, Johannes Schindelin wrote:

> On Wed, 11 Jun 2014, Paul van Schayck wrote:
>
> > The second one, is that this macro:
> >
> > open(imagePath);
> > run("8-bit");
> > setThreshold(1, 255);
> > run("Analyze Particles...", "size=300-Infinity circularity=0.40-1.00");
> > print(nResults);
> >
> > Will return 0.

Thank you for that reproducer script, it really saved me *a lot* of time.

What it uncovered, however, are two problems, not one. That's why it took
so long: in headless mode, we would not register legacy images properly,
and also -- a really, really longstanding bug! -- the headless generic
dialog never parsed numbers properly. It always returned 0. Therefore,
your circularity filter was converted into circularity=0.0-0.0.

I fear this bug goes way back into our first headless support days in
ImageJA, so it is really embarrassing, but all the better that you
uncovered it!

Unfortunately, I ran out of time today to clean up the fixes and upload
new file versions, but I promise to do that tomorrow.

Thank you for your patience,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Paul van Schayck]

Hey Johannes and Curtis,

Thank you for your quick responses.

On Thu, Jun 12, 2014 at 1:22 AM, Johannes Schindelin
<[hidden email]> wrote:

> Thank you for that reproducer script, it really saved me *a lot* of time.
>
> What it uncovered, however, are two problems, not one. That's why it took
> so long: in headless mode, we would not register legacy images properly,
> and also -- a really, really longstanding bug! -- the headless generic
> dialog never parsed numbers properly. It always returned 0. Therefore,
> your circularity filter was converted into circularity=0.0-0.0.
>
> I fear this bug goes way back into our first headless support days in
> ImageJA, so it is really embarrassing, but all the better that you
> uncovered it!
>
> Unfortunately, I ran out of time today to clean up the fixes and upload
> new file versions, but I promise to do that tomorrow.

I figured it was a longer standing bug (and should have mentioned
that) as I could only get the particle analyzer headless working on
rather old version of Fiji. I'll keep an eye on Jenkins and Github for
the fixes. I'm also trying to rewrite the macro to js, still
struggling a bit with that, should the bug also be visible there?

Regarding the SCIFIO file opening. Your fix fixed the
NullPointerException. However, now when running with SCIFIO enabled
the following error is shown "There are no images open.". This
regardless of whether I run headless or not, but only occurs when
passing the macro through "-macro" as it does not happen when running
from the script editor. Also adding a wait(4000); fixes the problem.

I fully agree on making SCIFIO default, I was very surprised when I
first discovered a few months a go it was not implemented like this
yet. It will add a lot of nice features.

Regarding the command line options. I'm interested in running in true
headless mode making -batch not directly interesting. So, if I
understand correctly, that while "-macro" is a ImageJ1 option it is
also the preferred way of starting the new ImageJ2 scripts (in js,
bsh, etc). And how exactly does this all relate to the "--ij1" and
"--ij2" flags.

Thank you,

Paul

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[dscho]

Hi Paul,

On Thu, 12 Jun 2014, Paul van Schayck wrote:

> On Thu, Jun 12, 2014 at 1:22 AM, Johannes Schindelin
> <[hidden email]> wrote:
> > Thank you for that reproducer script, it really saved me *a lot* of time.
> >
> > What it uncovered, however, are two problems, not one. That's why it
> > took so long: in headless mode, we would not register legacy images
> > properly, and also -- a really, really longstanding bug! -- the
> > headless generic dialog never parsed numbers properly. It always
> > returned 0. Therefore, your circularity filter was converted into
> > circularity=0.0-0.0.
> >
> > I fear this bug goes way back into our first headless support days in
> > ImageJA, so it is really embarrassing, but all the better that you
> > uncovered it!
> >
> > Unfortunately, I ran out of time today to clean up the fixes and
> > upload new file versions, but I promise to do that tomorrow.
>
> I figured it was a longer standing bug (and should have mentioned
> that) as I could only get the particle analyzer headless working on
> rather old version of Fiji.

Oh, that is interesting. So maybe at the time we started implementing the
headless hack that *kind of* replaced the GenericDialog with a purely
headless one in headless mode, there were no non-GUI methods in
GenericDialog? I will have to investigate when such functions, including
the parseDouble() one that was the culprit in your case, entered the fray.

This will probably uncover quite a few troubling bugs... please bear with
me, I want to fix this properly, rather than papering over the issue with
a workaround that causes even more problems in the long run [*1*].

> I'll keep an eye on Jenkins and Github for the fixes.

Hopefully this is not quite necessary. I try to follow up on mails when I
fixed the described problem... ;-)

> I'm also trying to rewrite the macro to js, still struggling a bit with
> that, should the bug also be visible there?

Yes. Both problems (that the image was not properly registered with ImageJ
1.x as well as the 0 particles found problem caused by the parseDouble()
method being incorrect in headless mode) would persist.

> Regarding the SCIFIO file opening. Your fix fixed the
> NullPointerException. However, now when running with SCIFIO enabled the
> following error is shown "There are no images open.". This regardless of
> whether I run headless or not, but only occurs when passing the macro
> through "-macro" as it does not happen when running from the script
> editor. Also adding a wait(4000); fixes the problem.

Yes, I think that I have a proper fix for that [*2*], but I still have to
verify that it does not cause other problems, add a unit test to guarantee
that the same bug does not reoccur, and then merge the topic branch. I
just ran out of time yesterday night, but I will work on this directly
after the first round of emails of the day.

> I fully agree on making SCIFIO default, I was very surprised when I
> first discovered a few months a go it was not implemented like this yet.
> It will add a lot of nice features.

Yes. I am rather proud of the system myself, having had a small part in
devising the technique we use. You could call it "HandleExtraFileTypes
Done Right", basically learning from years of experience with
HandleExtraFileTypes:

- it is truly extensible, i.e. different third and fourth parties can
  *both* extend the mechanism; there is not a single class that you have
  to modify in order to add your reader/writer, and if two parties
  try to modify it, only one version wins. The other readers/writers
  fall unsupported. This is particularly surprising when the overridden
  HandleExtraFileTypes.class file is stuck into a plugin's .jar file.

- SCIFIO is called *first*, before letting ImageJ try to open the image,
  i.e. it allows overriding the default implementation that us ImageJ
  developers might have made too limited,

- since SCIFIO's plugin concept is based on SciJava's, and hence the
  SCIFIO plugins are typed strongly, it is much easier to get SCIFIO
  readers  and writers right than it is to get ImageJ 1.x plugins right to
  do the same.

  A quick search for reader plugins failing to extend ImagePlus or to
  interpret a non-empty arg as path (essentially breaking batch mode
  operations with HandleExtraFileTypes) may demonstrate what I mean.
  Please understand that I do not blame any of the authors of those
  plugins: I myself was really surprised when I learned *years* after
  writing the initial version of the Amira file format reader that you
  should not call "imp.show();" at the end of the run() method, but
  instead make the plugin class "extend ImagePlus" and then call
  "setImage(imp);" at the end of the run() method.

The best thing about it, however, is how flexible SCIFIO is. By making it
a software library in its own right, SCIFIO can concentrate on its core
strength, and offer it to all users, including non-ImageJ ones. I have
experimental code that allows C++ programs to use SCIFIO, *without* even
the need for a Java Virtual Machine!

Of course, we have to stabilize the SCIFIO support first. But we're well
underway in that project, with outstanding help from the community,
including you!

> Regarding the command line options. I'm interested in running in true
> headless mode making -batch not directly interesting. So, if I
> understand correctly, that while "-macro" is a ImageJ1 option it is
> also the preferred way of starting the new ImageJ2 scripts (in js,
> bsh, etc). And how exactly does this all relate to the "--ij1" and
> "--ij2" flags.

Actually, my preferred way to run scripts is to call

        ImageJ-<platform> script.bsh

or

        ImageJ-<platform> script.js

directly. This will not spin up a context or for that matter, ImageJ, for
you, but makes it possible to use ImageJ truly as a library. You would
probably want to start up a context anyway, like this:

        importClass(Packages.net.imagej.ImageJ);

        ij = ImageJ();
        ij.command().run(...);

We might want to change this in the near future, though, because the
SciJava scripting framework -- which puts all scripting languages into a
common, consistent shape -- is just about ready to be used by the ImageJ
launcher. If you are interested in that project, let's just add a ticket
on GitHub which you can follow to keep you updated automatically.

Ciao,
Johannes

Footnote *1*: I have a workaround, but this was only put into the code on
a topic branch, intended to identify the bug, not to fix it:
https://github.com/imagej/ij1-patcher/compare/master...headless-dialog

Footnote *2*: the affected code lives in the imagej-legacy component (i.e.
the library that is supposed to leverage the extension points inserted by
ij1-patcher that lets ImageJ2 truly extend ImageJ 1.x):
https://github.com/imagej/imagej-legacy/compare/master...batch-mode-opener

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Paul van Schayck]

Dear Johannes,

Thank you for your extensive response.

On Thu, Jun 12, 2014 at 4:03 PM, Johannes Schindelin
<[hidden email]> wrote:
>> I'm also trying to rewrite the macro to js, still struggling a bit with
>> that, should the bug also be visible there?
>
> Yes. Both problems (that the image was not properly registered with ImageJ
> 1.x as well as the 0 particles found problem caused by the parseDouble()
> method being incorrect in headless mode) would persist.

Hmm.. this is contratary to what I found using this script:

importClass(Packages.ij.IJ)
importClass(Packages.ij.plugin.filter.ParticleAnalyzer)
importClass(Packages.ij.measure.Measurements)
importClass(Packages.ij.measure.ResultsTable)

imp = IJ.openImage(imagePath);
IJ.run(imp, "8-bit", "");
IJ.setThreshold(imp, 1, 255);
IJ.run(imp, "Convert to Mask", ""); // How to use ij.plugin.Thresholder?
table = ResultsTable()
pa = ParticleAnalyzer(0, Measurements.CENTER_OF_MASS, table, 300,
Number.POSITIVE_INFINITY, 0.4, 1.0)
pa.analyze(imp)
print(table.getCounter())

This does work headless. However, maybe you meant running the Analyse
particle command through IJ.run() will not work. In that case, ignore
this comment.

> Actually, my preferred way to run scripts is to call
>
>         ImageJ-<platform> script.bsh
>
> or
>
>         ImageJ-<platform> script.js

Running scripts this way in pure ImageJ has been giving this exception
for me for a few weeks. However, it does work in the daily Fiji build.

Exception in thread "main" java.lang.NullPointerException
        at net.imagej.launcher.ClassLauncher.launch(ClassLauncher.java:248)
        at net.imagej.launcher.ClassLauncher.run(ClassLauncher.java:184)
        at net.imagej.launcher.ClassLauncher.main(ClassLauncher.java:76)

> directly. This will not spin up a context or for that matter, ImageJ, for
> you, but makes it possible to use ImageJ truly as a library. You would
> probably want to start up a context anyway, like this:
>
>         importClass(Packages.net.imagej.ImageJ);
>
>         ij = ImageJ();
>         ij.command().run(...);
>

I'm not sure what you mean by this latest example. In which case would
you like to use this?

Thank you,

Paul

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Jan Eglinger]

Dear Paul,

I just noticed the comment in your example code:

On 13.06.2014, 1:24 PM, Paul van Schayck wrote:
 > IJ.run(imp, "Convert to Mask", ""); // How to use ij.plugin.Thresholder?


and wanted to share how I have done it [*1*]:

     import ij.WindowManager;
     import ij.plugin.Thresholder;

     [...]

     private Thresholder thr;

     [...]

     WindowManager.setTempCurrentImage(imp);
     thr.run("mask");
     WindowManager.setTempCurrentImage(null);

If anyone has a more elegant solution, I would be happy to hear it.

Cheers,
Jan

Footnote *1*: The reason why I was looking for a way to avoid calls of
IJ.run is that using IJ.run within an ExtendedPlugInFilter will break
macro recording when using the preview functionality (i.e. macro
commands will be recorded prematurely when IJ.run will be called from a
dialog containing a Preview checkbox). See this bug for background:
http://fiji.sc/bugzilla/show_bug.cgi?id=651



On 13.06.2014, 1:24 PM, Paul van Schayck wrote:

> Dear Johannes,
>
> Thank you for your extensive response.
>
> On Thu, Jun 12, 2014 at 4:03 PM, Johannes Schindelin
> <[hidden email]> wrote:
>>> I'm also trying to rewrite the macro to js, still struggling a bit with
>>> that, should the bug also be visible there?
>>
>> Yes. Both problems (that the image was not properly registered with ImageJ
>> 1.x as well as the 0 particles found problem caused by the parseDouble()
>> method being incorrect in headless mode) would persist.
>
> Hmm.. this is contratary to what I found using this script:
>
> importClass(Packages.ij.IJ)
> importClass(Packages.ij.plugin.filter.ParticleAnalyzer)
> importClass(Packages.ij.measure.Measurements)
> importClass(Packages.ij.measure.ResultsTable)
>
> imp = IJ.openImage(imagePath);
> IJ.run(imp, "8-bit", "");
> IJ.setThreshold(imp, 1, 255);
> IJ.run(imp, "Convert to Mask", ""); // How to use ij.plugin.Thresholder?
> table = ResultsTable()
> pa = ParticleAnalyzer(0, Measurements.CENTER_OF_MASS, table, 300,
> Number.POSITIVE_INFINITY, 0.4, 1.0)
> pa.analyze(imp)
> print(table.getCounter())
>
> This does work headless. However, maybe you meant running the Analyse
> particle command through IJ.run() will not work. In that case, ignore
> this comment.
>
>> Actually, my preferred way to run scripts is to call
>>
>>          ImageJ-<platform> script.bsh
>>
>> or
>>
>>          ImageJ-<platform> script.js
>
> Running scripts this way in pure ImageJ has been giving this exception
> for me for a few weeks. However, it does work in the daily Fiji build.
>
> Exception in thread "main" java.lang.NullPointerException
>          at net.imagej.launcher.ClassLauncher.launch(ClassLauncher.java:248)
>          at net.imagej.launcher.ClassLauncher.run(ClassLauncher.java:184)
>          at net.imagej.launcher.ClassLauncher.main(ClassLauncher.java:76)
>
>> directly. This will not spin up a context or for that matter, ImageJ, for
>> you, but makes it possible to use ImageJ truly as a library. You would
>> probably want to start up a context anyway, like this:
>>
>>          importClass(Packages.net.imagej.ImageJ);
>>
>>          ij = ImageJ();
>>          ij.command().run(...);
>>
>
> I'm not sure what you mean by this latest example. In which case would
> you like to use this?
>
> Thank you,
>
> Paul
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[dscho]

Hi Paul,

On Fri, 13 Jun 2014, Paul van Schayck wrote:

> Thank you for your extensive response.

You are welcome.

> On Thu, Jun 12, 2014 at 4:03 PM, Johannes Schindelin
> <[hidden email]> wrote:
> >> I'm also trying to rewrite the macro to js, still struggling a bit with
> >> that, should the bug also be visible there?
> >
> > Yes. Both problems (that the image was not properly registered with ImageJ
> > 1.x as well as the 0 particles found problem caused by the parseDouble()
> > method being incorrect in headless mode) would persist.
>
> Hmm.. this is contratary to what I found using this script:
>
> importClass(Packages.ij.IJ)
> importClass(Packages.ij.plugin.filter.ParticleAnalyzer)
> importClass(Packages.ij.measure.Measurements)
> importClass(Packages.ij.measure.ResultsTable)
>
> imp = IJ.openImage(imagePath);
> IJ.run(imp, "8-bit", "");
> IJ.setThreshold(imp, 1, 255);
> IJ.run(imp, "Convert to Mask", ""); // How to use ij.plugin.Thresholder?
> table = ResultsTable()
> pa = ParticleAnalyzer(0, Measurements.CENTER_OF_MASS, table, 300,
> Number.POSITIVE_INFINITY, 0.4, 1.0)
> pa.analyze(imp)
> print(table.getCounter())
>
> This does work headless. However, maybe you meant running the Analyse
> particle command through IJ.run() will not work. In that case, ignore
> this comment.

Yes, I should have been more clear on that account. But then, I am already
surprised that the "imp = IJ.openImage(imagePath);" does not return
null... ;-) Good that it works, though!

> > Actually, my preferred way to run scripts is to call
> >
> >         ImageJ-<platform> script.bsh
> >
> > or
> >
> >         ImageJ-<platform> script.js
>
> Running scripts this way in pure ImageJ has been giving this exception
> for me for a few weeks. However, it does work in the daily Fiji build.
>
> Exception in thread "main" java.lang.NullPointerException
>         at net.imagej.launcher.ClassLauncher.launch(ClassLauncher.java:248)
>         at net.imagej.launcher.ClassLauncher.run(ClassLauncher.java:184)
>         at net.imagej.launcher.ClassLauncher.main(ClassLauncher.java:76)

Hmm... I cannot reproduce in my Fiji.app/...

> > directly. This will not spin up a context or for that matter, ImageJ, for
> > you, but makes it possible to use ImageJ truly as a library. You would
> > probably want to start up a context anyway, like this:
> >
> >         importClass(Packages.net.imagej.ImageJ);
> >
> >         ij = ImageJ();
> >         ij.command().run(...);
> >
>
> I'm not sure what you mean by this latest example. In which case would
> you like to use this?

This is for the cases where you run ImageJ2 commands headlessly in a
script... I will elaborate on that more when we're out of the user-centric
phase of the release cycle: at the moment, I cannot focus on
developer-centric issues.  But I will definitely write a nice blog post
about the intended use in scripts!

Ciao,
Johannes

P.S.: I am *about* to merge the branch and then release, so that finally
your macro works again. Sorry for the delay!

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[dscho]

Hi Paul,

On Fri, 13 Jun 2014, Johannes Schindelin wrote:

> P.S.: I am *about* to merge the branch and then release, so that finally
> your macro works again. Sorry for the delay!

It is uploaded [*1*]. After updating your Fiji, your macro should work.
Oh, and you earned a piece of eternal fame: your name is part of the
development history of ImageJ2 now:

        https://github.com/imagej/imagej-legacy/commit/ddb3af62

Ciao,
Johannes

Footnote *1*: we briefly considered increasing the release candidate
number (visible when clicking on the status bar) when uploading any new
component to the update site, but these days updates are just happening
too fast...

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[dscho]

Hi Jan,

On Fri, 13 Jun 2014, Jan Eglinger wrote:

> I just noticed the comment in your example code:
>
> On 13.06.2014, 1:24 PM, Paul van Schayck wrote:
> > IJ.run(imp, "Convert to Mask", ""); // How to use ij.plugin.Thresholder?
>
>
> and wanted to share how I have done it [*1*]:
>
>     import ij.WindowManager;
>     import ij.plugin.Thresholder;
>
>     [...]
>
>     private Thresholder thr;
>
>     [...]
>
>     WindowManager.setTempCurrentImage(imp);
>     thr.run("mask");
>     WindowManager.setTempCurrentImage(null);
>
> If anyone has a more elegant solution, I would be happy to hear it.

You could also make use of the static methods in
fiji.threshold.Auto_Threshold (i.e. Gabriel Landini's Auto Threshold
plugin) with the histogram obtained via

        http://jenkins.imagej.net/job/ImageJ1-javadoc/javadoc/ij/process/ImageProcessor.html#getHistogram%28%29

then set the threshold via

        http://jenkins.imagej.net/job/ImageJ1-javadoc/javadoc/ij/process/ImageProcessor.html#setThreshold%28double,%20double,%20int%29

and if you need a mask, turn the threshold into a ROI via

        http://jenkins.imagej.net/job/ImageJ1-javadoc/javadoc/ij/plugin/filter/ThresholdToSelection.html#run%28ij.ImagePlus%29

and then make the mask by calling

        http://jenkins.imagej.net/job/ImageJ1-javadoc/javadoc/ij/process/ImageProcessor.html#setRoi%28ij.gui.Roi%29

and

        http://jenkins.imagej.net/job/ImageJ1-javadoc/javadoc/ij/process/ImageProcessor.html#getMask%28%29

I know, not very straight-forward... but thread-safe! Well, kinda...

Ciao,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Paul van Schayck]

Hi Johannes,

On Fri, Jun 13, 2014 at 7:27 PM, Johannes Schindelin
<[hidden email]> wrote:

> Hi Paul,
>
> On Fri, 13 Jun 2014, Johannes Schindelin wrote:
>
>> P.S.: I am *about* to merge the branch and then release, so that finally
>> your macro works again. Sorry for the delay!
>
> It is uploaded [*1*]. After updating your Fiji, your macro should work.
> Oh, and you earned a piece of eternal fame: your name is part of the
> development history of ImageJ2 now:
>
>         https://github.com/imagej/imagej-legacy/commit/ddb3af62

Thank you very much. The macro (and few parts more I haven't shared)
are working perfectly again. Also in SCIFIO mode. Thank you for the
credits on the unit (regression?) test macro.

A remaining issue is that when the macro (or js script for that
matter) finishes there is a delay (1 minute) in ImageJ fully exiting.
An explicit exit() does not seem to help. This does not happen in
Fiji. I saw a recent commit for this on Github, but that does not seem
to have completely solved it.

>> Running scripts this way in pure ImageJ has been giving this exception
>> for me for a few weeks. However, it does work in the daily Fiji build.
>>
>> Exception in thread "main" java.lang.NullPointerException
>>         at net.imagej.launcher.ClassLauncher.launch(ClassLauncher.java:248)
>>         at net.imagej.launcher.ClassLauncher.run(ClassLauncher.java:184)
>>         at net.imagej.launcher.ClassLauncher.main(ClassLauncher.java:76)
>
>Hmm... I cannot reproduce in my Fiji.app/...

No, exactly, it does work for me in Fiji.app/ but it does *not* work
in ImageJ.app/.

Thank you,

Paul

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[dscho]

Hi Paul,

On Mon, 16 Jun 2014, Paul van Schayck wrote:

> A remaining issue is that when the macro (or js script for that matter)
> finishes there is a delay (1 minute) in ImageJ fully exiting.  An
> explicit exit() does not seem to help. This does not happen in Fiji. I
> saw a recent commit for this on Github, but that does not seem to have
> completely solved it.

That is strange. We run our regression tests (including macros) pretty
frequently and no such behavior was recorded. Maybe this behavior does not
occur in headless mode?

In any case, do you know about the Ctrl+\ trick on the console to ask Java
for a thread dump? That might help you determine which thread is still
alive (and obviously blocking).

> >> Running scripts this way in pure ImageJ has been giving this exception
> >> for me for a few weeks. However, it does work in the daily Fiji build.
> >>
> >> Exception in thread "main" java.lang.NullPointerException
> >>         at net.imagej.launcher.ClassLauncher.launch(ClassLauncher.java:248)
> >>         at net.imagej.launcher.ClassLauncher.run(ClassLauncher.java:184)
> >>         at net.imagej.launcher.ClassLauncher.main(ClassLauncher.java:76)
> >
> >Hmm... I cannot reproduce in my Fiji.app/...
>
> No, exactly, it does work for me in Fiji.app/ but it does *not* work
> in ImageJ.app/.

I'll have another look. Thank you for pointing out that the problem occurs
in ImageJ.app/, it never occured to me that that was the issue.

Ciao,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[dscho]

Hi Paul,

On Mon, 23 Jun 2014, Johannes Schindelin wrote:

> On Mon, 16 Jun 2014, Paul van Schayck wrote:
>
> > >> Running scripts this way in pure ImageJ has been giving this exception
> > >> for me for a few weeks. However, it does work in the daily Fiji build.
> > >>
> > >> Exception in thread "main" java.lang.NullPointerException
> > >>         at net.imagej.launcher.ClassLauncher.launch(ClassLauncher.java:248)
> > >>         at net.imagej.launcher.ClassLauncher.run(ClassLauncher.java:184)
> > >>         at net.imagej.launcher.ClassLauncher.main(ClassLauncher.java:76)
> > >
> > >Hmm... I cannot reproduce in my Fiji.app/...
> >
> > No, exactly, it does work for me in Fiji.app/ but it does *not* work
> > in ImageJ.app/.
>
> I'll have another look. Thank you for pointing out that the problem occurs
> in ImageJ.app/, it never occured to me that that was the issue.

Hmm... I cannot reproduce with a current ImageJ.app/... Maybe it is
script-dependent? I tried with a .bsh file that contains just:
print("Hello");

Ciao,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Paul van Schayck]

Dear Johannes,

On Mon, Jun 23, 2014 at 6:31 PM, Johannes Schindelin
<[hidden email]> wrote:
> Hi Paul,
> That is strange. We run our regression tests (including macros) pretty
> frequently and no such behavior was recorded. Maybe this behavior does not
> occur in headless mode?

It is pretty consistent behaviour in headless mode. It takes about 30s
to a minute for it to return to the command line after running a macro
or script (only js, not bsh!). It also happens after having run the
updater (although the cause may be completely different there I
realise). It does not happen in Fiji.daily.

> In any case, do you know about the Ctrl+\ trick on the console to ask Java
> for a thread dump? That might help you determine which thread is still
> alive (and obviously blocking).

I've made a pastebin from this:
http://pastebin.com/uF9AMKNH

This one looks suspicious:
  java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000009d8b0070> (a
java.util.concurrent.SynchronousQueue$TransferStack)

>> >> Running scripts this way in pure ImageJ has been giving this exception
>> >> for me for a few weeks. However, it does work in the daily Fiji build.
>> >>
>> >> Exception in thread "main" java.lang.NullPointerException
>> >>         at net.imagej.launcher.ClassLauncher.launch(ClassLauncher.java:248)
>> >>         at net.imagej.launcher.ClassLauncher.run(ClassLauncher.java:184)
>> >>         at net.imagej.launcher.ClassLauncher.main(ClassLauncher.java:76)
>> >
>> >Hmm... I cannot reproduce in my Fiji.app/...
>>
>> No, exactly, it does work for me in Fiji.app/ but it does *not* work
>> in ImageJ.app/.
>
> I'll have another look. Thank you for pointing out that the problem occurs
> in ImageJ.app/, it never occured to me that that was the issue.

You were right, it's the language used. This works:
./ImageJ-linux64 --headless scripts/short.bsh

This throws the exception:
./ImageJ-linux64 --headless scripts/short.js

Once again, not in Fiji.daily. They are both running the same JRE (the
one distributed with Fiji.app).

Thanks,

Paul

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[dscho]

Hi Paul,

On Tue, 24 Jun 2014, Paul van Schayck wrote:

> On Mon, Jun 23, 2014 at 6:31 PM, Johannes Schindelin
> <[hidden email]> wrote:
> >
> > That is strange. We run our regression tests (including macros) pretty
> > frequently and no such behavior was recorded. Maybe this behavior does not
> > occur in headless mode?
>
> It is pretty consistent behaviour in headless mode. It takes about 30s
> to a minute for it to return to the command line after running a macro
> or script (only js, not bsh!). It also happens after having run the
> updater (although the cause may be completely different there I
> realise). It does not happen in Fiji.daily.
>
> > In any case, do you know about the Ctrl+\ trick on the console to ask Java
> > for a thread dump? That might help you determine which thread is still
> > alive (and obviously blocking).
>
> I've made a pastebin from this:
> http://pastebin.com/uF9AMKNH
>
> This one looks suspicious:
>   java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x000000009d8b0070> (a
> java.util.concurrent.SynchronousQueue$TransferStack)

Hmm... It looks as if the Context is not correctly disposed of. Is there
any chance that you can reduce your script to a minimal example of that
behavior and share that script with us? At least here I cannot reproduce
with a print("Hello"); script...

> >> >> Running scripts this way in pure ImageJ has been giving this exception
> >> >> for me for a few weeks. However, it does work in the daily Fiji build.
> >> >>
> >> >> Exception in thread "main" java.lang.NullPointerException
> >> >>         at net.imagej.launcher.ClassLauncher.launch(ClassLauncher.java:248)
> >> >>         at net.imagej.launcher.ClassLauncher.run(ClassLauncher.java:184)
> >> >>         at net.imagej.launcher.ClassLauncher.main(ClassLauncher.java:76)
> >> >
> >> >Hmm... I cannot reproduce in my Fiji.app/...
> >>
> >> No, exactly, it does work for me in Fiji.app/ but it does *not* work
> >> in ImageJ.app/.
> >
> > I'll have another look. Thank you for pointing out that the problem occurs
> > in ImageJ.app/, it never occured to me that that was the issue.
>
> You were right, it's the language used. This works:
> ./ImageJ-linux64 --headless scripts/short.bsh
>
> This throws the exception:
> ./ImageJ-linux64 --headless scripts/short.js
>
> Once again, not in Fiji.daily. They are both running the same JRE (the
> one distributed with Fiji.app).

Ah! That is easily explained. Javascript is included in Java 6. But in a
different package than Mozilla's original one (this is called "shading"
and prevents many problems when inconsistent versions of the same
component are on the same class path).

The launcher assumes the original packages, i.e. the Mozilla ones, but
that .jar file is only present in Fiji's jars/ directory.

We probably need to integrate the launcher better with the SciJava
scripting framework (which was designed, after all, to offer a
consistent interface to all scripting languages); that would fix your
problem with Javascript, make the command-line support for scripting more
easily extended (supporting e.g. running .java "scripts" from the
command-line!), and in general The Right Thing.

Having said that, I still feel that we have a little more work to do in
the user-centric phase of the work leading to 2.0.0 final, and this change
strikes me as a developer-centric one.

So I opened a ticket as a reminder for later:
https://github.com/imagej/imagej-launcher/issues/15

Feel free to log into GitHub (if you do not have an account yet: it is
free) and click the "Watch" button of that ticket to receive notification
mails about updates.

Ciao,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Paul van Schayck]

Hey Johannes,

On Tue, Jun 24, 2014 at 3:51 PM, Johannes Schindelin
<[hidden email]> wrote:

> On Tue, 24 Jun 2014, Paul van Schayck wrote:
>
>> On Mon, Jun 23, 2014 at 6:31 PM, Johannes Schindelin
>> It is pretty consistent behaviour in headless mode. It takes about 30s
>> to a minute for it to return to the command line after running a macro
>> or script (only js, not bsh!). It also happens after having run the
>> updater (although the cause may be completely different there I
>> realise). It does not happen in Fiji.daily.
>>
>> This one looks suspicious:
>>   java.lang.Thread.State: TIMED_WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x000000009d8b0070> (a
>> java.util.concurrent.SynchronousQueue$TransferStack)
>
> Hmm... It looks as if the Context is not correctly disposed of. Is there
> any chance that you can reduce your script to a minimal example of that
> behavior and share that script with us? At least here I cannot reproduce
> with a print("Hello"); script...

This already is a print("hello") script. Which is being run like this:
./ImageJ-linux64 --headless -macro scripts/short.js

For bsh script this only happens when being run with the -macro
options. NOT when being run like this:
./ImageJ-linux64 --headless scripts/short.bsh

I can also reproduce this on another 32 bit system, also running Ubuntu 14.04.

> Ah! That is easily explained. Javascript is included in Java 6. But in a
> different package than Mozilla's original one (this is called "shading"
> and prevents many problems when inconsistent versions of the same
> component are on the same class path).
>
> The launcher assumes the original packages, i.e. the Mozilla ones, but
> that .jar file is only present in Fiji's jars/ directory.
>
> We probably need to integrate the launcher better with the SciJava
> scripting framework (which was designed, after all, to offer a
> consistent interface to all scripting languages); that would fix your
> problem with Javascript, make the command-line support for scripting more
> easily extended (supporting e.g. running .java "scripts" from the
> command-line!), and in general The Right Thing.
>
> Having said that, I still feel that we have a little more work to do in
> the user-centric phase of the work leading to 2.0.0 final, and this change
> strikes me as a developer-centric one.

Thanks for the explanation. Regarding it being a developer-centric
issue. It did confuse the hell out of me at first.

Is it possible to clear up the --help message a bit? The --ij1 and
--ij2 options are quite confusing, and not clear to me. The ij1 -macro
and -batch options are not mentioned. The type of <files> could be
mentioned, so both image files and bsh script files.

Thanks,

Paul

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[dscho]

Hi Paul,

On Wed, 25 Jun 2014, Paul van Schayck wrote:

> On Tue, Jun 24, 2014 at 3:51 PM, Johannes Schindelin
> <[hidden email]> wrote:
> > On Tue, 24 Jun 2014, Paul van Schayck wrote:
> >
> >> On Mon, Jun 23, 2014 at 6:31 PM, Johannes Schindelin
> >> It is pretty consistent behaviour in headless mode. It takes about 30s
> >> to a minute for it to return to the command line after running a macro
> >> or script (only js, not bsh!). It also happens after having run the
> >> updater (although the cause may be completely different there I
> >> realise). It does not happen in Fiji.daily.
> >>
> >> This one looks suspicious:
> >>   java.lang.Thread.State: TIMED_WAITING (parking)
> >>         at sun.misc.Unsafe.park(Native Method)
> >>         - parking to wait for  <0x000000009d8b0070> (a
> >> java.util.concurrent.SynchronousQueue$TransferStack)
> >
> > Hmm... It looks as if the Context is not correctly disposed of. Is there
> > any chance that you can reduce your script to a minimal example of that
> > behavior and share that script with us? At least here I cannot reproduce
> > with a print("Hello"); script...
>
> This already is a print("hello") script. Which is being run like this:
> ./ImageJ-linux64 --headless -macro scripts/short.js
>
> For bsh script this only happens when being run with the -macro
> options.

Does it also happen if you run

        ./ImageJ-linux64 --headless -batch scripts/short.js

?

> Is it possible to clear up the --help message a bit? The --ij1 and --ij2
> options are quite confusing, and not clear to me. The ij1 -macro and
> -batch options are not mentioned. The type of <files> could be
> mentioned, so both image files and bsh script files.

Well, the launcher has its own options and passes all others to the
program to be launched. So -macro and -batch are actually not options that
the launcher really knows about... About the scripts accepted, it seems
that there is a bug and only --js is mentioned, but that documentation
actually mentions .js files explicitly:

        --js
                start Javascript (the Rhino engine) instead of
                ImageJ (this is the default when called with a
                file ending in .js)

Ciao,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Paul van Schayck]

Hey Johannes,

On Thu, Jun 26, 2014 at 1:00 AM, Johannes Schindelin
<[hidden email]> wrote:

> On Wed, 25 Jun 2014, Paul van Schayck wrote:
>
>> This already is a print("hello") script. Which is being run like this:
>> ./ImageJ-linux64 --headless -macro scripts/short.js
>>
>> For bsh script this only happens when being run with the -macro
>> options.
>
> Does it also happen if you run
>
>         ./ImageJ-linux64 --headless -batch scripts/short.js
>
> ?

Ehmm.. well when using -batch there is no output in the terminal, or
any other indication that the script is being run. Also, the delay in
closing is still there. Actually, when just running (so not script
specified):
./ImageJ-linux64 --headless -batch
The same delay is present in returning the command.

>> Is it possible to clear up the --help message a bit? The --ij1 and --ij2
>> options are quite confusing, and not clear to me. The ij1 -macro and
>> -batch options are not mentioned. The type of <files> could be
>> mentioned, so both image files and bsh script files.
>
> Well, the launcher has its own options and passes all others to the
> program to be launched. So -macro and -batch are actually not options that
> the launcher really knows about... About the scripts accepted, it seems
> that there is a bug and only --js is mentioned, but that documentation
> actually mentions .js files explicitly:
>
>         --js
>                 start Javascript (the Rhino engine) instead of
>                 ImageJ (this is the default when called with a
>                 file ending in .js)

Yes, fiji does show this extra option. However ImageJ2 does not. And
while I (now!) understand the technical reasons why the -macro option
is not mentioned it makes for a very confusing user experience. Right
now, for someone new to IJ it's very difficult to understand the
relation between IJ1 and IJ2. The help of the commands not being
complete adds to that.

Paul

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[ctrueden]

Hi Paul & everyone,

> Yes, fiji does show this extra option. However ImageJ2 does not. And
> while I (now!) understand the technical reasons why the -macro option
> is not mentioned it makes for a very confusing user experience. Right
> now, for someone new to IJ it's very difficult to understand the
> relation between IJ1 and IJ2. The help of the commands not being
> complete adds to that.

I agree. I have wanted to beef up the output of the launcher's "--help" for
a while now, but haven't had time. Since it is the "ImageJ launcher" I
think it should make a best effort to print out all of the command line
flags that ImageJ supports on the Java side, not just at the launcher level.

Of course, this may fall out of sync with what ImageJ supports, and in
ImageJ2 those command line arguments are fully extensible, so you might
turn on an update site that adds an option. We'll have to think a little
about how best to support that; I have some ideas.

I filed an issue for it:
https://github.com/imagej/imagej-launcher/issues/20

Regards,
Curtis


On Thu, Jun 26, 2014 at 1:40 AM, Paul van Schayck <[hidden email]> wrote:

> Hey Johannes,
>
> On Thu, Jun 26, 2014 at 1:00 AM, Johannes Schindelin
> <[hidden email]> wrote:
> > On Wed, 25 Jun 2014, Paul van Schayck wrote:
> >
> >> This already is a print("hello") script. Which is being run like this:
> >> ./ImageJ-linux64 --headless -macro scripts/short.js
> >>
> >> For bsh script this only happens when being run with the -macro
> >> options.
> >
> > Does it also happen if you run
> >
> >         ./ImageJ-linux64 --headless -batch scripts/short.js
> >
> > ?
>
> Ehmm.. well when using -batch there is no output in the terminal, or
> any other indication that the script is being run. Also, the delay in
> closing is still there. Actually, when just running (so not script
> specified):
> ./ImageJ-linux64 --headless -batch
> The same delay is present in returning the command.
>
> >> Is it possible to clear up the --help message a bit? The --ij1 and --ij2
> >> options are quite confusing, and not clear to me. The ij1 -macro and
> >> -batch options are not mentioned. The type of <files> could be
> >> mentioned, so both image files and bsh script files.
> >
> > Well, the launcher has its own options and passes all others to the
> > program to be launched. So -macro and -batch are actually not options
> that
> > the launcher really knows about... About the scripts accepted, it seems
> > that there is a bug and only --js is mentioned, but that documentation
> > actually mentions .js files explicitly:
> >
> >         --js
> >                 start Javascript (the Rhino engine) instead of
> >                 ImageJ (this is the default when called with a
> >                 file ending in .js)
>
> Yes, fiji does show this extra option. However ImageJ2 does not. And
> while I (now!) understand the technical reasons why the -macro option
> is not mentioned it makes for a very confusing user experience. Right
> now, for someone new to IJ it's very difficult to understand the
> relation between IJ1 and IJ2. The help of the commands not being
> complete adds to that.
>
> Paul
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html