when does setting the thread number gets effective

[Stephan Janosch]

Hey imagej and fiji users!

We have Fiji on our cluster here using the multiview registration.

And we want to specifiy the number of used threads dynamically (as a
parameter to the fiji call). So we do like this:


call script:
> $fijiApp -Dxml=$xml -Dsigma=$sigma -Dthreshold=$threshold -Dchannels=$channels -Dlabel=$labels -DthreadCount=$threadCount -- --console $beanShellScript > $xmlDir/fiji_detect$jobName$labels$channels.txt

$beanShellScript (only the relevant parts)
> import ij.Prefs;
> threadCount=Integer.parseInt(System.getProperty("threadCount"));
>
> Prefs.setThreads(threadCount);
> Prefs.savePreferences();

Does this running instance of Fiji actually knows about the new set
value? Or do I have to restart Fiji to pick up the new thread setting?

Or did I miss some command line parameter to set the number of used
threads at all?

Thanks and happy weekend community!
Stephan

--
Stephan Janosch
Software Engineer - TransgeneOme Database
https://transgeneome.mpi-cbg.de

Max Planck Institute of Molecular Cell Biology and Genetics
Pfotenhauerstr. 108
01307 Dresden
Germany

Room: 205
Phone: [phone]
Email: [hidden email]
Web: www.mpi-cbg.de
Twitter: https://twitter.com/TransgeneOme

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Michael Schmid]

Hi Stephan,

not for Fiji but for plain ImageJ:
You can set the muber of threads used for many commands using
PlugInFilters (processing stack slices in parallel, but also
parallelization of Gaussion Blur, Median, Mean, Minimum, Maximum filters
and some more) in Edit>Options>Memory and Threads. The number you enter
there becomes effective immediately, but I think it is not stored in the
Prefs.

The default number is the number of cores (hyperthreading counts as
additional cores), Runtime.getRuntime().availableProcessors();

Michael
_________________________________________________________

On Fri, March 27, 2015 16:10, Stephan Janosch wrote:

> Hey imagej and fiji users!
>
> We have Fiji on our cluster here using the multiview registration.
>
> And we want to specifiy the number of used threads dynamically (as a
> parameter to the fiji call). So we do like this:
>
>
> call script:
>> $fijiApp -Dxml=$xml -Dsigma=$sigma -Dthreshold=$threshold
>> -Dchannels=$channels -Dlabel=$labels -DthreadCount=$threadCount --
>> --console $beanShellScript >
>> $xmlDir/fiji_detect$jobName$labels$channels.txt
>
> $beanShellScript (only the relevant parts)
>> import ij.Prefs;
>> threadCount=Integer.parseInt(System.getProperty("threadCount"));
>>
>> Prefs.setThreads(threadCount);
>> Prefs.savePreferences();
>
> Does this running instance of Fiji actually knows about the new set
> value? Or do I have to restart Fiji to pick up the new thread setting?
>
> Or did I miss some command line parameter to set the number of used
> threads at all?
>
> Thanks and happy weekend community!
> Stephan
>
> --
> Stephan Janosch
> Software Engineer - TransgeneOme Database
> https://transgeneome.mpi-cbg.de
>
> Max Planck Institute of Molecular Cell Biology and Genetics
> Pfotenhauerstr. 108
> 01307 Dresden
> Germany
>
> Room: 205
> Phone: [phone]
> Email: [hidden email]
> Web: www.mpi-cbg.de
> Twitter: https://twitter.com/TransgeneOme
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Stephan Janosch]

Thanks for your answers Michael,

Am 28/03/15 um 20:27 schrieb Michael Schmid:
> You can set the muber of threads used for many commands using
> PlugInFilters (processing stack slices in parallel, but also
> parallelization of Gaussion Blur, Median, Mean, Minimum, Maximum filters
> and some more) in Edit>Options>Memory and Threads. The number you enter
> there becomes effective immediately, but I think it is not stored in the
> Prefs.

I don't have any access to Fiji while it's running on our cluster. So
thats why i need another way of setting the number of threads.

The number of threads is saved in IJ_Prefs!

> The default number is the number of cores (hyperthreading counts as
> additional cores), Runtime.getRuntime().availableProcessors();

On a cluster you need very fine control about the number of used
threads. That's why the multiview reconstruction plugin does not use

> Runtime.getRuntime().availableProcessors()

anymore, but the other mechanism mentioned above.

I'll keep trying. Now I try this in my beanshell script

> threadCount=Integer.parseInt(System.getProperty("threadCount"));
> IJ.run("Memory & Threads...", "parallel="+threadCount);

Maybe that works.

Stephan


--
Stephan Janosch
Software Engineer - TransgeneOme Database
https://transgeneome.mpi-cbg.de

Max Planck Institute of Molecular Cell Biology and Genetics
Pfotenhauerstr. 108
01307 Dresden
Germany

Room: 205
Phone: [phone]
Email: [hidden email]
Web: www.mpi-cbg.de
Twitter: https://twitter.com/TransgeneOme

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Paul van Schayck]

Hi Stephan,

On Fri, Mar 27, 2015 at 4:10 PM, Stephan Janosch <[hidden email]> wrote:
> Hey imagej and fiji users!
> Or did I miss some command line parameter to set the number of used
> threads at all?
>

We've also struggled to limit the number of threads ImageJ was
starting on our cluster. The problem is that nothing enforces plugins
and core code to actually obey to the "IJ.run("Memory & Threads...",
"parallel="+threadCount);" option.

In the end we have the best results by faking the number of cores
reported to Java. Java is getting this number from sysconf() (on unix,
which I assume you are using). We are using a small preprocessor that
intercepts this call and reports a different number.

This library is libsysconfcpus and can be found here:
http://www.kev.pulo.com.au/libsysconfcpus/

Paul van Schayck

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Stephan Janosch]

Thank you very much Paul!

> This library is libsysconfcpus and can be found here:
> http://www.kev.pulo.com.au/libsysconfcpus/

Cheers,
Stephan

--
Stephan Janosch
Software Engineer - TransgeneOme Database
https://transgeneome.mpi-cbg.de

Max Planck Institute of Molecular Cell Biology and Genetics
Pfotenhauerstr. 108
01307 Dresden
Germany

Room: 205
Phone: [phone]
Email: [hidden email]
Web: www.mpi-cbg.de
Twitter: https://twitter.com/TransgeneOme

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Stephan Janosch]

A little recap on this:

If the plugin takes the number of threads set via the prefs into
account, the code below works as intended. I tested around with the
http://fiji.sc/Multiview-Reconstruction plugin, which uses the setting
and voila, the number of used cores on our cluster node was controllable
like a charm.

Beanshell
> import ij.Prefs;
> Prefs.setThreads(threadCount);

Happy image crunching!
Stephan

PS. Paul van Schayk pointed to
http://www.kev.pulo.com.au/libsysconfcpus/ for all other plugins, which
rely on the JVM to detect the cores.




--
Stephan Janosch
Software Engineer - TransgeneOme Database
https://transgeneome.mpi-cbg.de

Max Planck Institute of Molecular Cell Biology and Genetics
Pfotenhauerstr. 108
01307 Dresden
Germany

Room: 205
Phone: [phone]
Email: [hidden email]
Web: www.mpi-cbg.de
Twitter: https://twitter.com/TransgeneOme

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html