Posted by
suendermann on
URL: http://imagej.273.s1.nabble.com/imageJ-distributed-computing-tp3686733p3686742.html
Hi,
> On Thu, 30 Sep 2010, suendermann wrote:
>
>>> For example, you will never be able to parallelize in general a given
>>> macro. But you could provide a way to do the Process>Batch>Macro... in
>>> a truly parallel manner, say, on a cluster.
>>
>> Maybe I don't get the point, but what do you mean with cluster?
>
> Except for some enterprisey marketing types's new definitions, a cluster
> is a well defined entity: a couple of computers with identical
> architecture and operating system (so-called nodes), controlled by a
> master node, which gets sent jobs using a so-called scheduler.
I'm aware what a cluster is. I just wanted to be sure we are speaking
about the same thing and not about cloud or a thing like the sun
grid-engine.
>> Just one PC with several cpus or a bigger thing with many nodes each
>> with its own cpus? We tried to get it run on a machine with 8 nodes with
>> 8 CPUs each and had to find out, that ImageJ is not able to communicate
>> over nodes. It stuck within the node and its cpus.
>
> I tried to get over the point in my previous mail: it is _not_ ImageJ! It
> is the _plugin_! ImageJ would be happy to run on as many cores as you let
> it. But ImageJ is inherently sequential, as it is designed as a desktop
> application (which I probably mentioned before).
Maybe I'm a little bit brain-blocked today, but eventually we don't
speak about the same. Running ImageJ on many cores isn't a problem. I
even don't want to parallelize a macro. I' interested in building a
plugin that uses as many cores as possible, like the
ExtendedPlugInFilter. At this point I didn't get it run on more than 8
cores (each of our nodes has 8 cores). If you did so, could you give me
a hint?
>> So our only solution was to start it on every node separatly and use a
>> script to distribute to "jobs" to the nodes.
>
> If you thought that there would be an automatic way to distribute
> the work load of a sequential macro magically onto a number of cores, then
> I have to disappoint you. It is easy to construct pathological examples
> where each step of the macro depends on the previous step, and where each
> individual step is not parallelizable. There is simply no way to
> parallelize this thing automatically.
No, I don't like magic... I want to control such stuff. Maybe Fiji's
jython interface is a starting point for that. It should be easy to
include the functions/plugins you need in a python script and distribute
that across your cluster.
>>> However. The big problem is that ImageJ was designed as a desktop
>>> application, and therefore many plugins require a graphical desktop
>>> even if they do not display anything. For example, if you instantiate
>>> any java.awt.Dialog on Linux, you need a graphical desktop.
>>>
>>> We tried to work around this issue in the 'headless' hack in Fiji
>>> (basically providing fake Dialog and Menus classes), but fact is: many
>>> plugins will fail to run with that hack, because they access more
>>> functionality than we provide, e.g. registering callbacks such as
>>> constraining the aspect ratio when downsampling. And our fake Dialog
>>> class does not provide all required fake methods for that.
>>
>> Do you plan to improve your fake class or do you search for another
>> solution?
>
> I will improve the fake Dialog and Menus classes as I need to. I will also
> happily accept improvements others needed to do. But it is a hack, and
> every hack shows its limitations if you use it too hard.
>
> The proper solution is a redesign of the plugin mechanism to allow
> headless execution (this is underway as far as I understand)
This sounds great!
> but of
> course this will only work with plugins written using the new interface.
> The old plugins will continue to have the problem.
>
> Hth,
> Johannes
Greetings
Fred