Posted by
dscho on
Sep 30, 2010; 8:22am
URL: http://imagej.273.s1.nabble.com/imageJ-distributed-computing-tp3686733p3686741.html
Hi,
On Thu, 30 Sep 2010, suendermann wrote:
> > For example, you will never be able to parallelize in general a given
> > macro. But you could provide a way to do the Process>Batch>Macro... in
> > a truly parallel manner, say, on a cluster.
>
> Maybe I don't get the point, but what do you mean with cluster?
Except for some enterprisey marketing types's new definitions, a cluster
is a well defined entity: a couple of computers with identical
architecture and operating system (so-called nodes), controlled by a
master node, which gets sent jobs using a so-called scheduler.
> Just one PC with several cpus or a bigger thing with many nodes each
> with its own cpus? We tried to get it run on a machine with 8 nodes with
> 8 CPUs each and had to find out, that ImageJ is not able to communicate
> over nodes. It stuck within the node and its cpus.
I tried to get over the point in my previous mail: it is _not_ ImageJ! It
is the _plugin_! ImageJ would be happy to run on as many cores as you let
it. But ImageJ is inherently sequential, as it is designed as a desktop
application (which I probably mentioned before).
> So our only solution was to start it on every node separatly and use a
> script to distribute to "jobs" to the nodes.
If you thought that there would be an automatic way to distribute
the work load of a sequential macro magically onto a number of cores, then
I have to disappoint you. It is easy to construct pathological examples
where each step of the macro depends on the previous step, and where each
individual step is not parallelizable. There is simply no way to
parallelize this thing automatically.
> > However. The big problem is that ImageJ was designed as a desktop
> > application, and therefore many plugins require a graphical desktop
> > even if they do not display anything. For example, if you instantiate
> > any java.awt.Dialog on Linux, you need a graphical desktop.
> >
> > We tried to work around this issue in the 'headless' hack in Fiji
> > (basically providing fake Dialog and Menus classes), but fact is: many
> > plugins will fail to run with that hack, because they access more
> > functionality than we provide, e.g. registering callbacks such as
> > constraining the aspect ratio when downsampling. And our fake Dialog
> > class does not provide all required fake methods for that.
>
> Do you plan to improve your fake class or do you search for another
> solution?
I will improve the fake Dialog and Menus classes as I need to. I will also
happily accept improvements others needed to do. But it is a hack, and
every hack shows its limitations if you use it too hard.
The proper solution is a redesign of the plugin mechanism to allow
headless execution (this is underway as far as I understand), but of
course this will only work with plugins written using the new interface.
The old plugins will continue to have the problem.
Hth,
Johannes