concepts for remote processing (render farm, job management)

[Rainer M. Engel]

Hello everyone,

I read a presentation from Volker Bäcker about Remote ImageJ.
Are there advances done, which can be shared?

Of course one can write scripts or macros to automate fantastic
operations possible with ImageJ, but I'm currently looking for a way to
automate the distribution of calculations between ImageJ clients.

Would be nice to hear some tech infos on how this or related things are
done around the globe.

If there isn't a way to do this in different scales, what would you like
to have? What is your workflow and what would make sense to ease bigger
calculation demands on your side?

Example:
I have 20 jobs that would consume a lot of automatic calculation and I
have 5 clients with ImageJ running.
The jobs would take 5 days on one multicore system to complete but the
calculation amount differs from job to job so that I would get idle
times when assigning these jobs to my five clients manually. I would
always have to check the clients and assign new jobs by hand.

Any thoughts are very much appreciated..


Regards,
Rainer


--
Rainer M. Engel, Dipl. Digital Artist
scientific|Media GbR
Pichelsdorfer Str. 143
13595 Berlin

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Volker Baecker]

Hello,
here the Remote ImageJ has mostly been sleeping, waiting for our Image
database project to advance.
The code is available in the svn from the redmine at:

http://dev.mri.cnrs.fr/projects/remote-imagej

If someone wants to try it please note that there are no security
considerations taken into account for the time being (i.e. make sure the
ports you use are closed from outside of your local network)

Server and client can communicate synchronously or asynchronously, the
server has a job queue.
 
To answer your question: What I would want is that users of our facility
use the pcs of the facility to run batch jobs on dedicated server
machines either from the pcs using ImageJ or from a web-interface. In
our experience for our current usage the most simple scheduling is good
enough.

Best regards,
Volker Baecker

On 19/03/13 12:29, Rainer M. Engel wrote:

> Hello everyone,
>
> I read a presentation from Volker Bäcker about Remote ImageJ.
> Are there advances done, which can be shared?
>
> Of course one can write scripts or macros to automate fantastic
> operations possible with ImageJ, but I'm currently looking for a way to
> automate the distribution of calculations between ImageJ clients.
>
> Would be nice to hear some tech infos on how this or related things are
> done around the globe.
>
> If there isn't a way to do this in different scales, what would you like
> to have? What is your workflow and what would make sense to ease bigger
> calculation demands on your side?
>
> Example:
> I have 20 jobs that would consume a lot of automatic calculation and I
> have 5 clients with ImageJ running.
> The jobs would take 5 days on one multicore system to complete but the
> calculation amount differs from job to job so that I would get idle
> times when assigning these jobs to my five clients manually. I would
> always have to check the clients and assign new jobs by hand.
>
> Any thoughts are very much appreciated..
>
>
> Regards,
> Rainer
>
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Tobias Kloepper]

Hi Rainer,
I am not sure if this is interesting for you. I use Gridgain for my
parallel work in Java. The current version is free for non-profit use
and older versions are available via LGPL.

For me the main challenge is the plugin structure in ImageJ 1 e.g. the
practice of not using packages for the plugins. I think this will be
changed in ImageJ 2.

I mainly use my own plugins and just transfer whatever else I need into
extra classes. This generates a bit of overhead, but for me it is worth
the extra work. The scalability of Gridgain is very good and it is easy
to learn and use.

I am happy to discuss this in more details if this is interesting for you.
Cheers,

Tobias

> Hello everyone,
>
> I read a presentation from Volker Bäcker about Remote ImageJ.
> Are there advances done, which can be shared?
>
> Of course one can write scripts or macros to automate fantastic
> operations possible with ImageJ, but I'm currently looking for a way to
> automate the distribution of calculations between ImageJ clients.
>
> Would be nice to hear some tech infos on how this or related things are
> done around the globe.
>
> If there isn't a way to do this in different scales, what would you like
> to have? What is your workflow and what would make sense to ease bigger
> calculation demands on your side?
>
> Example:
> I have 20 jobs that would consume a lot of automatic calculation and I
> have 5 clients with ImageJ running.
> The jobs would take 5 days on one multicore system to complete but the
> calculation amount differs from job to job so that I would get idle
> times when assigning these jobs to my five clients manually. I would
> always have to check the clients and assign new jobs by hand.
>
> Any thoughts are very much appreciated..
>
>
> Regards,
> Rainer
>
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Rainer M. Engel]

Thank you Volker and Tobias for providing this information.

@Volker
Probably "the most simple scheduling is good enough" as you said. I'm
not sure what kind of concept we will use here exactly, but I'll have a
look at remote imageJ. Thank you for that link.

@Tobias
That reads interesting. Do I understand it correctly that the main
thought of Gridgain is the use of RAM prior to spinning disks. So a job
that would take very long at one pc is distributed and done parallel in
smaller chunks on many pcs in their RAM?

We use expansion cards like Revo Drive which are nice too. But I think
the main speed boost comes from the point of working parallel. First
read the big stack, then do this, then that... is slower than doing it
parallel. Is that correct?


I'm not sure how fast distinct features will find their way to ImageJ2
but I guess that the development of modern multi-core architectures will
have its visible impact.


Like David Hasselhoff sung at the Eastside Gallery Berlin (some days ago)..

".. still the search goes on .. " ;-)

Regards,
Rainer







Am 19.03.2013 12:29, schrieb Rainer M. Engel:

> Hello everyone,
>
> I read a presentation from Volker Bäcker about Remote ImageJ.
> Are there advances done, which can be shared?
>
> Of course one can write scripts or macros to automate fantastic
> operations possible with ImageJ, but I'm currently looking for a way to
> automate the distribution of calculations between ImageJ clients.
>
> Would be nice to hear some tech infos on how this or related things are
> done around the globe.
>
> If there isn't a way to do this in different scales, what would you like
> to have? What is your workflow and what would make sense to ease bigger
> calculation demands on your side?
>
> Example:
> I have 20 jobs that would consume a lot of automatic calculation and I
> have 5 clients with ImageJ running.
> The jobs would take 5 days on one multicore system to complete but the
> calculation amount differs from job to job so that I would get idle
> times when assigning these jobs to my five clients manually. I would
> always have to check the clients and assign new jobs by hand.
>
> Any thoughts are very much appreciated..
>
>
> Regards,
> Rainer
>
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[dscho]

Hi Volker,

On Tue, 19 Mar 2013, Volker Baecker wrote:

> http://dev.mri.cnrs.fr/projects/remote-imagej
>
> If someone wants to try it please note that there are no security
> considerations taken into account for the time being (i.e. make sure the
> ports you use are closed from outside of your local network)

Larry Lindsey started something with security considerations in mind:

        http://fiji.sc/Fiji_Archipelago

I Cc'ed him in case you want to work together on a unified framework.
Larry, the full discussion can be seen here:

        http://thread.gmane.org/gmane.comp.java.imagej/27923/

Ciao,
Johannes

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html