Parallel image processing using ImageJ in the Cloud

[Jimmy Su]

We recently completed a Phase 1 SBIR project with OSD on the topic of
analytic tools in the Cloud.  To demonstrate our tool's ability to
construct image processing workflow and deploy the generated code to
the Cloud, we took ImageJ and added some Cloud processing capabilities
by using the MapReduce framework.

We added Cloud processing capability to ImageJ by adding a Hadoop
InputFormat to handle image types in HDFS (Hadoop Distributed File
System) and encapsulating ImageJ operations in map and reduce methods.
 This significantly increases ImageJ's throughput in processing
images.  Attached is the running time chart showing processing time
decreases from over 5 hours on two nodes to 15 minutes on 64 nodes on
Amazon EC2.  Are there any interests in the ImageJ community for
parallel processing in the Cloud?  What kind of applications are you
developing that needs ImageJ processing in the Cloud?  We would love
to hear your feedback.  Thank you very much for your time.

Jimmy

[Daniel James White]

Hi Jimmy,

On Apr 6, 2011, at 6:00 AM, IMAGEJ automatic digest system wrote:

> Date:    Mon, 4 Apr 2011 22:37:44 -0700
> From:    Jimmy Su <[hidden email]>
> Subject: Parallel image processing using ImageJ in the Cloud
>
> We recently completed a Phase 1 SBIR project with OSD on the topic of
> analytic tools in the Cloud.  To demonstrate our tool's ability to
> construct image processing workflow and deploy the generated code to
> the Cloud, we took ImageJ and added some Cloud processing capabilities
> by using the MapReduce framework.

Sounds very nice, and is certainly of interest to the imageJ community...
as we are moving towards systems biology approaches where
massively parallel computing is a requirement.

Around here we have a small cluster 100 nodes or so runbning sun grid engine,
and also access to the larger cluster in Dresden's TUD.

We don't have experience with the Amazon cloud systems ... yet...

ImageJ2 will have a better design for running in a headless (no GUI) manner,
which should help a great deal with this kind of stuff.
Many imageJ plugins are tied to a GUI, and cant be run headless...
in the future that will no longer be a limitation for a properly written plugin.

see
http://imagejdev.org/

maybe you might like to get involved?

>
> We added Cloud processing capability to ImageJ by adding a Hadoop
> InputFormat to handle image types in HDFS (Hadoop Distributed File
> System) and encapsulating ImageJ operations in map and reduce methods.
> This significantly increases ImageJ's throughput in processing
> images.  Attached is the running time chart showing processing time
> decreases from over 5 hours on two nodes to 15 minutes on 64 nodes on
> Amazon EC2.  

can you send that attachment to be personally, as thee mailing list ate it.

> Are there any interests in the ImageJ community for
> parallel processing in the Cloud?

Yes, very great interest.

>  What kind of applications are you
> developing that needs ImageJ processing in the Cloud?  

High content screening, many thousands to millions of images
to be processed in pipelines, or even with decisions made on the fly,
classification of phenotypes from images....

Cell profiler is currently better geared to this job than imageJ,
but they play nice together.

> We would love
> to hear your feedback.  Thank you very much for your time.

the ImageJ-dev and Fiji teams are where this stuff is happening...
there are mailing lists and IRC channels where you can chat to
those who know more about it than me....

and this is our invitation to communicate directly with the interested developers
of imageJ2 and Fiji
 (
http://pacific.mpi-cbg.de                Fiji -  is just ImageJ (Batteries Included)
)

cheers

Dan

>
> Jimmy

Dr. Daniel James White BSc. (Hons.) PhD
Senior Microscopist / Image Visualisation, Processing and Analysis
Light Microscopy and Image Processing Facilities
Max Planck Institute of Molecular Cell Biology and Genetics
Pfotenhauerstrasse 108
01307 DRESDEN
Germany

[phone] (German Mobile)
[phone] (Work phone at MPI-CBG)
[phone] (Fax MPI-CBG LMF)

http://www.bioimagexd.net  BioImageXD
http://pacific.mpi-cbg.de                Fiji -  is just ImageJ (Batteries Included)
http://www.chalkie.org.uk                Dan's Homepages
https://ifn.mpi-cbg.de  Dresden Imaging Facility Network
dan (at) chalkie.org.uk
( white (at) mpi-cbg.de )

[dscho]

Hi Jimmy,

On Mon, 4 Apr 2011, Jimmy Su wrote:

> We recently completed a Phase 1 SBIR project with OSD on the topic of
> analytic tools in the Cloud.

I only understood half the words in that sentence, but I'm quite used to
that :-)

> To demonstrate our tool's ability to construct image processing workflow
> and deploy the generated code to the Cloud, we took ImageJ and added
> some Cloud processing capabilities by using the MapReduce framework.

What exactly did you do in terms of image processing? Some Gaussian Blur,
or Find Edges, or some advanced plug-in? From a technical point of view,
there are huge differences there.

> We added Cloud processing capability to ImageJ by adding a Hadoop
> InputFormat to handle image types in HDFS (Hadoop Distributed File
> System) and encapsulating ImageJ operations in map and reduce methods.

That is _very_ interesting. For a long time I have been wanting to play
with Hadoop now.

> This significantly increases ImageJ's throughput in processing
> images.  Attached is the running time chart showing processing time
> decreases from over 5 hours on two nodes to 15 minutes on 64 nodes on
> Amazon EC2.  Are there any interests in the ImageJ community for
> parallel processing in the Cloud?  What kind of applications are you
> developing that needs ImageJ processing in the Cloud?  We would love
> to hear your feedback.

Two feedbacks from my side:

1) fantastic!
2) where can I get it?

Ciao,
Johannes

[Dean Kossives]

Every now and then something comes along that catches my eye. A
reduction in processing time from 5 hours to 15 min is great.

1) can you get the processing time down to under 1 minute?

Dean P. Kossives
Principal Engineer
Clear Align
2550 Boulevard of the Generals, Suite 280
Eagleville, PA 19403
phone [phone] X185
fax  [phone]
mailto:[hidden email]
www.clearalign.com
 
Clear Align is certified as a SB, 8(a) SDB & WOSB.
See us at SPIE Defense, Security, and Sensing (Booth 1017), April 25-29,
2011

This message and any attachments are solely for the use of the
individual or entity to which it is addressed and may contain
information that is privileged or confidential.  If you are not the
intended recipient, any disclosure, use or distribution of the
information contained herein is prohibited. If you have received this
communication in error, please notify the sender by reply e-mail and
immediately delete this message and any attachments.  In the event this
document(s) contains technical data within the definition of the
International Traffic in Arms Regulations, it is subject to the export
control laws of the U.S. Government.  Transfer of this data by any means
to a foreign person, whether in the United States or abroad, without an
export license or other approval from the U.S. Department of State, is
prohibited.

 P Please consider your environmental responsibilities before printing
this e-mail


-----Original Message-----
From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of
Johannes Schindelin
Sent: Wednesday, April 06, 2011 6:25 AM
To: [hidden email]
Subject: Re: Parallel image processing using ImageJ in the Cloud

Hi Jimmy,

On Mon, 4 Apr 2011, Jimmy Su wrote:

> We recently completed a Phase 1 SBIR project with OSD on the topic of
> analytic tools in the Cloud.

I only understood half the words in that sentence, but I'm quite used to

that :-)

> To demonstrate our tool's ability to construct image processing
workflow
> and deploy the generated code to the Cloud, we took ImageJ and added
> some Cloud processing capabilities by using the MapReduce framework.

What exactly did you do in terms of image processing? Some Gaussian
Blur,
or Find Edges, or some advanced plug-in? From a technical point of view,

there are huge differences there.

> We added Cloud processing capability to ImageJ by adding a Hadoop
> InputFormat to handle image types in HDFS (Hadoop Distributed File
> System) and encapsulating ImageJ operations in map and reduce methods.

That is _very_ interesting. For a long time I have been wanting to play
with Hadoop now.

> This significantly increases ImageJ's throughput in processing
> images.  Attached is the running time chart showing processing time
> decreases from over 5 hours on two nodes to 15 minutes on 64 nodes on
> Amazon EC2.  Are there any interests in the ImageJ community for
> parallel processing in the Cloud?  What kind of applications are you
> developing that needs ImageJ processing in the Cloud?  We would love
> to hear your feedback.

Two feedbacks from my side:

1) fantastic!
2) where can I get it?

Ciao,
Johannes

[Gary Sellani]

I should point out here that 8 core CPUs (AMD bulldozer) are due in June. A dual CPU bulldozer would be 16 cores and not depend on the cloud.

Basically what is powering the speed increase is the parallel processing, not the cloud itself.

-----Original Message-----
From: Dean Kossives <[hidden email]>
Sender: ImageJ Interest Group <[hidden email]>
Date: Wed, 6 Apr 2011 16:18:23
To: <[hidden email]>
Reply-To: ImageJ Interest Group <[hidden email]>
Subject: Re: Parallel image processing using ImageJ in the Cloud

Every now and then something comes along that catches my eye. A
reduction in processing time from 5 hours to 15 min is great.

1) can you get the processing time down to under 1 minute?

Dean P. Kossives
Principal Engineer
Clear Align
2550 Boulevard of the Generals, Suite 280
Eagleville, PA 19403
phone [phone] X185
fax  [phone]
mailto:[hidden email]
www.clearalign.com
 
Clear Align is certified as a SB, 8(a) SDB & WOSB.
See us at SPIE Defense, Security, and Sensing (Booth 1017), April 25-29,
2011

This message and any attachments are solely for the use of the
individual or entity to which it is addressed and may contain
information that is privileged or confidential.  If you are not the
intended recipient, any disclosure, use or distribution of the
information contained herein is prohibited. If you have received this
communication in error, please notify the sender by reply e-mail and
immediately delete this message and any attachments.  In the event this
document(s) contains technical data within the definition of the
International Traffic in Arms Regulations, it is subject to the export
control laws of the U.S. Government.  Transfer of this data by any means
to a foreign person, whether in the United States or abroad, without an
export license or other approval from the U.S. Department of State, is
prohibited.

 P Please consider your environmental responsibilities before printing
this e-mail


-----Original Message-----
From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of
Johannes Schindelin
Sent: Wednesday, April 06, 2011 6:25 AM
To: [hidden email]
Subject: Re: Parallel image processing using ImageJ in the Cloud

Hi Jimmy,

On Mon, 4 Apr 2011, Jimmy Su wrote:

> We recently completed a Phase 1 SBIR project with OSD on the topic of
> analytic tools in the Cloud.

I only understood half the words in that sentence, but I'm quite used to

that :-)

> To demonstrate our tool's ability to construct image processing
workflow
> and deploy the generated code to the Cloud, we took ImageJ and added
> some Cloud processing capabilities by using the MapReduce framework.

What exactly did you do in terms of image processing? Some Gaussian
Blur,
or Find Edges, or some advanced plug-in? From a technical point of view,

there are huge differences there.

> We added Cloud processing capability to ImageJ by adding a Hadoop
> InputFormat to handle image types in HDFS (Hadoop Distributed File
> System) and encapsulating ImageJ operations in map and reduce methods.

That is_very_ interesting. For a long time I have been wanting to play
with Hadoop now.

> This significantly increases ImageJ's throughput in processing
> images.  Attached is the running time chart showing processing time
> decreases from over 5 hours on two nodes to 15 minutes on 64 nodes on
> Amazon EC2.  Are there any interests in the ImageJ community for
> parallel processing in the Cloud?  What kind of applications are you
> developing that needs ImageJ processing in the Cloud?  We would love
> to hear your feedback.

Two feedbacks from my side:

1) fantastic!
2) where can I get it?

Ciao,
Johannes

[Jimmy Su]

On Wed, Apr 6, 2011 at 3:17 AM, Daniel James White <[hidden email]> wrote:

> Hi Jimmy,
>
> On Apr 6, 2011, at 6:00 AM, IMAGEJ automatic digest system wrote:
>
>> Date:    Mon, 4 Apr 2011 22:37:44 -0700
>> From:    Jimmy Su <[hidden email]>
>> Subject: Parallel image processing using ImageJ in the Cloud
>>
>> We recently completed a Phase 1 SBIR project with OSD on the topic of
>> analytic tools in the Cloud.  To demonstrate our tool's ability to
>> construct image processing workflow and deploy the generated code to
>> the Cloud, we took ImageJ and added some Cloud processing capabilities
>> by using the MapReduce framework.
>
> Sounds very nice, and is certainly of interest to the imageJ community...
> as we are moving towards systems biology approaches where
> massively parallel computing is a requirement.
>
> Around here we have a small cluster 100 nodes or so runbning sun grid engine,
> and also access to the larger cluster in Dresden's TUD.
>
> We don't have experience with the Amazon cloud systems ... yet...
>
> ImageJ2 will have a better design for running in a headless (no GUI) manner,
> which should help a great deal with this kind of stuff.
> Many imageJ plugins are tied to a GUI, and cant be run headless...
> in the future that will no longer be a limitation for a properly written plugin.
>

We certainly run into this problem in our project.  We can't pop up a
window when we are running on many machines concurrently in the Cloud.
 We ended up using the ImageProcessor type and performed some
operations such as shifting and histogram on the images.  We look
forward to learning more about this domain as we get involved in the
project.

> see
> http://imagejdev.org/
>
> maybe you might like to get involved?
>

Yes, definitely.

Jimmy

>>
>> We added Cloud processing capability to ImageJ by adding a Hadoop
>> InputFormat to handle image types in HDFS (Hadoop Distributed File
>> System) and encapsulating ImageJ operations in map and reduce methods.
>> This significantly increases ImageJ's throughput in processing
>> images.  Attached is the running time chart showing processing time
>> decreases from over 5 hours on two nodes to 15 minutes on 64 nodes on
>> Amazon EC2.
>
> can you send that attachment to be personally, as thee mailing list ate it.
>
>> Are there any interests in the ImageJ community for
>> parallel processing in the Cloud?
>
> Yes, very great interest.
>
>>  What kind of applications are you
>> developing that needs ImageJ processing in the Cloud?
>
> High content screening, many thousands to millions of images
> to be processed in pipelines, or even with decisions made on the fly,
> classification of phenotypes from images....
>
> Cell profiler is currently better geared to this job than imageJ,
> but they play nice together.
>
>> We would love
>> to hear your feedback.  Thank you very much for your time.
>
> the ImageJ-dev and Fiji teams are where this stuff is happening...
> there are mailing lists and IRC channels where you can chat to
> those who know more about it than me....
>
> and this is our invitation to communicate directly with the interested developers
> of imageJ2 and Fiji
>  (
> http://pacific.mpi-cbg.de               Fiji -  is just ImageJ (Batteries Included)
> )
>
> cheers
>
> Dan
>
>>
>> Jimmy
>
> Dr. Daniel James White BSc. (Hons.) PhD
> Senior Microscopist / Image Visualisation, Processing and Analysis
> Light Microscopy and Image Processing Facilities
> Max Planck Institute of Molecular Cell Biology and Genetics
> Pfotenhauerstrasse 108
> 01307 DRESDEN
> Germany
>
> [phone] (German Mobile)
> [phone] (Work phone at MPI-CBG)
> [phone] (Fax MPI-CBG LMF)
>
> http://www.bioimagexd.net       BioImageXD
> http://pacific.mpi-cbg.de               Fiji -  is just ImageJ (Batteries Included)
> http://www.chalkie.org.uk               Dan's Homepages
> https://ifn.mpi-cbg.de                  Dresden Imaging Facility Network
> dan (at) chalkie.org.uk
> ( white (at) mpi-cbg.de )
>
>
>
>
>
>
>
>
>
>
>
>

[Jimmy Su]

On Wed, Apr 6, 2011 at 3:24 AM, Johannes Schindelin
<[hidden email]> wrote:

> Hi Jimmy,
>
> On Mon, 4 Apr 2011, Jimmy Su wrote:
>
>> We recently completed a Phase 1 SBIR project with OSD on the topic of
>> analytic tools in the Cloud.
>
> I only understood half the words in that sentence, but I'm quite used to
> that :-)
>
>> To demonstrate our tool's ability to construct image processing workflow
>> and deploy the generated code to the Cloud, we took ImageJ and added
>> some Cloud processing capabilities by using the MapReduce framework.
>
> What exactly did you do in terms of image processing? Some Gaussian Blur,
> or Find Edges, or some advanced plug-in? From a technical point of view,
> there are huge differences there.
>

We used some operations from the ImageProcessor type such as shifting,
histogram, and edge detection and applied them to images.  The
parallelism comes from applying the same set of operations to many
images at the same time.  In parallel computing, this would be
classified as data parallelism.  We lack the expertise in image
processing, so the set of operations we used may not actually
represent correct ImageJ usage.  We look forward to learning more
about this domain as we get involved.

>> We added Cloud processing capability to ImageJ by adding a Hadoop
>> InputFormat to handle image types in HDFS (Hadoop Distributed File
>> System) and encapsulating ImageJ operations in map and reduce methods.
>
> That is _very_ interesting. For a long time I have been wanting to play
> with Hadoop now.
>
>> This significantly increases ImageJ's throughput in processing
>> images.  Attached is the running time chart showing processing time
>> decreases from over 5 hours on two nodes to 15 minutes on 64 nodes on
>> Amazon EC2.  Are there any interests in the ImageJ community for
>> parallel processing in the Cloud?  What kind of applications are you
>> developing that needs ImageJ processing in the Cloud?  We would love
>> to hear your feedback.
>
> Two feedbacks from my side:
>
> 1) fantastic!

Thank you.

> 2) where can I get it?
>

At its current state, it is not likely to be useful to the whole
ImageJ community yet, as we only have a few operations from
ImageProcessor running in the Cloud.  We would need to learn more
about which operations are most likely to benefit from
parallelization, and add the right interface for using them.  We will
definitely contribute our modifications back to ImageJ source tree
when that is done.

Jimmy

> Ciao,
> Johannes
>
>

[Jimmy Su]

On Wed, Apr 6, 2011 at 1:18 PM, Dean Kossives
<[hidden email]> wrote:
> Every now and then something comes along that catches my eye. A
> reduction in processing time from 5 hours to 15 min is great.
>
> 1) can you get the processing time down to under 1 minute?
>

The short answer is yes.  It would require throwing more processors at
the problem.  This would do the trick if applying this set of
operations to a single image would take under one minute.  Otherwise,
we would need to further parallelize the problem by using multiple
processors/cores on each image.

Jimmy

> Dean P. Kossives
> Principal Engineer
> Clear Align
> 2550 Boulevard of the Generals, Suite 280
> Eagleville, PA 19403
> phone [phone] X185
> fax  [phone]
> mailto:[hidden email]
> www.clearalign.com
>
> Clear Align is certified as a SB, 8(a) SDB & WOSB.
> See us at SPIE Defense, Security, and Sensing (Booth 1017), April 25-29,
> 2011
>
> This message and any attachments are solely for the use of the
> individual or entity to which it is addressed and may contain
> information that is privileged or confidential.  If you are not the
> intended recipient, any disclosure, use or distribution of the
> information contained herein is prohibited. If you have received this
> communication in error, please notify the sender by reply e-mail and
> immediately delete this message and any attachments.  In the event this
> document(s) contains technical data within the definition of the
> International Traffic in Arms Regulations, it is subject to the export
> control laws of the U.S. Government.  Transfer of this data by any means
> to a foreign person, whether in the United States or abroad, without an
> export license or other approval from the U.S. Department of State, is
> prohibited.
>
>  P Please consider your environmental responsibilities before printing
> this e-mail
>
>
> -----Original Message-----
> From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of
> Johannes Schindelin
> Sent: Wednesday, April 06, 2011 6:25 AM
> To: [hidden email]
> Subject: Re: Parallel image processing using ImageJ in the Cloud
>
> Hi Jimmy,
>
> On Mon, 4 Apr 2011, Jimmy Su wrote:
>
>> We recently completed a Phase 1 SBIR project with OSD on the topic of
>> analytic tools in the Cloud.
>
> I only understood half the words in that sentence, but I'm quite used to
>
> that :-)
>
>> To demonstrate our tool's ability to construct image processing
> workflow
>> and deploy the generated code to the Cloud, we took ImageJ and added
>> some Cloud processing capabilities by using the MapReduce framework.
>
> What exactly did you do in terms of image processing? Some Gaussian
> Blur,
> or Find Edges, or some advanced plug-in? From a technical point of view,
>
> there are huge differences there.
>
>> We added Cloud processing capability to ImageJ by adding a Hadoop
>> InputFormat to handle image types in HDFS (Hadoop Distributed File
>> System) and encapsulating ImageJ operations in map and reduce methods.
>
> That is _very_ interesting. For a long time I have been wanting to play
> with Hadoop now.
>
>> This significantly increases ImageJ's throughput in processing
>> images.  Attached is the running time chart showing processing time
>> decreases from over 5 hours on two nodes to 15 minutes on 64 nodes on
>> Amazon EC2.  Are there any interests in the ImageJ community for
>> parallel processing in the Cloud?  What kind of applications are you
>> developing that needs ImageJ processing in the Cloud?  We would love
>> to hear your feedback.
>
> Two feedbacks from my side:
>
> 1) fantastic!
> 2) where can I get it?
>
> Ciao,
> Johannes
>

[Jimmy Su]

On Wed, Apr 6, 2011 at 2:00 PM, Gary Sellani <[hidden email]> wrote:
> I should point out here that 8 core CPUs (AMD bulldozer) are due in June. A dual CPU bulldozer would be 16 cores and not depend on the cloud.
>
> Basically what is powering the speed increase is the parallel processing, not the cloud itself.
>

Gary's point is right on.  The Cloud is just another parallel
computing platform.  Hadoop implements the MapReduce parallel
processing framework, which has been used successfully in several
domains.

Sometimes a problem is not CPU bound.  It may be limited by the amount
of memory or disk on a single machine.  In those cases, having more
cores on a single processor is not going to help.  We ran into this
problem when we were parallelizing a decision tree training algorithm,
where it requires the training samples to be in memory.

Jimmy

> -----Original Message-----
> From: Dean Kossives <[hidden email]>
> Sender: ImageJ Interest Group <[hidden email]>
> Date: Wed, 6 Apr 2011 16:18:23
> To: <[hidden email]>
> Reply-To: ImageJ Interest Group <[hidden email]>
> Subject: Re: Parallel image processing using ImageJ in the Cloud
>
> Every now and then something comes along that catches my eye. A
> reduction in processing time from 5 hours to 15 min is great.
>
> 1) can you get the processing time down to under 1 minute?
>
> Dean P. Kossives
> Principal Engineer
> Clear Align
> 2550 Boulevard of the Generals, Suite 280
> Eagleville, PA 19403
> phone [phone] X185
> fax  [phone]
> mailto:[hidden email]
> www.clearalign.com
>
> Clear Align is certified as a SB, 8(a) SDB & WOSB.
> See us at SPIE Defense, Security, and Sensing (Booth 1017), April 25-29,
> 2011
>
> This message and any attachments are solely for the use of the
> individual or entity to which it is addressed and may contain
> information that is privileged or confidential.  If you are not the
> intended recipient, any disclosure, use or distribution of the
> information contained herein is prohibited. If you have received this
> communication in error, please notify the sender by reply e-mail and
> immediately delete this message and any attachments.  In the event this
> document(s) contains technical data within the definition of the
> International Traffic in Arms Regulations, it is subject to the export
> control laws of the U.S. Government.  Transfer of this data by any means
> to a foreign person, whether in the United States or abroad, without an
> export license or other approval from the U.S. Department of State, is
> prohibited.
>
>  P Please consider your environmental responsibilities before printing
> this e-mail
>
>
> -----Original Message-----
> From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of
> Johannes Schindelin
> Sent: Wednesday, April 06, 2011 6:25 AM
> To: [hidden email]
> Subject: Re: Parallel image processing using ImageJ in the Cloud
>
> Hi Jimmy,
>
> On Mon, 4 Apr 2011, Jimmy Su wrote:
>
>> We recently completed a Phase 1 SBIR project with OSD on the topic of
>> analytic tools in the Cloud.
>
> I only understood half the words in that sentence, but I'm quite used to
>
> that :-)
>
>> To demonstrate our tool's ability to construct image processing
> workflow
>> and deploy the generated code to the Cloud, we took ImageJ and added
>> some Cloud processing capabilities by using the MapReduce framework.
>
> What exactly did you do in terms of image processing? Some Gaussian
> Blur,
> or Find Edges, or some advanced plug-in? From a technical point of view,
>
> there are huge differences there.
>
>> We added Cloud processing capability to ImageJ by adding a Hadoop
>> InputFormat to handle image types in HDFS (Hadoop Distributed File
>> System) and encapsulating ImageJ operations in map and reduce methods.
>
> That is_very_ interesting. For a long time I have been wanting to play
> with Hadoop now.
>
>> This significantly increases ImageJ's throughput in processing
>> images.  Attached is the running time chart showing processing time
>> decreases from over 5 hours on two nodes to 15 minutes on 64 nodes on
>> Amazon EC2.  Are there any interests in the ImageJ community for
>> parallel processing in the Cloud?  What kind of applications are you
>> developing that needs ImageJ processing in the Cloud?  We would love
>> to hear your feedback.
>
> Two feedbacks from my side:
>
> 1) fantastic!
> 2) where can I get it?
>
> Ciao,
> Johannes
>

[vtali]

We have done alfa implementation of server-side execution for ImageJ on Simagis Live which is our platform for building image analysis web applications.

Main purpose of our implementation was to use ImageJ plugins/macros for Whole Slide Imaging so we run it as muliti-stream aperture processor for distributed tile arrays under dispatcher process. But it also works for distributed parallel processing of independent regular-sized images on server architectures including Amazon AWS Cloud.

In our assessment current implementation of ImageJ was not suitable for Amazon Map/Reduce deployment. I would be very interested to see if somebody could successfully deploy it for aperture processing on Amazon Map/Reduce. Or perhaps, point me to the next generation of ImageJ code that would run on Map/Reduce. We have the rest of the architecture (buckets, EC2 EBS disk RAIDs, turbo upload, web interface, process management etc.) and I would really like to give ImageJ a try on Map/Reduce.

On the other side, if anyone wants to get involved in our implementation, send me your info (Name, email) and I will set you up with account on development server so you can try your ImageJ apps on Amazon EC2 Instances.

Vitali

Smart Imaging Technologies
[phone]
http://live.simagis.com

[Vitali Khvatkov-2]

In out tests for parallel image processing with ImageJ on Amazon Could (as
well as on dedicated servers)  major performance hurdles were:
- Overloading HDD I/O with memory swapping (so we had to do some trickery
with RAID0 stuctures using Amazone EBS drives)
-  High process overhead (say 10% load by plugin code and 90% by ImageJ
starting and loading all its goodies even if we dont need any)
- Luck of code execution process control within ImageJ , mentioned by other
posters. To do "Reduce" step we need to known when process is done

NOTE: Amazon (and other clouds we seen), have high latency storage (even EBS
drives not to mention buckets) so this fact has to be considered.

Vitali.
Smart Imaging Technologies
713-589-3500
live.simagis.com

On Thu, Apr 7, 2011 at 10:30 AM, Jimmy Su <[hidden email]> wrote:

> On Wed, Apr 6, 2011 at 2:00 PM, Gary Sellani <[hidden email]> wrote:
> > I should point out here that 8 core CPUs (AMD bulldozer) are due in June.
> A dual CPU bulldozer would be 16 cores and not depend on the cloud.
> >
> > Basically what is powering the speed increase is the parallel processing,
> not the cloud itself.
> >
>
> Gary's point is right on.  The Cloud is just another parallel
> computing platform.  Hadoop implements the MapReduce parallel
> processing framework, which has been used successfully in several
> domains.
>
> Sometimes a problem is not CPU bound.  It may be limited by the amount
> of memory or disk on a single machine.  In those cases, having more
> cores on a single processor is not going to help.  We ran into this
> problem when we were parallelizing a decision tree training algorithm,
> where it requires the training samples to be in memory.
>
> Jimmy
>
> > -----Original Message-----
> > From: Dean Kossives <[hidden email]>
> > Sender: ImageJ Interest Group <[hidden email]>
> > Date: Wed, 6 Apr 2011 16:18:23
> > To: <[hidden email]>
> > Reply-To: ImageJ Interest Group <[hidden email]>
> > Subject: Re: Parallel image processing using ImageJ in the Cloud
> >
> > Every now and then something comes along that catches my eye. A
> > reduction in processing time from 5 hours to 15 min is great.
> >
> > 1) can you get the processing time down to under 1 minute?
> >
> > Dean P. Kossives
> > Principal Engineer
> > Clear Align
> > 2550 Boulevard of the Generals, Suite 280
> > Eagleville, PA 19403
> > phone [phone] X185
> > fax  [phone]
> > mailto:[hidden email]
> > www.clearalign.com
> >
> > Clear Align is certified as a SB, 8(a) SDB & WOSB.
> > See us at SPIE Defense, Security, and Sensing (Booth 1017), April 25-29,
> > 2011
> >
> > This message and any attachments are solely for the use of the
> > individual or entity to which it is addressed and may contain
> > information that is privileged or confidential.  If you are not the
> > intended recipient, any disclosure, use or distribution of the
> > information contained herein is prohibited. If you have received this
> > communication in error, please notify the sender by reply e-mail and
> > immediately delete this message and any attachments.  In the event this
> > document(s) contains technical data within the definition of the
> > International Traffic in Arms Regulations, it is subject to the export
> > control laws of the U.S. Government.  Transfer of this data by any means
> > to a foreign person, whether in the United States or abroad, without an
> > export license or other approval from the U.S. Department of State, is
> > prohibited.
> >
> >  P Please consider your environmental responsibilities before printing
> > this e-mail
> >
> >
> > -----Original Message-----
> > From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of
> > Johannes Schindelin
> > Sent: Wednesday, April 06, 2011 6:25 AM
> > To: [hidden email]
> > Subject: Re: Parallel image processing using ImageJ in the Cloud
> >
> > Hi Jimmy,
> >
> > On Mon, 4 Apr 2011, Jimmy Su wrote:
> >
> >> We recently completed a Phase 1 SBIR project with OSD on the topic of
> >> analytic tools in the Cloud.
> >
> > I only understood half the words in that sentence, but I'm quite used to
> >
> > that :-)
> >
> >> To demonstrate our tool's ability to construct image processing
> > workflow
> >> and deploy the generated code to the Cloud, we took ImageJ and added
> >> some Cloud processing capabilities by using the MapReduce framework.
> >
> > What exactly did you do in terms of image processing? Some Gaussian
> > Blur,
> > or Find Edges, or some advanced plug-in? From a technical point of view,
> >
> > there are huge differences there.
> >
> >> We added Cloud processing capability to ImageJ by adding a Hadoop
> >> InputFormat to handle image types in HDFS (Hadoop Distributed File
> >> System) and encapsulating ImageJ operations in map and reduce methods.
> >
> > That is_very_ interesting. For a long time I have been wanting to play
> > with Hadoop now.
> >
> >> This significantly increases ImageJ's throughput in processing
> >> images.  Attached is the running time chart showing processing time
> >> decreases from over 5 hours on two nodes to 15 minutes on 64 nodes on
> >> Amazon EC2.  Are there any interests in the ImageJ community for
> >> parallel processing in the Cloud?  What kind of applications are you
> >> developing that needs ImageJ processing in the Cloud?  We would love
> >> to hear your feedback.
> >
> > Two feedbacks from my side:
> >
> > 1) fantastic!
> > 2) where can I get it?
> >
> > Ciao,
> > Johannes
> >
>

--
Vitali Khvatkov
Smart Imaging Technologies Co.

[dscho]

Hi,

On Thu, 7 Apr 2011, Jimmy Su wrote:

> On Wed, Apr 6, 2011 at 3:24 AM, Johannes Schindelin
> <[hidden email]> wrote:
>
> > 2) where can I get it?
>
> At its current state, it is not likely to be useful to the whole ImageJ
> community yet, as we only have a few operations from ImageProcessor
> running in the Cloud.  We would need to learn more about which
> operations are most likely to benefit from parallelization, and add the
> right interface for using them.  We will definitely contribute our
> modifications back to ImageJ source tree when that is done.

Please note that "not likely to be useful to the whole ImageJ community"
implies all the developers who could start from your code base and extend
it to do something actually useful for image processing if you provided
that code base.

Ciao,
Johannes

[Antoinette]

Image processing tool within web asp.net application is developed by using asp.net and javascript controls without taking any memory space and zero-footprint. It can fullfill most of the commonly used image manipulating requirements, such as image rotating, image scaling, image cropping, and many more.