java plugin administration advice

[Kenneth Sloan-2]

I posted something similar a while ago and got no response.  Trying again.  

I'm in need of instruction on how to administer shared "utility" Java classes in the context
of ImageJ (and FIJI).  Any advice is welcome.

Our work revolves around the frequent creation of custom Java plugins.  Lately, I have been writing about 1 major plugin per month.  As you might expect, these tend to form "families" of similar (but definitely different) plugins.  Usually, the family resemblence depends on handling specific file formats.  For example, I have a class called "VOL.class" which I'll use as an example.  VOL manages a particular proprietary file format, and this file format is used by many of our custom Java plugins.  It changes infrequently,  but recently went through a major revision (mostly adding features).

I updated "VOL" while writing a new Java plugin.  When I ran the new plugin, it was obvious that it was using the old version of VOL at runtime.  In order to cause it to use the new version, I un-installed the old plugins, propagated the changes, rebuilt the old plugins, and all was well.  But, it's clear to me (now) that, at runtime, ImageJ is finding ONE specific instance of VOL and using it in all plugins.  

Our method for distributing plugins is to create a .jar file which contains everything necessary and using "drag&drop" to install the .jar file.  The intent is that these .jar files are independent (not everyone needs, or wants, all of the plugins).

So...the question is: how can I structure either my source code environment, or the .jar files, or...???
Our previous mindset was that the .jar file for a single plugin was completely independent of anything else.
One thought is to create separate .jar files for each shared class and install these (where?) separately.  This would (slightly) increase the complexity of installing a new plugin (sometimes this is done by customers who are blindly following an installation procedure).  Another is to change the names to avoid conflict (or perhaps use explicit paths to each instance).

I note that we have similar problems with "ij.jar".  Our development environment is 'ant' (and I am loathe to change this).  All plugins are in a "project" folder which includes ij.jar at the top level.  Each plugin has a src directory (and some have a "helpers" directory under that; I have recently eliminated the "helpers" sub-directory - might that have caused my problems?)

So...going down the path of "separate .jar files for shared utilities".  Assume that I separately build the shared classes and create .jar files for each one.  WHERE do they go, and how do I "install" them?

Or...are there other suggestions?  Again, I'd like to stay with 'ant', mostly because I want to spend time writing plugins and not coming up to speed on yet-another-development-environment.  If I had my druthers, I'd still be using 'make'.

--
Kenneth Sloan
[hidden email]
Vision is the art of seeing what is invisible to others.

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Olivier Burri]

Dear Kenneth,

I'm pretty sure you've heard the recommendations to use either Gradle or Maven when developing your plugins, so I'll avoid that :)

In ImageJ/Fiji utility classes reside in the 'jars' folder. But they can be in the plugins folder too.

So in your scenario
Your VOL class would exist in a VOL.jar file in the 'jars' folder.

Then while you develop other plugins that depend on this guy, you would point your IDE (Eclipse?) to this VOL.jar. Same thing for ij.jar, instead of copying it into the project, you simply point to its location. (The Java Build Path in Eclipse and for your ANT file, it would be using the <classpath> tag, if I recall)
Update VOL.class, and you would update the VOL.jar. All your plugins would access the same VOL.class file this way.

The simplest thing to 'administer' this would be for you to have an update site that stores all these utility classes. That way, when a new user arrives, a prerequisite is to activate your update site. If using Fiji, your update site will be pre-selectable on any new installation as long as you filled the list of update sites at http://imagej.net/List_of_update_sites 

You can then give them the actual plugin as you normally did, but everyone has the same 'utility' classes, even if they do not use it.

I just wanted to point out that the idea of plugins, unless you want them to work as an independent Java Application which bundles its own copy of ImageJ, is to avoid repackaging the same things, and avoid these redundancies. I think you're going in the right direction with your new approach!

All the best, hope this helps

Oli

-----Original Message-----
From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of Kenneth Sloan
Sent: Tuesday, February 13, 2018 5:11 PM
To: [hidden email]
Subject: java plugin administration advice

I posted something similar a while ago and got no response.  Trying again.  

I'm in need of instruction on how to administer shared "utility" Java classes in the context of ImageJ (and FIJI).  Any advice is welcome.

Our work revolves around the frequent creation of custom Java plugins.  Lately, I have been writing about 1 major plugin per month.  As you might expect, these tend to form "families" of similar (but definitely different) plugins.  Usually, the family resemblence depends on handling specific file formats.  For example, I have a class called "VOL.class" which I'll use as an example.  VOL manages a particular proprietary file format, and this file format is used by many of our custom Java plugins.  It changes infrequently,  but recently went through a major revision (mostly adding features).

I updated "VOL" while writing a new Java plugin.  When I ran the new plugin, it was obvious that it was using the old version of VOL at runtime.  In order to cause it to use the new version, I un-installed the old plugins, propagated the changes, rebuilt the old plugins, and all was well.  But, it's clear to me (now) that, at runtime, ImageJ is finding ONE specific instance of VOL and using it in all plugins.  

Our method for distributing plugins is to create a .jar file which contains everything necessary and using "drag&drop" to install the .jar file.  The intent is that these .jar files are independent (not everyone needs, or wants, all of the plugins).

So...the question is: how can I structure either my source code environment, or the .jar files, or...???
Our previous mindset was that the .jar file for a single plugin was completely independent of anything else.
One thought is to create separate .jar files for each shared class and install these (where?) separately.  This would (slightly) increase the complexity of installing a new plugin (sometimes this is done by customers who are blindly following an installation procedure).  Another is to change the names to avoid conflict (or perhaps use explicit paths to each instance).

I note that we have similar problems with "ij.jar".  Our development environment is 'ant' (and I am loathe to change this).  All plugins are in a "project" folder which includes ij.jar at the top level.  Each plugin has a src directory (and some have a "helpers" directory under that; I have recently eliminated the "helpers" sub-directory - might that have caused my problems?)

So...going down the path of "separate .jar files for shared utilities".  Assume that I separately build the shared classes and create .jar files for each one.  WHERE do they go, and how do I "install" them?

Or...are there other suggestions?  Again, I'd like to stay with 'ant', mostly because I want to spend time writing plugins and not coming up to speed on yet-another-development-environment.  If I had my druthers, I'd still be using 'make'.

--
Kenneth Sloan
[hidden email]
Vision is the art of seeing what is invisible to others.

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[ctrueden]

Hi Kenneth,

For development, you can use Ivy with Ant to manage dependencies.

http://ant.apache.org/ivy/history/latest-milestone/tutorial/dependence.html

We do not have any examples of developing ImageJ with Ant+Ivy, but you are
welcome to create something along the lines of the example-imagej-command
[1] or example-legacy-plugin [2].

For distribution, you can ship your component JARs using an ImageJ update
site:
  https://imagej.net/Update_Sites

Then users have the freedom to cherry pick which plugins they want, and the
ImageJ Updater will auto-install the needed shared dependencies.

> at runtime, ImageJ is finding ONE specific instance of VOL and using
> it in all plugins.

That is correct. The alternative would be for ImageJ to do more magic with
class loaders (like OSGi does) so that multiple versions of shared
libraries can coexist in the same runtime. I am seriously considering
updating ImageJ2 to work this way [3], but it would be a major change with
substantial downsides.

> Our method for distributing plugins is to create a .jar file which
> contains everything necessary and using "drag&drop" to install the
> .jar file.  The intent is that these .jar files are independent (not
> everyone needs, or wants, all of the plugins).

Yep -- the "uber-JAR" a.k.a. "fat JAR" file. See http://imagej.net/Uber-JAR

If your goal is to minimize "bloat" while benefiting from the advantages
offered by using dependencies, then component JARs (rather than uber-JARs)
are the way to go.

That said, many developers in this community opt for a more pragmatic
middle ground with only one shared dependent library, that they keep
updated to work with all their plugins. For example, Erik Meijering's
ImageScience project [4] consists of several plugins (NeuronJ, TransformJ,
etc.) that all have the core imagescience.jar as a dependency. This
approach keeps the dependency tree simple, but compromises the modularity
-- there may be functions in the single shared library that are not used by
many of the plugins downstream. It is good to think carefully about your
desired balance between modularity and convenience.

> All plugins are in a "project" folder which includes ij.jar at the top
> level.

If you use a dependency management tool like Ivy (or Maven or Gradle), you
can stop committing binaries to your source control -- instead, the build
pulls them down on demand from public repositories (caching them locally,
of course). All lettered versions of ij.jar are published to the Maven
Central repository.

> Assume that I separately build the shared classes and create .jar
> files for each one.

That is how the ImageJ software stack, including the Fiji plugins, are
structured. There is a separate Git repository for each JAR file. Shared
libraries are declared as dependencies. To keep track of which versions
work correctly with which other versions, there is a Bill of Materials
(BOM) [5] -- essentially a "grocery list" of components. Unlike a "single
JAR file" distribution, this modularity gives developers the flexibility to
consume any part of this code as independently as possible from the rest.

> WHERE do they go, and how do I "install" them?

As Oli said, your shared libraries go into jars/ and your plugins go into
plugins/.

(If you develop ImageJ2/SciJava commands, everything can go into jars --
the plugins folder is only needed for ImageJ 1.x.)

Not sure what you mean by 'how do I install them'.

- To install them locally, you can copy the build artifacts manually into
your ImageJ installation folder. (There is a Maven command for doing it
automatically including dependencies, but you said you don't want to use
Maven.)

- To make it easy for users to install them, create an ImageJ update site
and upload them there [6].

See also:
  https://imagej.net/Distribution

> If I had my druthers, I'd still be using 'make'.

Sure, why not?

  <project name="MyProject" default="build">
    <target name="build"><exec executable="make build"/></target>
    <target name="clean"><exec executable="make clean"/></target>
  </project>

Olivier Burri wrote:
> If using Fiji, your update site will be pre-selectable on any new
> installation as long as you filled the list of update sites

As a small point of clarification: this is not a Fiji-specific feature --
it is a feature of the ImageJ Updater, which is part of core ImageJ2. I.e.:
all ImageJ2 installations, not just Fiji-flavored ones, utilize the
built-in list of update sites wiki page [7]. (Not that there are many
vanilla ImageJ2 installations in the wild right now... but once the loose
ends of the Java 8 migration are tidied up, I expect this will change.)

Regards,
Curtis

[1] https://github.com/imagej/example-imagej-command
[2] https://github.com/imagej/example-legacy-plugin
[3] http://forum.imagej.net/t/8393
[4] https://imagej.net/ImageScience, https://github.com/imagescience
[5] https://imagej.net/BOM, https://github.com/scijava/pom-scijava
[6] https://imagej.net/How_to_set_up_and_populate_an_update_site
[7] https://imagej.net/List_of_update_sites

--
Curtis Rueden
LOCI software architect - https://loci.wisc.edu/software
ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden
Did you know ImageJ has a forum? http://forum.imagej.net/


On Tue, Feb 13, 2018 at 10:11 AM, Kenneth Sloan <[hidden email]>
wrote:

> I posted something similar a while ago and got no response.  Trying again.
>
> I'm in need of instruction on how to administer shared "utility" Java
> classes in the context
> of ImageJ (and FIJI).  Any advice is welcome.
>
> Our work revolves around the frequent creation of custom Java plugins.
> Lately, I have been writing about 1 major plugin per month.  As you might
> expect, these tend to form "families" of similar (but definitely different)
> plugins.  Usually, the family resemblence depends on handling specific file
> formats.  For example, I have a class called "VOL.class" which I'll use as
> an example.  VOL manages a particular proprietary file format, and this
> file format is used by many of our custom Java plugins.  It changes
> infrequently,  but recently went through a major revision (mostly adding
> features).
>
> I updated "VOL" while writing a new Java plugin.  When I ran the new
> plugin, it was obvious that it was using the old version of VOL at
> runtime.  In order to cause it to use the new version, I un-installed the
> old plugins, propagated the changes, rebuilt the old plugins, and all was
> well.  But, it's clear to me (now) that, at runtime, ImageJ is finding ONE
> specific instance of VOL and using it in all plugins.
>
> Our method for distributing plugins is to create a .jar file which
> contains everything necessary and using "drag&drop" to install the .jar
> file.  The intent is that these .jar files are independent (not everyone
> needs, or wants, all of the plugins).
>
> So...the question is: how can I structure either my source code
> environment, or the .jar files, or...???
> Our previous mindset was that the .jar file for a single plugin was
> completely independent of anything else.
> One thought is to create separate .jar files for each shared class and
> install these (where?) separately.  This would (slightly) increase the
> complexity of installing a new plugin (sometimes this is done by customers
> who are blindly following an installation procedure).  Another is to change
> the names to avoid conflict (or perhaps use explicit paths to each
> instance).
>
> I note that we have similar problems with "ij.jar".  Our development
> environment is 'ant' (and I am loathe to change this).  All plugins are in
> a "project" folder which includes ij.jar at the top level.  Each plugin has
> a src directory (and some have a "helpers" directory under that; I have
> recently eliminated the "helpers" sub-directory - might that have caused my
> problems?)
>
> So...going down the path of "separate .jar files for shared utilities".
> Assume that I separately build the shared classes and create .jar files for
> each one.  WHERE do they go, and how do I "install" them?
>
> Or...are there other suggestions?  Again, I'd like to stay with 'ant',
> mostly because I want to spend time writing plugins and not coming up to
> speed on yet-another-development-environment.  If I had my druthers, I'd
> still be using 'make'.
>
> --
> Kenneth Sloan
> [hidden email]
> Vision is the art of seeing what is invisible to others.
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html