request for guidance on managing suites of plugins

[Kenneth Sloan-2]

What started l as a small, focussed collection of FIJI plugins has exploded into a collection that is beginning to be a problem to manage.  I'm hoping someone here can point me at suggestions (and brief tutorial material) on current "best practices" for developing collections of loosely related FIJI plugins which share multiple mid-level Java classes.

Constraints:

a) about 50 (and growing) FIJI plugins, all written in Java
b) distributed as .jar files, usually not including source code
c) quasi-independent - perhaps 5 different classes of "customers", each with their own mix
d) many "customers" lack the ability to build from source (and many do not receive source)
e) a growing number of shared classes - these can be grouped into a small number of clusters.
f) Some plugins have a lifespan of 10 years or more...others are one-shot throwaways.
g) development on Mac - production use on varied platforms

Current practice:

a) I use 'ant', as set up by grad students, long ago.  This has been convenient, but I'm not all that fluent in, or committed to, 'ant'.  For a new plugin, I end up robotically editing 2 files (build.xml, plugins.config) and it "just works".  Perhaps my desired changes can be implemented in 'ant' - but I get the feeling that 'ant' is outdated (and a pain to install on some OS).  The pain involved in modifying the boilerplate 'ant' files that I use now is probably no less than simply starting over with something "better".
b) I compile each plugin separately, outside ImageJ, and create a .jar file.  There are issues here with matching the Java level to that used by ImageJ, and also providing a pointer to "ij.jar".  The latter is achieved by (periodically, when there's an observed problem) copying a new version of "ij.jar" to the development area and rebuilding everything.  The idea was to be independent of the location of "ij.jar" on the development machine - but this causes minor bobbles every so often when the API of our copy of "ij.jar" does not match the actual installed "ij.jar" when the plugin runs.  I could live with pointing 'ant' at the canonical location of "ij.jar" on *my* machine (rather than making a copy).
c) each plugin is "self-contained" - they are delivered as independent .jar files.  Users install the .jar files (typically drag&drop) and it "just works".  This is the practice that may need some change - in particular, the handling of shared classes.

The problems:
a) maintaining a copy of "ij.jar"
b) shared classes

It occurs to me (duh!) that it makes sense to create separate .jar files for each cluster of shared classes.  I have not done that yet out of  consideration for the end users.  For some of them, showing them how to drag&drop to install a single .jar file is challenging.  I wanted to simplify their lives as much as possible.

The problem (for those who don't see it immediately) is that on the rare occasion when I need to modify a shared class, it is necessary to remove, re-build, and re-install all plugins that use the shared class.  This turns out to be more of a hassle for naive end users than any requirement to install multiple files when adding a new plugin.

So...my plan is to distribute every plugin as a set of .jar files (the .jar file for the plugin itself, plus multiple .jar files for shared classes).

I also intend to incrementally move new development to a new environment, leaving the existing development files as an archive.
Plus, I feel the need to re-structure the entire set of plugins to better reflect the various types of end-users.

So, I'm ready to start from scratch, and move to a new environment one plugin at a time.

Assume that I know my way around development, but am woefully out of date on "modern" development tools.  Left to my own devices, I might be perfectly happy doing all of this in "make" - but I am reliably informed that there are "better" solutions out there.  At least, that's what my students told me when they created this 'ant' environment.


I need something suitable for 50+ Java plugins (each delivered individually, or in groups of up to 10), smooth interface with "ij.jar", plus about 5 (and growing) "clusters" of classes providing mid-level support.

A typical project looks like:

ij.jar # copy
<plugin-1>
        <ant stuff>
        bin
                <various>
        release
                <plugin-1>.jar    # suitable for drag&drop installation
        src
                <plugin-1>.java..  
                <plugin-1-specific-class>.java
                ...
                <package1>
                        <package1>.java
                        ...
                <package2>
                ...
<plugin-2>
...

So, I have multiple copies of the source code for the shared packages, and everything gets rolled into one .jar file (per plugin) - and the .jar files end up having multiple copies of the shared packages.  AND...the multiple copies may conflict!  This can actually feel like a good thing when developing, but not so good when it's time to distribute.

My desired setup is more like:

<plugin-1>
        <control stuff>
        bin
        release
        src
<plugin-2>
...

and also

<shared-1>
        <control stuff>
        bin
        release
        src
<shared-2>
...

Then, I think my users can handle:

        a) install <shared-1>
        b) install <shared-2>
        ...
        z) install <plugin>

although, a "one-step" install would be pleasant.

[lower level note: I assume that the .jar files for shared classes belong in "jars"?]

I suppose I also want all dependencies checked and built while developing/modifying a new plugin (but not strictly necessary).

I suppose I also want "make all" to bring *everything* up to snuff.

I am *not* particularly interested in source control (a la git) or the like.  This is a "single-developer" environment and the "single developer" is decidedly "old school".  My development tools are EMACS, a web browser (for documentation and APIs) and command line.  Left to. my own devices, I would probably do this in "make" - but I'm willing to be educated.

Suggestions?

--
Kenneth Sloan
[hidden email]
Vision is the art of seeing what is invisible to others.  

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Curtis Rueden-2]

Hi Ken,

> I suppose I also want all dependencies checked and built while
> developing/modifying a new plugin (but not strictly necessary).

Do you want to do monoversioning? Or version each JAR file separately?
https://imagej.net/Philosophy#Release_early.2C_release_often

Either way, Maven is your friend.
https://imagej.net/Maven

But how you structure your projects will depend on how you want to version
them.

> So...my plan is to distribute every plugin as a set of .jar files

Consider creating an ImageJ update site.
https://imagej.net/Update_Sites

See also https://imagej.net/Distribution

> I am *not* particularly interested in source control (a la git) or the
like.

I would urge you to reconsider.
https://stackoverflow.com/questions/1408450/why-should-i-use-version-control

Regards,
Curtis

--
Curtis Rueden
Software architect, LOCI/Eliceiri lab - https://loci.wisc.edu/software
ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden
Have you tried the Image.sc Forum? https://forum.image.sc/



On Mon, Jan 20, 2020 at 3:31 PM Kenneth Sloan <[hidden email]>
wrote:

> What started l as a small, focussed collection of FIJI plugins has
> exploded into a collection that is beginning to be a problem to manage.
> I'm hoping someone here can point me at suggestions (and brief tutorial
> material) on current "best practices" for developing collections of loosely
> related FIJI plugins which share multiple mid-level Java classes.
>
> Constraints:
>
> a) about 50 (and growing) FIJI plugins, all written in Java
> b) distributed as .jar files, usually not including source code
> c) quasi-independent - perhaps 5 different classes of "customers", each
> with their own mix
> d) many "customers" lack the ability to build from source (and many do not
> receive source)
> e) a growing number of shared classes - these can be grouped into a small
> number of clusters.
> f) Some plugins have a lifespan of 10 years or more...others are one-shot
> throwaways.
> g) development on Mac - production use on varied platforms
>
> Current practice:
>
> a) I use 'ant', as set up by grad students, long ago.  This has been
> convenient, but I'm not all that fluent in, or committed to, 'ant'.  For a
> new plugin, I end up robotically editing 2 files (build.xml,
> plugins.config) and it "just works".  Perhaps my desired changes can be
> implemented in 'ant' - but I get the feeling that 'ant' is outdated (and a
> pain to install on some OS).  The pain involved in modifying the
> boilerplate 'ant' files that I use now is probably no less than simply
> starting over with something "better".
> b) I compile each plugin separately, outside ImageJ, and create a .jar
> file.  There are issues here with matching the Java level to that used by
> ImageJ, and also providing a pointer to "ij.jar".  The latter is achieved
> by (periodically, when there's an observed problem) copying a new version
> of "ij.jar" to the development area and rebuilding everything.  The idea
> was to be independent of the location of "ij.jar" on the development
> machine - but this causes minor bobbles every so often when the API of our
> copy of "ij.jar" does not match the actual installed "ij.jar" when the
> plugin runs.  I could live with pointing 'ant' at the canonical location of
> "ij.jar" on *my* machine (rather than making a copy).
> c) each plugin is "self-contained" - they are delivered as independent
> .jar files.  Users install the .jar files (typically drag&drop) and it
> "just works".  This is the practice that may need some change - in
> particular, the handling of shared classes.
>
> The problems:
> a) maintaining a copy of "ij.jar"
> b) shared classes
>
> It occurs to me (duh!) that it makes sense to create separate .jar files
> for each cluster of shared classes.  I have not done that yet out of
> consideration for the end users.  For some of them, showing them how to
> drag&drop to install a single .jar file is challenging.  I wanted to
> simplify their lives as much as possible.
>
> The problem (for those who don't see it immediately) is that on the rare
> occasion when I need to modify a shared class, it is necessary to remove,
> re-build, and re-install all plugins that use the shared class.  This turns
> out to be more of a hassle for naive end users than any requirement to
> install multiple files when adding a new plugin.
>
> So...my plan is to distribute every plugin as a set of .jar files (the
> .jar file for the plugin itself, plus multiple .jar files for shared
> classes).
>
> I also intend to incrementally move new development to a new environment,
> leaving the existing development files as an archive.
> Plus, I feel the need to re-structure the entire set of plugins to better
> reflect the various types of end-users.
>
> So, I'm ready to start from scratch, and move to a new environment one
> plugin at a time.
>
> Assume that I know my way around development, but am woefully out of date
> on "modern" development tools.  Left to my own devices, I might be
> perfectly happy doing all of this in "make" - but I am reliably informed
> that there are "better" solutions out there.  At least, that's what my
> students told me when they created this 'ant' environment.
>
>
> I need something suitable for 50+ Java plugins (each delivered
> individually, or in groups of up to 10), smooth interface with "ij.jar",
> plus about 5 (and growing) "clusters" of classes providing mid-level
> support.
>
> A typical project looks like:
>
> ij.jar # copy
> <plugin-1>
>         <ant stuff>
>         bin
>                 <various>
>         release
>                 <plugin-1>.jar    # suitable for drag&drop installation
>         src
>                 <plugin-1>.java..
>                 <plugin-1-specific-class>.java
>                 ...
>                 <package1>
>                         <package1>.java
>                         ...
>                 <package2>
>                 ...
> <plugin-2>
> ...
>
> So, I have multiple copies of the source code for the shared packages, and
> everything gets rolled into one .jar file (per plugin) - and the .jar files
> end up having multiple copies of the shared packages.  AND...the multiple
> copies may conflict!  This can actually feel like a good thing when
> developing, but not so good when it's time to distribute.
>
> My desired setup is more like:
>
> <plugin-1>
>         <control stuff>
>         bin
>         release
>         src
> <plugin-2>
> ...
>
> and also
>
> <shared-1>
>         <control stuff>
>         bin
>         release
>         src
> <shared-2>
> ...
>
> Then, I think my users can handle:
>
>         a) install <shared-1>
>         b) install <shared-2>
>         ...
>         z) install <plugin>
>
> although, a "one-step" install would be pleasant.
>
> [lower level note: I assume that the .jar files for shared classes belong
> in "jars"?]
>
> I suppose I also want all dependencies checked and built while
> developing/modifying a new plugin (but not strictly necessary).
>
> I suppose I also want "make all" to bring *everything* up to snuff.
>
> I am *not* particularly interested in source control (a la git) or the
> like.  This is a "single-developer" environment and the "single developer"
> is decidedly "old school".  My development tools are EMACS, a web browser
> (for documentation and APIs) and command line.  Left to. my own devices, I
> would probably do this in "make" - but I'm willing to be educated.
>
> Suggestions?
>
> --
> Kenneth Sloan
> [hidden email]
> Vision is the art of seeing what is invisible to others.
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html