pure java HDF reader

[Zummy]

Is there such a thing as a pure java HDF reader.  Would be very useful in
websites.



Thanks,

 

John

[ctrueden]

Hi John,

Is there such a thing as a pure java HDF reader.  Would be very useful in
> websites.
>

Bio-Formats uses the Java NetCDF implementation version 4.0 (
http://www.unidata.ucar.edu/software/netcdf-java/) to handle HDF-based image
formats.

-Curtis

On Wed, Oct 28, 2009 at 5:19 PM, John Carlson <[hidden email]> wrote:

> Is there such a thing as a pure java HDF reader.  Would be very useful in
> websites.
>
>
>
> Thanks,
>
>
>
> John
>

[williamfgc]

There is not such a thing (netCDF is not pure Java). I was looking online for a while without success. This is the only reference I found http://www.cv.nrao.edu/fits/traffic/scidataformats/sdf.9709 and
http://www.hdfgroup.org/hdf5-quest.html#jpar  (scroll down 3/4ths of page)

Apparently, there is not much motivation to do a pure Java implementation to manipulate HDF files (NetCDF is not pure Java either and they do not specify that on their website). The argument is that the native C code is 100,000 lines and that's too much to implement. It will worth the effort since Java is a growing force and things like MPI and Lapack, Blas are already written in pure Java.
Thanks,
William

Zummy wrote

Is there such a thing as a pure java HDF reader.  Would be very useful in
websites.



Thanks,

 

John

[spota]

This is not completely correct:
netCDF has a pure java implementation (netCDF-All if I recall well), which allows you to read and write to the older but still usable netCDF-3 format.
The same implementation will allow you to read netCDF-4, which is in essence an HDF reader.
It will not allow you to write ho netCDF-4 or HDF though. For that, you still need to use the JNI that HDF provides and which used the C implementation underneath.

[williamfgc]

Thanks, so far I've been using JNI and NCSA libraries in C, 'cause I have to read HDF5 files that have the attributes that netCDF-4 can't access (read the list below). It's pure Java for what you mentioned, but I can't read all HDF5 files that I need with netCDF. What I stated has a lot of truth.
My question is: how hard would it be to download the source code in C (from HDFgroup) and make it pure Java since they claim that the advantage of HDF files is in their code and not the files themselves.  
Thanks!



From their website:

NetCDF-4 intentionally supports a simpler data model than HDF5, which means there are HDF5 files that cannot easily be converted to netCDF-4, including files that make use of features in the following list:

    * Multidimensional data that doesn't use shared dimensions implemented using HDF5 "dimension scales". (This restriction will be eliminated in netCDF 4.1, permitting access to HDF5 datasets that don't use dimension scales.)
    * Non-hierarchical organizations of Groups, in which a Group may have multiple parents or may be both an ancestor and a descendant of another Group, creating cycles in the subgroup graph. In the netCDF-4 data model, Groups form a tree with no cycles, so each Group (except the top-level unnamed Group) has a unique parent.
    * HDF5 "references" which are like pointers to objects and data regions within a file. The netCDF-4 data model omits references.
    * Additional primitive types not included in the netCDF-4 data model, including H5T_TIME, H5T_BITFIELD, and user-defined atomic types.
    * Multiple names for data objects such as variables and groups. The netCDF-4 data model requires that each variable and group have a single distinguished name.
    * Attributes attached to user-defined types.
    * Stored property lists
    * Object names that begin or end with a space

If you know that an HDF5 file conforms to the netCDF-4 enhanced data model, either because it was written with netCDF function calls or because it doesn't make use of HDF5 features in the list above, then it can be accessed using netCDF-4, and analyzed, visualized, and manipulated through other applications that can access netCDF-4 files.

A tool has been developed to convert HDF5-EOS Aura files to netCDF-4 files, and it is currently undergoing testing and documentation before release on the HDF5 web site.

[spota]

Good point, I wasn't aware of these differences (I don't use any of these features in my application).

As for your question on implementing the whole HDF5 C libraries into pure Java, I wonder about that myself.
What I have read on the HDF5 JNI mailing lists querying on that subject is that it's 100.000 lines of C code and quite complex and that basically nobody has dedicated the time or money to do it. It makes me wonder if this thing is spaghetti code or if it's been optimized beyond human recognition...

At my work, the resources are currently spread very thin and although I'm pushing for HDF5 to be  implemented all over the place as a standard, we won't have a team to complete such an effort all by ourselves any time soon... but If a project of this sort gets started and people need my humble help, I'm totally in!