http://imagej.273.s1.nabble.com/vsi-file-corruption-Mac-tp5018990p5019028.html
That's good for the short term. But, anyone who is aware of the fact that
need it. The folk who need a warning are the clueless ones (like me, last
> Hi Kenneth,
>
> once more thanks for bringing this issue to our attention. Adding some
> logic
> to warn the users of a missing folder makes sense and we have recorded
> it in the following card [1].
>
> Correcting my previous email, we have some private specification for the
> CellSens VSI format and will update our documentation accordingly.
> Hopefully this spec provides an unambiguous way to detect the existence of
> extra raw data files. We hope we have enough data for now but will contact
> you
> if we need additional files.
>
> For the time being, would it be sufficient to add a VSI-specific check in
> your plugin
> using the FormatReader.getUsedFiles() API and verify that an `*.ets` file
> has also
> been detected as part of the fileset?
> The mid-term solution is obviously for Bio-Formats to correctly handle
> partial VSI
> filesets as discussed in the card below.
>
> Best,
> Sebastien
>
> [1]
>
https://trello.com/c/fkWsjC4R/157-vsi-detect-warn-about-missing-ets-folder>
> On 30 Jun 2017, at 18:20, Kenneth Sloan <
[hidden email]<mailto:
>
[hidden email]>> wrote:
>
> Yes, please.
>
> Looking at the meta-data display, I suspect that there is an easy sanity
> check that
> could determine that a .ets file is required (but is not present). The
> difference in output
> is striking. Without checking - I think it might be easy to recognize the
> problem.
>
> What I (as a dumb user of .vsi/.ets files) see is a very truncated display
> of meta-data.
> This (mis-)led me to think that there might be a problem character
> embedded in the meta-data
> that led to incorrect parsing. I was about to try to chase down that
> theory with the cavalry
> arrived and solved my problem.
>
> To reproduce:
>
> a) start with a .vsi/.ets pair
> b) remove the .ets file
> c) open the .vsi file and display the meta data.
>
> I don’t know enough about the full range of .vsi/.ets files that CellSens
> can produce, so
> I hesitate to speculate. But, FOR US, I believe that data collection
> always produced a .vsi/.ets
> pair desribing 2 “series”. One is a depth stack, at one autofluorescence
> wavelength. These images are 1344x1024, monochrome, 16-bit. The other is
> a 3-layer stack, which appears to be autofluorescence at three different
> wavelengths. [if it really matters, please contact me off-list and I can
> elicit details from the person who actually
> gathered the data; I would also be more than happy to provide an example
> pair of files]. In our world, there is always a .ets file for each .vsi
> file, but I am ignorant about exactly how that happened. If it matters, I
> can find out.
>
> My current problem is that I’m implementing an ImageJ Java plugin which
> needs to read the 1st series (the mono, 16-bit, depth stack). WITH the
> .ets file present, this works just fine (we first used BioFormats Importer
> to set parameters and recorded the command. I then simply copied this
> command into an IJ.run(…) call. Again, if it will help, I’ll gladly send
> the source code for this plugin.
>
> When both files are present, it works just fine. When the .ets file is
> missing (because we moved “only the relevant files” to a working directory)
> BioFormats Importer silently reads in the SECOND series (smaller - perhaps
> 340x512?, and only 3 planes). This, of course, causes the rest of the
> plugin to be very confused!
>
> Recognizing the fact that the .ets file is required (and is missing) and
> generating a WARNING would have been a great help to me - it would have cut
> the debugging time down by 3 days).
>
> I suspect the above is sufficient to reproduce the problem, but if not,
> contact me at
[hidden email]<mailto:
[hidden email]>
> <mailto:
[hidden email]> and I can supply source code for the
> plugin and a .vsi/.ets file that reproduces the behavior.
>
> Given the specific nature of our input files, I am going to add a sanity
> check by looking at the xy dimensions of the stack imported by BioFormats.
> If it’s not the right size, I’ll issue my own ERROR message.
>
> Thanks very much for your attention to this.
>
> --
> Kenneth Sloan
>
[hidden email]<mailto:
[hidden email]> <mailto:
>
[hidden email]>
> Vision is the art of seeing what is invisible to others.
>
>
>
>
> On Jun 29, 2017, at 14:36 , Sebastien Besson (Staff) <
>
[hidden email]<mailto:
[hidden email]>> wrote:
>
> Hi Kenneth,
>
> glad to hear that Michael suggestion solved your issue.
>
> Unfortunately, we have CellSens VSI samples of both types i.e. one vsi
> files with or
> without ets subdirectory [1]. At the moment, we don’t have any
> specification allowing
> us to know whether a subdirectory is expected or not.
>
> Adding some debug level logging when no subdirectory is found would
> certainly be
> something we could consider. Would this be of any help in this case?
>
> Best,
> Sebastien
>
> [1]
>
https://www.openmicroscopy.org/site/support/bio-formats5.5/formats/dataset-table.html>
> On 29 Jun 2017, at 19:14, Kenneth Sloan <
[hidden email]<mailto:
>
[hidden email]><mailto:
[hidden email]>> wrote:
>
> This did the trick - THANK YOU!
>
> Note to BioFormats implementors - the fact that the .ets file is missing
> seems worthy of at least a WARNING message!
>
> --
> Kenneth Sloan
>
[hidden email]<mailto:
[hidden email]><mailto:
>
[hidden email]> <mailto:
[hidden email]>
> Vision is the art of seeing what is invisible to others.
>
>
>
>
> On Jun 29, 2017, at 10:44 , Kenneth R Sloan <
[hidden email]
> <mailto:
[hidden email]><mailto:
[hidden email]>> wrote:
>
> Ah...I'll look for bit this - thank you. We have the .ets files, but are
> not copying them. I want to say that we have had success with only the .vsi
> file, but I have to test that systematically.
>
> One more twist - the .vsi files were produced on a Windows machine and the
> copying is being done by macs. Might there be some convention clash?
> Escaped chars, CRLF vs NL? That's what I am looking at, today.
>
>
> On Thu, Jun 29, 2017 at 02:52 Michael Schmid <
[hidden email]
> <mailto:
[hidden email]><mailto:
[hidden email]> <mailto:
>
[hidden email]>> wrote:
> Hi Kenneth,
>
> the .vsi format can have an additional directory with the same name as
> the vsi file (but the ".vsi" stripped off), containing *.ets files.
> Maybe that directory did not get copied?
>
> Michael
>
> ________________________________________________________________
>
> On 28/06/2017 21:53, Kenneth Sloan wrote:
> One more thing: while copying the .vsi file tends to corrupt it, running a
> plugin which uses BioFormats Importer to read the file from the original
> location always works.
>
> The corruption is specific to COPYING the .vsi file.
>
> --
> Kenneth Sloan
>
[hidden email]<mailto:
[hidden email]><mailto:
>
[hidden email]> <mailto:
[hidden email]> <mailto:
>
[hidden email] <mailto:
[hidden email]>>
> Vision is the art of seeing what is invisible to others.
>
>
>
>
> On Jun 28, 2017, at 14:50 , Kenneth Sloan <
[hidden email]<mailto:
>
[hidden email]><mailto:
[hidden email]> <mailto:
>
[hidden email]>> wrote:
>
> This one is strange.
>
> Environment: Mac, OS/X (may be specific to iMacs running slightly older
> version of OS/X, may be specific to the use of an external disk drive.
>
> We have .vsi files which contain two series. One series is a depth stack
> of ~20 16bit grayscale images. The other is at lower resolution and
> contains 3 planes. We are using the first series.
>
> With a well-formed .vsi file, manually running BioFormats prompts for a
> choice of which series to use. Running BioFormats Importer from an ImageJ
> java plugin, we can successfully import the correct series.
>
> With a CORRUPTED .vsi file, manually runnint BioFormats simply opens the
> second series (the first one is invisible)
> Running BioFormats Importer from an ImageJ java plugin produces the second
> series (which is not useful).
>
> Looking at the meta-date, it appears that the CORRUPTED files contain LESS
> meta-data (details available to anyone who knows what to do with it).
>
> How do the files become CORRUPTED? It seems that simply copying the file
> (either to another disk, or even to the same disk, will SOMETIMES corrupt
> the file. Some copies have been successful - but some (probably most) have
> produced corrupted files.
>
> Both the well-formed and corrupted files are exactly the same length.
>
> It looks to me as if some character in the middle of the meta-data becomes
> mangled.
>
> NOTE: I have seen the corrupting AND correct copying behavior on the SAME
> FILE.
>
> I’m stumped. I’ll probably try to get hex dumps of a correct and a
> mangled file to try to generate more clues.
>
> Does this ring any bells? Is there a .vsi file expert out there? With
> some effort, I think I can provide a correct, and mangled, version of one
> of these files to anyone who has the necessary knowledge.
>
> Note that this *may* be either a Mac issue, or a disk issue. As near as I
> can make out, all the offending copies were FROM a specific external drive,
> and done on one of two iMacs.
>
> One last twist…the person who generated most of these files was German,
> and may have used a Mac with German settings at some point.
>
> But…note that I have personally done two copies FROM the primary external
> drive TO a different external drive. One corrupted…and the second one did
> not. So, whatever is going on is intermittent.
>
> It’s *possible* that there may be a way to REPAIR the damage, once done.
> That’s what I’ll be working on right now - comparing good and bad copies to
> see if there is an easy way to patch the bad copies. I’m not really
> optimistic about that.
>
> --
> Kenneth Sloan
>
[hidden email]<mailto:
[hidden email]><mailto:
>
[hidden email]> <mailto:
[hidden email]> <mailto:
>
[hidden email] <mailto:
[hidden email]>>
> Vision is the art of seeing what is invisible to others.
>
>
>
>
>
>
> --
> ImageJ mailing list:
http://imagej.nih.gov/ij/list.html <
>
http://imagej.nih.gov/ij/list.html>
>
>
> --
> ImageJ mailing list:
http://imagej.nih.gov/ij/list.html <
>
http://imagej.nih.gov/ij/list.html>
>
>
> --
> ImageJ mailing list:
http://imagej.nih.gov/ij/list.html>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
> --
> ImageJ mailing list:
http://imagej.nih.gov/ij/list.html>
>
> --
> ImageJ mailing list:
http://imagej.nih.gov/ij/list.html>
>
> The University of Dundee is a registered Scottish Charity, No: SC015096
>
> --
> ImageJ mailing list:
http://imagej.nih.gov/ij/list.html>