.vsi file corruption, Mac
Locked
 
|
This one is strange.
Environment: Mac, OS/X (may be specific to iMacs running slightly older version of OS/X, may be specific to the use of an external disk drive. We have .vsi files which contain two series. One series is a depth stack of ~20 16bit grayscale images. The other is at lower resolution and contains 3 planes. We are using the first series. With a well-formed .vsi file, manually running BioFormats prompts for a choice of which series to use. Running BioFormats Importer from an ImageJ java plugin, we can successfully import the correct series. With a CORRUPTED .vsi file, manually runnint BioFormats simply opens the second series (the first one is invisible) Running BioFormats Importer from an ImageJ java plugin produces the second series (which is not useful). Looking at the meta-date, it appears that the CORRUPTED files contain LESS meta-data (details available to anyone who knows what to do with it). How do the files become CORRUPTED? It seems that simply copying the file (either to another disk, or even to the same disk, will SOMETIMES corrupt the file. Some copies have been successful - but some (probably most) have produced corrupted files. Both the well-formed and corrupted files are exactly the same length. It looks to me as if some character in the middle of the meta-data becomes mangled. NOTE: I have seen the corrupting AND correct copying behavior on the SAME FILE. I’m stumped. I’ll probably try to get hex dumps of a correct and a mangled file to try to generate more clues. Does this ring any bells? Is there a .vsi file expert out there? With some effort, I think I can provide a correct, and mangled, version of one of these files to anyone who has the necessary knowledge. Note that this *may* be either a Mac issue, or a disk issue. As near as I can make out, all the offending copies were FROM a specific external drive, and done on one of two iMacs. One last twist…the person who generated most of these files was German, and may have used a Mac with German settings at some point. But…note that I have personally done two copies FROM the primary external drive TO a different external drive. One corrupted…and the second one did not. So, whatever is going on is intermittent. It’s *possible* that there may be a way to REPAIR the damage, once done. That’s what I’ll be working on right now - comparing good and bad copies to see if there is an easy way to patch the bad copies. I’m not really optimistic about that. -- Kenneth Sloan [hidden email] <mailto:[hidden email]> Vision is the art of seeing what is invisible to others. -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
Re: .vsi file corruption, Mac
|
|
One more thing: while copying the .vsi file tends to corrupt it, running a plugin which uses BioFormats Importer to read the file from the original location always works.
The corruption is specific to COPYING the .vsi file. -- Kenneth Sloan [hidden email] <mailto:[hidden email]> Vision is the art of seeing what is invisible to others. > On Jun 28, 2017, at 14:50 , Kenneth Sloan <[hidden email]> wrote: > > This one is strange. > > Environment: Mac, OS/X (may be specific to iMacs running slightly older version of OS/X, may be specific to the use of an external disk drive. > > We have .vsi files which contain two series. One series is a depth stack of ~20 16bit grayscale images. The other is at lower resolution and contains 3 planes. We are using the first series. > > With a well-formed .vsi file, manually running BioFormats prompts for a choice of which series to use. Running BioFormats Importer from an ImageJ java plugin, we can successfully import the correct series. > > With a CORRUPTED .vsi file, manually runnint BioFormats simply opens the second series (the first one is invisible) > Running BioFormats Importer from an ImageJ java plugin produces the second series (which is not useful). > > Looking at the meta-date, it appears that the CORRUPTED files contain LESS meta-data (details available to anyone who knows what to do with it). > > How do the files become CORRUPTED? It seems that simply copying the file (either to another disk, or even to the same disk, will SOMETIMES corrupt the file. Some copies have been successful - but some (probably most) have produced corrupted files. > > Both the well-formed and corrupted files are exactly the same length. > > It looks to me as if some character in the middle of the meta-data becomes mangled. > > NOTE: I have seen the corrupting AND correct copying behavior on the SAME FILE. > > I’m stumped. I’ll probably try to get hex dumps of a correct and a mangled file to try to generate more clues. > > Does this ring any bells? Is there a .vsi file expert out there? With some effort, I think I can provide a correct, and mangled, version of one of these files to anyone who has the necessary knowledge. > > Note that this *may* be either a Mac issue, or a disk issue. As near as I can make out, all the offending copies were FROM a specific external drive, and done on one of two iMacs. > > One last twist…the person who generated most of these files was German, and may have used a Mac with German settings at some point. > > But…note that I have personally done two copies FROM the primary external drive TO a different external drive. One corrupted…and the second one did not. So, whatever is going on is intermittent. > > It’s *possible* that there may be a way to REPAIR the damage, once done. That’s what I’ll be working on right now - comparing good and bad copies to see if there is an easy way to patch the bad copies. I’m not really optimistic about that. > > -- > Kenneth Sloan > [hidden email] <mailto:[hidden email]> > Vision is the art of seeing what is invisible to others. > > > > -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Hi Kenneth,
the .vsi format can have an additional directory with the same name as the vsi file (but the ".vsi" stripped off), containing *.ets files. Maybe that directory did not get copied? Michael ________________________________________________________________ On 28/06/2017 21:53, Kenneth Sloan wrote: > One more thing: while copying the .vsi file tends to corrupt it, running a plugin which uses BioFormats Importer to read the file from the original location always works. > > The corruption is specific to COPYING the .vsi file. > > -- > Kenneth Sloan > [hidden email] <mailto:[hidden email]> > Vision is the art of seeing what is invisible to others. > > > > >> On Jun 28, 2017, at 14:50 , Kenneth Sloan <[hidden email]> wrote: >> >> This one is strange. >> >> Environment: Mac, OS/X (may be specific to iMacs running slightly older version of OS/X, may be specific to the use of an external disk drive. >> >> We have .vsi files which contain two series. One series is a depth stack of ~20 16bit grayscale images. The other is at lower resolution and contains 3 planes. We are using the first series. >> >> With a well-formed .vsi file, manually running BioFormats prompts for a choice of which series to use. Running BioFormats Importer from an ImageJ java plugin, we can successfully import the correct series. >> >> With a CORRUPTED .vsi file, manually runnint BioFormats simply opens the second series (the first one is invisible) >> Running BioFormats Importer from an ImageJ java plugin produces the second series (which is not useful). >> >> Looking at the meta-date, it appears that the CORRUPTED files contain LESS meta-data (details available to anyone who knows what to do with it). >> >> How do the files become CORRUPTED? It seems that simply copying the file (either to another disk, or even to the same disk, will SOMETIMES corrupt the file. Some copies have been successful - but some (probably most) have produced corrupted files. >> >> Both the well-formed and corrupted files are exactly the same length. >> >> It looks to me as if some character in the middle of the meta-data becomes mangled. >> >> NOTE: I have seen the corrupting AND correct copying behavior on the SAME FILE. >> >> I’m stumped. I’ll probably try to get hex dumps of a correct and a mangled file to try to generate more clues. >> >> Does this ring any bells? Is there a .vsi file expert out there? With some effort, I think I can provide a correct, and mangled, version of one of these files to anyone who has the necessary knowledge. >> >> Note that this *may* be either a Mac issue, or a disk issue. As near as I can make out, all the offending copies were FROM a specific external drive, and done on one of two iMacs. >> >> One last twist…the person who generated most of these files was German, and may have used a Mac with German settings at some point. >> >> But…note that I have personally done two copies FROM the primary external drive TO a different external drive. One corrupted…and the second one did not. So, whatever is going on is intermittent. >> >> It’s *possible* that there may be a way to REPAIR the damage, once done. That’s what I’ll be working on right now - comparing good and bad copies to see if there is an easy way to patch the bad copies. I’m not really optimistic about that. >> >> -- >> Kenneth Sloan >> [hidden email] <mailto:[hidden email]> >> Vision is the art of seeing what is invisible to others. >> >> >> >> > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
Re: .vsi file corruption, Mac
|
|
Ah...I'll look for bit this - thank you. We have the .ets files, but are
not copying them. I want to say that we have had success with only the .vsi file, but I have to test that systematically. One more twist - the .vsi files were produced on a Windows machine and the copying is being done by macs. Might there be some convention clash? Escaped chars, CRLF vs NL? That's what I am looking at, today. On Thu, Jun 29, 2017 at 02:52 Michael Schmid <[hidden email]> wrote: > Hi Kenneth, > > the .vsi format can have an additional directory with the same name as > the vsi file (but the ".vsi" stripped off), containing *.ets files. > Maybe that directory did not get copied? > > Michael > > ________________________________________________________________ > > On 28/06/2017 21:53, Kenneth Sloan wrote: > > One more thing: while copying the .vsi file tends to corrupt it, running > a plugin which uses BioFormats Importer to read the file from the original > location always works. > > > > The corruption is specific to COPYING the .vsi file. > > > > -- > > Kenneth Sloan > > [hidden email] <mailto:[hidden email]> > > Vision is the art of seeing what is invisible to others. > > > > > > > > > >> On Jun 28, 2017, at 14:50 , Kenneth Sloan <[hidden email]> > wrote: > >> > >> This one is strange. > >> > >> Environment: Mac, OS/X (may be specific to iMacs running slightly older > version of OS/X, may be specific to the use of an external disk drive. > >> > >> We have .vsi files which contain two series. One series is a depth > stack of ~20 16bit grayscale images. The other is at lower resolution and > contains 3 planes. We are using the first series. > >> > >> With a well-formed .vsi file, manually running BioFormats prompts for a > choice of which series to use. Running BioFormats Importer from an ImageJ > java plugin, we can successfully import the correct series. > >> > >> With a CORRUPTED .vsi file, manually runnint BioFormats simply opens > the second series (the first one is invisible) > >> Running BioFormats Importer from an ImageJ java plugin produces the > second series (which is not useful). > >> > >> Looking at the meta-date, it appears that the CORRUPTED files contain > LESS meta-data (details available to anyone who knows what to do with it). > >> > >> How do the files become CORRUPTED? It seems that simply copying the > file (either to another disk, or even to the same disk, will SOMETIMES > corrupt the file. Some copies have been successful - but some (probably > most) have produced corrupted files. > >> > >> Both the well-formed and corrupted files are exactly the same length. > >> > >> It looks to me as if some character in the middle of the meta-data > becomes mangled. > >> > >> NOTE: I have seen the corrupting AND correct copying behavior on the > SAME FILE. > >> > >> I’m stumped. I’ll probably try to get hex dumps of a correct and a > mangled file to try to generate more clues. > >> > >> Does this ring any bells? Is there a .vsi file expert out there? With > some effort, I think I can provide a correct, and mangled, version of one > of these files to anyone who has the necessary knowledge. > >> > >> Note that this *may* be either a Mac issue, or a disk issue. As near > as I can make out, all the offending copies were FROM a specific external > drive, and done on one of two iMacs. > >> > >> One last twist…the person who generated most of these files was German, > and may have used a Mac with German settings at some point. > >> > >> But…note that I have personally done two copies FROM the primary > external drive TO a different external drive. One corrupted…and the second > one did not. So, whatever is going on is intermittent. > >> > >> It’s *possible* that there may be a way to REPAIR the damage, once > done. That’s what I’ll be working on right now - comparing good and bad > copies to see if there is an easy way to patch the bad copies. I’m not > really optimistic about that. > >> > >> -- > >> Kenneth Sloan > >> [hidden email] <mailto:[hidden email]> > >> Vision is the art of seeing what is invisible to others. > >> > >> > >> > >> > > > > > > -- > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
Re: .vsi file corruption, Mac
|
|
This did the trick - THANK YOU!
Note to BioFormats implementors - the fact that the .ets file is missing seems worthy of at least a WARNING message! -- Kenneth Sloan [hidden email] <mailto:[hidden email]> Vision is the art of seeing what is invisible to others. > On Jun 29, 2017, at 10:44 , Kenneth R Sloan <[hidden email]> wrote: > > Ah...I'll look for bit this - thank you. We have the .ets files, but are not copying them. I want to say that we have had success with only the .vsi file, but I have to test that systematically. > > One more twist - the .vsi files were produced on a Windows machine and the copying is being done by macs. Might there be some convention clash? Escaped chars, CRLF vs NL? That's what I am looking at, today. > > > On Thu, Jun 29, 2017 at 02:52 Michael Schmid <[hidden email] <mailto:[hidden email]>> wrote: > Hi Kenneth, > > the .vsi format can have an additional directory with the same name as > the vsi file (but the ".vsi" stripped off), containing *.ets files. > Maybe that directory did not get copied? > > Michael > > ________________________________________________________________ > > On 28/06/2017 21:53, Kenneth Sloan wrote: > > One more thing: while copying the .vsi file tends to corrupt it, running a plugin which uses BioFormats Importer to read the file from the original location always works. > > > > The corruption is specific to COPYING the .vsi file. > > > > -- > > Kenneth Sloan > > [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> > > Vision is the art of seeing what is invisible to others. > > > > > > > > > >> On Jun 28, 2017, at 14:50 , Kenneth Sloan <[hidden email] <mailto:[hidden email]>> wrote: > >> > >> This one is strange. > >> > >> Environment: Mac, OS/X (may be specific to iMacs running slightly older version of OS/X, may be specific to the use of an external disk drive. > >> > >> We have .vsi files which contain two series. One series is a depth stack of ~20 16bit grayscale images. The other is at lower resolution and contains 3 planes. We are using the first series. > >> > >> With a well-formed .vsi file, manually running BioFormats prompts for a choice of which series to use. Running BioFormats Importer from an ImageJ java plugin, we can successfully import the correct series. > >> > >> With a CORRUPTED .vsi file, manually runnint BioFormats simply opens the second series (the first one is invisible) > >> Running BioFormats Importer from an ImageJ java plugin produces the second series (which is not useful). > >> > >> Looking at the meta-date, it appears that the CORRUPTED files contain LESS meta-data (details available to anyone who knows what to do with it). > >> > >> How do the files become CORRUPTED? It seems that simply copying the file (either to another disk, or even to the same disk, will SOMETIMES corrupt the file. Some copies have been successful - but some (probably most) have produced corrupted files. > >> > >> Both the well-formed and corrupted files are exactly the same length. > >> > >> It looks to me as if some character in the middle of the meta-data becomes mangled. > >> > >> NOTE: I have seen the corrupting AND correct copying behavior on the SAME FILE. > >> > >> I’m stumped. I’ll probably try to get hex dumps of a correct and a mangled file to try to generate more clues. > >> > >> Does this ring any bells? Is there a .vsi file expert out there? With some effort, I think I can provide a correct, and mangled, version of one of these files to anyone who has the necessary knowledge. > >> > >> Note that this *may* be either a Mac issue, or a disk issue. As near as I can make out, all the offending copies were FROM a specific external drive, and done on one of two iMacs. > >> > >> One last twist…the person who generated most of these files was German, and may have used a Mac with German settings at some point. > >> > >> But…note that I have personally done two copies FROM the primary external drive TO a different external drive. One corrupted…and the second one did not. So, whatever is going on is intermittent. > >> > >> It’s *possible* that there may be a way to REPAIR the damage, once done. That’s what I’ll be working on right now - comparing good and bad copies to see if there is an easy way to patch the bad copies. I’m not really optimistic about that. > >> > >> -- > >> Kenneth Sloan > >> [hidden email] <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> > >> Vision is the art of seeing what is invisible to others. > >> > >> > >> > >> > > > > > > -- > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html <http://imagej.nih.gov/ij/list.html> > > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html <http://imagej.nih.gov/ij/list.html> -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
Hi Ken,
> Note to BioFormats implementors - the fact that the .ets file is > missing seems worthy of at least a WARNING message! For this feedback to be noticed, I suggest you use one of the OME communication channels; see: * https://www.openmicroscopy.org/site/support/bio-formats/about/bug-reporting.html Maybe file it as an issue here: * https://github.com/openmicroscopy/bioformats/issues Regards, Curtis -- Curtis Rueden LOCI software architect - https://loci.wisc.edu/software ImageJ2 lead, Fiji maintainer - https://imagej.net/User:Rueden Did you know ImageJ has a forum? http://forum.imagej.net/ On Thu, Jun 29, 2017 at 8:14 PM, Kenneth Sloan <[hidden email]> wrote: > This did the trick - THANK YOU! > > Note to BioFormats implementors - the fact that the .ets file is missing > seems worthy of at least a WARNING message! > > -- > Kenneth Sloan > [hidden email] <mailto:[hidden email]> > Vision is the art of seeing what is invisible to others. > > > > > > On Jun 29, 2017, at 10:44 , Kenneth R Sloan <[hidden email]> > wrote: > > > > Ah...I'll look for bit this - thank you. We have the .ets files, but are > not copying them. I want to say that we have had success with only the .vsi > file, but I have to test that systematically. > > > > One more twist - the .vsi files were produced on a Windows machine and > the copying is being done by macs. Might there be some convention clash? > Escaped chars, CRLF vs NL? That's what I am looking at, today. > > > > > > On Thu, Jun 29, 2017 at 02:52 Michael Schmid <[hidden email] > <mailto:[hidden email]>> wrote: > > Hi Kenneth, > > > > the .vsi format can have an additional directory with the same name as > > the vsi file (but the ".vsi" stripped off), containing *.ets files. > > Maybe that directory did not get copied? > > > > Michael > > > > ________________________________________________________________ > > > > On 28/06/2017 21:53, Kenneth Sloan wrote: > > > One more thing: while copying the .vsi file tends to corrupt it, > running a plugin which uses BioFormats Importer to read the file from the > original location always works. > > > > > > The corruption is specific to COPYING the .vsi file. > > > > > > -- > > > Kenneth Sloan > > > [hidden email] <mailto:[hidden email]> <mailto: > [hidden email] <mailto:[hidden email]>> > > > Vision is the art of seeing what is invisible to others. > > > > > > > > > > > > > > >> On Jun 28, 2017, at 14:50 , Kenneth Sloan <[hidden email] > <mailto:[hidden email]>> wrote: > > >> > > >> This one is strange. > > >> > > >> Environment: Mac, OS/X (may be specific to iMacs running slightly > older version of OS/X, may be specific to the use of an external disk drive. > > >> > > >> We have .vsi files which contain two series. One series is a depth > stack of ~20 16bit grayscale images. The other is at lower resolution and > contains 3 planes. We are using the first series. > > >> > > >> With a well-formed .vsi file, manually running BioFormats prompts for > a choice of which series to use. Running BioFormats Importer from an > ImageJ java plugin, we can successfully import the correct series. > > >> > > >> With a CORRUPTED .vsi file, manually runnint BioFormats simply opens > the second series (the first one is invisible) > > >> Running BioFormats Importer from an ImageJ java plugin produces the > second series (which is not useful). > > >> > > >> Looking at the meta-date, it appears that the CORRUPTED files contain > LESS meta-data (details available to anyone who knows what to do with it). > > >> > > >> How do the files become CORRUPTED? It seems that simply copying the > file (either to another disk, or even to the same disk, will SOMETIMES > corrupt the file. Some copies have been successful - but some (probably > most) have produced corrupted files. > > >> > > >> Both the well-formed and corrupted files are exactly the same length. > > >> > > >> It looks to me as if some character in the middle of the meta-data > becomes mangled. > > >> > > >> NOTE: I have seen the corrupting AND correct copying behavior on the > SAME FILE. > > >> > > >> I’m stumped. I’ll probably try to get hex dumps of a correct and a > mangled file to try to generate more clues. > > >> > > >> Does this ring any bells? Is there a .vsi file expert out there? > With some effort, I think I can provide a correct, and mangled, version of > one of these files to anyone who has the necessary knowledge. > > >> > > >> Note that this *may* be either a Mac issue, or a disk issue. As near > as I can make out, all the offending copies were FROM a specific external > drive, and done on one of two iMacs. > > >> > > >> One last twist…the person who generated most of these files was > German, and may have used a Mac with German settings at some point. > > >> > > >> But…note that I have personally done two copies FROM the primary > external drive TO a different external drive. One corrupted…and the second > one did not. So, whatever is going on is intermittent. > > >> > > >> It’s *possible* that there may be a way to REPAIR the damage, once > done. That’s what I’ll be working on right now - comparing good and bad > copies to see if there is an easy way to patch the bad copies. I’m not > really optimistic about that. > > >> > > >> -- > > >> Kenneth Sloan > > >> [hidden email] <mailto:[hidden email]> <mailto: > [hidden email] <mailto:[hidden email]>> > > >> Vision is the art of seeing what is invisible to others. > > >> > > >> > > >> > > >> > > > > > > > > > -- > > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html < > http://imagej.nih.gov/ij/list.html> > > > > > > > -- > > ImageJ mailing list: http://imagej.nih.gov/ij/list.html < > http://imagej.nih.gov/ij/list.html> > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
Re: .vsi file corruption, Mac
|
|
In reply to this post by Kenneth Sloan-2
Hi Kenneth,
glad to hear that Michael suggestion solved your issue. Unfortunately, we have CellSens VSI samples of both types i.e. one vsi files with or without ets subdirectory [1]. At the moment, we don’t have any specification allowing us to know whether a subdirectory is expected or not. Adding some debug level logging when no subdirectory is found would certainly be something we could consider. Would this be of any help in this case? Best, Sebastien [1] https://www.openmicroscopy.org/site/support/bio-formats5.5/formats/dataset-table.html On 29 Jun 2017, at 19:14, Kenneth Sloan <[hidden email]<mailto:[hidden email]>> wrote: This did the trick - THANK YOU! Note to BioFormats implementors - the fact that the .ets file is missing seems worthy of at least a WARNING message! -- Kenneth Sloan [hidden email]<mailto:[hidden email]> <mailto:[hidden email]> Vision is the art of seeing what is invisible to others. On Jun 29, 2017, at 10:44 , Kenneth R Sloan <[hidden email]<mailto:[hidden email]>> wrote: Ah...I'll look for bit this - thank you. We have the .ets files, but are not copying them. I want to say that we have had success with only the .vsi file, but I have to test that systematically. One more twist - the .vsi files were produced on a Windows machine and the copying is being done by macs. Might there be some convention clash? Escaped chars, CRLF vs NL? That's what I am looking at, today. On Thu, Jun 29, 2017 at 02:52 Michael Schmid <[hidden email]<mailto:[hidden email]> <mailto:[hidden email]>> wrote: Hi Kenneth, the .vsi format can have an additional directory with the same name as the vsi file (but the ".vsi" stripped off), containing *.ets files. Maybe that directory did not get copied? Michael ________________________________________________________________ On 28/06/2017 21:53, Kenneth Sloan wrote: One more thing: while copying the .vsi file tends to corrupt it, running a plugin which uses BioFormats Importer to read the file from the original location always works. The corruption is specific to COPYING the .vsi file. -- Kenneth Sloan [hidden email]<mailto:[hidden email]> <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> Vision is the art of seeing what is invisible to others. On Jun 28, 2017, at 14:50 , Kenneth Sloan <[hidden email]<mailto:[hidden email]> <mailto:[hidden email]>> wrote: This one is strange. Environment: Mac, OS/X (may be specific to iMacs running slightly older version of OS/X, may be specific to the use of an external disk drive. We have .vsi files which contain two series. One series is a depth stack of ~20 16bit grayscale images. The other is at lower resolution and contains 3 planes. We are using the first series. With a well-formed .vsi file, manually running BioFormats prompts for a choice of which series to use. Running BioFormats Importer from an ImageJ java plugin, we can successfully import the correct series. With a CORRUPTED .vsi file, manually runnint BioFormats simply opens the second series (the first one is invisible) Running BioFormats Importer from an ImageJ java plugin produces the second series (which is not useful). Looking at the meta-date, it appears that the CORRUPTED files contain LESS meta-data (details available to anyone who knows what to do with it). How do the files become CORRUPTED? It seems that simply copying the file (either to another disk, or even to the same disk, will SOMETIMES corrupt the file. Some copies have been successful - but some (probably most) have produced corrupted files. Both the well-formed and corrupted files are exactly the same length. It looks to me as if some character in the middle of the meta-data becomes mangled. NOTE: I have seen the corrupting AND correct copying behavior on the SAME FILE. I’m stumped. I’ll probably try to get hex dumps of a correct and a mangled file to try to generate more clues. Does this ring any bells? Is there a .vsi file expert out there? With some effort, I think I can provide a correct, and mangled, version of one of these files to anyone who has the necessary knowledge. Note that this *may* be either a Mac issue, or a disk issue. As near as I can make out, all the offending copies were FROM a specific external drive, and done on one of two iMacs. One last twist…the person who generated most of these files was German, and may have used a Mac with German settings at some point. But…note that I have personally done two copies FROM the primary external drive TO a different external drive. One corrupted…and the second one did not. So, whatever is going on is intermittent. It’s *possible* that there may be a way to REPAIR the damage, once done. That’s what I’ll be working on right now - comparing good and bad copies to see if there is an easy way to patch the bad copies. I’m not really optimistic about that. -- Kenneth Sloan [hidden email]<mailto:[hidden email]> <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> Vision is the art of seeing what is invisible to others. -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html <http://imagej.nih.gov/ij/list.html> -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html <http://imagej.nih.gov/ij/list.html> -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html The University of Dundee is a registered Scottish Charity, No: SC015096 -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
Re: .vsi file corruption, Mac
|
|
Yes, please.
Looking at the meta-data display, I suspect that there is an easy sanity check that could determine that a .ets file is required (but is not present). The difference in output is striking. Without checking - I think it might be easy to recognize the problem. What I (as a dumb user of .vsi/.ets files) see is a very truncated display of meta-data. This (mis-)led me to think that there might be a problem character embedded in the meta-data that led to incorrect parsing. I was about to try to chase down that theory with the cavalry arrived and solved my problem. To reproduce: a) start with a .vsi/.ets pair b) remove the .ets file c) open the .vsi file and display the meta data. I don’t know enough about the full range of .vsi/.ets files that CellSens can produce, so I hesitate to speculate. But, FOR US, I believe that data collection always produced a .vsi/.ets pair desribing 2 “series”. One is a depth stack, at one autofluorescence wavelength. These images are 1344x1024, monochrome, 16-bit. The other is a 3-layer stack, which appears to be autofluorescence at three different wavelengths. [if it really matters, please contact me off-list and I can elicit details from the person who actually gathered the data; I would also be more than happy to provide an example pair of files]. In our world, there is always a .ets file for each .vsi file, but I am ignorant about exactly how that happened. If it matters, I can find out. My current problem is that I’m implementing an ImageJ Java plugin which needs to read the 1st series (the mono, 16-bit, depth stack). WITH the .ets file present, this works just fine (we first used BioFormats Importer to set parameters and recorded the command. I then simply copied this command into an IJ.run(…) call. Again, if it will help, I’ll gladly send the source code for this plugin. When both files are present, it works just fine. When the .ets file is missing (because we moved “only the relevant files” to a working directory) BioFormats Importer silently reads in the SECOND series (smaller - perhaps 340x512?, and only 3 planes). This, of course, causes the rest of the plugin to be very confused! Recognizing the fact that the .ets file is required (and is missing) and generating a WARNING would have been a great help to me - it would have cut the debugging time down by 3 days). I suspect the above is sufficient to reproduce the problem, but if not, contact me at [hidden email] <mailto:[hidden email]> and I can supply source code for the plugin and a .vsi/.ets file that reproduces the behavior. Given the specific nature of our input files, I am going to add a sanity check by looking at the xy dimensions of the stack imported by BioFormats. If it’s not the right size, I’ll issue my own ERROR message. Thanks very much for your attention to this. -- Kenneth Sloan [hidden email] <mailto:[hidden email]> Vision is the art of seeing what is invisible to others. > On Jun 29, 2017, at 14:36 , Sebastien Besson (Staff) <[hidden email]> wrote: > > Hi Kenneth, > > glad to hear that Michael suggestion solved your issue. > > Unfortunately, we have CellSens VSI samples of both types i.e. one vsi files with or > without ets subdirectory [1]. At the moment, we don’t have any specification allowing > us to know whether a subdirectory is expected or not. > > Adding some debug level logging when no subdirectory is found would certainly be > something we could consider. Would this be of any help in this case? > > Best, > Sebastien > > [1] https://www.openmicroscopy.org/site/support/bio-formats5.5/formats/dataset-table.html > > On 29 Jun 2017, at 19:14, Kenneth Sloan <[hidden email]<mailto:[hidden email]>> wrote: > > This did the trick - THANK YOU! > > Note to BioFormats implementors - the fact that the .ets file is missing seems worthy of at least a WARNING message! > > -- > Kenneth Sloan > [hidden email]<mailto:[hidden email]> <mailto:[hidden email]> > Vision is the art of seeing what is invisible to others. > > > > > On Jun 29, 2017, at 10:44 , Kenneth R Sloan <[hidden email]<mailto:[hidden email]>> wrote: > > Ah...I'll look for bit this - thank you. We have the .ets files, but are not copying them. I want to say that we have had success with only the .vsi file, but I have to test that systematically. > > One more twist - the .vsi files were produced on a Windows machine and the copying is being done by macs. Might there be some convention clash? Escaped chars, CRLF vs NL? That's what I am looking at, today. > > > On Thu, Jun 29, 2017 at 02:52 Michael Schmid <[hidden email]<mailto:[hidden email]> <mailto:[hidden email]>> wrote: > Hi Kenneth, > > the .vsi format can have an additional directory with the same name as > the vsi file (but the ".vsi" stripped off), containing *.ets files. > Maybe that directory did not get copied? > > Michael > > ________________________________________________________________ > > On 28/06/2017 21:53, Kenneth Sloan wrote: > One more thing: while copying the .vsi file tends to corrupt it, running a plugin which uses BioFormats Importer to read the file from the original location always works. > > The corruption is specific to COPYING the .vsi file. > > -- > Kenneth Sloan > [hidden email]<mailto:[hidden email]> <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> > Vision is the art of seeing what is invisible to others. > > > > > On Jun 28, 2017, at 14:50 , Kenneth Sloan <[hidden email]<mailto:[hidden email]> <mailto:[hidden email]>> wrote: > > This one is strange. > > Environment: Mac, OS/X (may be specific to iMacs running slightly older version of OS/X, may be specific to the use of an external disk drive. > > We have .vsi files which contain two series. One series is a depth stack of ~20 16bit grayscale images. The other is at lower resolution and contains 3 planes. We are using the first series. > > With a well-formed .vsi file, manually running BioFormats prompts for a choice of which series to use. Running BioFormats Importer from an ImageJ java plugin, we can successfully import the correct series. > > With a CORRUPTED .vsi file, manually runnint BioFormats simply opens the second series (the first one is invisible) > Running BioFormats Importer from an ImageJ java plugin produces the second series (which is not useful). > > Looking at the meta-date, it appears that the CORRUPTED files contain LESS meta-data (details available to anyone who knows what to do with it). > > How do the files become CORRUPTED? It seems that simply copying the file (either to another disk, or even to the same disk, will SOMETIMES corrupt the file. Some copies have been successful - but some (probably most) have produced corrupted files. > > Both the well-formed and corrupted files are exactly the same length. > > It looks to me as if some character in the middle of the meta-data becomes mangled. > > NOTE: I have seen the corrupting AND correct copying behavior on the SAME FILE. > > I’m stumped. I’ll probably try to get hex dumps of a correct and a mangled file to try to generate more clues. > > Does this ring any bells? Is there a .vsi file expert out there? With some effort, I think I can provide a correct, and mangled, version of one of these files to anyone who has the necessary knowledge. > > Note that this *may* be either a Mac issue, or a disk issue. As near as I can make out, all the offending copies were FROM a specific external drive, and done on one of two iMacs. > > One last twist…the person who generated most of these files was German, and may have used a Mac with German settings at some point. > > But…note that I have personally done two copies FROM the primary external drive TO a different external drive. One corrupted…and the second one did not. So, whatever is going on is intermittent. > > It’s *possible* that there may be a way to REPAIR the damage, once done. That’s what I’ll be working on right now - comparing good and bad copies to see if there is an easy way to patch the bad copies. I’m not really optimistic about that. > > -- > Kenneth Sloan > [hidden email]<mailto:[hidden email]> <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> > Vision is the art of seeing what is invisible to others. > > > > > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html <http://imagej.nih.gov/ij/list.html> > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html <http://imagej.nih.gov/ij/list.html> > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > > > The University of Dundee is a registered Scottish Charity, No: SC015096 > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
Re: .vsi file corruption, Mac
|
|
Hi Kenneth,
once more thanks for bringing this issue to our attention. Adding some logic to warn the users of a missing folder makes sense and we have recorded it in the following card [1]. Correcting my previous email, we have some private specification for the CellSens VSI format and will update our documentation accordingly. Hopefully this spec provides an unambiguous way to detect the existence of extra raw data files. We hope we have enough data for now but will contact you if we need additional files. For the time being, would it be sufficient to add a VSI-specific check in your plugin using the FormatReader.getUsedFiles() API and verify that an `*.ets` file has also been detected as part of the fileset? The mid-term solution is obviously for Bio-Formats to correctly handle partial VSI filesets as discussed in the card below. Best, Sebastien [1] https://trello.com/c/fkWsjC4R/157-vsi-detect-warn-about-missing-ets-folder On 30 Jun 2017, at 18:20, Kenneth Sloan <[hidden email]<mailto:[hidden email]>> wrote: Yes, please. Looking at the meta-data display, I suspect that there is an easy sanity check that could determine that a .ets file is required (but is not present). The difference in output is striking. Without checking - I think it might be easy to recognize the problem. What I (as a dumb user of .vsi/.ets files) see is a very truncated display of meta-data. This (mis-)led me to think that there might be a problem character embedded in the meta-data that led to incorrect parsing. I was about to try to chase down that theory with the cavalry arrived and solved my problem. To reproduce: a) start with a .vsi/.ets pair b) remove the .ets file c) open the .vsi file and display the meta data. I don’t know enough about the full range of .vsi/.ets files that CellSens can produce, so I hesitate to speculate. But, FOR US, I believe that data collection always produced a .vsi/.ets pair desribing 2 “series”. One is a depth stack, at one autofluorescence wavelength. These images are 1344x1024, monochrome, 16-bit. The other is a 3-layer stack, which appears to be autofluorescence at three different wavelengths. [if it really matters, please contact me off-list and I can elicit details from the person who actually gathered the data; I would also be more than happy to provide an example pair of files]. In our world, there is always a .ets file for each .vsi file, but I am ignorant about exactly how that happened. If it matters, I can find out. My current problem is that I’m implementing an ImageJ Java plugin which needs to read the 1st series (the mono, 16-bit, depth stack). WITH the .ets file present, this works just fine (we first used BioFormats Importer to set parameters and recorded the command. I then simply copied this command into an IJ.run(…) call. Again, if it will help, I’ll gladly send the source code for this plugin. When both files are present, it works just fine. When the .ets file is missing (because we moved “only the relevant files” to a working directory) BioFormats Importer silently reads in the SECOND series (smaller - perhaps 340x512?, and only 3 planes). This, of course, causes the rest of the plugin to be very confused! Recognizing the fact that the .ets file is required (and is missing) and generating a WARNING would have been a great help to me - it would have cut the debugging time down by 3 days). I suspect the above is sufficient to reproduce the problem, but if not, contact me at [hidden email]<mailto:[hidden email]> <mailto:[hidden email]> and I can supply source code for the plugin and a .vsi/.ets file that reproduces the behavior. Given the specific nature of our input files, I am going to add a sanity check by looking at the xy dimensions of the stack imported by BioFormats. If it’s not the right size, I’ll issue my own ERROR message. Thanks very much for your attention to this. -- Kenneth Sloan [hidden email]<mailto:[hidden email]> <mailto:[hidden email]> Vision is the art of seeing what is invisible to others. On Jun 29, 2017, at 14:36 , Sebastien Besson (Staff) <[hidden email]<mailto:[hidden email]>> wrote: Hi Kenneth, glad to hear that Michael suggestion solved your issue. Unfortunately, we have CellSens VSI samples of both types i.e. one vsi files with or without ets subdirectory [1]. At the moment, we don’t have any specification allowing us to know whether a subdirectory is expected or not. Adding some debug level logging when no subdirectory is found would certainly be something we could consider. Would this be of any help in this case? Best, Sebastien [1] https://www.openmicroscopy.org/site/support/bio-formats5.5/formats/dataset-table.html On 29 Jun 2017, at 19:14, Kenneth Sloan <[hidden email]<mailto:[hidden email]><mailto:[hidden email]>> wrote: This did the trick - THANK YOU! Note to BioFormats implementors - the fact that the .ets file is missing seems worthy of at least a WARNING message! -- Kenneth Sloan [hidden email]<mailto:[hidden email]><mailto:[hidden email]> <mailto:[hidden email]> Vision is the art of seeing what is invisible to others. On Jun 29, 2017, at 10:44 , Kenneth R Sloan <[hidden email]<mailto:[hidden email]><mailto:[hidden email]>> wrote: Ah...I'll look for bit this - thank you. We have the .ets files, but are not copying them. I want to say that we have had success with only the .vsi file, but I have to test that systematically. One more twist - the .vsi files were produced on a Windows machine and the copying is being done by macs. Might there be some convention clash? Escaped chars, CRLF vs NL? That's what I am looking at, today. On Thu, Jun 29, 2017 at 02:52 Michael Schmid <[hidden email]<mailto:[hidden email]><mailto:[hidden email]> <mailto:[hidden email]>> wrote: Hi Kenneth, the .vsi format can have an additional directory with the same name as the vsi file (but the ".vsi" stripped off), containing *.ets files. Maybe that directory did not get copied? Michael ________________________________________________________________ On 28/06/2017 21:53, Kenneth Sloan wrote: One more thing: while copying the .vsi file tends to corrupt it, running a plugin which uses BioFormats Importer to read the file from the original location always works. The corruption is specific to COPYING the .vsi file. -- Kenneth Sloan [hidden email]<mailto:[hidden email]><mailto:[hidden email]> <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> Vision is the art of seeing what is invisible to others. On Jun 28, 2017, at 14:50 , Kenneth Sloan <[hidden email]<mailto:[hidden email]><mailto:[hidden email]> <mailto:[hidden email]>> wrote: This one is strange. Environment: Mac, OS/X (may be specific to iMacs running slightly older version of OS/X, may be specific to the use of an external disk drive. We have .vsi files which contain two series. One series is a depth stack of ~20 16bit grayscale images. The other is at lower resolution and contains 3 planes. We are using the first series. With a well-formed .vsi file, manually running BioFormats prompts for a choice of which series to use. Running BioFormats Importer from an ImageJ java plugin, we can successfully import the correct series. With a CORRUPTED .vsi file, manually runnint BioFormats simply opens the second series (the first one is invisible) Running BioFormats Importer from an ImageJ java plugin produces the second series (which is not useful). Looking at the meta-date, it appears that the CORRUPTED files contain LESS meta-data (details available to anyone who knows what to do with it). How do the files become CORRUPTED? It seems that simply copying the file (either to another disk, or even to the same disk, will SOMETIMES corrupt the file. Some copies have been successful - but some (probably most) have produced corrupted files. Both the well-formed and corrupted files are exactly the same length. It looks to me as if some character in the middle of the meta-data becomes mangled. NOTE: I have seen the corrupting AND correct copying behavior on the SAME FILE. I’m stumped. I’ll probably try to get hex dumps of a correct and a mangled file to try to generate more clues. Does this ring any bells? Is there a .vsi file expert out there? With some effort, I think I can provide a correct, and mangled, version of one of these files to anyone who has the necessary knowledge. Note that this *may* be either a Mac issue, or a disk issue. As near as I can make out, all the offending copies were FROM a specific external drive, and done on one of two iMacs. One last twist…the person who generated most of these files was German, and may have used a Mac with German settings at some point. But…note that I have personally done two copies FROM the primary external drive TO a different external drive. One corrupted…and the second one did not. So, whatever is going on is intermittent. It’s *possible* that there may be a way to REPAIR the damage, once done. That’s what I’ll be working on right now - comparing good and bad copies to see if there is an easy way to patch the bad copies. I’m not really optimistic about that. -- Kenneth Sloan [hidden email]<mailto:[hidden email]><mailto:[hidden email]> <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> Vision is the art of seeing what is invisible to others. -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html <http://imagej.nih.gov/ij/list.html> -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html <http://imagej.nih.gov/ij/list.html> -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html The University of Dundee is a registered Scottish Charity, No: SC015096 -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html The University of Dundee is a registered Scottish Charity, No: SC015096 -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
Re: .vsi file corruption, Mac
|
|
That's good for the short term. But, anyone who is aware of the fact that
.vsi files sometimes require an .ets file (and include the check) may not need it. The folk who need a warning are the clueless ones (like me, last week) who don't realize this is an issue. For example, I am modifying my plugin to check the xy dimensions of the imported stack. On Mon, Jul 3, 2017 at 08:38 Sebastien Besson (Staff) <[hidden email]> wrote: > Hi Kenneth, > > once more thanks for bringing this issue to our attention. Adding some > logic > to warn the users of a missing folder makes sense and we have recorded > it in the following card [1]. > > Correcting my previous email, we have some private specification for the > CellSens VSI format and will update our documentation accordingly. > Hopefully this spec provides an unambiguous way to detect the existence of > extra raw data files. We hope we have enough data for now but will contact > you > if we need additional files. > > For the time being, would it be sufficient to add a VSI-specific check in > your plugin > using the FormatReader.getUsedFiles() API and verify that an `*.ets` file > has also > been detected as part of the fileset? > The mid-term solution is obviously for Bio-Formats to correctly handle > partial VSI > filesets as discussed in the card below. > > Best, > Sebastien > > [1] > https://trello.com/c/fkWsjC4R/157-vsi-detect-warn-about-missing-ets-folder > > On 30 Jun 2017, at 18:20, Kenneth Sloan <[hidden email]<mailto: > [hidden email]>> wrote: > > Yes, please. > > Looking at the meta-data display, I suspect that there is an easy sanity > check that > could determine that a .ets file is required (but is not present). The > difference in output > is striking. Without checking - I think it might be easy to recognize the > problem. > > What I (as a dumb user of .vsi/.ets files) see is a very truncated display > of meta-data. > This (mis-)led me to think that there might be a problem character > embedded in the meta-data > that led to incorrect parsing. I was about to try to chase down that > theory with the cavalry > arrived and solved my problem. > > To reproduce: > > a) start with a .vsi/.ets pair > b) remove the .ets file > c) open the .vsi file and display the meta data. > > I don’t know enough about the full range of .vsi/.ets files that CellSens > can produce, so > I hesitate to speculate. But, FOR US, I believe that data collection > always produced a .vsi/.ets > pair desribing 2 “series”. One is a depth stack, at one autofluorescence > wavelength. These images are 1344x1024, monochrome, 16-bit. The other is > a 3-layer stack, which appears to be autofluorescence at three different > wavelengths. [if it really matters, please contact me off-list and I can > elicit details from the person who actually > gathered the data; I would also be more than happy to provide an example > pair of files]. In our world, there is always a .ets file for each .vsi > file, but I am ignorant about exactly how that happened. If it matters, I > can find out. > > My current problem is that I’m implementing an ImageJ Java plugin which > needs to read the 1st series (the mono, 16-bit, depth stack). WITH the > .ets file present, this works just fine (we first used BioFormats Importer > to set parameters and recorded the command. I then simply copied this > command into an IJ.run(…) call. Again, if it will help, I’ll gladly send > the source code for this plugin. > > When both files are present, it works just fine. When the .ets file is > missing (because we moved “only the relevant files” to a working directory) > BioFormats Importer silently reads in the SECOND series (smaller - perhaps > 340x512?, and only 3 planes). This, of course, causes the rest of the > plugin to be very confused! > > Recognizing the fact that the .ets file is required (and is missing) and > generating a WARNING would have been a great help to me - it would have cut > the debugging time down by 3 days). > > I suspect the above is sufficient to reproduce the problem, but if not, > contact me at [hidden email]<mailto:[hidden email]> > <mailto:[hidden email]> and I can supply source code for the > plugin and a .vsi/.ets file that reproduces the behavior. > > Given the specific nature of our input files, I am going to add a sanity > check by looking at the xy dimensions of the stack imported by BioFormats. > If it’s not the right size, I’ll issue my own ERROR message. > > Thanks very much for your attention to this. > > -- > Kenneth Sloan > [hidden email]<mailto:[hidden email]> <mailto: > [hidden email]> > Vision is the art of seeing what is invisible to others. > > > > > On Jun 29, 2017, at 14:36 , Sebastien Besson (Staff) < > [hidden email]<mailto:[hidden email]>> wrote: > > Hi Kenneth, > > glad to hear that Michael suggestion solved your issue. > > Unfortunately, we have CellSens VSI samples of both types i.e. one vsi > files with or > without ets subdirectory [1]. At the moment, we don’t have any > specification allowing > us to know whether a subdirectory is expected or not. > > Adding some debug level logging when no subdirectory is found would > certainly be > something we could consider. Would this be of any help in this case? > > Best, > Sebastien > > [1] > https://www.openmicroscopy.org/site/support/bio-formats5.5/formats/dataset-table.html > > On 29 Jun 2017, at 19:14, Kenneth Sloan <[hidden email]<mailto: > [hidden email]><mailto:[hidden email]>> wrote: > > This did the trick - THANK YOU! > > Note to BioFormats implementors - the fact that the .ets file is missing > seems worthy of at least a WARNING message! > > -- > Kenneth Sloan > [hidden email]<mailto:[hidden email]><mailto: > [hidden email]> <mailto:[hidden email]> > Vision is the art of seeing what is invisible to others. > > > > > On Jun 29, 2017, at 10:44 , Kenneth R Sloan <[hidden email] > <mailto:[hidden email]><mailto:[hidden email]>> wrote: > > Ah...I'll look for bit this - thank you. We have the .ets files, but are > not copying them. I want to say that we have had success with only the .vsi > file, but I have to test that systematically. > > One more twist - the .vsi files were produced on a Windows machine and the > copying is being done by macs. Might there be some convention clash? > Escaped chars, CRLF vs NL? That's what I am looking at, today. > > > On Thu, Jun 29, 2017 at 02:52 Michael Schmid <[hidden email] > <mailto:[hidden email]><mailto:[hidden email]> <mailto: > [hidden email]>> wrote: > Hi Kenneth, > > the .vsi format can have an additional directory with the same name as > the vsi file (but the ".vsi" stripped off), containing *.ets files. > Maybe that directory did not get copied? > > Michael > > ________________________________________________________________ > > On 28/06/2017 21:53, Kenneth Sloan wrote: > One more thing: while copying the .vsi file tends to corrupt it, running a > plugin which uses BioFormats Importer to read the file from the original > location always works. > > The corruption is specific to COPYING the .vsi file. > > -- > Kenneth Sloan > [hidden email]<mailto:[hidden email]><mailto: > [hidden email]> <mailto:[hidden email]> <mailto: > [hidden email] <mailto:[hidden email]>> > Vision is the art of seeing what is invisible to others. > > > > > On Jun 28, 2017, at 14:50 , Kenneth Sloan <[hidden email]<mailto: > [hidden email]><mailto:[hidden email]> <mailto: > [hidden email]>> wrote: > > This one is strange. > > Environment: Mac, OS/X (may be specific to iMacs running slightly older > version of OS/X, may be specific to the use of an external disk drive. > > We have .vsi files which contain two series. One series is a depth stack > of ~20 16bit grayscale images. The other is at lower resolution and > contains 3 planes. We are using the first series. > > With a well-formed .vsi file, manually running BioFormats prompts for a > choice of which series to use. Running BioFormats Importer from an ImageJ > java plugin, we can successfully import the correct series. > > With a CORRUPTED .vsi file, manually runnint BioFormats simply opens the > second series (the first one is invisible) > Running BioFormats Importer from an ImageJ java plugin produces the second > series (which is not useful). > > Looking at the meta-date, it appears that the CORRUPTED files contain LESS > meta-data (details available to anyone who knows what to do with it). > > How do the files become CORRUPTED? It seems that simply copying the file > (either to another disk, or even to the same disk, will SOMETIMES corrupt > the file. Some copies have been successful - but some (probably most) have > produced corrupted files. > > Both the well-formed and corrupted files are exactly the same length. > > It looks to me as if some character in the middle of the meta-data becomes > mangled. > > NOTE: I have seen the corrupting AND correct copying behavior on the SAME > FILE. > > I’m stumped. I’ll probably try to get hex dumps of a correct and a > mangled file to try to generate more clues. > > Does this ring any bells? Is there a .vsi file expert out there? With > some effort, I think I can provide a correct, and mangled, version of one > of these files to anyone who has the necessary knowledge. > > Note that this *may* be either a Mac issue, or a disk issue. As near as I > can make out, all the offending copies were FROM a specific external drive, > and done on one of two iMacs. > > One last twist…the person who generated most of these files was German, > and may have used a Mac with German settings at some point. > > But…note that I have personally done two copies FROM the primary external > drive TO a different external drive. One corrupted…and the second one did > not. So, whatever is going on is intermittent. > > It’s *possible* that there may be a way to REPAIR the damage, once done. > That’s what I’ll be working on right now - comparing good and bad copies to > see if there is an easy way to patch the bad copies. I’m not really > optimistic about that. > > -- > Kenneth Sloan > [hidden email]<mailto:[hidden email]><mailto: > [hidden email]> <mailto:[hidden email]> <mailto: > [hidden email] <mailto:[hidden email]>> > Vision is the art of seeing what is invisible to others. > > > > > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html < > http://imagej.nih.gov/ij/list.html> > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html < > http://imagej.nih.gov/ij/list.html> > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > > > The University of Dundee is a registered Scottish Charity, No: SC015096 > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > > > The University of Dundee is a registered Scottish Charity, No: SC015096 > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
Re: .vsi file corruption, Mac
|
|
In reply to this post by Sebastien Besson (Staff)
Sure - that would work. Although, what will work for us is educating the user to NOT move one file without the other!
-- Kenneth Sloan [hidden email] <mailto:[hidden email]> Vision is the art of seeing what is invisible to others. > On Jul 3, 2017, at 09:23 , Sebastien Besson (Staff) <[hidden email]> wrote: > > Hi Kenneth, > > once more thanks for bringing this issue to our attention. Adding some logic > to warn the users of a missing folder makes sense and we have recorded > it in the following card [1]. > > Correcting my previous email, we have some private specification for the > CellSens VSI format and will update our documentation accordingly. > Hopefully this spec provides an unambiguous way to detect the existence of > extra raw data files. We hope we have enough data for now but will contact you > if we need additional files. > > For the time being, would it be sufficient to add a VSI-specific check in your plugin > using the FormatReader.getUsedFiles() API and verify that an `*.ets` file has also > been detected as part of the fileset? > The mid-term solution is obviously for Bio-Formats to correctly handle partial VSI > filesets as discussed in the card below. > > Best, > Sebastien > > [1] https://trello.com/c/fkWsjC4R/157-vsi-detect-warn-about-missing-ets-folder > > On 30 Jun 2017, at 18:20, Kenneth Sloan <[hidden email]<mailto:[hidden email]>> wrote: > > Yes, please. > > Looking at the meta-data display, I suspect that there is an easy sanity check that > could determine that a .ets file is required (but is not present). The difference in output > is striking. Without checking - I think it might be easy to recognize the problem. > > What I (as a dumb user of .vsi/.ets files) see is a very truncated display of meta-data. > This (mis-)led me to think that there might be a problem character embedded in the meta-data > that led to incorrect parsing. I was about to try to chase down that theory with the cavalry > arrived and solved my problem. > > To reproduce: > > a) start with a .vsi/.ets pair > b) remove the .ets file > c) open the .vsi file and display the meta data. > > I don’t know enough about the full range of .vsi/.ets files that CellSens can produce, so > I hesitate to speculate. But, FOR US, I believe that data collection always produced a .vsi/.ets > pair desribing 2 “series”. One is a depth stack, at one autofluorescence wavelength. These images are 1344x1024, monochrome, 16-bit. The other is a 3-layer stack, which appears to be autofluorescence at three different wavelengths. [if it really matters, please contact me off-list and I can elicit details from the person who actually > gathered the data; I would also be more than happy to provide an example pair of files]. In our world, there is always a .ets file for each .vsi file, but I am ignorant about exactly how that happened. If it matters, I can find out. > > My current problem is that I’m implementing an ImageJ Java plugin which needs to read the 1st series (the mono, 16-bit, depth stack). WITH the .ets file present, this works just fine (we first used BioFormats Importer to set parameters and recorded the command. I then simply copied this command into an IJ.run(…) call. Again, if it will help, I’ll gladly send the source code for this plugin. > > When both files are present, it works just fine. When the .ets file is missing (because we moved “only the relevant files” to a working directory) BioFormats Importer silently reads in the SECOND series (smaller - perhaps 340x512?, and only 3 planes). This, of course, causes the rest of the plugin to be very confused! > > Recognizing the fact that the .ets file is required (and is missing) and generating a WARNING would have been a great help to me - it would have cut the debugging time down by 3 days). > > I suspect the above is sufficient to reproduce the problem, but if not, contact me at [hidden email]<mailto:[hidden email]> <mailto:[hidden email]> and I can supply source code for the plugin and a .vsi/.ets file that reproduces the behavior. > > Given the specific nature of our input files, I am going to add a sanity check by looking at the xy dimensions of the stack imported by BioFormats. If it’s not the right size, I’ll issue my own ERROR message. > > Thanks very much for your attention to this. > > -- > Kenneth Sloan > [hidden email]<mailto:[hidden email]> <mailto:[hidden email]> > Vision is the art of seeing what is invisible to others. > > > > > On Jun 29, 2017, at 14:36 , Sebastien Besson (Staff) <[hidden email]<mailto:[hidden email]>> wrote: > > Hi Kenneth, > > glad to hear that Michael suggestion solved your issue. > > Unfortunately, we have CellSens VSI samples of both types i.e. one vsi files with or > without ets subdirectory [1]. At the moment, we don’t have any specification allowing > us to know whether a subdirectory is expected or not. > > Adding some debug level logging when no subdirectory is found would certainly be > something we could consider. Would this be of any help in this case? > > Best, > Sebastien > > [1] https://www.openmicroscopy.org/site/support/bio-formats5.5/formats/dataset-table.html > > On 29 Jun 2017, at 19:14, Kenneth Sloan <[hidden email]<mailto:[hidden email]><mailto:[hidden email]>> wrote: > > This did the trick - THANK YOU! > > Note to BioFormats implementors - the fact that the .ets file is missing seems worthy of at least a WARNING message! > > -- > Kenneth Sloan > [hidden email]<mailto:[hidden email]><mailto:[hidden email]> <mailto:[hidden email]> > Vision is the art of seeing what is invisible to others. > > > > > On Jun 29, 2017, at 10:44 , Kenneth R Sloan <[hidden email]<mailto:[hidden email]><mailto:[hidden email]>> wrote: > > Ah...I'll look for bit this - thank you. We have the .ets files, but are not copying them. I want to say that we have had success with only the .vsi file, but I have to test that systematically. > > One more twist - the .vsi files were produced on a Windows machine and the copying is being done by macs. Might there be some convention clash? Escaped chars, CRLF vs NL? That's what I am looking at, today. > > > On Thu, Jun 29, 2017 at 02:52 Michael Schmid <[hidden email]<mailto:[hidden email]><mailto:[hidden email]> <mailto:[hidden email]>> wrote: > Hi Kenneth, > > the .vsi format can have an additional directory with the same name as > the vsi file (but the ".vsi" stripped off), containing *.ets files. > Maybe that directory did not get copied? > > Michael > > ________________________________________________________________ > > On 28/06/2017 21:53, Kenneth Sloan wrote: > One more thing: while copying the .vsi file tends to corrupt it, running a plugin which uses BioFormats Importer to read the file from the original location always works. > > The corruption is specific to COPYING the .vsi file. > > -- > Kenneth Sloan > [hidden email]<mailto:[hidden email]><mailto:[hidden email]> <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> > Vision is the art of seeing what is invisible to others. > > > > > On Jun 28, 2017, at 14:50 , Kenneth Sloan <[hidden email]<mailto:[hidden email]><mailto:[hidden email]> <mailto:[hidden email]>> wrote: > > This one is strange. > > Environment: Mac, OS/X (may be specific to iMacs running slightly older version of OS/X, may be specific to the use of an external disk drive. > > We have .vsi files which contain two series. One series is a depth stack of ~20 16bit grayscale images. The other is at lower resolution and contains 3 planes. We are using the first series. > > With a well-formed .vsi file, manually running BioFormats prompts for a choice of which series to use. Running BioFormats Importer from an ImageJ java plugin, we can successfully import the correct series. > > With a CORRUPTED .vsi file, manually runnint BioFormats simply opens the second series (the first one is invisible) > Running BioFormats Importer from an ImageJ java plugin produces the second series (which is not useful). > > Looking at the meta-date, it appears that the CORRUPTED files contain LESS meta-data (details available to anyone who knows what to do with it). > > How do the files become CORRUPTED? It seems that simply copying the file (either to another disk, or even to the same disk, will SOMETIMES corrupt the file. Some copies have been successful - but some (probably most) have produced corrupted files. > > Both the well-formed and corrupted files are exactly the same length. > > It looks to me as if some character in the middle of the meta-data becomes mangled. > > NOTE: I have seen the corrupting AND correct copying behavior on the SAME FILE. > > I’m stumped. I’ll probably try to get hex dumps of a correct and a mangled file to try to generate more clues. > > Does this ring any bells? Is there a .vsi file expert out there? With some effort, I think I can provide a correct, and mangled, version of one of these files to anyone who has the necessary knowledge. > > Note that this *may* be either a Mac issue, or a disk issue. As near as I can make out, all the offending copies were FROM a specific external drive, and done on one of two iMacs. > > One last twist…the person who generated most of these files was German, and may have used a Mac with German settings at some point. > > But…note that I have personally done two copies FROM the primary external drive TO a different external drive. One corrupted…and the second one did not. So, whatever is going on is intermittent. > > It’s *possible* that there may be a way to REPAIR the damage, once done. That’s what I’ll be working on right now - comparing good and bad copies to see if there is an easy way to patch the bad copies. I’m not really optimistic about that. > > -- > Kenneth Sloan > [hidden email]<mailto:[hidden email]><mailto:[hidden email]> <mailto:[hidden email]> <mailto:[hidden email] <mailto:[hidden email]>> > Vision is the art of seeing what is invisible to others. > > > > > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html <http://imagej.nih.gov/ij/list.html> > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html <http://imagej.nih.gov/ij/list.html> > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > > > The University of Dundee is a registered Scottish Charity, No: SC015096 > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > > > The University of Dundee is a registered Scottish Charity, No: SC015096 > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
«
Return to ImageJ
|
1 view|%1 views
| Free forum by Nabble | Edit this page |