Network drive slowness
Locked
 
|
Hi all,
I recently got our lab workstation connected through the 1Gbps network to the building's server. Among other reasons, I was hoping to process images directly from the network drive. However, even with the upgrade to 1Gbps, loading images in Fiji is still slow. For example, it takes me at least three times longer (often more) to load a 200MB file directly from the server than to download it to the desktop and then open it. Does anyone have experienced a similar situation when working with files directly from a mapped network drive? Is loading a file from a network drive unavoidably slower than downloading it to the desktop and working with it locally, despite the network speed? Is there a fix or a tweak to get loading files from mapped network drives faster in Fiji? Thank you Rafael -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
Re: Network drive slowness
|
|
How long is it taking you on average in both.scenarios?
On Nov 9, 2014 10:01 AM, "Rafael Buono" <[hidden email]> wrote: > Hi all, > > I recently got our lab workstation connected through the 1Gbps network to > the building's server. Among other reasons, I was hoping to process images > directly from the network drive. However, even with the upgrade to 1Gbps, > loading images in Fiji is still slow. > For example, it takes me at least three times longer (often more) to load > a 200MB file directly from the server than to download it to the desktop > and then open it. > Does anyone have experienced a similar situation when working with files > directly from a mapped network drive? > Is loading a file from a network drive unavoidably slower than downloading > it to the desktop and working with it locally, despite the network speed? > Is there a fix or a tweak to get loading files from mapped network drives > faster in Fiji? > > Thank you > > Rafael > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
In reply to this post by Rafael Buono
For us
The key was 1- how is the data stored on your server.... We had ours on external raids units and it was the interface to the server that was slow We moved to using an internal SSD to serve up data - Made a world of difference Sent from my iPhone > On Nov 9, 2014, at 11:01, "Rafael Buono" <[hidden email]> wrote: > > Hi all, > > I recently got our lab workstation connected through the 1Gbps network to the building's server. Among other reasons, I was hoping to process images directly from the network drive. However, even with the upgrade to 1Gbps, loading images in Fiji is still slow. > For example, it takes me at least three times longer (often more) to load a 200MB file directly from the server than to download it to the desktop and then open it. > Does anyone have experienced a similar situation when working with files directly from a mapped network drive? > Is loading a file from a network drive unavoidably slower than downloading it to the desktop and working with it locally, despite the network speed? > Is there a fix or a tweak to get loading files from mapped network drives faster in Fiji? > > Thank you > > Rafael > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
In reply to this post by Rafael Buono
Hi Christopher, thanks for the answer. I have been wondering if the snag is totally on the server side.
I have little access to how our data is handled on the server side. My strongest assumptions are that we have a few hard drives (the spinning type) running in RAID. No SSDs there. And likely not top performance. A simple test gave me a reading speed of ~40MB/s, and I know that it will fluctuate depending on the amount of other users trying to access it at the same time as me. The portion that puzzles me is that I can copy a file from the network drive to my desktop much faster than I can open the same file in Fiji. I should mention that after my original message I realized that Zeiss' ZEN 2012 is able to open the files faster than Fiji. That makes me wonder why Fiji is not taking advantage of the bandwidth available to it. Has anyone else experienced slow loading of files from network drives in Fiji? -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Am 09/11/14 um 20:28 schrieb Rafael Buono:
> Has anyone else experienced slow loading of files from network drives in Fiji? Hey Rafael, i have worked quite intensivly with .czi files over our network here. This is what I can share. - reading .czi in Fiji is always slower than reading a .tiff - with a 1 Gbps connection to a samba share I can read .tiffs with up to 80 MB/s, while .czi creeps along with 20 MB/s. - I even notice this speed differences locally, the network makes it just much more feelable. - If you really need fast .czi loading, put it at a local SSD. Because of all this behaviour we convert all our .czi into .tiff in an automated way. Here in house we have multiple approaches. I wrote myself a bigger macro which takes even care of double illumination. I will happily share my macro, if you want. Additionally we do this conversion on our local cluster, where we process many imaging sessions in parallel. This way we move our .czi's there and after starting the conversion job, you have the .tiff's after 2-3 hours. This way convert up to 500 GB of .czi into much more usable .tiffs. I do all this effort, because the tiff exporter of the Zeiss Software is barely usable. We have the Zeiss Black 2014 edition here, which does not even has the batch export mode. The Zeiss Blue 2012 edition is better suited for tiff export, but is also more painful to use. I can give you more infos if you want, Stephan -- Stephan Janosch Software Engineer - TransgeneOme Database https://transgeneome.mpi-cbg.de Max Planck Institute of Molecular Cell Biology and Genetics Pfotenhauerstr. 108 01307 Dresden Germany Room: 205 Phone: +49 351 210 2709 Email: [hidden email] Web: www.mpi-cbg.de Twitter: https://twitter.com/TransgeneOme -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
In reply to this post by Rafael Buono
Hi Carlos and Stephan,
As an example, I have a .czi of ~210MB that opens: - ZEN Blue Lite 2012: about 2 seconds - Copying and pasting to the Desktop: about 4 seconds - Fiji (updated today right before starting to work): 30 seconds I realize now that I was being optimistic with my "it take 3x longer in Fiji". Stephan, I am interested in you macro. Pardon my ignorance, but does it keep all the metadata intact? Now, to make the problem more intriguing, while testing the speeds above I found out an interesting effect. If I put the same file one folder deeper and try to open it in Fiji, the time grows from 30 seconds to 12 minutes. I am one of those people that organizes things by creating folder, inside folder, inside folder. In the following structure: O:\Folder1\Folder2\Folder3\Folder4\Folder5\Folder6\Folder7 If I have the czi file in Folder6, it takes 30 seconds to open. If I move it to Folder7, it takes 12 minutes to open. Is there an obvious reason for that? Is this how it works or should I do a bug report for it? -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
Hi Rafael,
> should I do a bug report for it? The CZI support is driven by Bio-Formats [1]. See this page for information on reporting Bio-Formats bugs: http://www.openmicroscopy.org/site/support/bio-formats/about/bug-reporting.html Regards, Curtis [1] http://imagej.net/Bio-Formats On Mon, Nov 10, 2014 at 11:56 AM, Rafael Buono <[hidden email]> wrote: > Hi Carlos and Stephan, > > As an example, I have a .czi of ~210MB that opens: > - ZEN Blue Lite 2012: about 2 seconds > - Copying and pasting to the Desktop: about 4 seconds > - Fiji (updated today right before starting to work): 30 seconds > > I realize now that I was being optimistic with my "it take 3x longer in > Fiji". > > Stephan, I am interested in you macro. Pardon my ignorance, but does it > keep all the metadata intact? > > Now, to make the problem more intriguing, while testing the speeds above I > found out an interesting effect. If I put the same file one folder deeper > and try to open it in Fiji, the time grows from 30 seconds to 12 minutes. > > I am one of those people that organizes things by creating folder, inside > folder, inside folder. > In the following structure: > O:\Folder1\Folder2\Folder3\Folder4\Folder5\Folder6\Folder7 > If I have the czi file in Folder6, it takes 30 seconds to open. If I move > it to Folder7, it takes 12 minutes to open. > Is there an obvious reason for that? > Is this how it works or should I do a bug report for it? > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
Re: Network drive slowness
|
|
Hi all,
On 10.11.2014 21:13, Curtis Rueden wrote: > Hi Rafael, > >> should I do a bug report for it? > > The CZI support is driven by Bio-Formats [1]. from my experience with loading / saving data from network shares, I'm not sure wheter this is related to the CZI format. Loading images using Bio-Formats directly from a network drive usually takes around 10x longer than opening the same file from a local disk (whereas the storage backend of the network drive is potentially a lot faster than the conventional HDD's we're using on most of our computers). Sometimes it can even be up to 100x slower than opening data locally. The same story goes the other way round, when writing OME-TIFFs to a network share. We've even had situations where the OME-TIFFs seem to be written correctly but upon re-opening them they were broken. Problem was that this was not reliably reproducible, so I didn't have anything to file a reasonable bug report. The baseline of this is that we keep on telling our users to *always* transfer the data onto the machines they're using to process it, do their work, save it locally and then eventually transfer everything back to their network share to ensure maximum speed and reliability. I'd be more than happy to help in improving this situation! Best regards, ~Niko -- Niko Ehrenfeuchter | Image Analysis Specialist | Biozentrum, University of Basel | Klingelbergstr. 50/70 | CH-4056 Basel Phone: +41 (61) 26 72673 | [hidden email] | www.biozentrum.unibas.ch | www.microscopynetwork.unibas.ch -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Dear All-
Just to reiterate Curtis' point, OME welcomes reports of any issues regarding Bio-Formats. See http://www.openmicroscopy.org/site/support/bio-formats5/about/bug-reporting.html or the Bio-Formats Forum at http://www.openmicroscopy.org/community/ Obviously, we won't be very helpful diagnosing the vagaries of your network performance. We will respond to and work on issues related to file opening times, etc. We've put a lot of work into CZI and other multi-platform file formats recently-- any and all feedback welcome. Thanks! Cheers, Jason On Mon, Nov 10, 2014 at 10:29 PM, Niko Ehrenfeuchter < [hidden email]> wrote: > Hi all, > > On 10.11.2014 21:13, Curtis Rueden wrote: > >> Hi Rafael, >> >> should I do a bug report for it? >>> >> >> The CZI support is driven by Bio-Formats [1]. >> > > from my experience with loading / saving data from network shares, I'm not > sure wheter this is related to the CZI format. > > Loading images using Bio-Formats directly from a network drive usually > takes around 10x longer than opening the same file from a local disk > (whereas the storage backend of the network drive is potentially a lot > faster than the conventional HDD's we're using on most of our computers). > Sometimes it can even be up to 100x slower than opening data locally. > > The same story goes the other way round, when writing OME-TIFFs to a > network share. We've even had situations where the OME-TIFFs seem to be > written correctly but upon re-opening them they were broken. Problem was > that this was not reliably reproducible, so I didn't have anything to file > a reasonable bug report. > > The baseline of this is that we keep on telling our users to *always* > transfer the data onto the machines they're using to process it, do their > work, save it locally and then eventually transfer everything back to their > network share to ensure maximum speed and reliability. > > I'd be more than happy to help in improving this situation! > > Best regards, > ~Niko > -- > Niko Ehrenfeuchter | Image Analysis Specialist | Biozentrum, University of > Basel | Klingelbergstr. 50/70 | CH-4056 Basel > Phone: +41 (61) 26 72673 | [hidden email] | > www.biozentrum.unibas.ch | www.microscopynetwork.unibas.ch > > > -- > ImageJ mailing list: http://imagej.nih.gov/ij/list.html > -- ************************** Centre for Gene Regulation & Expression College of Life Sciences MSI/WTB/JBC Complex University of Dundee Dow Street Dundee DD1 5EH United Kingdom phone (01382) 385819 Intl phone: 44 1382 385819 FAX (01382) 388072 email: [hidden email] Lab Page: http://www.lifesci.dundee.ac.uk/gre/staff/jason-swedlow Open Microscopy Environment: http://openmicroscopy.org ************************** -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
Re: Network drive slowness
|
|
Dear All,
maybe I should clarify a bit my previous message: I didn't intend *at all* to criticize Bio-Formats - in fact it is an outstanding project which is saving my ass every single day, and I *really* appreciate it a lot! My point was just about our general experience with using imaging data directly on network shares (which is even limited to the SMB/CIFS protocol, but this is probably true for the majority of the users here) and our persistent recommendation to work on local storage only. Actually, we've seen *much* worse behaviour in combination with network shares from commercial software (under paid maintenance!), where the vendor is even refusing to work on these problems. So my apologies if this came out the wrong way! Cheers ~Niko On 10.11.2014 23:49, Jason Swedlow wrote: > Dear All- > > Just to reiterate Curtis' point, OME welcomes reports of any issues > regarding Bio-Formats. See > > http://www.openmicroscopy.org/site/support/bio-formats5/about/bug-reporting.html > > or the Bio-Formats Forum at > > http://www.openmicroscopy.org/community/ > > Obviously, we won't be very helpful diagnosing the vagaries of your network > performance. We will respond to and work on issues related to file opening > times, etc. We've put a lot of work into CZI and other multi-platform file > formats recently-- any and all feedback welcome. > > Thanks! > > Cheers, > > Jason > > On Mon, Nov 10, 2014 at 10:29 PM, Niko Ehrenfeuchter < > [hidden email]> wrote: > >> Hi all, >> >> On 10.11.2014 21:13, Curtis Rueden wrote: >> >>> Hi Rafael, >>> >>> should I do a bug report for it? >>>> >>> >>> The CZI support is driven by Bio-Formats [1]. >>> >> >> from my experience with loading / saving data from network shares, I'm not >> sure wheter this is related to the CZI format. >> >> Loading images using Bio-Formats directly from a network drive usually >> takes around 10x longer than opening the same file from a local disk >> (whereas the storage backend of the network drive is potentially a lot >> faster than the conventional HDD's we're using on most of our computers). >> Sometimes it can even be up to 100x slower than opening data locally. >> >> The same story goes the other way round, when writing OME-TIFFs to a >> network share. We've even had situations where the OME-TIFFs seem to be >> written correctly but upon re-opening them they were broken. Problem was >> that this was not reliably reproducible, so I didn't have anything to file >> a reasonable bug report. >> >> The baseline of this is that we keep on telling our users to *always* >> transfer the data onto the machines they're using to process it, do their >> work, save it locally and then eventually transfer everything back to their >> network share to ensure maximum speed and reliability. >> >> I'd be more than happy to help in improving this situation! >> >> Best regards, >> ~Niko >> -- >> Niko Ehrenfeuchter | Image Analysis Specialist | Biozentrum, University of >> Basel | Klingelbergstr. 50/70 | CH-4056 Basel >> Phone: +41 (61) 26 72673 | [hidden email] | >> www.biozentrum.unibas.ch | www.microscopynetwork.unibas.ch >> >> >> -- >> ImageJ mailing list: http://imagej.nih.gov/ij/list.html >> > > > -- Niko Ehrenfeuchter | Image Analysis Specialist | Biozentrum, University of Basel | Klingelbergstr. 50/70 | CH-4056 Basel Phone: +41 (61) 26 72673 | [hidden email] | www.biozentrum.unibas.ch | www.microscopynetwork.unibas.ch -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
Is it possible to export in the older .lsm format and preserve metadata? That format was essentially a customized tiff with metadata, so the performance should be similar to tiff. In addition, there are non-bioformats options for reading that data if you want to troubleshoot. We save all confocal data as .lsm rather than .czi.
Jay -----Original Message----- From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of Niko Ehrenfeuchter Sent: Monday, November 10, 2014 5:20 PM To: [hidden email] Subject: Re: Network drive slowness Dear All, maybe I should clarify a bit my previous message: I didn't intend *at all* to criticize Bio-Formats - in fact it is an outstanding project which is saving my ass every single day, and I *really* appreciate it a lot! My point was just about our general experience with using imaging data directly on network shares (which is even limited to the SMB/CIFS protocol, but this is probably true for the majority of the users here) and our persistent recommendation to work on local storage only. Actually, we've seen *much* worse behaviour in combination with network shares from commercial software (under paid maintenance!), where the vendor is even refusing to work on these problems. So my apologies if this came out the wrong way! Cheers ~Niko On 10.11.2014 23:49, Jason Swedlow wrote: > Dear All- > > Just to reiterate Curtis' point, OME welcomes reports of any issues > regarding Bio-Formats. See > > http://www.openmicroscopy.org/site/support/bio-formats5/about/bug-repo > rting.html > > or the Bio-Formats Forum at > > http://www.openmicroscopy.org/community/ > > Obviously, we won't be very helpful diagnosing the vagaries of your > network performance. We will respond to and work on issues related to > file opening times, etc. We've put a lot of work into CZI and other > multi-platform file formats recently-- any and all feedback welcome. > > Thanks! > > Cheers, > > Jason > > On Mon, Nov 10, 2014 at 10:29 PM, Niko Ehrenfeuchter < > [hidden email]> wrote: > >> Hi all, >> >> On 10.11.2014 21:13, Curtis Rueden wrote: >> >>> Hi Rafael, >>> >>> should I do a bug report for it? >>>> >>> >>> The CZI support is driven by Bio-Formats [1]. >>> >> >> from my experience with loading / saving data from network shares, >> I'm not sure wheter this is related to the CZI format. >> >> Loading images using Bio-Formats directly from a network drive >> usually takes around 10x longer than opening the same file from a >> local disk (whereas the storage backend of the network drive is >> potentially a lot faster than the conventional HDD's we're using on most of our computers). >> Sometimes it can even be up to 100x slower than opening data locally. >> >> The same story goes the other way round, when writing OME-TIFFs to a >> network share. We've even had situations where the OME-TIFFs seem to >> be written correctly but upon re-opening them they were broken. >> Problem was that this was not reliably reproducible, so I didn't have >> anything to file a reasonable bug report. >> >> The baseline of this is that we keep on telling our users to *always* >> transfer the data onto the machines they're using to process it, do >> their work, save it locally and then eventually transfer everything >> back to their network share to ensure maximum speed and reliability. >> >> I'd be more than happy to help in improving this situation! >> >> Best regards, >> ~Niko >> -- >> Niko Ehrenfeuchter | Image Analysis Specialist | Biozentrum, >> University of Basel | Klingelbergstr. 50/70 | CH-4056 Basel >> Phone: +41 (61) 26 72673 | [hidden email] | >> www.biozentrum.unibas.ch | www.microscopynetwork.unibas.ch >> >> >> -- >> ImageJ mailing list: http://imagej.nih.gov/ij/list.html >> > > > -- Niko Ehrenfeuchter | Image Analysis Specialist | Biozentrum, University of Basel | Klingelbergstr. 50/70 | CH-4056 Basel Phone: +41 (61) 26 72673 | [hidden email] | www.biozentrum.unibas.ch | www.microscopynetwork.unibas.ch -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
|
|
In reply to this post by Rafael Buono
Hi all,
It is nice to know that I am not alone in this. I will compile the information I have and post the bug report. As Niko, I also consider Bio-Formats an outstanding project. It plays a big role in my current workflow, and I might dare to say that it makes certain things possible that otherwise would be impossible for me to achieve. About the conversion from .czi to .lsm (or saving directly in .lsm): I am not sure if the metadata structure is the same. Does anyone else knows? Rafael -- ImageJ mailing list: http://imagej.nih.gov/ij/list.html |
«
Return to ImageJ
|
1 view|%1 views
| Free forum by Nabble | Edit this page |