Issue with progress bar and locking files when saving

[Stein Rørvik]

Hello,

I see an issue with the progress bar when saving large files on external drives, and wonder if anyone else experience the same:

When saving a large TIFF file, say 5GB, onto an external drive, the ImageJ progress bar runs to completion suspiciously fast (about 3 seconds).
When it is "finished", the file-size is shown in the Windows as having its full size, but the file cannot be renamed or moved from there.
Also, there is no slice scroll bar at the bottom of the image in ImageJ, indicating that the file is still locked by ImageJ.

When the expected number of seconds for a complete save has elapsed (about the same number of seconds or minutes it usually takes to copy the file to or from the disk in question) the file is unlocked both in ImageJ and Windows explorer and can be moved and/or reopened showing its normal contents. So the file locking works as it should; it is the progress bar that is wrong.

I see this only happen when using external hard-drives; that is, directly connected to the PC via USB.
On internal disks this does not happen. Neither when saving to networked disks. The progress bar elapses correctly here.

So there seems to be a miscommunication between the OS and ImageJ when saving files to removable disks, giving the wrong progress bar speed.
This is a bit annoying when saving very large files (several tens of GB) as you have no indication to how long the file saving will actually need to complete.

I am using Windows 10/64, but the problem also occurs in Windows 7/64 and using older versions of ImageJ, and Fiji as well.

Stein

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Fred Damen]

Greetings Stein,

Most likely your USB disk is configured for performance. In this case, the
write to disk requests are cached by the OS and the throughput appears
better than if really is.  Most likely the close file request waits until
the cache has made it to disk.  This is configurable in the low level
system IO calls.  For USB drives you can configure for safe physical
removal, and the write requests will wait until the cache has made it to
disk; this will appear to work the way that you think it should.

Most drives are configured for performance, and thus why you should not
dop a hard shut off of you computer. The "remove device" icon on windows
and the /bin/sync command are there for insuring all the cache is flushed
to the physical disk.

Fred

https://www.howtogeek.com/410353/how-to-optimize-usb-storage-for-better-performance-on-windows-10/


On Tue, September 22, 2020 10:17 am, Stein Rørvik wrote:

> Hello,
>
> I see an issue with the progress bar when saving large files on external
> drives, and wonder if anyone else experience the same:
>
> When saving a large TIFF file, say 5GB, onto an external drive, the ImageJ
> progress bar runs to completion suspiciously fast (about 3 seconds).
> When it is "finished", the file-size is shown in the Windows as having its
> full size, but the file cannot be renamed or moved from there.
> Also, there is no slice scroll bar at the bottom of the image in ImageJ,
> indicating that the file is still locked by ImageJ.
>
> When the expected number of seconds for a complete save has elapsed (about
> the same number of seconds or minutes it usually takes to copy the file to
> or from the disk in question) the file is unlocked both in ImageJ and
> Windows explorer and can be moved and/or reopened showing its normal
> contents. So the file locking works as it should; it is the progress bar
> that is wrong.
>
> I see this only happen when using external hard-drives; that is, directly
> connected to the PC via USB.
> On internal disks this does not happen. Neither when saving to networked
> disks. The progress bar elapses correctly here.
>
> So there seems to be a miscommunication between the OS and ImageJ when
> saving files to removable disks, giving the wrong progress bar speed.
> This is a bit annoying when saving very large files (several tens of GB)
> as you have no indication to how long the file saving will actually need
> to complete.
>
> I am using Windows 10/64, but the problem also occurs in Windows 7/64 and
> using older versions of ImageJ, and Fiji as well.
>
> Stein
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Stein Rørvik]

Fred,

My USB disk is formatted as NTFS and is not regarded as a removable in the disk manager, so I cannot turn the caching on or off by any simple means. I assume the caching is always on. I agree to your suggestion that because of the disk caching, "the throughput appears better than it really is". It makes sense that when the file close request is sent by ImageJ, it is not honored until the file is actually residing on disk.

For USB disks I have always used "configure for performance" when that choice is available, and used the "safely remove media" icon to make sure that the cache is flushed. If the disk is accidentally removed while writing (for example if the disk is moved in a way that causes it to disconnect), the last written file is usually corrupt. Fortunately, the Windows disk repair feature is able to fix the file system errors that occur in these cases, so that we can re-write the file safely.

Stein

-----Original Message-----
Sent: 23. september 2020 01:37
Subject: Re: Issue with progress bar and locking files when saving

Greetings Stein,

Most likely your USB disk is configured for performance. In this case, the write to disk requests are cached by the OS and the throughput appears better than if really is.  Most likely the close file request waits until the cache has made it to disk.  This is configurable in the low level system IO calls.  For USB drives you can configure for safe physical removal, and the write requests will wait until the cache has made it to disk; this will appear to work the way that you think it should.

Most drives are configured for performance, and thus why you should not dop a hard shut off of you computer. The "remove device" icon on windows and the /bin/sync command are there for insuring all the cache is flushed to the physical disk.

Fred

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Fred Damen]

Greetings Stein,

The "configure for performance/safe removal' is a USB device setting and
should be independent of file system(s) and OS(s).  Whether, or how
exactly, an OS(s) and its file system implementation implements this
setting is a complex matter; albeit, this *should* be how you can turn
off/on caching on any USB drive.

In Unix file systems that existed long before NTFS came about, there were
IOTCL file handle calls that allowed setting the writes to be synchronous,
i.e., do not cache buffer and immediately return; the default being
asynchronous, i.e., cache and return, and thus the general fear of a hard
shutdown.

Fred

PS.  When I actually bother to set settings, I generally set this to safe
removal, as I am an idiot and tend to forget to safely remove the drive;
from the previously referenced article, it appears MS considers me normal.

On Thu, September 24, 2020 2:33 am, Stein Rørvik wrote:

> Fred,
>
> My USB disk is formatted as NTFS and is not regarded as a removable in the
> disk manager, so I cannot turn the caching on or off by any simple means.
> I assume the caching is always on. I agree to your suggestion that because
> of the disk caching, "the throughput appears better than it really is". It
> makes sense that when the file close request is sent by ImageJ, it is not
> honored until the file is actually residing on disk.
>
> For USB disks I have always used "configure for performance" when that
> choice is available, and used the "safely remove media" icon to make sure
> that the cache is flushed. If the disk is accidentally removed while
> writing (for example if the disk is moved in a way that causes it to
> disconnect), the last written file is usually corrupt. Fortunately, the
> Windows disk repair feature is able to fix the file system errors that
> occur in these cases, so that we can re-write the file safely.
>
> Stein
>
> -----Original Message-----
> Sent: 23. september 2020 01:37
> Subject: Re: Issue with progress bar and locking files when saving
>
> Greetings Stein,
>
> Most likely your USB disk is configured for performance. In this case, the
> write to disk requests are cached by the OS and the throughput appears
> better than if really is.  Most likely the close file request waits until
> the cache has made it to disk.  This is configurable in the low level
> system IO calls.  For USB drives you can configure for safe physical
> removal, and the write requests will wait until the cache has made it to
> disk; this will appear to work the way that you think it should.
>
> Most drives are configured for performance, and thus why you should not
> dop a hard shut off of you computer. The "remove device" icon on windows
> and the /bin/sync command are there for insuring all the cache is flushed
> to the physical disk.
>
> Fred
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html