Reducing virtual stack

[Stein Rørvik]

I have some large folders (usually 46GB each) of raw µCT data which I open as virtual stacks to quickly inspect if the data looks okay, before reconstructing it. This works as expected.

These days I am working on samples that require the instrument flux correction to be turned off, due to an asymmetry in the way the samples are mounted. I therefore need to play through the stack to inspect to what degree the actual flux has drifted, to decide if the data needs to be re-acquired or can be used as is. Since the folder resides on a network disk which is too slow to allow slice animation, I reduce it using the Image → Stacks → Tools → Reduce command. This works, but I have noticed that the result stack is no longer virtual, but residing in memory. This makes the operation slow, especially if I choose a small reduction factor, as a lot of data then has to be read.

So, is the conversion to memory resident stack really necessary, as I assume a virtual stack is just a list of pointers to files?  Then just the list could be reduced, instead of loading all files as a reduced stack.

Stein

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[jmutterer]

Dear Stein,
If you could list the remote directory, create a reduced list of files and
open this list with the "Stack from list" command.
See this example macro that builds and opens a list of files as a virtual
stack:

https://webcache.googleusercontent.com/search?q=cache:jd_q0eWMg2YJ:https://imagej.nih.gov/ij/macros/VirtualStackFromList.txt+&cd=1&hl=en&ct=clnk&gl=fr

Jerome.


On Sun, 2 May 2021 at 17:30, Stein Rørvik <[hidden email]> wrote:

> I have some large folders (usually 46GB each) of raw µCT data which I open
> as virtual stacks to quickly inspect if the data looks okay, before
> reconstructing it. This works as expected.
>
> These days I am working on samples that require the instrument flux
> correction to be turned off, due to an asymmetry in the way the samples are
> mounted. I therefore need to play through the stack to inspect to what
> degree the actual flux has drifted, to decide if the data needs to be
> re-acquired or can be used as is. Since the folder resides on a network
> disk which is too slow to allow slice animation, I reduce it using the
> Image → Stacks → Tools → Reduce command. This works, but I have noticed
> that the result stack is no longer virtual, but residing in memory. This
> makes the operation slow, especially if I choose a small reduction factor,
> as a lot of data then has to be read.
>
> So, is the conversion to memory resident stack really necessary, as I
> assume a virtual stack is just a list of pointers to files?  Then just the
> list could be reduced, instead of loading all files as a reduced stack.
>
> Stein
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Stein Rørvik]

Thanks Jerome,
yes this approach will work. I did not think of that solution.

I can then use a macro for doing what I want, by writing a reduced file list to a temporary file:
-----------------------------------------------------------------
tempfile = getDirectory("temp") + "list.txt";
path = "E:/SR 2021-05 BatCat T 2/SR20210503_BatCat-SiG_3mm_T-W-100kV-30uA-24dB-nf [2021-05-03 21.50.18]/";
files = getFileList(path);
images = Array.filter(files, ".tif");
stepsize = 15;
stepsize = getNumber("Reduction Factor: ", stepsize);
showMessage(tempfile);
f = File.open(tempfile);
for (i = 0; i < images.length; i += stepsize) {
        print(i, images[i]);
        print(f, path + images[i]);
}
File.close(f);
run("Stack From List...", "open=&tempfile use");
rename(File.getName(path));
-----------------------------------------------------------------

I still think this would have been simpler if Stack → Reduce could preserve the virtual state of a stack, like Stack → Reverse does.

Stein

-----Original Message-----
Sent: 2. mai 2021 20:39
Subject: Re: Reducing virtual stack

Dear Stein,
If you could list the remote directory, create a reduced list of files and open this list with the "Stack from list" command.
See this example macro that builds and opens a list of files as a virtual
stack:

https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwebcache.googleusercontent.com%2Fsearch%3Fq%3Dcache%3Ajd_q0eWMg2YJ%3Ahttps%3A%2F%2Fimagej.nih.gov%2Fij%2Fmacros%2FVirtualStackFromList.txt%2B%26cd%3D1%26hl%3Den%26ct%3Dclnk%26gl%3Dfr&amp;data=04%7C01%7Cstein.rorvik%40sintef.no%7Cece35ac6675340d3750508d90d99b525%7Ce1f00f39604145b0b309e0210d8b32af%7C1%7C1%7C637555776092804557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=llxwnYbx6XRXHOi3kaYFgp9iDL%2Bt6zrojEcmZ%2B1G%2B6s%3D&amp;reserved=0

Jerome.


On Sun, 2 May 2021 at 17:30, Stein Rørvik <[hidden email]> wrote:

> I have some large folders (usually 46GB each) of raw µCT data which I
> open as virtual stacks to quickly inspect if the data looks okay,
> before reconstructing it. This works as expected.
>
> These days I am working on samples that require the instrument flux
> correction to be turned off, due to an asymmetry in the way the
> samples are mounted. I therefore need to play through the stack to
> inspect to what degree the actual flux has drifted, to decide if the
> data needs to be re-acquired or can be used as is. Since the folder
> resides on a network disk which is too slow to allow slice animation,
> I reduce it using the Image → Stacks → Tools → Reduce command. This
> works, but I have noticed that the result stack is no longer virtual,
> but residing in memory. This makes the operation slow, especially if I
> choose a small reduction factor, as a lot of data then has to be read.
>
> So, is the conversion to memory resident stack really necessary, as I
> assume a virtual stack is just a list of pointers to files?  Then just
> the list could be reduced, instead of loading all files as a reduced stack.
>
> Stein
>
> --
> ImageJ mailing list:
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fimage
> j.nih.gov%2Fij%2Flist.html&amp;data=04%7C01%7Cstein.rorvik%40sintef.no
> %7Cece35ac6675340d3750508d90d99b525%7Ce1f00f39604145b0b309e0210d8b32af
> %7C1%7C1%7C637555776092804557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw
> MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=iw
> aW9JqRrZqqI%2BSQWEbIJtHD7HEZ1kF7fQxGIB0F2ew%3D&amp;reserved=0
>

--
ImageJ mailing list: https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fimagej.nih.gov%2Fij%2Flist.html&amp;data=04%7C01%7Cstein.rorvik%40sintef.no%7Cece35ac6675340d3750508d90d99b525%7Ce1f00f39604145b0b309e0210d8b32af%7C1%7C1%7C637555776092804557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=iwaW9JqRrZqqI%2BSQWEbIJtHD7HEZ1kF7fQxGIB0F2ew%3D&amp;reserved=0

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html