Recursive processing of a directory not working correctly

[Matthew Jacobs]

I've been combing this forum looking at other examples of code similar to mine but I still have not been able to solve the issue.

I am processing images in Fiji/imageJ using a macro code. The code is meant to recursively search a given directory for image files (in this case .vsi files) and convert them to .tiff using the built in bioformats functions.

I have cobbled the following code together from several examples (I am not a programmer, I have a biology degree), and I am returning an error that the specified file does not exist. Printout and debug console show the error occurs on the run("Bio-Formats... line of the macro.

For reference:

All of my images have the following general directory structure:

~/Brain_01/Slice_01/Slice_01_01.vsi

An actual example is as follows for the second image in this set:

~/K259_tlx3cre/K259_1of3_02/K259_1of3_01_02.vsi

I really, really appreciate all the help you guys provide! I am completely stuck right now and am just banging my head against the keyboard at this point. Also, I think I've followed all the rules for posting. If not please just point out the error of my ways.

Finally here is the macro code I have written so far is attached as a text file which formats well in notepad++, and I will also paste the code here but have no idea how it will format once the post goes up.


Dialog.create("File type");
Dialog.addString("File suffix: ", ".vsi", 5);
Dialog.show();
suffix = Dialog.getString();

inDir = getDirectory("Choose Directory Containing " + suffix + " Files ");
outDir = getDirectory("Choose Directory for TIFF Output ");
setBatchMode(true);
processFiles(inDir, outDir, "");
print("Done!");

function processFiles(inBase, outBase, sub) {
  flattenFolders = true; // this flag controls output directory structure
  print("Processing folder: " + sub);
  list = getFileList(inBase + sub);
  if (!flattenFolders) File.makeDirectory(outBase + sub);
  for (i=0; i<list.length; i++) {
    path = sub + list[i];
    //upath = toUpperCase(path); only need if the file extension is case sensitive
    upath = path; //avoids the previous line
    if (endsWith(path, "/")) {
      processFiles(inBase, outBase, path);
     
    }
    else if (endsWith(upath, suffix)) {

    print("Importing " + suffix + " = " + list[i]);

        run("Bio-Formats Windowless Importer", "open=inBase + path color_mode=Default view=Hyperstack stack_order=XYCZT");
       
        print("Blue...");
        selectImage(1);
        title1 = getTitle();
        run("Blue");

        print("Green...");
        selectImage(2);
        title2 = getTitle();
        run("Green");

        print("Red...");
        selectImage(3);
        title3 = getTitle();
        run("Red");
               
        print("Merging Channels...");
        run("Merge Channels...", "red=&title3 green=&title2 blue=&title1 gray=*None* cyan=*None* magenta=*None* yellow=*None* create keep");

        print("Converting to RGB");
        run("RGB Color");
               
        saveAs("Tiff", outbase + path);
        run("Close All");
    }
  }
}

[Krs5]

Dear matsojr22(???),

Your line: run("Bio-Formats Windowless Importer", "open=inBase + path color_mode=Default view=Hyperstack stack_order=XYCZT");

should read; run("Bio-Formats Windowless Importer", "open="+inBase + path+" color_mode=Default view=Hyperstack stack_order=XYCZT"); as 'inBase' and 'path' contain a string that will create the name of the file you want to open; you don't want to open the file with the name: "inBase + path.vsi"


There is also a typo in saveAs("Tiff", outbase + path); which needs outBase instead of outbase.

Hope this solves your problem.

Best wishes

Kees


Dr Ir K.R. Straatman
Senior Experimental Officer
Advanced Imaging Facility
Centre for Core Biotechnology Services
University of Leicester
http://www2.le.ac.uk/colleges/medbiopsych/facilities-and-services/cbs/lite/aif


-----Original Message-----
From: ImageJ Interest Group [mailto:[hidden email]] On Behalf Of matsojr22
Sent: 20 May 2014 00:15
To: [hidden email]
Subject: Recursive processing of a directory not working correctly

I've been combing this forum looking at other examples of code similar to mine but I still have not been able to solve the issue.

I am processing images in Fiji/imageJ using a macro code. The code is meant to recursively search a given directory for image files (in this case .vsi
files) and convert them to .tiff using the built in bioformats functions.

I have cobbled the following code together from several examples (I am not a programmer, I have a biology degree), and I am returning an error that the specified file does not exist. Printout and debug console show the error occurs on the /run("Bio-Formats.../ line of the macro.

For reference:

All of my images have the following general directory structure:

~/Brain_01/Slice_01/Slice_01_01.vsi

An actual example is as follows for the second image in this set:

~/K259_tlx3cre/K259_1of3_02/K259_1of3_01_02.vsi

I really, really appreciate all the help you guys provide! I am completely stuck right now and am just banging my head against the keyboard at this point. Also, I think I've followed all the rules for posting. If not please just point out the error of my ways.

Finally here is the macro code I have written so far is attached as a text file which formats well in notepad++, and I will also paste the code here but have no idea how it will format once the post goes up.


Dialog.create("File type");
Dialog.addString("File suffix: ", ".vsi", 5); Dialog.show(); suffix = Dialog.getString();

inDir = getDirectory("Choose Directory Containing " + suffix + " Files "); outDir = getDirectory("Choose Directory for TIFF Output "); setBatchMode(true); processFiles(inDir, outDir, ""); print("Done!");

function processFiles(inBase, outBase, sub) {
  flattenFolders = true; // this flag controls output directory structure
  print("Processing folder: " + sub);
  list = getFileList(inBase + sub);
  if (!flattenFolders) File.makeDirectory(outBase + sub);
  for (i=0; i<list.length; i++) {
    path = sub + list[i];
    //upath = toUpperCase(path); only need if the file extension is case sensitive
    upath = path; //avoids the previous line
    if (endsWith(path, "/")) {
      processFiles(inBase, outBase, path);
     
    }
    else if (endsWith(upath, suffix)) {

    print("Importing " + suffix + " = " + list[i]);

        run("Bio-Formats Windowless Importer", "open=inBase + path color_mode=Default view=Hyperstack stack_order=XYCZT");
       
        print("Blue...");
        selectImage(1);
        title1 = getTitle();
        run("Blue");

        print("Green...");
        selectImage(2);
        title2 = getTitle();
        run("Green");

        print("Red...");
        selectImage(3);
        title3 = getTitle();
        run("Red");
               
        print("Merging Channels...");
        run("Merge Channels...", "red=&title3 green=&title2 blue=&title1
gray=*None* cyan=*None* magenta=*None* yellow=*None* create keep");

        print("Converting to RGB");
        run("RGB Color");
               
        saveAs("Tiff", outbase + path);
        run("Close All");
    }
  }
}



--
View this message in context: http://imagej.1557.x6.nabble.com/Recursive-processing-of-a-directory-not-working-correctly-tp5007797.html
Sent from the ImageJ mailing list archive at Nabble.com.

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Matthew Jacobs]

Hey Dr. Straatman,

I really appreciate your help! The changes you suggested make perfect sense now that I see them, but I am still throwing an error while importing the file.

When I run the macro now, I get an error from fiji saying "The specified file (C:\Users\Matt) does not exist."

It aborts at the same step.

I get a printout; "Importing .vsi = K259_1of3_01_02.vsi" then I get the error. I am sure that this is an easy fix, and probably has something to do with the whole directory string not getting recorded correctly, but I have not been able to fix it today.

Any suggestions on how to fix this new error?

Thanks again to anyone and everyone who can help!

Matt

[Matthew Jacobs]

Let me also clarify that I am currently testing the macro on files I have copied to the desktop...

Input directory is C:\Users\Matt Jacobs\Desktop\K259_tlx3cre_Rabies\
Output directory is C:\Users\Matt Jacobs\Desktop\fiji output\

Also, I'm just recognizing that the error is not telling me that "C:\Users\Matt Jacobs\" does not exist... it is telling me the file "C:\Users\Matt" does not exist...

Am I having trouble just because I have spaces in the directory?

Thanks again,
Matt

[Michael Schmid]

Hi Matt,

you can put square brackets around the directory+filename, then spaces don't do any harm.

run("Bio-Formats Windowless Importer", "open=["+inBase + path+"] color_mode=Default view=Hyperstack stack_order=XYCZT");

If you have
  open=C:\Users\Matt Jacobs\Desktop\K259_tlx3cre_Rabies\whateverFile.vsi
the space is taken as a delimiter,
  open=C:\Users\Matt
and the next statement would be
  Jacobs\Desktop\K259_tlx3cre_Rabies\whateverFile.vsi
As the second part is not any keyword, it is probably ignored.
So it should be
  open=[C:\Users\Matt Jacobs\Desktop\K259_tlx3cre_Rabies\whateverFile.vsi]

---

One more point:
in your macro, you have
  if (endsWith(path, "/"))
to check whether this is a directory.
It would be better to use
  if(File.isDirectory(path))
The current code seems to work also under Windows because ImageJ replaces the backslash by a slash, but relying on this is not a nice solution.

Michael
________________________________________________________________
On May 20, 2014, at 22:42, matsojr22 wrote:

> Let me also clarify that I am currently testing the macro on files I have
> copied to the desktop...
>
> Input directory is C:\Users\Matt Jacobs\Desktop\K259_tlx3cre_Rabies\
> Output directory is C:\Users\Matt Jacobs\Desktop\fiji output\
>
> Also, I'm just recognizing that the error is not telling me that
> "C:\Users\Matt Jacobs\" does not exist... it is telling me the file
> "C:\Users\Matt" does not exist...
>
> Am I having trouble just because I have spaces in the directory?
>
> Thanks again,
> Matt
>
>
>
> --
> View this message in context: http://imagej.1557.x6.nabble.com/Recursive-processing-of-a-directory-not-working-correctly-tp5007797p5007811.html
> Sent from the ImageJ mailing list archive at Nabble.com.
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Kenton Arkill]

Hi Matt
Not sure if it helps but I use the following for a very similar problem
with .dm3 files where I want to make jpg mini versions for people (I've
taken out the size change bit):
dir = getDirectory("Choose a Directory");
list = getFileList(dir);
setBatchMode(true);
for (i=0; i<list.length; i++) {
path = dir+list[i];
if(endsWith(path, ".dm3")){
open(path);   // OR OTHER IMPORT PLUGGIN
run("Enhance Contrast", "saturated=3"); // Change the 0.35 to a higher
number if jpgs too gray eg 2 or 5
// OR OTHER TRANSFORMATIONS
dotIndex = lastIndexOf(path, ".");
if (dotIndex!=-1){
path = substring(path, 0, dotIndex); // remove extension
save(path+".jpg");
}
close();
}
}
showStatus("Done");


On 21 May 2014 08:39, Michael Schmid <[hidden email]> wrote:

> Hi Matt,
>
> you can put square brackets around the directory+filename, then spaces
> don't do any harm.
>
> run("Bio-Formats Windowless Importer", "open=["+inBase + path+"]
> color_mode=Default view=Hyperstack stack_order=XYCZT");
>
> If you have
>   open=C:\Users\Matt Jacobs\Desktop\K259_tlx3cre_Rabies\whateverFile.vsi
> the space is taken as a delimiter,
>   open=C:\Users\Matt
> and the next statement would be
>   Jacobs\Desktop\K259_tlx3cre_Rabies\whateverFile.vsi
> As the second part is not any keyword, it is probably ignored.
> So it should be
>   open=[C:\Users\Matt Jacobs\Desktop\K259_tlx3cre_Rabies\whateverFile.vsi]
>
> ---
>
> One more point:
> in your macro, you have
>   if (endsWith(path, "/"))
> to check whether this is a directory.
> It would be better to use
>   if(File.isDirectory(path))
> The current code seems to work also under Windows because ImageJ replaces
> the backslash by a slash, but relying on this is not a nice solution.
>
> Michael
> ________________________________________________________________
> On May 20, 2014, at 22:42, matsojr22 wrote:
>
> > Let me also clarify that I am currently testing the macro on files I have
> > copied to the desktop...
> >
> > Input directory is C:\Users\Matt Jacobs\Desktop\K259_tlx3cre_Rabies\
> > Output directory is C:\Users\Matt Jacobs\Desktop\fiji output\
> >
> > Also, I'm just recognizing that the error is not telling me that
> > "C:\Users\Matt Jacobs\" does not exist... it is telling me the file
> > "C:\Users\Matt" does not exist...
> >
> > Am I having trouble just because I have spaces in the directory?
> >
> > Thanks again,
> > Matt
> >
> >
> >
> > --
> > View this message in context:
> http://imagej.1557.x6.nabble.com/Recursive-processing-of-a-directory-not-working-correctly-tp5007797p5007811.html
> > Sent from the ImageJ mailing list archive at Nabble.com.
> >
> > --
> > ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Jan Eglinger]

Hi Matt and Michael,

as I pointed out on my answer to your question on stackoverflow.com [1]
a few days ago, Fiji's script editor contains a macro template for
processing folders recursively. You can open it in the script editor via
'Templates > Macros > Process Folder'.

 > One more point:
 > in your macro, you have
 >    if (endsWith(path, "/"))
 > to check whether this is a directory.
 > It would be better to use
 >    if(File.isDirectory(path))

I included Michael's suggestion in a pull request to said template:
https://github.com/fiji/Script_Editor/pull/1

Thanks and cheers,
Jan

[1]: http://stackoverflow.com/a/23709728/1919049



On 21.05.2014, 9:39 AM, Michael Schmid wrote:

> Hi Matt,
>
> you can put square brackets around the directory+filename, then spaces don't do any harm.
>
> run("Bio-Formats Windowless Importer", "open=["+inBase + path+"] color_mode=Default view=Hyperstack stack_order=XYCZT");
>
> If you have
>    open=C:\Users\Matt Jacobs\Desktop\K259_tlx3cre_Rabies\whateverFile.vsi
> the space is taken as a delimiter,
>    open=C:\Users\Matt
> and the next statement would be
>    Jacobs\Desktop\K259_tlx3cre_Rabies\whateverFile.vsi
> As the second part is not any keyword, it is probably ignored.
> So it should be
>    open=[C:\Users\Matt Jacobs\Desktop\K259_tlx3cre_Rabies\whateverFile.vsi]
>
> ---
>
> One more point:
> in your macro, you have
>    if (endsWith(path, "/"))
> to check whether this is a directory.
> It would be better to use
>    if(File.isDirectory(path))
> The current code seems to work also under Windows because ImageJ replaces the backslash by a slash, but relying on this is not a nice solution.
>
> Michael
> ________________________________________________________________
> On May 20, 2014, at 22:42, matsojr22 wrote:
>
>> Let me also clarify that I am currently testing the macro on files I have
>> copied to the desktop...
>>
>> Input directory is C:\Users\Matt Jacobs\Desktop\K259_tlx3cre_Rabies\
>> Output directory is C:\Users\Matt Jacobs\Desktop\fiji output\
>>
>> Also, I'm just recognizing that the error is not telling me that
>> "C:\Users\Matt Jacobs\" does not exist... it is telling me the file
>> "C:\Users\Matt" does not exist...
>>
>> Am I having trouble just because I have spaces in the directory?
>>
>> Thanks again,
>> Matt
>>
>>
>>
>> --
>> View this message in context: http://imagej.1557.x6.nabble.com/Recursive-processing-of-a-directory-not-working-correctly-tp5007797p5007811.html
>> Sent from the ImageJ mailing list archive at Nabble.com.
>>
>> --
>> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Matthew Jacobs]

Thanks Jan! I appreciate the help!
I actually started a few weeks ago with the template you suggested. I got it to work, but it seems to only like scanning all files within a single folder. I was unable to get it to scan recursively through a directory which is why I started writing my own code.

I used your suggestion and got the error due to spaces in my path. I have things working now and will post a reply with the working code for future readers to benefit from.

Thanks again,
Matt

[Matthew Jacobs]

Thanks to all of you I was able to get my code working well enough to do what I need to do! The code below is working well enough to batch convert a directory full of .vsi files to .tiff using the functions I need! You guys are the best.

Dialog.create("File type");
Dialog.addString("File suffix: ", ".vsi", 5);
Dialog.show();
suffix = Dialog.getString();

inDir = getDirectory("Choose Directory Containing " + suffix + " Files ");
outDir = getDirectory("Choose Directory for TIFF Output ");
setBatchMode(true);
processFiles(inDir, outDir, "");
print("Done!");

function processFiles(inBase, outBase, sub) {
  flattenFolders = true; // this flag controls output directory structure
  print("Processing folder: " + sub);
  list = getFileList(inBase + sub);
  if (!flattenFolders) File.makeDirectory(outBase + sub);
  for (i=0; i<list.length; i++) {
    path = sub + list[i];
    //upath = toUpperCase(path); only need if the file extension is case sensitive
    upath = path; //avoids the previous line
    if (File.isDirectory(path)) {
      processFiles(inBase, outBase, path);
     
    }
    else if (endsWith(upath, suffix)) {

    print("Importing " + suffix + " = " + list[i]);

                run("Bio-Formats Windowless Importer", "open=["+inBase + path+"] color_mode=Default view=Hyperstack stack_order=XYCZT");
               
                print("Stack to Images...");
                run("Stack to Images");
       
                print("Blue...");
                selectImage(1);
                title1 = getTitle();
                run("Blue");

                print("Green...");
                selectImage(2);
                title2 = getTitle();
                run("Green");

                print("Red...");
                selectImage(3);
                title3 = getTitle();
                run("Red");
               
                print("Merging Channels...");
                run("Merge Channels...", "red=&title3 green=&title2 blue=&title1 gray=*None* cyan=*None* magenta=*None* yellow=*None* create keep");

                print("Converting to RGB");
                run("RGB Color");
               
                saveAs("Tiff", outBase + path);
                run("Close All");
    }
  }
}

[Matthew Jacobs]

Oops... just a heads up to anyone trying to use this code in the future. I tried applying the suggested:

if (File.isDirectory(path))

in place of the endswith / from my original code and it is not working. I clearly need to change that bit of code  somewhat to make it work.

If you replace that single line with the original from my first post, things should work (even if it is not elegant in any way haha).

[Michael Entrup]

Hello Matthew,

in your macro 'path' is only the string that represents the file-names
and folder-names starting at the selected directory. You have to use
'inBase + path' to get the full path.

 > if (File.isDirectory(inBase + path))

will work. If you have a look at

 > run("Bio-Formats Windowless Importer", "open=["+inBase + path+"]
color_mode=Default view=Hyperstack stack_order=XYCZT");

there you use the same combination.

Best regards
Michael


Am 23.05.2014 01:12, schrieb Matthew Jacobs:

> Oops... just a heads up to anyone trying to use this code in the future. I
> tried applying the suggested:
>
> if (File.isDirectory(path))
>
> in place of the endswith / from my original code and it is not working. I
> clearly need to change that bit of code  somewhat to make it work.
>
> If you replace that single line with the original from my first post, things
> should work (even if it is not elegant in any way haha).
>
>
>
> -----
> Matthew Jacobs,
> Research Assistant
> Callaway Neuroscience Laboratory
> The Salk Institute for Biological Studies
> --
> View this message in context: http://imagej.1557.x6.nabble.com/Recursive-processing-of-a-directory-not-working-correctly-tp5007797p5007867.html
> Sent from the ImageJ mailing list archive at Nabble.com.
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Matthew Jacobs]

Michael,

You are awesome! Thanks for the info! I assumed it was something like this, but I have not had the time to try anything in our biophotonics core yet.

You are the best! Thanks again,
Matt

On May 23, 2014 2:07 AM, "Michael Entrup [via ImageJ]" <ml-node+[hidden email]> wrote:

Hello Matthew,

in your macro 'path' is only the string that represents the file-names
and folder-names starting at the selected directory. You have to use
'inBase + path' to get the full path.

 > if (File.isDirectory(inBase + path))

will work. If you have a look at

 > run("Bio-Formats Windowless Importer", "open=["+inBase + path+"]
color_mode=Default view=Hyperstack stack_order=XYCZT");

there you use the same combination.

Best regards
Michael


Am 23.05.2014 01:12, schrieb Matthew Jacobs:

> Oops... just a heads up to anyone trying to use this code in the future. I
> tried applying the suggested:
>
> if (File.isDirectory(path))
>
> in place of the endswith / from my original code and it is not working. I
> clearly need to change that bit of code  somewhat to make it work.
>
> If you replace that single line with the original from my first post, things
> should work (even if it is not elegant in any way haha).
>
>
>
> -----
> Matthew Jacobs,
> Research Assistant
> Callaway Neuroscience Laboratory
> The Salk Institute for Biological Studies
> --
> View this message in context: http://imagej.1557.x6.nabble.com/Recursive-processing-of-a-directory-not-working-correctly-tp5007797p5007867.html
> Sent from the ImageJ mailing list archive at Nabble.com.
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

---

If you reply to this email, your message will be added to the discussion below:

http://imagej.1557.x6.nabble.com/Recursive-processing-of-a-directory-not-working-correctly-tp5007797p5007868.html

To unsubscribe from Recursive processing of a directory not working correctly, click here.
NAML