Manipulate text files

[lechristophe]

Hi,

I want to manipulate big tab-delimited text files, like extract specific
columns, reorder them, generate new columns by simple calculations on
existing columns, and generate a new text file from this.This is typically
to translate data/results files obtained from various software, from one
format to another.

What is the most efficient to do this in IJ Javascript? For the moment I
import the whole text file as a string, make an array from it (parsing the
numerical values), manipulate the array, convert it back to a big string,
and save it. But for text files with 10^5-10^6 lines it takes ages to
process. Is it possible to read/write line by line from the input file to
the output file using IJ Javascript?

Alternatively, I'd be open to suggestions for using other than IJ
Javascript to do that, as it is not really image processing.

Thanks for your help,


Christophe

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Adam Hughes]

I'd recommend using pandas in python actually.  You also probably could do
this in SQL, Perl or R.  IMO, python pandas library will be the most
straightforward way.


On Fri, Nov 15, 2013 at 5:37 PM, Christophe Leterrier <
[hidden email]> wrote:

> Hi,
>
> I want to manipulate big tab-delimited text files, like extract specific
> columns, reorder them, generate new columns by simple calculations on
> existing columns, and generate a new text file from this.This is typically
> to translate data/results files obtained from various software, from one
> format to another.
>
> What is the most efficient to do this in IJ Javascript? For the moment I
> import the whole text file as a string, make an array from it (parsing the
> numerical values), manipulate the array, convert it back to a big string,
> and save it. But for text files with 10^5-10^6 lines it takes ages to
> process. Is it possible to read/write line by line from the input file to
> the output file using IJ Javascript?
>
> Alternatively, I'd be open to suggestions for using other than IJ
> Javascript to do that, as it is not really image processing.
>
> Thanks for your help,
>
>
> Christophe
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[lechristophe]

Hi ImageJers,

Answering my own question : using BufferedReader and BufferedWriter to read
the input file and write the output file line by line rather than
manipulating the whole file as a string, my javascript went from 791 s (!)
to 3.4 s for the processing of a 122,000 lines text file. So I guess it was
worth trying!

Here is the code for those interested, I'd be happy if you have comments or
enhancements.

Cheers,

Christophe




importClass(Packages.java.io.FileReader);
importClass(Packages.java.io.BufferedReader);
importClass(Packages.java.io.File);
importClass(Packages.java.io.FileWriter);
importClass(Packages.java.io.BufferedWriter);

// definition of constants
var ExposTime = 0.016;

// definition of column number for retrieved values in the input file
var XcCol = 3;
var YcCol = 4;
var FrameCol = 12;

// get the input file
var od = new OpenDialog("Choose a loc text file", "");
var inName = od.getFileName();
var inDir = od.getDirectory();
var inPath = inDir + inName;

var startTime = new Date().getTime();

// create the output file
var outPath = inDir + inName.substring(0, inName.length()-4) +
"_processed.txt";
var outFile = new File(outPath);
if (!outFile.exists()) outFile.createNewFile();

// initialize reader and writer
var br = new BufferedReader(new FileReader(inPath));
var bw = new BufferedWriter(new FileWriter(outFile));

// strips the header (skip first line of input file)
inLine = br.readLine();

// read/process/write line by line (read in the while condition)
while ((inLine = br.readLine()) != null) {
   // split input lines into values
   inValues = inLine.split("\t");
   // process the frame number column into frame time (* by exposure time)
   frameTime = Double.parseDouble(inValues[FrameCol]) * ExposTime;
   // output line : copy Xc and Yc directly (no parsing), append calculated
frame time
   outLine = inValues[XcCol] + "\t" + inValues[YcCol] + "\t" + frameTime;
   // write to output file
   bw.write(outLine);
   bw.newLine();
}

// close the reader and writer
br.close();
bw.close();

var stopTime = new Date().getTime();
var Time = stopTime - startTime;
IJ.showStatus("end after " + Time + " ms");




On Fri, Nov 15, 2013 at 11:37 PM, Christophe Leterrier <
[hidden email]> wrote:

> Hi,
>
> I want to manipulate big tab-delimited text files, like extract specific
> columns, reorder them, generate new columns by simple calculations on
> existing columns, and generate a new text file from this.This is typically
> to translate data/results files obtained from various software, from one
> format to another.
>
> What is the most efficient to do this in IJ Javascript? For the moment I
> import the whole text file as a string, make an array from it (parsing the
> numerical values), manipulate the array, convert it back to a big string,
> and save it. But for text files with 10^5-10^6 lines it takes ages to
> process. Is it possible to read/write line by line from the input file to
> the output file using IJ Javascript?
>
> Alternatively, I'd be open to suggestions for using other than IJ
> Javascript to do that, as it is not really image processing.
>
> Thanks for your help,
>
>
> Christophe
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html