Posted by
lechristophe on
Nov 18, 2013; 10:24pm
URL: http://imagej.273.s1.nabble.com/Manipulate-text-files-tp5005575p5005594.html
Hi ImageJers,
Answering my own question : using BufferedReader and BufferedWriter to read
the input file and write the output file line by line rather than
manipulating the whole file as a string, my javascript went from 791 s (!)
to 3.4 s for the processing of a 122,000 lines text file. So I guess it was
worth trying!
Here is the code for those interested, I'd be happy if you have comments or
enhancements.
Cheers,
Christophe
importClass(Packages.java.io.FileReader);
importClass(Packages.java.io.BufferedReader);
importClass(Packages.java.io.File);
importClass(Packages.java.io.FileWriter);
importClass(Packages.java.io.BufferedWriter);
// definition of constants
var ExposTime = 0.016;
// definition of column number for retrieved values in the input file
var XcCol = 3;
var YcCol = 4;
var FrameCol = 12;
// get the input file
var od = new OpenDialog("Choose a loc text file", "");
var inName = od.getFileName();
var inDir = od.getDirectory();
var inPath = inDir + inName;
var startTime = new Date().getTime();
// create the output file
var outPath = inDir + inName.substring(0, inName.length()-4) +
"_processed.txt";
var outFile = new File(outPath);
if (!outFile.exists()) outFile.createNewFile();
// initialize reader and writer
var br = new BufferedReader(new FileReader(inPath));
var bw = new BufferedWriter(new FileWriter(outFile));
// strips the header (skip first line of input file)
inLine = br.readLine();
// read/process/write line by line (read in the while condition)
while ((inLine = br.readLine()) != null) {
// split input lines into values
inValues = inLine.split("\t");
// process the frame number column into frame time (* by exposure time)
frameTime = Double.parseDouble(inValues[FrameCol]) * ExposTime;
// output line : copy Xc and Yc directly (no parsing), append calculated
frame time
outLine = inValues[XcCol] + "\t" + inValues[YcCol] + "\t" + frameTime;
// write to output file
bw.write(outLine);
bw.newLine();
}
// close the reader and writer
br.close();
bw.close();
var stopTime = new Date().getTime();
var Time = stopTime - startTime;
IJ.showStatus("end after " + Time + " ms");
On Fri, Nov 15, 2013 at 11:37 PM, Christophe Leterrier <
[hidden email]> wrote:
> Hi,
>
> I want to manipulate big tab-delimited text files, like extract specific
> columns, reorder them, generate new columns by simple calculations on
> existing columns, and generate a new text file from this.This is typically
> to translate data/results files obtained from various software, from one
> format to another.
>
> What is the most efficient to do this in IJ Javascript? For the moment I
> import the whole text file as a string, make an array from it (parsing the
> numerical values), manipulate the array, convert it back to a big string,
> and save it. But for text files with 10^5-10^6 lines it takes ages to
> process. Is it possible to read/write line by line from the input file to
> the output file using IJ Javascript?
>
> Alternatively, I'd be open to suggestions for using other than IJ
> Javascript to do that, as it is not really image processing.
>
> Thanks for your help,
>
>
> Christophe
>
--
ImageJ mailing list:
http://imagej.nih.gov/ij/list.html