Results Table CSV file- tab to comma-delimited?

[Gabriel Landini]

Hi
We have a plugin that takes a Results table and saves it.
That is a CVS file, which is tab-delimited. We then load that file to weka from
the plugin.

In weka v3.6 this cvs file could be loaded without any problems.  When updating
to 3.8  or 3.9 there is an error:
 
An error occurred: java.io.IOException: wrong number of values. Read 24,
expected 23, read Token[EOL], line 2 Problem encountered on line: 2      

The file is the same one, it can be loaded in one version of weka, but not in
the other.

I am guessing that the weka CVS reader is not behaving or expects a comma
delimited file, not just tab-delimited.

The code we are using to do this is:
ResultsTable rt= ResultsTable.getResultsTable();
int[] assignments = new int[rt.size()];
String IJdir = IJ.getDirectory("imagej");
rt.saveAs(IJdir+"datatobeclustered3.cvs");
CSVLoader loader = new CSVLoader();
loader.setSource(new File(IJdir+"datatobeclustered3.cvs"));

Does anbody have any idea of what might be going on?

Is there any way to save the Results table as comma-delimited instead of tab-
delimited?

Or is there a platform-agnostic way of converting tabs to , ?

Many thanks

Gabriel

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[John Hayes]

Hi Gabriel,

According to this you can change the field separate with the CSVLoader::setFieldSeparator function.
http://javadoc.imagej.net/Fiji/weka/core/converters/CSVLoader.html

HTH,
John

> On Feb 7, 2017, at 2:05 PM, Gabriel Landini <[hidden email]> wrote:
>
> Hi
> We have a plugin that takes a Results table and saves it.
> That is a CVS file, which is tab-delimited. We then load that file to weka from
> the plugin.
>
> In weka v3.6 this cvs file could be loaded without any problems.  When updating
> to 3.8  or 3.9 there is an error:
>
> An error occurred: java.io.IOException: wrong number of values. Read 24,
> expected 23, read Token[EOL], line 2 Problem encountered on line: 2      
>
> The file is the same one, it can be loaded in one version of weka, but not in
> the other.
>
> I am guessing that the weka CVS reader is not behaving or expects a comma
> delimited file, not just tab-delimited.
>
> The code we are using to do this is:
> ResultsTable rt= ResultsTable.getResultsTable();
> int[] assignments = new int[rt.size()];
> String IJdir = IJ.getDirectory("imagej");
> rt.saveAs(IJdir+"datatobeclustered3.cvs");
> CSVLoader loader = new CSVLoader();
> loader.setSource(new File(IJdir+"datatobeclustered3.cvs"));
>
> Does anbody have any idea of what might be going on?
>
> Is there any way to save the Results table as comma-delimited instead of tab-
> delimited?
>
> Or is there a platform-agnostic way of converting tabs to , ?
>
> Many thanks
>
> Gabriel
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Brandon Hurr]

The error suggests it is with your line ends. Like you have an extra column
that isn't named. Could you attach or make a gist of one of your files for
inspection?

Maybe it ignored this error before and now it doesn't.

I've not used these functions before though so I'm really not sure.

B
On Tue, Feb 7, 2017 at 11:07 Gabriel Landini <[hidden email]> wrote:

> Hi
> We have a plugin that takes a Results table and saves it.
> That is a CVS file, which is tab-delimited. We then load that file to weka
> from
> the plugin.
>
> In weka v3.6 this cvs file could be loaded without any problems.  When
> updating
> to 3.8  or 3.9 there is an error:
>
> An error occurred: java.io.IOException: wrong number of values. Read 24,
> expected 23, read Token[EOL], line 2 Problem encountered on line: 2
>
> The file is the same one, it can be loaded in one version of weka, but not
> in
> the other.
>
> I am guessing that the weka CVS reader is not behaving or expects a comma
> delimited file, not just tab-delimited.
>
> The code we are using to do this is:
> ResultsTable rt= ResultsTable.getResultsTable();
> int[] assignments = new int[rt.size()];
> String IJdir = IJ.getDirectory("imagej");
> rt.saveAs(IJdir+"datatobeclustered3.cvs");
> CSVLoader loader = new CSVLoader();
> loader.setSource(new File(IJdir+"datatobeclustered3.cvs"));
>
> Does anbody have any idea of what might be going on?
>
> Is there any way to save the Results table as comma-delimited instead of
> tab-
> delimited?
>
> Or is there a platform-agnostic way of converting tabs to , ?
>
> Many thanks
>
> Gabriel
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html
>

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Gabriel Landini]

On Tuesday, 7 February 2017 14:12:32 GMT John Hayes wrote:
> According to this you can change the field separate with the
> CSVLoader::setFieldSeparator function.
> http://javadoc.imagej.net/Fiji/weka/core/converters/CSVLoader.html

Dear John and Brandon,
Thanks for your prompt replies. That setOptions method was exactly what I was
looking for... not sure how I missed it. The CSV file was fine, the separator
was \t and it seems that it was  autodetected in 3.6 but not in 3.8 or 3.9.

Just in case it is useful to somebody else, loading the tab-delimited file is
done using the options "-F" and "/t" :
ResultsTable rt= ResultsTable.getResultsTable();
int[] assignments = new int[rt.size()];
String IJdir = IJ.getDirectory("imagej");
rt.saveAs(IJdir+"datatobeclustered3.cvs");
CSVLoader loader = new CSVLoader();
String[] opts = new String[]{"-F", "\t"};
loader.setOptions(opts);
loader.setSource(new File(IJdir+"datatobeclustered3.cvs"));

Many thanks again for your help.

Gabriel

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Rasband, Wayne (NIH/NIMH) [E]]

> On Feb 7, 2017, at 2:05 PM, Gabriel Landini <[hidden email]> wrote:
>
> Hi
> We have a plugin that takes a Results table and saves it.
>
>   <snip>
>
> The code we are using to do this is:
> ResultsTable rt= ResultsTable.getResultsTable();
> int[] assignments = new int[rt.size()];
> String IJdir = IJ.getDirectory("imagej");
> rt.saveAs(IJdir+"datatobeclustered3.cvs");
> CSVLoader loader = new CSVLoader();
> loader.setSource(new File(IJdir+"datatobeclustered3.cvs"));
>
> Does anbody have any idea of what might be going on?
>
> Is there any way to save the Results table as comma-delimited instead of tab-
> delimited?

The saveAs() and save() methods of the ResultsTable class save in CSV (comma-separated values) format if the name ends with ".csv”. Your code uses “.cvs” so the file will be saved in tab-dilimited format.

-wayne


--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Gabriel Landini]

On Wednesday, 8 February 2017 21:37:27 GMT Rasband, Wayne (NIH/NIMH) [E]
wrote:
> The saveAs() and save() methods of the ResultsTable class save in CSV
> (comma-separated values) format if the name ends with ".csv”. Your code
> uses “.cvs” so the file will be saved in tab-dilimited format.
 
I see!! Thanks Wayne for spotting that out!
The weka difference is, however, there. Newer versions do not autodetect the
tab-delimited files.

Cheers

Gabriel

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html