test cases

[David Strasfeld]

I am using ImageJ as part of test protocols to verify the performance of
medical device software that I am developing. Do written procedures exist
for verifying ImageJ itself ? This would likely be a list of test cases
that would help a user verify that ImageJ is working as expected.

Any help is greatly appreciated.

--
David Strasfeld
*Lumicell*
Senior Scientist
80 William St.
Suite 2-260
Wellesley, MA 02481
office: 781.684.0229
mobile: [phone]
www.lumicell.com

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Michael Schmid]

Hi David,

there are at least two macros used for testing whether various ImageJ functions behave properly. I guess that Wayne applies them for all new versions of ImageJ.

  http://rsb.info.nih.gov/ij/macros/RegressionTests.txt

This macro makes a lot of tests of the macro language and also calls the second one,

  http://rsb.info.nih.gov/ij/macros/FilterTester.txt

This macro tests whether various processing functions do the same as it did the last time that its results were written to a file. It also does a lot of consistency tests, e.g. whether the operation the same for a rotated or flipped image (isotropic operations only) and whether an operation works the same way on stacks as on single images.
This test macro cannot discover whether a function was working improperly (but consistently) since the beginning, but here the large number of critical users is a good test criterion - at least for all of the basic ImageJ functions,
A further restriction is that standard test set only includes a limited number of operations that are tested, but you can add your own ones.

Nevertheless, even in spite of Wayne's work (I don't think that there is any other piece of software in the world where bug reports are handled so well and so quickly!), also ImageJ is not an exception to the rule that there is no bug-free software above a certain level of complexity...

Michael
________________________________________________________________
On Jul 30, 2014, at 18:00, David Strasfeld wrote:

> I am using ImageJ as part of test protocols to verify the performance of
> medical device software that I am developing. Do written procedures exist
> for verifying ImageJ itself ? This would likely be a list of test cases
> that would help a user verify that ImageJ is working as expected.
>
> Any help is greatly appreciated.
>
> --
> David Strasfeld
> *Lumicell*
> Senior Scientist
> 80 William St.
> Suite 2-260
> Wellesley, MA 02481
> office: 781.684.0229
> mobile: [phone]
> www.lumicell.com
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Gabriel Landini]

On Wednesday 30 Jul 2014 18:33:19 Michael Schmid wrote:

> there are at least two macros used for testing whether various ImageJ
> functions behave properly. I guess that Wayne applies them for all new
> versions of ImageJ.
>
>   http://rsb.info.nih.gov/ij/macros/RegressionTests.txt
>
> This macro makes a lot of tests of the macro language and also calls the
> second one,
>
>   http://rsb.info.nih.gov/ij/macros/FilterTester.txt

Do these run OK in your machine?

Hm... I get *lots* of FAIL...
Are there any options settings that need to be set by default?
I even renamed my StartupMacros.txt file to see if that was the case and some
FAIL do not appear anymore, but there still *many* in the Smooth, Mean,
Gaussian blur and many other filters.

Occasionally there is one window popping up saying '"Conver Stack to Images"
requires a stack' but there is a HSB stack at the forefront...

I wonder it this has to do with the blobs.gif image that had the wrong LUT and
corrected no so long ago.
If I remove the Samples folder the fails do not show up, but I still get one

FAIL: Smooth:   type=8bit select=all nThreads=1, single image: mean=63.34
expected=62.8975

I also get at popup:
FilterTester -Stopped on Fail
NOTE:
Options>Memory&Threads may be changed

I am using 1.49e12 under linux with Oracle Java 1.6,0_35.
I also tried 1.49d.
Cheers

Gabriel

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Michael Schmid]

Hi Gabriel,

one problem is the 'blobs.gif' file - if ImageJ opens that from ImageJ/samples/blobs.gif via an 'open' command, the image has no inverted LUT (as it should have) but the same appearance, thus different pixel values. In that case you get 'fail' messages for every test.

If it opens 'blobs.gif it via File>Open Samples, it has the correct LUT and pixel values.
Thus, in the macro, you should have the 'blobsFile' variable not pointing at the location where it can fine 'blobs.gif'. As we have the possibility of caching sample images in ImageJ now, the macro code to directly read the image is obsolete anyhow.

Apart from that problem, I got no error, neither with 1.49e1 nor with 1.49e12.
(OS X 10.6.8, Java 1.6.0_65 [64-bit])

Your other error message looks strange:
> FAIL: Smooth:   type=8bit select=all nThreads=1, single image: mean=63.34
> expected=62.8975


This means that the simple "Smooth" command does something wrong on your machine!

Process>Smooth should have exactly the same result as Process>Filters>Mean with a radius of 1 pixel, but it uses the ImageProcessor's 'smooth' method instead of the RankFilters.

Could you please try the two commands and see whether there is any difference between the results, and if so, what went wrong?

Michael
________________________________________________________________
On Jul 31, 2014, at 10:46, Gabriel Landini wrote:

> On Wednesday 30 Jul 2014 18:33:19 Michael Schmid wrote:
>
>> there are at least two macros used for testing whether various ImageJ
>> functions behave properly. I guess that Wayne applies them for all new
>> versions of ImageJ.
>>
>>  http://rsb.info.nih.gov/ij/macros/RegressionTests.txt
>>
>> This macro makes a lot of tests of the macro language and also calls the
>> second one,
>>
>>  http://rsb.info.nih.gov/ij/macros/FilterTester.txt
>
> Do these run OK in your machine?
>
> Hm... I get *lots* of FAIL...
> Are there any options settings that need to be set by default?
> I even renamed my StartupMacros.txt file to see if that was the case and some
> FAIL do not appear anymore, but there still *many* in the Smooth, Mean,
> Gaussian blur and many other filters.
>
> Occasionally there is one window popping up saying '"Conver Stack to Images"
> requires a stack' but there is a HSB stack at the forefront...
>
> I wonder it this has to do with the blobs.gif image that had the wrong LUT and
> corrected no so long ago.
> If I remove the Samples folder the fails do not show up, but I still get one
>
> FAIL: Smooth:   type=8bit select=all nThreads=1, single image: mean=63.34
> expected=62.8975
>
> I also get at popup:
> FilterTester -Stopped on Fail
> NOTE:
> Options>Memory&Threads may be changed
>
> I am using 1.49e12 under linux with Oracle Java 1.6,0_35.
> I also tried 1.49d.
> Cheers
>
> Gabriel
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Gabriel Landini]

On Thursday 31 Jul 2014 11:20:32 you wrote:

> one problem is the 'blobs.gif' file - if ImageJ opens that from
> ImageJ/samples/blobs.gif via an 'open' command, the image has no inverted
> LUT (as it should have) but the same appearance, thus different pixel
> values. In that case you get 'fail' messages for every test.
>
> If it opens 'blobs.gif it via File>Open Samples, it has the correct LUT and
> pixel values. Thus, in the macro, you should have the 'blobsFile' variable
> not pointing at the location where it can fine 'blobs.gif'. As we have the
> possibility of caching sample images in ImageJ now, the macro code to
> directly read the image is obsolete anyhow.
>
> Apart from that problem, I got no error, neither with 1.49e1 nor with
> 1.49e12. (OS X 10.6.8, Java 1.6.0_65 [64-bit])

So odd..
 

> Your other error message looks strange:
> > FAIL: Smooth:   type=8bit select=all nThreads=1, single image: mean=63.34
> > expected=62.8975
> This means that the simple "Smooth" command does something wrong on your
> machine!
>
> Process>Smooth should have exactly the same result as Process>Filters>Mean
> with a radius of 1 pixel, but it uses the ImageProcessor's 'smooth' method
> instead of the RankFilters.
>
> Could you please try the two commands and see whether there is any
> difference between the results, and if so, what went wrong?

If I do smoothing and mean filter radius 1 on an image I get exactly the same
result, either for the whole image or for ROIs in them, so that seems OK, but
the the FilterTester still fails with the error above.

run("Blobs (25K)");
getStatistics(area, mean);
print ("Original mean: "+mean);
run("Smooth");
getStatistics(area, mean);
print ("Smooth mean: "+mean);
close();
run("Blobs (25K)");
run("Mean...", "radius=1");
getStatistics(area, mean);
print ("Mean mean: "+mean);
close();

gives 103.2677 for both versions of the filter.

The other errors do not appear now that I copied the two macros into the
/macros folder (so perhaps there was an old version in that folder?).

Cheers

Gabriel

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Gabriel Landini]

Sorted!
There is another file, the FilterTesterTasklist.txt needed on top of the 2
macros.
I had an old version of it, and since the previous email mentioned only 2
macros, when I copied those 2 macros, the tasklist was obsolete.

I only noted this when I run the files in Fiji and it complained of a missing
file.

All tests are are passed fine. Phew!

Sorry for having brought up this false alarm.

Cheers

Gabriel

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html

[Michael Schmid]

Hi Gabriel,

yes, I agree, you had an old version of the FilterTesterTasklist.txt file causing that error. The FilterTester macro reads its tasklist from the ImageJ/macros directory.

There was a change of the 'Smooth' command in ImageJ 1.44g, 18 September 2010. From the release notes:
> Thanks to Barry DeZonia, fixed a bug that caused Process>Smooth to not round results with 8-bit images.


This explains the difference between 63.34 (new value) and 62.8975 (value before 1.44g; I also found this number in one of my old FilterTesterTasklist files).

Michael
________________________________________________________________
On Jul 31, 2014, at 13:07, Gabriel Landini wrote:

> On Thursday 31 Jul 2014 11:20:32 you wrote:
>> one problem is the 'blobs.gif' file - if ImageJ opens that from
>> ImageJ/samples/blobs.gif via an 'open' command, the image has no inverted
>> LUT (as it should have) but the same appearance, thus different pixel
>> values. In that case you get 'fail' messages for every test.
>>
>> If it opens 'blobs.gif it via File>Open Samples, it has the correct LUT and
>> pixel values. Thus, in the macro, you should have the 'blobsFile' variable
>> not pointing at the location where it can fine 'blobs.gif'. As we have the
>> possibility of caching sample images in ImageJ now, the macro code to
>> directly read the image is obsolete anyhow.
>>
>> Apart from that problem, I got no error, neither with 1.49e1 nor with
>> 1.49e12. (OS X 10.6.8, Java 1.6.0_65 [64-bit])
>
> So odd..
>
>> Your other error message looks strange:
>>> FAIL: Smooth:   type=8bit select=all nThreads=1, single image: mean=63.34
>>> expected=62.8975
>> This means that the simple "Smooth" command does something wrong on your
>> machine!
>>
>> Process>Smooth should have exactly the same result as Process>Filters>Mean
>> with a radius of 1 pixel, but it uses the ImageProcessor's 'smooth' method
>> instead of the RankFilters.
>>
>> Could you please try the two commands and see whether there is any
>> difference between the results, and if so, what went wrong?
>
> If I do smoothing and mean filter radius 1 on an image I get exactly the same
> result, either for the whole image or for ROIs in them, so that seems OK, but
> the the FilterTester still fails with the error above.
>
> run("Blobs (25K)");
> getStatistics(area, mean);
> print ("Original mean: "+mean);
> run("Smooth");
> getStatistics(area, mean);
> print ("Smooth mean: "+mean);
> close();
> run("Blobs (25K)");
> run("Mean...", "radius=1");
> getStatistics(area, mean);
> print ("Mean mean: "+mean);
> close();
>
> gives 103.2677 for both versions of the filter.
>
> The other errors do not appear now that I copied the two macros into the
> /macros folder (so perhaps there was an old version in that folder?).
>
> Cheers
>
> Gabriel
>
> --
> ImageJ mailing list: http://imagej.nih.gov/ij/list.html

--
ImageJ mailing list: http://imagej.nih.gov/ij/list.html