[Gegl-developer] Test plan

[yahvuu]

Hi,

here are some thoughts about a static test framework for gegl operations.

* Functional Tests
These can be as simple as a set of files including a composition,
reference output and a test description. A perl/ruby script feeds the
composition through the gegl binary and compares the output with
reference data. Optionally some PNGs + HTML are created for visual inspection.

Test Case:
  foo.xml                     composition
  foo.xxx                     reference data; format to be determinded
  foo.specs.yaml              test specification
  foo.input1.xxx              optional input data

Output:
  output/foo.xxx              compositing result
  output/foo.result.yaml      pass/fail info + basic statistic measures

Optional report output:
  report/foo.output.png
  report/foo.reference.png
  report/foo.diff.png
  report/foo.html


The test specification foo.specs.yaml looks like:

  comment: Regression test for Bug #1234
  errorbounds:
      peak-absolute:   1e-6


The result foo.result.yaml something like:

  testname : foo
  result   : pass
  errors   :
      peak-absolute    : 1e-10
      mean-square      : 1e-15
      differing-pixels : 123
      [...]
  timing  :
      total     : 500       # ms


The data format .xxx should be a float format understood by ImageMagick,
candidates so far are MIFF, PFM, FITS, TIFF.

Reference data is most easily created using Octave. Probably a 'make reference' target
will generate them from some ground truth m-files.


* Unit Tests
Unit tests for operations are most comfortably expressed as functional tests.
However, there are some tests which are common for all/many operations:
- rendering must be independent of tile size
- operations with spatial parameters should be scale invariance, i.e.
  applying a operation on a downscaled composition using parameters which have been
  downscaled by the same factor should give a good approximation of applying the
  unscaled operation on the unscaled composition and downscaling afterwards.
- neutral settings


Somewhat unrelated, i'd also like to install a round trip test for GEGL's
XML format, but haven't found a good way to deal with value rounding yet.


Comments, other things to consider, do's and don'ts?
Is somebody already working on this?


peter