Skip to content

WIP: External tests

Jonathan Wilkes requested to merge external-tests into master

Some API changes and (upcoming) patches to automate the testing of Pd-l2ork externals.

"External tests" might be a bit of a misnomer as the tools should be usable for testing internal classes as well.

Ideally, the tests would do the following:

  1. load all libs in the relevant libdir
  2. fetch the list of new creator names that came from loading the libraries in the libdir
  3. try to create an object using each new creator name (with required args if necessary-- unfortunately some externals fail)
  4. for each successfully instantiated object, enumerate the methods and call each one with whatever args are required.
  5. check the output against the output we saved from the previous release (of course this implies saving changed state in each release as necessary)
  6. for each dsp method, output a single block of audio using [switch~] (probably want to check against known good state, too, but that may create a lot of state)
  7. report any new method and new creators

The nice thing is that we can use a nonzero arg to "menuquit" in order to signal to CI that there was a failure in any of these steps.

This will allow us to more easily make changes to the codebase. Not only external libraries but also the build system and core, as any soft failures (of which there are many in the build system) that screw up something in an external lib will (eventually) trigger a hard failure in this test framework.

Difficulties

  1. State-saving and other side-effect-based methods. For now we might have to exclude certain methods like "write" or "read" that are more difficult to test using such an automated framework.
  2. "helper" creators and methods. Some objects have "proxy" objects with their own class names. However, I think such objects have their constructors disabled. It may be possible to add a check for that in [classinfo] and thus skip these. (Not sure what to do about "helper" methods, though.
  3. Probably lots of other problems I'm not thinking of yet...

Merge request reports