[Automated Testing Framework](#automated-testing-framework) [Change to Double Precision Floating Point Format](#change-to-double-precision-floating-format) [API for HTML5 Web Apps](#api-for-html5-web-apps) [K12 Mode](#k12-mode) [Profile and Optimize Purr Data for Realtime Safety](#profile-and-optimize-purr-data-for-realtime-safety) [ASCII art to Purr Data diagram conversion](#ascii-art-to-purr-data-diagram-conversion) [Navigation of "Wireless" Objects](#navigation-of-wireless-objects) [Terminal REPL](#terminal-repl) [Core Accessibility](#core-accessibility) [Purr Data Message and DSP Profiler](#purr-data-message-and-dsp-profiler) [Streamlining Purr Data GUI-Pd communication](#streamlining-purr-data-gui-pd-communication) [Vintage Platform Audio Emulation Library](#vintage-platform-audio-emulation-library) [Library for Data-Over-Audio Communication](#data-over-audio-messaging) [Interaction with Audio Plugins](#plugins) [JIT-compiled signal graph for the audio engine](#jit) Automated Testing Framework --------------------------- ### Goal Build a testing framework that covers most methods of internal and external classes. This should include the "dsp" method of all signal-based externals, so that we can compare the block output of each across commits to serve as both a regression and code coverage test. ### Benefits This will help developers catch bugs earlier in the development process, leading to overall cleaner code in merge requests and less disruption to users from regressions. ### Difficulty level Moderate. One challenge is that not all classes have sane defaults for their methods. For example, the internal class [until] will iterate in an infinite loop when the "bang" method is invoked. Even with external libraries, such poor defaults are entrenched. A robust testing system must therefore work around these challenges. Another challenge is that many external libraries have never been tested at all. Many are not 64-bit clean. A maintainable and sensible testing system will therefore take an incremental approach, starting with the most well-tested and popular libraries and branching out from there to the more obscure ones. ### Languages Testing framework can be fully implemented in Purr Data itself. However, some familiarity with shell scripting tools and the platforms supported by Purr Data will be useful (i.e., Windows, OSX, and GNU/Linux). Change to Double Precision Floating Format ------------------------------------------ ### Goal Change Purr Data's numeric data type from single-precision floating point to double-precision floating point. ### Benefits Purr Data has a single numeric type to represent numeric data. Changing this type to double-precision has many benefits: * increased precision in the internal sample format used in the audio engine * increases the maximum index into an array of audio data without losing precision. Current single-point floating point indicies severely limit array indexing, requiring the user to manage both an index and an onset into the array to retain precision. * creates a consistent interface between the core audio engine and the HTML5 GUI which itself uses double-precision floating point for represented numbers. * makes it easier for cases where users want integers, as a double-precision floating point type can itself contain a 32-bit integer. (Single-precision floating point cannot.) * makes it easier for third-party developers to interface with Purr Data. For example, the upcoming MIDI revision uses 32-bit integers which can simply be converted to double (while single-precision would lose data and require a more complicated approach). ### Details This requires some changes to the core, leveraging Katja Vetter's previous work to find performant replacements for the core DSP classes. It also requires changes to some of the external libraries-- such as freeverb and others-- which either have separate APIs based on numeric precision or rely on the numeric type being single-precision. This work will benefit immensely from the automated testing framework listed above. That framework will immediately reveal many crashers and discrepancies in output of the "dsp" methods of external classes. Developers can then use the output of those tests to get a better sense of the scope of the work and prioritize which libraries to refactor first. ### Difficulty Hard. This is mitigated somewhat by the prior work of Katja Vetter and others who have done double-precision implementations for most of the core audio engine. Still, this change touches many different parts of the code and will no doubt result in some insidious crashes and runtime errors that will require a well-rounded knowledge of the C programming language, floating point type standards, and Purr Data's internals including the core DSP API and algorithms. ### Languages C, some C++ (for some of the external libraries coded in C++), plus basic shell scripting familiarity (both `find` and `grep` will come in handy here). API for HTML5 Web Apps ---------------------- ### Goal Create an API and simple user interface for loading a web app that can communicate with a Pd patch. ### Details Currently Pd patches are essentially just an HTML5 page with a window/application menu. It should be possible for the user to instead open an arbitrary HTML document that can communicate through a javascript interface with a Pd patch. Implementation suggestion: since the current pdgui.js module already has an interface for sending messages to the Pd backend, we should probably leverage that to communicate from the web app to Pd. Then it's only necessary to add a feature that allows a Pd Patch to send messages back to the web app. The user should then be able to load an HTML file using the existing File->Open dialog. Then the web app will need something like the following interface: * ability to send messages to Pd (already exists) * ability to tell Pd to load a particular patch in the same directory as the web app (or perhaps using Pd's search path). * ability to close a Pd patch * ability to specify whether the patch we're loading should default to being visible or hidden * ability to set callbacks for the web app to receive the following events: * the Pd patch we wanted to open has loaded * the Pd patch we loaded has closed ### Languages Javascript. Some basic knowledge of C will be helpful if we need to add a method in the Purr Data engine. However, such methods will probably be quite simple and won't require implementing any complex algorithms in the C language. K12 Mode -------- ### Goal Port the K12 Mode that Pd-l2ork introduced. ### Details Currently we ship a lot of the K12 abstractions with Purr Data. However, I never ported the special mode that starts the program up in a "K12" mode that replaces the window/application menubar with a more user-friendly, clickable object menu. That mode makes it easier for beginners to get started. In porting it, we can take advantage of the new HTML5 GUI: * The menu itself can leverage HTML/CSS to be more responsive, autohide, etc. * The entire K12 Mode could itself just be a web-app separate from normal Pd app operation. To figure out which is best, it's probably helpful to know more about the history of K12 mode, and whether the ability to also open and interact with normal Pd objects turned out to be an important feature. Ivica Bukvic has a lot of experience using this mode when teaching kids, so we can look to him for some more insight into the right path to take. Either way, this feature vastly improves the usability of Purr Data for new users, giving them a way to quickly generate sounds and experiment with the interface. ### Difficulty Hard. A lot of the initial implementation of K12 mode was made in the core engine in C. This will require at least reading a lot of the source code and making sense of the changes made for K12 mode, plus possibly refactoring some of that code so that it can be handled in the GUI instead. On the other hand, there is a working version of the old K12 mode which can be consulted as a reference implementation. ### Languages Javascript, C, CSS, HTML. Profile and Optimize Purr Data for Realtime Safety -------------------------------------------------- ### Goal Find the "pain points" in Purr Data's core message dispatcher and audio engine, and optimize the code where possible to improve the realtime scheduling of DSP. ### Details The core DSP algorithms of Purr Data tend to avoid system calls, unnecessary branching, and other calls which would make the performance of the audio process unreliable. However, there are many areas of Purr Data which are not optimized for realtime scheduling. The first order of business would be to profile Purr Data in several areas to see where problems may be. Some likely culprits are the following: * parse time for incoming messages from GUI to the audio/messaging engine * overhead of sending messages to the GUI, especially for visual arrays and drawing instructions for data structures * overhead of sending streams of "motion" messages from GUI to Pd to track mouse position * overhead of walking the linked list of objects in order to find the object under the mouse/mouseclick * overhead of tracking mouse position over a visual array * overhead of calculating bboxes in the audio process as opposed to GUI * overhead of calculating text positioning in audio process as opposed to GUI * overhead of counting UTF-8 code points in audio process to calculate line breaks * overhead of handling incoming/outgoing socket traffic in the same thread as the audio scheduler * probably other areas Once we get a sense of the pain points, we can tackle the problem of how to optimize in a maintainable manner. Some references for this: * [Guilio Moro's work](https://github.com/giuliomoro/purrdata/commit/9dc3223ece79be5f60a6a629450b52a79b9e050c) on using a separate thread for the socket connections from GUI to the core audio engine (plus all the other socket connections like netsend, netreceive, etc.) * Guilio Moro's work on a threaded microsleep for the event loop. * [Guilio Moro's work](https://github.com/giuliomoro/purrdata/commits/simpler-motion) to simplify GUI communication by handling more of the mouse motion/click logic in the GUI. This results in fewer messages from GUI to audio engine, but still requires a linked-list walk in the audio engine to find the relevant object. * Matt Barber's link to a new cosine wave generator algorithm that may be more performant than the current implementation. (Not so important for current performance, but this may become more relevant once we switch to double-precision for block samples.) * Possibility to vectorize DSP algos using SIMD. Also more crude experiments by just hand-unrolling one or two classes when N=64 (i.e., the most common block size) and measuring the performance impact (if any). Note: There may be some overlap with the other profiling idea listed below. Developers for both ideas may therefore benefit by periodically sharing their work with each other. ### Difficulty Moderate to hard. The initial profiling will take some time but isn't particularly challenging. Making changes to the core audio engine, however, will require some knowledge of Linux system interfaces and some of Purr Data's internals. Properly assessing and testing any threading techniques in C is also frought with peril and will require extreme care in order to keep the code maintainable and avoid insidious bugs. ### Languages Basic to advanced shell scripting, C, plus familiarity with profiling tools like gprof and others. ASCII art to Purr Data diagram conversion ----------------------------------------- ### Goal Make a GUI interface where the user can either type or paste in ASCII art for a Purr Data diagram and have it converted to a floating selection in the current Purr Data diagram. ### Details Often Purr Data users rely on ASCII art in forums, mailing lists and other documentation to give examples of Purr Data programs. This is convenient because it's quicker than taking a screenshot of the Purr Data GUI and takes up less space, too. The ability for the user to type ASCII art into an object box and get a new selection of a Purr Data object chain would be useful. This would let users paste ASCII art from the mailing list directly into the interface. It would also make it easier to create Purr Data diagrams using only the keyboard. ### Bonus Goal Make it possible to convert a subset of Purr Data diagrams into ASCII art. Bonus Goal Details: Unfortunately Purr Data allows arbitrary positioning of objects on an integer-based x/y axis. Thus, not all Purr Data diagrams can be represented as ASCII art. The bonus goal would add a feature to check if the currently selected Purr Data diagram is able to be represented using only ASCII art. This could be either a) a button that does an analysis, or b) realtime feedback to let the user know when they have positioned an object or connection in such a way that makes ASCII art impossible. This is a non-trivial problem. For example, one could add a button to Purr Data that would constrain object positioning to a predefined grid. Then one could disallow objects to overlap with each other. Still, the connections among the various objects (i.e., the arcs which connect the nodes) could overlap in such a way that makes unambiguous ASCII art impossible. ### Difficulty Easy to hard. This idea runs the gamut-- a simple GUI-side parser for simple ASCII art is a quick exercise. Handling some of the more complicated art with multiple and/or crossed connections is moderate. Scanning the mailing lists for examples and covering the majority of them is hard. Additionally, converting an existing Purr Data graphical diagram back to ASCII art is a difficult problem. This will require reading up on the current state of the art and comparing various approaches from other fields like OCR and some of the current work converting Photoshop mockups to HTML pages. However, Purr Data diagrams tend to be much simpler than arbitrary Photoshop art, so the problem may be more tractable here than in other fields. ### Languages Javascript if converting ASCII art to Purr Data diagram, C if attempting to convert from graphical diagram *back* to ASCII art. Navigation of Wireless Objects -------------------------------- ### Goal Make it possible for the user to navigate among all extant wireless objects that are bound to a particular symbol. ### Details Purr Data diagrams typically consist of objects-- English words in boxes-- connected by Bezier curves. In a complex diagram, however, the Bezier curves can end up obscuring the flow of the data in the diagram. For these instances, "wireless" objects like `[send]` and `[receive]` may be used to send data from one object to another without making an explicit connection. However, such "nonlocal" connections can quickly make diagrams difficult to maintain. It would be helpful if the user could query a particular "wireless" object to find out how many other objects it communicates with. Internally, all related "wireless" objects are bound to the same immutable symbol. This makes it easy to find all related objects for a given symbol. Probably the easiest UI for this would be a button to query all related connections. Once Purr Data's engine finishes the search, it can send back the ids of all the objects it found. Finally, the gui can just print a list of hyperlinks to the console. When the user clicks any hyperlink, the corresponding object can be highlighted (or the relevant diagram brought up with the object in it highlighted). ### Difficulty Moderate. Purr Data's bindelem and pd_bind API are a little tricky if one hasn't used them before. However, once that interface is understood it is all that is needed to return a complete list of all nonlocal objects bound to a particular symbol. The remainder of the problem is easy and can be done in Javascript. However, this problem "upgrades" gracefully. Once the initial UI is done, it can be tested on users and iteratively improved from there. ### Languages Javascript, C. Terminal REPL ------------- ### Goal Make a little REPL interface with which the user can interact with Purr Data programs and program state. ### Details Purr Data is being used in situations where the hardware is an embedded device. While the current GUI runs on most common hardware including the RPI, there are situations where it would be more convenient to simply interact using a text interface (locally or over ssh). The user can already communicate with Purr Data's audio engine over a socket connection. So the "read" and "evaluate" part already exists. However, Purr Data does not print a response nor loop in this situation. Additionally, the UX of sending raw messages to Purr Data's interpreter is quite lacking. The syntax for creating new environments and objects was not meant to be used directly. Objects are referenced by index number, and the diagrams themselves must be referenced using hex identifiers. It would be very beneficial to create a REPL UI that is more user-friendly and well-documented/specified. This way Purr Data users can always interact with and create programs easily on any embedded device, even if there is no direct display. (This would also be very handy for debugging purposes.) ### Difficulty Moderate. An initial REPL can be created with the current Purr Data API, but it won't be particularly user-friendly. To achieve that requires more work and an understanding of Purr Data's message dispatching system. ### Languages C, some shell scripting. Core Accessibility ------------------ ### Goal Ensure that Purr Data is accessible by coupling accessibility with the core UX ### Details Especially because Purr Data is a graphical environment, it's important to make sure the core functionality is accessible. Rather than tack on accessibility as an afterthought, Purr Data should have a UX that makes accessibility features a generally useful part of the programming environment. For example: how does one navigate the nodes of a Purr Data diagram? There should be a way to navigate among the nodes and their connections without using the mouse. If we make sure that each element in the diagram is annotated we can tackle accessibility and keyboard navigation at the same time. Thus, a robust keyboard navigation implementation will help make it possible for screen readers to give meaningful information about each node in the graph. Note: there may be some overlap with the REPL idea above, as the REPL could provide a sensible way for a user to traverse the diagram as an alternative to using the GUI. ### Difficulty Moderate. For example, it will be necessary to study the current GUI implementation to figure out how to extend it to add keyboard navigation. It will also be necessary to study pre-existing approaches to making SVG diagrams accessible and study the current state of HTML5 tools that facilitate this. ### Languages Javascript, HTML, CSS. Some basic C knowledge may be required to send a richer set of data about each object from the core to the GUI. However, there is already an interface that can do this-- it just needs to be hooked into the GUI. Purr Data Message and DSP Profiler ---------------------------------- ### Goal Measure the time it takes for each object in a Purr Data diagram to process its data and display the results in the diagram. ### Details Purr Data users would benefit greatly from the ability to profile their programs while they are running. This is easy to do for the program as a whole, but challenging to do per-object. A successful implementation of this feature will give an accurate measure of the time it takes each object to process its incoming data. This needs to support all of Purr Data's platforms: Windows, OSX, and GNU/Linux. A successful implementation will also be performant enough that the measurements themselves don't impact the realtime operation of Purr Data itself. A now defunct fork of Pure Data called ["DesireData"](http://artengine.ca/desiredata/) did an initial implementation of this idea using the x86 RDTSC instruction. (Though its unlikely this feature was actually stable at the time DesireData was in active development.) Though this instruction is no longer considered reliable on modern machines, the overall approach taken by DesireData of adding a field to the t_gobj struct for storing this timing data is probably a sound starting point. Note: There may be overlap with the other profiling idea listed above, as developers on both ideas will probably be using the same tools and can therefore benefit by periodically sharing their work with each other. ### Bonus goal Figure out a way to meaningfully profile DSP objects. DSP objects typically process data at a high sample rate (44,100 is common) so displaying the data in a user-friendly and meaningful way is tricky. ### Difficulty Moderate to Hard. This feature touches the main artery of the message dispatching system, and the bonus goal would touch the main DSP routine. In both cases realtime scheduling deadlines must be taken into account by careful profiling. ### Languages C for the profiling business logic, HTML5 for displaying the results in the GUI. Streamlining Purr Data GUI-Pd communication ------------------------------------------ ### Goal Move some of the GUI callbacks out of Purr Data's audio engine so that GUI interaction is less likely to cause dropouts. ### Details The Pd GUI is heavily entangled with the Pd audio backend. In fact, most of the "gestures" performed on the GUI are passed straight to the Pd engine for processing. The GUI gestures are then "analyzed" by the audio thread, which may respond with triggering a GUI action, changing the state of an object, or nothing. For instance, each mouse move triggers a `motion` message to the Pd backend, handled by `canvas_motion()` in `g_editor.h`. This calls `canvas_doclick(... doit = false)`, which in turn iterates through all the objects on the patch and asks each of them "does the cursor happen to be on top of you?" (`canvas_findhitbox()`/`canvas_hitbox()`), calling a callback function (`w_getrectfn()`) for each of those objects. Now, most of the time the cursor is not on an object (or patch cable) and the calls to `w_getrecfn()` have no effect, except for wasting CPU power. There are two notable exceptions: a) when the mouse pointer is on top of an object, or one of its inlets or outlets, or on top of a patch cord, or on top of a GUI object, the mouse pointer may change, plus, e.g.: flickering inlets/outlets. b) some objects use the calls to `w_getrecfn()` to track mouse position (e.g.: [mousestate] from cyclone). The above results in a plethora of CPU cycles being wasted, which may cause dropouts when using small blocksizes and/or embedded platforms. Besides - and perhaps most importantly - it seems the wrong approach that some GUI-specific actions (like the ones at a) above) have to be processed and validated by the audio engine, within the audio thread. We could therefore think of an improvement to the Purr-data architecture, where the GUI stuff (e.g.: point a) above) is delegated uniquely to the GUI, which makes for lower CPU usage and potentially a more responsive GUI. For instance, the GUI could be designed to only send `motion` messages when the mouse is on top of an object and it could send alongside with it the Pd "tag" of the object, so that `w_getrectfn()` can be called only for the relevant object). The optimal approach would involve handling all the graphics effects (in/outlet animation, mouse pointers) directly within the GUI, and only sending `motion` messages when something relevant to the Pd engine is _actually_ happening (e.g.: when connecting objects). Additionally, and looking forward, in order to address point b), objects that need to track mouse position should declare this at initialization and should be kept in a dedicated list, so that the `motion` messages from the GUI can be delivered only to them with minimal CPU waste. An alternative - and probably worse - approach to the problem, which could reduce peak CPU usage, would be for the Pd audio engine to maintain a "rasterized" cached map of the patch (e.g.: by calling `w_getrecfn()` for each object at each pixel). This way, it could simply look up the cached map in response to each `motion` message. The cache could be recomputed in a separate thread every time after a new object or patch cord is created. Threading issues may arise here, in case one of the objects is deleted while the cached map is being built. ### Difficulty: Moderate. This project comes with a number of challenges, including: potential threading issues between the engine and the GUI, the necessity to re-write the C code of some objects, providing complete documentation for creators of externals, maintaining - where possible (e.g.: excluding objects that track mouse position) - backwards compatibility with Pd. More details on a previous attempt at addressing the problem can be found [here]( http://disis.music.vt.edu/pipermail/l2ork-dev/2017-June/001383.html). ### Languages Javascript and C. Vintage Platform Audio Emulation Library ---------------------------------------- ### Goal Create a library with objects that emulate the hardware from old hardware like the atari 2600, NES, and others. ### Details There are a lot of resources online for emulating old hardware. Purr Data would benefit by having a library that provides a consistent interface for objects that take input into an emulation of a piece of hardware and output one or more audio signals. If possible, it would be beneficial if most of the interface could be built as a set of abstractions. That way more developers would be able to understand and improve the library. There is a TIA chip emulator written in C in externals/mmonoplayer that can be used as a starting point. ### Difficulty: Moderate. ### Languages Pd (Purr Data is a fork of the software Pure Data-- the visual language itself is usually referred to as Pd.) Also, C. Data Over Audio Messaging ------------------------- ### Goal Create a library that allows two instances of Purr Data to pass data messages to each other using sound as the transmission medium. ### Details Pd messages consist mainly of space-separated numbers and symbols. Semicolons mark the end of a message. Sometimes it would be helpful to be able to pass messages from one instance of Purr Data to another-- especially if each instance is on a different machine in the same room. This is currently done either by setting up socket listener/receiver between the two instances or by leveraging a separate message-passing system outside of Purr Data. Since Purr Data is concerned mainly with analyzing and sythesizing sound, machines running Purr Data typically have a mic and speakers connected to a running instance. If it were possible for the user to simply create objects which send/receive messages by sending audio signals to/from each other it would greatly simplify sending at least small amounts of data between machines. ### Difficulty: Moderate. ### Languages Pd (Purr Data is a fork of the software Pure Data-- the visual language itself is usually referred to as Pd.) However, the library may also be written in C. Interaction with Audio Plugins ------------------------------ ### Goal Make sure that Purr Data has a well-documented way to accept input from standard audio plugin APIs like VST, LADSP, and LV2. Also make sure Purr Data can be used as a plugin in other environments. ### Details There are multiple audio plugin APIs that aim at seamlessly mixing and matching audio filters, synthesizers, and analysis tools in different languages and applications. Purr Data has some libraries to interface with at least two of these standards (VST and LADSPA). There is also a library for LV2 that Purr Data can leverage. However, not all of these libraries run on all the supported platforms (OSX, Windows, Linux). Purr Data also has all the APIs necessary to act as a plugin itself in other applications. But work must be done to ensure this works properly and that it is properly documented how to do it. ### Difficulty: Moderate. There is a lot of pre-existing technology here, but it needs to be tested rigorously on all platforms. ### Languages C. Also, familiarity with shell scripting as well as Gnu make. Interaction with Audio Plugins ------------------------------ ### Goal Make sure that Purr Data has a well-documented way to accept input from standard audio plugin APIs like VST, LADSP, and LV2. Also make sure Purr Data can be used as a plugin in other environments. ### Details There are multiple audio plugin APIs that aim at seamlessly mixing and matching audio filters, synthesizers, and analysis tools in different languages and applications. Purr Data has some libraries to interface with at least two of these standards (VST and LADSPA). There is also a library for LV2 that Purr Data can leverage. However, not all of these libraries run on all the supported platforms (OSX, Windows, Linux). Purr Data also has all the APIs necessary to act as a plugin itself in other applications. But work must be done to ensure this works properly and that it is properly documented how to do it. ### Difficulty: Moderate. There is a lot of pre-existing technology here, but it needs to be tested rigorously on all platforms. ### Languages C. Also, familiarity with shell scripting as well as Gnu make. JIT-compiled Signal Graph for the Audio Engine ---------------------------------------------- ### Goal Leverage LLVM to make a jit-compiler for Purr Data's DSP graph. ### Details Alex Norman has shown that it is possible to build a jit compiler for one of Pd's "workhorse" libraries-- the "expr" library. This library essentially lets users specify a mathematical expression that can operate on both vectors and on the sample level. The current library parses the tokens of the expression and requires a separate function call for each unit. This is similar to the way Pd's DSP graph itself gets executed. The jit compiler produces assembly which has superior performance to the current "expr" library. Extending that process to the entire signal graph (or at least core classes within it) would benefit performance on a greater level. ### Difficulty: Moderate to difficult. Since there is pre-existing work done with the jit "expr" library, a good start would be to help test that library and improve it. Once the student gains a mental model of the process the student can begin to extend it to Purr Data's DSP graph itself. ### Languages C.