Goal: Find the "pain points" in Purr Data's core message dispatcher and
### Goal
Find the "pain points" in Purr Data's core message dispatcher and
audio engine, and optimize the code where possible to improve the realtime
scheduling of DSP.
Details: The core DSP algorithms of Purr Data tend to avoid system calls,
### Details
The core DSP algorithms of Purr Data tend to avoid system calls,
unnecessary branching, and other calls which would make the performance of
the audio process unreliable. However, there are many areas of Purr Data
which are not optimized for realtime scheduling.
...
...
@@ -203,24 +216,28 @@ Note: There may be some overlap with the other profiling idea listed below.
Developers for both ideas may therefore benefit by periodically sharing their
work with each other.
Difficulty: Moderate to hard. The initial profiling will take some time but
### Difficulty
Moderate to hard. The initial profiling will take some time but
isn't particularly challenging. Making changes to the core audio engine,
however, will require some knowledge of Linux system interfaces and some of
Purr Data's internals. Properly assessing and testing any threading techniques
in C is also frought with peril and will require extreme care in order to
keep the code maintainable and avoid insidious bugs.
Languages: basic to advanced shell scripting, C, plus familiarity with
### Languages
Basic to advanced shell scripting, C, plus familiarity with
profiling tools like gprof and others.
ASCII art to Purr Data diagram conversion
-----------------------------------------
Goal: make a GUI interface where the user can either type or paste in
### Goal
Make a GUI interface where the user can either type or paste in
ASCII art for a Purr Data diagram and have it converted to a floating selection
in the current Purr Data diagram.
Details: Often Purr Data users rely on ASCII art in forums, mailing lists
### Details
Often Purr Data users rely on ASCII art in forums, mailing lists
and other documentation to give examples of Purr Data programs. This is
convenient because it's quicker than taking a screenshot of the Purr Data
GUI and takes up less space, too.
...
...
@@ -230,7 +247,8 @@ selection of a Purr Data object chain would be useful. This would let users
paste ASCII art from the mailing list directly into the interface. It would
also make it easier to create Purr Data diagrams using only the keyboard.
Bonus Goal: make it possible to convert a subset of Purr Data diagrams into
### Bonus Goal
Make it possible to convert a subset of Purr Data diagrams into
ASCII art.
Bonus Goal Details: Unfortunately Purr Data allows arbitrary positioning
...
...
@@ -249,7 +267,8 @@ could disallow objects to overlap with each other. Still, the connections
among the various objects (i.e., the arcs which connect the nodes) could
overlap in such a way that makes unambiguous ASCII art impossible.
Difficulty: Easy to hard. This idea runs the gamut-- a simple GUI-side
### Difficulty
Easy to hard. This idea runs the gamut-- a simple GUI-side
parser for simple ASCII art is a quick exercise. Handling some of the more
complicated art with multiple and/or crossed connections is moderate. Scanning
the mailing lists for examples and covering the majority of them is hard.
...
...
@@ -261,16 +280,19 @@ OCR and some of the current work converting Photoshop mockups to HTML pages.
However, Purr Data diagrams tend to be much simpler than arbitrary Photoshop
art, so the problem may be more tractable here than in other fields.
Languages: Javascript if converting ASCII art to Purr Data diagram, C if
### Languages
Javascript if converting ASCII art to Purr Data diagram, C if
attempting to convert from graphical diagram *back* to ASCII art.
Navigation of "Wireless" Objects
--------------------------------
Goal: make it possible for the user to navigate among all extant wireless
### Goal
Make it possible for the user to navigate among all extant wireless
objects that are bound to a particular symbol.
Details: Purr Data diagrams typically consist of objects-- English words in
### Details
Purr Data diagrams typically consist of objects-- English words in
boxes-- connected by Bezier curves. In a complex diagram, however, the
Bezier curves can end up obscuring the flow of the data in the diagram. For
these instances, "wireless" objects like `[send]` and `[receive]` may be
...
...
@@ -291,7 +313,8 @@ of hyperlinks to the console. When the user clicks any hyperlink, the
corresponding object can be highlighted (or the relevant diagram brought up
with the object in it highlighted).
Difficulty: Easy to moderate. Purr Data's bindelem and pd_bind API are a little
### Difficulty
Easy to moderate. Purr Data's bindelem and pd_bind API are a little
tricky if one hasn't used them before. However, once that interface is
understood it is all that is needed to return a complete list of all
nonlocal objects bound to a particular symbol.
...
...
@@ -300,15 +323,18 @@ The remainder of the problem is easy and can be done in Javascript. However,
this problem "upgrades" gracefully. Once the initial UI is done, it can be
tested on users and iteratively improved from there.
Languages: Javascript, C.
### Languages
Javascript, C.
Terminal REPL
-------------
Goal: make a little REPL interface with which the user can interact with
### Goal
Make a little REPL interface with which the user can interact with
Purr Data programs and program state.
Details: Purr Data is being used in situations where the hardware is an embedded
### Details
Purr Data is being used in situations where the hardware is an embedded
device. While the current GUI runs on most common hardware including the RPI,
there are situations where it would be more convenient to simply interact using
a text interface (locally or over ssh).
...
...
@@ -327,19 +353,23 @@ and well-documented/specified. This way Purr Data users can always interact
with and create programs easily on any embedded device, even if there is
no direct display. (This would also be very handy for debugging purposes.)
Difficulty: Moderate. An initial REPL can be created with the current Purr
### Difficulty
Moderate. An initial REPL can be created with the current Purr
Data API, but it won't be particularly user-friendly. To achieve that requires
more work and an understanding of Purr Data's message dispatching system.
Languages: C, some shell scripting.
### Languages
C, some shell scripting.
Core Accessibility
------------------
Goal: ensure that Purr Data is accessible by coupling accessibility
### Goal
Ensure that Purr Data is accessible by coupling accessibility
with the core UX
Details: especially because Purr Data is a graphical environment, it's
### Details
Especially because Purr Data is a graphical environment, it's
important to make sure the core functionality is accessible. Rather than
tack on accessibility as an afterthought, Purr Data should have a UX that
makes accessibility features a generally useful part of the programming
...
...
@@ -357,13 +387,15 @@ Note: there may be some overlap with the REPL idea above, as the REPL could
provide a sensible way for a user to traverse the diagram as an alternative
to using the GUI.
Difficulty: Moderate. For example, it will be necessary to study the current
### Difficulty
Moderate. For example, it will be necessary to study the current
GUI implementation to figure out how to extend it to add keyboard navigation.
It will also be necessary to study pre-existing approaches to making SVG
diagrams accessible and study the current state of HTML5 tools that facilitate
this.
Languages: Javascript, HTML, CSS. Some basic C knowledge may be required to
### Languages
Javascript, HTML, CSS. Some basic C knowledge may be required to
send a richer set of data about each object from the core to the GUI. However,
there is already an interface that can do this-- it just needs to be hooked
into the GUI.
...
...
@@ -371,10 +403,12 @@ into the GUI.
Purr Data Message and DSP Profiler
----------------------------------
Goal: measure the time it takes for each object in a Purr Data diagram to
### Goal
Measure the time it takes for each object in a Purr Data diagram to
process its data and display the results in the diagram.
Details: Purr Data users would benefit greatly from the ability to profile
### Details
Purr Data users would benefit greatly from the ability to profile
their programs while they are running. This is easy to do for the program
as a whole, but challenging to do per-object.
...
...
@@ -399,37 +433,91 @@ Note: There may be overlap with the other profiling idea listed above, as
developers on both ideas will probably be using the same tools and can
therefore benefit by periodically sharing their work with each other.
Bonus goal: Figure out a way to meaningfully profile DSP objects. DSP objects
### Bonus goal
Figure out a way to meaningfully profile DSP objects. DSP objects
typically process data at a high sample rate (44,100 is common) so displaying
the data in a user-friendly and meaningful way is tricky.
Difficulty: Moderate to Hard. This feature touches the main artery of the
### Difficulty
Moderate to Hard. This feature touches the main artery of the
message dispatching system, and the bonus goal would touch the main DSP
routine. In both cases realtime scheduling deadlines must be taken into
account by careful profiling.
Languages: C for the profiling business logic, HTML5 for displaying the
### Languages
C for the profiling business logic, HTML5 for displaying the
results in the GUI.
Streamlining Purr Data GUI-Pd communication
------------------------------------------
The Pd GUI is heavily entangled with the Pd audio backend. In fact, most of the "gestures" performed on the GUI are passed straight to the Pd engine for processing. The GUI gestures are then "analyzed" by the audio thread, which may respond with triggering a GUI action, changing the state of an object, or nothing.
For instance, each mouse move triggers a `motion` message to the Pd backend, handled by `canvas_motion()` in `g_editor.h`. This calls `canvas_doclick(... doit = false)`, which in turn iterates through all the objects on the patch and asks each of them "does the cursor happen to be on top of you?" (`canvas_findhitbox()`/`canvas_hitbox()`), calling a callback function (`w_getrectfn()`) for each of those objects.
Now, most of the time the cursor is not on an object (or patch cable) and the calls to `w_getrecfn()` have no effect, except for wasting CPU power. There are two notable exceptions:
a) when the mouse pointer is on top of an object, or one of its inlets or outlets, or on top of a patch cord, or on top of a GUI object, the mouse pointer may change, plus, e.g.: flickering inlets/outlets.
b) some objects use the calls to `w_getrecfn()` to track mouse position (e.g.: [mousestate] from cyclone).
The above results in a plethora of CPU cycles being wasted, which may cause dropouts when using small blocksizes and/or embedded platforms. Besides - and perhaps most importantly - it seems the wrong approach that some GUI-specific actions (like the ones at a) above) have to be processed and validated by the audio engine, within the audio thread.
We could therefore think of an improvement to the Purr-data architecture, where the GUI stuff (e.g.: point a) above) is delegated uniquely to the GUI, which makes for lower CPU usage and potentially a more responsive GUI. For instance, the GUI could be designed to only send `motion` messages when the mouse is on top of an object and it could send alongside with it the Pd "tag" of the object, so that `w_getrectfn()` can be called only for the relevant object).
The optimal approach would involve handling all the graphics effects (in/outlet animation, mouse pointers) directly within the GUI, and only sending `motion` messages when something relevant to the Pd engine is _actually_ happening (e.g.: when connecting objects).
Additionally, and looking forward, in order to address point b), objects that need to track mouse position should declare this at initialization and should be kept in a dedicated list, so that the `motion` messages from the GUI can be delivered only to them with minimal CPU waste.
An alternative - and probably worse - approach to the problem, which could reduce peak CPU usage, would be for the Pd audio engine to maintain a "rasterized" cached map of the patch (e.g.: by calling `w_getrecfn()` for each object at each pixel). This way, it could simply look up the cached map in response to each `motion` message. The cache could be recomputed in a separate thread every time after a new object or patch cord is created. Threading issues may arise here, in case one of the objects is deleted while the cached map is being built.
This project comes with a number of challenges, including: potential threading issues between the engine and the GUI, the necessity to re-write the C code of some objects, providing complete documentation for creators of externals, maintaining - where possible (e.g.: excluding objects that track mouse position) - backwards compatibility with Pd.
More details on a previous attempt at addressing the problem can be found [here](http://disis.music.vt.edu/pipermail/l2ork-dev/2017-June/001383.html).
### Goal
Move some of the GUI callbacks out of Purr Data's audio engine so that GUI
interaction is less likely to cause dropouts.
### Details
The Pd GUI is heavily entangled with the Pd audio backend. In fact, most of the
"gestures" performed on the GUI are passed straight to the Pd engine for
processing. The GUI gestures are then "analyzed" by the audio thread, which may
respond with triggering a GUI action, changing the state of an object, or
nothing.
For instance, each mouse move triggers a `motion` message to the Pd backend,
handled by `canvas_motion()` in `g_editor.h`. This calls
`canvas_doclick(... doit = false)`, which in turn iterates through all the
objects on the patch and asks each of them "does the cursor happen to be on top
of you?" (`canvas_findhitbox()`/`canvas_hitbox()`), calling a callback function
(`w_getrectfn()`) for each of those objects.
Now, most of the time the cursor is not on an object (or patch cable) and the
calls to `w_getrecfn()` have no effect, except for wasting CPU power. There are
two notable exceptions:
a) when the mouse pointer is on top of an object, or one of its inlets or
outlets, or on top of a patch cord, or on top of a GUI object, the mouse
pointer may change, plus, e.g.: flickering inlets/outlets.
b) some objects use the calls to `w_getrecfn()` to track mouse position
(e.g.: [mousestate] from cyclone).
The above results in a plethora of CPU cycles being wasted, which may cause
dropouts when using small blocksizes and/or embedded platforms. Besides - and
perhaps most importantly - it seems the wrong approach that some GUI-specific
actions (like the ones at a) above) have to be processed and validated by the
audio engine, within the audio thread.
We could therefore think of an improvement to the Purr-data architecture, where
the GUI stuff (e.g.: point a) above) is delegated uniquely to the GUI, which
makes for lower CPU usage and potentially a more responsive GUI. For instance,
the GUI could be designed to only send `motion` messages when the mouse is on
top of an object and it could send alongside with it the Pd "tag" of the object,
so that `w_getrectfn()` can be called only for the relevant object).
The optimal approach would involve handling all the graphics effects (in/outlet
animation, mouse pointers) directly within the GUI, and only sending `motion`
messages when something relevant to the Pd engine is _actually_ happening
(e.g.: when connecting objects).
Additionally, and looking forward, in order to address point b), objects that
need to track mouse position should declare this at initialization and should
be kept in a dedicated list, so that the `motion` messages from the GUI can be
delivered only to them with minimal CPU waste.
An alternative - and probably worse - approach to the problem, which could
reduce peak CPU usage, would be for the Pd audio engine to maintain a
"rasterized" cached map of the patch (e.g.: by calling `w_getrecfn()` for each
object at each pixel). This way, it could simply look up the cached map in
response to each `motion` message. The cache could be recomputed in a separate
thread every time after a new object or patch cord is created. Threading issues
may arise here, in case one of the objects is deleted while the cached map is
being built.
### Difficulty: Moderate. This project comes with a number of
challenges, including: potential threading issues between the engine and the
GUI, the necessity to re-write the C code of some objects, providing complete
documentation for creators of externals, maintaining - where possible (e.g.:
excluding objects that track mouse position) - backwards compatibility with Pd.
More details on a previous attempt at addressing the problem can be found
