README.md 27.5 KB
Newer Older
1
[Purr Data as a Web App](#inject-purr-data-directly-into-the-web)
2

Jonathan Wilkes's avatar
add TOC    
Jonathan Wilkes committed
3
[K12 Mode](#k12-mode)
4

Jonathan Wilkes's avatar
add TOC    
Jonathan Wilkes committed
5
[Profile and Optimize Purr Data for Realtime Safety](#profile-and-optimize-purr-data-for-realtime-safety)
6

Jonathan Wilkes's avatar
add TOC    
Jonathan Wilkes committed
7
[ASCII art to Purr Data diagram conversion](#ascii-art-to-purr-data-diagram-conversion)
8

Jonathan Wilkes's avatar
add TOC    
Jonathan Wilkes committed
9
[Terminal REPL](#terminal-repl)
10

Jonathan Wilkes's avatar
add TOC    
Jonathan Wilkes committed
11
[Core Accessibility](#core-accessibility)
12

Jonathan Wilkes's avatar
add TOC    
Jonathan Wilkes committed
13
[Purr Data Message and DSP Profiler](#purr-data-message-and-dsp-profiler)
14

Jonathan Wilkes's avatar
add TOC    
Jonathan Wilkes committed
15
16
[Streamlining Purr Data GUI-Pd communication](#streamlining-purr-data-gui-pd-communication)

Jonathan Wilkes's avatar
Jonathan Wilkes committed
17
18
[Vintage Platform Audio Emulation Library](#vintage-platform-audio-emulation-library)

19
[Library for Data-Over-Audio Communication](#data-over-audio-messaging)
Jonathan Wilkes's avatar
Jonathan Wilkes committed
20

Jonathan Wilkes's avatar
Jonathan Wilkes committed
21
[Interaction with Audio Plugins](#interaction-with-audio-plugins)
Jonathan Wilkes's avatar
Jonathan Wilkes committed
22

Jonathan Wilkes's avatar
Jonathan Wilkes committed
23
[JIT-compiled signal graph for the audio engine](#jit-compiled-signal-graph-for-the-audio-engine)
Jonathan Wilkes's avatar
Jonathan Wilkes committed
24

Jonathan Wilkes's avatar
Jonathan Wilkes committed
25
[Use ref-counting to handle object lifetimes](#use-ref-counting-to-handle-object-lifetimes)
Jonathan Wilkes's avatar
Jonathan Wilkes committed
26

Jonathan Wilkes's avatar
Jonathan Wilkes committed
27
[Visual Diff](#visual-diff)
Jonathan Wilkes's avatar
Jonathan Wilkes committed
28

29
30
[Encapsulation Ergonomics](#encapsulation-ergonomics)

31
32
Inject Purr Data Directly into the Web
--------------------------------------
Jonathan Wilkes's avatar
Jonathan Wilkes committed
33

Jonathan Wilkes's avatar
Jonathan Wilkes committed
34
### Goal
35
Add a WebAssembly target and pure HTML5 GUI framework for running Purr
36
Data as a web app.
Jonathan Wilkes's avatar
Jonathan Wilkes committed
37

Jonathan Wilkes's avatar
Jonathan Wilkes committed
38
### Benefits
39
40
41
Installing Purr Data is often the biggest hurdle for newcomers. Running
it in a web browser simplifies this process and allows Purr Data to be run
on any device, with any architecture, as long as it has a web browser installed.
Jonathan Wilkes's avatar
Jonathan Wilkes committed
42

Jonathan Wilkes's avatar
Jonathan Wilkes committed
43
### Details
44
45
46
47
48
49
50
51
52
53
54
55
56
For the core, we want to produce a WebAssembly binary that can load a patch
and produce audio using the Web Audio API. If possible, we want to also
leverage Chromium's newer callback-based interface to compute realtime audio
as efficiently as possible.

For the GUI, we need a framework that can display and edit a Pd patch
within a container in a web page. It should be possible to run the core
WebAssembly binary with or without the GUI displayed.

The challenge with the GUI is that Purr Data currently uses multiple
toplevel windows to display a patch and its visible subcanvases, and
properties windows for certain widgets. The framework needs to allow
viewing all of these in a container within a *single* browser window.
57

58
59
### Challenges
There has been some prior art for building a WebAssembly binary
60
61
for the core audio engine. However, the difficulty will lie in building
the GUI framework that has a workable UX within a single browser window.
62

Jonathan Wilkes's avatar
Jonathan Wilkes committed
63
### Languages
64
C, HTML5, and Emscripten
65

66
67
68
K12 Mode
--------

Jonathan Wilkes's avatar
Jonathan Wilkes committed
69
70
### Goal
Port the K12 Mode that Pd-l2ork introduced.
71

Jonathan Wilkes's avatar
Jonathan Wilkes committed
72
73
### Details
Currently we ship a lot of the K12 abstractions with Purr Data.
74
75
76
77
78
79
80
81
82
83
84
85
86
However, I never ported the special mode that starts the program up in a
"K12" mode that replaces the window/application menubar with a more
user-friendly, clickable object menu.

That mode makes it easier for beginners to get started. In porting it, we
can take advantage of the new HTML5 GUI:
* The menu itself can leverage HTML/CSS to be more responsive, autohide, etc.
* The entire K12 Mode could itself just be a web-app separate from normal
  Pd app operation.

To figure out which is best, it's probably helpful to know more about the
history of K12 mode, and whether the ability to also open and interact with
normal Pd objects turned out to be an important feature.
Jonathan Wilkes's avatar
Jonathan Wilkes committed
87
88
89
Ivica Bukvic has a lot of experience using this mode when teaching
kids, so we can look to him for some more insight into the right path to
take.
90
91
92
93

Either way, this feature vastly improves the usability of Purr Data for new
users, giving them a way to quickly generate sounds and experiment with the
interface.
94

95
96
### Challenges
A lot of the initial implementation of K12 mode was made
97
98
99
100
101
102
in the core engine in C. This will require at least reading a lot of the
source code and making sense of the changes made for K12 mode, plus possibly
refactoring some of that code so that it can be handled in the GUI instead.
On the other hand, there is a working version of the old K12 mode which can
be consulted as a reference implementation.

103
104
105
106
107
108
109
110
111
112
One additional feature is the possibility of
using a touchscreen to control the GUI while sound is being produced.
The HTML5 interface of nw.js makes this fairly straightforward to implement.
But it will take some extra work to ensure all of the common interactivity
of a Pd patch is accessible through touch.

As an additional challenge, it should be possible to edit a patch using
the touchscreen as well. K12 mode doesn't require the same UX complexity
as regular editing, so this should be doable as well.

Jonathan Wilkes's avatar
Jonathan Wilkes committed
113
114
### Languages
Javascript, C, CSS, HTML.
115

116
117
118
### Potential Mentor
Ivica Bukvic, Jonathan Wilkes

119
120
121
Profile and Optimize Purr Data for Realtime Safety
--------------------------------------------------

Jonathan Wilkes's avatar
Jonathan Wilkes committed
122
123
### Goal
Find the "pain points" in Purr Data's core message dispatcher and
124
125
126
audio engine, and optimize the code where possible to improve the realtime
scheduling of DSP.

Jonathan Wilkes's avatar
Jonathan Wilkes committed
127
128
### Details
The core DSP algorithms of Purr Data tend to avoid system calls,
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
unnecessary branching, and other calls which would make the performance of
the audio process unreliable. However, there are many areas of Purr Data
which are not optimized for realtime scheduling.

The first order of business would be to profile Purr Data in several areas
to see where problems may be. Some likely culprits are the following:

* parse time for incoming messages from GUI to the audio/messaging engine
* overhead of sending messages to the GUI, especially for visual arrays and
  drawing instructions for data structures
* overhead of sending streams of "motion" messages from GUI to Pd to track
  mouse position
* overhead of walking the linked list of objects in order to find the object
  under the mouse/mouseclick
* overhead of tracking mouse position over a visual array
* overhead of calculating bboxes in the audio process as opposed to GUI
* overhead of calculating text positioning in audio process as opposed to GUI
* overhead of counting UTF-8 code points in audio process to calculate
  line breaks
* overhead of handling incoming/outgoing socket traffic in the same thread
  as the audio scheduler
* probably other areas

Once we get a sense of the pain points, we can tackle the problem of how
to optimize in a maintainable manner. Some references for this:

* [Guilio Moro's work](https://github.com/giuliomoro/purrdata/commit/9dc3223ece79be5f60a6a629450b52a79b9e050c) on using a separate thread for the socket connections from GUI to the core audio engine (plus all the other socket connections
like netsend, netreceive, etc.)
* Guilio Moro's work on a threaded microsleep for the event loop.
* [Guilio Moro's work](https://github.com/giuliomoro/purrdata/commits/simpler-motion) to simplify GUI communication by handling more of the mouse motion/click
  logic in the GUI. This results in fewer messages from GUI to audio engine,
  but still requires a linked-list walk in the audio engine to find the
  relevant object.
* Matt Barber's link to a new cosine wave generator algorithm that may be
  more performant than the current implementation. (Not so important for
  current performance, but this may become more relevant once we switch to
  double-precision for block samples.)
166
167
168
* Possibility to vectorize DSP algos using SIMD. Also more crude experiments
  by just hand-unrolling one or two classes when N=64 (i.e., the most common
  block size) and measuring the performance impact (if any).
169

170
171
172
173
Note: There may be some overlap with the other profiling idea listed below.
Developers for both ideas may therefore benefit by periodically sharing their
work with each other.

174
175
### Challenges
The initial profiling will take some time but
176
177
178
179
180
181
isn't particularly challenging. Making changes to the core audio engine,
however, will require some knowledge of Linux system interfaces and some of
Purr Data's internals. Properly assessing and testing any threading techniques
in C is also frought with peril and will require extreme care in order to
keep the code maintainable and avoid insidious bugs.

Jonathan Wilkes's avatar
Jonathan Wilkes committed
182
183
### Languages
Basic to advanced shell scripting, C, plus familiarity with
184
185
profiling tools like gprof and others.

186
187
188
ASCII art to Purr Data diagram conversion
-----------------------------------------

Jonathan Wilkes's avatar
Jonathan Wilkes committed
189
190
### Goal
Make a GUI interface where the user can either type or paste in
191
192
193
ASCII art for a Purr Data diagram and have it converted to a floating selection
in the current Purr Data diagram.

Jonathan Wilkes's avatar
Jonathan Wilkes committed
194
195
### Details
Often Purr Data users rely on ASCII art in forums, mailing lists
196
197
198
199
200
201
202
203
204
and other documentation to give examples of Purr Data programs. This is
convenient because it's quicker than taking a screenshot of the Purr Data
GUI and takes up less space, too.

The ability for the user to type ASCII art into an object box and get a new
selection of a Purr Data object chain would be useful. This would let users
paste ASCII art from the mailing list directly into the interface. It would
also make it easier to create Purr Data diagrams using only the keyboard.

Jonathan Wilkes's avatar
Jonathan Wilkes committed
205
206
### Bonus Goal
Make it possible to convert a subset of Purr Data diagrams into
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
ASCII art.

Bonus Goal Details: Unfortunately Purr Data allows arbitrary positioning
of objects on an integer-based x/y axis. Thus, not all Purr Data diagrams
can be represented as ASCII art.

The bonus goal would add a feature to check if the currently selected Purr
Data diagram is able to be represented using only ASCII art. This could be
either a) a button that does an analysis, or b) realtime feedback to let the
user know when they have positioned an object or connection in such a way
that makes ASCII art impossible.

This is a non-trivial problem. For example, one could add a button to Purr
Data that would constrain object positioning to a predefined grid. Then one
could disallow objects to overlap with each other. Still, the connections
among the various objects (i.e., the arcs which connect the nodes) could
overlap in such a way that makes unambiguous ASCII art impossible.

225
226
### Challenges
This idea runs the gamut-- a simple GUI-side
227
228
229
230
231
232
233
234
235
236
237
parser for simple ASCII art is a quick exercise. Handling some of the more
complicated art with multiple and/or crossed connections is moderate. Scanning
the mailing lists for examples and covering the majority of them is hard.

Additionally, converting an existing Purr Data graphical diagram back to
ASCII art is a difficult problem. This will require reading up on the current
state of the art and comparing various approaches from other fields like
OCR and some of the current work converting Photoshop mockups to HTML pages.
However, Purr Data diagrams tend to be much simpler than arbitrary Photoshop
art, so the problem may be more tractable here than in other fields.

Jonathan Wilkes's avatar
Jonathan Wilkes committed
238
239
### Languages
Javascript if converting ASCII art to Purr Data diagram, C if
240
241
attempting to convert from graphical diagram *back* to ASCII art.

242
243
244
Terminal REPL
-------------

Jonathan Wilkes's avatar
Jonathan Wilkes committed
245
246
### Goal
Make a little REPL interface with which the user can interact with
247
248
Purr Data programs and program state.

Jonathan Wilkes's avatar
Jonathan Wilkes committed
249
250
### Details
Purr Data is being used in situations where the hardware is an embedded
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
device. While the current GUI runs on most common hardware including the RPI,
there are situations where it would be more convenient to simply interact using
a text interface (locally or over ssh).

The user can already communicate with Purr Data's audio engine over a socket
connection. So the "read" and "evaluate" part already exists. However, Purr
Data does not print a response nor loop in this situation.

Additionally, the UX of sending raw messages to Purr Data's interpreter is
quite lacking. The syntax for creating new environments and objects was not
meant to be used directly. Objects are referenced by index number, and the
diagrams themselves must be referenced using hex identifiers.

It would be very beneficial to create a REPL UI that is more user-friendly
and well-documented/specified. This way Purr Data users can always interact
with and create programs easily on any embedded device, even if there is
no direct display. (This would also be very handy for debugging purposes.)
268

269
270
### Challenges
An initial REPL can be created with the current Purr
271
272
273
Data API, but it won't be particularly user-friendly. To achieve that requires
more work and an understanding of Purr Data's message dispatching system.

Jonathan Wilkes's avatar
Jonathan Wilkes committed
274
275
### Languages
C, some shell scripting.
276
277
278
279

Core Accessibility
------------------

Jonathan Wilkes's avatar
Jonathan Wilkes committed
280
281
### Goal
Ensure that Purr Data is accessible by coupling accessibility
282
283
with the core UX

Jonathan Wilkes's avatar
Jonathan Wilkes committed
284
285
### Details
Especially because Purr Data is a graphical environment, it's
286
287
288
289
290
291
292
293
294
295
296
297
298
important to make sure the core functionality is accessible. Rather than
tack on accessibility as an afterthought, Purr Data should have a UX that
makes accessibility features a generally useful part of the programming
environment.

For example: how does one navigate the nodes of a Purr Data diagram? There
should be a way to navigate among the nodes and their connections
without using the mouse. If we make sure that each element in the diagram
is annotated we can tackle accessibility and keyboard navigation at the
same time. Thus, a robust keyboard navigation implementation will help
make it possible for screen readers to give meaningful information about
each node in the graph.

299
300
301
302
Note: there may be some overlap with the REPL idea above, as the REPL could
provide a sensible way for a user to traverse the diagram as an alternative
to using the GUI.

303
304
### Challenges
For example, it will be necessary to study the current
305
306
307
308
309
GUI implementation to figure out how to extend it to add keyboard navigation.
It will also be necessary to study pre-existing approaches to making SVG
diagrams accessible and study the current state of HTML5 tools that facilitate
this.

Jonathan Wilkes's avatar
Jonathan Wilkes committed
310
311
### Languages
Javascript, HTML, CSS. Some basic C knowledge may be required to
312
313
314
315
316
317
318
send a richer set of data about each object from the core to the GUI. However,
there is already an interface that can do this-- it just needs to be hooked
into the GUI.

Purr Data Message and DSP Profiler
----------------------------------

Jonathan Wilkes's avatar
Jonathan Wilkes committed
319
320
### Goal
Measure the time it takes for each object in a Purr Data diagram to
321
322
process its data and display the results in the diagram.

Jonathan Wilkes's avatar
Jonathan Wilkes committed
323
324
### Details
Purr Data users would benefit greatly from the ability to profile
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
their programs while they are running. This is easy to do for the program
as a whole, but challenging to do per-object.

A successful implementation of this feature will give an accurate measure
of the time it takes each object to process its incoming data. This needs to
support all of Purr Data's platforms: Windows, OSX, and GNU/Linux.

A successful implementation will also be performant enough that the
measurements themselves don't impact the realtime operation of Purr Data
itself.

A now defunct fork of Pure Data called ["DesireData"](http://artengine.ca/desiredata/)
did an initial
implementation of this idea using the x86 RDTSC instruction. (Though its
unlikely this feature was actually stable at the time DesireData was
in active development.) Though this instruction is no longer considered
reliable on modern machines, the overall approach taken by DesireData of
adding a field to the t_gobj struct for storing this timing data is
probably a sound starting point.

345
346
347
348
Note: There may be overlap with the other profiling idea listed above, as
developers on both ideas will probably be using the same tools and can
therefore benefit by periodically sharing their work with each other.

Jonathan Wilkes's avatar
Jonathan Wilkes committed
349
350
### Bonus goal
Figure out a way to meaningfully profile DSP objects. DSP objects
351
352
353
typically process data at a high sample rate (44,100 is common) so displaying
the data in a user-friendly and meaningful way is tricky.

354
355
### Challenges
This feature touches the main artery of the
356
357
358
359
message dispatching system, and the bonus goal would touch the main DSP
routine. In both cases realtime scheduling deadlines must be taken into
account by careful profiling.

Jonathan Wilkes's avatar
Jonathan Wilkes committed
360
361
### Languages
C for the profiling business logic, HTML5 for displaying the
362
results in the GUI.
363
364
365
366

Streamlining Purr Data GUI-Pd communication
------------------------------------------

Jonathan Wilkes's avatar
Jonathan Wilkes committed
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
### Goal
Move some of the GUI callbacks out of Purr Data's audio engine so that GUI
interaction is less likely to cause dropouts.

### Details
The Pd GUI is heavily entangled with the Pd audio backend. In fact, most of the
"gestures" performed on the GUI are passed straight to the Pd engine for
processing. The GUI gestures are then "analyzed" by the audio thread, which may
respond with triggering a GUI action, changing the state of an object, or
nothing.

For instance, each mouse move triggers a `motion` message to the Pd backend,
handled by `canvas_motion()` in `g_editor.h`. This calls
`canvas_doclick(... doit = false)`, which in turn iterates through all the
objects on the patch and asks each of them "does the cursor happen to be on top
of you?" (`canvas_findhitbox()`/`canvas_hitbox()`), calling a callback function
(`w_getrectfn()`) for each of those objects.

Now, most of the time the cursor is not on an object (or patch cable) and the
calls to `w_getrecfn()` have no effect, except for wasting CPU power. There are
two notable exceptions: 
a) when the mouse pointer is on top of an object, or one of its inlets or
outlets, or on top of a patch cord, or on top of a GUI object, the mouse
pointer may change, plus, e.g.: flickering inlets/outlets.
b) some objects use the calls to `w_getrecfn()` to track mouse position
(e.g.: [mousestate] from cyclone).

The above results in a plethora of CPU cycles being wasted, which may cause
dropouts when using small blocksizes and/or embedded platforms. Besides - and
perhaps most importantly - it seems the wrong approach that some GUI-specific
actions (like the ones at a) above) have to be processed and validated by the
audio engine, within the audio thread.

We could therefore think of an improvement to the Purr-data architecture, where
the GUI stuff (e.g.: point a) above) is delegated uniquely to the GUI, which
makes for lower CPU usage and potentially a more responsive GUI. For instance,
the GUI could be designed to only send `motion` messages when the mouse is on
top of an object and it could send alongside with it the Pd "tag" of the object,
so that `w_getrectfn()`  can be called only for the relevant object).

The optimal approach would involve handling all the graphics effects (in/outlet
animation, mouse pointers) directly within the GUI, and only sending `motion`
messages when something relevant to the Pd engine is _actually_ happening
(e.g.: when connecting objects).

Additionally, and looking forward, in order to address point b), objects that
need to track mouse position should declare this at initialization and should
be kept in a dedicated list, so that the `motion` messages from the GUI can be
delivered only to them with minimal CPU waste.

An alternative - and probably worse - approach to the problem, which could
reduce peak CPU usage, would be for the Pd audio engine to maintain a
"rasterized" cached map of the patch (e.g.: by calling `w_getrecfn()` for each
object at each pixel). This way, it could simply look up the cached map in
response to each `motion` message. The cache could be recomputed in a separate
thread every time after a new object or patch cord is created. Threading issues
may arise here, in case one of the objects is deleted while the cached map is
being built.

426
427
### Challenges
This project comes with a number of
Jonathan Wilkes's avatar
Jonathan Wilkes committed
428
429
430
431
432
433
434
435
436
437
challenges, including: potential threading issues between the engine and the
GUI, the necessity to re-write the C code of some objects, providing complete
documentation for creators of externals, maintaining - where possible (e.g.:
excluding objects that track mouse position) - backwards compatibility with Pd.

More details on a previous attempt at addressing the problem can be found
[here]( http://disis.music.vt.edu/pipermail/l2ork-dev/2017-June/001383.html).

### Languages
Javascript and C.
438

439
440
441
### Potential Mentor
Giulio Moro

442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
Vintage Platform Audio Emulation Library
----------------------------------------

### Goal
Create a library with objects that emulate the hardware from
old hardware like the atari 2600, NES, and others.

### Details
There are a lot of resources online for emulating old
hardware. Purr Data would benefit by having a library
that provides a consistent interface for objects that
take input into an emulation of a piece of hardware and
output one or more audio signals.

If possible, it would be beneficial if most of the interface
could be built as a set of abstractions. That way more
developers would be able to understand and improve the library.

There is a TIA chip emulator written in C in externals/mmonoplayer
that can be used as a starting point.

463
464
465
466
### Challenges
Finding a common interface that makes it easy for users to leverage
these classes while at the same time being expressive enough to allow
decent control of the chip being emulated.
467
468
469
470
471

### Languages
Pd (Purr Data is a fork of the software Pure Data-- the visual language itself
is usually referred to as Pd.) Also, C.

472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
Data Over Audio Messaging
-------------------------

### Goal
Create a library that allows two instances of Purr Data to pass data messages
to each other using sound as the transmission medium.

### Details
Pd messages consist mainly of space-separated numbers and symbols.
Semicolons mark the end of a message.

Sometimes it would be helpful to be able to pass messages from
one instance of Purr Data to another-- especially if each instance is
on a different machine in the same room. This is currently done either
by setting up socket listener/receiver between the two instances or by
leveraging a separate message-passing system outside of Purr Data.

Since Purr Data is concerned mainly with analyzing and sythesizing sound,
machines running Purr Data typically have a mic and speakers connected to
a running instance. If it were possible for the user to simply create
objects which send/receive messages by sending audio signals to/from each
other it would greatly simplify sending at least small amounts of data
between machines.

496
497
498
### Challenges
Finding a decent interface for users without relying on a big set
of dependencies.
499
500
501
502
503

### Languages
Pd (Purr Data is a fork of the software Pure Data-- the visual language itself
is usually referred to as Pd.) However, the library may also be written in
C.
Jonathan Wilkes's avatar
Jonathan Wilkes committed
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527

Interaction with Audio Plugins
------------------------------

### Goal
Make sure that Purr Data has a well-documented way to accept input from 
standard audio plugin APIs like VST, LADSP, and LV2. Also
make sure Purr Data can be used as a plugin in other environments.


### Details
There are multiple audio plugin APIs that aim at seamlessly mixing and
matching audio filters, synthesizers, and analysis tools in different
languages and applications.

Purr Data has some libraries to interface with at least two of these
standards (VST and LADSPA). There is also a library for LV2 that
Purr Data can leverage. However, not all of these libraries run on
all the supported platforms (OSX, Windows, Linux).

Purr Data also has all the APIs necessary to act as a plugin itself
in other applications. But work must be done to ensure this works
properly and that it is properly documented how to do it.

528
529
### Challenges
There is a lot of pre-existing technology 
Jonathan Wilkes's avatar
Jonathan Wilkes committed
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
here, but it needs to be tested rigorously on all platforms.

### Languages
C. Also, familiarity with shell scripting as well as Gnu make.

JIT-compiled Signal Graph for the Audio Engine
----------------------------------------------

### Goal
Leverage LLVM to make a jit-compiler for Purr Data's DSP graph.

### Details
Alex Norman has shown that it is possible to build a jit compiler
for one of Pd's "workhorse" libraries-- the "expr" library. This
library essentially lets users specify a mathematical expression
that can operate on both vectors and on the sample level. The
current library parses the tokens of the expression and requires
a separate function call for each unit. This is similar to the
way Pd's DSP graph itself gets executed.

The jit compiler produces assembly which has superior performance
to the current "expr" library. Extending that process to the entire
signal graph (or at least core classes within it) would benefit
performance on a greater level.

555
556
### Challenges
Finding a way to incrementally build up the functionality
557
558

Since there is pre-existing
Jonathan Wilkes's avatar
Jonathan Wilkes committed
559
560
561
562
563
564
work done with the jit "expr" library, a good start would be to help
test that library and improve it. Once the student gains a mental
model of the process the student can begin to extend it to Purr Data's
DSP graph itself.

### Languages
565
C++. Knowledge of LLVM will also come in handy here.
Jonathan Wilkes's avatar
Jonathan Wilkes committed
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596

Use ref-counting to handle object lifetimes
-------------------------------------------

### Goal
Use ref-counting to track object lifetimes in a more robust manner.

### Details
Currently there are many places in the message dispatching system where an
object may disappear before a message bound for it gets dispatched. This,
along with dynamic patching and arbitrary patch loading can easily cause
Purr Data to crash.

Additionally, this limits the possibility for reporting errors from within
a given call stack. Currently we have no way to know if any of the objects
participating in a given call still exist at the time of an error. This
means we're limited to checking after the fact for any signs of mutation
at error time, and then simply refusing to trace in that case. That prevents
valuable analytical data for the most complex error cases.

### Challenges
There aren't currently any tests for the cases where attempting to
dereference a pointer to a freed object causes a crash. Also, there are
two "stacks"-- one is the messaging system proper. The other is the
loading mechanism used to open a file or abstraction.

Additionally, Purr Data has a binding system which can dynamically bind
or unbind symbols even within a single call stack.

### Languages
C.
Jonathan Wilkes's avatar
Jonathan Wilkes committed
597
598


599
Visual Diff
Jonathan Wilkes's avatar
Jonathan Wilkes committed
600
601
602
-----------

### Goal
603
604
Make a tool to visually show the changes in a Purr Data patch, possibly
with animation.
Jonathan Wilkes's avatar
Jonathan Wilkes committed
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621

### Details
Pd has a source file format that is difficult to read. Furthermore,
changes in z-order in the GUI can make changes in the source nearly
impossible to convey. This makes it difficult to glean anything of
substance by viewing a diff of a Pd file in git.


### Challenges
This is a general problem with visual programming languages. There may
be some prior art with some languages that attempt to solve the problem.
But unlike Pd, those languages were probably built from the ground up
to solve that problem, whereas Pd's file format doesn't lend itself to
solving that problem.

### Languages
C, possibly HTML5.
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656


Encapsulation Ergonomics
------------------------

### Goal
Improve Purr Data's techniques for encapsulation in the language.

### Details
Pd currently has two basic techniques for organizing diagrams. One is simply
to copy part of the diagram and paste it into a subpatch. That part of the
diagram becomes hidden inside a new object that appears on the parent window.

The other is to use so-called "abstractions", which are other Pd files that
are instantiated as an object on the parent.

Both cases present problems for developers. For example, the program should
be able to take a selection of objects and immediately put it into a subpatch
with the relevant inlets and outlets created inside the subpatch. Instead,
the user must manually cut, paste, and redraw the connections in the editor.

Second, users who want to encapsulate and reuse code must first save a
patch to the filesystem in the current Pd path, and then create a new object
with the same name as that file (minus the ".pd" extension). This makes
proper encapsulation more difficult than simply cutting/pasting subpatches,
leading to quick-and-dirty copy/pasta where more robust code reuse through
abstractions would have been preferred.

### Challenges
Much of Pd's current code assumes that abstractions must be loaded from
the file system instead of from some template that already exists in the
current patch file.

Additionally, a new class must be added to define a patch template that
fills the role that ".pd" files currently do for abstractions.