STCV-Synth Demo Videos

• Performance – Short Demo

• Rundown

Description

STCV-Synth is the first prototype in a series of devices designed for people with disabilites to express themselves creatively. Secondarily, these devices are designed to create novel approaches for interdisciplinary art performances. It is our hope that we bring new opportunities of expression to artists, non-artists, and any other individual.

The STCV-Synth consists of an audio DSP engine that is operated via two controllers, an STMicroelectronics SensorTile (ST), and Computer Vision (CV). The combination of these controllers create performance approaches analogous to conducting an orchestra.

Note that you will need at least Python 3.7 to run STCV-Synth, as it heavily relies on the asyncio module. We are currently working in developing a workaround for running it on Python 3.6, which is what the NVIDIA Jetson Nano ships out with.

Created by David Cardona and Robert Fischer.

Installation

Please refer to the included installation file.

Prepare SensorTile

Please note that as of right now, the quaternion functionality will only work when the SensorTile is soldered onto a cradle board. Using the SensorTile on a cradle expansion board will result in an error unpacking the data received via GATT. We are working on this and hope to upload a updated version of the firmware that guarantees quaternion compatibility in both the cradle and the cradle expansion boards.

Flashing the binaries is enough for prepping the SensorTile. See the documentation on how to do this within the ST_Firmware folder.

If you're interested in modifying the firmware, see the modified_files folder and the enclosed md file.

(Note that building on Linux is currently not possible as Linux is case-sensitive OS, and there are some case issues in the STMicroelectronics that are inconsistent with this requirement.)

Execution

After completing the installation steps for your OS, and flashing you SensorTile, you may run the STCV-Synth from the project's root folder by running: python main.py

References

• asyncio

  • Async IO in Python: A Complete Walkthrough
  • Documentation
  • Signal Library

• asyncio - Modify Queue

  • CPython - Queue File
  • Modify Queue to drop old items to make space for new ones

• Bleak

  • Documentation
  • GitHub
  • Enable Notifications Example
  • SensorTag Example
  • Service Explorer Example

• MediaPipe

  • Build MediaPipe Python Package
  • Installation Instructions

• Numpy

  • Numpy Math Fuctions

• OpenCV

  • Video Capture
  • Video Capture Properties

• Orientation

  • 3-DOF Orientation Tracking - Stanford
  • Orientation from Acceleration
  • Quaternions for Orientation
  • STMicroelectronics - Orientation

• Pyo

  • Documentation
  • GitHub
  • Map Function
  • Tips for Improving Real-Time Performance

• Threading

  • Python Threading

• Ubuntu

  • Upgrade from Python 3.6 to 3.7 on Ubuntu 18.04

Further Goals

• Add provisions for real-time OpenCV execution, potentially using Docker and/or other processes enabled by RabbitMQ.

• Enable GPU acceleration.

• Optimize OpenCV performance.

• Gray out GUI plus and minus buttons when the current value is equal to the minimum or maximum.

• Use the ST wearing hand menu to only detect the other hand for polyphony, and the ST hand for GUI operation. This may be unnecessary and different approaches may be taken to determine the functionality that will come from the ST wearing hand vs the other one.

• Implement interactivity using various sensors.

• Add event handler

  • Set 'esc' key for quitting the program.
  • Set 'p' key for triggering a pause state.

• Add modes of operation based on automatic controller detection upon launch.

• Implement 'Sustain' mode of operation. (Determine how to start and stop notes. Perhaps a tapping gesture?)

• Add polyphonic control. (Voice class abstraction?)

• Create shell script for flashing binaries.

• Verify Quaternion computations (i.e., usage of Real value w).

• Map Quaternion calculated Euler angles to the synth.

• Verify value interpolation in Pyo when updating ADSR 'mul' within a loop.

• Decouple ST async Queues for real-time improvements.

• Video Synth (potentially use PyOpenGL + shaders for GPU execution)

• Add new musical gestures

  • Switch chord type, e.g., major, minor, suspended, augmented, diminished, etc. (this might be quite complicated because of scales are defined in such a way that the data gets quantizes to specific scale steps).

  • Switch chord inversion: assuming a C major chord (with notes C, E, and G), the chord is said to be in its fundamental position when the lowest note is C; it is said to be in first inversion if the bottom note is E (E, G, and C); and it is said to be in second inversion if the bottom note is G (G, C, and E). All of these permutations reflect the same major chord.

  • Add/Remove audio effects.