" Music technologies could benefit from a little AI. "
Background
Learning a new instrument typically requires developing a particular set of skills and musical knowledge which can be frustrating and tedious. Machine learning software and sensor technologies allow for new forms of musical interaction, expression and playing styles. One no longer needs to be an expert musician to create pleasant sounding music.
Project goal
Create a digital music hardware controller which explores new modes of music interaction and is accessible to novice musicians. Experiment with technologies and materials.
Target audience
Amateur and professional musicians.
Result
The Ml.cubes controllers allow musicians to easily explore an extensive palette of sounds and musical phrases on-the-fly. The cubes simplify music interaction in a somewhat similar manner to the way a drum machine allows for instant music creation. Wireless sensor and machine learning technologies are used to enable a more embodied and playful musical performance.
Skills
  • Interaction design
  • Sound design
  • Coding
  • Machine learning
Introduction

Machine learning technologies are frequently used in visual design software and applications. However, there are far fewer examples of these technologies being used for audio applications and instrument design.

The ml.cubes prototype is meant to demonstrate how this tech might allow for new forms of musical interaction, expression and playing styles.

A photo of Ml.cubes prototype next to a laptop showing a graphical user interface. Also on the wooden desk is a small recording console.
ml.cubes prototype
Inspiration

I was inspired to begin this project after taking an online course which teaches how to use machine learning technologies for the purpose of building real-time controllers for music, games and interactive art.

My main personal goal for this project was to learn through making and experimentation. I had no particular design outcomes which I wanted to achieve, but was inspired by the materials and technology that I was working with.

A couple of curiosities I had while beginning this project:

  • What unconventional musical interactions can be designed using machine learning?
  • How might machine learning tech enable the rapid prototyping of instrument designs?
  • What do users expect while interacting with musical instruments and devices, and how might physical material selection, form and appearance influence this?
A photo of ml.cubes product prototype on a wooden desk next to a music speaker with a hand reaching out to grab one.
colourful painted sides
Exploration

I created a moodboard to establish the tone that I wanted to set with my instrument design. I came up with three themes which I wanted this product to express:

  • simple and natural
  • abstract, experimental and expressive
  • modern, but timeless

The final design uses wooden cube enclosures painted in a hard-edge style. The cubes don't have any controls, switches or lights and simply start to play sound the moment in which they are moved.

A mood board collage of image inspiration for ml.cube project. Images include hard edge style art and modern art inspired instrument design.
ml.cubes moodboard
Design & technology

The ml.cubes send wireless accelerometer data via a Bluetooth connection to the ml.cubes program. This data is fed to an SVM machine learning model to detect cube face positions. Each unique cube position combination triggers audio effect changes. The wireless design enables the cubes to be easily positioned on top of a table, piano, guitar amplifier, or wherever else feels appropriate during a live performance.

I designed physically modelled software instruments to achieve realistic sounds. I also created a simple graphical user interface to signal which position either cube is currently in.

Technologies I used for this project include:

  • MicroBit wireless accelerometers
  • Ml.lib SVM machine learning library
  • IRCAM's Modalys
  • Cycling '74 Max software
A photo of a micro-computer inside a wooden cube with a solid white background.
wireless controller
Reflection

My biggest takeaway from this project was how personal experimentation and interaction with physical materials and tech can lead to more unconventional product designs. For example, the more I experimented with the wooden cubes the more I understood their physical affordances.

The cube shapes invite users to turn and flip the controllers and I used these natural movements and transitions to design intuitive interactions. The number of combinations that are possible with the design suggest to the user that experimentation is the best way forward.

Overhead photo of Ml.cubes prototype next to a laptop showing a graphical user interface. Also on the wooden desk is a small recording console.
performance set-up
Next project
© 2022 Steve Savage
Work
Play
Contact