Real-time What/when Expectation System

From Emcap

Contents

Expect: Real Time What/When Expectation System

Amaury Hazan, Pompeu Fabra University 2008

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.

To get the source code and updates please mail me at ahazan@iua.upf.edu

Download Expect Package (Windows XP)

Download Expect Package (MACOSX)

Sound examples for the Expect package (Reviewers use only)

What is the Module?

Expect is an application demonstrating the use of various what/when expectation models applied to musical audio. Salient events in the incoming music stream are detected and categorized.

For each timbre category, expectation of the future events to be heard is computed. This expectation can in turn affect the transcription process.

The analysis and visualization are done real-time, which enables an interactive experience.

What are the inputs/outputs

The most important input to be provided to Expect is a live audio stream. This stream can either be fed into the system by a microphone or via a wave file using a VST plug-in host.

The output of the system is an audiovisual feedback describing the output of transcription and expectation processes.

What are the parameters?

  • Detection sensitivity

This parameter is related to the transient detection behaviour of the system. If the slider is on the on left side, the perceptual hit detection is very sensible, if the slider is on the right side, the perceptual hit detection is less sensible.

  • Strength of expectation feedback

This parameter is related to the expectation-driven top-down control of the onset detection. If the slider is on the left-most side, no feedback is applied at all. If the slider is on the right-most side, a greater feedback is applied. This means the transient detection is less sensible when events are not expected, and more sensible when events are expected.

What are the system requirements?

Expect can be run in a self contained demo on Windows XP Simply click on install.bat (Expect dependencies will be copied on C:\Expect_deps), and then click on Expect icon (pink square) after unzipping the package. If the program gives an error at startup, select an audio buffer size smaller than 512 (see example) and re-run it.

Apart from this, Expect is available as a VST plug-in, which enables its use in various software hosts and provides a MIDI output. The XP and MACOSX binary are provided here. This software compiles on XP, MACOSX and Linux environment, however the Linux binary has not been tested.

The plugin included in the package can be run in any VST Plugin host. We recomend to use the Bidule host, which has a free trial version for both XP and MacOSX, and is very stable. http://www.plogue.com/index.php?option=com_remository&Itemid=0&func=selectcat&cat=1

How to Use

Quick reference

The most important settings are available in the following screenshot.

Quick reference
Enlarge
Quick reference

About the visualization

The visualization is divided into two parts: past and future. The vertical red line indicates the present. At each instant, when a perceptual hit is detected, a point (filled colored ball) appears at the present. The color and row correspond to the detected timbre category. Based on the detected events, expectations are generated and appear in the future. When a detected event matches and expected event, this mean that the expectation is fulfilled.

Example

  • From windows explorer, double-click on the Expect icon
  • Activate wav player

click on:

 Engine->recorder->add to input
  • Set audio latency

Expect requires working with low latency driver (audio buffer size < 512 samples) To change your audio buffer settings settings click on:

 Device->Wave->buffer and set the buffer to a low value (e.g. 128)
  • Load a sound file

Expect aims at processing *real world recordings*. Choose any recording in wav format with a sample rate of 44100.

Examples files are provided in the test_sound folder. For instance load the file Techno_Piano.wav.

  • Click on the play button. Each Timbre-specific onset appears in distinct rows with distinct colors. Enjoy

What are the dependencies

The standalone demo is made thanks to SAVI standalone host.

Expect is built on top of the Billaboop (Audio Analysis) and Juce (Graphical user interface) C++ libraries, which are published under the General Public License.

The main dependencies of the Billaboop library are already bundled with it. They are:

  • FFTReal (public domain)
  • libsndfile (LGPL)
  • libsvm (LGPL)

How to Cite

Expect is the output of the following research article:

 Hazan, A., Marxer, R., Brossier, P., Purwins, H., Herrera, P., Serra, X. (2009).  
 What/when causal expectation modelling applied to audio signals.  Connection Science. 

Bibtex

 @article {hazan2009,
 	title = {What/when causal expectation modelling applied to audio signals},
 	journal = {Connection Science},
 	year = {2009},
 	URL = {http://www.mtg.upf.es/files/publications/whatwhen_connection.pdf},
 	author = {Hazan, A. and Marxer, R. ;Brossier, P. and Purwins, H.;Herrera, P. and Serra, X.}
 }