Event Sequence Evaluation Module

From Emcap

Contents

Event Sequence Evaluation

by Ricard Marxer <email@ricardmarxer.com>

Music Technology Group, Universitat Pompeu Fabra

2008/2/4

Copyright (c) 2006-2008 All Rights Reserved.

Download Emerge 1.0 (GNU/Linux)

Download Emerge 1.0 (Windows)

What Is The Module?

It is in charge of evaluating the correctness of an estimated sequence in comparison to a groundtruth sequence. Sequences are represented as an ordered list of timestamp and symbols. Additionally taxonomic symbols will interleave taxonomy operations in the sequence in the same manner the COBWEB module does.

This module can be used in several ways:

  • between the output of a classifier and the groundtruth to test the accuracy of transcription
  • between the output of an expectator and the output of a classifier to test the accuracy of expectation versus transcription
  • between the output of an expectator and the groundtruth to test the accuracy of expectation
  • between the output of a feature extractor and the groundtruth to test the accuracy of onset detection

What Are The Inputs/Outputs?

The input to the module consists of a stream of events. These can be treated as a sequence of tuples timestamp+symbol, sequences of symbols, by ignoring the timestamp or as sequence of timestamps by ignoring the symbol.

The module outputs a set of measures, such as the ROC (Receiver Operator Characteristics) and the Mapping accuracy. Additionally it may output a plot of the mapping, showing the estimated and groutndtruth sequence, one above the other, with a coloring encoding the correctness of a match.

What Are The Parameters?

The parameters available to control this module are the ignoring flags, the starting and ending indexes of each of the input sequences and the plotting option.

Usage: emem-eveval [options]
Options:
 -h, --help            show this help message and exit
 -t, --ignore-timestamps                              
                       Evaluate ignoring timestamps, only applicable when
                       same number of annotated and analyzed events [default:
                       False]                                                
 -s, --ignore-symbols  Evaluate ignoring symbols. Equivalent to doing onset  
                       evaluation [default: False]                           
 -e ESTIMATEDLIMITS, --estimated-limits=ESTIMATEDLIMITS                      
                       Define the limits of the estimated stream for which to
                       make the comparaison. [default: (None, None)]         
 -g GROUNDTRUTHLIMITS, --groundturth-limits=GROUNDTRUTHLIMITS                
                       Define the limits of the groundtruth stream for which 
                       to make the comparaison. [default: (None, None)]      
 -p, --plot            plot the representation module during exposure        
                       [default: False]                                      


What Are The System Requirements?

GNU/Linux

The package can be installed in several different ways, depending on your system. The easiest way is to install the setuptools package. On Debian based systems:

$ sudo apt-get install python-setuptools

and then just write the following:

$ sudo easy_install http://www.ricardmarxer.com/emcap/Emerge-1.0-py2.5.egg

If this method does not work correctly, one must install manually all the dependencies, and build the package himself. For the software to fully work you must have the following dependencies installed:

Then one must download the package here and run:

$ python setup.py build
$ sudo python setup.py install

Windows

A binary package has been made for ease of installation and use. To run, one must unzip the package somewhere and run the following executable from the command line:

emem-inckmeans.exe [OPTIONS] INPUT_FILENAME OUTPUT_FILENAME

How To Cite

@incollection{marx07,
 author = {Ricard Marxer and Piotr Holonowicz and Hendrik Purwins},
 title = {Dynamical Hierarchical Self-Organization of Harmonic, Motivic, and
 Pitch Categories},
 address = {Vancouver, Canada},
 booktitle = {Music, Brain and Cognition. Part 2: Models of Sound and
 Cognition, held at NIPS)},
 note = {(to appear)},
 year = {2007}
}

How To Use

$ emem-eveval -e 1 -1 -g 0 -2 -t -p expected.cev transcribed.rev

or

$ emem-eveval -s -p featureextracted.fev groundtruth.rev

Input File Format

With the -s option (ignoring symbols): a file containing one feature event per line. The events are of the form:

ONSET-TIME,ONSET-TIME-VAR;FEAT1-VAL,FEAT1-VAR FEAT2-VAL,FEAT2-VAR ... FEATn-VAL,FEATn-VAR

With the -t or no option (ignoring or not the timestamp): a file containing one representation event per line. The events are of two kinds:

  • Assignments of classes to incoming feature events in the form
ONSET-TIME,ONSET-TIME-VAR;CLASS-ID1 CLASS-ID2 ... CLASS-IDn
  • Changes to the representation space in the form:
CREATE CLASS-IDnew IN CLASS-IDparent
REMOVE CLASS-IDremoved
MERGE CLASS-ID1 CLASS-ID2 ... CLASS-IDn IN CLASS-IDparent

Output File Format

The output gets displayed on the screen.

Example Of Use

$ emem-eveval -e 1 -1 -g 0 -2 -t -p expected.rev transcribed.rev