Dance Recognition
Closing Report
Travis Simpson
Dr. Bradford Bennett
Motion Analysis & Motor Performance Lab
August 12, 2011
The purpose of this work is to be able to uniquely identify codified dance moves from motion capture data of a move with distinct start/stop positions.
The first challenge is deciding how to read in the raw data and what parts of the data are desirable. The Vicon .c3d files are processed using Bodybuilder and then exported as a text file. The exact parameters to be exported are chosen at the export screen prompt in Bodybuilder. The processed text files already available contain the following markers/joints for each side: ankle, heel, toe, sacrum, ASIS, and knee. These will most likely need to be expanded over time but are currently sufficient.
A script (readtextfile.m) reads in all of the parameters saved to the text file. The problem that results is choosing a start/stop point for all the data. Since all the markers analyzed are compared against each other at some point, the same time frame of data needs to be used across the board. To accommodate this, the starting time frame in which data is available from each parameter is pulled and compared. The latest starting point between all the moves is used as the first frame to use. A similar designation is used for the end point, with the earliest end point used by all markers being the last frame to use. Furthermore, there will be periodic gaps in the middle of the parameter sets, so the code is geared to disregard these sections when necessary. From these procedures, the raw data is now available to be manipulated in MATLAB.
There are multiple methods of pattern recognition that could be used to solve this problem. I chose two methods, the first of which being classification using dance characteristics. This is important because it allows not only the development of which move a dance sequencemimics but what the move physically does. At this stage, there is a somewhat limited set of name-identifiable moves (100 or so out of 300), but because of classification, they can still be described in dance terminology. As will be shown later, adding the rest of the moves will not be difficult to accomplish.First, the methods of recognition will be described (combined together to form idMove()) and then the details of how model moves (those that the trial is compared against) are developed will be explained.
In past versions, the actual classification followed in order of increasing complexity and specificity based on the moves remaining. If after the most basic analysis is performed there are still multiple possible move left, specific analyses were chosen to separate them. By specific it is not only meant that the analysis may not work for all moves but also that depending on what moves are left, the type of analysis will change. The disadvantage to this method was that every move must be “hard-coded” into the classification system after manually tweaking the code for an optimal result, as a move may need a special/new analysis.
Currently, classification recognition is used up to a point to categorize the move, which eliminates many possible moves. This looks at three basic characteristics: jump/rise/grounded, sacral movement, and cumulative hip rotation. Then, the fourth characteristic depends on whether or not the move is a jump, rise, or neither (grounded). In the case of neither, this characteristic is null. However, if it is a jump or lift, it represents the number of jumps or liftsthe move contains, respectively. Additionally, information is noted in the classification description about what type of take-off/set-down there is (1 foot, 2 feet), and whether or not the jump contains lifts in addition to jumps.
Although some of the classification methods I have developed are not currently used in the code, they could be useful for a more specific classification analysis. There is a function called xCross() that searches for markers coming within a certain threshold of each other, which was previously used to see if the ankles crossed or the ankle crossed the knee during jump/lift periods. The functions findPos() and xPos() were used to determine the positions involved during a grande plie. It could be used for other moves, although it is limited in that it searches from the beginning to end of the move for the starting position and vice versa for the end position. It chooses the position based on marker proximity and only needs to encounter the condition for one frame. Another function, squat(), looks for the sacrum to drop past a certain threshold position based on the subject’s approximate femur length. The functions onToe() and transCheck() look for instances where the dancer is en pointe. transCheck() ensures that the en pointe is not preceding a jump based on a threshold frequency. In addition to these functions, there are very detailed arrays describing useful parameters, such as time frames for jumps/lifts, embedded throughout the code. Much of the basic information needed to perform other, more specific and advanced classification analysis is probably already in place.
Although the classification method is effective because of its description of moves, it can be bulky, among other things previously discussed. To combat these weaknesses, a second pattern recognition method was implemented which utilizes cross-correlation. This analysis follows the basic classification recognition and builds on its results. Covariance was used which is a built-in MATLAB function for assessing cross-correlation between equal length data series. This requires time normalizing for all data that is to be compared. This can be problematic because some trials of a single move are slower than others but it is also important to remember this work is for codified dance moves at this point. Although covariance will analyze for time lag and normalization of the variable set against time, it will not optimize time scaling. This has not been shown to be very influential in the results of my work with a single subject but with future implementation and additional subjects it could be. A better cross-correlating function should be developed which also optimizes time scaling for even better correlations for differently paced and different length moves.
The code currently looks at a total of three trends: hip minus knee elevation, knee minus heel elevation, knee to knee proximity in the transverse plane, andheel to heel proximity in the transverse plane. Note, for the first two trends, they must be correlated for each side since it is a unilateral analysis. This produces a total of two sets of 6 individual trend correlation coefficients (6 assuming same side orientation of move and 6 more assuming opposite). For the combined correlation coefficient, I chose to use the root mean square as opposed to the average or some other mean measure because it factors in standard deviations; root mean square tends to create a larger discrepancy between significant and insignificant coefficients, making the results clearer. The closest matching move is found by maximizing the combined coefficient for some orientation. In the pursuit of simply naming moves, this pattern recognition method seems to be superior since each move tends to have unique and reproducible trends in relative joint motion and elevation. Over time more characteristics may be added to further separate matches, considering the code is designed to accommodate such changes.
Now, that the methods for recognition have been briefly discussed it is important to describe how codified move sets are chosen and compared to the input move. If the classification system was able to look at moves consistently and accurately it would be possible to create a learning system, in which hard coding is unnecessary. This is the premise behind only determining four basic characteristics with classification recognition to narrow down possible moves for the cross-correlation analysis. The first step in developing a model move set requires that the text files desired as “models” be placed into the modelMoves folder and read in by a function called modelMoves(). For each move in the folder, the pre-described classification analysisis performed, exactly as would be done in idMove(). Then the series data is scaled and indexed using the spline fit function spfit() so that it will be ready for cross-correlation.The resulting data is archived to a *.mat, as well as the names of the moves which are pulled from the text file’s title (title format: movename XX.txt). This information is ultimately used as a database to compare the input move against when the actual identification program, idMove(), is called. Which moves are model moves are initially chosen arbitrarily then manually analyzed to make sure the expected trends are present and consistent with other trials, as well as there being full marker data. This script currently takes a few minutes to run with approximately 100 model moves. Although this may seem to be a long time, it is only required to be ran once every time new model moves are added or code is changed, since the data produced is permanently saved for use in idMove().
When performing the cross-correlation within idMove(), only model moves with the same first four characteristics are compared, since this information is recorded in the *.mat. Based on the maximum combined correlation coefficient, it is simple to tell what model move the input file most closely resembles. From this, the script will suggest what move the input trial most likely is.
In past versions, the classification system was used exclusively, with a lot of complex, hard-coded logic. Although it worked fairly well, the maintenance required to add new moves was extensive and too much work to realistically expand the model move database past 40 or so moves. Using the current combined analysis, close to 100 moves can be distinguished. The limit for the number of identifiable moves is theoretically endless as all one has to do is add more files to the modelMoves folder and rerun modelMovesto archive each move. With more moves, it is expected that more correlation characteristics be developed and probably more classification characteristics as well.
Currentlysome of the classification code is developed using thresholds that are user-defined and not necessarily a function of the individual performing the move (as they should be). Even when the thresholds are subject related, there are often moves which teeter along a threshold and going one way or the other can cause the sequence to register as an undesirable move and skew the rest of the analysis. These thresholds need to be made more accurate or implement some sort of correlation for the classification system that does not require the use of absolute thresholds.
Even with the progress made thus far, there are still many issues that need to be addressed. Due to odd start/stop points and gaps in the middle of start/stop, sometimes an incomplete sequence of the move is caught. Most moves that have this limitation are currently unidentified; there are a few documented moves with this problem. If a move such as this is used as a mode, all subsequent correlations involving it will be misguided and irrelevant. Another resulting problem from gaps in data is determination of number of rises and the accompanying points at which the foot leaves and re-contacts the ground. As shown in the MATLAB command window when modelMoves is ran, there are numerous moves that find foot offs but no foot downs. Eventually, the model moves will need to be hand-picked to make sure they contain the marker data needed to accurately and reproducibly portray the move. In some cases, it is obvious which trials are bad due to built in errors, but others will be more subtle and require human verification.
Each move needs to be tested and verified as a viable template, because of the variability of quality motion capture data. This task has been started and there is an excel file (Dance Moves list Upd.xls) which contains all the move names that were recorded during all dance recordings performed prior to the summer of 2011. In this file, which trial is used as the model and which other trials do and don’t work with this model are noted. Green highlighted moves indicate those which have been verified to work with all trials. Those not highlighted are not perfect but work for most trials. Those highlighted in red have issues that need to be addressed directly. In the description and non-working trials columns, information can be found about what is wrong with the file(s). Note, that some of the columns included in this spreadsheet are obsolete, as they were used with the pure classification style of recognition.