Steven Roebert

In the previous entry I talked about clustering the descriptors. Even though the clustering algorithm was run on a really fast machine with loads of memory, it was still running after two days. It was not even finished with clustering the HoG descriptors, while it still had to do the HoF descriptors afterwards (which takes at least the same amount of time). As we I had to continue with my work a faster clustering algorithm was necessary.

Luckily Ivo had a solution, careful seeding. More is explained in the paper by Arthur and Vassilvitskii: The Advantages of Careful Seeding. This proved to be a good temporary solution. The clustering was now done in less than 5 minutes (note that we don't know if this is optimal, but it useable non the less).

After the initial clustering was done, it was time for the creation of feature vectors for each video sample. To do this, we simply go through all HoG and HoF descriptors in the video sample and measure the distance to each cluster (Euclidean distance in our case). Then for each cluster we sum up the amount of descriptors that have the closest distance to it. Finally we normalize the all the values, based on the number of descriptors. All the values for each cluster together form the feature vector for the video sample. The following code block contains the code to calculate these feature vectors.

As there are 1700 video samples (each containing a few thousand descriptors), it would once again take to long to run the script on my own laptop. Instead Ivo made use of this opportunity to teach me how to work with the national computer cluster Lisa. One can simply write a bash script and submit it as a job to Lisa, which will run the script as soon as it has an available spot on the cluster. For each script you have to specify how much time and memory you will be needing. So we split up the script into 18 pieces, all calculating the feature vector for up to 100 samples at a time. This took about 3 hours to complete.

The Hollywood2 dataset also contains labels for each video sample, describing wether a certain action is performed in the video. Using these labels and the feature vectors we created, it was possible to create a classifier and to see how well it performs on our data. As we do not have the specific frames where each action is performed, the classifier will simply be able to specify if a certain action happened somewhere in a given video sample. It will not be able to tell us the exact location of the action. Although we obviously want to know at what point in the video the action is taking place, it is is nevertheless a good starting point to know in which video the action is taking place. The results from the first classification experiments look quite promising. The errors are quite low for most actions (under 0.1).

The next step is to build a multi-class classifier to distinguish between different actions. Also I will be experimenting with methods for finding the specific frames where the action is taking place. However, the latter problem does require me to go through the whole dataset, as this information is not in the Hollywood2 dataset by default (a daunting task...).