We present a novel approach towards the creation of vision based recognition tasks. A lot of domain specific recognition systems have been presented in the past which make use of the large amounts of available video data. The creation of ground truth data sets for the training of theses systems remains difficult and tiresome. We present a system which automatically creates clusters of 2D trajectories. The results of this clustering can then be used to perform the actual labeling of the data, or rather the selection of events or features of interest by the user. The selected clusters can be used as positive training data for a user defined recognition task – without the need to adapt the system. The proposed technique reduces the necessary user interaction and allows the creation of application independent ground truth data sets with minimal effort. In order to achieve the automatic clustering we have developed a distance metric based on the Hidden Markov Model representations of three sequences – movement, speed and orientation – derived from the initial trajectory. The proposed system yields promising results and could prove to be an important steps towards mining very large data sets.