The thesis objective is to develop a robust tracking pipeline for a Baxter robot system. The tracking pipeline includes the applications of robot tracking and object tracking, which localizes the surrounding objects for robot manipulation. By combining, extending, and comparing state-of-theart approaches for object tracking and image segmentation, this thesis estimates the initial poses of the objects such that their pose can be used to auto-initialize object tracking algorithms. The pipeline is able to handle unfamiliar and dynamic environments. The system was implemented in simulation and can be applied to a real robotic system. The resulting pipeline uses an RGB image, a depth image, and the 3D object models as input. The outputs are the tracked object poses in real time. A dataset was generated to train the instance segmentation network. Furthermore, the pipeline was evaluated with several conditions to test the robustness of the tracking. A comparison to other 6D pose estimation approaches is provided in the results. The code of the pipeline is available on GitHub: https://github.com/timschaeferde/rai_baxter