In the last years, convolutional neural network (CNN) based methods are becoming more and more popular to estimate optical flow. Recently, state-of-the art optical flow methods often use multiple frames to make use of temporal information. However, a prediction based on previous frames was not studied separately from the flow estimation for CNN based learning approaches. In this thesis various network structures are tested, compared and improved for this task. The best results were obtained by using warped backward and forward flows from two previous frames. It was shown that in this setting even a simple linear CNN structure produces better results than a prediction based on the reversed backward flow.