| Bibliography | Hierlinger, Johannes: An investigation of matching cost dependent and independent convex upsampling of optical flow in a single-scale setting. University of Stuttgart, Faculty of Computer Science, Electrical Engineering, and Information Technology, Bachelor Thesis No. 118 (2023). 57 pages, english.
|
| Abstract | Optical flow estimation is one of the main objectives in the field of computer vision as optical flow has many areas of use. There are many approaches that aim to solve the optical flow problem, one of them being RAFT which gained a lot of popularity for its novel architecture and excellent accuracy. The architecture of RAFT includes a feature and context extractor, extracting image and context features, a correlation pyramid which contains dot products between all pairs of image feature vectors, a lookup operator, generating a feature map by indexing from the correlation pyramid and an iterative update operator to update the optical flow and a latent hidden-state at each iteration, eventually converging to a final estimate. Due to the high computational demands involved in estimating optical flow at the original resolution, RAFT reduces the spatial size of the estimated optical flow to 1/8 resolution. Since RAFT is being trained supervised by comparing the estimated optical flow to the ground truth optical flow at the original resolution, the optical flow should be upsampled to the original resolution. RAFT achieves this by introducing a hidden-state based convex upsampling process with the upsampling factor of 8, which is dependent on the looked up costs of the lookup operator as the hidden-state is computed with the use of these costs. Since the ×8 upsampling factor is very large, and likely not optimal, we investigate whether combining smaller upsampling factors yield to better results or not. It would be impossible to achieve this combination of smaller upsampling factors with the use of cost dependent hidden-state based convex upsampling as this would require too many computational resources for looking up the costs at larger scales of the correlation pyramid. That is why in this thesis, we introduce context based convex upsampling which gets computed with the use of the context features and therefore is fully independent of the looked up costs. Furthermore, we combine this cost independent context based upsampling with the cost dependent hidden-state based convex upsampling. To this end, we introduce several different upsampling methods as well as several upsampling strategies to upsample the flow to the original resolutions. All our introduced approaches are used in combination with RAFT and aim to change its upsampling module. In the end, we provide a comparison of the performances of our best performing approaches and RAFT. Our results show that some approaches outperform the cost dependent convex upsampling process of RAFT with the best ones achieving a 2, 94% increase on the Sintel clean train set, a 5, 39% increase on its final train set and a 10, 46% increase on the FlyingChairs data set.
|