Test Time Training for Long Videos via Frame Forgetting Network

Test Time Training (TTT) is a mechanism in which a modeladapts to an incoming test-sample by performing some self-supervised(SSL) task and updating its weights even during inference. This proce-dure does not require labels at test-time. This paper focuses on TTT forlong-videos. A major concern with existing approaches is: 1) they per-form TTT updates using a sliding window containing frames in the past,whose compute increases linearly with the size of window. This becomescomputationally intractable when the videos are hours long. 2) TTT isperformed even when temporally close frames look similar, thereby con-suming a lot of compute.We present the Frame Forgetting Network (FFN) that: 1) operates ononly three frames within the sliding window, namely the frame that ex-its, the current frame and the frame after that. The model still managesto retain temporal context and work for hours long-videos; 2) mathemat-ically define a ‘surprise’ metric: how much ‘new information’ the incom-ing frame contains with respect to the past seen frame. This facilitatesdetermining how to modify the effective window size during TTT andconstitutes the core mechanism of an adaptive windowing algorithm. Ad-ditionally, we curate a dataset EpicTours containing up to 3 hour longvideos of walking city-tours, whereas earlier datasets on this problemwere only 5 min long. We demonstrate FFN’s empirical effectiveness ondense-segmentation, video classification tasks, generalization to depth-estimation, and multi-hour long videos. The project page can be foundat https://github.com/rajatmodi62/ffn.