Arbitrary video super-resolution is a challenging task that involves generating high-resolution videos of arbitrary sizes from low-resolution videos while preserving fine details and ensuring temporal consistency between consecutive frames. In this study, we present ArbVSR, an efficient and effective framework for arbitrary-scale video super-resolution. Specifically, ArbVSR builds upon flow-guided recurrent unit to capture temporal dependency and utilizes local window aggregation to exploit future frames. To better leverage scale information, we generate spatially varying maps via all stages in pre-trained deep neural networks as structural and textural priors, which can identify regions with a high probability of containing texture. The priors effectively guide the super-resolution process to generate more visually pleasing and accurate results across different scale factors. In the upsampling phase, we propose a scale-sensitive and data-independent hypernetwork to generate continuous upsampling weights for arbitrary-scale video super-resolution, which can be computed during pre-processing to improve efficiency. Extensive experiments demonstrate the significant advantages of our method in terms of both performance and efficiency. The source code and trained models will be publicly available.