Video summarization
Background and Related Work The concept of video summarization was formally introduced in the 1990s. 1 Early methods primarily relied on optical flow analysis to identify local minima of motion within video frames, emphasizing the importance of static images to extract summary segments. In 1996, Wolf et al. proposed viewing a video sequence as a trajectory curve in a high-dimensional feature space, where the intersection points of curve segments were selected as keyframes for multi-level video summarization. 2 In 2000, Doulamis et al. [3] introduced a method based on constructing multidimensional fuzzy histograms of video frames, identifying keyframes by eliminating shots or frames with similar content. These approaches represent several early traditional techniques for video summarization. 3 With the rapid advancement of machine learning in recent years, deep neural networks have been successfully applied to the field of video summarization, achieving promising results. Deep learning-based methods can be broadly categorized into supervised (and weakly supervised) and unsupervised learning approaches. 2.1 Supervised Learning Supervised methods based on deep learning treat video summarization as a prediction problem, estimating the frame-level importance by modelling temporal dependencies between frames. During training, the network receives a sequence of video frames along with user-annotated ground-truth summaries. The predicted importance scores are compared with the ground-truth annotations, and the model is optimized through backpropagation using a designated loss function. Based on this
approach, Zhang et al. [4] were the first to propose, in 2016, the use of Long Short-Term Memory (LSTM) networks to model temporal dependencies among video frames. 4 They employed a Multilayer Perceptron (MLP) to predict frame-level importance scores and incorporated a Determinantal Point Process (DPP) to enhance the diversity of visual content in the generated summary. The architecture is illustrated in Figure 1. Building on this framework, Zhao et al. proposed a two-layer LSTM architecture in 2017. 5 The first layer encodes structural information from the video, while the
Figure 1: The video summarization algorithm
based on supervised model
1 Barbieri M., Agnihotri L. and Dimitrova, N. (2003) ‘Video summarization: methods and landscape’, Internet
Multimedia Management Systems 4: 1–13. 2 Wolf, W. et al. (1996) ‘Key frame selection by motion analysis’, IEEE Computer Society : 1228–1231 (available at https://ieeexplore.ieee.org/document/543588). 3 Doulamis, A. et al. (2000) ‘Fuzzy video content representation for video summarization and content-based retrieval’, Signal Processing 80.6: 1049–1067. 4 Zhang. K. al. (2016) ‘Video summarization with long short-term memory,’ European Conference on Computer Vision 2016: 766–782 (available at https://link.springer.com/chapter/10.1007/978-3-319-46478-7_47). 5 Zhao, B. et al. (2017) ‘Hierarchical recurrent neural network for video summarization’, in ACM International
Conference on Multimedia (available at https://dl.acm.org/doi/pdf/10.1145/3123266.3123328).
97
Made with FlippingBook - PDF hosting