Video summarization
performance and robustness. In 2021, Zhao et al. used both audio and visual information for video summarization by extracting features from each modality using LSTM networks, as illustrated in Figure 4. 18 Zhang et al. introduced a semantic video encoder to extract high-level semantic information, which was then combined with visual feature representations to enhance the model’s sensitivity to important content. 19 In addition, multimodal features can also be augmented through human-centric signals. For example, Paul and Salehin proposed a novel video summarization framework based on eye-tracking data. 20 This method calculates motion saliency scores by measuring the distance between the viewer’s current and previous fixation points, and uses these scores as auxiliary features alongside visual features to guide the summarization process. 2.2 Unsupervised Learning Although supervised video summarization methods can yield relatively good results, they heavily depend on manually annotated datasets, which are costly and time-consuming to produce. To address this limitation, unsupervised learning methods have emerged. In the absence of explicit supervision (i.e., ground-truth labels used in supervised learning), most unsupervised approaches adopt heuristic
strategies that define constraints based on quantifiable characteristics of summaries, aiming to generate segments that best represent the original video content. The main idea of clustering-based methods is to first aggregate similar shots together and select cluster centers as components of the summary segments. In 1998, Y. Zhuang et al. first used this method, directly clustering video frames with similar features and selecting cluster center frames to form the final video summary. 21 In 2006, Y. Hadi et al. improved the clustering accuracy by using
Figure 5: Video summarization model based on
clustering methods
18 Zhao, B. et al. (2021). ‘Audiovisual video summarization,’ IEEE Transactions on Neural Networks and Learning Systems 34.8: 5181–5188. 19 Zhang, Y.et al. (2023) ‘VSS-Net: Visual Semantic Self-mining Network for Video Summarization,’ IEEE Transactions on Circuits and Systems for Video Technology (available at https://ieeexplore.ieee.org/document/10239534). 20 Paul, M. & Salehin, M. (2019) ‘Spatial and motion saliency prediction method using eye tracker data for video summarization’, IEEE Transactions on Circuits and Systems for Video Technology 29. 6: 1856–1867. 21 Zhuang, Y.et al. (1998) ‘Adaptive keyframe extraction using unsupervised clustering,’ in IEEE International
Conference on Image Processing, 1998, pp. 866–870 (available at
https://ieeexplore.ieee.org/document/723655).
100
Made with FlippingBook - PDF hosting