Video summarization
second layer estimates the importance of video segments based on this representation and selects the key segments for summarization. Subsequently, in 2018, Zhao et al. further integrated a trained module designed to detect shot-level temporal structures within the video. This prior knowledge was then used to estimate the importance of each shot, resulting in a final video summary composed of key shots. 6 In 2019, Lebron Casas et al. extended the LSTM-based framework by incorporating an attention mechanism to model user interest. 7 The attention-enhanced information was then used to estimate frame-level importance and select keyframes for constructing a video storyboard. It is advisable to keep all the given values. In addition, several approaches have adopted the Sequence-to-Sequence (Seq2Seq) architecture, originally developed in the field of natural language processing, to compute importance scores through an encoder–decoder framework. In this structure, the encoder is responsible for extracting video features, while the decoder predicts the frame-level importance scores. In 2019, Ji et al.
formulated video summarization as a Seq2Seq learning problem and proposed an LSTM-based encoder–decoder network with an intermediate attention layer, as illustrated in Figure 2. 8 In the same year, Lal et al. improved the decoder design by employing a convolutional LSTM to better model the spatio-temporal relationships within the video, leading to improved performance. 9 In 2021, Gao et al. also adopted this structure and achieved promising results. 10
Figure 2: The video summarization
algorithm based on supervised model
With the remarkable success of Transformers in the field of computer vision, several studies have incorporated the self-attention mechanism to replace LSTM for video feature extraction. In 2019, Fajtl et al. proposed a video summarization network that employs self-attention to extract video features,
6 Zhao, B. et al. (2018) ‘HSA-RNN: Hierarchical structure-adaptive RNN for video summarization’, IEEE
Conference on Computer Vision and Pattern Recognition (available at
https://ieeexplore.ieee.org/document/8578871). 7 Lebron Casas, L. et al. (2019) ‘Video summarization with LSTM and deep attention models,’ International Conference on MultiMedia Modelling: 175–187. 8 Ji Z, et al. (2019) ‘Video summarization with attention-based encoder–decoder networks’, IEEE Transactions on Circuits and Systems for Video Technology 30.6: 1709–1717. 9 Lal, S. et al. ‘Online video summarization: Predicting future to better summarize present’, IEEE Winter Conference on Applications of Computer Vision 2019, 471–480 (available at https://ieeexplore.ieee.org/document/8659035). 10 Gao, J. et al. (2021) ‘Unsupervised video summarization via relation-aware assignment learning’, IEEE
Transactions on Multimedia 23: 3203–3214.
98
Made with FlippingBook - PDF hosting