Video summarization
followed by a two-layer fully connected network to compute frame-level importance scores. The model architecture is illustrated in Figure 3. 11
In 2023, Khan et al. directly used the Vision Transformer (ViT) model as the feature extractor and introduced a pyramid structure to refine multi-scale features. 12 In another study, the standard multi-head attention mechanism was enhanced with channel attention and spatial attention modules to more effectively capture inter-feature dependencies. 13 Furthermore, a spatio- temporal Transformer architecture was proposed to explicitly model the correlations between non-adjacent frames, resulting in improved summarization performance. 14 In addition, building
Figure 3: video summarization model
based on self-attention algorithm
upon the Seq2Seq framework, Rochan et al. 15 introduced a novel approach that combines Fully Convolutional Networks (FCNs) 16 with the DeepLab semantic segmentation model to analyse video content, offering a new perspective for video summarization. 17
Given the multimodal nature of video data, many approaches incorporate features from different modalities to enrich the extracted representations and achieve more accurate summarization results.
Commonly used features include audio and textual information, which, when combined with visual features, provide more comprehensive context for video analysis and help improve both model
Figure 4: multimodal feature video
summarization algorithm example diagram
11 Fajtl, H. et al. (2019) ‘Summarizing Videos with Attention’, Asian Conference on Computer Vision Workshops , 2019: 39–54. (See https://doi.org/10.1007/978-3-030-21074-8_4.) 12 Khan, H.et al. (2023) ‘Deep multi-scale pyramidal features network for supervised video summarization’, Expert Systems with Applications 221: 121288. 13 Puthige, I. et al. (2023) ‘Attention over attention: an enhanced supervised video summarization approach’, Procedia Computer Science 218: 2359–2368. 14 Hsu, T. et al. (2023) ‘Video summarization with spatiotemporal vision transformer,’ IEEE Transactions on Image Processing (available at https://ieeexplore.ieee.org/document/10124837). 15 Rochan, M. et al. (2018) ‘Video summarization using fully convolutional sequence networks,’ in ECCV 2018:
347–363 (available at
https://openaccess.thecvf.com/content_ECCV_2018/papers/Mrigank_Rochan_Video_Summarization_Using_
ECCV_2018_paper.pdf. 16 Long, J. et al. (2015) ‘Fully convolutional networks for semantic segmentation’, IEEE Conference on Computer Vision and Pattern Recognition, 2015: 3431–3440. 17 Chen, L. et al. (2017) ‘DeepLab: Semantic image segmentation with deep convolutional nets, atrous
convolution, and fully connected CRFs’ , IEEE Transactions on Pattern Analysis and Machine Intelligence 40.4:
834–848.
99
Made with FlippingBook - PDF hosting