Finally, a spatio-temporal multi-modal graph (STMG) is constructed to model the interaction among multiple faces. Face and audio branches encode audio signals and faces, respectively. Visual branch has a two-stream architecture to capture spatial and temporal information. VAM-Net consists of three branches corresponding to visual, audio and face modalities. Guided by these findings, a visual-audio multi-task network (VAM-Net) is introduced to predict saliency and locate sound source. Using this database, we find that sound influences human attention, and conversly attention offers a cue to determine sound source on multi-face video. Specifically, we first introduce a large-scale database of multi-face video in visual-audio condition (MVVA), containing eye-tracking data and sound source annotations. In this paper, we propose a multitask learning method for visual-audio saliency prediction and sound source localization on multi-face video by leveraging visual, audio and face information. However, most existing saliency prediction works ignore the influence of audio and only consider vision modality. Visual and audio events simultaneously occur and both attract attention. Finally, the experimental results show that our DeepVS2.0 method advances the state-of-the-art video saliency prediction. Moreover, the center-bias dropout and sparsity-weighted loss are embedded in SS-ConvLSTM, to consider the center-bias and sparsity of human attention maps. Therefore, a saliency-structured convolutional long short-term memory network (SS-ConvLSTM) is developed in DeepVS2.0 to predict inter-frame saliency, using the extracted features of OM-CNN as the input.

We further find from our database that human attention has a temporal correlation with a smooth saliency transition across video frames. Accordingly, we propose an object-to-motion convolutional neural network (OM-CNN) in DeepVS2.0 to learn spatio-temporal features for predicting the intra-frame saliency via exploring the information of both objectness and object motion. Through the statistical analysis of LEDOV, we find that human attention is normally attracted by objects, particularly moving objects or the moving parts of objects. Specifically, we establish a large-scale eye-tracking database of videos (LEDOV), which provides sufficient data to train the DNN models for predicting video saliency. In this paper, we propose a novel DNN-based video saliency prediction method, called DeepVS2.0. However, few works apply DNNs to predict the saliency of generic videos. Deep neural networks (DNNs) have exhibited great success in image saliency prediction.

Attention map vs. saliency map