[Defense] A Multimodal Approach to Detecting and Assessing Objectionable Content in Online Media
Monday, July 1, 2024
9:30 am - 11:30 am
In
Partial
Fulfillment
of
the
Requirements
for
the
Degree
of
Doctor
of
Philosophy
Elaheh
Baharlouei
will
defend
her
dissertation
听A
Multimodal
Approach
to
Detecting
and
Assessing
Objectionable
Content
in
Online
Media
Abstract
In this dissertation, we address the challenge of detecting questionable content in online media, specifically the subcategory of comic mischief. This type of content combines elements such as violence, adult content, or sarcasm with humor, making it difficult to detect. Employing a multimodal approach is vital to capture the subtle details inherent in comic mischief content. To tackle this problem, we propose a novel end-to-end multimodal system for the task of comic mischief detection. As part of this contribution, we release a novel dataset for the targeted task consisting of three modalities: video, text (video captions and subtitles), and audio. We also design a HIerarchical Cross-attention model with CAPtions (HICCAP) to capture the intricate relationships among these modalities. We employ a hybrid pretraining approach that merges contrastive learning with multimodal matching tasks allowing for the joint learning of language, audio, and video representations. Our system also innovatively employs automatic video captioning to compensate for missing subtitles, thereby enriching the dataset and improving detection efficacy. The results show that the proposed approach makes a significant improvement over robust baselines and state-of-the-art models for comic mischief detection and its type classification. This emphasizes the potential of our system to empower users, to make informed decisions about the online content they choose to see.
Monday,
July
1,
2024
9:30
AM
-
11:30
AM
CT
PGH 550
Dr. Thamar Solorio, dissertation advisor
Faculty, students, and the general public are invited.
- Location
- PGH 550