Audio interval retrieval using convolutional neural networks

Ievgeniia Kuzminykh*, Dan Shevchuk, Stavros Shiaeles, Bogdan Ghita

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

17 Downloads (Pure)

Abstract

Modern streaming services are increasingly labeling videos based on their visual or audio content. This typically augments the use of technologies such as AI and ML by allowing to use natural speech for searching by keywords and video descriptions. Prior research has successfully provided a number of solutions for speech to text, in the case of a human speech, but this article aims to investigate possible solutions to retrieve sound events based on a natural language query, and estimate howeffective and accurate they are. In this study, we specifically focus on theYamNet, AlexNet, and ResNet-50 pre-trained models to automatically classify audio samples using their respective melspectrograms into a number of predefined classes. The predefined classes can represent sounds associated with actions within a video fragment. Two tests are conducted to evaluate the performance of the models on two separate problems: Audio classification and intervals retrieval based on a natural language query. Results show that the benchmarked models are comparable in terms of performance, with YamNet slightly outperforming the other two models. YamNet was able to classify single fixed-size audio samples with 92.7% accuracy and 68.75% precisionwhile its average accuracy on intervals retrieval was 71.62% and precision was 41.95%. The investigated method may be embedded into an automated event marking architecture for streaming services.

Original languageEnglish
Title of host publicationInternet of Things, Smart Spaces, and Next Generation Networks and Systems - 20th International Conference, NEW2AN 2020 and 13th Conference, ruSMART 2020, Proceedings
EditorsOlga Galinina, Sergey Andreev, Sergey Balandin, Yevgeni Koucheryavy
PublisherSpringer
Pages229-240
Number of pages12
ISBN (Print)9783030657253, 9783030657260
DOIs
Publication statusPublished - 22 Dec 2020
Event20th International Conference on Next Generation Teletraffic and Wired/Wireless Advanced Networks and Systems, NEW2AN 2020 and 13th Conference on the Internet of Things and Smart Spaces, ruSMART 2020 - St. Petersburg, Russian Federation
Duration: 26 Aug 202028 Aug 2020

Publication series

NameLecture Notes in Computer Science
PublisherSpringer
Volume12525
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference20th International Conference on Next Generation Teletraffic and Wired/Wireless Advanced Networks and Systems, NEW2AN 2020 and 13th Conference on the Internet of Things and Smart Spaces, ruSMART 2020
Country/TerritoryRussian Federation
CitySt. Petersburg
Period26/08/2028/08/20

Keywords

  • Audio classification
  • Convolutional neural network
  • Deep learning
  • Intervals retrieval
  • Natural language query

Fingerprint

Dive into the research topics of 'Audio interval retrieval using convolutional neural networks'. Together they form a unique fingerprint.

Cite this