BIROn - Birkbeck Institutional Research Online

    Real time anomaly detection of NoSQL systems based on resource usage monitoring

    Spyridon, Chouliaras and Stelios, Sotiriadis (2019) Real time anomaly detection of NoSQL systems based on resource usage monitoring. IEEE Transactions on Industrial Informatics , ISSN 1551-3203. (In Press)

    [img] Text
    bare_jrnlBBK-library.pdf - Author's Accepted Manuscript
    Restricted to Repository staff only until 9 December 2021.

    Download (2MB) | Request a copy

    Abstract

    Today, with the emergence of the industry revolu- tion systems such as Industry 4.0, Internet of Things and big data frameworks pose new challenges in terms of storage and processing of real time data. As systems scale in humongous sizes, a crucial task is to administer the variety of different sub-systems and applications to ensure high performance. This is directly related with the identification and elimination of system failures and errors, while the system runs. In particular, database systems, may experience abnormalities related with decreased throughput or increased resource usage, that in turn affects system performance. In this work, we focus on NoSQL database systems, that are ideal for storing sensor data in the concept of Industry 4.0. This typically includes a variety of applications and workloads that are difficult to online monitor, thus making anomaly detection a challenging task. Creating a robust platform to serve such infrastructures with minimum hardware or software failures is a key challenge. In this work, we propose RADAR, an anomaly detection system that works on real-time. RADR is a data-driven decision-making system for NoSQL systems by providing process information extraction during resource monitoring and by associating resource usage with the top processes, to identify anomalous cases. In this work, we focus on anomalies such as hardware failures or software bugs that could lead to abnormal application runs, without necessarily stopping system functionality e.g. due to a system crash, but by affecting its performance e.g. decreased database system throughput. Although, different patterns may occur through time, we focus on periodic running workloads (e.g. monitoring daily usage) that are very common for NoSQL systems, and Internet of Things scenarios where data streams are forwarded to the Cloud for storage and processing. We apply various machine learning algorithms such as autoregressive integrated moving average (ARIMA), seasonal ARIMA and long-short-term memory recurrent neural networks. We experimentally analyse our solution to demonstrate the benefits of supporting online erroneous state identification and characterisation for modern applications.

    Metadata

    Item Type: Article
    Additional Information: (c) 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.
    School: School of Business, Economics & Informatics > Computer Science and Information Systems
    Depositing User: Stelios Sotiriadis
    Date Deposited: 03 Mar 2020 09:54
    Last Modified: 02 Jul 2021 22:42
    URI: https://eprints.bbk.ac.uk/id/eprint/30971

    Statistics

    Downloads
    Activity Overview
    3Downloads
    75Hits

    Additional statistics are available via IRStats2.

    Archive Staff Only (login required)

    Edit/View Item Edit/View Item