Deduplicação de dados não-estruturados de processos streaming em tempo real

Abstract

Data duplication is a widespread problem in data streams processing applications that can occur due to software errors or the adoption of data loss prevention measures, this problem is usually treated after ingesting the data into a repository, either by an auxiliary process or by the analyzes themselves about the data disregarding duplicates. However, with the need for analyzes made as close to the moment of data creation and in the shortest possible time, both approaches become insufficient to meet both requirements, making it necessary for deduplication to occur during ingestion. This work explores methods that can be used with the Apache Spark library to deal with data deduplication in real time, analyzing the use of resources and the delivery time of each method and identifying their use cases. This work investigates Apache Spark native deduplication operators (distinct and drop Duplicates) and auxiliary tools (RocksDB, Apache Ignite and Apache Hudi) that provide data deduplication and fault tolerance mechanisms to the application. The experimental results show that there is an increase in the data delivery time when using external mechanisms, but that these mechanisms become essential for an ingestion process to ensure that there is no loss of data used during deduplication, ensuring that no duplicates are persisted. In addition, other factors influence the choice of the best deduplication method, such as the use of computational resources and the size of the persisted data