Functional dependency (FD) is a concept for describing regularities in data, that traditionally was used for schema normalization. Recently, a different problem has emerged: For a given dataset, find FDs that are contained in it. Efficient FD discovery is of great importance due to FD usage in many tasks such as data cleaning, schema recovery, and query optimization. In this paper we consider a particular class of FD discovery algorithms that rely on partition intersection. We present a simple approach for caching intermediate results that are generated during run times of these algorithms. Our approach is essentially a heuristic that selects which partitions to cache based on measures of disorder in data. For this purpose, we adopt such well-known measures as Entropy and Gini Impurity, and propose a novel one—Inverted Entropy. Our approach has a negligible computational overhead, and can be used with a number of FD discovery algorithms, both exact and approximate. Our experiments demonstrate that our heuristic allows to decrease algorithm run times by up to 40% while simultaneously requiring up to an order of magnitude less space compared to the state-of-the-art caching approaches.
Original languageEnglish
Title of host publicationNew Trends in Databases and Information Systems
PublisherSpringer Nature
Pages52-59
Number of pages8
ISBN (Electronic)978-3-030-30278-8
ISBN (Print)978-3-030-30277-1
DOIs
StatePublished - 1 Jan 2019
Event23rd European Conference on Advances in Databases and Information Systems, ADBIS 2019 - Bled, Slovenia
Duration: 8 Sep 201911 Sep 2019
Conference number: 231719

Publication series

NameCommunications in Computer and Information Science
PublisherSpringer Nature
Volume1064
ISSN (Print)1865-0929

Conference

Conference23rd European Conference on Advances in Databases and Information Systems, ADBIS 2019
Country/TerritorySlovenia
CityBled
Period8/09/1911/09/19

    Research areas

  • Caching, Entropy, Functional dependency, Gini impurity, Partition caching, Partition intersection, PYRO, TANE

ID: 113380819