CORDS: automatic discovery of correlations and soft functional dependencies

Ilyas, I.F. and Markl, V. and Haas, P.J. and Brown, Paul and Aboulnaga, A. (2004) CORDS: automatic discovery of correlations and soft functional dependencies. In: Weikum, G. and Konig, A.C. and Dessloch, S. (eds.) Proceedings of the ACM SIGMOD International Conference on Management of Data. Association for Computing Machinery, pp. 647-658. ISBN 9781581138597.

Full text not available from this repository.

Official URL: https://doi.org/10.1145/1007568.1007641

Abstract

The rich dependency structure found in the columns of real-world relational databases can be exploited to great advantage, but can also cause query optimizers---which usually assume that columns are statistically independent---to underestimate the selectivities of conjunctive predicates by orders of magnitude. We introduce CORDS, an efficient and scalable tool for automatic discovery of correlations and soft functional dependencies between columns. CORDS searches for column pairs that might have interesting and useful dependency relations by systematically enumerating candidate pairs and simultaneously pruning unpromising candidates using a flexible set of heuristics. A robust chi-squared analysis is applied to a sample of column values in order to identify correlations, and the number of distinct values in the sampled columns is analyzed to detect soft functional dependencies. CORDS can be used as a data mining tool, producing dependency graphs that are of intrinsic interest. We focus primarily on the use of CORDS in query optimization. Specifically, CORDS recommends groups of columns on which to maintain certain simple joint statistics. These "column-group" statistics are then used by the optimizer to avoid naive selectivity estimates based on inappropriate independence assumptions. This approach, because of its simplicity and judicious use of sampling, is relatively easy to implement in existing commercial systems, has very low overhead, and scales well to the large numbers of columns and large table sizes found in real-world databases. Experiments with a prototype implementation show that the use of CORDS in query optimization can speed up query execution times by an order of magnitude. CORDS can be used in tandem with query feedback systems such as the LEO learning optimizer, leveraging the infrastructure of such systems to correct bad selectivity estimates and ameliorating the poor performance of feedback systems during slow learning phases.

Metadata

Item Type:	Book Section
School:	Birkbeck Faculties and Schools > Faculty of Science > School of Computing and Mathematical Sciences
Depositing User:	Sarah Hall
Date Deposited:	23 Feb 2021 20:34
Last Modified:	09 Aug 2023 12:50
URI:	https://eprints.bbk.ac.uk/id/eprint/43163

Statistics

Archive Staff Only (login required)

Edit/View Item