A Scalable Data Mining Infrastructure for Association Rule Discovery on Large Market Basket Datasets

International Journal of Computer Science and Technology | Vol. 10, No. 6, June 2008 | pp. 41–48

DOI: 10.46882/2008/IJCST/106015

Research Article

Title: A Scalable Data Mining Infrastructure for Association Rule Discovery on Large Market Basket Datasets

Names of Authors: Thomas E. Keller¹, Sarah J. Fletcher¹

Authors’ Affiliations: ¹Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL, USA

Abstract: Extracting meaningful consumer purchase patterns from massive enterprise data warehouses demands highly efficient data mining frameworks. Standard Apriori-based mining implementations are frequently bottlenecked by the computational cost of iterative database scans and substantial memory requirements for candidate generation. This paper presents a scalable data mining infrastructure featuring an optimized Frequent Pattern Growth (FP-Growth) algorithm that uses a localized horizontal data partitioning paradigm. The architecture distributes the overhead of tree construction across multiple concurrent memory spaces, drastically shrinking global input/output wait times. The system was stress-tested using synthetic transactional datasets scaling up to 10 million individual items. Benchmark logs reveal a 4.2x acceleration in transaction processing speeds relative to traditional single-threaded FP-growth configurations. The infrastructure maintained linear scalability, keeping peak RAM consumption stable at 1.2 GB ± 0.1 GB during high-density itemset passes, rendering it highly viable for real-time retail decision-support systems.

Keywords: Data mining, association rules, market basket analysis, FP-growth, scalable architecture, parallel processing

Manuscript Timeline: Received March 3, 2008; Revised April 15, 2008; Accepted May 12, 2008; Published June 4, 2008

Citation: Keller, T. E., & Fletcher, S. J. (2008). A Scalable Data Mining Infrastructure for Association Rule Discovery on Large Market Basket Datasets. International Journal of Computer Science and Technology, 10(6), 41–48.

Subscribe to read the full article: https://www.internationalscholarsjournals.org/subscribe-to-read