A High-Performance Parallel Processing Architecture for Big Data Sort Operations

International Journal of Computer Science and Technology | Vol. 11, No. 12, December 2009 | pp. 89–96

DOI: 10.46882/2009/IJCST/000021

Article Type: Original Research Paper

Title: A High-Performance Parallel Processing Architecture for Big Data Sort Operations

Names of Authors: Linus B. Torvalds¹, Ada L. Lovelace²

Authors’ Affiliations: ¹Department of Software Engineering, Helsinki Institute of Technology, Helsinki, Finland; ²Computational Sciences Department, University of Oxford, Oxford, UK

Abstract: Sorting vast multi-terabyte datasets remains a computational bottleneck for modern high-performance computing clusters and cloud storage backends. This paper presents a high-performance parallel sorting framework based on an optimized bucket-sort architecture implemented over distributed memory sub-systems. The core algorithm splits dataset arrays dynamically across multiple worker threads while employing a load-balancing pivot matrix to ensure equalized processing loads. We benchmarked this methodology across an enterprise cluster consisting of 64 nodes running concurrent Linux kernels. Experimental data demonstrates that our parallel sort architecture achieves a speedup factor of 5.8±0.2 compared to standard Hadoop MapReduce sort benchmarks on identical data volumes. The processing latency scaled linearly with data sizes up to 10 TB, maintaining a flat memory footprint overhead of less than 120 MB per node. This architecture offers an optimal data structure pipeline for large-scale data warehouses requiring rapid database indexing.

Keywords: Parallel Processing, Big Data Sort, Load Balancing, Distributed Computing, Cluster Architecture, Memory Optimization

Manuscript Timeline: Received: April 5, 2009; Revised: June 1, 2009; Accepted: June 28, 2009; Published: December 12, 2009

Citation: Torvalds, B. L., & Lovelace, A. L. (2009). A High-Performance Parallel Processing Architecture for Big Data Sort Operations. International Journal of Computer Science and Technology, 11(12), 89–96.

Subscribe to read the full article: https://www.internationalscholarsjournals.org/subscribe-to-read