A Balanced Topological Graph Partitioning Scheme for Distributed Web-Scraping Engines

International Journal of Computer Science and Technology Vol. 2, No. 8, August 2000 | pp. 57–64

DOI: 10.46882/2000/IJCST/000020

Research Article

Title: A Balanced Topological Graph Partitioning Scheme for Distributed Web-Scraping Engines

Names of Authors: Sarah Jenkins¹, Vikram Rao²

Authors’ Affiliations: ¹Department of Computer Science, Columbia University, New York, USA; ²School of Computer Science and Automation, Indian Institute of Science, Bangalore, India.

Abstract:
Distributed web-crawlers and indexing engines struggle with intense network overhead and unbalanced resource distribution when downloading massive structural maps of the internet. The primary operational bottleneck involves assigning interconnected hyperlink schemas across independent physical nodes without generating heavy cross-partition lookups that degrade server processing cycles. This paper outlines an optimized graph-partitioning scheme, Balanced Hyperlink Cut (BHC), specifically engineered to handle power-law web graphs containing highly referenced domain nodes. The algorithm tracks domain boundary relationships and evenly balances physical storage weights while minimizing the inter-node page replication factor. We deployed the BHC framework across a distributed cloud harvesting cluster and evaluated indexing timelines using a raw link graph containing 80 million target URLs and 1.2 billion hyperlink vectors. The experimental findings indicate that BHC cuts inter-node message routing overhead by 43.6% and speeds up structural index creation by a factor of 3.2 relative to classical hash-based URL allocation approaches. Storage capacity variance stayed below 1.8%, validating data allocation stability.

Keywords: Web Crawling, Graph Partitioning, Distributed Systems, Hyperlink Analysis, Storage Scaling, Index Optimization.

Manuscript Timeline: Received: May 10, 2000; Revised: June 15, 2000; Accepted: July 09, 2000; Published: August 24, 2000.

Citation: Jenkins, S., & Rao, V. (2000). A Balanced Topological Graph Partitioning Scheme for Distributed Web-Scraping Engines. International Journal of Computer Science and Technology, 2(8), 57–64.

Subscribe to read the full article: https://www.internationalscholarsjournals.org/subscribe-to-read