Hierarchical Hybrid Indexing for Ultra-High-Dimensional Similarity Retrieval in Large-Scale Information Systems

• Bilal Ahmed

  Ghazi University, Multan–D.G. Khan Road No. 15, Department of Information Technology, Dera Ghazi Khan, Punjab, Pakistan

  Author

• Imran Siddiq

  Lasbela University of Agriculture, Water and Marine Sciences, Uthal Road No. 28, Department of Computer Systems, Uthal, Balochistan, Pakistan

  Author

• Ariful Islam

  Dhaka, Bangladesh

  Author

Large-scale information systems increasingly represent content, users, and interactions as vectors whose dimensionality can reach tens of thousands or more due to multi-modal encoders, token-level pooling, and feature concatenation across heterogeneous signals. Similarity retrieval in this regime is constrained by two interacting effects: the geometric concentration that weakens distance contrast as dimension grows, and the systems-level pressure to serve low-latency top-$k$ results under high concurrency while storing billions of vectors. This paper develops a hierarchical hybrid indexing approach for ultra-high-dimensional similarity retrieval that combines multi-resolution routing, adaptive compression, and locality-preserving neighborhood navigation across memory and secondary storage tiers. The proposed design decomposes the search process into a coarse routing stage that narrows the candidate region using structured projections and coarse quantization, a mid-level filtering stage that applies subspace-adaptive binary sketches to control candidate fanout, and a fine stage that performs graph-based navigation with residual-aware distance estimation and selective exact refinement. The hierarchy is constructed to make the dominant costs depend primarily on intrinsic dimension and local neighborhood complexity rather than ambient dimension. A mathematical model connects quantization distortion, sketch collision probability, and graph expansion to expected recall and latency, enabling explicit parameterization under fixed compute and I/O budgets. Implementation considerations are addressed for distributed deployments, including shard-aware routing, tiered caching, incremental updates, and failure isolation. Empirical characterization on representative workloads indicates that the hybrid hierarchy can sustain high recall at bounded tail latency while reducing memory footprint relative to single-level graph or pure quantization baselines.

• PDF

Copyright (c) 2026 International Journal of Science, Engineering and Applied Technologies

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.