GITHUB UNIVERSE

GitHub Universe is an experimental data visualization project that maps public GitHub activity into an explorable cosmos. Using a custom Python scraper with query-bucket partitioning and deep-scrape follower modes, the pipeline statically compiles millions of data points into a high-performance 4-level SVG and WebGL drill-down interface: Countries (Level 0), Cities (Level 1), Users (Level 2), and Repositories (Level 3).

Data Pipeline

Visualization UI

Engineering Decisions

• Static-Data Pipeline: Bypasses runtime backend requirements by compiling scraped data into `data/index.json` and partitioned `data/countries/{CODE}.json` files, achieving zero-latency data serving on Vercel.
• Query-Bucket Partitioning: The scraper logic segments API requests by 'follower' and 'year' buckets to circumvent top-result API caps, yielding a higher density of data representation per country node.
• Adaptive Rendering: Incorporates an automatic performance degradation engine (auto/high/mid/low) with a live FPS HUD, culling off-screen elements via wide-spread layouts and de-crowding algorithms to maintain 60 FPS in dense views.

The Goal: A Cosmic Map

The primary objective is to map the vast, invisible landscape of the open-source community into a tangible, explorable cosmos. By treating GitHub not just as a database, but as a digital universe, the visualization allows users to navigate geographically—zooming seamlessly from a macro-galactic view of global countries down into specific cities, and eventually into the individual users (stars) and their repositories (satellites). It transforms raw, tabular API data into an interactive, spatial map of global software engineering density.

The Challenge: Scale & API Limits

The most formidable engineering hurdle was the sheer scale of data retrieval combined with GitHub's strict API rate limits. Scraping a global population sequentially takes an unfeasible amount of time, and the API hard-caps pagination at 1,000 results per query. To bypass this, the scraper was engineered with an advanced "Query-Bucket Partitioning" algorithm. By dynamically slicing search requests into granular sub-buckets (based on exact follower counts and account creation years), the system forces the API to expose deeper layers of hidden users. Additionally, a fault-tolerant state-resume mode was built to handle network drops over continuous multi-day scraping operations.