Task 1: Designing a Scalable GitHub Data Aggregation System

Objective

Design an architecture for a system that efficiently aggregates repository data from GitHub and serves it to a website while minimizing API usage and ensuring scalability from 300 to 10,000+ repositories.

Executive Summary

This document outlines a robust data aggregation system for WebiU capable of keeping 300 to 10,000+ GitHub repositories synchronized in near real-time. The core philosophy is evolution over revolution: the architecture strictly leverages WebiU’s existing technology stack (NestJS, GraphQL, Mongoose, Axios, Angular SSR), avoiding unnecessary microservices. It proposes a decoupled worker-queue pattern within the existing backend using two minimal, standard additions: Redis and BullMQ.

Architecture Overview

Core Components

1. Ingestion Layer (Event-Driven and Scheduled)

The ingestion layer detects changes without blocking the main API thread.

Primary: GitHub Organization Webhooks

An organization-level webhook listens to push, star, repository, and issues events. The NestJS webhook controller validates incoming payloads using the X-Hub-Signature-256 header mapped against a secret stored in environment variables. Upon validation, the controller immediately responds 200 OK to GitHub (preventing webhook timeouts) and pushes a lightweight job object to the BullMQ queue.

Fallback: Nightly Reconciliation Cron

Scheduled at off-peak hours (e.g., 0 2 * * * for 2:00 AM). It iterates through all active repositories in MongoDB and pushes a sync job to the queue. This guarantees eventual consistency if a webhook delivery fails.

2. Processing Layer (Message Broker and Workers)

Message Broker (BullMQ backed by Redis)

Job payload structure:

{
  "repoName": "webiu",
  "owner": "c2siorg",
  "lastEtag": "W/\"hash\"",
  "priority": 1
}

Webhook-triggered jobs are assigned higher priority than nightly cron jobs.

Worker Nodes (NestJS Workers)

Workers consume jobs from the queue with controlled concurrency (e.g., concurrency: 5) to abide by GitHub’s secondary abuse rate limits. Each worker executes Axios requests, normalizes the GitHub API JSON response, and triggers cache invalidation.

3. Storage and Caching Layer

MongoDB/Mongoose (Persistent Storage)

Stores only vital metadata required for rendering the UI and handling ETags:

@Schema({ timestamps: true })
export class Repository {
  @Prop({ required: true, unique: true }) repoId: number;
  @Prop({ required: true }) fullName: string;
  @Prop() description: string;
  @Prop() language: string;
  @Prop({ default: 0 }) stargazersCount: number;
  @Prop({ default: 0 }) forksCount: number;
  @Prop() latestEtag: string;
  @Prop({ default: 'active' }) status: 'active' | 'archived';
}

Redis Cache (Volatile Storage)

Uses @nestjs/cache-manager. Keys are structured logically: webiu:repo:c2siorg:webiu. TTL is set to 10 minutes, shielding the frontend from database read spikes.

Rate Limit Handling

Authentication Upgrade: GitHub App vs. PAT

A standard Personal Access Token allows 5,000 requests/hour. For 10,000 repos, a single sync exhausts this instantly. Authenticating as a GitHub App raises the limit to 15,000 requests per hour per installation.

The ETag Strategy (Conditional Requests)

Every successful fetch stores the ETag header returned by GitHub. On subsequent fetches, the worker attaches If-None-Match: latestEtag headers.

The math: if 90% of 10,000 repos have not changed in the last 24 hours, GitHub returns 304 Not Modified. 304 responses cost zero rate-limit points. Instead of 10,000 API points, a full sync costs roughly 1,000 points, leaving 14,000 available for dynamic user actions.

Scalability Plan: 300 to 10,000 Repos

At 10,000 repositories, synchronous for loops or Promise.all() arrays cause memory leaks and block the Node.js event loop.

By decoupling the scheduler from the execution environment, the cron job runs in milliseconds (writing 10k JSON objects to Redis). The actual API fetching is handled by BullMQ workers. Because they are queue-driven, horizontal scaling is straightforward: if one worker takes 2 hours, spinning up 4 worker instances processes the queue in 30 minutes, all while respecting a global concurrency limit.

Data Flow

GitHub Webhook / Nightly Cron
  -> NestJS Controller validates payload
  -> Pushes job to BullMQ (Redis)
  -> Worker fetches GitHub API with ETag headers
  -> Normalizes response, writes to MongoDB
  -> Invalidates Redis cache

Frontend Request (Angular SSR)
  -> GraphQL query to NestJS API Gateway
  -> Apollo Resolver checks Redis cache
  -> Cache hit: returns in <15ms
  -> Cache miss: queries MongoDB, repopulates cache

Failure Handling

Technology Stack

References

• Conditional requests and 304 behaviour
• GitHub Apps rate limits
• Secondary rate limits