WebiU: API Optimization, Scalable Architecture, and AI Enhancements

Personal Information

Abstract

WebiU is a dynamic organization website that fetches project data in real time from GitHub repositories using reusable component architecture. It currently serves as the public face for C2SI’s 300+ projects. While the existing platform works, it faces scalability bottlenecks in its API layer, lacks automated deployment infrastructure, and does not leverage any form of intelligent content generation.

This proposal targets four concrete areas: refactoring APIs for faster and lighter responses, introducing a serverless-compatible real-time data pipeline, automating testing and deployment through GitHub Actions CI/CD, and adding lightweight AI features for project summarization and discoverability. Each area is scoped to deliver a working, testable increment by the end of its timeline phase.

Motivation

WebiU is C2SI’s primary public-facing platform. Every GSoC contributor, potential collaborator, and organizational stakeholder interacts with it. The quality of this platform directly affects how the organization is perceived.

I am drawn to this project for three reasons. First, the problem is real and operational, not academic. Optimizing API response times, handling rate limits at scale, and generating useful project summaries are engineering challenges that exist in production systems everywhere. Second, the existing codebase is mature enough to build on but has clear, well-defined gaps. Third, I have already invested significant effort understanding the WebiU architecture through pre-GSoC contributions, which gives me confidence that the proposed timeline is realistic.

Pre-GSoC Contributions

I completed both pre-GSoC tasks for the WebIU 2026 selection process, demonstrating familiarity with the codebase and the problem domain.

Project Description

Current State

WebiU is built on NestJS (backend), Angular with SSR (frontend), GraphQL/Apollo (API layer), and MongoDB/Mongoose (storage). It fetches repository data from the GitHub API and renders it through configurable component templates.

The current limitations:

1. API performance. Responses include full repository payloads without field-level filtering. There is no response compression. Repeated requests for the same data hit the GitHub API every time.
2. Scalability ceiling. The data fetching runs synchronously in the request path. At 300+ repositories, this produces noticeable latency. At 1,000+, it will become unusable.
3. No deployment automation. Builds and deployments are manual. There is no automated test pipeline, no rollback capability, and no staging environment.
4. No intelligent content. Project descriptions are raw GitHub descriptions. There is no summarization, no automated tech-stack detection, and no semantic search capability.

Proposed Solution

The work is organized into four pillars, each delivering independently testable value.

Pillar 1: API Optimization

Goal: Reduce response times and payload sizes for all frontend-facing endpoints.

Approach:

• Implement field-level selection in the GraphQL schema so the Angular frontend requests only the data it needs for each view (project list, project detail, search results).
• Add GZIP/Brotli compression at the NestJS response layer using compression middleware.
• Introduce an in-memory cache layer (Redis with @nestjs/cache-manager) with a 10-minute TTL between the GraphQL resolvers and MongoDB. This absorbs repeated reads without touching the database.
• Add ETag support for GitHub API requests. Unchanged repositories return 304 Not Modified at zero rate-limit cost.

Measurable outcome: Average API response time drops below 50ms for cached requests. Payload sizes reduce by at least 40% on list views.

Pillar 2: Scalable Real-Time Architecture

Goal: Decouple data ingestion from the request path so the system can handle 1,000+ repositories without degradation.

Approach:

This pillar implements the architecture I designed in Pre-GSoC Task 1:

• Set up Redis as a shared cache and message broker.
• Introduce BullMQ for background job processing within the existing NestJS backend.
• Configure organization-level GitHub webhooks to trigger immediate data refreshes on push, star, repository, and issues events.
• Add a nightly reconciliation cron job (@nestjs/schedule) as a consistency fallback.
• Implement worker concurrency controls to respect GitHub’s secondary rate limits.
• Add a Dead Letter Queue for jobs that fail repeatedly.

Measurable outcome: Full data sync for 1,000 repositories completes in under 15 minutes. Frontend requests never wait for GitHub API calls.

Pillar 3: CI/CD Integration

Goal: Automate testing, building, and deployment with rollback capability.

Approach:

• Create a GitHub Actions workflow for the main branch: lint, unit tests, build, and deploy to the production environment.
• Create a separate workflow for pull requests: lint, unit tests, and build verification without deployment.
• Add automated Lighthouse performance audits on pull requests to catch regressions before merge.
• Implement deployment rollback using Vercel’s instant rollback API, triggered automatically if the post-deploy health check fails.
• Set up a staging environment that deploys from the develop branch for mentor review before production merges.

Measurable outcome: Every merged PR is automatically tested, built, and deployed. A failed deployment rolls back within 60 seconds.

Pillar 4: AI Enhancements

Goal: Improve project presentation and discoverability without increasing system complexity.

Approach:

• Generate concise project summaries (2-3 sentences) from GitHub README content and repository metadata using an LLM API call (OpenAI or a self-hosted alternative). Summaries are generated once and cached in MongoDB alongside repository data.
• Detect technology stacks automatically by parsing repository language data, dependency manifests (package.json, requirements.txt, go.mod, etc.), and CI configuration files. This replaces manually maintained tech badges.
• Enable optional natural-language project search by mapping user queries to existing metadata fields (language, topic, description) using lightweight embedding similarity. This does not require a vector database; it uses pre-computed embeddings stored as arrays in MongoDB.
• All AI outputs are cached and refreshed only when the underlying repository data changes (detected via the webhook/cron pipeline from Pillar 2).

Measurable outcome: Every repository has an auto-generated summary and accurate tech-stack badges. Natural-language search returns relevant results for queries like “machine learning projects in Python.”

Deliverables

Required

Stretch Goals

Timeline

Pre-GSoC (Now through May)

• Continue studying the WebiU codebase, focusing on the NestJS module structure and Angular SSR rendering pipeline.
• Submit additional PRs for bug fixes or small improvements to deepen codebase familiarity.
• Set up local development environment with Redis for early prototyping.

Community Bonding (May 26 to June 15)

• Discuss implementation details with mentors Mahender and Charith.
• Finalize the Redis caching key structure and BullMQ job schema.
• Agree on the GraphQL schema changes and field-selection approach.
• Resolve open questions about AI provider selection and cost constraints.
• Set up staging environment on Vercel.

Coding Period

Availability

I can commit approximately 30 to 35 hours per week during the summer period (June through August). If academic coursework resumes in September, I will adjust to 20 to 25 hours per week but will compensate by front-loading implementation work in the earlier weeks. I have no other GSoC applications or competing commitments during this period.

Post-GSoC Plans

I plan to remain an active contributor to WebIU after GSoC. Specific areas I intend to continue working on:

• Maintaining and iterating on the CI/CD pipeline as new deployment requirements emerge.
• Extending the AI enhancements with the natural-language search feature if it is not completed during GSoC.
• Mentoring future contributors on the real-time architecture and caching systems.
• Contributing to other C2SI projects where the patterns built here (queue-based ingestion, conditional API requests, AI summarization) can be reused.

Technology Choices

References

• WebiU GitHub Repository
• GitHub REST API: Conditional Requests
• GitHub Apps Rate Limits
• BullMQ Documentation
• NestJS Caching
• Pre-GSoC Task 1: System Design
• Pre-GSoC Task 2: Analyzer Tool