Skip to main content

Optimizing Feed APIs for Performance and Scalability

LikeMinds recently undertook an important initiative to optimize several core APIs powering our feed system—one of the most high-traffic components in our platform. These endpoints directly influence content creation, consumption, and interaction, and are critical for ensuring a seamless user experience. Performance and reliability at scale were the central focus.

Overview of APIs Optimized​

  • Create Feed Post API: Allows users to create new posts within a community.
  • Feed Post Like API: Enables users to express appreciation for posts.
  • Feed Post Comment API: Supports commenting on feed posts and nested discussions.
  • Fetch Topics(s) API: Retrieves topic-based post collections (used in feed tab structure).
  • Fetch Single Feed Post API: Provides detailed data for an individual feed post, including media, metadata, and interactions.

These endpoints are foundational to our engagement model, serving both read-heavy and write-intensive operations across communities.

Why Was Optimization Needed?​

Through internal load testing simulations, we observed that several feed APIs exhibited degraded performance under high-concurrency scenarios—especially beyond 500 to 1000 concurrent users. Some endpoints experienced over 10% failure rates, with requests hitting gateway timeout thresholds due to excessive response latency.

Although the platform follows a horizontally scalable microservices architecture, scaling services linearly is not an optimal or cost-effective solution. API-level inefficiencies—like blocking I/O, redundant computation, and sequential execution—can lead to subpar system behavior even with sufficient hardware provisioning.

Our goal was to minimize latency and maximize throughput, all while using minimal computational resources. This meant refining internal API logic, leveraging caching strategies, and introducing concurrency wherever applicable.

Load Testing Insights​

Using tools such as JMeter and internal benchmarking scripts, we simulated real-world concurrency levels and identified bottlenecks:

  • Latency spikes under load due to sequential processing
  • I/O blocking when interacting with external services (e.g., user access checks)
  • Repeated downstream API calls across endpoints
  • Unnecessary data computation and serialization

These insights shaped the foundation for our optimization efforts and enabled a measurable improvement cycle based on controlled re-tests.

Key Optimization Strategies​

1. Caching Authorization Checks​

A core internal method, fetch member access, was responsible for validating user roles and permissions by interfacing with our central identity service (Django-based). This method was invoked by nearly all feed-related APIs.

To reduce network-bound latency and improve time-to-response, we implemented Redis-based caching for the results of this method. The cached access control state was intelligently invalidated on permission changes, ensuring both performance and correctness.

2. Concurrent Execution of Independent Tasks​

We identified independent logical units within API request lifecycles that were previously executed sequentially. By using concurrency constructs such as sync.WaitGroup in Go, we parallelized these units to reduce blocking and aggregate results efficiently:

  • Fetching access control metadata
  • Parsing tagged users from post/comment text
  • Checking post moderation status in Redis
  • Generating user activity logs
  • Preparing platform-specific headers

This concurrency-first refactor significantly improved response time without introducing race conditions or logical conflicts.

3. Endpoint-Specific Improvements​

a. Create Feed Post API​

Functionality: Enables community members to publish new feed posts with optional media, mentions, and tags.

Optimizations Applied:

  • Downstream access checks and text parsing made concurrent.
  • Deferred validation logic for conditional fields to avoid redundant operations.
  • Reduced I/O latency by eliminating synchronous waits on optional attributes.

b. Feed Post Like API​

Functionality: Allows users to like or unlike posts.

Optimizations Applied:

  • Parallelized retrieval of post metadata and like state to avoid serialized DB hits.
  • Short-circuited logic paths when redundant conditions were met.
  • Improved idempotency and error handling under high request volumes.

c. Feed Post Comment API​

Functionality: Allows users to comment on posts and mention other users.

Optimizations Applied:

  • Early returns implemented for validation failures to reduce compute waste.
  • Metadata generation (user ID, platform version, client info) centralized and reused.
  • Notification dispatch logic was moved to asynchronous execution to minimize response blocking.
  • Concurrent execution of comment persistence and activity logging using wait groups.

d. Fetch Single Feed Post API​

Functionality: Retrieves a specific feed post along with related metadata (likes, comments, media).

Optimizations Applied:

  • Independent I/O operations—such as comment retrieval and media resolution—were made concurrent.
  • The fetchPostWithReplies() logic was decomposed into subtasks for better control and performance.
  • Enhanced lazy-loading support for large media blocks to improve initial response time.

4. Other Enhancements​

  • Caching Layer: Redis used to cache semi-static data like topics, roles, and post metadata. Custom TTLs ensured balance between speed and freshness.
  • Payload Slimming: Trimmed unnecessary fields in API responses, reducing JSON payload size and client-side parse time.
  • Batching Writes: Introduced queuing and batching logic for frequent write operations like likes and comments (ongoing phased rollout).
  • Async Offloading: Time-consuming side effects—like logging, notification triggers, and activity writes—moved to background workers via goroutines or Celery, reducing end-user perceived latency.

Results & Impact​

Following the optimization rollout, our APIs demonstrated measurable improvements:

  • Response Time improved by 30–70% on average across endpoints
  • DB Query Count reduced significantly, minimizing CPU usage and connection pool pressure
  • Error Rate dropped to <1% even under 1000+ concurrent requests
  • Throughput increased considerably, with APIs sustaining higher request-per-second (RPS) rates without infrastructure scaling

These enhancements collectively improved system stability, reduced error recovery overhead, and ensured a more seamless experience for users during peak usage.

Conclusion​

Optimizing APIs is not just a performance exercise—it’s about building resilient, scalable, and cost-effective backend systems. At LikeMinds, we believe that proactive optimization is as important as feature delivery.

This project reinforced a fundamental engineering principle: "Measure frequently, validate rigorously, and iterate systematically."

By combining load testing, caching, concurrency, and architectural adjustments, we successfully built a feed system that performs reliably—even under extreme concurrency scenarios.