preload-icon
Technology
Apr 09.2019

Scaling Node.js microservices on AWS to handle 5M requests per minute

The goal of this article is to share the experience of how to scale a cloud solution (AWS) to handle 5M requests per minute (83K req/sec). (Read more: Scaling Amazon Aurora MySQL)

Load Testing

To find bottlenecks,  load tests were used— 50 tests were held. Every test was started with minimum resources on the cloud to monitor how it scales.

Apache Bench and WRK were used for load testing and Ansible to control bombers.

Scaling Node.js microservices on AWS to handle 5M requests per minute. minute (83K req-sec) Load Testing

Metrics

We logged each test to Google Sheets with metrics, changes, and a description of bottlenecks. CloudWatch provided all the metrics we required.

ELB/ALB

  • HealthyHostCount
  • RequestCount

RDS

  • CPUUtilization
  • DatabaseConnections
  • Queries

Scaling Node.js microservices on AWS to handle 5M requests per minute. minute (83K req-sec)

Microservices on EC2

Let’s start with a simple Microservice architecture on EC2 instances (without ECS). Each microservice was placed on a single EC2 instance. ELB and Auto Scale Group were used for load balancing between instances and for adding new instances on high load.

Scaling Node.js microservices on AWS to handle 5M requests per minute. minute (83K req-sec). Microservises on EC2

In the test, we used three Node.js microservices with three RDS clusters for each service. Microservice 1 receives all requests, authorizes users with Microservice 2 and fetches some data from Microservice 3.

Here are some bottlenecks we found:

Babel — JavaScript compiler

Some microservices uses Babel (babel-node) to compile and execute files. It already uses the latest Node.js version which supports everything we need. We just added ESM to leave ES modules.

AWS stolen CPU

On constant load, CloudWatch shows that CPU load drops from 90% to 40%, but Htop shows 100% CPU load. Auto-scale policy can’t handle this because of the incorrect metric.

This problem is known as “AWS stolen CPU.” If your service depends on constant high CPU load, you should move away from T2 (CPU Credits and Burstable Performance) to instances like C4 or C5 (Compute Intensive).

AWS Limits

657 instances of C4.large type were used to reach 82,000 req/sec. Then we found this was the limit of subnet size. The easiest solution is to use a more powerful instance type, such as C4.4xlarge.

Inter-service communication

On C4.4xlarge, I ran Node.js with PM2 cluster mode and 16 processes. But I still couldn’t reach even 40% CPU load.

Communications between microservices were REST API over HTTP/1.1. The problem was due to the extra overhead caused by the TCP protocol.

The solution was to use the KeepAlive flag and reuse TCP connections for HTTP requests. Other possible solutions would be to use REST over HTTP/2 or gRPC (HTTP/2 under the hood).

Scaling Node.js microservices on AWS to handle 5M requests per minute. minute (83K req-sec). Inter-service communication

Scaling Node.js microservices on AWS to handle 5M requests per minute. minute (83K req-sec). Inter-service communication2

 

So we successfully reached the goal with microservices on EC2. Now let’s try serverless microservices with ECS Fargate.

Microservices on ECS Fargate

ECS is a container orchestration service managed by AWS. Fargate is a compute engine that allows us to set resources for each service and forget about servers. This is a serverless microservice.

Scaling Node.js microservices on AWS to handle 5M requests per minute. minute (83K req-sec)2

Example of Serverless Microservice architecture on ECS Fargate.

Example of Serverless Microservice architecture on ECS Fargate

ECS Fargate

Pros

  • Easy to start with a simple configuration
  • You don’t care about OS, security patches, support, AWS configuration
  • Container Image Repository.

Cons

  • One task (container) is limited to 4 vCPU and 30 GB RAM
  • Lower performance compared to EC2
  • Increasing Fargate limits in our case took a couple of weeks.

EC2 vs. Fargate performance and cost

EC2

  • 82,000 req/sec
  • 657 C4.large (2 vCPU)
  • 2 vCPU, 7.5 GB (C4.xlarge), $0.199 per hour

Fargate

  • 58,667 req/sec
  • 724 tasks (4 vCPU)
  • 4 vCPU, 8 GB, $0.197 per hour

Fargate gives us serverless microservices, but it has a performance and cost penalty in comparison to an EC2 instance.

Conclusion

  • Your AWS VPC subnet should support your future workloads.
  • Check your AWS EC2 and Fargate limits up front.
  • Enable KeepAlive for all interservice communications (can dramatically increase requests performance).
  • Never use Babel for production backend (can add bottlenecks to asynchronous operations and memory leaks).
  • Don’t use T2/T3 for CPU intensive tasks.
  • Don’t use Nginx in front of a microservice — an extra layer that you need to properly configure; it can slow down your req/sec.
  • Use JWT where you can.

TAKEAWAYS

This article shares our knowlege which we use on practice. See our last case study: Increasing the scalability of a cloud-based system for IoT products. If you need some advice with our cloud system, Sirin Software has expertise in this area and is ready to share its knowledge with you. Contact us and we’ll be happy to find the best solution together with you!

Leave a comment

Your Name:

Enter correct name

Email:

Enter correct email

Enter correct message

Check captcha field

HOW CAN WE HELP YOU?

We will get in touch with you within 24 business hours