Protect APIs with Rate Limiting via NGINX Configuration
Practical guide to implementing rate-limiting on APIs deployed on Minikube server
In this post, we'll play around with NGINX config to limit requests coming to our web app deployed in the cluster.
While working on his problem initially I was able to find a lot of blogs, and websites that tell us all low-level configurations on how to rate limit but rarely anything which would have helped me implement the configuration using Configmap, and ingress files on our servers (For DevOps noobs).
My objective is to string together all the distinct resources to practically implement rate limiting on our system in one go and make you comfortable enough to start fine-tuning your servers.
Let's follow the lucid Kubernetes documentation to set a mini cluster on your Minikube:
After you are done deploying your web apps and applying the ingress files, we can move to config changes!
Now let's list out the pods in namespace "ingress-nginx"
~ kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-xt9qb 0/1 Completed 0 4d3h
ingress-nginx-admission-patch-7psbh 0/1 Completed 0 4d3h
ingress-nginx-controller-5959f988fd-rmcsp 1/1 Running 5 (13m ago) 4d3h
To find the default Nginx configuration we can go inside the ingress-nginx-controller pod:
~ kubectl exec -it ingress-nginx-controller-5959f988fd-rmcsp -n ingress-nginx /bin/bash
Find the file located in the path /etc/nginx/nginx.conf
~ cat nginx.conf
daemon off;
worker_processes x;
....
http {
.....
.....
location / {
set $namespace "";
set $ingress_name "";
....
....
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
...
...
proxy_cookie_domain off;
proxy_cookie_path off;
}
# backend for when default-backend-service is not configured or it does not have endpoints
server {
listen xxxx default_server reuseport backlog=xxx;
......
location / {
return 404;
}
}
# default server, used for NGINX healthcheck and access to nginx stats
server {
listen 127.0.0.1:xxxx;
....
# Stream Snippets
}
Now at this point, we might get intimated by the code in the conf. Let's dive into some basics of NGinx config
The HTTP/server/location snippets in the Nginx.conf file can be customised for our server's use case
Nginx config comprises modules and directives using which we can customise the request handling by our servers
For instance, using the location directive we can configure the request with a specified URI to return 404 or redirect.
We can set headers to allow specific methods such as POST, and GET requests and restrict DELETE requests if needed.
Now for our use case to limit requests by users to protect our servers from a DDoS attack, we would be diving into the limit_req directives
The limit_req directive, which needs a zone parameter, and also provides optional burst and nodelay parameters.
There are multiple concepts at play here:
zonelets you define a bucket, a shared ‘space’ in which to count the incoming requests. All requests coming into the same bucket will be counted in the same rate limit. This is what allows you to limit per URL, per IP.burstis optional. If set, it defines how many exceeding requests you can accept over the base rate.nodelayis also optional and is only useful when you also set aburstvalue.
Before we assign a variable x = 5, we need to declare the variable before it can hold a value in memory. Similarly, before we can use the zone we need to define/declare it like you would in any programming language.
Step 1: Define the limit_req_zone
When you set a zone, you define a rate, like 300r/m to allow 300 requests per minute, or 5r/s to allow 5 requests each second.
For instance:
limit_req_zone $binary_remote_addr zone=zone1:10m rate=300r/m;limit_req_zone $binary_remote_addr zone=zone2:10m rate=5r/s;
Here, the states are kept in a 10 megabyte zone “zone1”, and an average request processing rate for this zone cannot exceed 5 requests per second or 300 requests per minute
A client IP address serves as a key. Note that instead of $remote_addr, the $binary_remote_addr variable is used here. The $binary_remote_addr variable’s size is always 4 bytes for IPv4 addresses or 16 bytes for IPv6 addresses. The stored state always occupies 64 bytes on 32-bit platforms and 128 bytes on 64-bit platforms. One megabyte zone can keep about 16 thousand 64-byte states or about 8 thousand 128-byte states.
To generalise,
Syntax: |
|
|---|---|
Default: | — |
Context: |
|
Sets parameters for a shared memory zone that will keep states for different keys. In particular, the state stores the current number of excessive requests. The key can contain text, variables, and their combination. Requests with an empty key value are not accounted for.
Here's configmap file we will apply to define limit_req_zone
Now we'll again go inside the ingress-nginx-controller pod and try to find the below change under http snippet:
# Custom code snippet configured in the configuration configmap
limit_req_zone $binary_remote_addr zone=one_zone:10m rate=1r/m;
At this point you can try to play around with the example code to add any custom configuration and see what is allowed.
List of directives to play around with!
Step 2: Apply limit_req using Ingress file
Next on we will be applying the directive defined to our server's endpoints using ingress files
kubectl apply -f rate-limit-ingress.yaml
At this point, you can strategically use different annotations to position your zone under different snippets
nginx.ingress.kubernetes.io/configuration-snippet: |
limit_req zone=one_zone burst=6 nodelay;
limit_req_status 429;
Using the configuration snippet the zone would be placed under each of the location blocks in our config file, hence limiting requests for all the endpoints specified.
location = /v2 {
set $namespace "default"; set $ingress_name "rate-limit-ingress"; set $service_name "web2"; set $service_port "8080"; set $location_path "/v2"; set $global_rate_limit_exceeding n;
....
# mitigate HTTPoxy Vulnerability # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/ proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_request_buffering on; proxy_http_version 1.1;
proxy_cookie_domain off; proxy_cookie_path off;
...
limit_req zone=one_zone burst=6 nodelay;
limit_req_status 429;
...
}
Similarly, different snippets like server, http, stream can be customised using the directives we define using Configmap.
That's it our web app is now fortified against DDoS attacks!
Common errors you might face while applying the ingress file:
➜ ~ kubectl apply -f rate-limit-ingress.yaml
Error from server (BadRequest): error when creating "rate-limit-ingress.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request:
-------------------------------------------------------------------------------
Error: exit status 1
...
...
nginx: [emerg] unknown "limit_key" variable
nginx: configuration file /tmp/nginx/nginx-xxxxxxxxx test failed
This is the indication that your config map hasn't declared the variable in the nginx.conf, hence you can't use it duh
Time to dig into the nginx.conf file and investigate your config file for any errors
~ kubectl apply -f nginx.configmap.yaml
The request is invalid: patch: Invalid value: "map[data:map[http-snippet:<nil>] metadata:map[annotations:map[kubectl.kubernetes.io/last-applied-configuration:{......}]] server-snippet:limit_req_zone $binary_remote_addr zone=one_zone:10m rate=1r/m;\n]": strict decoding error: unknown field "server-snippet
This means that we cannot declare our limit_req_zone directive under server block. It has to be under the http block which encompasses the server block itself.
Here are a few resources to dive deeper into the low-level configuration :
https://www.freecodecamp.org/news/nginx-rate-limiting-in-a-nutshell-128fe9e0126c/
https://www.nginx.com/blog/microservices-march-protect-kubernetes-apis-with-rate-limiting/
https://nginx.org/en/docs/http/ngx_http_core_module.html#limit_rate
Hopefully you found it easy to understand and comprehensive.
PS: Feedback on the blog content is much appreciated and thanks for reading :D
Cheers!