AWS EKS IP Exhaustion: Causes, Solutions, and Best Practices
#aws#networking#ipv4#kubernetes
Ravi Kyada

Ravi Kyada @ravikyada

About: A Cloud Enthusiast DevOps Engineer working with AWS, Cloud Computing, Automation, CI/CD, and Monitoring. Core Technical Skills: Linux, AWS, Dockers and Kubernetes, Key Technical Skills: Linux, Unix,

Joined:
Apr 22, 2023

AWS EKS IP Exhaustion: Causes, Solutions, and Best Practices

Publish Date: May 23
0 0

Imagine your AWS EKS cluster running smoothly in production, all CI/CD pipelines are working without any issues, services are scaling gracefully, and everything feels like autopilot mode. — Until one day, new pods fail to start.

So what’s going on?

You dive into kubectl get pods and there it is — a bunch of pods stuck in Pending state.

The logs say it all:

Failed to assign an IP address: ENI IP address limit reached.

That’s when it hits you — this isn’t about compute power. It’s a network-level bottleneck. You’re out of allocatable IP addresses in your cluster.

At first, everything seems normal. But as your workloads scale, pods suddenly get stuck in a Pending state. The culprit? IP exhaustion.

Networking is the backbone of Kubernetes on AWS, and if not planned correctly, it can lead to serious scaling issues.

This guide explores why IP exhaustion happens in AWS EKS, how to detect it early, and what best practices you should follow to prevent it.

Understanding IP Exhaustion in AWS EKS

AWS EKS assigns an IP address to every pod using the AWS VPC CNI (Container Network Interface). While this provides native networking integration with AWS, it also means:

  • Every pod consumes an IP from the subnet CIDR block.
  • AWS reserves five IPs per subnet , reducing the available range.
  • Small CIDR blocks fill up quickly, leading to eventual IP exhaustion.

The available IP pool shrinks as your cluster scales, preventing new pods from running.

How AWS EKS Assigns IPs

AWS VPC CNI and Its Role

The AWS VPC CNI plugin manages pod-to-IP allocation using the primary VPC CIDR block. Here’s how it works:

  • Each Kubernetes node gets an Elastic Network Interface (ENI).
  • The ENI gets multiple secondary IPs , assigned to running pods.
  • The number of available IPs depends on the node instance type.

For example, an m5.large instance supports 10 pods per ENI, meaning if you run more, you’ll need another ENI — consuming even more IPs.

Why Does IP Exhaustion Happen?

  1. Subnet Size Limitations
  2. AWS reserves five IPs per subnet.
  3. Small subnets (e.g., /25) run out of addresses faster.
  4. Scaling Issues
  5. More pods = more IPs required.
  6. Without proper planning, clusters grow beyond their available IPs.
  7. Incorrect CIDR Block Allocation.
  8. Using a /25 CIDR for a large cluster is insufficient.
  9. A /20 or /21 CIDR is recommended for high-scale deployments.

Signs That Your Cluster Is Running Out of IPs

  • Pods are stuck in Pending state.
  • aws-node logs show IP exhaustion errors.
  • CloudWatch logs indicate subnet IP depletion.

Run the following command to check available IPs:

kubectl get pods -A -o wide | grep Pending
Enter fullscreen mode Exit fullscreen mode

Or monitor subnet usage with AWS CLI:

aws ec2 describe-subnets --filters "Name=vpc-id,Values=<your-vpc-id>" --query 'Subnets[*].[SubnetId,AvailableIpAddressCount]'
Enter fullscreen mode Exit fullscreen mode

Best Practices to Prevent IP Exhaustion

1. Use a Larger CIDR Block

  • Start with a /20 or /21 instead of /25 or /24.
  • This provides thousands of IPs for your cluster.
  • Example setup: 
aws ec2 create-vpc --cidr-block 10.0.0.0/20
Enter fullscreen mode Exit fullscreen mode

2. Use Nitro-Based Instances

  • Nitro instances support Prefix Delegation , reducing IP overhead.
  • Example: c6g, m6g, and r6g instances allow prefix-based IP allocation.

3. Leverage Secondary CIDR Blocks

  • Add extra CIDR blocks to your VPC.
  • This prevents a single subnet from exhausting all available IPs. 
aws ec2 associate-vpc-cidr-block --vpc-id <your-vpc-id> --cidr-block 10.1.0.0/16
Enter fullscreen mode Exit fullscreen mode

4. Enable Prefix Delegation

Prefix delegation allows ENIs to allocate smaller subnet prefixes instead of full IPs. This significantly reduces IP waste.

To enable it:

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
Enter fullscreen mode Exit fullscreen mode

5. Use AWS PrivateLink for Services

  • Avoid assigning public IPs where unnecessary.
  • AWS PrivateLink routes traffic internally , saving IPs.

Monitoring IP Usage in AWS EKS

Use CloudWatch and aws-node logs to track available IPs:

kubectl logs -n kube-system aws-node | grep "IP exhaustion"
Enter fullscreen mode Exit fullscreen mode

Set up alerts with Amazon CloudWatch:

aws cloudwatch put-metric-alarm --alarm-name IP-Exhaustion-Alert \
  --metric-name AvailableIPCount --namespace AWS/VPC \
  --threshold 10 --comparison-operator LessThanThreshold \
  --evaluation-periods 1 --alarm-actions arn:aws:sns:region:account-id:topic-name
Enter fullscreen mode Exit fullscreen mode

Debugging IP Exhaustion Issues

1. Identify Stuck Pods 

kubectl get pods --all-namespaces -o wide | grep Pending
Enter fullscreen mode Exit fullscreen mode

2. Check IP Availability 

aws ec2 describe-subnets --query 'Subnets[*].[SubnetId,AvailableIpAddressCount]'
Enter fullscreen mode Exit fullscreen mode

3. Restart aws-node DaemonSet 

kubectl rollout restart daemonset aws-node -n kube-system
Enter fullscreen mode Exit fullscreen mode

Case Study: Preventing IP Exhaustion in a Growing EKS Deployment

A SaaS startup faced scaling issues as its EKS cluster outgrew a /25 subnet. Pods were failing, and deployments stalled. The solution:

✅ Migrated to a /20 CIDR block for long-term scalability.

✅ Switched to Nitro instances for Prefix Delegation support.

✅ Used AWS PrivateLink to reduce public IP allocation.

Within a few weeks, they eliminated IP exhaustion issues and achieved seamless scaling.

Conclusion

AWS EKS networking must be planned in advance to avoid IP exhaustion issues. Using a proper CIDR block , enabling Prefix Delegation , and monitoring IP consumption will prevent costly scaling failures.

By implementing these best practices, you can ensure your Kubernetes workloads scale smoothly without hitting network roadblocks. 🚀

FAQ

1. What is AWS VPC CNI, and how does it assign IPs?

AWS VPC CNI assigns IPs from the VPC CIDR block to each pod running in EKS. It does this via Elastic Network Interfaces (ENIs) attached to EC2 nodes.

2. How can I check if my AWS EKS cluster is running out of IPs?

Run:

kubectl get pods -A -o wide | grep Pending
Enter fullscreen mode Exit fullscreen mode

Or check available IPs in subnets:

aws ec2 describe-subnets --query 'Subnets[*].[AvailableIpAddressCount]'
Enter fullscreen mode Exit fullscreen mode

3. What’s the best CIDR block size for an EKS cluster?

A /20 or /21 CIDR block is recommended for large-scale deployments to avoid early IP exhaustion.

4. How does Prefix Delegation improve IP efficiency?

Prefix Delegation allows ENIs to allocate IPs more efficiently , reducing waste and increasing scalability.

5. Can I add more IPs to an existing AWS EKS cluster?

Yes! You can associate a secondary CIDR block to your VPC to expand available IPs without downtime.

By implementing these strategies, you’ll future-proof your AWS EKS networking and avoid painful IP exhaustion issues. 🚀

Thank you so much for reading the article till the end! 🙌🏻 Your time and interest truly mean a lot. 😁📃

If you have any questions or thoughts about this blog, feel free to connect with me:

🔗 LinkedIn: Ravi Kyada

🐦 Twitter: @ravijkyada

Until next time, ✌🏻 Cheers to more learning and discovery! 🇮🇳 🚀

Comments 0 total

    Add comment