More complicated EKS scenarios: EKS managed worker nodes without internet access

Are you using EKS managed worker pools? If you don’t have any specific reasons for not using them, you should. It saves tons of time plus it boosts the “managed Kubernetes” feeling. However, this pretty new offering did not cover one specific use case: cluster with no Internet access.

By default, workers need Internet access so they can pull Docker images and register to the control plane. On the other hand, no internet access is a pretty common requirement, especially in regulated businesses.

So how to find some common ground here?

For the record, it really does not work by default. When we put everything to the private subnets with no outbound Internet access, workers can’t join the cluster.

Community for the rescue

If you know me, you know what I did. Search engine! And I was very successful cause I’ve found this GitHub issue after five minutes of searching. What’s going on there?

Mike Stefaniak from the EKS team is saying there that it can work as long as you set up the other required PrivateLink endpoints correctly.

Moreover, there’s also a link to the GitHub repository with some CloudFormation samples. Challenge accepted. I’m gonna build it 💪

Terraforming the CloudFormation stack

I’m not that much into CloudFormation. Not yet. So I had to rewrite it to Terraform first. Check the key components below, I’ll also add some explanation.

VPC settings

Note the property enable_dns_support, this part is required by private EKS endpoint. See more details in the documentation.

resource "aws_vpc" "main" {
  cidr_block           = "10.20.0.0/16"
  enable_dns_support   = true
  tags = {
    "kubernetes.io/cluster/cl01" = "shared"
  }
}

Private subnets

Also, let’s add some basic private networks.

resource "aws_subnet" "private-0" {
  availability_zone = "eu-central-1a"
  cidr_block        = "10.20.1.0/24"
  vpc_id            = aws_vpc.main.id

  tags = {
    "kubernetes.io/cluster/cl01" = "shared"
  }
}

resource "aws_subnet" "private-1" {
  availability_zone = "eu-central-1b"
  cidr_block        = "10.20.2.0/24"
  vpc_id            = aws_vpc.main.id

  tags = {
    "kubernetes.io/cluster/cl01" = "shared"
  }
}

resource "aws_subnet" "private-2" {
  availability_zone = "eu-central-1c"
  cidr_block        = "10.20.3.0/24"
  vpc_id            = aws_vpc.main.id

  tags = {
    "kubernetes.io/cluster/cl01" = "shared"
  }
}

Endpoint for EC2 API

resource "aws_security_group" "endpoint_ec2" {
  name   = "endpoint-ec2"
  vpc_id = aws_vpc.main.id
}

resource "aws_security_group_rule" "endpoint_ec2_443" {
  security_group_id = aws_security_group.endpoint_ec2.id
  type              = "ingress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  cidr_blocks = [
    10.20.1.0/24, // private subnet 1
    10.20.2.0/24, // private subnet 2
    10.20.3.0/24, // private subnet 3
  ]
}

resource "aws_vpc_endpoint" "ec2" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.eu-central-1.ec2"
  vpc_endpoint_type   = "Interface"
  private_dns_enabled = true
  subnet_ids = [
    aws_subnet.private-0.id,
    aws_subnet.private-1.id,
    aws_subnet.private-2.id,
  ]

  security_group_ids = [
    aws_security_group.endpoint_ec2.id,
  ]
}

Endpoint for ECR/Docker APIs

resource "aws_security_group" "endpoint_ecr" {
  name   = "endpoint-ecr"
  vpc_id = aws_vpc.main.id
}

resource "aws_security_group_rule" "endpoint_ecr_443" {
  security_group_id = aws_security_group.endpoint_ecr.id
  type              = "ingress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  cidr_blocks = [
    10.20.1.0/24, // private subnet 1
    10.20.2.0/24, // private subnet 2
    10.20.3.0/24, // private subnet 3
  ]
}

resource "aws_vpc_endpoint" "ecr_api" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.eu-central-1.ecr.api"
  vpc_endpoint_type   = "Interface"
  private_dns_enabled = true
  subnet_ids = [
    aws_subnet.private-0.id,
    aws_subnet.private-1.id,
    aws_subnet.private-2.id,
  ]

  security_group_ids = [
    aws_security_group.endpoint_ecr.id,
  ]
}

resource "aws_vpc_endpoint" "ecr_dkr" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.eu-central-1.ecr.dkr"
  vpc_endpoint_type   = "Interface"
  private_dns_enabled = true
  subnet_ids = [
    aws_subnet.private-0.id,
    aws_subnet.private-1.id,
    aws_subnet.private-2.id,
  ]

  security_group_ids = [
    aws_security_group.endpoint_ecr.id,
  ]
}

Endpoint for s3 API

In this part, we’re associating s3 private gateway with our private subnets’ routing tables. See more details in the documentation. We need this API as images are stored in S3 buckets. So when it’s not enabled, nodes won’t be able to start essential services (pods).

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.eu-central-1.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids = [
    aws_route_table.private-0.id,
    aws_route_table.private-1.id,
    aws_route_table.private-2.id,
  ]
}

resource "aws_route_table" "private-0" {
  vpc_id = aws_vpc.main.id
}

resource "aws_route_table" "private-1" {
  vpc_id = aws_vpc.main.id
}

resource "aws_route_table" "private-2" {
  vpc_id = aws_vpc.main.id
}

resource "aws_route_table_association" "private-0" {
  subnet_id      = aws_subnet.private-0.id
  route_table_id = aws_route_table.private-0.id
}

resource "aws_route_table_association" "private-1" {
  subnet_id      = aws_subnet.private-1.id
  route_table_id = aws_route_table.private-1.id
}

resource "aws_route_table_association" "private-2" {
  subnet_id      = aws_subnet.private-2.id
  route_table_id = aws_route_table.private-2.id
}

EKS

Now are somehow ready to start the control plane. Please note I don’t mention there IAM resources and Security Groups as it would make the whole post even bigger. But don’t worry, Hashicorp prepared a beautiful tutorial so you can use those.

In the following example note the property vpc_config.endpoint_private_access. This is the private EKS endpoint I was talking about.

resource "aws_eks_cluster" "master" {
  name     = var.cluster_name
  role_arn = aws_iam_role.default-master.arn

  vpc_config {

    subnet_ids = [
      aws_subnet.private-0.id,
      aws_subnet.private-1.id,
      aws_subnet.private-2.id,
    ]

    endpoint_private_access = true
    endpoint_public_access  = true
    public_access_cidrs = [
      "0.0.0.0/0"
    ]
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_master_default_policy,
    aws_iam_role_policy_attachment.eks_master_default_service_policy,
  ]
}

And that’s pretty much it for the control plane. Let’s proceed to managed worker nodes!

resource "aws_eks_node_group" "aza" {
  cluster_name    = aws_eks_cluster.master.name
  node_group_name = "AZa"
  node_role_arn   = aws_iam_role.default-worker.arn
  instance_types  = ["t3.small"]
  subnet_ids = [
    aws_subnet.private-0.id,
  ]

  scaling_config {
    desired_size = 1
    max_size     = 2
    min_size     = 1
  }

  lifecycle {
    ignore_changes = [
      scaling_config[0].desired_size
    ]
  }

  depends_on = [
    aws_iam_role_policy_attachment.default-AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.default-AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.default-AmazonEC2ContainerRegistryReadOnly,
  ]
}

resource "aws_eks_node_group" "azb" {
  cluster_name    = aws_eks_cluster.master.name
  node_group_name = "AZb"
  node_role_arn   = aws_iam_role.default-worker.arn
  instance_types  = ["t3.small"]
  subnet_ids = [
    aws_subnet.private-1.id,
  ]

  scaling_config {
    desired_size = 1
    max_size     = 2
    min_size     = 1
  }

  lifecycle {
    ignore_changes = [
      scaling_config[0].desired_size
    ]
  }

  depends_on = [
    aws_iam_role_policy_attachment.default-AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.default-AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.default-AmazonEC2ContainerRegistryReadOnly,
  ]
}

resource "aws_eks_node_group" "azc" {
  cluster_name    = aws_eks_cluster.master.name
  node_group_name = "AZc"
  node_role_arn   = aws_iam_role.default-worker.arn
  instance_types  = ["t3.small"]
  subnet_ids = [
    aws_subnet.private-2.id,
  ]

  scaling_config {
    desired_size = 1
    max_size     = 2
    min_size     = 1
  }

  lifecycle {
    ignore_changes = [
      scaling_config[0].desired_size
    ]
  }

  depends_on = [
    aws_iam_role_policy_attachment.default-AmazonEKSWorkerNodePolicy,
    aws_iam_role_policy_attachment.default-AmazonEKS_CNI_Policy,
    aws_iam_role_policy_attachment.default-AmazonEC2ContainerRegistryReadOnly,
  ]
}

Please note that we’re using 3 managed node groups. That’s because we are following the official recommendation about the stateful applications and cluster autoscaler. In any perspective, this setup can’t cause any harm.

Rise of the EKS cluster

We’re ready to go. Let’s give it a spin! The creation of such a setup takes almost 20 minutes. I usually use such time for emailing or Slack conversations but there might be a better way to utilize that 😄

terraform apply

After 20 minutes, we can view our brand new cluster in the console.

It’s truly beautiful and guess what, even all the Kubernetes components are up and running! This means that worker nodes were able to pull everything via configured private endpoints.

NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   aws-node-687nk             1/1     Running   0          88s
kube-system   aws-node-mppfc             1/1     Running   0          78s
kube-system   aws-node-s9d4z             1/1     Running   0          82s
kube-system   coredns-5b5455fd88-mdzs5   1/1     Running   0          5m19s
kube-system   coredns-5b5455fd88-znqmz   1/1     Running   0          5m19s
kube-system   kube-proxy-d5jbs           1/1     Running   0          82s
kube-system   kube-proxy-fhsxt           1/1     Running   0          88s
kube-system   kube-proxy-w2sbp           1/1     Running   0          78s

Wrap

Let’s summarize the specification of this setup:

  • 3 private subnets
  • VPC endpoints for ECR
  • EC2 endpoint
  • S3 endpoint
  • EKS control plane with private endpoint
  • no internet access

We’ve just witnessed that it is absolutely possible to create totally isolated clusters even for the most demanding customers. Moreover, official documentation will reflect this setup soon! Don’t forget that such setup can’t pull images from the public repositories, so there’s always an extra step needed for each application deployed: pushing its images to the ECR repository.

Seriously, I’m really glad that AWS is visible on GitHub and they actually care about the users of their products. I’d be totally lost without that advice I got there.