cables in a tunnel by chatgpt
cables in a tunnel by chatgpt

Recently stood up a new ECS container for the Store Locator Plus® staging app. We can connect to the app and login, but the app cannot talk to the outside world. It is not connecting to WordPress news service and cannot validate a WP SMTP Pro license.

This is the notebook for resolving that issue.

Research

Use Private Subnets : Stack Overflow

A Stack Overflow article that nearly describes our situation.

You are using the awsvpc network mode. This means that each ECS task gets its own Elastic Network Interface (ENI). With this configuration, the ECS tasks do not use the underlying EC2 instance’s network connection, they have their own network connection, with their own IP addresses.

  • You are currently disabling public IP assignment to your ECS tasks in the ECS service network_configuration block. You will need to change assign_public_ip to true in order to have ECS assign public IP addresses to the ECS Task’s ENIs, so that the ECS tasks can access resources outside of the VPC.
    • I forgot you can’t use public IP with awsvpc ECS deployed to EC2. You can only do that with Fargate deployments. So your options are to use a different network mode, so you can use the EC2 instance’s public IP from your ECS task: docs.aws.amazon.com/AmazonECS/latest/bestpracticesguide/… or switch to private subnets and a NAT Gateway.

Amazon ECS Networking Documents

Enable VPC internet access using internet gateways

“An internet gateway enables resources in your public subnets (such as EC2 instances) to connect to the internet if the resource has a public IPv4 address or an IPv6 address. “

Configuration for Internet Access

  • Add a route to the route table for the subnet that directs internet-bound traffic to the internet gateway.
  • Ensure that instances in your subnet have a public IPv4 address or IPv6 address. For more information, see Instance IP addressing in the Amazon EC2 User Guide.
  • Ensure that your security groups and network access control lists allow the desired internet traffic to flow to and from your instances.

Connect Amazon ECS applications to the internet

ECS Stack

EC2 Instance

  • ID i-0a1fa80a12dda9903
  • VPC: vpc-1b5e2c7c (slp-cluster-vpc)
    • Route Table: rtb-f0a96096 (slp-cluster-default-router)
      • 0.0.0.0/0 => igw-ab334cf
  • Security Group: sg-020cdb78 (default)
  • Network ACL: acl-e088aa87 (slp-cluster-acl)
  • EC2 Subnet: subnet-7b232951 (slp-cluster-east-1c)

ECS Service Details

Task Definition: slp_saas_staging:4
arn:aws:ecs:us-east-1:744590032041:task-definition/slp_saas_staging:4

Load Balancer: application load balancer myslp-staging-alb
arn:aws:elasticloadbalancing:us-east-1:744590032041:loadbalancer/app/myslp-staging-alb/2eae5893f2db5c1b

Target Group: ecs-myslp-staging-target
arn:aws:elasticloadbalancing:us-east-1:744590032041:targetgroup/ecs-myslp-staging-target/331cd16e4b3c52e1

VPC: slp-cluster-vpc

From AWS Support

Summary of problem

ECS instances using AWSVPC have no public IP address on the running instance/container and thus cannot route through the Internet Gateway despite it being on the VPC and subnet where the EC2 instance and container are attached.

Resolution Summary

To resolve this issue, we need to separate the subnets for your ECS tasks and the ALB, and configure the routing appropriately.

Keep Existing ALB / IG Subnets Unchanged

Keep the existing public subnet(s) for your ALB unchanged, with the Internet Gateway attached.

myslp-staging-alb

DNS Name: myslp-staging-alb-1533129727.us-east-1.elb.amazonaws.com

VPC: slp-cluster-vpc (vpc-1b5e2c7c)

Subnets:
slp-cluster-east-1a : subnet-7c8a8124 us-east-1a (use1-az6)
slp-cluster-east-1c : subnet-7b232951 us-east-1c (use1-az2)
slp-cluster-east-1d : subnet-5213e91b us-east-1d (use1-az4)
slp-cluster-east-1e : subnet-d00210ed us-east-1e (use1-az3)

Internet Gateway: slp-cluster-gateway (igw-ab3d34cf)
attached to slp-cluster-vpc

Create A NAT Gateway

Create New Private Subnets To Match The Public Subnets

Create A New Route Table And Associate With The Private Subnets

And add a route to the general internet (0.0.0.0/0) that goes through the NAT gateway.

ECS Service Changes

Note: You cannot update the network configuration of an existing ECS service. Therefore, you need to recreate the service.

Put the new service on the private subnets only.

Autoscaling Group Updates

Update the auto scaling group to add the private subnet with the NAT gateway.

Resolution Summary

The container using AWSVPC will not have a public IP address. That means the automatic routing for outbound connections will never use the Internet Gateway.

You need to setup a VPC with a public subnet (we have 4 zones , A C D E) and private subnet in those same zones.

The cluster will setup an automatic scaling group, something like Infra-ECS-Cluster….* which will define the Auto Scaling group via the infrastructure subcomponent of the cluster.

The Auto Scaling Group(ASG) needs to include both the private and public subnets.

The EC2 instances it spins up can be in the private subnet (let ASG decide).

The cluster service will setup the application load balancer (ALB) and target group. The service must be placed in the private subnet only. This will ensure the subsequent tasks (container instances) run on the private subnet. The ALB that is created must be assigned to the public subnets on the VPC to allow general inbound traffic from the internet to find its way over to the container on the private subnet. As a side note, the target group listens on HTTPS port 443 and routes to the container HTTP port 80. Use the service to create the ALB and target groups.

On the VPC…

Make sure the default routing table is explicitly assigned to the public subnets and NOT the private subnets.

Create an Internet Gateway (IG) and attach it to the VPC. This will allow inbound traffic from the internet to any service on the VPC with a public IP, in our case the application load balancer listener on port 443.

Create a NAT Gateway. Assign all the private subnets to be part of the NAT Gateway.

Create a second routing table and assign the private subnets to this table. Add a route to 0.0.0.0/0 that goes through the NAT Gateway.

If there are any tasks, containers, or EC2 instances already running stop and reinstantiate each of them. If the service was created originally on the public subnet of the VPC it will need to be deleted and recreated on the private VPC subnets.