To update an EC2 instance image for the SaaS servers:

  • Go to Cloud Formation for the cluster to be updated (find it under ECS Configuration | CloudFormation stack).
  • Create a Change Set:
    • standard, use existing template.
    • provide a description “updating instance AMI”
    • for LatestECSOptimizedAMI enter /aws/service/ecs/optimized-ami/amazon-linux-2023/arm64/recommended/image_id
    • Keep everything else unchanged
    • On the last page to review the change set it should show the new AMI ID
      Use this , versus guessing from the AMI picker under the Launch Template editor
    • Create the change set, validate it, and deploy it.
    • This should create a new launch template version (it will not deploy or set to default)

To deploy the new instances:

  • Go to EC2 Launch Template
  • Review the newest version on the list, make sure it has the new AMI ID
  • Set the new version of the template to the default version
  • Go to EC2 Auto Scaling Groups
  • Find Instance Refresh and execute that

This should roll the instances to the new template with the new AMI in place.

KEEP TRACK OF THE PRIOR DEFAULT LAUNCH TEMPLATE ID.
You may need to set the launch template back to the prior valid version of the launch template ID and refresh instances to roll back to the older template if something goes wrong.

When this was done in August 2026, the Auto Scaling Group had some invalid subnets attached. This kept launching EC2 instances that would not auto-attach to the target group and load balancer. This took the production SaaS offline for 5-10 minutes until the services were rolled back. LUCKILY some of the replacement instances randomly picked the right subnet and it worked. It turned out not to be an AMI issue, so the roll back to the prior default only helped by pure change – but it highlights the importance of being able to quickly reset the state of the instances.

It also highlighted the fact that the ASG can deem an instance health if it booted and is running, but the ALB and associated Target Group may still deem the instance “unhealthy” and not allow anything to connect to it making the fault-tolerant scaling group a bit of a false sense of security.

EC2 Auto Scaling Groups

Manages the number of EC2 hosts available.
Here you can run an instance refresh to use the defined launch template and default version of the template to replace running EC2 hosts with new hosts.
If configured properly these should automatically attach to the target groups which then are automatically picked up by the ECS SaaS service.
* The Scaling Group determines which subnets the launched EC2 instances attach to. For the production server this must be the PRIVATE subnets (slp-saas-private-1d), the public cluster subnets will not work.

EC2 Launch Templates

define the general characteristics of the host instance.
This includes the OS image (AMI).
Latest versions are setup to use Amazon Linux 2023.
* see note on CloudFormation below for picking the AMI

Manually launching a new instance via EC2 Launch Templates should auto-connect to the EC2 Target Group and then the ECS Infrastructure stack of Container Instances if the launch template is configured correctly (and connecting to the proper subnet).

EC2 Target Groups

(ecs-slp-saas-production) will automatically connect to EC2 instances via the associated EC2 application load balancer (slp-saas-production). The ALB is connected to the target group by the HTTPS:443 listener.
Should not need to be changed.

EC2 Application Load Balancer

Routes traffic and balances it between multiple EC2 hosts as needed.
Should not need to be changed.

Cloud Formation

Creates a template for building connected AWS resources into an IT stack.
You can use Cloud Formation to create a change set and have it automatically search and find the latest compatible AMI.
for LatestECSOptimizedAMI enter /aws/service/ecs/optimized-ami/amazon-linux-2023/arm64/recommended/image_id

Leave a Reply