Why can't my ECS service register available EC2 instances with my ELB?

Question

I've got an EC2 launch configuration that builds the ECS optimized AMI. I've got an auto-scaling group that ensures that I've got at least two available instances at all times. Finally, I've got a load balancer.

I'm trying to create an ECS service that distributes my tasks across the instances in the load balancer.

After reading the documentation for ECS load balancing, it's my understanding that my ASG should not automatically register my EC2 instances with the ELB, because ECS takes care of that. So, my ASG does not specify an ELB. Likewise, my ELB does not have any registered EC2 instances.

When I create my ECS service, I choose the ELB and also select the ecsServiceRole. After creating the service, I never see any instances available in the ECS Instances tab. The service also fails to start any tasks, with a very generic error of ...

service was unable to place a task because the resources could not be found.

I've been at this for about two days now and can't seem to figure out what configuration settings are not properly configured. Does anybody have any ideas as to what might be causing this to not work?

I think this may have something to do with the ECS_CLUSTER user data setting.

In my EC2 auto-scaling launch configuration, if I leave the user data input completely empty, the instances are created with an ECS_CLUSTER value of "default". When this happens, I see an automatically-created cluster, named "default". In this default cluster, I see the instances and can register tasks with the ELB like expected. My ELB health check (HTTP) passes once the tasks are registered with the ELB and all is good in the world.

But, if I change that ECS_CLUSTER setting to something custom I never see a cluster created with that name. If I manually create a cluster with that name, the instances never become visible within the cluster. I can't ever register tasks with the ELB in this scenario.

Any ideas?

Answer 1

ECS needs the EC2 instance’s public IP addresses to directly communicate with them. I think you might have provided the Private IPs of the instances to the ECS cluster. Try this answer and check if it works or not.