Bobcares

wesupport

CLIENT AREACall Us 1-800-383-5193
Bobcares

wesupport

Call Us 1-800-383-5193
Bobcares

wesupport

Call Us 1-800-383-5193

Need help?

Our experts have had an average response time of 13.52 minutes in October 2021 to fix urgent issues.

We will keep your servers stable, secure, and fast at all times for one fixed price.

Container Instances for Amazon ECS Disconnected

by | Aug 20, 2021

Container Instances for Amazon ECS Disconnected? We can help you.

Generally, these change events are normal. However, if the container agent remains disconnected, then it can’t operate as part of the ECS cluster.

The agent is disconnected when agentConnected returns false.

Here, at Bobcares, we assist our customers with several AWS queries as part of our AWS Support Services.

Today, let us see how we can fix this issue.

 

Container Instances for Amazon ECS Disconnected

This can occur due to the following:

  • Networking issues prevent communication between the instance and Amazon ECS.
  • The container agent doesn’t have the required AWS IAM permissions to communicate with Amazon ECS endpoints.
  • Problems with the host or Docker service inside the container instance.

 

Resolution

Moving ahead, let us see how our Support Techs fix this issue.

These steps apply to Amazon ECS-optimized Amazon Linux 2 AMIs.

 

Verify that the Docker service is running on the container instance

1. To verify that the Docker service runs on the affected container instance, we run:

sudo systemctl status docker
docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2019-06-28 03:23:52 UTC; 1 day 12h ago
Docs: https://docs.docker.com
Process: 5519 ExecStartPre=/usr/libexec/docker/docker-setup-runtimes.sh (code=exited, status=0/SUCCESS)
Process: 5509 ExecStartPre=/bin/mkdir -p /run/docker (code=exited, status=0/SUCCESS)
Main PID: 5531 (dockerd)
Tasks: 60
Memory: 55.4M
CGroup: /system.slice/docker.service
├─5531 /usr/bin/dockerd --default-ulimit nofile=1024:4096
├─5570 docker-containerd --config /var/run/docker/containerd/containerd.toml
├─5782 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/...
├─6006 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/...
└─6284 docker-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/...

If the Docker service is inactive, then we restart the Docker service:

sudo systemctl restart docker

2. Eventually, to start the container agent, we run:

sudo systemctl start ecs

 

Verify that the container agent is running on the container instance

To verify that the container agent runs on the affected container instance, we run:

sudo systemctl status ecs
ecs.service - Amazon Elastic Container Service - container agent
Loaded: loaded (/usr/lib/systemd/system/ecs.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2019-06-29 15:45:57 UTC; 4min 5s ago
Docs: https://aws.amazon.com/documentation/ecs/
Process: 18896 ExecStopPost=/usr/libexec/amazon-ecs-init post-stop (code=exited, status=0/SUCCESS)
Process: 18818 ExecStop=/usr/libexec/amazon-ecs-init stop (code=exited, status=0/SUCCESS)
Process: 19422 ExecStartPre=/usr/libexec/amazon-ecs-init pre-start (code=exited, status=0/SUCCESS)
Main PID: 19455 (amazon-ecs-init)
Tasks: 7
Memory: 2.7M
CGroup: /system.slice/ecs.service
└─19455 /usr/libexec/amazon-ecs-init start

Suppose the output fails to show the active status of the service. Then we restart it:

sudo systemctl restart ecs

 

Review log files for the container agent and Docker

The container instance may still be disconnected. In that case, we review the log files on the container host for the container agent and Docker.

To output the log files for the container agent and Docker, we run:

sudo journalctl -u ecs
sudo journalctl -u docker

in addition, we run the Amazon ECS logs collector to collect log information from the container instance.

 

Verify if the IAM instance profile has the necessary permissions

In case the container agent is still disconnected, we verify the IAM instance profile associated with the container instance has the necessary IAM permissions.

1. To do so, we connect to the instance via SSH.

2. To view the instance metadata on the instance profile associated with the instance, we run:

curl http://xxx.xxx.xxx.xxx/latest/meta-data/iam/info
{
"Code" : "Success",
"LastUpdated" : "2019-06-29T15:47:03Z",
"InstanceProfileArn" : "arn:aws:iam::1122334455:instance-profile/ecsInstanceRole",
"InstanceProfileId" : "AIPAJ5WF3LZVY7PLUHV72"
}

3. Then we ensure that the IAM role contains the correct permissions for the container instances.

4. To verify specific credential errors with the container agent and to check the container agent log for a list of ECS logs, we run:

cat /var/log/ecs/ecs-agent.log.YYYY-MM-DD-**

We will receive the following error if the container agent doesn’t have the necessary credentials:

2019-06-29T16:10:09Z [ERROR] Unable to register as a container instance with ECS: AccessDeniedException: User: arn:aws:sts::1122334455:assumed-role/ecsInstanceRole/i-0052b2e858b1891ef is not authorized to perform: ecs:RegisterContainerInstance on resource: arn:aws:ecs:us-east-1:1122334455:cluster/exampleCluster
status code: 400, request id: 0b73e260-5088-4688-a425-6f35f1ef440f

2019-06-29T16:10:09Z [ERROR] Error re-registering: AccessDeniedException: User: arn:aws:sts::1122334455:assumed-role/ecsInstanceRole/i-0052b2e858b1891ef is not authorized to perform: ecs:RegisterContainerInstance on resource: arn:aws:ecs:us-east-1:1122334455:cluster/exampleCluster
status code: 400, request id: 0b73e260-5088-4688-a425-6f35f1ef440f

[Need help with the fix? We’d be happy to assist]

 

Conclusion

In short, we saw how our Support Techs fix the Amazon ECS error for our customers.

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

GET STARTED

var google_conversion_label = "owonCMyG5nEQ0aD71QM";

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Reviews