Need help?

Our experts have had an average response time of 13.52 minutes in October 2021 to fix urgent issues.

We will keep your servers stable, secure, and fast at all times for one fixed price.

ECR issues with Amazon EKS: How to troubleshoot

by | Sep 4, 2021

Wondering how to troubleshoot ECR issues with Amazon EKS? We can help you with this!

As a part of our AWS Support Services, we often receive similar requests from our AWS customers.

Today, let’s see the steps followed by our Support Techs to help our customers to fix ECR issues with Amazon EKS.

 

ECR issues with Amazon EKS

 
Because of one of the following reasons we can’t pull images from Amazon ECR:

  1. Unable to communicate with Amazon ECR endpoints.

2. If we don’t have the proper permissions in the instance profileto pull images from an Amazon ECR repository.

We can resolve the above issues by following any of the options below:

  1. Troubleshooting the communication between worker nodes and Amazon ECR endpoints.

2. Updating the instance profile of worker nodes and confirming that repository policies are correct.
 

Troubleshooting the communication between worker nodes and Amazon ECR endpoints

 
We could receive the following error message if the worker nodes can’t communicate with the Amazon ECR endpoints:

Failed to pull image "ACCOUNT.dkr.ecr.REGION.amazonaws.com/imagename:tag": rpc error: code = Unknown desc =
Error response from daemon: Get https://ACCOUNT.dkr.ecr.REGION.amazonaws.com/v2/: net/http:
request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

We should confirm the following to resolve this error:

  • Subnet for the worker node has a route to the internet.So we should check the route table associated with the subnet.
  • The security group related to the worker node allows outbound internet traffic.
  • The incoming and outgoing rule for the network ACLs allows access to the internet.

 

Updating the instance profile of the worker nodes

 
We could receive the following error message from your Amazon EKS pod if the worker node’s instance profile doesn’t have permission to pull images from Amazon ECR:

Warning Failed 14s (x2 over 28s) kubelet, ip-000-000-000-000.us-west-2.compute.internalFailed to pull image "ACCOUNT.dkr.ecr.REGION.amazonaws.com/imagename:tag": rpc error: code = Unknown desc = Error response from daemon: Get https://ACCOUNT.dkr.ecr.REGION.amazonaws.com/v2/imagename/manifests/tag: no basic auth credentials
Warning Failed 14s (x2 over 28s) kubelet, ip-000-000-000-000.us-west-2.compute.internal Error: ErrImagePull
Normal BackOff 2s (x2 over 28s) kubelet, ip-000-000-000-000.us-west-2.compute.internal Back-off pulling image "ACCOUNT.dkr.ecr.REGION.amazonaws.com/imagename:tag"
Warning Failed 2s (x2 over 28s) kubelet, ip-000-000-000-000.us-west-2.compute.internal Error: ImagePullBackOff

We need to confirm that the worker nodes use the AmazonEC2ContainerRegistryReadOnly AWS Identity and IAM policy to fix this error. The other option is by using the following IAM permissions to update theAmazon EC2 instance profile of worker nodes:

"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:GetRepositoryPolicy",
"ecr:DescribeRepositories",
"ecr:ListImages",
"ecr:DescribeImages",
"ecr:BatchGetImage",
"ecr:GetLifecyclePolicy",
"ecr:GetLifecyclePolicyPreview",
"ecr:ListTagsForResource",
"ecr:DescribeImageScanFindings"

It is aways better to use the AmazonEC2ContainerRegistryReadOnly policy instead of creating a duplicate policy.This helps our worker nodes the permissions to access Amazon ECR and pull images through the kubelet. The kubelet helps in fetching and periodically refreshing Amazon ECR credentials.
 

To confirm the repository policies are correct

 
The repository policies are a subset of IAM policies. It control access to individual Amazon ECR repositories.

The IAM policies are used to apply permissions for the entire Amazon ECR service. It can also control access to specific resources.

1. Firstly log in to the AWS Mnagement console and open the Amazon ECR console for the account.

2. Select Repositories, and then select the repository.

3. Select Permissions, and then check if the repository has the correct permissions.

For example:

{
"Version": "2009-10-16",
"Statement": [
{
"Sid": "ECR Repository Policy",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::098765432112:user/MyUsername"
},
"Action": [
"ecr:DescribeImages",
"ecr:DescribeRepositories"
]
}
]
}

[Need help with more AWS queries? We’d be happy to assist]
 

Conclusion

 
To conclude, today we discussed the steps followed by our Support Engineers to help our customers to resolve ECR issues with Amazon EKS.

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Reviews

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF