Need help?

Our experts have had an average response time of 13.52 minutes in October 2021 to fix urgent issues.

We will keep your servers stable, secure, and fast at all times for one fixed price.

High target latency on an AWS DMS – How to resolve

by | Aug 26, 2021

Wondering how to troubleshoot high target latency on an AWS DMS? We can help you.

As a part of our AWS Support Services, we often receive similar requests from our AWS customers.

Today, let’s see the steps followed by our Support Techs to help our customers.

 

How to troubleshoot high target latency on an AWS DMS?

You can use Amazon CloudWatch metrics to monitor your replication task’s statistics.

Specifically, you can monitor CDCLatencySource and CDCLatencyTarget to identify replication latency in the ongoing replication phase (CDC).

The CDCLatencySource metric is the latency between the source and the replication instance.

The CDCLatencyTarget metric is the latency between replication instance and target.

High CDCLatencySource means that the process of capturing changes from the source is delay.

And high CDCLatencyTarget means that the process of applying the change events to the target is delay.

If both CDCLatencySource and CDCLatencyTarget are high, investigate CDCLatencySource first, because target latency is always the same or greater than the source latency.

High CDCLatencyTarget is most likely the result of the delay in capturing the change events from the source.

If the CDCLatencySource isn’t high, but the CDCLatencyTarget is high, the latency could be caused by the following:

  • Firstly, there are no primary keys or indexes in the target
  • Secondly, there are resource bottlenecks in the target
  • There are resource bottlenecks in the replication instance
  • There is a network issue between the replication instance and the target

 

Today, let us see the steps followed by our Support Techs to troubleshoot it.

No primary keys or indexes in the target

By default, AWS DMS writes changes to the target by data manipulation language (DML) statements—such as INSERT, UPDATE, or DELETE—like any other application.

If the required indexes aren’t in place, then changes like UPDATEs and DELETEs can result in full table scans.

Full table scans can cause performance issues on the target. These scans then result in target latency.

Check your target database schemas, especially if you created the target schema manually.

Identify slow queries using target database mechanisms, like the slow query log for MySQL, pg_stat_activity for PostgreSQL, or a query plan.

If your target is Amazon Redshift, also check the distribution style for your table.

All distribution styles can cause target latency because they take longer to INSERT or UPDATE data into tables.

 

Resource bottlenecks in the target

If your target doesn’t have sufficient resources, then the target can’t accept changes at the rate that AWS DMS sends them.

This can cause resource bottlenecks on the target and target latency.

This can also happen if other processes are consuming resources in the target.

If the target is hosted on AWS, then check the resource statistics from the CloudWatch metrics.

 

Resource bottlenecks in the replication instance

Choose a replication instance that has enough resources to handle your migration—CPU, memory, network, or iOPS.

You can monitor your replication instance resources using CloudWatch metrics.

Network issue between replication instance and target

Network bandwidth and latency issues can also cause latency issues, especially when your target is an on-premises database, or if you use AWS DMS for cross-Region replication.

 

Best practices and troubleshooting

If your target is Amazon Relational Database Service (Amazon RDS) follow the best practices for Improving the Performance of an AWS DMS Migration.

Amazon RDS has an automated backup mechanism that starts within the backup window, and Amazon RDS backs up the moved data.

If a snapshot of the target DB instance was in the process of capture, AWS DMS can have issues applying changes to the target.

As a result, the target latency increases until the snapshot capture is completed.

If your target is Amazon Elastic Compute Cloud (Amazon EC2) or an on-premises database, check your target database’s backup mechanism.

Some task settings can cause changes to be written slowly to the target.

If you run ongoing replication from a source where the rate of change is high, consider using BatchApplyEnabled.

To set BatchApplyEnabled to True, run the modify-replication-task command using the AWS Command Line Interface (AWS CLI):

aws dms modify-replication-task --replication-task-arn arn:aws:dms:ap-northeast-1:123456789012:task:ABCDEFGHIJKLMNOPQRSTUVWXYZ --replication-task-settings "{\"TargetMetadata\":{\"BatchApplyEnabled\":true}}"

 

 

[Need help with more AWS queries? We’d be happy to assist]

Conclusion

To conclude, today we saw how our Support Engineers troubleshoot high target latency on an AWS DMS.

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

GET STARTED

var google_conversion_label = "owonCMyG5nEQ0aD71QM";

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF