Need help?

Our experts have had an average response time of 11.7 minutes in August 2021 to fix urgent issues.

We will keep your servers stable, secure, and fast at all times for one fixed price.

AWS DMS : data not migrated from S3 endpoint

by | Jul 30, 2021

Stuck with AWS DMS : data not migrated from S3 endpoint error? We can help you.

Here, at Bobcares, we assist our customers with several AWS queries as part of our AWS Support Services.

Today, let us see how our Support Techs resolve it for our customers.

 

What causes AWS DMS : data not migrated from S3 endpoint?

Some of the reasons why an AWS DMS task is successful but no data is migrate are as following:

  1. Firstly, the task status is Load complete, replication ongoing, but no data is load on the target.
  2. The task status is Load complete, replication ongoing, but no table is in the Table statistics section.
  3. The task status is Running and the table is created in the target endpoint, but no data is loaded. Also, you receive a No response body error in the replication log.

 

How to resolve it?

Today, let us see the steps followed by our Support Techs to resolve it.

The task status is Load complete, replication ongoing, but no data is loaded on the target

Firstly, confirm that the Amazon S3 path defined for the source endpoint is correct.

From the replication log, review the log entries.

Identify entries that indicate that AWS DMS can’t find the data files in the Amazon S3 path that is defined for the source endpoint.

Below is example replication log entry:

[SOURCE_UNLOAD ]I: Unload finished for table 'dms_schema'.'dms_table' (Id = 1). 0 rows sent. (streamcomponent.c:3396)
[TARGET_LOAD ]I: Load finished for table 'dms_schema'.'dms_table' (Id = 1). 0 rows received. 0 rows skipped. Volume transferred 0. (streamcomponent.c:3667)

In Amazon S3, the data file for the full load phase (data.csv) and the data file for on-going changes (change_data.csv) are stored like this:

S3-bucket/dms-folder/sub-folder/dms_schema/dms_table/data.csv
S3-bucket/dms-folder/sub-folder/dms-cdc-path/dms-cdc-sub-path/change_data.csv

For the Amazon S3 source endpoint to find the data files, there are three important fields:

  • Bucket folder
  • Change data capture (CDC) path
  • Table structure

In the example file paths listed previously, dms-folder/sub-folder is the Bucket folder.

The CDC path that you enter when creating the Amazon S3 source endpoint is dms-cdc-path/dms-cdc-sub-path.

The following example Table structure uses the same example file path listed previously:

{
"TableCount": 1,
"Tables": [
{
"TableColumns": […],
"TableColumnsTotal": "1",
"TableName": "dms_table",
"TableOwner": "dms_schema",
"TablePath": "dms_schema/dms_table/"
}
]
}

Please don’t include the bucket folder path (dms-folder/sub-folder) in the TablePath of the table structure.

When specifying your Endpoint configuration, consider the following:

  • The bucket folder is optional. If a bucket folder is specified, then the CDC path and table path must be in the same folder in Amazon S3.
  • The Bucket folder field of the Amazon S3 source endpoint can be any folder directory between the S3 bucket name and the schema name of the table structure. In the example above, it’s dms-schema.
  • Bucket folders or CDC paths can be individual folders or they can include sub folders, such as dms-folder or dms-folder/sub-folder.

 

If your DMS task setting uses Amazon S3 as the source endpoint, then you must include the schema and table in the table mapping.

This is required to successfully migrate data to the target.

If you use Drop tables on target as the table preparation mode for the task, then DMS creates the target table dms_schema.dms_table.

 

The task status is Load complete, replication ongoing, but no table is in the Table statistics section

You might see that no table was create in the target endpoint when you use Drop tables on target.

This means that the issue might cause by the table structure that is specified for the Amazon S3 source endpoint.

Confirm that the Amazon S3 path for the source endpoint is correct.

Then, verify that your data type is supported by the Amazon S3 endpoint.

After confirming that the Amazon S3 path is correct and it supports your data type, check the filter that is define by the table mapping of your DMS task.

Then, check to see if the filter is the cause of the missing tables.

Review the table that is require within the task table mapping and check that the table is define in the table structure of the Amazon S3 source endpoint.

 

The task status is Running and the table is created in the target end, but no data is loaded

If AWS DMS can’t get the content from the Amazon S3 path, you can find errors in the replication log.

See the following examples:

[SOURCE_CAPTURE ]E: No response body. Response code: 403 [1001730] (transfer_client.cpp:589)

[SOURCE_CAPTURE ]E: failed to download file </dms-folder/sub-folder/dms_schema/dms_table/data.csv> from bucket <dms-test> as </rdsdbdata/data/tasks/NKMBA237MEB4UFSRDF5ZAF3EZQ/bucketFolder/dms-folder/sub-folder/dms_schema/dms_table/data.csv>,
status = 4 (FAILED) [1001730] (transfer_client.cpp:592)

This error occurs when the AWS Identity and Access Management (IAM) role for the Amazon S3 source endpoint doesn’t have the correct permissions: s3:GetObject.

To resolve this error, confirm that the data file exists in the Amazon S3 path that is in the error message.

Then, confirm that the IAM user has permissions for s3:GetObject.

 

[Need help with the process? We’d be happy to assist you]

Conclusion

To conclude, here we saw how our Support Techs resolve AWS DMS : data not migrated from S3 endpoint error.

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

GET STARTED

var google_conversion_label = "owonCMyG5nEQ0aD71QM";

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF