It is possible to delete s3 data in the AWS data pipeline using the console or using the AWS CLI.

Here at Bobcares, we have seen several such AWS related queries as part of our AWS Support Services for AWS users, and online service providers.

Today we’ll take a look at how to delete s3 data in the AWS data pipeline.

Know more about AWS Data Pipeline

AWS Data Pipeline is a web service. It is mainly designed to make it easier for users to integrate data spread across multiple AWS services and analyze it from a single location.

Here are some of its features.

Simple and cost-efficient

The drag-and-drop feature makes the creation of pipelines easier on the console. Also, the visual pipeline creator provides a library of pipeline templates.

Reliable

Its infrastructure is designed for fault-tolerant execution activities. In case if any failures occur in the activity logic or data sources, then AWS Data Pipeline automatically retries the activity. However, if the failure continues, then it will send a failure notification.

Flexible

AWS Data Pipeline provides many features that include scheduling, tracking, error handling, etc. We can configure it such a way that it can take actions like run Amazon EMR jobs, execute SQL queries directly against databases, execute custom applications running on Amazon EC2, etc.

Scalable

AWS Data Pipeline makes it easy to dispatch work to one machine or many, in serial or parallel. With its flexible design, processing a million files is as easy as processing a single file.

How to Set Up Data Pipeline

To create the pipeline, our Support Engineers follow the steps below:

1. First, we sign-in to the AWS account.
2. Next, we use this link to Open AWS Data Pipeline console − https://console.aws.amazon.com/datapipeline/
3. Here we select the region in the navigation bar.
4. Then we click the Create New Pipeline button.
5. After that, we fill in the required details in the respective fields.

• In the Source field, we choose ‘Build using a template’ and then select this template − Getting Started using ShellCommandActivity.
  
• The Parameters section opens only when we select the template. We leave the S3 input folder and Shell command to run with their default values. After that, we click the folder icon next to the S3 output folder and select the buckets.
• In Schedule, we leave the values as default.
• In Pipeline Configuration, we leave the logging as enabled. Then we click the folder icon under the S3 location for logs and select the buckets.
• After that, in Security/Access, we leave IAM roles values as default.
• Finally, we click the Activate button.

How to Delete S3 data in AWS Data Pipeline

When we no longer need a pipeline, such as a pipeline that we created during application testing, we should delete it to remove it from active use. Deleting a pipeline will put it into a deleting state.

When the pipeline is in the deleted state, its run history and pipeline definition are gone. Therefore, we will no longer be able to perform operations on the pipeline, including describing it. We can’t restore a pipeline once we delete it. Moreover, deleting a pipeline will also delete all its associated objects.

To delete an AWS data pipeline using the console

• First, on the List Pipelines page, we select the pipeline.
• Next, we click the Actions button and then choose Delete.
• Finally, a confirmation prompt window opens. Here we click Delete.

To delete an AWS data pipeline using the AWS CLI

In order to delete a pipeline, we use the delete-pipeline command. We run the below command to delete a specified pipeline.

aws datapipeline delete-pipeline –pipeline-id df-00627471SOVYZEXAMPLE

[Need any further assistance in deleting data in AWS data pipeline? – We are here to help you.]

Conclusion

In short, we can delete s3 data in the AWS data pipeline using various ways. Today, we saw how our Support Engineers delete it.