Need help?

Our experts have had an average response time of 11.7 minutes in August 2021 to fix urgent issues.

We will keep your servers stable, secure, and fast at all times for one fixed price.

How To Copy Data To and From EFS on EC2 Instance

by | Sep 6, 2021

Are you finding it difficult to copy data to and from EFS on EC2 instance? Fortunately, there are a few easy fixes for that. Listen to what our experienced Support Techs have to say about the matter.

Recently one of our long-time customers faced the same dilemma while attempting to copy and delete a huge number of files. They were attempting to run these jobs in parallel on an Amazon EFS on their Amazon EC2 instance. Our amazing AWS Support Techs stepped in and resolved this issue without further delay.

Tips to Copy Data To and From EFS on EC2 Instance

You can copy or delete a large number of files parallelly on an EFS on Amazon EC2 instance with the right tools. Let’s take a look at some of the suggestions from our support techs:

  • GNU Parallel
  • msrsync
  • fpsync

GNU parallel

  1. Initially, install GNU parallel

Run the following code for RHEL 6 and Amazon Linux:

$ sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
$ sudo yum install parallel nload -y

Run the following code For RHEL 7:

$ sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
$ sudo yum install parallel nload -y

Run the following code for Amazon Linux 2:

$ sudo amazon-linux-extras install epel
$ sudo yum install nload sysstat parallel -y

Run the following code for Ubuntu:

$ sudo apt-get install parallel
  1. Then use rsync to copy the required files to Amazon EFS.

We recommend using either of the following two commands

$ sudo time find -L /src -type f | parallel rsync -avR {} /dst

or

$ sudo time find /src -type f | parallel -j 32 cp {} /dst
  1. Next, monitor network traffic and bandwidth with the nload console application
$ sudo nload -u M

msrsync: Copy Data To & From EFS on EC2 Instance

Did you know that msrsync is actually a Python wrapper that runs several rsync processes in parallel?

  1. First, install msrsync wiht the following command
$ sudo curl -s https://raw.githubusercontent.com/jbd/msrsync/master/msrsync -o /usr/local/bin/msrsync && sudo chmod +x /usr/local/bin/msrsync
  1. Specifiy the number of rsync processes that you require to run on parallel with the -p option. Replace Z with the total number of sync processes. The -P option indicates the progress of each job.
$ sudo time /usr/local/bin/msrsync -P -p Z --stats --rsync "-artuv" /src/ /dst/

fpsync : Copy Data To & From EFS on EC2 Instance

Our team recommends using the fpsync tool to synchronize directories in parallel using rsync and fpart. You can execute multiple rsync processes locally or also launch rsync transfers on multiple nodes through SSH.

  1. First, enable EPEL repository. After that, install the fpart package.

Run the following command for Amazon Linux and RHEL 6:

$ sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
$ sudo yum install fpart -y

Run the following command for RHEL 7:

$ sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
$ sudo yum install fpart -y

Run the following command for Amazon Linux 2:

$ sudo amazon-linux-extras install epel
$ sudo yum install fpart -y

Run the following command for Ubuntu:

$ sudo apt-get install fpart

Remember that fpsync is part of the fpart package in Ubuntu.

  1. Then Use fpsync and synchronize the /src and /dst directories. Remember to replace Z with the exact number of rsync processes that you plan on running in parallel.
$ sudo fpsync -n Z /src /dst

[Need further assistance? We are happy to help you out!]

Conclusion

In short, our Support Techs were able to copy data to and from EFS on EC2 instance for our client.

PREVENT YOUR SERVER FROM CRASHING!

Never again lose customers to poor server speed! Let us help you.

Our server experts will monitor & maintain your server 24/7 so that it remains lightning fast and secure.

GET STARTED

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF