Bobcares

How to configure Lookup and ForEach in Azure Data Factory

by | Jun 10, 2024

Configure Lookup activity and ForEach in Azure Data Factory in a couple of easy steps with help from our experts. Our DevOps Support team is here to help you with your questions and concerns.

How to configure Lookup and ForEach in Azure Data Factory

Azure Data Factory (ADF) offers several tools for handling dynamic data and performing iterative tasks. These include the Lookup activity and the ForEach loop. These features enable the creation of flexible and efficient data workflows.

How to configure Lookup and ForEach in Azure Data Factory

Let’s dive into their functionalities and see how they can work together to streamline our data processing tasks.

About Lookup Activity: Fetching Dynamic Data

The Lookup activity is designed to retrieve data from various sources, such as databases, files, or query execution results.

The output can range from a single value to an array of objects or records.

Here are a few use cases:

  • Dynamically gather configuration data based on pipeline execution.
  • Obtain a list of file names for subsequent processing.
  • Get a set of IDs or other data points for downstream activities.

For example:
We can use a Lookup activity to fetch a list of files from a storage account before copying them to another location or to retrieve a value from a database table to use as a parameter in subsequent activities. The retrieved data can be stored in a dataset or used directly within the pipeline.

ForEach Loop: Iterating Through Data

The ForEach loop iterates through a collection of data, typically an array, to perform repeated tasks on each item.

It executes a set of activities within the loop for each item in the collection.

Here are a few use cases:

    • Handle each record from a Lookup activity’s output.
    • Copy each file in a list to a different location.
    • Execute separate activities for each ID retrieved.

The ForEach activity supports iterating over an array, a dataset, or another activity’s output. Inside the loop, we can define a child pipeline that includes any combination of supported activities, dynamically passing the current item from the collection to this pipeline to perform specific operations.

How to create Dynamic and Iterative Workflows

  1. First, define the source and structure of the data we want to retrieve.
  2. Then, use dynamic content (`@activity(‘lookupActivity’).output.value`) to access the Lookup activity’s output in the ForEach loop settings.
  3. Now, it is time to configure ForEach Loop:
    • Set “Items” to reference the Lookup activity’s output.
    • Define activities to execute for each item within the loop.
    • Access Loop Items with the `@item()` function.

For example:
We can configure a Lookup activity to fetch a list of files from a storage account and then set up a ForEach loop to iterate over each file, performing actions like copying each file to a different location or processing its contents.

Benefits of Lookup and ForEach activities

  • The combination of Lookup and ForEach activities allows for flexible data processing based on the retrieved data.
  • Efficiently handles repetitive operations on multiple records, making data workflows more streamlined.
  • Breaks down complex pipelines into smaller, reusable activities, enhancing maintainability and scalability.

With the Lookup activity and ForEach loop in Azure Data Factory, we can create powerful, dynamic, and efficient data workflows tailored to our needs.

[Need assistance with a different issue? Our team is available 24/7.]

Conclusion

With the Lookup activity and ForEach loop in Azure Data Factory, we can create powerful, dynamic, and efficient data workflows tailored to our needs.

In brief, our Support Experts introduced us to Lookup activity and ForEach in Azure Data Factory and its benefits.

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Never again lose customers to poor
server speed! Let us help you.

Privacy Preference Center

Necessary

Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

PHPSESSID - Preserves user session state across page requests.

gdpr[consent_types] - Used to store user consents.

gdpr[allowed_cookies] - Used to store user allowed cookies.

PHPSESSID, gdpr[consent_types], gdpr[allowed_cookies]
PHPSESSID
WHMCSpKDlPzh2chML

Statistics

Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

_ga - Preserves user session state across page requests.

_gat - Used by Google Analytics to throttle request rate

_gid - Registers a unique ID that is used to generate statistical data on how you use the website.

smartlookCookie - Used to collect user device and location information of the site visitors to improve the websites User Experience.

_ga, _gat, _gid
_ga, _gat, _gid
smartlookCookie
_clck, _clsk, CLID, ANONCHK, MR, MUID, SM

Marketing

Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

IDE - Used by Google DoubleClick to register and report the website user's actions after viewing or clicking one of the advertiser's ads with the purpose of measuring the efficacy of an ad and to present targeted ads to the user.

test_cookie - Used to check if the user's browser supports cookies.

1P_JAR - Google cookie. These cookies are used to collect website statistics and track conversion rates.

NID - Registers a unique ID that identifies a returning user's device. The ID is used for serving ads that are most relevant to the user.

DV - Google ad personalisation

_reb2bgeo - The visitor's geographical location

_reb2bloaded - Whether or not the script loaded for the visitor

_reb2bref - The referring URL for the visit

_reb2bsessionID - The visitor's RB2B session ID

_reb2buid - The visitor's RB2B user ID

IDE, test_cookie, 1P_JAR, NID, DV, NID
IDE, test_cookie
1P_JAR, NID, DV
NID
hblid
_reb2bgeo, _reb2bloaded, _reb2bref, _reb2bsessionID, _reb2buid

Security

These are essential site cookies, used by the google reCAPTCHA. These cookies use an unique identifier to verify if a visitor is human or a bot.

SID, APISID, HSID, NID, PREF
SID, APISID, HSID, NID, PREF