Configure Lookup activity and ForEach in Azure Data Factory in a couple of easy steps with help from our experts. Our DevOps Support team is here to help you with your questions and concerns.

How to configure Lookup and ForEach in Azure Data Factory

Azure Data Factory (ADF) offers several tools for handling dynamic data and performing iterative tasks. These include the Lookup activity and the ForEach loop. These features enable the creation of flexible and efficient data workflows.

Let’s dive into their functionalities and see how they can work together to streamline our data processing tasks.

About Lookup Activity: Fetching Dynamic Data

The Lookup activity is designed to retrieve data from various sources, such as databases, files, or query execution results.

The output can range from a single value to an array of objects or records.

Here are a few use cases:

• Dynamically gather configuration data based on pipeline execution.
• Obtain a list of file names for subsequent processing.
• Get a set of IDs or other data points for downstream activities.

For example:
We can use a Lookup activity to fetch a list of files from a storage account before copying them to another location or to retrieve a value from a database table to use as a parameter in subsequent activities. The retrieved data can be stored in a dataset or used directly within the pipeline.

ForEach Loop: Iterating Through Data

The ForEach loop iterates through a collection of data, typically an array, to perform repeated tasks on each item.

It executes a set of activities within the loop for each item in the collection.

Here are a few use cases:

• • Handle each record from a Lookup activity’s output.
  • Copy each file in a list to a different location.
  • Execute separate activities for each ID retrieved.

The ForEach activity supports iterating over an array, a dataset, or another activity’s output. Inside the loop, we can define a child pipeline that includes any combination of supported activities, dynamically passing the current item from the collection to this pipeline to perform specific operations.

How to create Dynamic and Iterative Workflows

1. First, define the source and structure of the data we want to retrieve.
2. Then, use dynamic content (`@activity(‘lookupActivity’).output.value`) to access the Lookup activity’s output in the ForEach loop settings.
3. Now, it is time to configure ForEach Loop:
   • Set “Items” to reference the Lookup activity’s output.
   • Define activities to execute for each item within the loop.
   • Access Loop Items with the `@item()` function.

For example:
We can configure a Lookup activity to fetch a list of files from a storage account and then set up a ForEach loop to iterate over each file, performing actions like copying each file to a different location or processing its contents.

Benefits of Lookup and ForEach activities

• The combination of Lookup and ForEach activities allows for flexible data processing based on the retrieved data.
• Efficiently handles repetitive operations on multiple records, making data workflows more streamlined.
• Breaks down complex pipelines into smaller, reusable activities, enhancing maintainability and scalability.

With the Lookup activity and ForEach loop in Azure Data Factory, we can create powerful, dynamic, and efficient data workflows tailored to our needs.

[Need assistance with a different issue? Our team is available 24/7.]

Conclusion

With the Lookup activity and ForEach loop in Azure Data Factory, we can create powerful, dynamic, and efficient data workflows tailored to our needs.

In brief, our Support Experts introduced us to Lookup activity and ForEach in Azure Data Factory and its benefits.