Broadcast state in Apache Flink is just a key-value map that is read-only and shared to all concurrent instances (subtasks) of a streaming application. At Bobcares, with our Server Management Service, we can handle your Apache issues.

Broadcast State in Apache Flink

Spread across all parallel subtasks of a streaming app, the broadcast state in Flink is effectively a read-only key-value map. This is usually a small enough state to be stored in memory and copied across all instances.

Working

To merge and process two streams of events simultaneously in a certain manner, we can use the Broadcast State. All of the operator’s parallel instances get the events of the first stream, which keeps them in their original state.

While the events of the other stream are processed alongside the events of the broadcasted stream, they are not broadcast. Instead, they are transmitted to distinct instances of the same operator. Apps need to link a high-throughput and low-throughput stream. Else, requires dynamic processing logic updates for the new broadcast state.

When we have a big stream of events (the non-broadcast stream) and a smaller stream of configuration data (the broadcast stream), which need to be linked with the mainstream using a shared key, we can use a broadcast state.

Flink effectively broadcasts the state from the broadcast stream to all parallel instances of the app when processing the mainstream. Every instance keeps a local memory copy of the broadcast state. Flink searches the local copy of the broadcast state for the matching values when events from the mainstream reach each instance. As a result, joins can be completed quickly and effectively without requiring network contact.

After the broadcast, the broadcast state cannot be changed; it is immutable. A fresh copy of the broadcast state is created. Then, sent out to all instances whenever the data that has been broadcast has to be modified. Flink offers methods for managing state changes, like rebroadcasting the new state on a regular basis or initiating a state refresh under specific conditions.

Memory Management

It is crucial to make sure that the size of the broadcast state stays reasonable and does not beyond the memory resources that are available, as the broadcast state is repeated among all instances. Broadcast state memory allocation can be configured using the tools provided by Flink.

[Need to know more? Get in touch with us if you have any further inquiries.]

Conclusion

To sum up, our Tech team went over the details of the Broadcast state in Apache Flink in this article.