Stuck with modifying the Spark configuration in an Amazon EMR notebook? Find out how the experienced Support Engineers at Bobcares go about it.

If you want to customize your configuration for a Spark job in an Amazon EMR notebook, you have come to the right place. Our Support Engineers are well versed with Server Management issues, big and small.

About Amazon EMR notebook & modifying the Spark configuration in an Amazon EMR notebook

Did you know the Amazon EMR notebook is actually a serverless Jupyter notebook? Furthermore, it uses Sparkmagic kernel as a client. It allows it to interactively work with Spark in the remote cluster via an Apache Livy server. Moreover, Spark configuration is configured using Sparkmagic commands. In addition, you need a custom configuration to do the following:

• To edit executor cores and executor memory for a Spark Job.
• To change Spark’s resource allocation.

Modifying the Spark configuration in an Amazon EMR notebook

Modify current Session

1. First, run %%configure on a Jupyter notebook cell to modify the job configuration. For instance:

   %%configure -f
   {"executorMemory":"4G"}

2. After that, pass the –conf option for more configurations. For instance, you can use a nested JSON as seen below:

   %%configure -f
   {"conf":{"spark.dynamicAllocation.enabled":"false"}}

Verify the configuration change was successful

1. First, run %%info on the client side to verify current session configurations. For instance, here is a sample output:

   Current session configs: {'executorMemory': '4G', 'conf': {'spark.dynamicAllocation.enabled': 'false'}, 'kind': 'pyspark'}

2. Next, verify /var/log/livy/livy-livy-server.out log on the EMR cluster. In case, the SparkSessions has started, you will get a log entry similar to this:

   20/06/24 10:11:22 INFO InteractiveSession$: Creating Interactive session 2: [owner: null, request: [kind: pyspark, proxyUser: None, executorMemory: 4G, conf: spark.dynamicAllocation.enabled -> false, heartbeatTimeoutInSecond: 0]]

[Looking for assistance with Server Management? Give us a call.]

Conclusion

In conclusion, we saw how to modify the Spark configuration in an Amazon EMR notebook without too much trouble. Without a doubt, the experts at Bobcares are proficient at handling different Server Management challenges.