Stuck with AWS EMR managed scaling? Read on to find out what our expert Support Engineers have to say.

Learn how to use AWS CLI to configure managed scaling from the skilled Support Team at Bobcares.

Dealing with AWS EMR managed scaling

Let’s take a look at look at how to configure managed scaling for EMR with CLI commands. You can opt to use a shorthand syntax with the JSON configuration inline included in the relevant commands or reference the file with the configuration JSON.

Enabling the managed scaling during cluster launch

This example demonstrates how to enable managed scaling during cluster launch,

aws emr create-cluster \
 --service-role EMR_DefaultRole \
 --release-label emr-5.33.0 \
 --name EMR_Managed_Scaling_Enabled_Cluster \
 --applications Name=Spark Name=Hbase \
 --ec2-attributes KeyName=keyName,InstanceProfile=EMR_EC2_DefaultRole \
 --instance-groups InstanceType=m4.xlarge,InstanceGroupType=MASTER,InstanceCount=1 InstanceType=m4.xlarge,InstanceGroupType=CORE,InstanceCount=2 \
 --region us-east-1 \
 --managed-scaling-policy ComputeLimits='{MinimumCapacityUnits=2,MaximumCapacityUnits=4,UnitType=Instances}'

Furthermore, you can specify the managed policy configuration via the –managed-scaling-policy option while using create-cluster.

Applying managed scaling policy to an existing cluster

The following example applies managed scaling to an existing cluster:

aws emr put-managed-scaling-policy  
--cluster-id j-123456  
--managed-scaling-policy ComputeLimits='{MinimumCapacityUnits=1,
MaximumCapacityUnits=10,  MaximumOnDemandCapacityUnits=10, UnitType=Instances}'

Moreover, this can be done also via the AWS EMR put-managed-scaling-policy command. For instance, this example makes a reference to managedscaleconfig.json, a JSON file.

aws emr put-managed-scaling-policy --cluster-id j-123456 --managed-scaling-policy file://./managedscaleconfig.json

Here is a look at the contents of the managedscaleconfig.json file. It defines the managed scaling policy.

{
    "ComputeLimits": {
        "UnitType": "Instances",
        "MinimumCapacityUnits": 1,
        "MaximumCapacityUnits": 10,
        "MaximumOnDemandCapacityUnits": 10
    }
}

Retrieve an EMR managed scaling policy configuration

In order to retrieve the policy configuration, use the GetManagedScalingPolicy command. For instance, it is used here to retrieve the configuration of the cluster with the cluster id j-123455.

aws emr get-managed-scaling-policy --cluster-id j-123455

This results in the following output:

{
   "ManagedScalingPolicy": { 
      "ComputeLimits": { 
         "MinimumCapacityUnits": 1,
         "MaximumOnDemandCapacityUnits": 10,
         "MaximumCapacityUnits": 10,
         "UnitType": "Instances"
      }
   }
}

Removing EMR managed scaling policy

The RemoveManagedScalingPolicy is used to remove the managed scaling policy. For instance, this command removes the policy of the cluster with cluster id j-123455:

aws emr remove-managed-scaling-policy --cluster-id j-123455

[Need help with Server Management? We are here to help.]

Conclusion

In brief, we learned how to use AWS CLI to configure AWS EMR managed scaling. With the guidance of the skilled Support Engineers at Bobcares, Server management is now a breeze.