Autoscale based on Incomming Requests

There is no AWS supported way of implementing autoscaling based on incomming web traffic. Hence the post.

There are 2 scripts –

1. To read the access log on the host machine (placed on all host machines)
2. To collect access log values, take average and upload it to cloudwatch so that the necessary alarms are triggered.(placed on the monitoring server)

@Monitoring Server :

Installing Java
#apt-get install  openjdk-7-jre-headless
#export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

Setting cloudwatch api
#export  AWS_CLOUDWATCH_HOME=/root/cloudwatch
#export PATH=$PATH:${AWS_CLOUDWATCH_HOME}/bin

NOTE: if you are an IAM user make sure you attach Cloudwatch-FullAccess Policy in the IAM User ABC’s -> User Policies.

NOTE – update in /etc/profile and source it.

Check that your setup works properly, run the following command:
#mon-cmd –help

Creating New Metric!

Simply store new values and the new metric will be created automatically!!
#mon-put-data -m “FreeMemory” –namespace UnniCustomMetric –dimensions “instance=i-16e02478,servertype=apache” –value 100 -u Bytes

-m          : name of the metric to be gathered.Required.
–namespace : The namespace the given Metric is valid in. Required.
-d          : Dimension – different info about a particular instance to help us uniquely identify to get its metric data (or a group of instancesls)
-v          : The value of the metric datum being put in.
-u          : The unit that the metric is being reported in.  Valid unit values are Seconds, Bytes, Bits, Percent, Count, Bytes/Second, Bits/Second, CountSecond,None.
-s          : The statistics to be provided for the given metric. Valid key values are: SampleCount, Sum, Maximum, and Minimum.
Note
When you create a new metric using the mon-put-data command, it can take up to two minutes before you can retrieve statistics on the new metric using the mon-get-stats command. However, it can take up to fifteen minutes before the new metric appears in the list of metrics retrieved using the mon-list-metrics command.

Once the metric is created, we need to continuously feed data to this metric.

LIST OF METRICS FOR EC2 : http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/CW_Support_For_AWS.html#ec2-metricscollected

AUTOSCALE SETUP
There are 3 steps
1.Create Autoscale Launch Configuration
2.Create Autoscale Group
3.Create Autoscale Policy

Creating Launch Configuration:
#as-create-launch-config unni-autoscale –image-id ami-5766173e –region us-east-1 –instance-type t1.micro –key unni –group unni

Verify Lauch Configuration:
#as-describe-launch-configs –region us-east-1

Creating Autoscaling Group
#as-create-auto-scaling-group unni-autoscale-test –launch-configuration unni-autoscale –region us-east-1 –availability-zones us-east-1a –min-size 1 –max-size 3 –load-balancers unni-autoscale

Verify Autoscaling Group
#as-describe-auto-scaling-groups unni-autoscale-test –headers –region us-east-1

Autoscale Policy has 3 Options
1.Maintain a fixed number of instances
2.Scale UP and Down based on the demand
3.Scale UP and Down based on schedule

Auto Scaling will use the Cloudwatch alarms, to observe the defined metrics (CPU Utilization, Network I/O, Memory Utilization, etc) over a period of time. The Cloudwatch alarms send a message to Auto Scaling when one of the alarms is in the Alert state. The user will have a corresponding policy with Auto Scaling, which instructs on what action to take (add / remove resources) when the Alert state is reached.

Creating Autoscaling Policy

scaleup – #as-put-scaling-policy scale-up-unnias –auto-scaling-group unni-autoscale-test –adjustment=1 –type=ChangeInCapacity –cooldown 60
scalein – #as-put-scaling-policy scale-in-unnias –auto-scaling-group unni-autoscale-test –adjustment=-1 –type=ChangeInCapacity –cooldown 10

Configure Cloudwatch Alarm Metrics

General Metrics for which you want to set the cloud watch alerts. For each instance there are seven metrics available:
a)     CPU Utilization
b)     DiskReadBytes
c)     DiskReadOps
d)     DiskWriteBytes
e)     DiskWriteOps
f)     NetworkIn
g)     NetworkOut

Findout AS Policy – #as-describe-policies

Configure the Cloudwatch Alarm with the following command :

#mon-put-metric-alarm –alarm-name scaleup-unni –alarm-description “testt-alarm” –metric-name RequestCount –namespace UnniCustomMetric –statistic Maximum –period 60 –threshold 27000 –comparison-operator GreaterThanThreshold –dimensions “AutoScalingGroupName=unni-autoscale-test” –evaluation-periods 2 –alarm-actions arn:aws:autoscaling:us-east-1:528830004426:scalingPolicy:05d4b447-ab6c-405a-adfe-4f99073c5ed5:autoScalingGroupName/unni-autoscale-test:policyName/scale-up-unnias –region us-east-1 –unit Bytes

At this point, login to AWS Cloudwatch and modify the changes mentioned in the above command to fit the bill.

CHECKING MY AUTOSCALE SETUP OVERALL
#as-describe-auto-scaling-groups unni-autoscale-test –headers –region us-east-1
#as-describe-policies –headers
#as-describe-launch-configs

Monitor Server Script

Host Machine Script

#!/bin/bash
hit=`cat /var/log/apache2/access.log | wc -l`
echo $hit
echo 0 >  /var/log/apache2/access.log

Cleaning UP
Delete Policy
Terminate Instance
Delete Autoscaling Group
Delete Autoscale Launch Configuration

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s