Amazon Elastic MapReduce

Today, Amazon annouced the availability of it’s newest web servrice – Elastic MapReduce.

Amazon Elastic MapReduce is a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3).

Amazon Elastic MapReduce automatically spins up a Hadoop implementation of the MapReduce framework on Amazon EC2 instances, sub-dividing the data in a job flow into smaller chunks so that they can be processed (the “map” function) in parallel, and eventually recombining the processed data into the final solution (the “reduce” function). Amazon S3 serves as the source for the data being analyzed, and as the output destination for the end results.

Amazon Elastic MapReduce applications can be authored in Java, Ruby, Perl, Python, PHP, R, or C++.  Elastic MapReduce applications only run on Linux/Unix instances running in the US region on EC2. Both reserved and on-demand instances are supported. If you have any on-demand instances, they will be used first.

Amazon Elastic MapReduce pricing is in addition to standard EC2 rates.

Current Amazon MapReduce Pricing

Amazon EC2 and Amazon S3 charges are billed separately. Pricing for Amazon Elastic MapReduce is per instance-hour consumed for each instance type, from the time job flow began processing until it is terminated. Each partial instance-hour consumed will be billed as a full hour.

No related posts.

EC2 uses Xen virtualization for its instances (servers). EC2 instances come in 3 sizes Small, Large, and Extra large. Amazon sizes instances on what’s called “EC2 Compute Units”. One EC2 Compute Unit (ECU) provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.

EC2 Standard Instances
Most applications are well suited for these types of instances:

• Small Instance: 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of instance storage, 32-bit platform
• Large Instance: 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of instance storage, 64-bit platform
• Extra Large Instance: 15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform

EC2 High-CPU Instances
This family of instances has proportionally more CPU resources than memory and is better suited for computational-intensive applications.

• High-CPU Medium Instance: 1.7 GB of memory, 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each), 350 GB of instance storage, 32-bit platform
• High-CPU Extra Large Instance: 7 GB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform

For most blogs, company websites and low-traffic ecommerce sites, a Small Instance will be fine. You can always migrate up to a larger instance.

Getting Started
For this article we are going to launch a 32-bit Ubuntu 8.04 LTS Hardy AMI (Amazon Machine Instance) from Alestic. Amazon provides a few AMI’s designed to get you up and running quickly. They also have over 1,700 Community AMI’s. You can select from Fedora, Ubuntu, Windows, CentOS and other flavors of Linux.

Before you can create your first instance, you will have to Sign Up for Amazon EC2.

Amazon EC2 Signup

After signing up, visit the AWS Management Console page. Click on the Go to Amazon EC2 Console button.

Launch EC2 Management Console

Before you create and launch your EC2 instance. You should first create a Key Pair. You will use this key pair to login to your instance after it’s been created.
To create your Key Pair, click Key Pairs in the left Navigation menu. Next, click the Create Key Pair button in the Key Pairs window. Enter a descriptive name for your new Key Pair in the Create Key Pair dialog window.

Create Amazon EC2 Key Pair

Next you will be prompted to download your newly created Key Pair. Download your newly created Key Pair; you will need this to login to your new instance later.

Save EC2 Key Pair

NOTE: AMAZON DOES NOT STORE YOUR PRIVATE KEY PAIR. IF YOU DO NOT DOWNLOAD YOUR KEY FILE AT THIS POINT YOU WILL NOT BE ABLE TO LOGIN TO ANY INSTANCES THAT WERE CREATED USING THIS PRIVATE KEY PAIR.

Now let’s go ahead and find our Amazon Ubuntu AMI and fire it up. Click the Dashboard link in the left hand navigation menu and then click the Launch Instances button.

Launch Amazon EC2 Instance

Select the Community AMI’s tab. Enter ami-71fd1a18 in the search field. This is the AMI id of the Alestic 32-bit Ubuntu Hardy server. Click the Select button to continue.

Alestic Ubuntu Hardy AMI

Now we are going to set some options for our Ubuntu AMI. First type in the number of instances you want to create, we are only going to create 1 for this article.
Next, select the Small instance type. In the Key Pair Name drop-down, select the new Key Pair we just created. In the Security Groups listbox, select default. We will skip the Advanced Options for now.

Amazon EC2 AMI Options

Click Launch to start you new Ubuntu Amazon instance!

Amazon will now attempt to start you instance. This may take anywhere from 30 seconds to 5 minutes. To check on the status of your instance click the Instance link in the Navigation window.

Amazon EC2 Instance Status

You can connect to your instance with any SSH client. Find the Key Pair file you created previously for this instance. Make note of the Public DNS address of your new instance. For example, to connect to my test instance I just created, I would use the following SSH command:

ssh -i Web1.pem root@ec2-67-202-37-5.compute-1.amazonaws.com

Web1.pem is the private Key Pair that I created for this instance. If you did not save your Key Pair, then you will have to destroy this instance and create a new Key Pair and start a new instance with the new Key Pair.

Once you have finished playing around with your instance, make sure you terminate the instance so you won’t be charged for the idle instance.
In future articles, I will show you how to setup Elastic IP’s, Elastic Block Storage, run shell scripts to setup your new instance and creating an instance using only Amazon’s EC2 API.

No related posts.