EMR tricks
use Elastic MapReduce CLI.
Create a Spark 0.8.1 cluster:
elastic-mapreduce –create –alive \
–name “Spark/Shark Cluster” \
–bootstrap-action s3://elasticmapreduce/samples/spark/0.8.1/install-spark-shark.sh \
–bootstrap-name “Spark/Shark” \
–instance-type m3.xlarge \
–instance-count 3
[Doesn’t work] Create a Spark 1.0.0 cluster (notice –ami-version):
elastic-mapreduce –create –alive –name “Spark/Shark Cluster 1.0.0” \
–bootstrap-action s3://elasticmapreduce/samples/spark/1.0.0/install-spark-shark.rb \
–bootstrap-name “Spark/Shark” \
–instance-type m3.xlarge \
–instance-count 3 \
–ami-version 3.0.3
Show all running clusters:
elastic-mapreduce –list –active
Install tricks
Installing AWS cli on a Mac is dead easy with pip:
pip install awscli
# If you get ‘ValueError: unknown locale: UTF-8’
# edit ~/.profile, add lines (use en_US if appropriate):
# export LC_ALL=da_DK.UTF-8
# export LANG=da_DK.UTF-8
Make an AWS CLI profile (if you have multiple accounts):
# Have Access Key ID and Secret Access Key ready
aws configure –profile [profile name]
# Using the profile
aws –profile [profile name]
Create shortcut for using AWS CLI profile, e.g. with profile foo:
alias aws-foo=”aws –profile foo”
S3 tricks
Public read-access bucket policy (change ‘yourbucketname’ to actual bucket):
{
“Version”: “2008-10-17”,
“Statement”: [
{
“Sid”: “AllowPublicRead”,
“Effect”: “Allow”,
“Principal”: {
“AWS”: “*”
},
“Action”: [
“s3:GetObject”
],
“Resource”: [
“arn:aws:s3:::yourbucketname/*”
]
}
]
}
used s3funnel, but now I use the AWS cli.
Get help on AWS cli s3 commands:
aws s3 [command] help
# command one of: cp, ls, mb, mv, rb, rm, sync, website
# actually, the reference at http://aws.amazon.com/cli/ is better IMO
Create a new buckets in S3:
aws s3 mb s3://skipperkongen-test
# creates bucket called ‘skipperkongen-test’
List all your buckets in S3:
aws s3 ls
# 2014-07-17 23:04:53 skipperkongen-test
Move a file to S3:
aws s3 mv testfile.txt s3://skipperkongen-test/testfile.txt
Move a whole directory to S3:
aws s3 sync mylocaldir s3://skipperkongen-test/mys3dir
List files in an S3 bucket:
aws s3 ls s3://skipperkongen-test
Delete an entire S3 bucket (must be empty or use –force):
aws s3 rb –force s3://skipperkongen-test
# individually deletes each file, may take a while
theassyrianblog 