Building OTP applications with Rake for EC2

I uploaded a mashup of other’s code this morning to github. Namely charpi’s Rakefile, used to build the Duke of Erl’s ec2nodefinder.

Rakefile

It’s way more than a Rakefile, actually. It’s a whole project organization.

Your OTP applications go into lib/, and you can have as many as you want in there.

Just typing rake will build the release files, doing all the chores (creating app files and rel files … BORING !)

Here are all the defined tasks.

rake clean # Remove any temporary products.
rake clobber # Remove any generated file.
rake compile # Compile all project
rake dialyzer # Run dialyzer
rake edoc[name] # Buid Application documentation
rake edocs # Buid all application documentation
rake erlang_applications # Build application resource file
rake erlang_modules # Compile Erlang sources
rake erlang_release_files # Build erlang boot files
rake erlang_releases # Build release tarball
rake erlang_target_systems[n] # Build release tarball with erts
rake run[name,node] # Runs application
rake tests # Runs EUnit tests
rake upload # Sends release to S3 (presumably for deployment on EC2)

ec2nodefinder

In a nutshell, makes all erlang nodes in a single EC2 security group discover each other automatically.

Your applications on those instances can have processes joining pg2 groups, and you get scalability with very little efforts. However pg2 is quite crude in choosing a process as its only strategy is closest process or random.

ec2nodefinder is described here. Works well and very handy. I however would like to remove the dependency on ec2-describe-instances (which in turn requires a JRE on the deployment server). It’s working as advertised but JVM startup slows down discovery.

AMIs

Check out Kevin Smith’s blog. He is building erlang centric AMI based on Eric Hammond’s Ubuntu AMIs

Next cool things I’d like to do or see

Run OTP releases on <n> instances, with just one command :

ec2-run-instances ami-XXXXX -n <n> -d "release=<s3bucket>/<otp-release>"

It won’t be that easy, because the AMIs will need credentials to access S3 (and EC2 admin commands)

Actually that’s my 10€ question. Where are AWS credentials best stored ?

• have the cert / private key on S3 and pass the S3 AWSACCESS* values as parameters :

ec2-run-instances ami-XXXXX -n <n> -d "release=<s3bucket>/<otp-release>&AWS_ACCESS_KEY=XXXXX&AWS_SECRET_ACCESS_KEY=XXXX++XXXX"

• pass everything in a tar.gz (-f flag of ec2-run-instances)

Not decided yet.

relup, appup … hot code upgrade

No rake magic yet for those. Watch this space or charpi’s !

Code

Get the code on github.

Comments

Hi, I have just written a module for finding available erlang node on EC2. The module uses amazon ec2 api via https.

http://github.com/stevenchan/ec2_node.erl/tree/master

Posted by Seven Chan on 29 Jan 2009 at 14:45

Add comment