fo r

guides on clo ud tical c a co pr

ting pu m

OpenNebula and Amazon EC2 integration

Table of Contents
Hardware/software requirements :...................................................................................................2 Setup OpenNebula Host...................................................................................................................2 Configure the OpenNebula host:.................................................................................................2 Install OpenNebula in Ubuntu server :........................................................................................4 Prepare Amazon EC2 :......................................................................................................................7 Install EC2 API Library and tools in OpenNebula and configure OpenNebula ........................7 Create EC2 Host and VM in OpenNebula .................................................................................9 Connecting to your new EC2 instance :.....................................................................................11 Terminate an EC2 instance through OpenNebula.....................................................................11

place
fo r

al guides on clou ctic dc a r o p

ing put m

place

ANIL KUMAR A N Chennai cloud.b.lab@zoho.com admin@cloud-b-lab.co.in

fo r

guides on clo ud tical c a co pr

ting pu m

OpenNebula and Amazon EC2 integration

Hardware/software requirements :
A VT enabled hardware with Ubuntu 11.04 or 11.10 server installed Hostname : OneHost IP address : 192.168.1.104 Gateway : 192.168.1.1 username : localadmin

Setup OpenNebula Host

Before installation of OpenNebula in Ubuntu Server , just check the following to ensure that you have a proper setup.
# The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet static address 192.168.1.104 netmask 255.255.255.0 network 192.168.1.0 broadcast 192.168.1.255 gateway 192.168.1.1 # dns-* options are implemented by the resolvconf package, if installed dns-nameservers 192.168.1.1 dns-search example.com /etc/hosts 127.0.0.1 localhost 192.168.1.104 OneHost.example.com search example.com nameserver 192.168.1.1 OneHost

/etc/network/interfaces

/etc/resolv.conf

Configure the OpenNebula host:

Create a folder "cloud" and create a group named "cloud"

sudo mkdir -p /srv/cloud/ sudo groupadd -g 10000 cloud

place

fo r

guides on clo ud tical c a co pr

ting pu m

Create a user "oneadmin" , add user to group "cloud" and have /srv/cloud/one as home folder.
sudo useradd -u 10000 -m oneadmin -d /srv/cloud/one -s /bin/bash -g cloud

Setup password for "oneadmin" and make oneadmin owner of "/srv/cloud"

sudo passwd oneadmin - I chose redhat123 as the password sudo chown -R oneadmin:cloud /srv/cloud/

Test by logging as user "oneadmin" and exit

su -l oneadmin exit

Install Network file Server [NFS] Not a must for EC2 integration
sudo apt-get install nfs-kernel-server

edit /etc/exports and add the following line to make folder /srv/cloud/one/var shareable with ESXi server. We will configure ESXi server later.
/srv/cloud/one/var 192.168.1.0/24(rw,sync,no_subtree_check,all_squash,anonuid=10000,anongid=10000 )

Restart NFS server

sudo /etc/init.d/nfs-kernel-server start

create a SSH key for oneadmin and disable host key checking all hostkeys known on the OpenNebula node.
su -l oneadmin ssh-keygen {Note - all defaults, and no passphrase.} cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys nano ~/.ssh/config [add below two lines to SSH config file] Host * StrictHostKeyChecking no exit

place

else make

fo r

guides on clo ud tical c a co pr

ting pu m

Install OpenNebula in Ubuntu server :

Login to 192.168.1.98[OneHost] and download OpenNebula Release 3.2.1

su -l oneadmin Download stable opennebula release for ubuntu [opennebula-3.2.1.tar.gz] from http://downloads.opennebula.org/ and save it in /srv/cloud/one folder. Also add oneadmin to sudoers file with permissions same as root,

Un-tar the build

tar xzf opennebula-3.2.1.tar.gz cd opennebula-3.2.1//

Before installing OpenNebula, install all pre-requisite

place

packages

sudo apt-get install libsqlite3-dev libxmlrpc-c3-dev g++ ruby libopenssl-ruby libssl-dev ruby-dev sudo apt-get install libxml2-dev libmysqlclient-dev libmysql++-dev libsqlite3-ruby libexpat1-dev sudo apt-get install libc6 libgcc1 libmysqlclient16 libpassword-ruby libsequel-ruby libsqlite3-0 libssl0.9.8 libstdc++6 libxml2 libxmlrpc-c3-0 libxmlrpc-core-c3-0 sudo apt-get install ruby rubygems libmysql-ruby libsqlite3-ruby libamazonec2-ruby sudo apt-get install rake rubygems libxml-parser-ruby1.8 libxslt1-dev genisoimage scons sudo gem install nokogiri rake xmlparser sudo apt-get install opennebula-common [optional] sudo apt-get install mysql-server [ set the password when asked. I normally give mygreatsecret as the pwd]

configure MYSql: <refer below screen shot in case of any doubt) mysql -uroot -pmygreatsecret CREATE USER 'oneadmin'@'localhost' IDENTIFIED BY 'oneadmin'; CREATE DATABASE opennebula; GRANT ALL PRIVILEGES ON opennebula.* TO 'oneadmin' IDENTIFIED BY 'oneadmin'; quit;

<Screen shot as shown below>

fo r

guides on clo ud tical c a co pr

ting pu m

Before installing OpenNebula, configure mysql support as oneadmin user

[change your folder to opennebula source]

cd ~/opennebula-3.2.1 scons sqlite=no mysql=yes

Install openebula in /srv/cloud/one accessible user "oneadmin"

place

by group cloud and as

./install.sh -u oneadmin -g cloud -d /srv/cloud/one

Create a profile file[~/.bash_profile] to set ENVIRONMENT VARIABLES required to start and use services rendered by "one"

nano ~/.bash_profile export ONE_LOCATION=/srv/cloud/one export ONE_AUTH=$ONE_LOCATION/.one/one_auth export ONE_XMLRPC=http://localhost:2633/RPC2 export PATH=$ONE_LOCATION/bin:/usr/local/bin:/var/lib/gems/1.8/bin/:/var/lib/gems/1.8/:$PATH

execute the profile file and set the environment variables

source ~/.bash_profile

[Note: Anytime you open a new SSH window for OneHost, change user to oneadmin and source ~/.bash_profile before issuing any one command]
Create and store OpenNebula user and password in a file. Substitute <THE_PASSWORD> with value

mkdir ~/.one echo "oneadmin:<THE_PASSWORD>" > ~/.one/one_auth

Make required changes in OpenNebula configuration file ~/etc/oned.conf

nano ~/etc/oned.conf a. comment following line # Line 58 or near by [c hange if your password for oneadmin is some different] #DB = [ backend = "sqlite" ] b. Set SQL as MYSQL-uncomment #lines 61 through 66 or near by DB = [ backend = "mysql", server = "localhost", port = 0, user = "oneadmin", passwd = "oneadmin", db_name = "opennebula" ]

fo r

guides on clo ud tical c a co pr

ting pu m

Start OpenNebula
{ Note: it should start with no error messages}

one start

Now You can test OpenNebula services by typing onevm list on the $ prompt. The command should execute with no error.The list will be empty for now.

place

fo r

guides on clo ud tical c a co pr

ting pu m

Prepare Amazon EC2 :

Create an Amazon account Go to http://aws.amazon.com, and click Create an AWS Account. Follow the on-screen instructions. Part of the sign-up procedure involves receiving a phone call and entering a PIN using the phone keypad. Refer:
http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/SignUp.html

Create credentials make a folder ec2 with in /srv/cloud/one/

After signing up, you should end up at the EC2 console Create a key pair and download the private key Click Key Pairs under Networking and Security in the Navigation pane and then click the Create Key Pair button (save it in e.g. ~/.ec2/ec2.pem). This private key is for making SSH connections to newly created instances. You will also need to set up your Amazon API credentials. Go to Account->Security Credentials click X.509 Certificates tab Create a new Certificate Download the private key and the certificate (save them in e.g. ~/.ec2/certXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem and ~/.ec2/pkXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem). Never share these files . Make your credential files private: chmod go-rwx ~/.ec2/*.pem 5. Scroll to the bottom of the page and note your account ID (a number of the form XXXX-XXXXXXXX). Create a Ubuntu instance in EC2 [http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/LaunchInstance.html] Note the AMI- ID ,as we need to specify it in onevm template later Create an Elastic IP using NETWORK and SECURITY section and note the IP. Install EC2 API Library and tools in OpenNebula and configure OpenNebula Install EC2 API tools in Openenbula frontend machine

Login to openNebula Front-end machine [onehost] as root user and perform the following steps apt-get update Install the Amazon EC2 Query API library: gem install amazon-ec2 Install the Sinatra web framework and the thin web server: [if you get Invalid gemspec in [/var/lib/gems/1.8/specifications/tilt-1.3.3.gemspec]: invalid date format in specification: "2011-08-25 00:00:00.000000000Z"

place

fo r

guides on clo ud tical c a co pr

ting pu m

Type below on the command prompt as root user: sudo sed -i 's/ 00:00:00.000000000Z//' /var/lib/gems/1.8/specifications/* ] gem install sinatra gem install thin gem install uuid gem install curb apt-get install libsqlite3-ruby apt-get install libcurl4-gnutls-dev apt-get install openssl libruby1.8 gem install sqlite3-ruby apt-get install openjdk-6-jdk The service is configured through the $ONE_LOCATION/etc/econe.conf file Edit $ONE_LOCATION/etc/econe.conf and update the following

:SERVER=<IP of onehost/OpenNebula frontend> :PORT=4567

Check if multiverse is enabled in Ubuntu 11:10 server

sudo nano /etc/apt/sources.list if commented , uncomment the following deb http://in.archive.ubuntu.com/ubuntu/ oneiric multiverse deb-src http://in.archive.ubuntu.com/ubuntu/ oneiric multiverse deb http://in.archive.ubuntu.com/ubuntu/ oneiric-updates multiverse deb-src http://in.archive.ubuntu.com/ubuntu/ oneiric-updates multiverse Login as oneadmin user Make sure you have the following environment variables set up in your shell profile. This is accomplished by adding the following lines to your ~/.bash_profile :

sudo nano ~/.bash_profile

export EC2_KEYPAIR=<your keypair name> # name only, not the file name export EC2_URL=https://ec2.<your region>.amazonaws.com # example: ec2.us-east-1.amazonaws.com export EC2_PRIVATE_KEY=<PATH>/<where your private key is>/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem value of the EC2_PRIVATE_KEY environment variable export EC2_CERT=<PATH>/<where your certificate is>/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem # certificate to use when constructing requests to Amazon EC2 export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/

place

# The

The X.509

Load the changes into the current shell environment:

source ~/.bash_profile Check to see if it's working by running the following command:

ec2-describe-images -o self -o amazon If above step fails due to "Client.AuthFailure" then ensure you have signed up for both AWS and ec2 with amazon.com, and have provided valid payment details. Also double check that the EC2_PRIVATE_KEY and EC2_CERT point to the correct

fo r

guides on clo ud tical c a co pr

ting pu m

locations Create a AMI using a running/stopped EC2 instance

ec2-create-image <instance id> --name <name for new instance> -K <EC2_PRIVATE_KEY> -C <EC2_CERT> -v -U <EC2_URL> example: I am using the already sourced environment values: ec2-create-image i-5ccd593b --name ubuntu1110x64 -K $EC2_PRIVATE_KEY -C $EC2_CERT -v -U $EC2_URL Edit File: /srv/cloud/one/etc/vmm_ec2/vmm_ec2rc and add below lines nano /srv/cloud/one/etc/vmm_ec2/vmm_ec2rc EC2_HOME=/usr
program itself } # find the installed location of ec2-api-tools and provide the path here. For me it is /usr {/bin will appended by the

EC2_URL=https://ec2.us-east-1.amazonaws.com EC2_PRIVATE_KEY=<PATH>/<where your private key is>/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem EC2_CERT=<PATH>/<where your certificate is>/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem JAVA_HOME=/usr/lib/jvm/java-6-openjdk/ Edit File: /srv/cloud/one/etc/oned.conf and make the follwing changes

# EC2 Information Driver Manager Configuration #------------------------------------------------------------------------------IM_MAD = [ name = "im_ec2", executable = "one_im_ec2", arguments = "im_ec2/im_ec2.conf" ] #------------------------------------------------------------------------------VM_MAD = [ name = "vmm_ec2", executable = "one_vmm_ec2", arguments = "-u https://ec2.us-east-1.amazonaws.com -k <PATH>/<where your private key is>/pkXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem -c <PATH>/<where your certificate is>/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem -h <path where ec2-api tools are installed> vmm_ec2/vmm_ec2.conf", type = "xml" ] #------------------------------------------------------------------------------# Dummy Transfer Manager Driver Configuration #------------------------------------------------------------------------------TM_MAD = [ name = "tm_dummy", executable = "one_tm", arguments = "tm_dummy/tm_dummy.conf" ] #-------------------------------------------------------------------------------

Create EC2 Host and VM in OpenNebula

Start one server as oneadmin user

source ~/.bash_profile one start Add EC2 host to OpenNebula

place

fo r

guides on clo ud tical c a co pr

ting pu m

onehost create ec2 im_ec2 vmm_ec2 tm_dummy dummy You will get an output like 0 RVM TCPU FCPU ACPU TMEM FMEM AMEM STAT 500 500 500 8.5G 8.5G 8.5G on

ID NAME 0 ec2

Create a OpenNebula template file to launch an Instance in Amazon EC2

nano ~/ec2.one Add the following content to ec2.one , save and exit

#EC2 template to submit a VM to EC2 CPU = 0.5 MEMORY = 128 EC2 = [ AMI="<ami id of ubuntu instance>", KEYPAIR="<Name of keypair>", ELASTICIP="<Generated Elastic IP>", AUTHORIZED_PORTS="22", INSTANCETYPE=m1.small] #Add this if you want to use only EC2 cloud #REQUIREMENTS = 'NAME = "ec2"' Launch the Instance in Amazon EC2

onevm create ec2.one Monitor the log files in ~/var/oned.log and ~/var/<vmid>/vm.log

oned.log: Mon Apr 16 01:03:51 2012 [DiM][I]: New VM state is ACTIVE. Mon Apr 16 01:03:51 2012 [LCM][I]: New VM state is PROLOG. Mon Apr 16 01:03:51 2012 [VM][I]: Virtual Machine has no context Mon Apr 16 01:03:51 2012 [DiM][D]: Deploying VM 1 Mon Apr 16 01:03:51 2012 [LCM][I]: New VM state is BOOT Mon Apr 16 01:14:16 2012 [VMM][D]: Message received: LOG I 1 ExitCode: 0 Mon Apr 16 01:14:10 2012 [VMM][I]: Monitoring VM 1. Mon Apr 16 01:14:16 2012 [VMM][D]: Message received: POLL SUCCESS 1 USEDMEMORY=0 USEDCPU=0 NETTX=0 NETRX=0 STATE=a IP=ec2-23-23200-8.compute-1.amazonaws.com vm.log: Mon Apr 16 01:03:51 2012 [TM][D]: Message received: TRANSFER SUCCESS 1 Mon Apr 16 01:04:04 2012 [VMM][D]: Message received: LOG I 1 ExitCode: 0 Mon Apr 16 01:03:51 2012 [VMM][I]: Generating deployment file: /srv/cloud/one/var/1/deployment.0 Mon Apr 16 01:04:04 2012 [VMM][D]: Message received: DEPLOY SUCCESS 1 i-5ab7263d Mon Apr 16 01:03:58 2012 [VMM][I]: ExitCode: 0 Mon Apr 16 01:04:04 2012 [VMM][I]: ExitCode: 0 Mon Apr 16 01:04:04 2012 [LCM][I]: New VM state is RUNNING Mon Apr 16 01:14:16 2012 [VMM][W]: Adding custom monitoring attribute: IP ec2-23-23-200-8.compute-1.amazonaws.com Mon Apr 16 01:14:16 2012 [VMM][D]: Monitor Information:

Just note the monitoring attribute: IP {ec2-23-23-200-8.compute-1.amazonaws.com }from the log, [You can get it from EC2 console also]. We need it to SSH to the instance

place

fo r

guides on clo ud tical c a co pr

ting pu m

On successful creation you will get following output

onevm list ID USER GROUP NAME 1 oneadmin oneadmin one-1 STAT CPU runn 0 MEM 0K HOSTNAME ec2 TIME 00 00:02:21

Login in AMAZON EC2 management console and check the status of the instance. It should be running in awhile

Connecting to your new EC2 instance : In order to log in to your instance, you will need to authorize network access to the ssh port for the default security group: {if you have created a different security group and already added SSH to it, just skip below step} ec2-authorize default -p 22 You may then log in to the instance using ssh: ssh -i <private SSH key file> ubuntu@<external-host-name> example : ssh -i ~/.ec2/ec2.pem ubuntu@ec2-23-23-200-8.compute-1.amazonaws.com The <private SSH key file> is the filename of the private SSH key that corresponds to the Amazon Key Pair that you specified in the ec2-run-instances command. The <external-host-name> can be found using the ec2-describe-instances command. An example SSH command: Once you have logged in, you may begin to set up and use the instance just like any other Ubuntu machine.

Terminate an EC2 instance through OpenNebula onevm shutdown <vmid>

place

fo r

guides on clo ud tical c a co pr

ting pu m

That's it. You have successfully launched an Amazon EC2 instance using OpenNebula If you liked this tutorial , just post your comments to ANIL KUMAR A N cloud.b.lab@zoho.com admin@cloud-b-lab.co.in

place