Testbed Operator Documentation

OMF Installation Guide

Introduction

In an OMF testbed, everything is a resource. The entity that controls one or multiple resources is called Resource Controller (RC). The RC can run directly on the resource (e.g. a PC), or run on a separate computer that can control the resources (e.g. a bunch of sensor nodes or an OpenFlow switch). OMF is designed to support many different types of resources, and additional RCs for those resources can make use of our OMF APIs.

All communication in OMF is done via Publish/Subscribe (PubSub). By default OMF uses AMQP for that, but other messaging layers such as XMPP are supported. Each OMF component must be able to talk to at least one PubSub server, although there can be multiple PubSub servers used in an experiment. The examples in this guide are using XMPP.

The experimenter uses the Experiment Controller (EC) to run the experiment script and steer the resources. The EC can be installed on a user-facing machine inside the testbed, or alternatively on the user's own computer.

Prepare Your Environment

OMF could run under Ruby with versions not earlier than 1.9.3. Your operating system most likely have this version available by the time of writing this documentation. If you do require information regarding installing Ruby, please see official ruby site, or simply search for it.

Some additional packages shall be installed in order to compile native extensions for certain dependent libraries, for example 'Eventmachine':

Ubuntu/Debian

sudo apt-get install ruby-dev build-essential libssl-dev

If you decided, for some particular reasons, to choose XMPP over AMQP as your communication layer, you also need additional libraries which handle XML parsing.

sudo apt-get install libxml2-dev libxslt-dev

Fedora

sudo yum update -y audit
sudo yum install -y ruby-devel make gcc gpp gcc-c++ openssl-devel

Again, ONLY needed for XMPP and XML

sudo yum install -y libxml2 libxml2-devel libxslt libxslt-devel

Install communication server

OMF6 currently supports XMPP and AMQP for its message transport. Only one message transport is required.

We RECOMMEND AMQP for its superior performance and pub-sub support

TEST OPERATORS need to install a communication server in their testbed. It should be reachable from the outside Internet if any of the testbed users are on the outside of your organisation or if you want to federate your testbed with another organisation over the public Internet. DEVELOPERS might just run a server on their local machine for development.

AMQP server (RabbitMQ) Installation

For OMF, we RECOMMEND using RabbitMQ. Other AMQP servers may work as well, but have not been tested.

• Uninstall any other AMQP servers first
• Make sure port TCP 5672 is open in your firewall
• Install RabbitMQ on Ubuntu / Debian:

  sudo apt-get install rabbitmq-server
  

​Installation instructions for other operating systems

Common Issues When Installing RabbitMQ

• Allow the guest user to connect from a remote host

If you wish to allow the guest user to connect from a remote host, you should set the loopback_users configuration item to []. A complete rabbitmq.config which does this would look like:

[[{loopback_users, []]}].

​More details

• Client (EC or RC) cannot connect to AMQP broker (aka rabbitmq-server)

​Older installation issues When installing on old Ubuntu and some Debian distro, the default RabbitMQ version you get may be old. For example on Debian 6 you will get RabbitMQ version below 2. However, the AMQP library used by EC and RC expect a more recent version of RabbitMQ. ​Follow the instruction to upgrade your rabbitmq-server After upgrading rabbitmq-server as mentioned above, you should clean any existing queue data from the previous version by removing all files under /var/lib/rabbitmq/mnesia/

• What PORTS should I open when using clustering setup

​Firewall instructions In the most common configuration you will need to open ports 4369 and 25672 for clustering to work.

• Troubleshooting other RabbitMQ issues

​Troubleshooting instructions

Verify AQMP Server Installation =

If you want to verify that your RabbitMQ server is responding, you can follow this "Hello World" tutorial.

XMPP Server (Openfire) Installation

For OMF, we recommend using Openfire. Other XMPP servers may work as well, but this has not been extensively tested. Ejabberd is known to have some issues regarding node permissions and database cleanup. If are upgrading from an older Openfire release and you have trouble running OMF, please wipe your Openfire database and recreate the pubsub nodes.

• Uninstall any other XMPP servers first
• Make sure ports TCP 5222, 5269 and 9090 are open in your firewall
• Openfire 3.8.1 requires Java, but has some issues with OpenJDK. Instead we recommend to install Oracle Java 7. Ubuntu users can follow this guide.
• After installing Java, on Ubuntu 12.04 or higher, run the following commands as root:

  wget http://www.igniterealtime.org/downloadServlet?filename=openfire/openfire_3.8.1_all.deb
  dpkg -i downloadServlet\?filename\=openfire%2Fopenfire_3.8.1_all.deb
  

• On any other OS, download Openfire and install it alongside the Oracle Java JRE
• Check if openfire is running: {{{ ps aux | grep openfire . Startup can take a while, please be patient.
• If openfire failed to start, you may need to edit /etc/init.d/openfire and set the correct JAVA_HOME for your Java installation, e.g.:

  JAVA_HOME=/usr/lib/jvm/java-7-oracle
  

• Direct your web browser to ​http://localhost:9090 and begin the setup wizard
• Choose your language and click continue
• Enter the XMPP server's hostname (e.g. xmpp.domain.com or localhost) in the Domain field and click continue
• Choose the embedded database. You can also use other database types, but you will need to install the database server first and manually create a user and a database for Openfire.
• If you are using mySQL, you need to change the length of the Pubsub node names in the mySQL database. Basically you need to run the commands:

  mysql -u openfire -p
  use openfire;
  alter table ofPubsubNode modify name VARCHAR(100);
  quit;
  

• Choose the default profile and click continue
• Enter an admin password and click continue, then wait until the installation is finished
• Log on to the web GUI at ​http://localhost:9090 with the user/password you chose in the wizard
• Click 'Server', 'Server Settings', 'Registration & Login', enable 'Inband Account Registration' and disable 'Anonymous Login'
• To avoid duplicate subscriptions to the same topic in order to reduce the volume of messages, we recommend turning the multiple subscriptions feature off. Inside the Openfire system properties panel, add Property Name: xmpp.pubsub.multiple-subscriptions Property Value: false
• OPTIONAL: for server2server connectivity (used for testbed federation), you need to set up a DNS name "pubsub.yourhost.name" which can be resolved by public DNS servers. It can be an A record pointing to the same IP address as your host, or a CNAME pointing to the hostname. There is no need for a DNS SRV record. Learn more about S2S here.
• OPTIONAL: if your testbed consists of several hundred nodes, you may consider increasing the maximum open file descriptor limit for the "openfire" user. This can be done by editing /etc/security/limits.conf and adding the lines root soft nofile 100000 and root hard nofile 100000, which allows a maximum of 100K open sockets/files per process. The default in Ubuntu is 1024. If this is not increased you may run into a problem where openfire does not accept any more incoming TCP connections.
• Restart Openfire to apply any changes made: /etc/init.d/openfire restart

Verify XMPP Server Installation

To verify that Openfire has been set up correctly, we are providing a test script (also available here). Run:

curl -O https://raw.github.com/mytestbed/omf/master/doc/openfire_test.rb
ruby openfire_test.rb <xmpp_hostname>

Watch the output for any lines starting with ERROR. Double check your Openfire settings as described above and re-run the test until you receive no more ERRORs.

Install Resource Controller (RC)

On a typical node, you could install the RC as root:

gem install omf_rc --no-ri --no-rdoc

If you'd like to start the Resource Controller upon booting your node, please run the following as root (THIS COMMAND WILL OVERWRITE /etc/omf_rc/config.yml if the file exists):

install_omf_rc -i -c

This installs an init script for your distribution (currently supports Debian, Ubuntu and Fedora) and installs a default RC configuration file at /etc/omf_rc/config.yml.

You should edit this file and set the correct values for your testbed before you can run the RC successfully. Here's an example configuration file:

environment: production

communication:
  url: amqp://localhost

resources:
- type: node
  uid: <%= Socket.gethostname %>

Ruby's ERB syntax is used here to set the UID (topic name) to the output of the Ruby command Socket.gethostname. We assume that the host name of each node is unique (e.g. assigned via DHCP), so we can use the same configuration file for all of them.

Start RC

If you want to start the RC using an init script and you installed this script with the '-i' flag above, you can start the daemon like this:

Ubuntu:

start omf_rc

Debian & Fedora:

/etc/init.d/omf_rc start

Check for the RC process by running:

ps aux | grep omf_rc

If there is no "omf_rc" process, check the syslog for any errors that the startup script may have thrown. Also check /var/log/omf_rc.log for any errors.

To verify that your RC is responding correctly, we are providing a test script (also available here). Open a new terminal and run:

curl -O https://raw.github.com/mytestbed/omf/master/doc/rc_alive_check.rb
ruby rc_alive_check.rb xmpp://<xmpp_hostname> <your hostname>

Using the same XMPP URI as in the RC config file and your local machine's hostname (run hostname to find out), unless you changed the :uid: parameter in the RC config file to another name. Watch the output for any lines starting with ERROR. If the RC could be contacted successfully at the "test" topic, your resource is ready to run experiments.

Configure RC

RC could be configured via changing the configuration files. Examples can be found under ​https://github.com/mytestbed/omf/tree/master/omf_rc/config/

Only load certain resource proxies and set default property value ​https://github.com/mytestbed/omf/blob/master/omf_rc/config/config_node_topo_check.yml.example

Customise Logging ​https://github.com/mytestbed/omf/blob/master/omf_rc/config/with_custom_logging.yml

ALWAYS RESTART RC TO MAKE CHANGES EFFECTIVE

Test RC

To check if a particular resource controller responds to FRCP messages properly, simply send a FRCP REQUEST message to the resource, and expect an answer from it. OMF provides a tiny utility to do exactly this:

omf_send_request -r amqp://localhost/interlagos uid

It basically asks a resource called 'interlagos' about its 'uid' property. If such RC runs properly, you would see:

interlagos
  uid: interlagos
-----------------

Otherwise you would see NOTHING.

Install Experiment Controller (EC)

Install EC =

gem install omf_ec --no-ri --no-rdoc

Run EC

omf_ec <PATH_TO_YOUR_OEDL_FILE>

This, without any parameters will use built in default values. It will:

Connect to amqp server on localhost Write info level log messages to STDOUT Write all log messages to rolling log files in /var/tmp/ as omf_ec..log Configuring EC You could use command line options or configuration file to set up your experiment controller.

It is always useful to run EC help page for available options

omf_ec -h

Some common command line options are:

• Use another AQMP server

  omf_ec -u amqp://another_amqp_server <PATH_TO_YOUR_OEDL_FILE>
  

• Use OML server at localhost:3003

  omf_ec -u amqp://another_amqp_server --oml-uri tcp:localhost:3003 <PATH_TO_YOUR_OEDL_FILE>
  

• Show :debug log message in STDOUT

  omf_ec -u amqp://another_amqp_server --oml-uri tcp:localhost:3003 -d <PATH_TO_YOUR_OEDL_FILE>
  

Options can also be set by using a configuration file, you can store options like the communication URI and the OML URI here to avoid entering them on every EC run, and provide the configure file using '-c' option. For example (​simple configuration file):

communication:
  url: amqp://localhost

oml_uri: tcp:localhost:3003

Save the file as my_ec.yml for example and then start experiment controller by:

omf_ec -c my_ec.yml <PATH_TO_YOUR_OEDL_FILE>

The other example config files can be found at ​OMF github

Customise Logging

You could ​customise logging by describing level, appenders, layout etc. in the logging configuration file.

Does My EC Work?

To verify that you can run experiments on the testbed, we are providing an experiment script (also available here). Run:

curl -O https://raw.github.com/mytestbed/omf/master/doc/oedl_simple_test.rb

omf_ec -u <xmpp|amqp>://<pubsub_hostname> exec oedl_simple_test.rb -- --res1 <hrn>
Replace <pubsub_hostname> with the hostname of the testbed's XMPP server or AMQP server. Replace <hrn> with the ID of a PC-style resource controller in the testbed (this is usually the testbed node's hostname).

18:36:59 INFO  XMPP::Communicator: Connecting to '<xmpp_hostname>' ...
18:36:59 INFO  Object: Connected
18:36:59 INFO  Object: Start experiment: 2013-03-07T07:36:59Z
18:36:59 INFO  OmfEc: res1 = "test" (String)
18:36:59 INFO  OmfEc: Subscribed to 249bf4cd-f799-4fe3-aee6-72f92af002db
18:36:59 INFO  OmfEc: Subscribed to test
18:36:59 INFO  OmfEc: Config test to join Actor
18:37:00 INFO  OmfEc: Newly discovered resource >> test
18:37:00 INFO  OmfEc: Event triggered: 'ALL_UP'
18:37:03 INFO  Object: TEST - allGroups
18:37:03 INFO  OmfEc: Subscribed to 249bf4cd-f799-4fe3-aee6-72f92af002db_application
18:37:03 INFO  OmfEc: Resource xmpp://ccce4bf3-c8f7-4984-8e8e-1ade28d132a6@norbit.npc.nicta.com.au created
18:37:03 INFO  OmfEc: Newly discovered resource >> ccce4bf3-c8f7-4984-8e8e-1ade28d132a6
18:37:03 INFO  OmfEc: Event triggered: 'Actor_application_/bin/date_created_ba8b4fad-8d47-4782-b12b-9c93dd215bf2'
18:37:03 INFO  OmfEc: APP_EVENT STARTED from app /bin/date - msg: env -i /bin/date
18:37:03 INFO  OmfEc: APP_EVENT STDOUT from app /bin/date - msg: Thu Mar  7 18:37:03 EST 2013
18:37:03 INFO  OmfEc: APP_EVENT DONE.OK from app /bin/date - msg: status: pid 35135 exit 0
18:37:06 INFO  Object: TEST - group
18:37:06 INFO  OmfEc: Subscribed to 249bf4cd-f799-4fe3-aee6-72f92af002db_application
18:37:06 INFO  OmfEc: Newly discovered resource >> d3416b01-51d9-48d4-b6b5-442454e8e7fa
18:37:06 INFO  OmfEc: Resource xmpp://d3416b01-51d9-48d4-b6b5-442454e8e7fa@norbit.npc.nicta.com.au created
18:37:06 INFO  OmfEc: Event triggered: 'Actor_application_/bin/hostname -f_created_50da62ba-c8b5-49b1-8182-7632d68f312a'
18:37:06 INFO  OmfEc: APP_EVENT STARTED from app /bin/hostname -f - msg: env -i /bin/hostname -f
18:37:06 INFO  OmfEc: APP_EVENT STDOUT from app /bin/hostname -f - msg: y68.dynhost.nicta.com.au
18:37:06 INFO  OmfEc: APP_EVENT DONE.OK from app /bin/hostname -f - msg: status: pid 35136 exit 0
18:37:09 INFO  OmfEc: Exit in up to 15 seconds...
18:37:19 INFO  OmfEc: Release applications and network interfaces
18:37:19 INFO  OmfEc: Subscribed to ccce4bf3-c8f7-4984-8e8e-1ade28d132a6
18:37:19 INFO  OmfEc: Subscribed to 249bf4cd-f799-4fe3-aee6-72f92af002db
18:37:19 INFO  OmfEc: Subscribed to d3416b01-51d9-48d4-b6b5-442454e8e7fa
18:37:19 INFO  OmfEc: Subscribed to 249bf4cd-f799-4fe3-aee6-72f92af002db
18:37:24 INFO  XMPP::Communicator: Disconnecting ...

Gaining access to an OMF testbed

Contact an OMF testbed operator for an account. Make a resource reservation if required and gather the HRNs of the resources you'd like to use. Also find out what the testbed's PubSub server hostname is.