OpenStack Nova ZeroMQ RPC Driver

Deployment guide^[a]

---

Abstract

With Folsom, Nova introduces an optional messaging system using ZeroMQ.  For some Nova deployments it may be desirable to use a broker-less RPC mechanism.  This document provides deployment information for this driver.

Topics are used to identify the destination for an RPC call. There are two types of topics, bare topics and directed topics.  Bare topics look like “compute” , while directed topics look like “compute.machine1”.

Enabling

To enable the driver, the ‘rpc_backend’ flag must be passed to each Nova service. The value of this flag should be ‘nova.rpc.impl_zmq’.

DNS

All TCP connections are made to hostnames. To specify, these are hostnames, not FQDNs. Host resolution is mandatory. The best practice is to deploy viable nameservers which can resolve all hostnames in the cloud under a single domain. Machines should then have dns search domain to resolve the bare hostnames.

i.e.

        dns zone for lab.example.com on 192.168.1.1

        c1p509 IN A 192.168.2.1

        resolv.conf:

        search lab.example.com

        nameserver 192.168.1.1

Match Making

The ZeroMQ driver implements a matching capability to discover hosts available for communication when sending to a bare topic. This allows broker-less communications.

The MatchMaker is pluggable and may also be used to enforce sending messages through a BROKER.

This driver provides four MatchMaker classes in the Essex release:^[b]

1. MatchMakerBroker - forces all bare topics through a broker. (DEFAULT)
2. MatchMakerTopicScheduler - brokers a ‘get_worker’ request, returning a hostname to create a directed topic.
3. MatchMakerRing - loads a static hash table from a JSON file, cycles hosts per bare topic to create a directed topic.
4. MatchMakerFanoutRing - loads a static hash table from a JSON file, cycles hosts per bare topic. Supports broker-less Fanout messaging.^[c] Normally, acts like MatchMakerRing, but on fanout messages returns an array of directed topics (messages are sent to all destinations).

To set the MatchMaker class, use option ‘rpc_zmq_matchmaker’.

To specify the ring file for MatchMakerRing, use option ‘rpc_zmq_matchmaker_ringfile’.

Message Broker^[d]

A MatchMaker capable of Broker-less fanout should be available by the April release.

A message broker is OPTIONAL, depending on the matchmaker chosen. Some MatchMaking services also depend on a broker, but MatchMaker modules may also reduce or eliminate the requirement for a broker. Simply, the MatchMaker allows the requirement for a broker or other discovery mechanism to be pluggable.

The IP address of this broker is set via option ‘rpc_zmq_broker_ip’.  

Message Receivers

Each machine running Nova, or sending RPC messages, must run the ‘nova-rpc-zmq-receiver’ daemon. This receives replies to call requests and routes responses via IPC to blocked callers.

Thread pool  (OPTIONAL)

Each service will launch threads for incoming requests. These threads are maintained via a pool, the maximum number of threads is limited by rpc_thread_pool_size. The default value in Nova is 1024. (This is a common RPC configuration variable, also applicable to Kombu and Qpid)

Listening Address  (OPTIONAL)^[e]

All services bind to an IP address or ethernet adapter. By default, all services bind to ‘*’, effectively binding to 0.0.0.0.  This may be changed with the option ‘rpc_zmq_bind_address’, which accepts a wildcard, IP address, or ethernet adapter.

Listening Ports (OPTIONAL)

The ZeroMQ driver may consume an unhealthy number of TCP ports. The starting port is configured with 'rpc_zmq_start_port', which defaults to 9500. Expect the driver to consume up to 50 ports, so allow and reserve ports 9500-9550 for ZeroMQ communications.^[f]

MatchMaker Ringfile  (OPTIONAL)^[g]

If deploying the MatchMakerRing, a ringfile is required as specified by the option ‘rpc_zmq_matchmaker_ringfile’.

Format

The MatchMaker ring is a JSON file containing a hash where each key is a base topic and the values are arrays.

i.e.

{

  ‘scheduler’: [ “o1p409”, “o2p409” ],

  ‘network’: [ “o3p409”, “o4p409” ]

}

Topic names

Base topics in the ring file must match the VALUES of the following nova flags:

            'compute_topic',

            'console_topic',

            'scheduler_topic',

            'volume_topic',

            'network_topic',

            'vsa_topic'

The default values of the above flags are, respectfully:

            'compute',

            'console',

            'scheduler',

            'volume',

            'network',

            'vsa'

It is NOT expected that we should need to provide an array of all compute machines unless we deploy nova-volume as it seems neither fanout cast nor a bare topic are used for ‘compute_topic’.  However, the best practice would describe all applicable hosts under the ‘compute’ and ‘volume’ topics (if applicable) within the ring-file. This would be forward-compatible with potential Nova changes and would be safest, overall.

Encryption

Since it is asked frequently, the answer is: No, the driver does NOT encrypt communications. Discussions on how and if to implement encryption are ongoing.

This driver should be deployed in a trusted environment.

Authors

Christopher MacGown (chris@pistoncloud.com)
Duncan McGreggor (duncan@dreamhost.com)
Eric Windisch (eric@cloudscaling.com)
Zed Shaw (zedshaw@zedshaw.com)

Contact

Eric Windisch (eric@cloudscaling.com)

Download

This project is available in three different varients:

• Nova-Essex Integrated (full nova source with ZeroMQ) - branch: stable/essex-zeromq-rpc (source download)
• Nova-Essex Plugin - branch: zeromq-rpc-plugin (source download
• Nova Folsom Integrated (upstream-merge candidate) - branch: zeromq-rpc

You can clone the project with Git by running:

$ git clone git://github.com/ewindisch/nova