MidoNet is an Open Source network virtualization software for IaaS infrastructure.
It decouples your IaaS cloud from your network hardware, creating an intelligent software abstraction layer between your end hosts and your physical network.
This network abstraction layer allows the cloud operator to move what has traditionally been hardware-based network appliances into a software-based multi-tenant virtual domain.
This definition from MidoNet documentation explains what MidoNet is and what MidoNet does.
At this I will post cover my experiences integrating MidoNet with OpenStack.
I used the following configurations:
All servers have CentOS 7.2 installed
OpenStack has been previously installed from RDO packages with multinode Packstack
- x3 NSDB nodes (Casandra and Zookeeper services)
- x2 Gateway Nodes (Midolman Agent)
- x1 OpenStack Controller (MidoNet Cluster)
- x1 OpenStack compute node (Midolman Agent)
NSDB NODES
Disable SElinux
setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/sysconfig/selinux
Install OpenStack Mitaka release repository
sudo yum install -y centos-release-openstack-mitaka
Add Cassandra repository
cat <<EOF>/etc/yum.repos.d/datastax.repo [datastax] name = DataStax Repo for Apache Cassandra baseurl = http://rpm.datastax.com/community enabled = 1 gpgcheck = 1 gpgkey = https://rpm.datastax.com/rpm/repo_key EOF
Add Midonet repository
cat <<EOF>/etc/yum.repos.d/midonet.repo [midonet] name=MidoNet baseurl=http://builds.midonet.org/midonet-5.2/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-openstack-integration] name=MidoNet OpenStack Integration baseurl=http://builds.midonet.org/openstack-mitaka/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-misc] name=MidoNet 3rd Party Tools and Libraries baseurl=http://builds.midonet.org/misc/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key EOF
Clean repo cache and update packages
yum clean all yum update
Zookeeper Configuration
Install Zookeeper, java and dependencies
yum install -y java-1.7.0-openjdk-headless zookeeper zkdump nmap-ncat
Edit zookeeper configuration file
vi /etc/zookeeper/zoo.cfg
Add all NSDB nodes at the configuration file
server.1=nsdb1:2888:3888 server.2=nsdb2:2888:3888 server.3=nsdb3:2888:3888 autopurge.snapRetainCount=10 autopurge.purgeInterval =12
Create zookeeper folder on which zookeeper will store data, change the owner to zookeeper user
mkdir /var/lib/zookeeper/data chown zookeeper:zookeeper /var/lib/zookeeper/data
Create myid file at zookeeper data folder, the ID should match with the NSDB node number, insert that number as follows:
#NSDB1 echo 1 > /var/lib/zookeeper/data/myid #NSDB2 echo 2 > /var/lib/zookeeper/data/myid #NSDB3 echo 3 > /var/lib/zookeeper/data/myid
Create java folder and create a softlink to it
mkdir -p /usr/java/default/bin/ ln -s /usr/lib/jvm/jre-1.7.0-openjdk/bin/java /usr/java/default/bin/java
Start and enable Zookeeper service
systemctl enable zookeeper.service systemctl start zookeeper.service
Test if zookeeper is working locally
echo ruok | nc 127.0.0.1 2181 imok
Test if zookeeper is working at NSDB remote nodes
echo stat | nc nsdb3 2181 Zookeeper version: 3.4.5--1, built on 02/08/2013 12:25 GMT Clients: /192.168.100.172:35306[0](queued=0,recved=1,sent=0) Latency min/avg/max: 0/0/0 Received: 1 Sent: 0 Connections: 1 Outstanding: 0 Zxid: 0x100000000 Mode: follower Node count: 4
Cassandra configuration
Install Java and Cassandra dependencies
yum install -y java-1.8.0-openjdk-headless dsc22
Edit cassandra yaml file
vi /etc/cassandra/conf/cassandra.yaml
Change cluster_name to midonet
Configure seed_provider seeds to match all NSDB nodes
Configure listen_address and rpc_address to match the hostname of the self node
cluster_name: 'midonet'
....
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "nsdb1,nsdb2,nsdb3"
listen_address: nsdb1
rpc_address: nsdb1
Edit cassandra’s init script in order to fix a bug in the init script
vi /etc/init.d/cassandra
Add the next two lines after #Casandra startup
case "$1" in
start)
# Cassandra startup
echo -n "Starting Cassandra: "
mkdir -p /var/run/cassandra #Add this line
chown cassandra:cassandra /var/run/cassandra #Add this line
su $CASSANDRA_OWNR -c "$CASSANDRA_PROG -p $pid_file" > $log_file 2>&1
retval=$?
[ $retval -eq 0 ] && touch $lock_file
echo "OK"
;;
Start and enable Cassandra service
systemctl enable cassandra.service systemctl start cassandra.service
Check if all NSDB nodes join the cluster
nodetool --host 127.0.0.1 status Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.100.172 89.1 KB 256 70.8% 3f1ecedd-8caf-4938-84ad-8614d2134557 rack1 UN 192.168.100.224 67.64 KB 256 60.7% cb36c999-a6e1-4d98-a4dd-d4230b41df08 rack1 UN 192.168.100.195 25.78 KB 256 68.6% 4758bae8-9300-4e57-9a61-5b1107082964 rack1
Configure OpenStack Controller Nodes (On which Neutron Server is running)
Disable SElinux
setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/sysconfig/selinux
Install OpenStack Mitaka release repository
sudo yum install -y centos-release-openstack-mitaka
Add Midonet Repository
cat <<EOF>/etc/yum.repos.d/midonet.repo [midonet] name=MidoNet baseurl=http://builds.midonet.org/midonet-5.2/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-openstack-integration] name=MidoNet OpenStack Integration baseurl=http://builds.midonet.org/openstack-mitaka/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-misc] name=MidoNet 3rd Party Tools and Libraries baseurl=http://builds.midonet.org/misc/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key EOF
Clean repos cache and update the system
yum clean all yum update
Create an OpenStack user for MidoNet, change the password to match your own
# openstack user create --password temporal midonet +----------+----------------------------------+ | Field | Value | +----------+----------------------------------+ | email | None | | enabled | True | | id | ac25c5a77e7c4e4598ccadea89e09969 | | name | midonet | | username | midonet | +----------+----------------------------------+
Add admin role at tenant services to Midonet user
# openstack role add --project services --user midonet admin +-----------+----------------------------------+ | Field | Value | +-----------+----------------------------------+ | domain_id | None | | id | bca2c6e1f3da42b0ba82aee401398a8a | | name | admin | +-----------+----------------------------------+
Create MidoNet service at Keystone
# openstack service create --name midonet --description "MidoNet API Service" midonet +-------------+----------------------------------+ | Field | Value | +-------------+----------------------------------+ | description | MidoNet API Service | | enabled | True | | id | 499059c4a3a040cfb632411408a2be4c | | name | midonet | | type | midonet | +-------------+----------------------------------+
Clean up neutron server
Stop neutron services
openstack-service stop neutron
Remove neutron database and recreate it again
mysql -u root -p DROP DATABASE neutron; Query OK, 157 rows affected (11.50 sec) MariaDB [(none)]> CREATE DATABASE neutron; Query OK, 1 row affected (0.00 sec) MariaDB [(none)]> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'ab4f81b1040a495e'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'%' IDENTIFIED BY 'ab4f81b1040a495e'; Query OK, 0 rows affected (0.00 sec) MariaDB [(none)]> exit Bye
Remove plugin.ini symbolic link to ml2_conf.ini
#rm /etc/neutron/plugin.ini rm: remove symbolic link ‘/etc/neutron/plugin.ini’? y
Remove br-tun tunnel used by neutron in all the nodes
ovs-vsctl del-br br-tun
Install MidoNet packages and remove ml2 package
yum install -y openstack-neutron python-networking-midonet python-neutronclient yum remove openstack-neutron-ml2
Make a backup of neutron configuration file
cp /etc/neutron/neutron.conf neutron.conf.bak
Edit neutron configuration file
vi /etc/neutron/neutron.conf
Most of the options are already configured by our older neutron configuration, change the ones who apply to match this configuration
[DEFAULT] core_plugin = midonet.neutron.plugin_v2.MidonetPluginV2 service_plugins = midonet.neutron.services.l3.l3_midonet.MidonetL3ServicePlugin dhcp_agent_notification = False allow_overlapping_ips = True rpc_backend = rabbit auth_strategy = keystone notify_nova_on_port_status_changes = true notify_nova_on_port_data_changes = true nova_url = http://controller:8774/v2 [database] connection = mysql+pymysql://neutron:ab4f81b1040a495e@controller/neutron [oslo_messaging_rabbit] rabbit_host = controller rabbit_userid = guest rabbit_password = guest [keystone_authtoken] auth_uri = http://controller:5000/v2.0 admin_user=neutron admin_tenant_name=services identity_uri=http://controller:35357 admin_password=d88f0bd060d64c33 [nova] region_name = RegionOne auth_url = http://controller:35357 auth_type = password password = 9ca36d15e4824d93 project_domain_id = default project_name = services tenant_name = services user_domain_id = default username = nova [oslo_concurrency] lock_path = /var/lib/neutron/tmp
At my deployment these are the options I had to change to configure midonet
diff /etc/neutron/neutron.conf neutron.conf.bak 33c33 < core_plugin = midonet.neutron.plugin_v2.MidonetPluginV2 --- > core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin 37c37 < service_plugins = midonet.neutron.services.l3.l3_midonet.MidonetL3ServicePlugin --- > service_plugins =router 120c120 < dhcp_agent_notification = False --- > #dhcp_agent_notification = true 1087c1087,1088 < lock_path = /var/lib/neutron/tmp --- > lock_path = $state_path/lock >
Create midonet plugins folder
mkdir /etc/neutron/plugins/midonet
Create a file called midonet.ini
vi /etc/neutron/plugins/midonet/midonet.ini
Configure midonet.ini file to match your own configuration options
[MIDONET] # MidoNet API URL midonet_uri = http://controller:8181/midonet-api # MidoNet administrative user in Keystone username = midonet password = temporal # MidoNet administrative user's tenant project_id = services
Create a symbolic link from midonet.ini to plugin.ini
ln -s /etc/neutron/plugins/midonet/midonet.ini /etc/neutron/plugin.ini
Sync and populate database tables with Midonet plugin
su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/midonet/midonet.ini upgrade head" neutron su -s /bin/sh -c "neutron-db-manage --subproject networking-midonet upgrade head" neutron
Restart nova api and neutron server services
systemctl restart openstack-nova-api.service systemctl restart neutron-server
Install midonet cluster package
yum install -y midonet-cluster
Configure midonet.conf file
vi /etc/midonet/midonet.conf
Add all NSDB nodes at zookeeper_hosts
[zookeeper] zookeeper_hosts = nsdb1:2181,nsdb2:2181,nsdb3:2181
Configure midonet to make use of NSDB nodes as Zookeeper and cassandra hosts
cat << EOF | mn-conf set -t default
zookeeper {
zookeeper_hosts = "nsdb1:2181,nsdb2:2181,nsdb3:2181"
}
cassandra {
servers = "nsdb1,nsdb2,nsdb3"
}
EOF
Set cassandra replication factor to 3
echo "cassandra.replication_factor : 3" | mn-conf set -t default
Grab your admin token
#egrep ^admin_token /etc/keystone/keystone.conf admin_token = 7b84d89b32c34b71a697eb1a270807ab
Configure Midonet to auth with keystone
cat << EOF | mn-conf set -t default
cluster.auth {
provider_class = "org.midonet.cluster.auth.keystone.KeystoneService"
admin_role = "admin"
keystone.tenant_name = "admin"
keystone.admin_token = "7b84d89b32c34b71a697eb1a270807ab"
keystone.host = controller
keystone.port = 35357
}
EOF
Start and enable midonet cluster service
systemctl enable midonet-cluster.service systemctl start midonet-cluster.service
Install midonet CLI
yum install -y python-midonetclient
Create a file at you home directory with midonet auth info
#vi ~/.midonetrc [cli] api_url = http://controller:8181/midonet-api username = admin password = temporal project_id = admin
Configure Compute nodes
Disable SElinux
setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/sysconfig/selinux
Install OpenStack Mitaka release repository
sudo yum install -y centos-release-openstack-mitaka
Add Midonet repository
cat <<EOF>/etc/yum.repos.d/midonet.repo [midonet] name=MidoNet baseurl=http://builds.midonet.org/midonet-5.2/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-openstack-integration] name=MidoNet OpenStack Integration baseurl=http://builds.midonet.org/openstack-mitaka/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-misc] name=MidoNet 3rd Party Tools and Libraries baseurl=http://builds.midonet.org/misc/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key EOF
Clean repos cache and update the system
yum clean all yum update
Edit qemu.conf
vi /etc/libvirt/qemu.conf
Configure with the following options, by default all these options are commented, you can paste it all wherever you want
user = "root"
group = "root"
cgroup_device_acl = [
"/dev/null", "/dev/full", "/dev/zero",
"/dev/random", "/dev/urandom",
"/dev/ptmx", "/dev/kvm", "/dev/kqemu",
"/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
"/dev/net/tun"
]
Restart libvirtd service
systemctl restart libvirtd.service
Install nova-network package
yum install -y openstack-nova-network
Disable Nova Network service and restart Nova compute service
systemctl disable openstack-nova-network.service systemctl restart openstack-nova-compute.service
Install Midolman agent and java packages
yum install -y java-1.8.0-openjdk-headless midolman
Configure midolman.conf
vi /etc/midolman/midolman.conf
Add all nsdb nodes as zookeeper hosts
[zookeeper] zookeeper_hosts = nsdb1:2181,nsdb2:2181,nsdb3:2181
Configure each compute node with an appropiate flavor located at /etc/midolman/ folder, the have different hardware resources configured, use the one that better match your compute host capabilities
mn-conf template-set -h local -t agent-compute-medium cp /etc/midolman/midolman-env.sh.compute.medium /etc/midolman/midolman-env.sh
Configure metadata, issue the following commands only once, it will automatically populate the configuration to all midonet agents
echo "agent.openstack.metadata.nova_metadata_url : \"http://controller:8775\"" | mn-conf set -t default echo "agent.openstack.metadata.shared_secret : 2bfeb930a90d435d" | mn-conf set -t default echo "agent.openstack.metadata.enabled : true" | mn-conf set -t default
Allow metadata trafic at iptables
iptables -I INPUT 1 -i metadata -j ACCEPT
Remove br-tun bridge
ovs-vsctl del-br br-tun
Start and enable midolman agent service
systemctl enable midolman.service systemctl start midolman.service
Gateway nodes configuration
Disable SElinux
setenforce 0 sed -i 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/sysconfig/selinux
Install OpenStack Mitaka release repository
sudo yum install -y centos-release-openstack-mitaka
Add Midonet repository
cat <<EOF>/etc/yum.repos.d/midonet.repo [midonet] name=MidoNet baseurl=http://builds.midonet.org/midonet-5.2/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-openstack-integration] name=MidoNet OpenStack Integration baseurl=http://builds.midonet.org/openstack-mitaka/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key [midonet-misc] name=MidoNet 3rd Party Tools and Libraries baseurl=http://builds.midonet.org/misc/stable/el7/ enabled=1 gpgcheck=1 gpgkey=https://builds.midonet.org/midorepo.key EOF
Clean repos cache and update the system
yum clean all yum update
Install Midolman agent and java packages
yum install -y java-1.8.0-openjdk-headless midolman
Configure midolman.conf
vi /etc/midolman/midolman.conf
Add all nsdb nodes as zookeeper hosts
[zookeeper] zookeeper_hosts = nsdb1:2181,nsdb2:2181,nsdb3:2181
Configure each gateway node with an appropiate flavor located at /etc/midolman/ folder, the have different hardware resources configured, use the one that better match your gateway host capabilities
mn-conf template-set -h local -t agent-gateway-medium cp /etc/midolman/midolman-env.sh.gateway.medium /etc/midolman/midolman-env.sh
Grab the metadata shared secret located at nova.conf at any of your nova nodes
# egrep ^metadata_proxy_shared_secret /etc/nova/nova.conf metadata_proxy_shared_secret =2bfeb930a90d435d
Allow metadata trafic at iptables
iptables -I INPUT 1 -i metadata -j ACCEPT
Start and enable midolman agent service
systemctl enable midolman.service systemctl start midolman.service
Configure encapsulation and register nodes
Enter to midonet CLI from a controller node
midonet-cli
Create the tunnel zone with VXLAN encapsulation
midonet> tunnel-zone create name tz type vxlan tzone0 midonet> list tunnel-zone tzone tzone0 name tz type vxlan
List hosts discovered by midonet, should be all the nodes where you configured midonet agents(midolman)
midonet> list host host host0 name gateway2 alive true addresses fe80:0:0:0:0:11ff:fe00:1102,169.254.123.1,fe80:0:0:0:0:11ff:fe00:1101,127.0.0.1,0:0:0:0:0:0:0:1,192.168.200.176,fe80:0:0:0:5054:ff:fef9:b2a0,169.254.169.254,fe80:0:0:0:7874:d6ff:fe5b:dea8,192.168.100.227,fe80:0:0:0:5054:ff:fed9:9cc0,fe80:0:0:0:5054:ff:fe4a:e39b,192.168.1.86 flooding-proxy-weight 1 container-weight 1 container-limit no-limit enforce-container-limit false host host1 name gateway1 alive true addresses 169.254.169.254,fe80:0:0:0:3cd1:23ff:feac:a3c2,192.168.1.87,fe80:0:0:0:5054:ff:fea8:da91,127.0.0.1,0:0:0:0:0:0:0:1,fe80:0:0:0:5054:ff:feec:92c1,192.168.200.232,fe80:0:0:0:0:11ff:fe00:1102,169.254.123.1,fe80:0:0:0:0:11ff:fe00:1101,192.168.100.141,fe80:0:0:0:5054:ff:fe20:30fb flooding-proxy-weight 1 container-weight 1 container-limit no-limit enforce-container-limit false host host2 name compute1 alive true addresses fe80:0:0:0:0:11ff:fe00:1101,169.254.123.1,127.0.0.1,0:0:0:0:0:0:0:1,fe80:0:0:0:0:11ff:fe00:1102,192.168.100.173,fe80:0:0:0:5054:ff:fe06:161,fe80:0:0:0:5054:ff:fee3:eb48,192.168.200.251,fe80:0:0:0:5054:ff:fe8d:d22,192.168.1.93,169.254.169.254,fe80:0:0:0:48cb:adff:fe69:f07b flooding-proxy-weight 1 container-weight 1 container-limit no-limit enforce-container-limit false
Register each of the nodes at the VXLAN zone we created before
midonet> tunnel-zone tzone0 add member host host0 address 192.168.100.227 zone tzone0 host host0 address 192.168.100.227 midonet> tunnel-zone tzone0 add member host host1 address 192.168.100.141 zone tzone0 host host1 address 192.168.100.141 midonet> tunnel-zone tzone0 add member host host2 address 192.168.100.173 zone tzone0 host host2 address 192.168.100.173
Create Networks at Neutron
Create an external network
# neutron net-create ext-net --router:external Created a new network: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2016-07-03T14:47:30 | | description | | | id | dc15245e-4391-4514-b489-8976373046a3 | | is_default | False | | name | ext-net | | port_security_enabled | True | | provider:network_type | midonet | | router:external | True | | shared | False | | status | ACTIVE | | subnets | | | tags | | | tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 | | updated_at | 2016-07-03T14:47:30 | +-----------------------+--------------------------------------+
Create an external subnet in the network we created before, use you own IP ranges to match your environment
# neutron subnet-create ext-net 192.168.200.0/24 --name ext-subnet \
--allocation-pool start=192.168.200.225,end=192.168.200.240 \
--disable-dhcp --gateway 192.168.200.1
Created a new subnet:
+-------------------+--------------------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------------------+
| allocation_pools | {"start": "192.168.200.225", "end": "192.168.200.240"} |
| cidr | 192.168.200.0/24 |
| created_at | 2016-07-03T14:50:46 |
| description | |
| dns_nameservers | |
| enable_dhcp | False |
| gateway_ip | 192.168.200.1 |
| host_routes | |
| id | 234dcc9a-2878-4799-b564-bf3a1bd52cad |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | ext-subnet |
| network_id | dc15245e-4391-4514-b489-8976373046a3 |
| subnetpool_id | |
| tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 |
| updated_at | 2016-07-03T14:50:46 |
+-------------------+--------------------------------------------------------+
Create a tenant network and a subnet on it
# neutron net-create demo-net
Created a new network:
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| admin_state_up | True |
| created_at | 2016-07-03T14:51:39 |
| description | |
| id | 075ba699-dc4c-4625-8e0d-0a258a9aeb7d |
| name | demo-net |
| port_security_enabled | True |
| provider:network_type | midonet |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | |
| tags | |
| tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 |
| updated_at | 2016-07-03T14:51:39 |
+-----------------------+--------------------------------------+
# neutron subnet-create demo-net 10.0.20.0/24 --name demo-subnet
Created a new subnet:
+-------------------+----------------------------------------------+
| Field | Value |
+-------------------+----------------------------------------------+
| allocation_pools | {"start": "10.0.20.2", "end": "10.0.20.254"} |
| cidr | 10.0.20.0/24 |
| created_at | 2016-07-03T14:52:32 |
| description | |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.0.20.1 |
| host_routes | |
| id | b299d899-33a3-4bfa-aff4-fda071545bdf |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | demo-subnet |
| network_id | 075ba699-dc4c-4625-8e0d-0a258a9aeb7d |
| subnetpool_id | |
| tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 |
| updated_at | 2016-07-03T14:52:32 |
+-------------------+----------------------------------------------+
Create a tenant router
# neutron router-create router1 Created a new router: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | description | | | external_gateway_info | | | id | 258942d8-9d82-4ebd-b829-c7bdfcc973f5 | | name | router1 | | routes | | | status | ACTIVE | | tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 | +-----------------------+--------------------------------------+
Attach the tenant subnet interface we created before to the router
# neutron router-interface-add router1 demo-subnet Added interface 06c85a56-368c-4d79-bbf0-4bb077f163e5 to router router1.
Set the external network as router gateway
# neutron router-gateway-set router1 ext-net Set gateway for router router1
Now, you can create an instance at tenant network
# nova boot --flavor m1.tiny --image 80871834-29dd-4100-b038-f5f83f126204 --nic net-id=075ba699-dc4c-4625-8e0d-0a258a9aeb7d test1
+--------------------------------------+-----------------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-0000000a |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | q2Cq4kxePSLL |
| config_drive | |
| created | 2016-07-03T15:46:19Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | b8aa46f9-186c-4594-8428-f8dbb16a5e16 |
| image | cirros image (80871834-29dd-4100-b038-f5f83f126204) |
| key_name | - |
| metadata | {} |
| name | test1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 |
| updated | 2016-07-03T15:46:20Z |
| user_id | a2482a91a1f14750b372445d28b07c75 |
+--------------------------------------+-----------------------------------------------------+
# nova list
+--------------------------------------+-------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+---------------------+
| b8aa46f9-186c-4594-8428-f8dbb16a5e16 | test1 | ACTIVE | - | Running | demo-net=10.0.20.11 |
+--------------------------------------+-------+--------+------------+-------------+---------------------+
Ensure the instance gets IP and the metadata service is properly running
# nova console-log test1 ...#Snipp from the output Sending discover... Sending select for 10.0.20.11... Lease of 10.0.20.11 obtained, lease time 86400 cirros-ds 'net' up at 7.92 checking http://169.254.169.254/2009-04-04/instance-id successful after 1/20 tries: up 8.22. iid=i-0000000a ...
If you login to the instance through VNC you should be able to ping another instances
Edge router configuration
Create a new router
# neutron router-create edge-router Created a new router: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | description | | | external_gateway_info | | | id | 5ecadb64-cb0d-4f95-a00e-aa1dd20a2012 | | name | edge-router | | routes | | | status | ACTIVE | | tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 | +-----------------------+--------------------------------------+
Attach the external subnet interface to the router
# neutron router-interface-add edge-router ext-subnet Added interface e37f1986-c6b1-47f4-8268-02b837ceac17 to router edge-router.
Create an uplink network
# neutron net-create uplink-network --tenant_id admin --provider:network_type uplink Created a new network: +-----------------------+--------------------------------------+ | Field | Value | +-----------------------+--------------------------------------+ | admin_state_up | True | | created_at | 2016-07-03T14:57:15 | | description | | | id | 77173ed4-6106-4515-af1c-3683897955f9 | | name | uplink-network | | port_security_enabled | True | | provider:network_type | uplink | | router:external | False | | shared | False | | status | ACTIVE | | subnets | | | tags | | | tenant_id | admin | | updated_at | 2016-07-03T14:57:15 | +-----------------------+--------------------------------------+
Create a subnet in the uplink network
# neutron subnet-create --tenant_id admin --disable-dhcp --name uplink-subnet uplink-network 192.168.1.0/24
Created a new subnet:
+-------------------+--------------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------------+
| allocation_pools | {"start": "192.168.1.2", "end": "192.168.1.254"} |
| cidr | 192.168.1.0/24 |
| created_at | 2016-07-03T15:06:28 |
| description | |
| dns_nameservers | |
| enable_dhcp | False |
| gateway_ip | 192.168.1.1 |
| host_routes | |
| id | 4e98e789-20d3-45fd-a3b5-9bcf02d8a832 |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | uplink-subnet |
| network_id | 77173ed4-6106-4515-af1c-3683897955f9 |
| subnetpool_id | |
| tenant_id | admin |
| updated_at | 2016-07-03T15:06:28 |
+-------------------+--------------------------------------------------+
Create a port for each of the gateway nodes, interface should match with the NIC you want to use for binding the gateway nodes and a IP address for the same purposes
# neutron port-create uplink-network --binding:host_id gateway1 --binding:profile type=dict interface_name=eth1 --fixed-ip ip_address=192.168.1.199
Created a new port:
+-----------------------+--------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | compute1 |
| binding:profile | {"interface_name": "eth1"} |
| binding:vif_details | {"port_filter": true} |
| binding:vif_type | midonet |
| binding:vnic_type | normal |
| created_at | 2016-07-03T15:10:06 |
| description | |
| device_id | |
| device_owner | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "4e98e789-20d3-45fd-a3b5-9bcf02d8a832", "ip_address": "192.168.1.199"} |
| id | 7b4f54dd-2b41-42ba-9c5c-cda4640dc550 |
| mac_address | fa:16:3e:44:a8:c9 |
| name | |
| network_id | 77173ed4-6106-4515-af1c-3683897955f9 |
| port_security_enabled | True |
| security_groups | 0cf3e33e-dbd6-4b42-a0bd-6679b5eed4e1 |
| status | ACTIVE |
| tenant_id | 2f7ee2716b3b4140be57b4a5b26401e3 |
| updated_at | 2016-07-03T15:10:06 |
+-----------------------+--------------------------------------------------------------------------------------+
Attach each of the ports to the edge router
# neutron router-interface-add edge-router port=7b4f54dd-2b41-42ba-9c5c-cda4640dc550 Added interface 7b4f54dd-2b41-42ba-9c5c-cda4640dc550 to router edge-router.
At this point you have to decide if use border routers with BGP enabled or static routes.
Use one of the following links to configure your use case:
https://docs.midonet.org/docs/latest/operations-guide/content/bgp_uplink_configuration.html
https://docs.midonet.org/docs/latest/operations-guide/content/static_setup.html
Issues I faced during configuration of Midonet
Midolman agent don't start:
It was caused because midolman-env.sh file has more RAM configured as the one of my server.
Edit the file to match your server resources
# egrep ^MAX_HEAP_SIZE /etc/midolman/midolman-env.sh MAX_HEAP_SIZE="2048M"
Instances doesn't boot with the following error:
could not open /dev/net/tun: Permission denied
I had to remove br-tun bridges at ovs, if not, ovs locks the device and midolman cannot create the tunnel beetwen compute nodes and gateway nodes.
ovs-vsctl del-br br-tun
This post is my experience integrating Midonet into OpenStack, maybe some things are not correct, if you find any issue, please advise me to fix it.
Regards, Eduardo Gonzalez