There is a common problem where the instance has booted and shows its state to be active, but cannot be pinged or reached with ssh even though security groups allow this traffic. There are many reasons why this might happen, so this post just describes how nova-compute, quantum-openvswitch-plugin-agent and quantum-dhcp-agent setup the connectivity between the instance and dnsmasq that is required for the DHCP packets. It shows what Linux commands can be used and what log files to check.
First verify that the instance did not get an IP address by checking its console log. This can be done from the web dashboard or the nova command and you should see something like this if there was a problem and you are using the cirros test image:
$ nova console-log <instance name or uuid>
...
udhcpc (v1.17.2) started
Sending discover...
Sending discover...
Sending discover...
No lease, forking to background
...
cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id
wget: can't connect to remote host (169.254.169.254): Network is unreachable
Next you need to find out where the problem is. This is where you need a picture of the all the components involved and how they work together. This will vary depending on which of the following are being used:
- Hypervisor (KVM, QEMU, etc)
- VIF driver (LibvirtOpenVswitchVirtualPortDriver , LibvirtOpenVswitchDriver, LibvirtHybridOVSBridgeDriver etc)
- Security groups (Nova, Quantum, none)
- Quantum plugin (OVS, Linux bridge etc)
- If OVS plugin, how virtual networks span nodes (GRE, VLAN, flat)
- dhcp_driver (Dnsmasq, etc)
This post will provide a working example of one particular configuration: a multinode Grizzy on Ubuntu 12.10, libvirt/QEMU, LibvirtHybridOVSBridgeDriver vif driver, Quantum security groups, Open vSwitch Quantum plugin using GRE, dnsmasq, IP namespaces enabled as setup by the script in this post.
The instance was booted with a nic attached to Quantum network with name net1:
vagrant@controller:/$ quantum net-show net1
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 6b71dbb8-e91c-47f0-92c4-47882007115d |
| name | net1 |
| provider:network_type | gre |
| provider:physical_network | |
| provider:segmentation_id | 1 |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | ad970f3f-4ceb-4565-b897-1cd0fe34cd5b |
| tenant_id | 208914ef96eb492fa3cd066bdcb7bb2b |
+---------------------------+--------------------------------------+
vagrant@controller:/$ quantum subnet-show ad970f3f-4ceb-4565-b897-1cd0fe34cd5b
+------------------+--------------------------------------------------+
| Field | Value |
+------------------+--------------------------------------------------+
| allocation_pools | {"start": "172.241.0.2", "end": "172.241.0.254"} |
| cidr | 172.241.0.0/24 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 172.241.0.1 |
| host_routes | |
| id | ad970f3f-4ceb-4565-b897-1cd0fe34cd5b |
| ip_version | 4 |
| name | |
| network_id | 6b71dbb8-e91c-47f0-92c4-47882007115d |
| tenant_id | 208914ef96eb492fa3cd066bdcb7bb2b |
+------------------+--------------------------------------------------+
vagrant@controller:/$ nova boot --flavor micro --image cirros-030-x86_64 \
--nic net-id=6b71dbb8-e91c-47f0-92c4-47882007115d \
--security-groups test-vms test-instance1
...
vagrant@controller:/$ nova list
+--------------------------------------+----------------+--------+------------------+
| ID | Name | Status | Networks |
+--------------------------------------+----------------+--------+------------------+
| 44e362ba-e8a4-4bae-b0ea-5477666632c9 | test-instance1 | ACTIVE | net1=172.241.0.2 |
+--------------------------------------+----------------+--------+------------------+
vagrant@controller:/$ quantum port-list --device_id=44e362ba-e8a4-4bae-b0ea-5477666632c9
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
| 1408934f-93f9-415c-9c3a-3528f1a1f445 | | fa:16:3e:1e:63:66 | {"subnet_id": "ad970f3f-4ceb-4565-b897-1cd0fe34cd5b", "ip_address": "172.241.0.2"} |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
(mistake: I though 172.241.0.0/24 was in RFC1918 - but its not)
After a successful boot, there are four new network devices on the compute node. The first 11 characters of the Quantum port-id are used for naming the network devices associated with it:
root@compute1:/# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:88:0c:a6 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:2a:5c:70 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:bc:f6:83 brd ff:ff:ff:ff:ff:ff
5: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
link/ether e6:1f:7f:53:26:45 brd ff:ff:ff:ff:ff:ff
8: br-tun: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
link/ether 4a:98:b5:e7:ff:41 brd ff:ff:ff:ff:ff:ff
13: qbr1408934f-93: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 22:e3:98:fb:50:8a brd ff:ff:ff:ff:ff:ff
14: qvo1408934f-93: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 0e:ff:72:e2:90:de brd ff:ff:ff:ff:ff:ff
15: qvb1408934f-93: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbr1408934f-93 state UP qlen 1000
link/ether 22:e3:98:fb:50:8a brd ff:ff:ff:ff:ff:ff
16: tap1408934f-93: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master qbr1408934f-93 state UNKNOWN qlen 500
link/ether fe:16:3e:1e:63:66 brd ff:ff:ff:ff:ff:ff
Note: 'ip -s link' will show statistics which are useful for troubleshooting. Also note OVS bridges br-int and br-tun are DOWN - this is ok and they should not be put UP.
The LibvirtHybridOVSBridgeDriver plug method running under nova-compute created a Linux bridge named qbr1408934f-93 and plugged the instance's vif (tap1408934f-93) into it. It also created a veth pair qvb1408934f-93 and qvo1408934f-93 to connect the Linux bridge to the OVS bridge br-int. It does all this because OVS and iptables are not compatible when rules are applied directly on vif ports - see limitations at the end of this. Check /var/log/nova/nova-compute.log if there are problems here.
root@compute1:/# brctl show
bridge name bridge id STP enabled interfaces
br-int 0000.e61f7f532645 no qvo1408934f-93
br-tun 0000.4a98b5e7ff41 no
qbr1408934f-93 8000.22e398fb508a no qvb1408934f-93
tap1408934f-93
Iptables
The quantum-plugin-openvswitch-agent configures these on the compute-node to implement the Quantum security groups. The ovs_quantum_plugin.ini has:
And these rules were added:
[SECURITYGROUP]
# Firewall driver for realizing quantum security group function
firewall_driver = quantum.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
root@compute1:/# iptables-save
# Generated by iptables-save v1.4.12 on Wed May 8 14:45:07 2013
*nat
:PREROUTING ACCEPT [19:5732]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [96:5760]
:POSTROUTING ACCEPT [115:11492]
:quantum-openvswi-OUTPUT - [0:0]
:quantum-openvswi-POSTROUTING - [0:0]
:quantum-openvswi-PREROUTING - [0:0]
:quantum-openvswi-float-snat - [0:0]
:quantum-openvswi-snat - [0:0]
:quantum-postrouting-bottom - [0:0]
-A PREROUTING -j quantum-openvswi-PREROUTING
-A OUTPUT -j quantum-openvswi-OUTPUT
-A POSTROUTING -j quantum-openvswi-POSTROUTING
-A POSTROUTING -j quantum-postrouting-bottom
-A quantum-openvswi-snat -j quantum-openvswi-float-snat
-A quantum-postrouting-bottom -j quantum-openvswi-snat
COMMIT
# Completed on Wed May 8 14:45:07 2013
# Generated by iptables-save v1.4.12 on Wed May 8 14:45:07 2013
*filter
:INPUT ACCEPT [5938:1326085]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [7179:1276153]
:quantum-filter-top - [0:0]
:quantum-openvswi-FORWARD - [0:0]
:quantum-openvswi-INPUT - [0:0]
:quantum-openvswi-OUTPUT - [0:0]
:quantum-openvswi-i1408934f-9 - [0:0]
:quantum-openvswi-local - [0:0]
:quantum-openvswi-o1408934f-9 - [0:0]
:quantum-openvswi-sg-chain - [0:0]
:quantum-openvswi-sg-fallback - [0:0]
-A INPUT -j quantum-openvswi-INPUT
-A INPUT -p gre -j ACCEPT
-A FORWARD -j quantum-filter-top
-A FORWARD -j quantum-openvswi-FORWARD
-A OUTPUT -j quantum-filter-top
-A OUTPUT -j quantum-openvswi-OUTPUT
-A quantum-filter-top -j quantum-openvswi-local
-A quantum-openvswi-FORWARD -m physdev --physdev-out tap1408934f-93 --physdev-is-bridged -j quantum-openvswi-sg-chain
-A quantum-openvswi-FORWARD -m physdev --physdev-in tap1408934f-93 --physdev-is-bridged -j quantum-openvswi-sg-chain
-A quantum-openvswi-INPUT -m physdev --physdev-in tap1408934f-93 --physdev-is-bridged -j quantum-openvswi-o1408934f-9
-A quantum-openvswi-i1408934f-9 -m state --state INVALID -j DROP
-A quantum-openvswi-i1408934f-9 -m state --state RELATED,ESTABLISHED -j RETURN
-A quantum-openvswi-i1408934f-9 -p icmp -j RETURN
-A quantum-openvswi-i1408934f-9 -p tcp -m tcp --dport 22 -j RETURN
-A quantum-openvswi-i1408934f-9 -s 172.241.0.3/32 -p udp -m udp --sport 67 --dport 68 -j RETURN
-A quantum-openvswi-i1408934f-9 -j quantum-openvswi-sg-fallback
-A quantum-openvswi-o1408934f-9 -m mac ! --mac-source FA:16:3E:1E:63:66 -j DROP
-A quantum-openvswi-o1408934f-9 -p udp -m udp --sport 68 --dport 67 -j RETURN
-A quantum-openvswi-o1408934f-9 ! -s 172.241.0.2/32 -j DROP
-A quantum-openvswi-o1408934f-9 -p udp -m udp --sport 67 --dport 68 -j DROP
-A quantum-openvswi-o1408934f-9 -m state --state INVALID -j DROP
-A quantum-openvswi-o1408934f-9 -m state --state RELATED,ESTABLISHED -j RETURN
-A quantum-openvswi-o1408934f-9 -j RETURN
-A quantum-openvswi-o1408934f-9 -j quantum-openvswi-sg-fallback
-A quantum-openvswi-sg-chain -m physdev --physdev-out tap1408934f-93 --physdev-is-bridged -j quantum-openvswi-i1408934f-9
-A quantum-openvswi-sg-chain -m physdev --physdev-in tap1408934f-93 --physdev-is-bridged -j quantum-openvswi-o1408934f-9
-A quantum-openvswi-sg-chain -j ACCEPT
-A quantum-openvswi-sg-fallback -j DROP
COMMIT
So each port gets its own input and output chains (ending with i1408934f-9 and o1408934f-9). For DHCP the output chain has:
-A quantum-openvswi-o1408934f-9 -p udp -m udp --sport 68 --dport 67 -j RETURN
and the input chain has this rule to allow packets from the IP address running the DHCP server (172.241.0.3):
-A quantum-openvswi-i1408934f-9 -s 172.241.0.3/32 -p udp -m udp --sport 67 --dport 68 -j RETURN
There is also a rule on the output chain to prevent the instance from running a dhcp server on the segment, and rules to prevent the instance spoofing MAC and IP addresses.
If you think there is a problem with iptables, then you can use 'iptables-save -c' to show counters and see which chains are being hit, something like this:
iptables-save -c >/tmp/sample1
# make the instance do a dhcp request
iptables-save -c >/tmp/sample2
diff /tmp/sample1 /tmp/sample2
Open vSwitch on the compute-node
There are two OVS bridges on the compute-node. Br-int is for connecting instances (indirectly with the hybrid vif driver) and br-tun for connecting physical nodes when using GRE mode.
root@compute1:/# ovs-vsctl show
3f7c02b4-00d2-4b4e-871d-eafdc22eb67b
Bridge br-int
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port br-int
Interface br-int
type: internal
Port "qvo1408934f-93"
tag: 2
Interface "qvo1408934f-93"
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "gre-1"
Interface "gre-1"
type: gre
options: {in_key=flow, out_key=flow, remote_ip="10.0.20.11"}
ovs_version: "1.4.0+build0"
Here the quantum-plugin-openvswitch-agent has put port qvo1408934f-93 into VLAN 2 (tag 2) on br-int. Ports on different Quantum networks will be put in different VLANs for isolation. If the port is not tagged then there is something wrong with the ovs agent - check /var/log/quantum/openvswitch-agent.log. Note: these VLANs are internal to OVS and should not be confused with "real" VLANs from physical switches.
Br-int is connected to br-tun with a patch cable. Br-tun is for inter-connecting data-forwarding nodes (network and compute) using GRE tunnels. Each node will be connected to every other node by a GRE tunnel which results in a full mesh topology. Also br-tun uses MAC flooding and learning to learn which ports the destination MAC addresses of instances are on. Special flows are required to prevent loops as this topology is not loop-free when there are three or more nodes.
root@compute1:/# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=6845.074s, table=0, n_packets=55, n_bytes=4950, priority=3,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=mod_vlan_vid:2,output:1
cookie=0x0, duration=6845.411s, table=0, n_packets=772, n_bytes=172410, priority=4,in_port=1,dl_vlan=2 actions=set_tunnel:0x1,NORMAL
cookie=0x0, duration=6844.75s, table=0, n_packets=182, n_bytes=46000, priority=3,tun_id=0x1,dl_dst=fa:16:3e:1e:63:66 actions=mod_vlan_vid:2,NORMAL
cookie=0x0, duration=10496.078s, table=0, n_packets=13, n_bytes=974, priority=1 actions=drop
Each flow is made up of statistics, a match critera and an action. To map the port numbers to names:
root@compute1:/# ovs-ofctl show br-tun | grep '^ [0-9]'
1(patch-int): addr:22:29:a4:f9:45:8b
2(gre-1): addr:ce:d9:0a:15:d8:2c
The first flow accepts incoming multicasts (including broadcasts) with GRE key 0x1 (this is the provider:segmentation_id of net1) and tags it for VLAN 2 and send it to br-int via the patch port. There will be one of these flows for each VLAN on br-int.
The second flow is for outgoing traffic from br-int VLAN 2 - it sets the GRE key to 0x1 and gives it the NORMAL action. NORMAL means MAC-learning - so if br-tun does not know which port the destination MAC is on, then it will flood all ports and learn from the source address in reply packets. There will be one of these flows for each VLAN on br-int.
The third flow accepts incoming unicasts with the destination MAC of the instance. There will one of these for each instance running on the compute node.
Any traffic that does not match the first three flows will be dropped by the fourth which has the lowest priority.
Open vSwitch on the network node
This is very similar to the compute-node. The quantum-dhcp-agent created a tap interface plugged into br-int.
root@netnode:/# ovs-vsctl show
20f4084e-0758-41bd-8dfe-5f54883c28e0
Bridge br-ex
Port "qg-dff8f187-92"
Interface "qg-dff8f187-92"
type: internal
Port br-ex
Interface br-ex
type: internal
Port "eth3"
Interface "eth3"
Bridge br-int
Port "tap9a41d8fa-a7"
tag: 1
Interface "tap9a41d8fa-a7"
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port br-int
Interface br-int
type: internal
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "gre-2"
Interface "gre-2"
type: gre
options: {in_key=flow, out_key=flow, remote_ip="10.0.20.12"}
ovs_version: "1.4.0+build0"
root@netnode:/# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=16029.492s, table=0, n_packets=1465, n_bytes=382914, priority=3,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=mod_vlan_vid:1,output:1
cookie=0x0, duration=16029.712s, table=0, n_packets=534, n_bytes=115374, priority=4,in_port=1,dl_vlan=1 actions=set_tunnel:0x1,NORMAL
cookie=0x0, duration=2.16s, table=0, n_packets=0, n_bytes=0, priority=3,tun_id=0x1,dl_dst=fa:16:3e:27:d6:30 actions=mod_vlan_vid:1,NORMAL
cookie=0x0, duration=17440.954s, table=0, n_packets=26, n_bytes=2720, priority=1 actions=drop
The tap device tap9a41d8fa-a7 on br-int is for the DHCP server. Its type is internal which means you can put an IP address on it. Note net1 is given VLAN 1 here - these tags are provisioned locally and may vary for the same network from node to node. The flows in br-tun ensure the translation to and from the segmentation-id in the GRE key field to the local VLAN is correct.
The DHCP agent
The DHCP agent is configured to use OVS and dnsmasq:
root@netnode:/# grep -v '^#\|^\s*$' /etc/quantum/dhcp_agent.ini
[DEFAULT]
interface_driver = quantum.agent.linux.interface.OVSInterfaceDriver
dhcp_driver = quantum.agent.linux.dhcp.Dnsmasq
Update: add root_helper = sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf to workaround this bug.
The agent uses another Quantum port on net1:
vagrant@controller:/$ quantum port-show 9a41d8fa-a745-4411-b9f8-daa182f38527
+----------------------+------------------------------------------------------------------------------------+
| Field | Value |
+----------------------+------------------------------------------------------------------------------------+
| admin_state_up | True |
| binding:capabilities | {"port_filter": true} |
| binding:vif_type | ovs |
| device_id | dhcpa01fb276-fc1a-5a98-90b4-3413b63a3276-6b71dbb8-e91c-47f0-92c4-47882007115d |
| device_owner | network:dhcp |
| fixed_ips | {"subnet_id": "ad970f3f-4ceb-4565-b897-1cd0fe34cd5b", "ip_address": "172.241.0.3"} |
| id | 9a41d8fa-a745-4411-b9f8-daa182f38527 |
| mac_address | fa:16:3e:27:d6:30 |
| name | |
| network_id | 6b71dbb8-e91c-47f0-92c4-47882007115d |
| security_groups | |
| status | ACTIVE |
| tenant_id | 208914ef96eb492fa3cd066bdcb7bb2b |
+----------------------+------------------------------------------------------------------------------------+
The tap device for the DHCP server (dnsmasq) exists within an IP namespace with a name that starts with qdhcp- and ends with the Quantum network-id:
root@netnode:/# ip netns | grep dhcp
qdhcp-6b71dbb8-e91c-47f0-92c4-47882007115d
root@netnode:/# ip netns exec qdhcp-6b71dbb8-e91c-47f0-92c4-47882007115d ip address
13: tap9a41d8fa-a7: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:27:d6:30 brd ff:ff:ff:ff:ff:ff
inet 172.241.0.3/24 brd 172.241.0.255 scope global tap9a41d8fa-a7
inet6 fe80::f816:3eff:fe27:d630/64 scope link
valid_lft forever preferred_lft forever
14: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
And dnsmasq runs within this IP namespace using this interface:
root@netnode:/# ip netns exec qdhcp-6b71dbb8-e91c-47f0-92c4-47882007115d netstat -pan
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 172.241.0.3:53 0.0.0.0:* LISTEN 8920/dnsmasq
tcp6 0 0 fe80::f816:3eff:fe27:53 :::* LISTEN 8920/dnsmasq
udp 0 0 172.241.0.3:53 0.0.0.0:* 8920/dnsmasq
udp 0 0 0.0.0.0:67 0.0.0.0:* 8920/dnsmasq
udp6 0 0 fe80::f816:3eff:fe27:53 :::* 8920/dnsmasq
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ] DGRAM 3170559 8920/dnsmasq
root@netnode:/# ps -ef | grep 8920
nobody 8920 1 0 19:15 ? 00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap9a41d8fa-a7 --except-interface=lo --pid-file=/var/lib/quantum/dhcp/6b71dbb8-e91c-47f0-92c4-47882007115d/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/6b71dbb8-e91c-47f0-92c4-47882007115d/host --dhcp-optsfile=/var/lib/quantum/dhcp/6b71dbb8-e91c-47f0-92c4-47882007115d/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,172.241.0.0,static,120s --conf-file= --domain=openstacklocal
root 8921 8920 0 19:15 ? 00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap9a41d8fa-a7 --except-interface=lo --pid-file=/var/lib/quantum/dhcp/6b71dbb8-e91c-47f0-92c4-47882007115d/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/6b71dbb8-e91c-47f0-92c4-47882007115d/host --dhcp-optsfile=/var/lib/quantum/dhcp/6b71dbb8-e91c-47f0-92c4-47882007115d/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,172.241.0.0,static,120s --conf-file= --domain=openstacklocal
root 25444 1618 0 19:33 pts/2 00:00:00 grep --color=auto 8920
You should see two copies of the dnsmasq process here. Wait minute and re-run netstat or ps and check that the pid stays the same - there is a bug where dnsmasq continuously respawns sometimes.
The lease time is 120 seconds and if you leave tcpdump running you will see the instance re-up about once per minute (lease time / 2).
The hosts file should contain the MAC/IP pairs for the instances:
root@netnode:/# cat /var/lib/quantum/dhcp/6b71dbb8-e91c-47f0-92c4-47882007115d/host
fa:16:3e:1e:63:66,172-241-0-2.openstacklocal,172.241.0.2
fa:16:3e:27:d6:30,172-241-0-3.openstacklocal,172.241.0.3
Dnsmasq logs to /var/log/syslog in this Ubuntu installation.
May 8 19:15:54 netnode dnsmasq[8920]: started, version 2.59 cachesize 150
May 8 19:15:54 netnode dnsmasq[8920]: compile time options: IPv6 GNU-getopt DBus i18n DHCP TFTP conntrack IDN
May 8 19:15:54 netnode dnsmasq[8920]: warning: no upstream servers configured
May 8 19:15:54 netnode dnsmasq-dhcp[8920]: DHCP, static leases only on 172.241.0.0, lease time 2m
May 8 19:15:54 netnode dnsmasq[8920]: cleared cache
May 8 19:15:54 netnode dnsmasq-dhcp[8920]: read /var/lib/quantum/dhcp/6b71dbb8-e91c-47f0-92c4-47882007115d/host
May 8 19:15:54 netnode dnsmasq-dhcp[8920]: read /var/lib/quantum/dhcp/6b71dbb8-e91c-47f0-92c4-47882007115d/opts
May 8 19:17:23 netnode dnsmasq-dhcp[8920]: DHCPDISCOVER(tap9a41d8fa-a7) 172.241.0.2 fa:16:3e:1e:63:66
May 8 19:17:23 netnode dnsmasq-dhcp[8920]: DHCPOFFER(tap9a41d8fa-a7) 172.241.0.2 fa:16:3e:1e:63:66
May 8 19:17:23 netnode dnsmasq-dhcp[8920]: DHCPREQUEST(tap9a41d8fa-a7) 172.241.0.2 fa:16:3e:1e:63:66
May 8 19:17:23 netnode dnsmasq-dhcp[8920]: DHCPACK(tap9a41d8fa-a7) 172.241.0.2 fa:16:3e:1e:63:66 172-241-0-2
The agent communicates with the main Quantum service over rpc. When new ports are added (or removed), the agent updates the host and opts files and sends the dnsmasq the HUP signal which causes it to reload them. This can be be seen in /var/log/quantum/dhcp-agent.log and the syslog.
Debugging with tcpdump
If the instance does not have an IP yet, you can still access it using vnc from the web dashboard. If you are using the cirros test image, do 'sudo udhcpc' and a dhcp request will be sent about once per second until a reply is received.Then start tcpdump somewhere in the middle and look for the DHCP request and reply packets:
root@compute1:/# tcpdump -n -i eth2 | grep -v IP6
...
14:30:53.389538 IP 10.0.20.12 > 10.0.20.11: GREv0, key=0x1, length 334: IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from fa:16:3e:1e:63:66, length 280
14:30:53.409163 IP 10.0.20.11 > 10.0.20.12: GREv0, key=0x1, length 375: IP 172.241.0.3.67 > 172.241.0.2.68: BOOTP/DHCP, Reply, length 321
Then depending on what you see, move left or right on diagram tcpdumping on devices until you find where the request or reply packets gets lost. Then check the logs for the agent that should have setup that stage. It might be necessary to restart that agent.
Once the instance gets an IP, you will need to associate a floating IP in order to access it from outside. Or you can use the dhcp namespace on the network node:
root@netnode:/# ip netns exec qdhcp-6b71dbb8-e91c-47f0-92c4-47882007115d ssh cirros@172.241.0.2
