1. Instance makes request
There are no special routes, so the request goes out the default gateway. Of course a Quantum router needs to have an interface on the subnet.
cirros$ curl http://169.254.169.254/openstack
2012-08-10
2013-04-04
latest
cirros$ ip -4 address show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
inet 172.17.17.3/24 brd 172.17.17.255 scope global eth0
cirros$ ip route
default via 172.17.17.1 dev eth0
172.17.17.0/24 dev eth0 src 172.17.17.3
cirros$ ip route get 169.254.169.254
169.254.169.254 via 172.17.17.1 dev eth0 src 172.17.17.3
2. Namespace proxy receives request
The default gateway 172.17.17.1 exists within a Quantum router namespace on the network node. The quantum-l3-agent started a namespace proxy in this namespace and added some iptables rules to redirect metadata requests to it.
root@netnode:/# ip netns exec qrouter-7a44de32-3ac0-4f3e-92cc-1a37d8211db8 ip -4 address show dev qr-123f9b7f-43
9: qr-123f9b7f-43: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
inet 172.17.17.1/24 brd 172.17.17.255 scope global qr-123f9b7f-43
root@netnode:/# ip netns exec qrouter-7a44de32-3ac0-4f3e-92cc-1a37d8211db8 iptables-save | grep 9697
-A quantum-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
-A quantum-l3-agent-INPUT -d 127.0.0.1/32 -p tcp -m tcp --dport 9697 -j ACCEPT
root@netnode:/# ip netns exec qrouter-7a44de32-3ac0-4f3e-92cc-1a37d8211db8 netstat -anpt
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9697 0.0.0.0:* LISTEN 11937/python
root@netnode:/# ps -f --pid 11937 | fold -s -w 82
UID PID PPID C STIME TTY TIME CMD
root 11937 1 0 08:43 ? 00:00:00 python
/usr/bin/quantum-ns-metadata-proxy
--pid_file=/var/lib/quantum/external/pids/7a44de32-3ac0-4f3e-92cc-1a37d8211db8.pid
--router_id=7a44de32-3ac0-4f3e-92cc-1a37d8211db8 --state_path=/var/lib/quantum
--debug
--log-file=quantum-ns-metadata-proxy7a44de32-3ac0-4f3e-92cc-1a37d8211db8.log
--log-dir=/var/log/quantum
The nameserver proxy adds two HTTP headers to the request:
- X-Forwarded-For: with the instance's IP address
- X-Quantum-Router-ID: with the uuid of the Quantum router
and proxies it to a Unix domain socket with name /var/lib/quantum/metadata_proxy.
3. Metadata agent receives request and queries the Quantum service
The metadata agent listens on this Unix socket. It is a normal Linux service that runs in the main operating system IP namespace, and so it is able to reach the Quantum and Nova metadata services. Its configuration file has all the information required to do so.
root@netnode:/# netstat -lxp | grep metadata
unix 2 [ ACC ] STREAM LISTENING 123940 11254/python /var/lib/quantum/metadata_proxy
root@netnode:/# lsof /var/lib/quantum/metadata_proxy
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 11254 quantum 4u unix 0xffff88001b6d4000 0t0 123940 /var/lib/quantum/metadata_proxy
root@netnode:/# ps -f --pid 11254 | fold -w 80 -s
UID PID PPID C STIME TTY TIME CMD
quantum 11254 1 0 08:43 ? 00:00:01 python
/usr/bin/quantum-metadata-agent --config-file=/etc/quantum/quantum.conf
--config-file=/etc/quantum/metadata_agent.ini
--log-file=/var/log/quantum/metadata-agent.log
root@netnode:/# grep -v '^#\|^\s*$' /etc/quantum/metadata_agent.ini
[DEFAULT]
debug = True
auth_url = http://10.0.10.10:35357/v2.0
auth_region = RegionOne
admin_tenant_name = service
admin_user = quantum
admin_password = quantum
nova_metadata_ip = 10.0.10.10
nova_metadata_port = 8775
metadata_proxy_shared_secret = seCr3t
It reads the X-Forwarded-For and X-Quantum-Router-ID headers in the request and queries the Quantum service to find the ID of the instance that created the request.
4. Metadata agent proxies request to Nova metadata service
It then adds these headers:- X-Instance-ID: the instance ID returned from Quantum
- X-Instance-ID-Signature: instance ID signed with the shared-secret
- X-Forwarded-For: the instance's IP address
5. Nova metadata service receives request
The metadata service was started by nova-api. The handler checks the X-Instance-ID-Signature with the shared key, looks up the data and returns the response which travels back via the two proxies to the instance.
root@controller:/# grep metadata /etc/nova/nova.conf
service_quantum_metadata_proxy = True
quantum_metadata_proxy_shared_secret = seCr3t
enabled_apis=osapi_compute,metadata
root@controller:/# netstat -pant | grep 8775
tcp 0 0 0.0.0.0:8775 0.0.0.0:* LISTEN 1038/python
root@controller:/# ps -f --pid 1038
UID PID PPID C STIME TTY TIME CMD
nova 1038 1 0 08:22 ? 00:00:01 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
root@controller:/# grep metadata /var/log/nova/nova-api.log
...
2013-06-11 08:23:01.183 1038 DEBUG nova.wsgi [-] Loading app metadata from /etc/nova/api-paste.ini load_app /usr/lib/python2.7/dist-packages/nova/wsgi.py:481
2013-06-11 08:23:01.199 1038 INFO nova.wsgi [-] metadata listening on 0.0.0.0:8775
2013-06-11 08:23:01.460 1644 INFO nova.metadata.wsgi.server [-] (1644) wsgi starting up on http://0.0.0.0:8775/
...
2013-06-11 09:28:44.665 1644 INFO nova.metadata.wsgi.server [-] (1644) accepted ('10.0.10.11', 57614)
2013-06-11 09:28:47.966 1644 INFO nova.metadata.wsgi.server [-] 172.17.17.3,10.0.10.11 "GET /openstack HTTP/1.1" status: 200 len: 144 time: 3.2876389
...
Troubleshooting
The l3-agent starts the namespace proxy, so check the l3_agent.log if it is not running. Check quantum-ns-metadata-proxy<router-uuid>.log and metadata-agent.log for errors - they should have a log entry for each request when log=debug in l3_agent.ini, metadata_agent.ini and quantum.conf. Finally there should be a message in nova-api.log for each request - see above.
