1. No category

Secure LXC networking
CEO of 1H Ltd.
CTO of GetClouder.com
Marian HackMan Marinov
<mm@1h.com>
Who am I?
System Administrator since 1998
CEO of 1H Ltd.
CTO of GetClouder Ltd.
Head of DevOps for Siteground.com
Organizer of OpenFest, BG Perl workshops and
others
➢ This year I helped with the organization of
YAPC europe and EuroBSDcon in Sofia
➢ In my spare time I teach Linux System
Administration and Network Security courses in
Sofia University
➢ For the past year I'm playing mainly with
containers!
➢
➢
➢
➢
➢
We don't really need networking...
MAC addresses
➢ Keep a central DB with all MAC addresses to
prevent collisions
➢ Use a reliable way to generate MAC addresses
➢ Keep in mind:
➢local or global
➢unicast or multicast
generate MAC address in bash
function gen_mac() {
mac_vars=(0 1 2 3 4 5 6 7 8 9 a b c d e f)
mac_base='52:00:01:'
ret=''
for i in {1..6}; do
n=$RANDOM
let 'n %= 16'
ret="${ret}${mac_vars[$n]}"
if [ $i -eq 2 ] || [ $i -eq 4 ]; then
ret="${ret}:"
fi
done
echo "${mac_base}${ret}"
}
generate mac address in PLPGSQL
CREATE OR REPLACE FUNCTION generate_mac() RETURNS text
LANGUAGE plpgsql
AS $$DECLARE
mac TEXT;
a CHAR;
count INTEGER;
BEGIN
mac='52:00:01:';
FOR count IN 1..6 LOOP
SELECT substring('0123456789abcdef' FROM (random()*16)::int + 1 FOR 1) INTO a;
-- This fixes an issue, where the above SELECT returns NULL or empty string
-- If for some reason we concatenate with a NULL string, the result will be NULL string
WHILE a IS NULL OR a = '' LOOP
SELECT substring('0123456789abcdef' FROM (random()*16)::int + 1 FOR 1)
INTO a;
END LOOP;
mac = mac || a;
IF count = 2 OR count = 4 THEN
mac = mac || ':';
END IF;
END LOOP;
RETURN mac;
END;$$;
generate MAC address in Python
#/usr/bin/python
import random
mac = [random.choice(range(256)) for i in range(6)]
mac[0] |= 0x02
mac[0] &= 0xfe
print ':'.join('%02x' % m for m in mac)
Types of LXC networking
➢ none
➢ empty
➢ macvlan
➢ macvtap (did not have time to test it)
➢ veth (linux or ovs bridge)
➢ vlan
➢ physical
➢ VPN device(haven't tried that either)
None
lxc.network.type = none
lxc.network.hwaddr = 00:16:3a:61:45:a6
lxc.network.flags = up
Empty
lxc.network.type = empty
lxc.network.hwaddr = 00:16:3a:61:45:a6
lxc.network.flags = up
VETH
lxc.network.type = veth
lxc.network.veth.pair = vethc3070
lxc.network.flags = up
lxc.network.name = eth0
lxc.network.ipv4 = X.X.X.X/24
lxc.network.ipv4.gateway = X.X.X.1
lxc.network.hwaddr = 00:16:3e:28:73:b3
VETH
lxc.network.veth.pair = vethc3070
11: vethD6YPJ1:
<BROADCAST,MULTICAST,PROMISC,UP,LOWE
R_UP> mtu 1500 qdisc pfifo_fast master lxcbr0
state UP qlen 1000
link/ether f2:0:32:02:55:2f brd ff:ff:ff:ff:ff:ff
valid_lft forever preferred_lft forever
MACVLAN
lxc.network.type = macvlan
lxc.network.macvlan.mode = bridge
lxc.network.flags = up
lxc.network.link = lxcbr1
lxc.network.name = eth0
lxc.network.ipv4 = X.X.X.X/24
lxc.network.ipv4.gateway = X.X.X.1
lxc.network.hwaddr = 00:16:3e:28:73:b3
MACVLAN
➢ If you want to manually setup the networking
ip link add link PARENT [NAME] type macvlan [address MAC]
➢ Auto generated MAC adresses
# ip link add link eth0 lxc0 type macvlan
➢ Manually assigned
# ip link add link eth0 lxc1 type macvlan address f0:de:f1:81:0a:2a
➢ Additional parameter: mode
➢ macvlan mode { private | vepa | bridge | passthru }
MACVLAN
➢ private (filter all incoming packets)
➢ bridge (all packets on the same iface can be seen from all
vlans)
➢ pasthru (requires enabled STP)
➢ VEPA (Virtual Ethernet Port Aggregator)
MACVLAN
➢ Edge Virtual Bridging EVB
➢ Top-of-Rack (ToR)
➢ End-of-Row (EoR)
➢ Virtual Ethernet Bridge (VEB)
➢ Linux bridge
➢ OpenVswitch
➢ Virtual Ethernet Port Aggregator (VEPA)
➢ used for EVB
➢ VEPA 802.1Qab - HP, IBM, Brocade, Juniper
➢ Standard mode
➢ Multi-channel VEPA (Q-in-Q)
➢ VN-Tag 802.1Qbh - Cisco
VLAN
lxc.network.type = vlan
lxc.network.vlan.id = 10
lxc.network.flags = up
lxc.network.link = eth0
lxc.network.name = eth0
lxc.network.ipv4 = X.X.X.X/24
lxc.network.ipv4.gateway = X.X.X.1
lxc.network.hwaddr = 00:16:3e:28:73:b3
VLAN
# vconfig add eth0 10
# ip link add link eth0 vlan10 type vlan id 10
# ip link show vlan10
10: vlan10@eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP>
mtu 1500 qdisc noqueue state LOWERLAYERDOWN mode
DEFAULT
link/ether f0:de:f1:81:0a:2a brd ff:ff:ff:ff:ff:ff
Physical
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = eth2
lxc.network.name = eth0
lxc.network.ipv4 = X.X.X.X/24
lxc.network.ipv4.gateway = X.X.X.1
lxc.network.hwaddr = 00:16:3e:28:73:b3
Bridging :)
➢ Linux Bridge
➢ setup with brctl
➢ setup with ip route
➢ OpenVswitch (OVS)
➢ setup with its tools
Bridging :)
➢ What is OpenVswitch
➢ multilayer virtual switch
➢ Why OpenVswitch
➢ greater flexibility
➢ more control over the traffic
➢ native VXLAN support
Bridging :)
# brctl show
bridge name
interfaces
bridge id
STP enabled
# brctl addbr br0
# brctl show
bridge name
interfaces
br0
bridge id
8000.000000000000
STP enabled
no
Bridging :)
# brctl addif br0 eth0
# brctl show
bridge name
br0
bridge id
8000.f0def1810a2a
adding a veth device
# brctl addif br0 vethc3070
adding a vlan
# brctl addif br0 eth0.4
STP interfaces
no eth0
Bridging :)
# ip link add name lxcbr0 type bridge
# ip link set dev lxcbr0 up
# ip link show lxcbr0
7: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc noqueue state UNKNOWN mode DEFAULT
link/ether fe:d8:b2:55:ce:5b brd ff:ff:ff:ff:ff:ff
# ip link set dev eth0 promisc on
# ip link set dev eth0 up
# ip link set dev eth0 master bridge_name
# ip link set dev eth0 nomaster
Securing all of these
➢ Do not allow traffic out of the container with
MAC address that was not assigned to the
container
➢ Do not allow traffic out of the container with IP
address that was not assigned to the container
➢ Do not allow multicast traffic to go to container
which is not part of the multicast group
➢ Actually if possible allow network traffic only to
its gateway :)
Securing all of these
➢ Do not use NAT for connecting your containers
➢ NAT is susceptible to DoS. By spoofing many
connections from one container can block the
connectivity of the whole machine!
Broadcasts...
➢ It depends on your network design
➢ Generally limit the broadcast destinations that a
container can reach
➢ If possible use source routing to route the traffic
directly to where it is supposed to go
OpenVswitch security
➢ Implement OpenFlow rules to enforce the previous rules
➢ For each container
hard_timeout=0,idle_timeout=0,cookie=$cookie,priority=150 dl_type=0x0800 in_port=$input_port nw_dst=$container_gw
actions=NORMAL
hard_timeout=0,idle_timeout=0,cookie=$cookie,priority=100 dl_type=0x0800 in_port=$input_port
nw_dst=$container_network actions=drop
hard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port dl_src=$container_mac nw_src=$container_ip
dl_type=0x0806 priority=50 actions=NORMAL
hard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port dl_src=$container_mac dl_type=0x0800
nw_src=$container_ip priority=25 actions=NORMAL
hard_timeout=0,idle_timeout=0,cookie=$cookie,priority=20 dl_type=0x0800 dl_src=$container_mac nw_dst=$container_ip
actions=output:$input_port
hard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port priority=5 actions=drop
➢ For each additional IP on the container
hard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port dl_type=0x0800 dl_src=$container_mac
nw_src=$additional_ip priority=10 actions=NORMAL
hard_timeout=0,idle_timeout=0,cookie=$cookie,in_port=$input_port dl_src=$container_mac nw_src=$additional_ip
dl_type=0x0806 priority=50 actions=NORMAL
OpenVswitch security
➢ OpenVswitch networking DOES NOT go trough the
normal linux networking so you CAN NOT use
ipatables/ebtables to limit the traffic
➢ Even if you use net_cls it still DON'T WORK
Thank you!
Questions?
Marian Marinov <mm@1h.com>
http://getclouder.com
Jabber: hackman@jabber.org
IRC: irc.freenode.net HackMan
ICQ: 7556201
GitHub: http://github.com/hackman