QEMU-KVM, libvirt, brctl and VLANs or (Headless Linux Virtuilzation with external VLAN isolation)

Sun 22 September 2019
Tech

This article is aimed at those who use a headless linux server. It is split into two parts: Virtulization and Networking.

1) Virtulization

Setting up QEMU-KVM!

QEMU is a sick "generic and open source machine emulator and virtualizer." It's probably available on most Linux distros.

QEMU exists as a binary file. You simply call it (/usr/lib/x86_64-linux-gnu/qemu) and pass the relevant arguments.

KVM is a component of the Linux kernel. You could think of it as an API for virtualization. Simply; It allows for the QEMU process to communicate with the CPU more efficiently. It also exists as a binary (/usr/bin/kvm) which serves as a wrapper for the QEMU binary.

Installation is trivial, simply install qemu-kmv and supporting packages.

apt install qemu-kvm libvirt-clients libvirt-daemon-system virtinst

Then you can setup a virtual machine like so

virt-install \
    --name=windows10 \
    --virt-type=kvm \
    --ram=4096 \
    --cpu=host \
    --vcpus=2 \
    --os-type=windows \
    --os-variant=win10 \
    --disk path=/mnt/thicc/data/windows10.qcow2,size=40 \
    --cdrom /mnt/thicc/data/public/Software/WindowsX.iso \
    --network bridge=br1337 \
    --graphics vnc,listen=0.0.0.0

According to this guys page you need the Windows VirtIO drivers, but YMMV.

Anyway, when you run that command virt-install will bring the Virtual Machine (VM) up and then wait for the VM to shutdown as signal that the guest Operating System (OS) installation has concluded. Now would be a good time to CTRL+Z and bg that process, as it will hold your tty.

I'd recommend installing a VNC viewer on your local machine and then port forwarding to your remote remote linux system (if you're on a different network). This is done like so: ssh -L 5900:127.0.0.1:5900 username@my.remote.host Your SSH client will listen on 5900 at your local system and tunnel data to 127.0.0.1:5900 on my.remote.host

You can use ss -tulpn to figure out which TCP port QEMU is hosting the VNC server on. It should be 5900 for your first running VM, and increments by 1 for every existing instance.

Use the VNC Viewer to complete the OS installation.

Controlling your Virtual Machine

Now that you've setup a VM, you're probably eager to learn how to start it. Enter virsh

virsh is a command line utility provided by libvirt. It's actually super powerful and extends beyond QEMU - it can be used to manage LXC, Xen, and OpenVZ hosts too, all from the single command line.

Pretty neat, eh?

VMs are referred to as domains and often you need to specify a domain argument as --domain myvmname.

Here's a ~~Top 10 Anime Betrayals~~ top few commands you should know:

destroy     destroy (stop) a domain             -   Terminates a running VM immedaitly.
edit        edit XML configuration for a domain -   Edits and reloads the XML description for a domain.
undefine    undefine a domain                   -   Removes a domain from inventory.
vol-delete  delete a vol                        -   Deletes a volume. Requires a pool argument, if unconfigured use '--pool default'

Yeah so that's cool. You should read the next section on networking for information on how to isolate your virtual machines into segregated networks.

Modifying your Virtual Machine

Pretty straight forward, you can use virsh and then run edit --domain myvmname and it'll open up the domain's descriptor file in your editor. When you exit, it'll automatically reload the domain refreshing the changes into inventory.

This is useful if you later setup a VLAN, and then want to change an existing VM to run on said VLAN. You can use edit and then search for interface and modify accordingly.

2) Networking

Bridges with brctl

Debian uses /etc/network/interfaces as it's persistent interface configuration file.

A typical interfaces file may look something like this:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto enp37s0
allow-hotplug enp37s0
iface enp37s0 inet dhcp

This is pretty standard, one local loopback device and one ethernet device.

SO BRIDGES are pretty good. They're essentially virtual switches. The kernel (and everything else) treats them just like it would a physical network interface.

apt install bridge-utils

Bridges functionally put a physical ethernet device into a slave state and then provide a virtual interface to use instead (eg: br0). It is br0 that your IP address would be bound to.

You can then attach other NICs to that bridge interface, or attach virtual machine NICs to it.

(Out of scope: If you want to use bridge-utils to route incoming packets over network segments, see net.ipv4.ip_forward=1)

Bridges can be setup transitively via brctl, or persistently within /etc/network/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug enp37s0
iface enp37s0 inet manual

auto br0
iface br0 inet dhcp
    bridge-ports enp37s0
    up ifconfig enp37s0 up

Note: Physical device enp37s0 was changed from dhcp to manual. Check the manual for bridge-utils-interfaces - it contains everything you need.

(I'm not sure if you really need up ifconfig enp37s0 up ¯\(ツ)/¯. bridge-utils should bring up the interfaces it needs automatically.)

You can use brctl show to validate the currently configured and operating bridges.

Please be careful when working with headless/remote systems. If you fuck up the network stack you're not going to have a very good time.

Now you've got your VMs operating on your LAN, great! When one of your VMs gets owned the attacker will have access to everything else on your network :D

pfSense and VLANs

First, get yourself a bigboy router. Something that's running pfSense would be great. Checkout ebay or wherever for boards featuring something like the Intel J1800/J1900 or Intel E3845 (has AES-NI) CPU and has at least TWO NICs. Intel would be best, but Realtek is fine too.

A good place to start is literally searching for "pfSense router". Once you've got that covered:

Head to Interfaces -> Assignments -> VLANs and register a VLAN. (1337 is used as demo throughout this article)

Now Interfaces -> Assignments and assign your VLAN to an interface.

Be sure to enable the interface, set the IPv4 to static (or whatever, it's your network), and specify an address and subnet size. Use an RFC1918 private network.

Go to Firewall -> Aliases and create an alias for the RFC1918 networks.

Go to Firewall -> Rules and set some rules. In the above screenshot I have:

Allowed DNS traffic to pfSense.
Blocked traffic to other RFC1915 networks
Allowed all other traffic

That's pretty much it. You might want to setup a DHCP instance too. Depending on your switch you may also have to explicitly allow trunking on ports.

SO VLANs THEN

Assuming you've already got some VLANs setup on your network, it's super easy to create more bridge devices on those VLANs. Adjust your /etc/network/interfaces to include something like:

auto br1337
iface br1337 inet manual
    bridge-ports enp37s0.1337

allow-hotplug enp37s0.1337
iface enp37s0.1337 inet manual
    vlan-raw-device enp37s0

In case you are lost, that's a VLAN (1337) interface set to manual and a bridge named br1337 to slave it. Also note that the br1337 device is set to manual. This is so my Virtual Machine host does not interact with the 1337 vlan - it merely exists as a physical host for the bridge interface, and downstream VM.

Anyway, you create an interface to TX/RX on the vlan (iface enp37s0.1337 inet manual) and then slave it to a bridge. The parent interface is specified, delimited by a period, and then the VLAN ID is appended. (You can also use vlan### if you want. Eg enp37s0.vlan1337)

(In writing this, I wonder if you could do vlan-raw-device enp37s0.1337 and skip creating the enp37s0.1337 sub-interface. 🤔)

You should know how to use the ip command suite by now. If you don't heres a crash course:

ip a    -   'ip address'    -   shows network addresses and broadcast domains       -   replacement for ifconfig
ip l    -   'ip link'       -   shows interface status                              -   replacement for ifconfig
ip r    -   'ip route'      -   shows routing information                           -   replacement for route
ip n    -   'ip neighbor'   -   shows other known hosts on the network              -   replacement for arp

If you have more than a couple of VLANs and more than a couple of VMs, that's gonna be a huge list of interfaces and addresses now. You’re a real pro now - congratulations! condolences?

As I described earlier in Modifying your Virtual Machine, you can set the VM to operate on a VLAN bridge interface. In brctl show that looks like this:

Extra Resources

https://wiki.archlinux.org/index.php/QEMU

http://bart.vanhauwaert.org/hints/installing-win10-on-KVM.html

https://wiki.archlinux.org/index.php/Network_bridge

https://wiki.debian.org/QEMU

https://wiki.debian.org/KVM

A meme

A m80 of mine made this when I fucked up my networking stack.