Dec 31, 2021

15 min read

OpenWRT Raspberry Pi Docker & VLAN Project

tl;dr Turn your Raspberry Pi into a router, managing devices and running services through Docker and VLANs! 😮

At the end of this guide you should have:

Raspberry Pi powered via Power over Ethernet running OpenWRT
Using its single gigabit ethernet LAN port to manage/route traffic
Connected to a managed gigabit switch with attached devices
Running Docker inside OpenWRT VLANs
[Docker] Plex server with mapped SSD drive
[Docker] Ad blocking managed by PiHole
DNS over HTTPS upstream servers configured for PiHole
OpenVPN client installed
[Docker] Transmission torrent client with forced traffic via VPN
Policy-based routing for network separation (LAN via WAN, Transmission via VPN)

Items Used

Raspberry Pi v4 4GB
1TB SSD USB 3.0 Drive (for Plex/Torrents)
64GB V30 MicroSD Card
GPIO Header Extender
Heatsink Case
Power Over Ethernet Hat
Netgear GS108PE v3 Managed Switch
TP-Link TL-WA1201 Access Point

Installing the PoE Hat

The Pi 4 gets a little toasty without a heatsink or fan. There are PoE Hats which include a fan, but since I already had a case, I wanted to try and work with it. If you’ve looked at the case already, then you would notice that the hat won’t fit the case without some sort of altering. First thing needed was a GPIO header extender so that the PoE hat can access the pins, but unfortunately the holes in the case didn’t allow for a normal square 4 pin extension to fit for the PoE headers, so I had to make my own.

With some spare extender cables lying about, I pulled off the pin heads:

I then wrapped each one in electrical tape and poked it on the end of the PoE headers:

End result:

Download & Install OpenWRT

Initially I used this guide to install OpenWRT. It allowed me to use the entire 64GB via EXT4 partition instead of the preset 100MB. Although a problem I had was not all x64 ARM packages supported EXT4, especially Docker (which I need). So I had to revert to SquashFS and then expand the partition.

Use your favourite SD formatting tool once downloaded. Mine is Rufus.

I used this post as a guide to expanding the partition. But to sum it up:

Install required packages:

opkg update
opkg install cfdisk resize2fs

Resize the bottom-most partition (it typically goes to max size automatically for you):

cfdisk /dev/mmcblk0

Reboot
Set the size:

resize2fs /dev/loop0

Reboot

VLAN Setup

It was this guide which really got me thinking that the Pi could be a pretty awesome home router. My problem has always been not having enough power sockets in the room for which this would be located. After reading this I figured if I could get a PoE hat, technically I don’t need an extra power socket. Hazaar!

Managed Switch

Following that guide, I decided to set up my switch as follows:

Take note of:

ports that are Tagged
VLAN 1: 1
VLAN 2: 1
ports that are UnTagged
VLAN 1: 2–7
VLAN 2: 8
ports that are Blank
VLAN 1: 8
VLAN 2: 2–7

OpenWRT

I pretty much mirrored the config from that guide, but changed the network CIDR for my setup. Here’s a snippet of /etc/config/network:

config interface 'loopback'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'
        option device 'lo'config interface 'lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ipaddr '192.168.0.1'
        option device 'eth0.1'config interface 'wan'
        option proto 'dhcp'
        option device 'eth0.2'

TIP: it’s a good idea to make a backup of your config before you start screwing around with it: cp /etc/config/network /etc/config/network.bk

That’s pretty much all you need to do to set your Pi up on a managed switch and route traffic from your local network ports on 2–7.

Open up a browser and navigate to the LAN IP address of your Raspberry Pi, in my case it’s 192.168.0.1, and view the Luci web interface. Then go to Network -> Interfaces [Interfaces/Devices]. So what you should have at the end of this are two interfaces shown below, and two devices (ignore all else, they will come later):

WiFi Access Point

This part is pretty simple, I just enabled my access point to be only just that, and nothing else. Not even DHCP, since that will be handled by the Pi. Cable goes from the access point to the swtich, and I’m sure you can figure out how to set an SSID and password yourself.

Adding the USB Drive & Creating Samba Shares

In order to access the 1TB SSD drive remotely (where my media files are stored for Plex) from my Windows PC I need to install Samaba and create some file shares. This will be installed from a package inside OpenWRT that I can then configure to share out folders on my LAN via the Raspberry Pi’s IP. I used this wiki as a reference for adding a USB drive. But to sum it up:

Install packages, create mount folder and map drive:

opkg update
opkg install ntfs-3g
mkdir -p /mnt/1TB && \
    ntfs-3g /dev/sda1 /mnt/1TB -o rw,lazytime,noatime,big_writes && \
    ls -la /mnt/1TB

To auto mount a partition at startup (with hard drive plugged) edit /etc/rc.local and add the following lines:

sleep 1
ntfs-3g /dev/sda1 /mnt/1TB -o rw,lazytime,noatime,big_writes
exit 0

So it looks like this:

Install Samba

opkg install samba4-server samba4-client luci-app-samba4

You might need to reboot the device or at least reset the service once done: service samba4 restart
Use Luci to set up your shares: Luci -> Services -> Network Shares

Docker

I’m a big fan of Docker. I had been using my Pi 4 with Ubuntu running Docker, hosting a bunch of containers (such as PiHole, Plex and Transmission), and using Portainer to manage them. I still wanted to use these services, but in a much more controlled environment via split networks and routing.

Install Docker

This is pretty straight forward with the following packages:

opkg update
opkg install dockerd docker-compose luci-app-dockerman kmod-macvlan

Take a note of kmod-macvlan, this is something we need later in order to do network separation for Docker MACVLAN.

Install Plex

This setup is pretty simple, for now. I don’t really need it to be separated from any other network, it can just sit on the device and connected via Docker bridge. On my 1TB SSD I have a subfolder called docker, for which I map all my container configurations to; it saves me a headache when having recreate everything should I need to. Also on that drive are some folders used to host music and other media. For now, my Docker command is simple, and maybe I should really turn it into a docker-compose YAML script at some point.

I use linuxserver images for Plex. These guys have some good, easy to use stuff.

docker run --detach \
    --name plex \
    --net=host \
    --restart unless-stopped \
    -e PGID=1000 \
    -e PUID=1000 \
    -e UMASK=022\
    -e VERSION=docker \
    -v /mnt/1TB/dMusic:/data/dmusic \
    -v /mnt/1TB/docker/plex:/config \
    -v /mnt/1TB/docker/plex:/transcode \
    -v /mnt/1TB/Music:/data/music \
    linuxserver/plex

Head on over to your newly created container from your browser to finish setting it up. Mine is http://192.168.0.1:32400

Install PiHole …and do the other networking stuff to support it

So this part is where it starts to really get interesting, as we’ll be using VLANs with Docker to separate the network and make things a whole lot easier for DNS setup.

I’ve read a bunch of forum posts and tutorials about using an Ad Blocker on OpenWRT, but most of them actually install the software into OpenWRT and do some funky port forwarding/routing (which is a headache). I want to run it in a container. Cos, why not? It’ll be much easier to remove and not leave a bunch of crap around if I want to get rid of it.

Initially I used this post as a guide to get it up and running on my Pi. It worked, albeit I didn’t understand it all fully and had to confirm a few things (such as the installation of kmod-macvlan), but also I think it added too much networking for what was actually needed, and thus confused me even more when I wanted to replicate the setup to apply to Transmission (more on that later). Following this guide not only created a MACVLAN on OpenWRT, but also a bridge network on Docker, for which the container connected to both. But it didn't need to.

It took me a few internet searchings to fully understand VLANs, MACVLANs and IPVLANs, for both standard and Docker setup. I’m not going to even bother explaining that here, but for a good explanation I suggest you visit this site, specifically this post about bridges and MACVLANs. It might be old, but it’s still worthy.

Now to further extend upon that learning, I also suggest this YouTube to help understand more about MACVLANs in Docker. Perhaps you don’t need to, but it did clarify a few things for me. Namely that a MACVLAN in Docker is really just an extension of whatever existing network your ethernet is on, or in my case, VLAN. And because of that, it made me realise that you don’t need any bridging that was done in the forum post just above, nor did I need any of the routing.

To extend up on that ‘no routing required’ statement…

I don’t need to create a route in OpenWRT, as it will sit in my current LAN firewall zone ⤵️:

config route
        option interface 'macvlan'
        option target '192.168.30.3'
        option netmask '255.255.255.255'

I don’t need to create 10-fixroutes.sh inside the container, because it's not using the Docker bridge network 'on top' of the existing VLAN ⤵️:

#!/usr/bin/with-contenv bash
set -eecho "fixing routes"
ip route del default
ip route add default via 172.18.0.1

OpenWRT VLAN Setup

Thus, my /etc/config/network extended to ONLY include:

config interface 'vlan20'
        option proto 'static'
        option ipaddr '192.168.20.1'
        option netmask '255.255.255.0'
        option device 'eth0.20'config device
        option type 'macvlan'
        option ifname 'eth0'
        option mode 'bridge'
        option name 'eth0.20'
        option acceptlocal '1'
        option ipv6 '0'

Where you can see that… Under Luci -> Network -> Interfaces [Interfaces] I have:

Under Luci -> Network -> Interfaces [Devices] I have:

An important part of that setup if you notice was to add the MACVLAN to the same firewall zone to which the LAN sits. It may not be 100% necessary, as you could add it to another zone and allow for routing and whatnot, but I figured it doesn’t hurt.

You can also see this in the Firewall Zone settings for LAN, too:

Docker Setup

Now that my MACVLAN/VLAN was set up, I can now deploy my container. Using the docker-compose.yml file below, I run docker-compose up -d pihole.

version: "3.3"
services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    hostname: pihole.lan
    environment:
      TZ: 'Europe/London'
      WEBPASSWORD: 'asdf'
    volumes:
      - '/mnt/1TB/docker/pihole/pihole/:/etc/pihole/'
      - '/mnt/1TB/docker/pihole/dnsmasq.d/:/etc/dnsmasq.d/'
    cap_add:
      - NET_ADMIN
    restart: unless-stopped
    networks:
      lan20:
        ipv4_address: 192.168.20.3networks:
  lan20:
    name: lan20
    driver: macvlan
    driver_opts:
      parent: eth0.20
    ipam:
      config:
        - subnet: 192.168.20.0/24
          gateway: 192.168.20.1

From Luci, you can see the following setup:

Now forward OpenWRT DNS lookup settings, Luci -> Network -> DHCP and DNS, to point to your newly created PiHole.

…and finally FORCE all devices on your network to go through your PiHole; else you might end up with them using their own DNS and avoiding the blocklist, or even having DNS leaks (more on that with DNS over HTTPS). You will need to add the PREROUTING rule in the firewall custom rules section; Luci -> Network -> Firewall [Custom Rules]:

iptables -t nat -A PREROUTING -i eth0 -p udp --dport 53 -j DNAT --to 192.168.20.3
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 53 -j DNAT --to 192.168.20.3

From here you can set up your Upstream DNS servers and be done with it, but I wasn’t. I also want to configure DNS over HTTPS.

DNS over HTTPS

What you are configuring here is an upstream server that the PiHole will turn to when the domain doesn’t exist in its blocklist. The data sent to that resolver will be encrypted and returned back to the PiHole, and finally back to the original caller.

I used this guide to setup DNS over HTTPS. Most of it is pretty straight forward, so I’ll just whack the commands in with some screen shots.

opkg update
opkg install https-dns-proxy luci-app-https-dns-proxy

After that’s installed, head over to Luci -> Services -> DNS HTTPS Proxy Settings, enable/start the service; might be worth performing a full reboot on the device to be on the safe side, and update the config to your taste:

Finally, configure PiHole to use this server for its upstream DNS lookup. Take note of the IP addresses used and the “hashes” to denote the port.

Docker #2

Install Transmission …and do the other networking stuff to support it

Now that we have a better understanding of Docker, MACVLANs and VLANs, suppose we can create Docker torrent container that is separated from the rest of the network. Say, make sure it connects out via a VPN and not over normal WAN showing our home IP. For this we’re going to use the same MACVLAN/VLAN setup as we did for the PiHole, but this time we don’t put it in the same firewall zone as our LAN. The problem with that, of course, is that our LAN is set to go out the WAN, so how does the container get out to the net? Via VPN. The magic of why this actually works, I can’t tell you, it still somewhat confuses me…

I had some random issues where my LAN wasn’t working through my WAN, or something like that, I don’t know. The container worked via VPN, but it screwed up the LAN somehow. To be honest, that day was highly confusing with other stuff, that, after I sorted with policy based routing (more on that later), I just let go of trying to figure out what went wrong.

To start off with, create your interface and device as you had done previously with the PiHole:

Create/build your Docker container and network, too:

version: "3.3"
services:
  transmission:
    container_name: transmission
    image: linuxserver/transmission:latest
    hostname: transmission.lan
    environment:
      TZ: 'Europe/London'
    volumes:
      - '/mnt/1TB/docker/transmission/config/:/config'
      - '/mnt/1TB/docker/transmission/downloads/:/downloads'
      - '/mnt/1TB/docker/transmission/watch/:/watch'
    cap_add:
      - NET_ADMIN
    restart: unless-stopped
    networks:
      internal:
      lan30:
        ipv4_address: 192.168.30.3networks:
  internal:
    name: transmission_internal
    driver: bridge
  lan30:
    name: lan30
    driver: macvlan
    driver_opts:
      parent: eth0.30
    ipam:
      config:
        - subnet: 192.168.30.0/24

Install OpenVPN

There are many great tutorials out there showing you how to set up OpenVPN on OpenWRT. I’m not going to reinvent the wheel, but I do recommend you having a look at this YouTube guide. Van Tech Corner has done a lot of videos around OpenWRT that are a great starting place to look.

Once you have set that up, I expect your Interfaces (Luci -> Network -> Interface [Interfaces]) to now have this:

Obviously mine is called VPN_TOR; named so I can run multiple VPNs cleary stating their use.

It’s also worth noting that you should specify a tunnel device in your OpenVPN configuration file (Luci -> VPN -> OpenVPN [Edit a config]). This way you can set up many VPN interfaces/devices and force certain traffic out through specific ones:

TIP: you don’t need to create a device to use it, as I’ve found OpenVPN creates one on the fly. When you do run/start OpenVPN, you will notice the device show up in the devices tab, and when that happens you can finish setting up the interface, specifying the tunX device that showed up, like Van Tech Corner explains.

Firewall Zone Configurations

Now head over to Luci -> Network -> Firewall and set some new zones, where it will end up looking something like:

TORVLAN

Specifies your Transmission (torrent/TORVLAN/VLAN30) LAN to get to the VPN service

TORVPN

Enables the VPN service to be used by your LAN and VLAN30

LAN2TOR

Allows your LAN to access VLAN30, but not the other way around

Policy Based Routing

Right now if you connect to VPN your LAN is now going out via your VPN (I think, I dunno, that day confused me), where you might not want that to happen. We want the container to, but not the LAN. There are a few tutorials out there which updates the route tables, directs traffic when/when-not needed which you can look read up on. A good example of this is this guide; it initially gave me the idea of splitting my network, and running a container out a VPN, but I couldn’t adjust it for my use case for my little monkey brain to comprehend. Thankfully Van Tech Corner to the rescue, again 😁 with this YouTube video. It seemed that this add-in was to make something hard, simple. And honestly, it really does.

A caveat to doing this, as mentioned in the video, is that you need to update DNSMASQ to the full version to install IPSET. A quick read on their website lead me to this post, where people need to be aware that you have the potential to lose the brain of your DNS if you uninstall DNSMASQ temporarily, like I did. The guide says to download the OPKG file first, and then update it. But for me I had dependencies missing that I needed, that could only be obtained by downloading them once the DNSMASQ Lite was uninstalled… which required DNS… which I didn’t have…

Thankfully a temporary fix was to update /etc/resolv.conf and point to a nameserver that could resolve the packages for me, then I could continue with the guide. Just update the following to something like 9.9.9.9, and when DNSMASQ full is installed, it'll revert back. Magic!

Once all is installed and done, head over to the settings page to configure as you wish; Luci -> VPN -> VPN and WAN Policy-Based Routing. Here's a snapshot of what mine looks like.

You should notice that I’m forcing all my LAN out via my normal WAN interface, with the exception of rutracker.org. This way I can spin up a Windows Sandbox VM on my home PC and browse to torrent sites without having to worry about running VPN on my PC. Also, you will notice that my Transmission network will force all traffic out the VPN. MAGIC!

END

I hope you found this tutorial useful, or perhaps generated ideas for a current project of your own. I’m not claiming to be an expert at any of this, so if I have something wrong or incorrect, please tell me and I will update where need be.