Episode 0x03: Networking - Part 1: VLANs with pfSense

Note: I strongly recommend setting up at least one Ubuntu machine with a UI for this part. We are going to configure an internal network, which entails many trials and errors, static IP assignments for diagnostic purposes, and potentially lots of hair-pulling. An easily configurable VM can simplify verifying your configurations.

Intro

As mentioned in part 1 of the series, we aim to isolate our Kubernetes cluster from other devices in our network. However, what we truly intend is for external devices to be able to initiate connections to our cluster but prevent the reverse. This setup allows us to locally connect to our cluster.

Below is an image that illustrates the overall structure of our network setup:

We have two main subnets in our network. The first, 192.168.0.1/24, is the main subnet where my modem assigns IPs to connected devices. The second, 10.100.0.1/24, is where we want our Kubernetes cluster to reside. To achieve this, we need to utilize a DHCP server, firewall, and VLANs. These are typically provided by modems, there are several solutions like pfSense, OPNsense, and OpenWrt to emulate a modem. We will use pfSense due to its ease of setup and flexibility.

pfSense

Installation

First, download the pfSense ISO image from their download page and upload it to Proxmox ISO images (local storage > ISO images).

Create a VM and select the pfSense image from the OS tab. Here are the hardware specifications for the pfSense VM:

Start the VM, and the pfSense installer will appear.

1. Accept the terms and conditions.
2. Select Install pfSense.
3. Choose Auto (ZFS) for the partition type.
4. Proceed with the installation.
5. Select Stripe - no redundancy.
6. Choose the virtual I/O block device (press space) and confirm.
7. Reboot when the installation is complete.
8. Wait for pfSense to boot up.

You’ll then be prompted with several questions.

1. Should VLANs be set up now? - Enter no.
2. Enter the WAN interface - Enter the first option, which should be vtnet0.
3. Enter the LAN interface - Leave this empty and press enter.

Proceed by confirming yes to the configuration questions. If the setup completes successfully, you should see a result similar to the image below:

Next, set a static IP address for the WAN interface to prevent pfSense being assigned a new IP on every restart. Enter 2 to set a static IP. After this, you should be able to log in to the pfSense portal by navigating to the set IP in your web browser, e.g., http://192.168.0.156. The default username and password are admin and pfsense.

After logging into pfSense for the first time, you will be greeted by a setup wizard. Follow the wizard steps and ensure to change the admin password.

Configuring WAN & Routing

Navigate to Interfaces > WAN. If you have set the static IP during installation, this should be correct. If not, set the IPv4 configuration type to static, choose an IP address with a 24 netmask, click + Add a new gateway, and define the gateway. Save the settings.

Next, go to System > Routing. You should see a WANGW gateway. Set this as the default gateway for IPv4 and apply the changes.

Setting Up VLAN

Navigate to Interfaces > Assignments > VLANs. Create a VLAN with the tag 100 and save.

Return to Interfaces > Interface Assignments and click the green Add button to add an interface assignment for VLAN 100.

Click on the LAN interface to configure it. Ensure you check the Enable box. Set the IPv4 configuration type to Static, the IPv6 type to None, and assign an IPv4 address for the subnet you wish to use with a 24 mask. I used 10.100.0.1 and renamed it to KubernetesVLAN100. Leave the IPv6 Upstream Gateway as None since this subnet won’t have a direct internet connection; it will connect through the WAN.

Save and apply the changes. You should see KubernetesVLAN100 among the interfaces in the dashboard.

Note: After enabling VLAN, you may not be able to connect to pfSense from your work machine. This occurs because pfSense thinks there is an accessible LAN network, thus disabling dashboard access through its WAN. To proceed, utilize a VM (the Ubuntu machine I advised to setup) and access the pfSense portal from it. Since we haven’t enabled DHCP on the VLAN yet, we need to set a static IP.

Configure your Ubuntu VM network interface in Proxmox, setting the VLAN Tag to 100.

Then, start your Ubuntu machine and set a static IP for your wired connection.

You should now be able to access the pfSense admin panel at https://10.100.0.1.

Configuring DHCP

If you prefer not to manually assign IP addresses on the VLAN, you can enable a DHCP server. This allows for easier network diagnostics while still supporting static IP assignments.

First, go to System > Advanced > Networking and ensure that the DHCP Server Backend is set to Kea DHCP. Save the settings.

Then navigate to Services > DHCP Server > KubernetesVLAN100. Enable the DHCP server and define an address pool. I’ve chosen 10.100.0.10 to 10.100.0.127 to reserve lower range IPs for future VLAN machines and high-range IPs for static assignments. 10.100.0.128 - 255 will later be used for MetalLB to expose Kubernetes services. More on this in future episodes.

Save and apply the changes.

Firewall

Now, we need to configure our firewall for both the WAN and KubernetesVLAN100 to ensure proper isolation. First, go to Firewall > Aliases > IP and add a new alias named RFC1918, which contains all local IPs defined by RFC1918.

Next, navigate to Firewall > Rules > WAN and set the following rules:

• Deny all traffic by default.
• Allow pings (ICMP) from WAN subnets to KubernetesVLAN100.
• Allow incoming connections from WAN subnets to KubernetesVLAN100 on ports 80 (HTTP), 443 (HTTPS), 6443 (used by kubectl to connect to the cluster), and 22 (SSH, for manual node connections and later used by Ansible).
• Keep pfSense Admin portal accessible from WAN subnets.

I’ve also added a rule (disabled by default) for test purposes. This allows connections to services exposed via LoadBalancer in Kubernetes on random ports, like Minio or Prometheus. Generally, services will be accessed through port 80 or 443 using ingress.

Then head to Firewall > Rules > KubernetesVLAN100 and set the following rules:

• Allow outgoing connections from KubernetesVLAN100 to any destination by default to ensure VLAN-connected VMs can access the internet.
• Disallow any initiated connections from KubernetesVLAN100 to local networks defined by the RFC1918 alias to ensure Kubernetes applications can’t access the local network.
• Allow DNS queries to be forwarded to pfSense from VLAN VMs, essential for proper internet access.

The first rule is an Anti-Lockout Rule automatically added by pfSense to ensure the admin portal remains accessible on LAN.

Test Internet Access

With the above configurations, the VLAN setup is essentially complete. Set the VLAN Tag to 100 on your Ubuntu machine, bridge its network interface to vmbr0, and boot it up. You should have internet access, your Ubuntu machine should be accessible from your local network, but you should not be able to access or ping local devices from your Ubuntu machine within the VLAN.

Stay tuned till networking part 2: 🚀