cilium

Why Cilium for My On-Prem K8s Cluster

A little background. I built my own home Kubernetes cluster using kubeadm, sticking to the basics to start with. Excited to experiment, I deployed a few sample apps, created some secrets, and finally tried exposing an NGINX service to see how I could access it from outside my cluster. And that's the beginning.

Story

I created a simple NGINX Pod and a default Service to map to the Pod’s ports.

By default, a Service is of type ClusterIP, which means it’s only reachable inside the cluster. But I wanted to reach it from my local network — so my options were limited to NodePort or LoadBalancer.

Since I’m running on-premises, I don’t have the luxury of a cloud provider that can automatically provision a load balancer for me. So I exposed it using NodePort: this opens a specific port on each node’s IP address. And voila — it worked!

However, this approach is not scalable. My plan is to expose more services, but with NodePort, I would have to:

Open a separate port on the node for each service.
Remember which port maps to which service.

This quickly becomes a hassle.

A better way

A more flexible solution is to have something that:

Watches for Services of type LoadBalancer.
Assigns them a real IP from your local network.
Announces this IP to your network so other devices can reach it directly.

This can be done in two ways:

Use a hardware load balancer
Use a software load balancer

A hardware load balancer might sound easier, but I don’t own one, I have never seen one, let alone know how to use one — and I don’t want to add extra hardware to my stack just for this.

Mighty MetalLB

So, a software load balancer is the practical choice — and the most common solution for on-prem clusters is MetalLB (a simple google search will show you it's most peoples' default choice).

What MetalLB does:

Watches for Services of type LoadBalancer.
Assigns each service an IP from a pool that you define.
Announces this IP to your network using ARP (Layer 2) or BGP (Layer 3).

With this setup, your cluster can handle inbound traffic on real IPs without needing external hardware.

However, MetalLB’s scope is limited:

It only does IP assignment and announcement — it doesn’t improve how traffic is routed or balanced inside the cluster.

If you prefer to keep external load balancing minimal and rely more on internal routing with Ingress Controllers, you may find MetalLB redundant — it mainly exists just to hand out a couple of IPs.

Enter Cilium

I knew Cilium was a CNI plugin and has been making steady strides with its eBPF-based traffic routing and kube-proxy replacement.

However, after some more research, I discovered that Cilium can:

Provide LoadBalancer IPs (like MetalLB).
Announce these IPs to your local network (Layer 2).
Replace kube-proxy entirely by using fast eBPF programs that run directly in the Linux kernel, instead of thousands of iptables rules.

In short, Cilium:

Watches for Services of type LoadBalancer.
Assigns real IPs from a pool you define.
Announces these IPs to your network.
Handles Pod-to-Pod, Node-to-Node, and Service routing efficiently inside the cluster.

This means fewer moving parts and higher performance — all using the same tool.

For my home cluster, this setup felt cleaner:

Cilium handles networking, load balancing, and IP announcements.
Traefik works as my Ingress Controller to handle Layer 7 routing for HTTP(S) traffic.

So far, I’m very happy with Cilium and Traefik. I love it especially when I expose a new LoadBalancer service and it just get an IP in the local network. How cool is that?

What else Cilium can do?

Cilium isn’t just a networking replacement — it’s a full Cloud Native Networking & Security platform with features like:

Network Policies: Fine-grained traffic control between Pods.
Cluster Mesh: Connect multiple clusters together.
Ingress Controller: Cilium can even act as an L7 Ingress Controller (HTTP/HTTPS) using eBPF and Envoy proxy.
Hubble: Deep visibility and observability for your cluster’s network flows.

I haven’t explored all of these yet — but it’s reassuring to know they’re available if I need them in the future.

Why I use Traefik for Ingress?

Although Cilium can handle Ingress too, I decided to stick with Traefik as my Ingress Controller for now.

Why?

I’m already familiar with Traefik.
It’s simple to configure for TLS certificates and routing.
It has a great dashboard and ecosystem.

So my current setup is:

Cilium handles networking, LoadBalancer IPs, and kube-proxy replacement.
Traefik manages smart HTTP/HTTPS routing for incoming web traffic (Layer 7).

This split gives me the best of both worlds — with the option to experiment more with Cilium’s Ingress later on.

Finally..

If you’re running Kubernetes on-premises:

MetalLB is great if you just need to expose Services with real IPs.
Cilium does the same — plus it replaces kube-proxy and gives you modern, high-performance networking with eBPF.

Less to maintain, more power under the hood and many many LoadBalancer services (I use mostly for local development and testing)