Prerequisites:
- Have a nginx vm, container or pod / deployment
- Have at least 2 servers for backend purposes
Introduction
As I like to toy around with salt I decided to leave my nginx vm intact rather than migrate it to a k8s deployment. However when I added an extra node to my k3s cluster, it made me realize I should have som form of loadbalancing in the event that one of the nodes goes down. There are solutions like metallb. But this as just as good.
Please be aware that however loadbalancing is possible the focus of this article lies in failover
Upstream explained
Nginx upstream is a tool that gives you the ability to loadbalance. BAsically this means that you configure 1n (or more) backend servers to which it can failover.
There are several possibilites in loadbalancing which nginx offers:
please note that I only use the opensource variant of nginx and not nginx plus
- Round Robin – Requests are distributed evenly across the servers, with server weights taken into consideration. This method is used by default (there is no directive for enabling it):
upstream backend {
# no load balancing method is specified for Round Robin
server backend1.example.com;
server backend2.example.com;
}
- Least Connections – A request is sent to the server with the least number of active connections, again with server weights taken into consideration:
upstream backend {
least_conn;
server backend1.example.com;
server backend2.example.com;
}
- IP Hash – The server to which a request is sent is determined from the client IP address. In this case, either the first three octets of the IPv4 address or the whole IPv6 address are used to calculate the hash value. The method guarantees that requests from the same address get to the same server unless it is not available.
upstream backend {
ip_hash;
server backend1.example.com;
server backend2.example.com;
}
If one of the servers needs to be temporarily removed from the load‑balancing rotation, it can be marked with the down parameter in order to preserve the current hashing of client IP addresses. Requests that were to be processed by this server are automatically sent to the next server in the group:
upstream backend {
server backend1.example.com;
server backend2.example.com;
server backend3.example.com down;
}
- Generic Hash – The server to which a request is sent is determined from a user‑defined key which can be a text string, variable, or a combination. For example, the key may be a paired source IP address and port, or a URL:
upstream backend {
hash $request_urI consistent;
server backend1.example.com;
server backend2.example.com;
}
I think in most cases round-robin will suffice. Especially in a homelab ;). However each possibilty has it’s own usescase and you should decide for yourself which one is best.
Additional notable features:
- Server Weights
By default, NGINX distributes requests among the servers in the group according to their weights using the Round Robin method. The weight parameter to the server directive sets the weight of a server; the default is 1:
upstream backend {
server backend1.example.com weight=5;
server backend2.example.com;
server 192.0.0.1 backup;
}
What this does is the following. For each 6 requests, 5 are sent to backend1 and 1 is sent to backend2. The server marked with backup will only start serving requests when backend1 and backend2 are down.
- Server Slow-Start
The server slow‑start feature prevents a recently recovered server from being overwhelmed by connections, which may time out and cause the server to be marked as failed again.
upstream backend {
server backend1.example.com slow_start=30s;
server backend2.example.com;
server 192.0.0.1 backup;
}
Usecase: loadbalancing k8s pods
I have the following situation:
- 1 k3s cluster
- 2 k3s nodes
I want to be able to failover or loadbalance. Failover means that when one of the nodes goes down. The other one will take over. This will make sure that the applications in the k8s deployment stays reachable. However deployments are reached typically via ingress through a service and not based on <ipaddres>:<port>, but via <sub><domain>.<ext>. This means that instead of balancing on <ipaddress>:<port>, you should also think about proper fqdn’s for your hosntames.
At my home I have configured this as follows in my BIND DNS servers.
- domain.local
- domain.net
Now let’s say we have portainer deployment. Which should be reachable at all times.
Which means I have 2 domains I maintain in my DNS. Underneath are the examples of the DNS records for my nginx, k3s node1 and k3s node2.
$ORIGIN .
$TTL 86400 ; 1 day
domain.local IN SOA ns1.domain.local. hostmaster.domain.local. (
2021102901 ; serial
300 ; refresh (5 minutes)
600 ; retry (10 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS ns1.domain.local.
NS ns2.domain.local.
$ORIGIN domain.local.
ns1 A 192.168.2.4
ns2 A 192.168.2.5
; machines
; 168.1.*
nginx A 192.168.1.101
k3snode1 A 192.168.1.123
k3snode2 A 192.168.1.124
; 168.2.*
DNS1 A 192.168.2.4
DNS2 A 192.168.2.5
As you can see there are only DNS records for the nginx vm, k3s nodes and DNS VM’s.
However in the domain.net config, things already look different:
$ORIGIN .
$TTL 86400 ; 1 day
domain.net IN SOA ns1.domain.net. hostmaster.domain.net. (
2021102901 ; serial
300 ; refresh (5 minutes)
600 ; retry (10 minutes)
604800 ; expire (1 week)
86400 ; minimum (1 day)
)
NS ns1.domain.net.
NS ns2.domain.net.
$ORIGIN domain.net.
ns1 A 192.168.2.4
ns2 A 192.168.2.5
; nginx proxy:
nginx A 192.168.1.104
; apps / websites
portainer CNAME nginx
; nginx CNAMES
proxy CNAME nginx
Here we can see that the portainer is pointing to nginx because of the .net domain (the use of .net is because of let’s encrypt and it looks cooler ;) )
In another blog post I have explained how I deploy nginx vhosts with salt. Please refer to the examples / configuration in this post.
Underneath you can see the eventual configuraton for portainer. The programming language is jinja2 as the template is used in salt.
When configuring upstream a couple things are important. Ofcourse make sure that the right fqdn’s are used behind server. But also to specify which port they backend should respond too. Also bear in mind that this is configured specifically for k8s pod backends. Nginx will ‘read’ backend(-whatevervalue) as a variable, so you don’t need to worry about it. As long as you keep it unique for each vhost. Trust me, you will get duplicate errors if you don’t ;)
In the server configuration part. change the hostname / ipaddress to the variable which you choose to name at upstream.
These lines which for this blog are most import::
note: please be aware that I cut some extra lines to make the changes visibible
{# if using multiple vhosts, then name the upstream to upstream-appname here to prevent duplicate errors #}
upstream backend-<yourapporwebsite>{
server k3snode1.domain.local:{{ port }} weight=5;
server k3snode2.domain.local:{{ port }};
}
{% endif %}
```bash
server {
# SSL configuration
listen 443 ssl http2;
location / {
proxy_pass {{ protocol }}://backend-<yourapporwebsite>;
}
}
Before configuring upstream, your configuration would look like this:
server {
# SSL configuration
listen 443 ssl http2;
location / {
proxy_pass {{ protocol }}://{{ ipaddress }}:{{ port }};
}
}
The configuration as deployed in nginx will look as followed:
$ cat /etc/nginx/sites-enabled/portainer.conf
# Managed by salt #
map $http_upgrade $connection_upgrade {
default Upgrade;
'' close;
}
upstream backend-portainer{
server k3snode1.domain.local:443 weight=5;
server k3snode2.domain.local:443;
}
server {
# SSL configuration
listen 443 ssl http2;
server_name portainer.domain.net;
client_max_body_size 520M;
# 443 logging
error_log /etc/nginx/logs/portainer_error_443.log warn;
access_log /etc/nginx/logs/portainer_access_443.log;
location / {
proxy_read_timeout 900s;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $proxy_add_x_forwarded_for;
proxy_pass_request_headers on;
add_header X-location websocket always;
proxy_pass https://backend-portainer;
proxy_ssl_verify off;
#proxy_buffering off;
fastcgi_buffers 16 16k;
fastcgi_buffer_size 32k;
proxy_buffering on;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
}
ssl_certificate /etc/letsencrypt/live/domain.net/fullchain.
ssl_certificate_key /etc/letsencrypt/live/domain.net/privkey.pem;
#ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off; # Requires nginx >= 1.5.9
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
}
{% endif %}
server {
listen 80;
server_name portainer.domain.net;
# this is not needed in http
# fastcgi_buffers 16 16k;
# fastcgi_buffer_size 32k;
# redirect to https
location / {
return 301 https://$host$request_uri;
}
#logging:
error_log /etc/nginx/logs/portainer_error.log warn;
access_log /etc/nginx/logs/portainer_access.log;
}
k8s deployment example
For those of you interested. Underneath you can find the yaml files of my ingress, service and deployment for portainer. Try it out. Please keep in mind that I use custom storage. You’ll have to change that part to your personal situation.
Ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: portainerx
namespace: portainer
annotations:
kubernetes.io/ingress.class: "traefik"
traefik.frontend.passHostHeader: "true"
traefik.backend.loadbalancer.sticky: "true"
spec:
rules:
- host: portainer.domain.net
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: portainerx-service
port:
number: 9000
Service:
apiVersion: v1
kind: Service
metadata:
namespace: portainer
name: portainerx-service
spec:
type: NodePort
selector:
app: portainerx
ports:
- name: http
port: 9000
targetPort: 9000
nodePort: 30778
protocol: TCP
- port: 8000
targetPort: 8000
protocol: TCP
name: edge
nodePort: 30775
Deployment:
apiVersion: v1
kind: Namespace
metadata:
name: portainer
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: portainer-sa-clusteradmin
namespace: portainer
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: portainer-crb-clusteradmin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: portainer-sa-clusteradmin
namespace: portainer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: portainerx
namespace: portainer
labels:
app: portainerx
spec:
replicas: 1
selector:
matchLabels:
app: portainerx
template:
metadata:
labels:
app: portainerx
spec:
restartPolicy: Always
serviceAccountName: portainer-sa-clusteradmin
volumes:
- name: iscsi-data-portainer
iscsi:
targetPortal: 1.1.1.1:3260
iqn: <someiqn>:k3s-portainer
lun: 5
fsType: xfs
readOnly: false
chapAuthSession: false
containers:
- name: portainerx
image: "portainerci/portainer:develop"
imagePullPolicy: "Always"
ports:
- containerPort: 9000
name: http-9000
protocol: TCP
- name: tcp-edge
containerPort: 8000
protocol: TCP
volumeMounts:
- name: iscsi-data-portainer
mountPath: "/data"
note: I use the develop tag on the portainer image to meet specific k3s needs