However, when it comes to rendering and validating manifests locally, I found myself frustrated with the existing tools (or lack thereof). For me, I found that working with manifest generators like helm or kustomize often involved a repetitive cycle:

1. Run helm template, kustomize build, or similar commands
2. Search through many pages of output looking for specific resources
3. Find some issue and make a change to the source files
4. Re-run the rendering commands
5. Re-run whatever search I originally did
6. Find another issue and make a change to the source files
7. Repeat ad nauseam

So, I set out to build something that would make this process easier and more efficient. After a few months of work, I’m excited to introduce you to kat!

What is kat?

kat automatically invokes manifest generators like helm or kustomize, and provides a persistent, navigable view of rendered resources, with support for live reloading, integrated validation, and more.

It is made of two main components, which can be used together or independently:

1. A rule-based engine for automatically rendering and validating manifests
2. A terminal UI for browsing and debugging rendered Kubernetes manifests

Together, these deliver a seamless development experience that maintains context and focus while iterating on Helm charts, Kustomize overlays, and other manifest generators.

If you’re interested in giving kat a try, there are installation and usage instructions available in the repo’s README: github.com/macropower/kat

Features

Manifest Browsing: Rather than outputting a single long stream of YAML, kat organizes the output into a browsable list structure. Navigate through any number of rendered resources using their group/kind/ns/name metadata.

Live Reload: Just use the -w flag to automatically re-render when you modify source files, without losing your current position or context when the output changes.

Integrated Validation: Run tools like kubeconform, kyverno, or custom validators automatically on rendered output through configurable hooks. Additionally, you can define custom “plugins”, which function the same way as k9s plugins (i.e. commands invoked with a keybind).

Flexible Configuration: kat allows you to define profiles for different manifest generators (like Helm, Kustomize, etc.). Profiles can be automatically selected based on output of CEL expressions, allowing kat to adapt to your project structure.

And Customization: kat can be configured with your own keybindings, as well as custom themes!

How do I use it?

Let’s use a simple example with helm to illustrate how kat works.

Note that configuration for helm is included in kat’s default configuration.

First, we need to define a profile for helm. This profile will specify how to render manifests using helm template, as well as:

• Any init, preRender, and/or postRender hooks we want to apply
• Any plugins that should be available in this context
• Any UI settings that should differ from the global config

# yaml-language-server: $schema=./config.v1beta1.json
apiVersion: kat.jacobcolvin.com/v1beta1
kind: Configuration
profiles:
  helm:
    command: helm
    args: [template, ., --generate-name]
    env: []
    # Reload on edits to YAML and template files.
    source: >-
      files.filter(f, pathExt(f) in [".yaml", ".yml", ".tpl"])      
    # Inherit helm environment variables from the caller process.
    envFrom: &helmEnvFrom
      - callerRef:
          pattern: ^HELM_.+
    hooks:
      # Ensure that helm is installed.
      init:
        - command: helm
          args: [version]
      # Build any helm dependencies before rendering.
      preRender:
        - command: helm
          args: [dependency, build]
          envFrom: *helmEnvFrom
      # Validate rendered resources with kubeconform.
      postRender:
        - command: kubeconform
          args: ["-strict", "-summary"]
    ui:
      # Use a custom theme for the helm profile.
      theme: "dracula"
    plugins:
      # Add a plugin to invoke helm dry-run when `H` is pressed.
      dry-run:
        description: invoke helm dry-run
        keys:
          - code: H
        command: helm
        args: [install, ., -g, --dry-run]
        envFrom: *helmEnvFrom

Now, we can use this profile to render manifests. For example, if we have a Helm chart in the current directory, we can run:

kat . helm

Next, we can define a rule, so that we automatically select the helm profile when we run kat in a directory containing a helm chart:

# yaml-language-server: $schema=./config.v1beta1.json
apiVersion: kat.jacobcolvin.com/v1beta1
kind: Configuration
profiles:
  # ...
rules:
  # If there is a chart file in the current directory, and
  # it defines `apiVersion: v2`, use the `helm` profile.
  - match: >-
      files.exists(f,
        pathBase(f) in ["Chart.yaml", "Chart.yml"] &&
        yamlPath(f, "$.apiVersion") == "v2"
      )      
    profile: helm

Now, if we have a Helm chart in the current directory, we can simply run:

kat

And kat will automatically select the helm profile, render, and validate the helm chart for us.

You can continue to define additional profiles and rules to handle other manifest generators like Kustomize, Jsonnet, CUE, KCL, and more. You can also express more complex rules using CEL expressions to match specific project structures or configurations (such as using flux-local if your project contains some Fluxtomizations).

To learn more, check out the README and kat’s CEL Expression Guide.

Conclusion

kat solved my specific workflow problems when working with Kubernetes manifests locally. And while it may not be a perfect fit for everyone, I hope it can help others who find themselves in a similar situation.

If you’re interested in giving it a try, check out the repo here:

github.com/macropower/kat (please ⭐ if you like it!)

Also, a huge thanks to the authors of the following projects (that provided inspiration and/or code):

• k9s - A terminal UI to interact with your Kubernetes clusters.
• bat - A cat(1) clone with wings.
• task - A task runner for Go.
• glow - Render markdown on the CLI, with pizzazz!
• soft-serve - The mighty, self-hostable Git server for the command line.
• wishlist - The SSH directory.
• viddy - A modern watch command.
• charmbracelet/bubbletea - A powerful TUI framework for Go.
  • …plus many other fantastic libraries from charm
• alecthomas/chroma - A general-purpose syntax highlighter in pure Go.
• google/cel-go - A fast, portable, and safe expression evaluation engine.
• goccy/go-yaml - YAML support for Go.
• fsnotify - Cross-platform filesystem notifications.
• invopop/jsonschema - JSON Schema generation.
• santhosh-tekuri/jsonschema - JSON Schema validation.

Setting up multi-cluster communication with Linkerd is straightforward under ideal conditions. However, it can be more challenging if one of the clusters cannot create services of type LoadBalancer (with a public, or otherwise routable IP address). This is the case for me, as I have clusters both at home and in Hetzner, with my home Kubernetes cluster running behind NAT.

Fortunately, there are ways to work around this! In this post, I’ll walk you through my design and implementation process, discuss the challenges I faced, and share the workarounds I came up with to make everything run smoothly. Additionally, I’ll explore some alternative options for multi-cluster communication and potential improvements to my current setup.

My exact and up-to-date implementation of everything in this article can be found in my homelab repo! https://github.com/MacroPower/homelab

The Basics

One thing that was not immediately clear to me, when I was following the multi-cluster setup docs for the first time, was how services are mirrored bi-directionally between clusters. The docs give examples of how to link a theoretical “east” cluster to a “west” cluster, which can be done via this command:

linkerd --context=east multicluster link --cluster-name east |
  kubectl --context=west apply -f -

However, this only allows you to mirror services from the “east” cluster to the “west” cluster. If you want to mirror services from the “west” cluster to the “east” cluster, you need to run this command a second time, but in the inverse order:

linkerd --context=west multicluster link --cluster-name west |
  kubectl --context=east apply -f -

This means that if you want bi-directional mirroring between two clusters, each cluster needs to have the ability to create services of type LoadBalancer, with an IP address that can be reached by the other cluster.

To say it differently, any clusters acting as a source for service mirrors must be routable from any destination clusters. If the source isn’t routable, you have to find a way to make it so, regardless of the networking situation on the other cluster.

The Recommended Solution

Linkerd’s multi-cluster docs recommend looking into inlets. The concept is very cool and also pretty simple.

Basically, in your home / non-routable cluster, you can have a client acting as a sort of proxy to any local services. The client establishes a tunnel to a server running somewhere accessible from the cloud / routable cluster. This means that you should be able to just point Linkerd to the inlets server, and from there it will be routed to the client, and then to the previously non-routable Linkerd gateway!

However, inlets is no longer the open-source project it once was. The author stopped maintaining the open-source version a while ago, before eventually deleting all of the source code. Now, it’s available for purchase as a monthly subscription, with personal licenses starting at $20/month. For me, this is a completely infeasible price to pay. There are still parts of the project that are open-source, notably inlets-operator, but it spins up an entire VPS for the tunnel, which we then have to pay for, when we already have a perfectly good K8s cluster we could be hosting it on.

Luckily, a fork of inlets was created, cubed-it/inlets. This will allow us to manually create a client in our home / non-routable cluster, and a server in our cloud / routable cluster. Again, very luckily, the fork adds support for tunneling TCP ports (as opposed to HTTP), which we will need for both Linkerd and also the K8s API server.

My Implementation

Conceptually, what I wanted to do was this:

┌────────────────────────────────────────┐     ┌─────────────────────────────────────────┐
│             Cloud Cluster              │  │  │              Home Cluster               │
│                                        │     │                                         │
│ ┌───────────────┐     ┌─────────────┐  │  │  │ ┌─────────────┐      ┌─────────────┐    │
│ │ Inlets Server │◀────│   Ingress   │◀─┼─────┼─│Inlets Client│──┬──▶│   Linkerd   │──┐ │
│ └───────────────┘     └─────────────┘  │  │  │ └─────────────┘  │   │   Gateway   │  │ │
│         ▲                              │     │                  │   └─────────────┘  │ │
│         ├───────────────────┐          │  │  │                  │   ┌─────────────┐  │ │
│         │                   │          │     │                  └──▶│   K8s API   │  │ │
│         │                   │          │  │  │                      └─────────────┘  │ │
│ ┌───────────────┐ ╔══════════════════╗ │     │                      ╔═════════════╗  │ │
│ │Linkerd Service│ ║      Foobar      ║ │  │  │                      ║   Foobar    ║  │ │
│ │    Mirror     │ ║      Mirror      ║─│─ ─ ─│─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─▶║   Service   ║◀─┘ │
│ └───────────────┘ ╚══════════════════╝ │  │  │                      ╚═════════════╝    │
│         │                   ▲          │     │                                         │
│         └────────Creates────┘          │  │  │                                         │
└────────────────────────────────────────┘     └─────────────────────────────────────────┘

In this implementation, the inlets client and server is entirely contained within Kubernetes. This is great, because we don’t have to pay anything extra, and also it’s great from a security perspective because the only thing we need to expose outside the cluster is a single endpoint for the tunnel, which can be done via our normal ingress.

In this section, I will walk you through my implementation for setting up a secure tunnel between my home and cloud Kubernetes clusters using inlets. We will be deploying inlets server and client using Helm charts I created for both the inlets server and client, which works with cubed-it/inlets, and then addressing some of the challenges encountered with Linkerd during the process.

First, add a new namespace in every cluster:

kubectl create namespace inlets

And create a secret in every cluster with the same token:

token=$( head -c 16 /dev/urandom | shasum | cut -d" " -f1 )
kubectl create secret generic linkerd-tunnel-token \
  --from-literal=token=${token}

In our cloud cluster, we can use the inlets-server chart:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
  - name: inlets-server
    repo: https://jacobcolvin.com/helm-charts/
    version: "0.1.1"
    releaseName: linkerd-tunnel
    namespace: inlets
    valuesInline:
      inlets:
        # Port is the main port that serves any HTTP traffic.
        # Other TCP ports are assigned on the client.
        port: 4191
        disableTransportWrapping: true
        tokenSecretName: linkerd-tunnel-token

      service:
        data-plane:
          type: ClusterIP
          ports:
            kube:
              port: 6443
              protocol: TCP
            proxy:
              port: 4143
              protocol: TCP
            admin:
              port: 4191
              protocol: TCP

      ingress: {}
      #  main:
      #    enabled: true
      #    hosts:
      #      - host: linkerd-tunnel.example.com
      #        paths:
      #          - path: /
      #            pathType: Prefix
      #    tls:
      #      - hosts: [linkerd-tunnel.example.com]
      #    annotations: {}

In our home cluster, we can use the inlets-client chart:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
  - name: inlets-client
    repo: https://jacobcolvin.com/helm-charts/
    version: "0.1.2"
    releaseName: linkerd-tunnel
    namespace: inlets
    valuesInline:
      inlets:
        # The url points to the ingress of the other cluster.
        url: wss://linkerd-tunnel.example.com
        # Since we don't want to restrict the hostnames the other cluster can
        # use, strictForwarding should be false.
        strictForwarding: false
        tokenSecretName: linkerd-tunnel-token

        # Configure upstreams for Linkerd. Any traffic coming to `match` will
        # be forwarded to `target`. If the `match` value is `tcp:PORT`, the
        # server will automatically create a server listening on that port.
        upstreams:
          - # Accessible on inlets.port / 4191.
            target: http://linkerd-gateway.linkerd-multicluster.svc.cluster.local:4191
          - match: tcp:6443
            target: kubernetes.default.svc.cluster.local:443
          - match: tcp:4143
            target: linkerd-gateway.linkerd-multicluster.svc.cluster.local:4143

In theory, you would expect to then be able to Link the clusters, just by overriding the defaults for the gateway and API server addresses:

linkerd --context=home multicluster link --cluster-name home \
    --gateway-addresses "linkerd-tunnel-data-plane.inlets.svc.cluster.local" --gateway-port 4143 \
    --api-server-address "https://linkerd-tunnel-data-plane.inlets.svc.cluster.local:6443" |
  kubectl --context=cloud apply -f -

But this is not the case. There are multiple interactions between the normal output of linkerd multicluster link and the inlets tunnel that need to be accounted for.

Fixing the probe-gateway service

First of all, the linkerd multicluster link command creates a probe-gateway service, which points to the gateway’s health endpoint. However, in this case, that health endpoint is actually another Kubernetes service. Now, I’m not confident on exactly why this is, but this is not a configuration that works in Kubernetes. The probe-gateway service will time out every time, even though the endpoint it points to will work just fine. To work around this issue, we need to change the probe-gateway service so that it’s of type ExternalName, with an externalName of the inlets server service.

apiVersion: v1
kind: Service
metadata:
  name: probe-gateway-home
  namespace: linkerd-multicluster
  labels:
    mirror.linkerd.io/mirrored-gateway: "true"
    mirror.linkerd.io/cluster-name: home
spec:
  type: ExternalName
  externalName: linkerd-tunnel-data-plane.inlets.svc.cluster.local
  ports:
  - name: mc-probe
    port: 4191
    protocol: TCP

Once this is done, the linkerd multicluster link command will work, and the linkerd multicluster check command will actually succeed as well. However, if you then create a service mirror, it will not work.

Fixing the service mirror

The service mirror suffers from the same issue as the probe gateway. It creates a service that points to the gateway’s proxy endpoint, which is another Kubernetes service. Unfortunately, we can’t solve this so easily since the service mirror is created dynamically by Linkerd. But, we can get around this issue by changing the way that we expose the gateway’s proxy endpoint. Instead of using a normal service, we can create a LoadBalancer service, which is very annoying, but I couldn’t find any better workarounds for it. Note that this service does not need to be exposed outside the cluster, so don’t feel a need to add a firewall exception or anything like that.

This introduces an additional issue. If we already have a LoadBalancer in our cluster for the gateway (for services being mirrored in the other direction), we can’t reuse the same port.

You can implement all of this by first making a slight modification to the upstreams declared in the inlets-client chart, to remap the ports:

upstreams:
  - target: http://linkerd-gateway.linkerd-multicluster.svc.cluster.local:4191
  - match: tcp:6443
    target: kubernetes.default.svc.cluster.local:443
  - match: tcp:6143
    target: linkerd-gateway.linkerd-multicluster.svc.cluster.local:4143

And also changing the services declared in the inlets-server chart:

inlets:
  port: 6191
service:
  data-plane:
    type: ClusterIP
    ports:
      kube:
        port: 6443
        protocol: TCP
      admin:
        port: 6191
        protocol: TCP
  data-plane-lb:
    type: LoadBalancer
    # Add any annotations you need, e.g. for MetalLB.
    annotations: {}
    ports:
      proxy:
        port: 6143
        protocol: TCP

And the probe-gateway service that we changed, we need to change it again for the new mc-probe port:

apiVersion: v1
kind: Service
metadata:
  name: probe-gateway-home
  namespace: linkerd-multicluster
  labels:
    mirror.linkerd.io/mirrored-gateway: "true"
    mirror.linkerd.io/cluster-name: home
spec:
  type: ExternalName
  externalName: linkerd-tunnel-data-plane.inlets.svc.cluster.local
  ports:
  - name: mc-probe
    port: 6191 # <-- The new port.
    protocol: TCP

Meshing the inlets pods

As a bit of a plus, using different ports from linkerd-proxy means that we can mesh the inlets pods themselves. You can do this by adding linkerd.io/inject: enabled to the namespace annotations:

apiVersion: v1
kind: Namespace
metadata:
  name: inlets
  annotations:
    linkerd.io/inject: enabled

The only change you will need to make is on the client. You will also need to skip the Linkerd outbound ports, which you can do by adding the following:

podAnnotations:
  config.linkerd.io/skip-outbound-ports: "4143,4191"

Linking the clusters

With all these workarounds in place, the architecture looks like this:

┌──────────────────────────────────────────────────────────┐     ┌─────────────────────────────────────┐
│                      Cloud Cluster                       │  │  │            Home Cluster             │
│                                                          │     │                                     │
│ ┌───────────────┐   ┌───────────────┐   ┌─────────────┐  │  │  │ ┌───────────────┐   ┌─────────────┐ │
│ │ Inlets Server │◀──│ Service: 8123 │◀──│   Ingress   │◀─┼─────┼─│ Inlets Client │──▶│   K8s API   │ │
│ └───────────────┘   └───────────────┘   └─────────────┘  │  │  │ └───────────────┘   └─────────────┘ │
│         ▲                                                │     │         │                           │
│         ├─────────────────┬───────────────────┐          │  │  │         │                           │
│         │                 │                   │          │     │         │                           │
│ ┌───────────────┐ ┌───────────────┐ ┌──────────────────┐ │  │  │         │                           │
│ │    K8s API    │ │Gateway Health │ │  Gateway Proxy   │ │     │         └──────────────────┐        │
│ │ Service: 6443 │ │ Service: 6191 │ │ Private LB: 6143 │ │  │  │                            │        │
│ └───────────────┘ └───────────────┘ └──────────────────┘ │     │                            │        │
│         ▲                 ▲                   ▲          │  │  │                            │        │
│         │                 │                   │          │     │                            ▼        │
│ ┌───────────────┐  ┌─────────────┐  ╔══════════════════╗ │  │  │  ╔═════════════╗    ┌─────────────┐ │
│ │Linkerd Service│  │probe-gateway│  ║      Foobar      ║ │     │  ║   Foobar    ║    │   Linkerd   │ │
│ │    Mirror     │─▶│ExternalName │  ║      Mirror      ║─│─ ┼ ─│─▶║   Service   ║◀───│   Gateway   │ │
│ └───────────────┘  └─────────────┘  ╚══════════════════╝ │     │  ╚═════════════╝    └─────────────┘ │
│                                                          │  │  │                                     │
└──────────────────────────────────────────────────────────┘     └─────────────────────────────────────┘

Now we can link the two clusters (but for real this time):

linkerd --context=home multicluster link --cluster-name home \
    --gateway-addresses "<The address of your LoadBalancer>" --gateway-port 6143 \
    --api-server-address "https://linkerd-tunnel-data-plane.inlets.svc.cluster.local:6443" |
  kubectl --context=cloud apply -f -

Other Options

There are tons of other avenues that could be explored for this. I went down this particular path because it was recommended in the Linkerd docs. However, basically any solution that would allow our cloud cluster to talk directly to our homelab would have worked. For example, setting up something like Wireguard would also have probably been a very reasonable solution.

There are also other options for multi-cluster communication, besides Linkerd. However, I believe they will all have the same or worse networking requirements. For example, some solutions require that all individual nodes be routable across all clusters.

Future Work

• I don’t think there’s any reason why a K8s provisioner couldn’t be added to inlets-operator, which means we wouldn’t have to use the forked version of inlets.

• There may also be a way to modify the service mirrors created by Linkerd, such that they can be directly routed through the tunnel, instead of having to hit a LoadBalancer first, e.g. by creating them as type ExternalName.

Conclusion

I hope this was helpful. To learn more, you can check out the following links:

• Linkerd: https://github.com/linkerd/linkerd2
• Linkerd Multi-Cluster: https://linkerd.io/2.12/tasks/multicluster/
• Inlets: https://blog.alexellis.io/ingress-for-your-local-kubernetes-cluster/
• Inlets Operator: https://github.com/inlets/inlets-operator
• Inlets Fork by cubed-it: https://github.com/cubed-it/inlets

And finally, again, if you would like to see my exact and up-to-date implementation of everything above, check out my homelab repo:

• https://github.com/MacroPower/homelab

As of writing, I have made a purchase (I went with the MyElectronics chassis) and am waiting for it to arrive. I will eventually be writing another article covering my exact setup.

1U

1U chassis will not fit a Turing Pi. So, I did not evaluate these.

1.5U

I gave up on this idea for a couple reasons.

One, we don’t yet know whether newer compute modules will fit. CM4s should, but there’s no telling how much clearance will be needed for RK1, etc.

Two, the second system, which I want to be a more normal x86 system, is not optimized for 1.5U. Next to no fans are going to fit in 1.5U, and fanless are all optimized for 1U with those loud and awful fans.

For this reason the 2U list is likely more complete.

OnLogic MK150

Items:

• MK150
• AKDB-MK15X

Notes: Normal mini-itx chassis that can be adapted into a dual-mainboard chassis.

Cost: $202 + $15

DV Industrial Computer DS12

Items:

• DS12 TWIN

Notes: Deep but with no hotswap. Thermals make no sense to me.

Cost: I have no idea what it costs.

2U

MyElectronics Dual Mini-ITX

Items:

• MyElectronics 6875

Notes: No hotswap, but is short depth. Looks nice, makes sense from a thermals perspective. Is built specifically for PicoPSU, and a normal PSU will not fit. Allows you to have front-io even without headers, via adapters that plug into the back of your system.

Cost: ~$295

S208

Items:

• GeneSys S208B-TWIN-ITX
• PlinkUSA TWIN-ITX-S2082

Notes: Hotswap. Nice layout. I reached out via email but could not get in contact with them.

Cost: $290

Threads:

• https://forums.servethehome.com/index.php?threads/2u-dual-itx-case.9386/

Travla / TAWA Series

Items:

• TAWA-T2240
• TAWA-T2241
• TAWA-T2242
• TAWA-T2280
• TAWA-T2900

All these are also available under the “Travla” brand here.

Notes: Number of different options. They tend to have really odd layouts internally. TAWA-T2900 has all front i/o which is unique.

Cost: I have no idea what it costs.

Threads:

• https://forums.servethehome.com/index.php?threads/travla-2u-t2280-t2281.10288/
• https://forums.servethehome.com/index.php?threads/dual-itx-cases.8230/
• https://www.reddit.com/r/sffpc/comments/ejek9k/travla_t2241_2u_dual_miniitx_case_19l_with_2x/

Cablematic CK018

Items:

• Cablematic CK01800

Notes: Deep but with no hotswap.

Cost: ~$125

RM-2270

Items:

• Circotech RM-2270
• KRI RM-2270

Notes: Deep but with no hotswap.

Cost: $279

iStarUSA D-218M2-ITX

Items:

• iStarUSA D-218M2-ITX

Notes: Deep but with no hotswap. There are some packages you can get that include PSUs, but they were out of stock when I looked.

Cost: ~$130

Threads:

• https://forums.servethehome.com/index.php?threads/dual-mitx-rackmount-2u-istarusa-d-218m2-itx.1303/

Misc Threads and other Articles

• https://www.reddit.com/r/homelab/comments/onfh15/2u4u_dual_itx_case/
• https://www.reddit.com/r/homelab/comments/u7fecx/looking_to_build_a_dual_mini_itx_server_rack/
• https://www.reddit.com/r/homelab/comments/l7ifrb/dual_mini_itx_rackmount_servers/
• https://www.reddit.com/r/homelab/comments/vskcrw/looking_for_unique_case_looking_for_a_rack_mount/
• https://forums.servethehome.com/index.php?threads/looking-for-dual-mini-itx-cases.26835/
• https://forums.servethehome.com/index.php?threads/4-mini-itx-boards-in-1u-chassi.10453/
• https://www.reddit.com/r/homelab/comments/4evfn6/dual_miniitx_rackmount_chassis/
• https://www.reddit.com/r/sffpc/comments/aeldtb/has_anyone_ever_made_a_smallish_itx_dual_system/
• https://www.reddit.com/r/sffpc/comments/hm4sr4/are_there_any_dual_itx_cases/
• https://www.reddit.com/r/sffpc/comments/wjsm4n/any_dual_itx_cases/
• https://www.reddit.com/r/homelab/comments/en84r2/mitx_multi_node_case/
• https://www.reddit.com/r/homelab/comments/u6062c/dual_server_chassis/

Recently I have been moving my homelab to Kubernetes. This has presented the need for a backup solution for any persistent data I might have there. For quite some time, I have been using Duplicati for my backups, but I haven’t been completely content with its performance, and have heard many horror stories of restores not working properly. So, I wanted to take this opportunity to find a backup solution that worked well for my personal computers (I have Windows, Linux, and Darwin hosts), my storage servers (UnRaid and FreeNAS), as well as Kubernetes. Assuming that such a solution exists, of course!

Choosing a Tool

There were a few solutions that I had heard mentioned lot on /r/homelab, and I took a look at all of them. Those being Duplicacy, Borg, and Restic.

Duplicacy seems like a good solution for some people, and the interface looked very nice. However, you do need to purchase a license to use all of its features, so I chose to avoid it unless I couldn’t find anything else that worked. I also wasn’t sure how I’d use it for any of my Kubernetes needs; it didn’t seem like a very popular use-case for the tool.

Borg and Restic both seemed like great tools. Ultimately, I decided to go with Restic purely because of its ecosystem, but again, they both seem like very nice solutions. Borg also probably has a better ecosystem for the majority of users, but I believe Restic’s is better in my particular case.

Restic didn’t have any particularly nice client GUIs that I could find, but for someone like me who likes to use version control as much as they can, there’s resticprofile, which is a fantastic tool that makes managing Restic on client machines very easy, and it works very well on my Windows, Linux, and Darwin hosts. I also found that Restic Browser could serve as a very usable GUI for doing restores. It’s still very bare-bones, but it does the job. Restic also has several solutions for interacting with K8s that looked very promising. Furthermore, Restic and all of these other tools are written in Go, which I very much prefer to Python, which is what Borg is written in. I assume this is one of the main reasons the Kubernetes ecosystem around Restic is so much more developed.

Integrating with Kubernetes

There are several tools out there that exist to make backing up persistent storage on Kubernetes with Restic much easier. Typically, they are operators that allow you define things like a backup schedule and what PVCs you want to be in which Restic repo. Again, I took a look at three relatively popular options.

The first product that I found was Stash. Stash is interesting because it has CRDs for a lot of different things you might want to backup or restore. I reached out to the sales team to see what an Enterprise License would cost (enterprise is needed for the most useful features), but they did not reply to me, I assume because I only have a few Kubernetes nodes to my name. From there, I was going to see if I could just build from source with license checks disabled, but it’s clear to me that at least some enterprise functionality isn’t present in the normal public repo, so that’s off the table as well.

Another very popular choice is Velero. However, I was immediately very apprehensive about it because it was made by Hepito, who sold out to VMware some time ago. This has led to a good amount of abandonware. It does look like Velero is still being supported, but it’s still important to realize that this acquisition altered the goals of the project. And I would have to pray that VMware does not alter them further. Additionally and very annoyingly, despite heavily using Restic, Velero does not support the Restic REST server backend. Meaning I would be hugely limited in my potential storage options.

Ultimately, I ended up going with K8up. In stark contrast to the other solutions I outlined, K8up is an active CNCF sandbox project, which makes me much more comfortable with using it. I really didn’t see any downsides to it for me personally, as it included most (if not all) of the enterprise features from Stash (such as support for backing up databases), and it also supports using the Restic REST server as a backend, which Velero was missing.

My Implementation

Below is a minimized account of how I implemented everything. For all the exact, ugly details, feel free to take a look at my homelab GitHub repo.

First, I created a backup namespace for everything centralized:

apiVersion: v1
kind: Namespace
metadata:
  name: backup

Next, I set up my Restic REST server. I used Rclone to do this, which basically allows you to use anything that Rclone supports as storage for Restic. I ended up creating a new helm chart for Rclone, just because I couldn’t find any existing ones that I liked very much. Unlike many others, it just runs rclone rcd, so you can use this chart for basically anything, and just send commands to serve/copy/sync/etc as needed.

Basically, the only extra values I supplied were to set my config file and add an extra port for Restic. This is my Kustomization:

helmCharts:
  - name: rclone
    repo: https://jacobcolvin.com/helm-charts/
    version: '0.3.0'
    releaseName: rclone
    namespace: backup
    valuesInline:
      image:
        repository: rclone/rclone
        tag: '1.60.1'

      configSecretName: rclone-config

      extraPorts:
        - name: restic
          containerPort: 50001
          protocol: TCP

I SSHed to the container and set up my remote ResticRemote. Then I saved this to my secret provider for the rclone-config secret, so it won’t be lost.

I then created a Job to run the following command after the sync completes to start the Restic server:

curl -v -X POST -H 'Content-Type: application/json' -d '{
  "_async": true,
  "_group": "job/restic",
  "command": "serve",
  "arg": ["restic", "ResticRemote:/"],
  "opt": {
    "addr": ":50001"
  }
}' http://rclone.backup.svc.cluster.local:5572/core/command

There are probably a lot of different ways to handle this, and I’m sure it’s mostly down to preference. So I won’t go into further detail on exactly how my Job is setup and such, but if you’re curious, it’s all public on my GitHub repo.

For my machines I wanted to back up, I used Traefik as an ingress for this. This is where I added things like authentication, certs, and such. Normally I use Cloudflare to proxy traffic, but in this case I thought it’d be better to not do that, as I am potentially sending quite a lot of data back and forth and don’t want to have to deal with any potential complications there. I also use both external-dns and cert-manager, so this was as simple as adding/replacing a few annotations, to disable proxying and switch to my Let’s Encrypt issuer:

'external-dns.alpha.kubernetes.io/cloudflare-proxied': 'false'
'cert-manager.io/issuer': 'letsencrypt-prod'

From there I was able to start using Restic on my personal machines. I used resticprofile to do the vast majority of the heavy lifting here. If you would like to see examples of the profiles I configured, you can check out my dotfiles repo.

Moving on to using this infrastructure to actually start backing up PVCs and such that are also hosted by Kubernetes. First, I installed the K8up Backup Operator. Note that the resources part is required, because they don’t include CRDs in the helm repo. You can also download the CRD and point to the file. Also, the BACKUP_GLOBAL_OPERATOR_NAMESPACE environment variable is important. It tells any Jobs in other namespaces that they should use the operator from the backup namespace. Obviously, you’d want to configure this differently if there were lots of people using one cluster.

helmCharts:
  - name: k8up
    repo: https://k8up-io.github.io/k8up
    version: '4.0.1'
    releaseName: k8up
    namespace: backup
    valuesInline:
      k8up:
        envVars:
          - name: BACKUP_GLOBAL_OPERATOR_NAMESPACE
            value: backup

resources:
  - https://github.com/k8up-io/k8up/releases/download/k8up-4.0.1/k8up-crd.yaml

With that installed, we can now use the Schedule CR to start backing things up. While in this configuration, the operator is centralized, the Schedule is not. There should be one CR in each namespace containing things you want to back up. Here’s an example Schedule for a foobar namespace.

apiVersion: k8up.io/v1
kind: Schedule
metadata:
  name: foobar-schedule
  namespace: foobar
spec:
  backend:
    rest:
      url: http://rclone-restic.backup.svc.cluster.local:50001/macropower/foobar
    repoPasswordSecretRef:
      name: restic-credentials
      key: repo-key
  backup:
    schedule: '0 4 * * *' # 04:00
    failedJobsHistoryLimit: 2
    successfulJobsHistoryLimit: 2
  check:
    schedule: '0 1 * * 1' # 01:00 on Monday
    failedJobsHistoryLimit: 2
    successfulJobsHistoryLimit: 2
  prune:
    schedule: '0 1 * * 0' # 00:00 on Monday
    failedJobsHistoryLimit: 2
    successfulJobsHistoryLimit: 2
    retention:
      keepLast: 3
      keepDaily: 7
      keepWeekly: 5
      keepMonthly: 12

Note that this Schedule has its very own repo to use at macropower/foobar, and also its own encryption key in the restic-credentials secret. A different namespace with its own Schedule could have its own repo, credentials, or any other attributes that aren’t configured on the operator.

Once the Schedule is created, anything inside the namespace it lives in (foobar in this example) can be backed up via an annotation. For example, below I have two PVCs, one for music and one for anime:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: music
  namespace: foobar
  annotations:
    'k8up.io/backup': 'true'
spec:
  # ...

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: anime
  namespace: foobar
spec:
  # ...

music has the backup annotation, so it will be backed up every day per our Schedule. However, anime does not have this annotation, so it will not be included in backups.

Here’s a diagram showing how everything works together:

Databases and Other Edge Cases

Lastly, to deal with databases, you of course can’t simply backup their PVC. Thankfully, K8up has a really simple way of addressing databases and basically any other edge cases. You can add annotations on the Pod itself, and K8up can run commands inside your containers to collect and backup data. Personally, I use TimescaleDB which is backed up almost exactly in the same way as Postgres. I was able to just add the following annotations:

podAnnotations:
  'k8up.io/backup': 'true'
  'k8up.io/backupcommand': sh -c 'PGUSER="postgres" PGPASSWORD="$PATRONI_SUPERUSER_PASSWORD" pg_dumpall --clean'
  'k8up.io/file-extension': .sql

This just creates a snapshot of the db.sql file resulting from the pg_dump command. I am not sure how or even if I could automate restores with Timescale, because they do require a bit of extra work compared to vanilla Postgres. But, hopefully this isn’t something I’ll have to do very often.

Conclusion

I hope someone found this article helpful. If you would like to see my exact and up-to-date implementation of everything above, please check out my homelab repo:

• https://github.com/MacroPower/homelab

And of course a huge thanks to the authors of the following projects:

• Restic
• K8up
• resticprofile
• Restic Browser

Read the blog post: https://grafana.com/blog/2021/07/30/how-to-use-grafana-and-prometheus-to-rickroll-your-friends-or-enemies/

ltrace -i -C ./baby

We’re prompted and can start off by inserting random value, e.g. asd.

A call to strcmp is intercepted by ltrace.

strcmp("asd\n", "ab[REDACTED]13\n")

Let’s start by trying to pass ab[REDACTED]13.

In this case it was just that simple, and we receive the flag.

HTB{B[REDACTED]Z}

There were probably a lot of additional ways this one could have been solved, and I think I got lucky by starting in this direction.