I'm back. Sorry for the delays. You'll see very quickly why things have been so quiet.
After reaching a new stage of lab hardware upgrades, I am doing an A/B transition from an old Docker Compose setup to start using Podman instead. The big reason being that Podman is inherently rootless by design, meaning that if a container breakout were to occur, the chances of them getting root privileges and causing serious damage is slightly dampened.
This post will cover my experience moving a Docker lab with quite a few containers over to Podman, and some of the hurdles that comes with some of them.
What's In A Pod
Podman is what I'd call a gateway technology that sits somewhere between both Docker itself and Kubernetes. Docker is great for homelab hackers who may have interest in becoming DevOps engineers, while Kubernetes is an at-scale compute solution that is not totally wieldy for homelabbers like myself. While I do plan on slowly learning how to leverage Kubernetes at home just for the learning experience, Podman is a pretty good place to start as it lifts a lot of Docker grammar and prepares you for a world of Kubernetes.
Docker has a lot of Docker-specific things like swarms, which is Docker's engine of managing large clusters of Docker hosts. Kubernetes' setup is not exactly simple, as there are lots of hoops to jump through, while Docker has swarm built into Docker out of the box. But, in my time perusing the internet, I simply do not see Docker swarm mentioned anywhere on the internet anymore, leading me to think it's not exactly the go-to choice for organizations and developers anymore.
Podman transitions users away from the idea of using a compose file to write their projects and instead lets users write their projects as Kube YAML instead. Podman has the ability to function as a Kubernetes runtime, so you could very well use Podman as the test bed and the real time deployment if you wish. There are projects to support compose files so the transition feels easier like podman-compose, but I feel you don't want to rely on that too much and are better off learning the commands yourself to generate fresh YAML easier. There could always be a slim chance of a compose file not working the way you think.
Lastly, there's the idea of the Pod itself. Podman could quite literally be interpreted as "pod manager", so surely what's the pod part imply? Kubernetes' main function is to inspect the state of a live system and determine if the live state matches that of the desired system describe from a manifest file. A selection of containers can be grouped up into a "pod" so that Kubernetes can see that the containers are designed to work together, and also make sure that networking-wise, same-pod containers can all talk to each other.
For my blog here, I run a reverse proxy and an HTTP file server. The proxy will handle outside internet traffic and direct it to the HTTP file backend. These two belong in a pod together, this way they can see each other and be managed together. If one crashes, it can be easily replaced and repaired. Or if I needed to scale up my hosting, the pod can easily be duplicated across cluster hosts. I can also manage multiple services on my host by separating them out into different pods as well, so managing multiple services like media servers or chat services are all independent from each other.
Okay, enough prefacing, let's dive in.
Guix and Podman
To make my technical setup worse, I am running my new server on Guix as part of my remote deployment setup. Guix is not a widely supported system on the internet, but it does have Podman support in a service called rootless-podman-service. I will install this into my existing services in order to start using Podman on my system.
(services
(cons*
(service dbus-root-service-type)
(service elogind-service-type)
(service openssh-service-type)
(service dhcpcd-service-type)
(service ntp-service-type)
(service iptables-service-type)
(let ([%ids (subid-range (name "user"))])
(service rootless-podman-service-type
(rootless-podman-configuration
(subgids %ids)
(subuids %ids))))
%base-services))
The Podman service per Guix needs to know the UID and GID information of the users expecting to use Podman, so nothing too complicated there. This works pretty much out of the box with no additional setup needed. Perform a sudo guix system reconfigure config.scm and then run the following:
podman run --name basic_httpd -d \
-p 8080:80/tcp docker.io/nginx
Then visit your http://localhost:8080 and you should see the Nginx starter page, verifying Podman is downloading OCI images and running them fine. For once, no complicated Guix problems.
Converting Services
I have a very long list of things I like to use. I would prefer if they were all part of one file such that if the computer reboots, it can use one file to replay the entire container setup with no issue. But that being said, the concept of pod grouping here would make more sense so that certain service containers are isolated from overlapping services. Sure you could make the case to share a Postgre container with other containers that rely on Postgre, but then you enter versioning hell where if two containers require different versions of Postgre, then you'd likely have to detangle them anyway.
Ideally, that shouldn't happen so much, but if you have compute bandwidth, then it really shouldn't be so much of an issue. The services I am going to try and experiment with here are:
- My website
- Pi-Hole
- Home Assistant server
- Immich
Website
Let's first examine the website. My website is a pretty plain HTML served website via Caddy's built-in file HTTP server. Caddy is great for reverse proxy traffic through VPN interfaces. Caddy is also pretty easy to use without much extra configuration, such that you can boot a Caddy service pretty much exclusively from the command line.
There isn't too much here, but for me to operate my website, I need to upload files somewhere. My dream later in life is to streamline my blog such that it can build new images on the fly and distribute across an array of hosts, but for now I need a place to put my files. We will leverage Caddy to serve files on a specific port using it's CLI arguments to do much of the lifting, meaning we don't really need a Caddyfile mounted if we can do it all from the arguments.
apiVersion: v1
kind: Pod
metadata:
labels:
app: blog
name: blog
spec:
containers:
- command:
- caddy
- file-server
- --listen
- :2020
- --root
- /www/public
- --browse
image: docker.io/library/caddy:latest
name: caddy
ports:
- containerPort: 2020
hostPort: 8002
volumeMounts:
- mountPath: /www
name: blog-data
readOnly: true
restartPolicy: Always
volumes:
- hostPath:
path: /home/steve/ste5e.net
type: Directory
name: blog-data
There's nothing terribly complicated. You can generate this from a podman run command and it simply passes in all the arguments that were used in that command into Kube YAML format. It's much simpler to start launching containers from the run command and then tweak it by putting them all into a Kube YAML and relaunching the pod with the file. After a while, Kube YAML starts looking like Docker Compose.
Pi-Hole
A few weeks ago we had a power outage that once again, for probably the third or fourth time in my life, completely took my Raspberry Pi 2 offline. And that may have marked the end of my journey with the Pi 2.
SD cards have a protection in mind that we don't really often see: a shut-down mechanism such that if the SD card is treading on being permanently damaged, in this case of an emergency shutdown due to loss of power, the SD card will protect itself by going read-only. Connecting it to an operating system, if you tried to write to the blocks itself via a /dev/sdX channel, the OS will act like everything is fine. However, if you mount the device and the filesystem, you will find yourself not being able to write to it at all anymore. Forever.
This flaw does not get seen very often because SD cards pair very well with battery-powered devices. If a device is about to be shut down, the system can safely stop writing to the SD card and shut down gracefully. It knows when the end of the system is because of the DC power in the batteries. However, a non-battery system like a Raspberry Pi does not have these guarantees at all. And with a device this old, the Raspberry Pi's reliance on the SD card cannot be switched, as it will always read from the SD card slot over anything else. You can mount a flash drive to the Raspberry Pi and redirect disk I/O to it instead, but the SD card is always needed for the boot process.
There are two options: get a new SD card, get a UPS backup system and ensure that voltage is always constant for the Raspberry Pi, or... Stop using it. I would need hundreds of dollars for a UPS, and even then they're very hefty still. The RPI2 I had in place functioned fine for almost a year and a half straight with very little downtime. Random storms probably don't warrant me spending more money at the moment. I would have to consistently buy SD cards for the RPI2 in the hopes to keep it going only for the SD card to be killed every now and again, unless I bought a UPS, which seems a little overkill for me still.
I want to cut out the SD card middleman, so I'm going with a different choice: leveraging my homelab hardware with disk drives instead of SD cards. Disks are a little more durable and easier to use. Pi-Hole also supports a container version so you don't necessarily need a Raspberry Pi to even use the thing. But this won't come as a free action, and we will have to do a couple intermediary steps.
First, DNS needs port 53 to be bindable. My router does not support routing DNS queries to non-standard DNS port servers, so it absolutely has to be bound to port 53. I am using Guix, and opening up the port access here means going into the sysctl configuration and changing the port range we can access with user programs. Podman needs ports to be made available on container creation, and if the port isn't bindable, then we can't do this.
The change is easy: add a sysctl-configuration that changes the privileged port range all the way down to 53. Is this risky? A little, but I trust my hardware to be fine for the most part. If I had a supported router that would do DNS queries on different ports, this would be a different story altogether. But, I decided to throw this into my config.scm and it works out for us by verifying it against sysctl -a.
(modify-services %base-services
(sysctl-service-type config =>
(sysctl-configuration
(settings
(append
(if dns
'(("net.ipv4.ip_unprivileged_port_start" . "53"))
'())
%default-sysctl-settings)))))
One day I might have a custom router computer instead and maybe the non-flexible DNS lookup port can be modified to avoid going into the restricted range. But anyways, the port is opened up and now we can bind. Pi-Hole is thankfully only one container image, so not a whole lot to do here other than to make the DNS port available to the pod, pass through the web traffic from a non-overlapping port, and we will have an operational Pi-hole server once the DNS assignment gets changed on whatever router you give it. Plus or minus a couple hours for DNS to start propogating across your devices.
Now, what's the container look like? Nothing terribly crazy. A container image, a mount for files so we can edit files outside of the container, and binding the pod to port 53.
apiVersion: v1
kind: Pod
metadata:
labels:
app: pihole
name: pihole
spec:
containers:
- env:
- name: TZ
value: America/New_York
- name: FTLCONF_dns_listeningMode
value: ALL
envFrom:
- secretRef:
name: pihole-secrets
image: docker.io/pihole/pihole:latest
name: server
ports:
- containerPort: 53
hostPort: 53
- containerPort: 53
hostPort: 53
protocol: UDP
- containerPort: 80
hostPort: 8001
volumeMounts:
- mountPath: /etc/pihole
name: pihole-data
volumes:
- hostPath:
path: /home/steve/pihole/pihole_etc
type: Directory
name: pihole-data
The web server does use a password for basic authentication. That password is stored in a Secrets file separately. I will talk more about that later, but for now, pretend the WEBPASSWORD environment variable is there and has a password you can use to log in with. I would do some debugging on this pod with tools like dig or dnslookup to ensure that I can actually make DNS queries, then swapped it over on my router and let the queries slowly propogate through the Pi-Hole pod itself. It takes a while but eventually everything will get routed through the Pi-Hole pod.
Immich
Immich is my go-to photo and mixed media manager. I don't really care much for organizing photos all the time, but Immich makes it pretty dead simple. The way we have it now, photos can come from all over, but the flow on organizing photos and media into albums and such is not an easily shared story across various devices. If you upload everything into Immich by default, you can manage it in one place.
Immich works well, but it has a lot of moving parts: a web server written in Typescript, a database like Postgre, a Redis-like server implementation like Valkey, and a machine learning service to identify faces and help tag by people. I have used it for a year and it's a relatively decent web hosting option for photos and videos, though I can't say if it's a true end-to-end backup solution or not.
This setup is pretty easy if you go down the list and convert their recent Docker Compose file and run it through Podman one at a time. You will end up with something like this.
apiVersion: v1
kind: Pod
metadata:
name: immich
labels:
app: immich
annotations:
spec:
volumes:
- hostPath:
path: /home/steve/Selfhosted/immich/library
type: DirectoryOrCreate
name: immich-data-host
- hostPath:
path: /home/steve/Selfhosted/immich/model-cache
type: DirectoryOrCreate
name: immich-model-cache-host
- name: immich-psql
hostPath:
path: /home/steve/Selfhosted/immich/postgre
type: DirectoryOrCreate
- hostPath:
path: /home/steve/Selfhosted/immich/redis
type: DirectoryOrCreate
name: immich-redis-host
containers:
- name: server
image: ghcr.io/immich-app/immich-server:v3
volumeMounts:
- mountPath: /usr/src/app/upload
name: immich-data-host
ports:
- containerPort: 2283
hostPort: 8088
envFrom:
- configMapRef:
name: immich-config
optional: false
- name: machine-learning
args:
- ./start.sh
image: ghcr.io/immich-app/immich-machine-learning:v3
volumeMounts:
- mountPath: /cache
name: immich-model-cache-host
envFrom:
- configMapRef:
name: immich-config
optional: false
- name: psql
image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: immich-psql
env:
- name: POSTGRES_USER
valueFrom:
configMapKeyRef:
name: immich-config
key: DB_USERNAME
- name: POSTGRES_PASSWORD
valueFrom:
configMapKeyRef:
name: immich-config
key: DB_PASSWORD
- name: POSTGRES_DB
valueFrom:
configMapKeyRef:
name: immich-config
key: DB_DATABASE_NAME
- name: POSTGRES_INITDB_ARGS
value: "--data-checksums"
- name: redis
image: docker.io/valkey/valkey:9@sha256:3acc0687f2a2e1091fae6450d7842dd658c941338cf0a873ddd9e14b9e4ea4dd
restartPolicy: Always
This one is terribly long since there's so much, and that's not even including all the secrets we need from the accompanying secrets file they provide. You will have to map those out as a ConfigMap or Secret object as well. But if everything works out, you can connect to the web port and start uploading photos pretty immediately.
The one downside to Immich is that its migration story is always a little suspect to change, but I feel this will get better with a few more years behind it. My previous server was on a v2 setup, and it didn't have good functionality for exporting all my data like albums and face tags. When migrating to v3, I ended up downloading all my photos locally and re-uploading them again, which is sort of annoying. But, I made it a note to not rely on Immich for all my photos but rather the photos I want to put online. Not every photo I take with my phone is deserving of the internet.
Another thing to note is that if you wish to run the machine learning, Immich by default will refer to the Docker hostname of the ML service, and you have to change it in settings to refer simply to localhost:3003 because Podman does not generate host identifiers per container, and everything is simply part of the pod with a localhost address. If you don't do this, then Immich will never be able to locate the ML service. The ML service is needed for face scanning so you can find and tag people in photos, which is quite handy for when people ask about photos of themselves from events.
Home Assistant
I use Home Assistant for very basic things like turning on a power plug or adjusting the occasional lightbulb or two. It helps to have some electronics on a timer so you can have lights be dimmed or turned off entirely. I go to bed pretty early and wake up early, so having lights turned off when I'm going to bed and turned on when I wake up help me a lot.
Home Assistant also happens to be the most annoying to customize in a declarative manner because of some nuances to how Home Assistant works: there are no environment variables you can inject at runtime, and the configuration is mostly done from the interface once the server is running. That means something like say choosing the port to open on the pod for Home Assistant is not strictly tied with the actual container, but configuration information exterior to the manifest itself.
Then there's the Matter server, something we need in order to communicate with physical devices. I picked Matter smart plugs over any other protocol like Zigbee or Z-wave because it seems like Matter is where we are slowly all moving towards. However, Home Assistant is a proxy to communicate with different protocol services, not necessarily the thing that can speak to all these different protocols. End of the day, we need a Matter server, so we can use the MatterJS server. I don't know of any other ones currently, but I pray they're all roughly the same in terms of functionality.
This setup was frankly the most painful in terms of solving. Matter needs mDNS device discovery in order to talk to smart devices on a network. When Home Assistant wants to connect a Matter device to its setup, it has to pass a Matter server IP for the device to talk to in order to relay data. IP discovery is sort of shaky in a rootless container setup, and the best I can do is use Podman's host network mode to skip over some of the weird nuances of both Home Assistant and MatterJS.
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2026-08-13T21:04:04Z"
labels:
app: homeassist
name: homeassist
spec:
restartPolicy: Always
# no macvlan, pure host-mode a la Docker
hostNetwork: true
hostname: boringbox
containers:
- image: docker.io/homeassistant/home-assistant:stable
name: ha-server
volumeMounts:
- mountPath: /config
name: homeassistant-config-data
- image: ghcr.io/matter-js/matterjs-server:stable
name: matterserver
volumeMounts:
- mountPath: /data
name: matter-config-data
volumes:
- hostPath:
path: /home/steve/homeassistant/config
type: DirectoryOrCreate
name: homeassistant-config-data
# this folder must be chmod 0777 for non-root user
- hostPath:
path: /home/steve/homeassistant/matterdata
type: DirectoryOrCreate
name: matter-config-data
I tried doing dual-pod setups so that Matter would live on its own IP address with a host network attachment, but then it created weird issues of getting in contact with Home Assistant that lived in a different isolated pod. Simply exposing Home Assistant to the host network namespace proved to be far simpler here. I'm sure with enough patience you could play around with pod settings and have it working without needing host mode, but I'm not sure if I have the willpower to keep experimenting.
I log into Home Assistant on http://host:8123, create an account, go to Settings > Network and change the web listening port through the interface. Then swap over to your ideal port so you're not camping on the default Home Assistant port, and change your URL to the new port and you should still be set. Then set up a Matter bridge with ws://localhost:5580/ws as the URL and you can start adding Matter-supported devices.
One thing to note, at least with my TP-Link Tapo plugs, these only work on a 2.4Ghz network. I had to disable Self-organizing WiFi to separate my home WiFi into two channels in order to use these plugs. The SON can deliberate which band channel would be best for a device, and if a smart device used that, then it would bounce between an unsupported 5Ghz band and the 2.4Ghz regular one.
Combining and Autostarting
Podman has good support to create what are called quadlets that allow easier integration into systemd, but with Guix we can't use this. Instead, we'll have to do a little manual work to ensure an auto-start so our services will resume if the computer reboots while we're away.
It's possible to launch multiple pods at once, and if that's your thing, you can treat each pod as if it were a docker-compose.yml and maintain it that way. But, I feel it slightly easier to keep all pods in one file and launch it with one simple podman kube play pods.yaml.
The one thing I've yet to mention is how to manage environment variables and secrets. Podman has a secrets system, but I'm still pretty amateur-ish. My secrets are mostly scrambled passwords that I make up for a system, and that system will pretty much never surface online anyway. I find it easier to manage a file by storing it in something like a .env, which is the lazier approach. The secret system is far superior, but adds another management layer.
The option I went with is creating a secrets.yaml file that I supply into the pods playbook at the end of a command like podman kube play pods.yaml secrets.yaml. As the YAML is parsed, it looks to see if there are any YAMLs that match kind: Secret. We can write a Secret object with the following:
apiVersion: v1
kind: Secret
metadata:
name: pod1-secrets
type: Opaque
stringData:
database_password: "Hello!"
---
apiVersion: v1
kind: Secret
metadata:
name: pod2-secrets
type: Opaque
stringData:
other_password: "World!"
---
...
These are then injected into the env section of a Pod YAML.
apiVersion: v1
kind: Pod
metadata:
labels:
app: pod1
name: pod1
spec:
containers:
- env:
- name: HardCoded
value: FlatValue123
envFrom:
- secretRef:
name: pod1-secrets
All the secrets for that pod are then loaded into the container's runtime. You can break up your secrets by container, or by pod, or however you prefer it really. The less information shared the better, you will most likely not want to share all secrets with all containers, so you should split it up where it makes sense to reduce information leakage. If a pod has a database and the password needs to be shared, then it makes sense to share it at a pod level.
The thing that needs to happen is for Podman to be able to run this at startup, meaning we need some user-level service that will execute a Podman playbook. This can be done in two ways by leveraging features of the Guix OS that we build, or we can leverage a Guix Home profile. Contextually, I believe it makes more sense to use Home, so I will show that in a little bit.
Same as the secrets, we can glue specifications together in the same file by using --- separators. This will get read into the system in bulk, and Podman will pick apart anything that looks like a kind: Pod and begin playing the entire pod out from scratch. So while I've been careful in doing very simple container runs adding them to pods, we can now export all our pods and merge them into one new YAML file that can launch all pods at once.
apiVersion: v1
kind: Pod
metadata:
labels:
name: pod1
name: pod1
spec:
- containers:
...
---
apiVersion: v1
kind: Pod
metadata:
labels:
name: pod2
name: pod2
spec:
- containers:
...
---
...
Lastly, we need some code that will start up the pods via podman kube play pods.yaml secrets.yaml command. Here is what I ended up going with as a Guix Home profile.
(simple-service 'autostart-podman home-shepherd-service-type
(list
(shepherd-service
(provision '(podman-pods))
(documentation "Launch podman pods")
(one-shot? #t)
(start #~(lambda ()
(zero? (system* #$(file-append podman "/bin/podman")
"play" "kube" "--replace"
(string-append (getenv "HOME") "/pods.yaml")
(string-append (getenv "HOME") "/secrets.yaml")))))
(stop #~(lambda ()
(zero? (system* #$(file-append podman "/bin/podman")
"play" "kube" "--down"
(string-append (getenv "HOME") "/pods.yaml"))))))))
Adding this to the home services, it will attempt to boot up the pods and it will generally be fine so long as the pods were battle tested before. It will attempt to bring them all down gracefully if the system is shutting down with the inverse command of pod play kube --down.
One thing to note, if you replace all the pods forcefully during testing, if you are force replacing the Pi-Hole pod itself, expect your internet to become a little wonky. For new DNS requests, if the DNS server is down, then you may have some turbulent internet while the pod is brought back up. It doesn't take terribly long, but obviously it is a little scary with containers that the connectivity of your internet relies on a container coming back.
Final Thoughts
I didn't have much reason before to move off Docker. Docker works, and it is largely fine. Millions of people will still use Docker, the world still uses Docker in various random parts.
But, the rise of security vulnerabilities appearing across the globe has made Docker appear like a lesser choice. Kubernetes replaced Docker entirely. Docker tried cementing themselves as the de facto container system by making sure they were the only ones viewed when the name "container" was brought up in conversation. It'd be defamatory of me to say Docker is bad, when it's really not all that bad and will continue to work for a long time. My personal experience is that I think I simply like Podman a little better so far.
I'm not sure for how long Podman will also be until a different solution inevitably arrives. Maybe there's a critical glaring issue with Podman I don't see right now, and maybe more open source tooling around OCI will evolve. Maybe there will be a more native way to approach containers in Guix (even though there sort of is already). Podman feels like a good replacement right now, and maybe one day I'll finally settle on using full-blown Kubernetes instead.
I apologize if this post is terribly long, but I thought it would be a good post to showcase that it is not that difficult to leave Docker with a bit of determination. Once I did Podman long enough, it sort of became second nature, and once you learn about the networking aspect between containers, its sort of easy to figure out from there what you can and can't do.
If anything happens to this website, you can most likely blame my Podman skills.
Addendum: Hardware Changes
I have three servers in my lab, and one third of them is now using my on-going Guix configuration project. The Guix one is managed remotely via guix deploy, and the other servers are both Fedora Server boxes, simply because it was an easier upfront choice time-wise rather than learning more Guix at the time.
Most of the Docker files sat on a Fedora host, and now that my containers are going to live on the Guix host (which has a better CPU anyway), the inferior Fedora host can be wiped and replaced with Guix. At least, that's the intention. Once I reach parity on my Podman server with my Fedora Docker host, the Fedora can be replaced with Guix, and then managed via my Guix machine deploy configuration.
The last Fedora server will take a little more time as that's my NAS that I use for backups. I want to revisit that one later and come up with some scripts to do routine backups to external hard drives. Now that we are in a period of time with expensive hardware, it is time to squeeze every last penny out of everything I have laying around.