Ste5e's Site

Purging CA Certs

2026-10-23T20:00:00Z

This week has been a very weird week. A few days ago, eMudhra's private key was leaked. It's private key was revealed to be the value 4; yes, the actual integer four. With this knowledge, it would be fully possible for someone to sign a valid SSL certificate and make it appear valid to the gigantic trust chain system, meaning anyone who employs an eMudhra certificate would fully believe a forged certificate.

This is a pretty large issue for the internet due to how certificate authorities work. Each one consigns trust to one another, meaning that if one authority trusts the other, then a certificate signed under one CA would be trusted by the other due to the math of elliptic curves. However, this only holds if private keys are never known, which eMudhra's key is now widely available.

So in short, it's pretty bad. But also may have funny implications elsewhere, but I'll theorize about that later.

Updating Guix

Upon breaking news of eMudhra going sour, I waited until Guix had a commit land to purge the CA from nss-certs. As of 2143f3c7cd, the NSS certificate package was updated to purge eMudrha's CA entirely.

Ideally at the end of it all, a curl call should look something like this:

$ curl https://dv-g3b-emudhra.we-have-a-problem.com/
curl: (60) SSL certificate verification failed: certificate signer not trusted. (CAfile: /run/current-system/profile/etc/ssl/certs/ca-certificates.crt CRLfile: none)
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.

Guix works in a few steps, but the best place to start is by doing guix pull. If you define your operating system with a system configuration and a home profile, and perhaps servers with deploy, the logical execution would look something like this:

guix pull
sudo guix system reconfigure file.scm
guix home reconfigure home.scm
guix deploy servers.scm

This is going to be a long update since a lot of people are downloading similar updates to get ahead of any future attack vectors. It took me a few hours to pull all the commits and files in to update my system and all my servers as well.

At the end of it, after a reboot, I can no longer curl an eMudhra-signed website. Hooray!

Podman / Docker

Now for problem number two: these CA trusts were absorbed into containers when the containers were built from images. If you did a docker-compose up or podman run at any point prior to this week, you are most likely not safe. The images will reference the operating system's store at build time, meaning the images are frozen with the CA trusts, and will not reflect any changes without being completely rebuilt.

But, this isn't so bad right? Containers are meant to be immutable and stateless, with the state being stored on the filesystem that gets mounted into the containers. If you run your containers correctly, then your software versions should all be pinned to a frozen version, you take everything down, delete all the containers and images, and simply pull it all down as you do your container plays. Not too bad, right?

My use-case is that my Pi-hole lives in my pod stack, and for me to do this would mean I would have to go into my router and disable my IPV4 DNS setting. Sucks, but it is what it is, otherwise once you tear down everything, you will no longer be able to query for the image server. Woops! Don't make it a habit to do this very often, but this is a pretty emergency scenario after all.

My Podman steps went as follows:

podman kube play --down pods.yaml  # take everything down
podman rmi -a --force              # wipe container images
podman kube play pods.yaml         # play all the pods

Docker is a little similar.

docker-compose down
docker container prune
docker image prune -a
docker-compose up -d

If you now run something like this:

$ podman exec -it <container-id> curl https://dv-g3b-emudhra.we-have-a-problem.com/
curl: (60) SSL certificate verification failed: certificate signer not trusted. (CAfile: /run/current-system/profile/etc/ssl/certs/ca-certificates.crt CRLfile: none)
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.

The container should have curl to use as a test, but point being if you remove the CA from your operating system, then the containers should be good once you re-pull everything in.

Conclusions

Update your CA stores, please! Not every application in the world will calculate trust correctly. Upgrade your browsers, upgrade your operating systems soon, and follow any advisories for programming languages specifically. For example, Python's pip package manager maintains its own CA store separately, meaning pip needs to get updated to 26.2.2!

New Looks

2026-10-18T15:00:00Z

Hello! Welcome to the new site. In this I will cover the changes we are undergoing currently.

What's New

Per my most recent post, I said I was going to start converting over my site from Zola to Haunt. I had a lot of hesitation in doing this since it would involve a very non-trivial amount of effort. Zola's posts are formatted with a hybrid of TOML and Markdown, while Haunt supports other types of formats but can also support CommonMark.

Changing the format for posts to be stored meant a lot of changes to individual files. I have been writing posts for five years, so this wasn't an easy thing to motivate myself to do. As I started, I realized instead of trying to support all my posts over the course of five years, I should take the time to trim the fat: get rid of things I wasn't totally happy with.

In turn, a lot of my older posts are going in the dumpster. There were many posts I wasn't totally satisfied with; information no longer relevant, code no longer supported, or projects I was not working on. Many dead-end projects I had ended up spanning multiple posts, and I felt it not totally right to hold onto pages for things no longer actively in use.

I am a fan of my work on deep-dive projects like my work exploring [MyHouse.wad](), or write-ups on hardware like my efforts on my [data vault](). I am proud of those quite a bit. I didn't care for my monthly updates anymore, as I no longer follow any strict schedule system for this website. It didn't feel right to do a lot of labor for posts I am no longer happy with.

Reasons for Haunt

Haunt I had known about for a while, but felt it was too early to consider seriously using it. Haunt even slightly resembles what I eventually derived on my own in Racket in an older project I no longer use! I felt slight enticement to join the Haunt users out there because some things started to make sense in my head.

Instead of templates being literal HTML files with substitute insertion points, templates are simply Scheme expression trees
Incorrect template structure can now be verified by the Guile runtime itself, instead of needing a template parser to first read a file and attempt to parse it into a template tree
Standard library functions for the template system is simply Guile itself, you can do a lot more when you don't need a template engine to define a standard library for you
The template format itself is not likely to change; Lisp syntax hasn't changed in decades
Being able to control the builder functions for your site to dictate what actions will get performed, quite a bit like Makefiles, but again, validated by the Guile runtime for a small layer of correctness

I had a lot of shared ideas while working on my own implementation with Racket, but eventually dropped it due to the amount of time required. Knowing Haunt has some similar traits, I'm more than happy to start using Haunt and restart my website.

What's Next

It's going to take a lot of time for me to properly implement all the things this website needs for me to be happy. I am using a basic CSS definition for now, but it is going to need some improvements slowly. My posts I am keeping are all here, and posts I no longer want are technically gone. Then, once things settle down, I will migrate my entire homelab over to the new server, which I still have not totally finished yet.

Huge shout-out to Dave Thompson for creating Haunt and basically acting as my source material for my change-over to Haunt. Excited to start tweaking it to my own custom design and implement some new ideas using Haunt as the basis. There may be some growing pains for a bit here, but I feel it's what I need to get creative again.

Where Next

2026-10-09T20:00:00Z

I haven't taken much time to sit down and write anything here for a while. I am on a new work schedule, so my after-work hours are usually a little drained. I said I would write more openly about things I am actively engaged with as far as projects, but I don't have a main project focus currently as there's too much going on in real life to think about big projects in that sense. Respectively, everything I write here on out may be considered skill enhancement learning so I can better my knowledge in various weak spots.

Road Ahead

Adjusting to my new schedule has proven challenging in what it permits me to accomplish. While I still bring my Surface Pro 7 with me most days, I sometimes feel the need to decouple myself from computers being involved in every waking moment of my life. I started bringing a moleskine notebook so I can start jotting down math problems, like I used to do for fun 10+ years ago.

I came down with a sickness I am still dealing with last week, but I am hoping to start immediately doing work changing everything here over to Haunt relatively soon. I will be copying this entire repo into a new folder and making the changes until I have a successful built site. The layout may change too based on how difficult it is to do the conversion.

Hoping by the end of the month, I will have my new homelab online and possibly under a new domain. I plan to self-host my own Git repositories with Forgejo to make that my online portfolio of sorts, and am still exploring what else I'd like to host online publicly. I think the interesting thing about adding a VPN to the mix is the ability to choose what I make public and what I would expose only to devices connected on the VPN, accessible only by IP addresses on the network. Something for me to consider.

So, next time I write here, this will be a new site in many different ways. Cheers!

Podman

2026-08-22T20:00:00Z

I'm back. Sorry for the delays. You'll see very quickly why things have been so quiet.

After reaching a new stage of lab hardware upgrades, I am doing an A/B transition from an old Docker Compose setup to start using Podman instead. The big reason being that Podman is inherently rootless by design, meaning that if a container breakout were to occur, the chances of them getting root privileges and causing serious damage is slightly dampened.

This post will cover my experience moving a Docker lab with quite a few containers over to Podman, and some of the hurdles that comes with some of them.

What's In A Pod

Podman is what I'd call a gateway technology that sits somewhere between both Docker itself and Kubernetes. Docker is great for homelab hackers who may have interest in becoming DevOps engineers, while Kubernetes is an at-scale compute solution that is not totally wieldy for homelabbers like myself. While I do plan on slowly learning how to leverage Kubernetes at home just for the learning experience, Podman is a pretty good place to start as it lifts a lot of Docker grammar and prepares you for a world of Kubernetes.

Docker has a lot of Docker-specific things like swarms, which is Docker's engine of managing large clusters of Docker hosts. Kubernetes' setup is not exactly simple, as there are lots of hoops to jump through, while Docker has swarm built into Docker out of the box. But, in my time perusing the internet, I simply do not see Docker swarm mentioned anywhere on the internet anymore, leading me to think it's not exactly the go-to choice for organizations and developers anymore.

Podman transitions users away from the idea of using a compose file to write their projects and instead lets users write their projects as Kube YAML instead. Podman has the ability to function as a Kubernetes runtime, so you could very well use Podman as the test bed and the real time deployment if you wish. There are projects to support compose files so the transition feels easier like podman-compose, but I feel you don't want to rely on that too much and are better off learning the commands yourself to generate fresh YAML easier. There could always be a slim chance of a compose file not working the way you think.

Lastly, there's the idea of the Pod itself. Podman could quite literally be interpreted as "pod manager", so surely what's the pod part imply? Kubernetes' main function is to inspect the state of a live system and determine if the live state matches that of the desired system describe from a manifest file. A selection of containers can be grouped up into a "pod" so that Kubernetes can see that the containers are designed to work together, and also make sure that networking-wise, same-pod containers can all talk to each other.

For my blog here, I run a reverse proxy and an HTTP file server. The proxy will handle outside internet traffic and direct it to the HTTP file backend. These two belong in a pod together, this way they can see each other and be managed together. If one crashes, it can be easily replaced and repaired. Or if I needed to scale up my hosting, the pod can easily be duplicated across cluster hosts. I can also manage multiple services on my host by separating them out into different pods as well, so managing multiple services like media servers or chat services are all independent from each other.

Okay, enough prefacing, let's dive in.

Guix and Podman

To make my technical setup worse, I am running my new server on Guix as part of my remote deployment setup. Guix is not a widely supported system on the internet, but it does have Podman support in a service called rootless-podman-service. I will install this into my existing services in order to start using Podman on my system.

(services
 (cons*
  (service dbus-root-service-type)
  (service elogind-service-type)
  (service openssh-service-type)
  (service dhcpcd-service-type)
  (service ntp-service-type)
  (service iptables-service-type)
  (let ([%ids (subid-range (name "user"))])
   (service rootless-podman-service-type
    (rootless-podman-configuration
     (subgids %ids)
     (subuids %ids))))
  %base-services))

The Podman service per Guix needs to know the UID and GID information of the users expecting to use Podman, so nothing too complicated there. This works pretty much out of the box with no additional setup needed. Perform a sudo guix system reconfigure config.scm and then run the following:

podman run --name basic_httpd -d \
  -p 8080:80/tcp docker.io/nginx

Then visit your http://localhost:8080 and you should see the Nginx starter page, verifying Podman is downloading OCI images and running them fine. For once, no complicated Guix problems.

Converting Services

I have a very long list of things I like to use. I would prefer if they were all part of one file such that if the computer reboots, it can use one file to replay the entire container setup with no issue. But that being said, the concept of pod grouping here would make more sense so that certain service containers are isolated from overlapping services. Sure you could make the case to share a Postgre container with other containers that rely on Postgre, but then you enter versioning hell where if two containers require different versions of Postgre, then you'd likely have to detangle them anyway.

Ideally, that shouldn't happen so much, but if you have compute bandwidth, then it really shouldn't be so much of an issue. The services I am going to try and experiment with here are:

My website
Pi-Hole
Home Assistant server
Immich

Website

Let's first examine the website. My website is a pretty plain HTML served website via Caddy's built-in file HTTP server. Caddy is great for reverse proxy traffic through VPN interfaces. Caddy is also pretty easy to use without much extra configuration, such that you can boot a Caddy service pretty much exclusively from the command line.

There isn't too much here, but for me to operate my website, I need to upload files somewhere. My dream later in life is to streamline my blog such that it can build new images on the fly and distribute across an array of hosts, but for now I need a place to put my files. We will leverage Caddy to serve files on a specific port using it's CLI arguments to do much of the lifting, meaning we don't really need a Caddyfile mounted if we can do it all from the arguments.

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: blog
  name: blog
spec:
  containers:
  - command:
    - caddy
    - file-server
    - --listen
    - :2020
    - --root
    - /www/public
    - --browse
    image: docker.io/library/caddy:latest
    name: caddy
    ports:
    - containerPort: 2020
      hostPort: 8002
    volumeMounts:
    - mountPath: /www
      name: blog-data
      readOnly: true
  restartPolicy: Always
  volumes:
  - hostPath:
      path: /home/steve/ste5e.net
      type: Directory
    name: blog-data

There's nothing terribly complicated. You can generate this from a podman run command and it simply passes in all the arguments that were used in that command into Kube YAML format. It's much simpler to start launching containers from the run command and then tweak it by putting them all into a Kube YAML and relaunching the pod with the file. After a while, Kube YAML starts looking like Docker Compose.

Pi-Hole

A few weeks ago we had a power outage that once again, for probably the third or fourth time in my life, completely took my Raspberry Pi 2 offline. And that may have marked the end of my journey with the Pi 2.

SD cards have a protection in mind that we don't really often see: a shut-down mechanism such that if the SD card is treading on being permanently damaged, in this case of an emergency shutdown due to loss of power, the SD card will protect itself by going read-only. Connecting it to an operating system, if you tried to write to the blocks itself via a /dev/sdX channel, the OS will act like everything is fine. However, if you mount the device and the filesystem, you will find yourself not being able to write to it at all anymore. Forever.

This flaw does not get seen very often because SD cards pair very well with battery-powered devices. If a device is about to be shut down, the system can safely stop writing to the SD card and shut down gracefully. It knows when the end of the system is because of the DC power in the batteries. However, a non-battery system like a Raspberry Pi does not have these guarantees at all. And with a device this old, the Raspberry Pi's reliance on the SD card cannot be switched, as it will always read from the SD card slot over anything else. You can mount a flash drive to the Raspberry Pi and redirect disk I/O to it instead, but the SD card is always needed for the boot process.

There are two options: get a new SD card, get a UPS backup system and ensure that voltage is always constant for the Raspberry Pi, or... Stop using it. I would need hundreds of dollars for a UPS, and even then they're very hefty still. The RPI2 I had in place functioned fine for almost a year and a half straight with very little downtime. Random storms probably don't warrant me spending more money at the moment. I would have to consistently buy SD cards for the RPI2 in the hopes to keep it going only for the SD card to be killed every now and again, unless I bought a UPS, which seems a little overkill for me still.

I want to cut out the SD card middleman, so I'm going with a different choice: leveraging my homelab hardware with disk drives instead of SD cards. Disks are a little more durable and easier to use. Pi-Hole also supports a container version so you don't necessarily need a Raspberry Pi to even use the thing. But this won't come as a free action, and we will have to do a couple intermediary steps.

First, DNS needs port 53 to be bindable. My router does not support routing DNS queries to non-standard DNS port servers, so it absolutely has to be bound to port 53. I am using Guix, and opening up the port access here means going into the sysctl configuration and changing the port range we can access with user programs. Podman needs ports to be made available on container creation, and if the port isn't bindable, then we can't do this.

The change is easy: add a sysctl-configuration that changes the privileged port range all the way down to 53. Is this risky? A little, but I trust my hardware to be fine for the most part. If I had a supported router that would do DNS queries on different ports, this would be a different story altogether. But, I decided to throw this into my config.scm and it works out for us by verifying it against sysctl -a.

(modify-services %base-services
 (sysctl-service-type config =>
   (sysctl-configuration
     (settings
       (append
         (if dns
             '(("net.ipv4.ip_unprivileged_port_start" . "53"))
             '())
         %default-sysctl-settings)))))

One day I might have a custom router computer instead and maybe the non-flexible DNS lookup port can be modified to avoid going into the restricted range. But anyways, the port is opened up and now we can bind. Pi-Hole is thankfully only one container image, so not a whole lot to do here other than to make the DNS port available to the pod, pass through the web traffic from a non-overlapping port, and we will have an operational Pi-hole server once the DNS assignment gets changed on whatever router you give it. Plus or minus a couple hours for DNS to start propogating across your devices.

Now, what's the container look like? Nothing terribly crazy. A container image, a mount for files so we can edit files outside of the container, and binding the pod to port 53.

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: pihole
  name: pihole
spec:
  containers:
  - env:
    - name: TZ
      value: America/New_York
    - name: FTLCONF_dns_listeningMode
      value: ALL
    envFrom:
      - secretRef:
          name: pihole-secrets
    image: docker.io/pihole/pihole:latest
    name: server
    ports:
    - containerPort: 53
      hostPort: 53
    - containerPort: 53
      hostPort: 53
      protocol: UDP
    - containerPort: 80
      hostPort: 8001
    volumeMounts:
    - mountPath: /etc/pihole
      name: pihole-data
  volumes:
  - hostPath:
      path: /home/steve/pihole/pihole_etc
      type: Directory
    name: pihole-data

The web server does use a password for basic authentication. That password is stored in a Secrets file separately. I will talk more about that later, but for now, pretend the WEBPASSWORD environment variable is there and has a password you can use to log in with. I would do some debugging on this pod with tools like dig or dnslookup to ensure that I can actually make DNS queries, then swapped it over on my router and let the queries slowly propogate through the Pi-Hole pod itself. It takes a while but eventually everything will get routed through the Pi-Hole pod.

Immich

Immich is my go-to photo and mixed media manager. I don't really care much for organizing photos all the time, but Immich makes it pretty dead simple. The way we have it now, photos can come from all over, but the flow on organizing photos and media into albums and such is not an easily shared story across various devices. If you upload everything into Immich by default, you can manage it in one place.

Immich works well, but it has a lot of moving parts: a web server written in Typescript, a database like Postgre, a Redis-like server implementation like Valkey, and a machine learning service to identify faces and help tag by people. I have used it for a year and it's a relatively decent web hosting option for photos and videos, though I can't say if it's a true end-to-end backup solution or not.

This setup is pretty easy if you go down the list and convert their recent Docker Compose file and run it through Podman one at a time. You will end up with something like this.

apiVersion: v1
kind: Pod
metadata:
  name: immich
  labels:
    app: immich
  annotations:
spec:
  volumes:
    - hostPath:
        path: /home/steve/Selfhosted/immich/library
        type: DirectoryOrCreate
      name: immich-data-host
    - hostPath:
        path: /home/steve/Selfhosted/immich/model-cache
        type: DirectoryOrCreate
      name: immich-model-cache-host
    - name: immich-psql
      hostPath:
        path: /home/steve/Selfhosted/immich/postgre
        type: DirectoryOrCreate
    - hostPath:
        path: /home/steve/Selfhosted/immich/redis
        type: DirectoryOrCreate
      name: immich-redis-host
  containers:
    - name: server
      image: ghcr.io/immich-app/immich-server:v3
      volumeMounts:
      - mountPath: /usr/src/app/upload
        name: immich-data-host
      ports:
      - containerPort: 2283
        hostPort: 8088
      envFrom:
      - configMapRef:
          name: immich-config
          optional: false
    - name: machine-learning
      args:
      - ./start.sh
      image: ghcr.io/immich-app/immich-machine-learning:v3
      volumeMounts:
      - mountPath: /cache
        name: immich-model-cache-host
      envFrom:
      - configMapRef:
          name: immich-config
          optional: false
    - name: psql
      image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:bcf63357191b76a916ae5eb93464d65c07511da41e3bf7a8416db519b40b1c23
      volumeMounts:
      - mountPath: /var/lib/postgresql/data
        name: immich-psql
      env:
      - name: POSTGRES_USER
        valueFrom:
          configMapKeyRef:
            name: immich-config
            key: DB_USERNAME
      - name: POSTGRES_PASSWORD
        valueFrom:
          configMapKeyRef:
            name: immich-config
            key: DB_PASSWORD
      - name: POSTGRES_DB
        valueFrom:
          configMapKeyRef:
            name: immich-config
            key: DB_DATABASE_NAME
      - name: POSTGRES_INITDB_ARGS
        value: "--data-checksums"
    - name: redis
      image: docker.io/valkey/valkey:9@sha256:3acc0687f2a2e1091fae6450d7842dd658c941338cf0a873ddd9e14b9e4ea4dd
  restartPolicy: Always

This one is terribly long since there's so much, and that's not even including all the secrets we need from the accompanying secrets file they provide. You will have to map those out as a ConfigMap or Secret object as well. But if everything works out, you can connect to the web port and start uploading photos pretty immediately.

The one downside to Immich is that its migration story is always a little suspect to change, but I feel this will get better with a few more years behind it. My previous server was on a v2 setup, and it didn't have good functionality for exporting all my data like albums and face tags. When migrating to v3, I ended up downloading all my photos locally and re-uploading them again, which is sort of annoying. But, I made it a note to not rely on Immich for all my photos but rather the photos I want to put online. Not every photo I take with my phone is deserving of the internet.

Another thing to note is that if you wish to run the machine learning, Immich by default will refer to the Docker hostname of the ML service, and you have to change it in settings to refer simply to localhost:3003 because Podman does not generate host identifiers per container, and everything is simply part of the pod with a localhost address. If you don't do this, then Immich will never be able to locate the ML service. The ML service is needed for face scanning so you can find and tag people in photos, which is quite handy for when people ask about photos of themselves from events.

Home Assistant

I use Home Assistant for very basic things like turning on a power plug or adjusting the occasional lightbulb or two. It helps to have some electronics on a timer so you can have lights be dimmed or turned off entirely. I go to bed pretty early and wake up early, so having lights turned off when I'm going to bed and turned on when I wake up help me a lot.

Home Assistant also happens to be the most annoying to customize in a declarative manner because of some nuances to how Home Assistant works: there are no environment variables you can inject at runtime, and the configuration is mostly done from the interface once the server is running. That means something like say choosing the port to open on the pod for Home Assistant is not strictly tied with the actual container, but configuration information exterior to the manifest itself.

Then there's the Matter server, something we need in order to communicate with physical devices. I picked Matter smart plugs over any other protocol like Zigbee or Z-wave because it seems like Matter is where we are slowly all moving towards. However, Home Assistant is a proxy to communicate with different protocol services, not necessarily the thing that can speak to all these different protocols. End of the day, we need a Matter server, so we can use the MatterJS server. I don't know of any other ones currently, but I pray they're all roughly the same in terms of functionality.

This setup was frankly the most painful in terms of solving. Matter needs mDNS device discovery in order to talk to smart devices on a network. When Home Assistant wants to connect a Matter device to its setup, it has to pass a Matter server IP for the device to talk to in order to relay data. IP discovery is sort of shaky in a rootless container setup, and the best I can do is use Podman's host network mode to skip over some of the weird nuances of both Home Assistant and MatterJS.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2026-08-13T21:04:04Z"
  labels:
    app: homeassist
  name: homeassist
spec:
  restartPolicy: Always
  # no macvlan, pure host-mode a la Docker
  hostNetwork: true
  hostname: boringbox
  containers:
  - image: docker.io/homeassistant/home-assistant:stable
    name: ha-server
    volumeMounts:
    - mountPath: /config
      name: homeassistant-config-data
  - image: ghcr.io/matter-js/matterjs-server:stable
    name: matterserver
    volumeMounts:
    - mountPath: /data
      name: matter-config-data
  volumes:
  - hostPath:
      path: /home/steve/homeassistant/config
      type: DirectoryOrCreate
    name: homeassistant-config-data
  # this folder must be chmod 0777 for non-root user
  - hostPath:
      path: /home/steve/homeassistant/matterdata
      type: DirectoryOrCreate
    name: matter-config-data

I tried doing dual-pod setups so that Matter would live on its own IP address with a host network attachment, but then it created weird issues of getting in contact with Home Assistant that lived in a different isolated pod. Simply exposing Home Assistant to the host network namespace proved to be far simpler here. I'm sure with enough patience you could play around with pod settings and have it working without needing host mode, but I'm not sure if I have the willpower to keep experimenting.

I log into Home Assistant on http://host:8123, create an account, go to Settings > Network and change the web listening port through the interface. Then swap over to your ideal port so you're not camping on the default Home Assistant port, and change your URL to the new port and you should still be set. Then set up a Matter bridge with ws://localhost:5580/ws as the URL and you can start adding Matter-supported devices.

One thing to note, at least with my TP-Link Tapo plugs, these only work on a 2.4Ghz network. I had to disable Self-organizing WiFi to separate my home WiFi into two channels in order to use these plugs. The SON can deliberate which band channel would be best for a device, and if a smart device used that, then it would bounce between an unsupported 5Ghz band and the 2.4Ghz regular one.

Combining and Autostarting

Podman has good support to create what are called quadlets that allow easier integration into systemd, but with Guix we can't use this. Instead, we'll have to do a little manual work to ensure an auto-start so our services will resume if the computer reboots while we're away.

It's possible to launch multiple pods at once, and if that's your thing, you can treat each pod as if it were a docker-compose.yml and maintain it that way. But, I feel it slightly easier to keep all pods in one file and launch it with one simple podman kube play pods.yaml.

The one thing I've yet to mention is how to manage environment variables and secrets. Podman has a secrets system, but I'm still pretty amateur-ish. My secrets are mostly scrambled passwords that I make up for a system, and that system will pretty much never surface online anyway. I find it easier to manage a file by storing it in something like a .env, which is the lazier approach. The secret system is far superior, but adds another management layer.

The option I went with is creating a secrets.yaml file that I supply into the pods playbook at the end of a command like podman kube play pods.yaml secrets.yaml. As the YAML is parsed, it looks to see if there are any YAMLs that match kind: Secret. We can write a Secret object with the following:

apiVersion: v1
kind: Secret
metadata:
  name: pod1-secrets
type: Opaque
stringData:
  database_password: "Hello!"
---
apiVersion: v1
kind: Secret
metadata:
  name: pod2-secrets
type: Opaque
stringData:
  other_password: "World!"
---
...

These are then injected into the env section of a Pod YAML.

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: pod1
  name: pod1
spec:
  containers:
  - env:
    - name: HardCoded
      value: FlatValue123
    envFrom:
      - secretRef:
          name: pod1-secrets

All the secrets for that pod are then loaded into the container's runtime. You can break up your secrets by container, or by pod, or however you prefer it really. The less information shared the better, you will most likely not want to share all secrets with all containers, so you should split it up where it makes sense to reduce information leakage. If a pod has a database and the password needs to be shared, then it makes sense to share it at a pod level.

The thing that needs to happen is for Podman to be able to run this at startup, meaning we need some user-level service that will execute a Podman playbook. This can be done in two ways by leveraging features of the Guix OS that we build, or we can leverage a Guix Home profile. Contextually, I believe it makes more sense to use Home, so I will show that in a little bit.

Same as the secrets, we can glue specifications together in the same file by using --- separators. This will get read into the system in bulk, and Podman will pick apart anything that looks like a kind: Pod and begin playing the entire pod out from scratch. So while I've been careful in doing very simple container runs adding them to pods, we can now export all our pods and merge them into one new YAML file that can launch all pods at once.

apiVersion: v1
kind: Pod
metadata:
  labels:
    name: pod1
  name: pod1
spec:
  - containers:
    ...
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    name: pod2
  name: pod2
spec:
  - containers:
    ...
---
...

Lastly, we need some code that will start up the pods via podman kube play pods.yaml secrets.yaml command. Here is what I ended up going with as a Guix Home profile.

(simple-service 'autostart-podman home-shepherd-service-type
  (list
    (shepherd-service
      (provision '(podman-pods))
      (documentation "Launch podman pods")
      (one-shot? #t)
      (start #~(lambda ()
        (zero? (system* #$(file-append podman "/bin/podman")
               "play" "kube" "--replace"
               (string-append (getenv "HOME") "/pods.yaml")
               (string-append (getenv "HOME") "/secrets.yaml")))))
    	(stop #~(lambda ()
          (zero? (system* #$(file-append podman "/bin/podman")
            "play" "kube" "--down"
            (string-append (getenv "HOME") "/pods.yaml"))))))))

Adding this to the home services, it will attempt to boot up the pods and it will generally be fine so long as the pods were battle tested before. It will attempt to bring them all down gracefully if the system is shutting down with the inverse command of pod play kube --down.

One thing to note, if you replace all the pods forcefully during testing, if you are force replacing the Pi-Hole pod itself, expect your internet to become a little wonky. For new DNS requests, if the DNS server is down, then you may have some turbulent internet while the pod is brought back up. It doesn't take terribly long, but obviously it is a little scary with containers that the connectivity of your internet relies on a container coming back.

Final Thoughts

I didn't have much reason before to move off Docker. Docker works, and it is largely fine. Millions of people will still use Docker, the world still uses Docker in various random parts.

But, the rise of security vulnerabilities appearing across the globe has made Docker appear like a lesser choice. Kubernetes replaced Docker entirely. Docker tried cementing themselves as the de facto container system by making sure they were the only ones viewed when the name "container" was brought up in conversation. It'd be defamatory of me to say Docker is bad, when it's really not all that bad and will continue to work for a long time. My personal experience is that I think I simply like Podman a little better so far.

I'm not sure for how long Podman will also be until a different solution inevitably arrives. Maybe there's a critical glaring issue with Podman I don't see right now, and maybe more open source tooling around OCI will evolve. Maybe there will be a more native way to approach containers in Guix (even though there sort of is already). Podman feels like a good replacement right now, and maybe one day I'll finally settle on using full-blown Kubernetes instead.

I apologize if this post is terribly long, but I thought it would be a good post to showcase that it is not that difficult to leave Docker with a bit of determination. Once I did Podman long enough, it sort of became second nature, and once you learn about the networking aspect between containers, its sort of easy to figure out from there what you can and can't do.

If anything happens to this website, you can most likely blame my Podman skills.

Addendum: Hardware Changes

I have three servers in my lab, and one third of them is now using my on-going Guix configuration project. The Guix one is managed remotely via guix deploy, and the other servers are both Fedora Server boxes, simply because it was an easier upfront choice time-wise rather than learning more Guix at the time.

Most of the Docker files sat on a Fedora host, and now that my containers are going to live on the Guix host (which has a better CPU anyway), the inferior Fedora host can be wiped and replaced with Guix. At least, that's the intention. Once I reach parity on my Podman server with my Fedora Docker host, the Fedora can be replaced with Guix, and then managed via my Guix machine deploy configuration.

The last Fedora server will take a little more time as that's my NAS that I use for backups. I want to revisit that one later and come up with some scripts to do routine backups to external hard drives. Now that we are in a period of time with expensive hardware, it is time to squeeze every last penny out of everything I have laying around.

Netbook Guix

2026-07-22T20:00:00Z

I've talked about the S10e before, a Thinkpad-style Lenovo netbook powered by an Intel Atom. It's an old piece of hardware, doesn't offer much on paper for performance, and realistically has no use in the future here on out. It's about sixteen years old, and only getting older with each passing day.

When I originally bought this, I wanted to use it as an experimental test bed for computing purposes. Sometimes it's nice to have an old piece of hardware to make benchmarks, and I thought adding it to my rotation would be fine. The only problem it has: how do I maintain an operating system for it?

Junk Shopping

The story of this humble S10e starts with me perusing eBay, looking for opportunities as I was searching for what is now my Surface Pro 7. I was looking casually at netbooks in case there was something there, and really there wasn't much in the netbook or Chromebook family that would have been great. Laptops that small are often really cheap, underpowered, underequipped, and are generally low quality. Lots of cheap plastic and crappy cables. Not my favorite.

However, the S10e must have found itself to me somehow by sheer luck. Through a series of searches and recommendations, I discovered it. The auction had no bids and was $20, so I bid, and days later, won, to no surprise. It shipped, arrived, and turned on immediately to a Linux Mint. Nice! I didn't ask for it, but Linux was a great fit of course and it felt fine on first use.

However, I didn't keep Mint for long. It simply put has absolutely no horsepower under this hood. It's an early generation Intel Atom, it can barely browse the web with an old version of Firefox. Updating it was a pain, since it's stuck sort of in a weird generation of 32-bit CPUs right before mostly everything became 64-bit by default. I took some time to check out some other systems in search of less memory use over the traditional memory-hog systemd.

The only systems I could mostly get to work were in fact Void Linux and antiX Linux. These sytems provided i686 ISO installers and they worked fine. But, desktop traditional software in general is not easy here, due to the sheer fact that 32-bit support is long gone for a lot of programs. On both systems I had no desktop software, and had gotten used to just using raw TTY modes on it. And actually, I had zero qualms about it too.

Much like the new theme of creating custom 3D-printed cyberdecks with DIY screens attached to weird folding cases for Raspberry Pi computers, the S10e is weirdly also a bit cyberdeck-feeling. It's age is clearly showing, but with the right software use, this thing can still function just fine for specific purposes. In fact, much of this blog post is written with the S10e. It's just so cute and lightweight. There may be things I don't like about it, like how small the Shfit key is on the right-hand side, but those are muscle memory problems I have.

After a while, I stopped using the S10e with alternative Linux systems and sort of left this one to hang on the side. I do use my Surface a lot more for actual coding, and this thing is not the coding beast it wishes it could be. The lack of widespread software support means this thing is basically sort of dead in many places. Not a whole lot of operating systems willing to build for this low marketshare computer.

Revisiting Guix

In parts, I thought Guix would be both a good idea and bad idea to put on the S10e. It's not a very high-power system and comes with a lot of challenges. Guix is good at constructing an operating system, but could it even work for an operating system that has much worse hardware specs on paper?

My first time installing Guix on the S10e was basic, but also a failed attempt. I had leveraged the guix system command to directly write to the disk an operating system. I don't remember entirely how long it took, but it wrote to disk, I plugged the disk and closed it up, turned the S10e on, and was met with a blank screen. I couldn't swap to a TTY, and after having spent some days playing around with different systems on it, this is where I sort of gave up and moved on.

But, this time I'm determined to work through the motions to get a bootable system and take advantage of Guix features I've never really used before.

Minimum Bootable System

A Guix system is really easy to build once you get the hang of things. The hard parts are tucked away far from our eyes, with the easy stuff being you simply tell Guix what type of programs you'd like to have installed. I know the S10e is going to have problems, so first it'd be a good idea to establish what a baseline operating system would look like, and use that to build future iterations.

For now, let us start with this:

(operating-system
  (hostname "tinythonk")
  
  (packages %base-packages)
  (services %base-services)

  ; filesystem info left out...
  )

This is a generic, very low abstraction of an operating system: take all the packages that Guix defines as the baseline, and put it into something that resembles an operating system. There's not a lot of other complicated details like what type of CPU architecture we want to use, because that is going to be the job of the guix system to dictate. Guix has a list of possible targets and supports doing cross builds, meaning we can build software packages on x86_64 and supply binaries to an aarch64 chipset.

guix system image --target=linux-i686-gnu config.scm

That will produce an image file, something we can write to a disk byte-for-byte, and it will have a full Linux kernel and root file system produced. A raw file will literally produce something that only uses as much space as it needs. If the filesystem was desired to be larger, then the raw image file would need to be larger to compensate for the fact that we'd have to copy more empty bytes onto the disk. If you want a 10GB file system, you will have to copy over 10GB of bytes.

Using QEMU, a utility for testing operating systems which routinely blows my mind, we can boot up an image file and treat it like bootable media so long as it has a bootloader. This will at least verify if our system even works under perfect conditions, but can at least eliminate any fears of the image file being incorrect.

qemu-system-i386 -drive format=raw,media=s10e.img -m 512

{{ img(src="qemu1.png" desc="Booting Guix into a QEMU session. From here you can test manual login with a password, or port forward the virtual ports to a real network interface and try using SSH.") }}

It boots all the way through with the Linux libre kernel with no issue. Having learned how to finally use these tools, I'm in a much better place than where I was blindly installing images hoping they just work. Now that I know that I can emulate an operating system file, let's move onto the next problem.

Trimming Guix

The earlier operating-system configuration above doesn't actually work. I saved this bit for later, but during some builds, a package was failing for a reason I didn't quite understand. I noticed alsa-utils, a package related to ALSA, was causing an issue.

I scanned the definition for both %base-packages and %base-services and saw no reference to ALSA anywhere. A common way to figure this out is to leverage guix graph to graph out all our dependencies in DOT format. But in this case, it doesn't help us since that's to scan for a package, not an operating system's configuration. Fortunately, I did notice something while scanning %base-services.

(service udev-service-type
  (udev-configuration
    (rules (list lvm2 fuse crda alsa))))

udev is linking to alsa-utils, and frankly I'm not too sure how to pick up on this connection using standard graph tools. Using guix system extension-graph shows a story about the service requirements, but doesn't really show between the two quite as easily since it's based on services, not exactly the packages themselves.

{{ img(src="udev_graph.png" desc="ALSA connects to etc, but not directly to udev it seems. Guix being able to export connection graphs of packages and services really shows off another cool feature of declarative package management.") }}

udev is responsible for pulling in alsa-utils at runtime, which was causing my build to crash. Could I simply just remove the 'alsa mention here and proceed onwards?

(modify-services %base-services
  (udev-service-type config =>
    (udev-configuration
      (rules (list lvm2 fuse crda))))

And surprise surprise, we continue to build until we hit another wall: kernel modules.

Since Guix can make use of either free or non-free kernels, I was trying to build this out with the free kernel until it became problematic, so I tried non-free. Same problem. Seems to be something about the firmware modules cannot be found, most likely because they probably don't even exist. I found that you can go through the Guix code to find the list of all the kernel modules used for initrd, and this is basically what I came up with for what works and doesn't work.

(define my-initrd
  (list
    ;"virtio_pci" "virtio_balloon" "virtio_blk" "virtio_net"
    ;"virtio_console" "virtio-rng" "virtio_mmio" "virtio_scsi"
    "ahci" "usb-storage"
    ;"uas"
    "usbhid" "hid-generic"
    ;"mmc_block" "dm-crpt"
    "xts"
    ;"serpent_generic" "wp512"
    "nls_iso8859-1"
    ;"pata_acpi" "pata_atiixp" "isci" "nvme"
    ))

Passing these in as initrd-modules to the operating-system makes the errors go away. I'll take a stab in the dark and say it could be due to the fact that we're cross-compiling x86/64 to i686, a platform that is not really used by anyone anymore. Could just be that nobody thought to port nvme to i686? Maybe, not sure. There's some other modules here tied to cryptography for perhaps things like LUKS, but I think we're well past the point of security being the dealbreaker here. I can mess with these later if I'm feeling experimental, I'm simply trying to get a bootable system.

Generating the Image

Guix has the ability to leverage the guix system command to generate an image file through another subcommand called image. With no external arguments, it will generate the entire image file you need byte-for-byte like I mentioned earlier. Nothing new here.

guix system image --target=linux-i686-gnu config.scm

However, we will run into another challenge: the root partition is not volatile. Accordingly, it seems like to generate the system image, we need a volatile root partition in order to build. But, there's no way to expose that setting from the CLI. We need to build an image object in Guile to pass into the image command instead. This has a more rigid interface for building images than passing CLI arguments.

This is an important section for another reason: this partition will also define the filesystem. When we copy an image file, we're also in turn copying bytes over to the disk, meaning we're copying the filesystem as well. Any partitions we define from earlier need to get mapped here. Normally on a Guix install, you do a prep step to write a filesystem to the disk before you begin doing the Guix install steps usually with parted or cfdisk.

Now, I have a 120GB SSD in this thing because I was feeling generous. That's the lowest size SSD I had, and frankly it made for a good test to see if it worked fine. But now that yields a different problem: we can't simply copy over 120GB of bytes. An image is simply copying 1:1 the bytes in the image file to the disk, and if we wanted to size this image to match the exact specifications, well... Our build would generate a 120GB image file. Not exactly a great idea for anyone's SSD lifespan.

To work around this, it's better if we expand the partition from the device when we're done, rather than burn millions of zeroes into the disk. It's good to start with a small image max size, something you could fit on a USB stick or such, and then expand the partition later via parted.

(image
 (inherit mbr-disk-image)
 (name 's10e)
 (format 'disk-image)
 (operating-system os)
 (partitions
  (list
   (partition
    (offset root-offset)
    (label "swap")
    (file-system "swap")
    (size (* 1 1024 1024 1024)))
   (partition
    (offset (+ root-offset (* 1 1024 1024 1024)))
    (label "root")
    (flags '(boot))
    (size (* 9 1024 1024 1024)))
   ))
 (volatile-root? #false))

The above is the partition format I went with. The goal is to have an MBR bootloader sector for the byte-0 bootload initializer code, then use the offset from that to have a 1GB swap space, and the remainder can go to the root partition. The reason for this order is that if you wanted to expand the root, but it came before the swap, the swap would have to be deleted and toggled via swapon/swapoff in order to move it to the end of the disk. By keeping it in front, we can more easily resize the root partition.

Our S10e has very little memory, so we will take advantage of swap space. I hate to sound like someone that has to explain the importance of swap each time I bring it up, but here it's pretty necessary since the S10e only comes equipped with 1GB of RAM. Should we go over that in memory assignment, at least programs will use swap instead of getting terminated by the OoM killer.

Let's make an image and verify our work by using parted's print command.

$ sudo cp /gnu/store/imagehashabcdefg-s10e s10e.img
$ sudo chown user s10e.img
$ sudo chgrp users s10e.img
$ chmod 0755 s10e.img
$ parted s10e.img print
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system     Flags
 1      1049kB  1075MB  1074MB  primary  linux-swap(v1)  swap
 2      1075MB  10.7GB  9664MB  primary  ext4            boot

Okay, so we get a roughly 11GiB image we can burn. Now we can boot it with QEMU to give it a test run.

qemu-system-i386 -drive file=s10e.img,format=raw -m 512

QEMU will appear with a tiny box and after doing the initial boot process, we get a login prompt. If you did not supply a login, you simply will not be able to log in. You can modify the password field of the user account, but it would be better to bind an SSH key to the user account so you can log in with ssh immediately, then use passwd to change the password on your OS.

At this point, I unplug the S10e's drive, plug it into a SATA-to-USB adapter (very handy to have FYI), and burn the file using dd just like I would any live Linux ISO file.

$ sudo dd bs=4M if=/home/steve/s10e.img of=/dev/sd<X>
$ sync

I throw an extra sync at the end of a dd call to immediately flush any write buffers to the disk to avoid having it done lazily. Now the only thing left to do is try and see if it turns on.

Unfortunately, after all that effort we're back at the same step: a blank screen. Turns out there's something that happens after the device turns on: the system tries to activate a VGA screen picture mode, and seemingly, if there is nothing there to take over the handoff, then the VGA screen will simply never do anything. Since we stripped out a ton of packages and desktop-related items, we can turn off any video outputs by disabling it in the kernel arguments, which gets loaded into the bootloader entry as extra arguments.

(define os (operating-system
 (host-name %hostname)
 (timezone "America/New_York")
 (locale "en_US.utf8")
 (keyboard-layout (keyboard-layout "us"))
 (kernel linux)
 (kernel-arguments '("quiet" "nomodeset"))
 (firmware (list linux-firmware))
 (initrd microcode-initrd)
 (initrd-modules my-initrd)
 ...

Had I known this some months ago, maybe this would've went a lot differently. Oh well, one quick reflash and let's try again.

We are now in a complete TTY mode and able to log in using either a dummy password or an SSH keyfile.

Deploying Remotely

This is obviously not a scalable way of doing upgrades. Not in the slightest. I do not want to have to unscrew the back plate of the S10e, remove a drive and pop it into my desktop each time I want to perform an upgrade. The wireless card it has does not seem to want to work either, even with nonfree drivers included. The ethernet port, however, is fine, and I can even ping outside IPs and ssh to another machine on the local network.

But now we enter the second stage of this project: how do we update and manage the S10e without having to re-flash the disk? Well you could simply guix pull on the device itself to keep Guix commits up-to-date and pray that substitutes are in place. Or we can leverage a Guix feature called deploy. We're going to need a few extra steps to incorporate this feature.

Guix Keyring

guix deploy is a feature that takes a list of machines paired with operating system configurations and attempts to build the entire thing, not unlike systems like Kubernetes. One machine will act as the build machine, while we supply a list of targets to apply said updates to. We can pin 100 operating systems to 100 machines, or one operating system to 100 machines. For our sake, we map one operating system definition to our S10e.

A Guix machine will not accept updates from a random machine, and I think we can understand why. Beyond the connection public and private keys, some kind of trust should also be established between the build machine and target host. For this, the target host needs a key identifying the build machine, and we unfortunately will have to pack a new image to burn to disk because of this change.

First we need to create our signing key.

guix archive --generate-key

This is part of the archive subcommand of Guix that deals with archiving packages, and it also maintains the list of keys that a machine could accept when downloading archived packages. Should we want to add a new key to our target host, we can do so with another Guix command.

guix archive --authorize < /etc/guix/signing-key.pub

But, that's not the long term solution. If you were to re-apply an OS upgrade, that key would no doubt get wiped as it's not a correct means of managing it. Instead, we should add the key directly through the configuration itself so that it is defined declaratively instead. The minute you do a guix deploy with the manual authorization, you will not be able to do another deploy after until you authorize the key again.

To accomplish this, we need to take our signing key and attach it to our configuration. We can simply verbatim paste it into the config as an exercise, but later it should be moved to something a little more permanent so it doesn't stay an ugly blob forever. We need to append our signing key to our list of approved signing keys. And yes here we will have to burn the disk image for the last time to make this change permanent.

(services
  (modify-services %base-services
    (guix-service-type config =>
      (authorized-keys
        (cons*
          (local-file "signing-key.pub" "(pubkey ...)")
          %default-guix-auth-keys))))))

While we're here writing our disk for the final time, I will skip a secret and mention that we are going to need passwordless sudo. The account connecting needs root permissions to modify the system, and as such we need modify the sudoers file to give us the ability to do that.

(sudoers-file
 (plain-file "sudoers"
   (string-append
     (plain-file-content %sudoers-specification)
     (format #f "~a ALL = NOPASSWD: ALL~%" %user))))

Another image rebuild, another dd to the disk, and we'll have an OS with our signing key set. So long as you're not making a habit of regenerating your keys, this should be a decently long term solution until key rotation becomes a threat model.

Guix Deployment

The last step before we can begin a guix deploy is to create a manifest for the deployment. We need a list of targets, hostnames, information on how to connect to them, the user... The works. It's not as simple as telling Guix to connect to a host and do its thing.

The manifest for the deployment isn't really that complicated. It's a list of things to connect and perform a deployment to. We connect, we inspect, and we generate a difference payload and submit the changes. Each host could be on a different type of state, and frankly I don't have enough field experience to know what happens when one machine is a little further ahead than another. Or if the build host is behind a target host. I'd be interested to find out though.

Here's a sketch of my S10e, using an operating-system record defined as %os.

(list
  (machine
    (hostname %hostname)
    (user %user)
    (operating-system %os)
    (identity "/path/to/your/ssh/id_ed25519")))

The scariest part of all is now actually performing the deploy. But the transactional property of Guix ensures that if a build cannot happen, then it does not impact the running system. Going from our previous image build to our new deploy system shouldn't add anything new into the system, so we can simply run it with no modifications to the system manifest.

guix deploy config.scm

Even if we only have my tiny S10e here, it's pretty cool to visualize how you can control and deploy your entire lab or network of hosts with just one command. You can deploy hundreds of hosts, all with the same OS, and deploy upgrades with one quick tap in the CLI. Since it's all Guile Scheme underneath, you have a full-blown programming language at your disposal as well, so you can generate your own automations with basic Scheme code.

Fundamentally, I now believe this is the end game right here for all operating system automation and reproducibility.

Closing Out

The S10e largely hasn't been part of the story much. Functionally, it's been pretty okay. The one drawback is that there is no desktop software on this thing yet. But that hasn't been a drawback for me at all still. I like the TTY, I know how to navigate and use it, so it's fine for my uses. In fact, on a battery this old, higher-performance applications would no doubt contribute to significantly worse battery life while I'm on the go.

Because of its architecture and the extreme limitations, there's not too much I can do here. Guix commits will focus on safe builds and compile specific targets so long as x86 can compile down to i686 fine. If I can get a certain commit to compile some software correctly, then I can feel confident to pin the packages to a specific commit and use the time-machine functionality to freeze Guix in place for it. I can also use that s10e.img I generated earlier and throw it onto my NAS for safe keeping and use it as a reliable OS backup when something with the disk inevitably goes wrong.

I think the S10e currently as it stands fits weirdly as a nice little laptop for an escape from modernity and can be used for when you really want to focus on tasks at hand. No browsers, no internet, no distractions, you sit in a TTY and install the tools you need and focus on the task at hand. It's a polar opposite from my Surface where I have tabs and tabs of Firefox open looking at docs and writing code. Sometimes I may want one or the other, and adding the S10e to a rotation of devices I can use is great, and saves it from the landfill for a few more years.

Using htop to benchmark the system, it doesn't really use too much memory. Although my fonts are a little wonky. Accordingly, it's not really using anything more than roughly 100MB of memory.

{{ img(src="htop.png" desc="I think the font packaging may have gotten slightly messed up, but it's still very usable regardless. htop is a mainstay program for any computer admin.") }}

Let's say I forgot to install vim on the S10e. How long would it take for me to build it from my desktop and install it? I ran a time guix deploy config.scm to see what would happen.

$ time guix deploy s10e/config.scm
The following 1 machine will be deployed:
  tinythonk
...
guix deploy: successfully deployed tinythonk

real	1m41.506s
user	0m2.618s
sys	0m0.159s

{{ img(src="vim.png" desc="I normally am an Emacs user, but without some heavy customization, normal Vim for me is just fine. In fact I tend to use Vim bindings in Emacs since it makes Emacs way easier to use.") }}

I even have an action shot of Vim on the train, although it predates Guix.

{{ img(src="train.png" desc="For a while I was bringing the S10e on the train with me to try it out. Pretty healthy battery life for being over a decade old.") }}

What about basic work, can it run Python? Well, if you install it just like have been doing, then you will get an i686-compatible Python runtime, meaning you could probably install most Python third-party packages so long as no x86-specific extensions were needed, or hard 64-bit types in the CPython runtime. One quick python added to our packages and deploy later:

{{ img(src="python.png" desc="The best part of runtime-based languages is that they generally play pretty well anywhere they can compile. I'm sure Haskell would work just fine too.") }}

$ echo "<html><h1>Hello World</h1></html>" > index.html
$ python3 -m http.server

{{ img(src="helloworld.png" desc="Hosting a Hello World webpage on my S10e using the standard Python HTTP server. Should we host websites on our netbooks? And other ridiculous questions we could ask ourselves.") }}

While maybe not the most useful desktop experience in the world to some, I think it's an interesting piece of hardware to have equipped with Guix as shown. I've learned my fair share of lessons with Guix, things I ordinarily would not have dug into I got to experience first hand. The oddly exciting part about i686 is the learning opportunity to design new software that supports older architectures while not feeling totally stuck in the past. Low memory and low performance means all sorts of new challenges to think about when writing programs for myself. Maybe I can start with some basic applications and slowly work my way up the ladder to eventually make a game cross-platform support i686 with a language like Rust.

My apologies for the long wait. This story took several months to finally finish, and I still wouldn't say it's totally complete. I need to figure out what combination of software can be used to give me a minimum desktop experience so I can toy around with graphical applications, perhaps XFCE or a window manager like Openbox.

The code listed here is not going to be kept up-to-date. The best place would be to follow my Guix repository and check it out for yourself. I am doing my best to keep detailed notes with all my hardware to keep note of what works and what doesn't as I go along. I can take what I learned here and start managing my homelab with guix deploy for multi-machine setups.

Thanks for spending the time reading! Lots of work ahead of me, but hopefully I'll have something not Guix related next time. May finally start doing audio work like I've been putting off for months. If you're interested in declarative Guix operating systems, hopefully what I've detailed will help you start on the right track.

Tablet Tale

2026-06-21T20:00:00Z

My XP-Pen Artist Pro 13.3 has been a reasonable source of headaches for a long time since swapping over to Guix as a full-time system. I thought this might be the most appropriate place for me to discuss the depths of which I've traveled to the bottom of to arrive at my final solution. Join me for the dumbest journey possibly ever seen.

Bootstrapping XP-Pen

XP-Pen provides a closed source "driver" as we could call it that interacts with the device over USB to change various settings like key mapping or pen pressure range. Using this software proved me no issues on Arch Linux since somebody kindly maintained a xp-pen package on the AUR. But... Well, let's not talk about what's been going on with AUR recently.

Now on Guix, nobody has a package for XP-Pen because it's a closed source binary, and the main derivative of proprietary bullshit Nonguix doesn't have an entry for it either. I've always assumed because nobody has a tablet uses Guix, or because it simply wasn't worth the headaches. I never asked in IRC on the #guix channel frankly, not worth annoying fellow Guixers about it. It's my burden, I'll work out a solution surely.

My first approach is the idea that the XP-Pen driver called PenTablet needs a bunch of packages to be made usable on the system. To prove that it can run, I don't need to make a package, I can simply use a guix shell. The shell command will pull in packages into an isolation chamber of sorts so you can build programs or run things without impacting your main system. Good for testing if a package is going to work on your system or not before installing it. I have a very long Guix shell command in my history that is certainly going to be fun to look at.

guix shell --container -F -u steve --network \
  --share=$XDG_RUNTIME_DIR --share=/dev \ 
  --share=/dev/input --share=/dev/bus/usb \
  --share=/dev/usb --share=/run --share=/etc/localtime \
  --share=/etc/udev/udev.conf=/etc/udev/udev.conf \
  -- preserve='^WAYLAND_DISPLAY$' --share=/sys \
  --preserve='^XDG_RUNTIME_DIR$' \
  bash coreutils gcc-toolchain glibc icu4c openssl \
  zlib sed grep libevdev libusb libx11 libxtst libxrandr \
  libglvnd glib dbus qtbase@5 qtwayland@5 usbutils less \
  strace libgudev eudev gtk libappindicator libnotify

The meat and potatoes of this command is the --container and -F flag, where -F means --emulate-fhs. Because Guix does not have a typical Linux standard filesystem layout, we are missing things like /usr/lib which is used for linking programs to shared libraries on a system. This will effectively collapse any and all packages along a few paths so shared libraries will be compressed into the /usr/lib folder.

After a lot of trial and error, I am able to boot the PenTablet program... But it's not quite working as expected.

The only problem is that it doesn't pick up my USB device at all. The program is currently screaming about not being able to find a configuration file along the /usr/lib/pentablet path, because Guix does not have a normal FHS path that would make that exist. I can't debug the program any further than that, really.

...Or can I?

Stracing

I don't consider myself an amazing programmer. By any stretch of the imagination, I write in fun languages and don't need to interact with low-level debugging involving Linux syscalls. I rarely get opportunities to debug things or solve problems. This program, however, is just another opportunity to learn something perhaps. This is where strace kicks in.

strace is a tool to trace syscall operations made by a program, so we can listen and inspect what a program is actually doing by observing the syscalls it makes. Maybe we can learn something about what is going wrong. Or, feed it into a large language model (sigh) and see if you can burp up an ideas based on their output.

Heading back into the shell, we can use strace inside a container to observe the syscalls. I will insert my own comments based on what I think is happening. With no access to source code, this is one of the few things I can do. I observed the following series of syscalls and I can only infer what is going on with some comments added.

### Scans through devices iteratively
openat(AT_FDCWD, "/sys/bus/usb/devices/usb5/bConfigurationValue", O_RDONLY|O_CLOEXEC) = 30
openat(AT_FDCWD, "/sys/bus/usb/devices/usb5/bConfigurationValue", O_RDONLY|O_CLOEXEC) = 30

### Finds my tablet as a matching device
openat(AT_FDCWD, "/gnu/store/1sbkaxg5r5ryn3m6jmqbkgsmi7a2zir3-glibc-for-fhs-2.41/lib/gconv/UTF-16.so", O_RDONLY|O_CLOEXEC) = 30
manufacturer:UGTABLET product:13.3 inch PenDisplay Get Max X:0x72b2 Y:0x407e  Press:0x1fff

### Looks for the config.xml file, can't find it (ENOENT)
openat(AT_FDCWD, "/usr/lib/pentablet/conf/xppen/config.xml", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
Function  LoadConfig Error: open file "/usr/lib/pentablet/conf/xppen/config.xml" failed, error: "No such file or directory"
ReadXml Failed!
------------------load xml failed: P1315Pro8K1W
DeviceListLabel: ""

### It can open up the /dev/uinput USB inputs just fine however
openat(AT_FDCWD, "/dev/uinput", O_WRONLY|O_NONBLOCK) = 29
open uinput ok

openat(AT_FDCWD, "/dev/uinput", O_WRONLY|O_NONBLOCK) = 30
open uinput ok

openat(AT_FDCWD, "/dev/uinput", O_WRONLY|O_NONBLOCK) = 31
open uinput ok

openat(AT_FDCWD, "/dev/uinput", O_WRONLY|O_NONBLOCK) = 32
open uinput absmouse ok

### Opens other USB files
openat(AT_FDCWD, "/dev/bus/usb", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 35
openat(AT_FDCWD, "/sys/bus/usb/devices", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 35
openat(AT_FDCWD, "/sys/bus/usb/devices/usb5/busnum", O_RDONLY|O_CLOEXEC) = 36
openat(AT_FDCWD, "/sys/bus/usb/devices/usb5/devnum", O_RDONLY|O_CLOEXEC) = 36
openat(AT_FDCWD, "/sys/bus/usb/devices/usb5/speed", O_RDONLY|O_CLOEXEC) = 36

I've looked at other syscall error messages and didn't find much more beyond that. There's polling syscalls, but those didn't really reveal anything helpful to me. I basically turned on every syscall I could think of related to opening file streams and beginning mmap'ing memory to start reading device events, but I couldn't make sense of it. Knowing this works on Arch but not on Guix, one can assume it's simply a Guix problem.

It could also be that because this is a container, it's slightly restricted from things it needs because of cgroups. I eventually started prodding around trying to see how I could maybe lower some "barriers" across cgroup processes to give the program more room to breath, but no luck.

I can go further and try and attempt to debug it with gdb, but because I have no access to source code, I would effectively only be able to debug it via registers. Without source code, I am left to guess what is going on through reverse engineering, but I think that's a rabbit hole I'm not quite ready for.

OpenTabletDriver

A program that I was able to prove that also works is OpenTabletDriver. This is a pretty nice program that works with a wide array of tablets, and for anyone using tablets on Linux (read: not Guix) systems, this would be my recommended option. OpenTabletDriver (OTD) runs a server daemon that clients can connect to and will look for devices on the host system. A client can connect to the server to manipulate the devices as needed, so you can change settings with basic scripts or using a GUI that they also provide.

As luck would have it, Guix does not have a package for OTD. Upon further inspection, OTD is written in C#/.NET, meaning it has strong ties to Microsoft-based tooling, and I already knew where this was going. C# is a pretty big problem for Guix because for Linux users to write C# or F#, they need to go through Mono. Mono has not one, not two, but eleven different software licenses. Mono does exist in packages, but it won't go very because most use-cases for Mono may typically involve the .NET SDK. And boy does that SDK have a nice looking software license.

But let me not start shitting all over Microsoft's stuff just yet. I happen to really like F# as a language. C# I really don't like because it's so many terrible Java/C++ ideas shoved into such a mundane and uninteresting language, and generally it doesn't offer any added advantages over a language like Golang, a language that has far less features but a much larger developer pool. Half lucky for us, a dotnet package exists in Nonguix. The bad news, we can't use it. It's too old and OTD requires a specific version for .NET. So... What now.

I went with two routes, and both lead to dead ends:

Compile the OTD source code by combining a manually downloaded .NET SDK with the mono package
Run the OTD pre-built binaries in a Guix shell like the other driver

I can confirm that both do work; the only problem is they also do not find my USB device. I have to only assume that due to the non-FHS environment, these drivers just don't work correctly. Despite that fact, there is even a Nix package for OTD. I've looked it up and down and I can't find the reason why my tablet simply won't get picked up.

I've looked in OTD's source code for days understanding how it works, and it relies on .NET libraries and shared object linked code, which lead me to believe that the problem lies in me simply not packaging my code and treating things as real Guix packages. If OpenTabletDriver were written in a language such as Rust, then it would be quite portable and could easily drop the .NET dependency. Oh well.

Only thing left I can think of: actually trying to create a Guix package myself.

Packaging XP-Pen

Instead of trying to go through the hell of updating the dotnet package which I couldn't get to work, I figured it was easier to simply work with the XP-Pen driver as a baseline. I know it worked, so if I could find the links then it wouldn't be a problem. By running ldd on PenTablet we can see that it is missing a lot of things.

$ ldd xppen/App/usr/lib/pentablet/PenTablet
steve@stovetop ~$ ldd XPPEN1/App/usr/lib/pentablet/PenTablet
    linux-vdso.so.1 (0x00007fca9eee4000)
    libusb-1.0.so.0 => not found
    libX11.so.6 => not found
    libXtst.so.6 => not found
    libXrandr.so.2 => not found
    libQt5Widgets.so.5 => not found
    libQt5Gui.so.5 => not found
    libQt5Xml.so.5 => not found
    libQt5DBus.so.5 => not found
    libQt5Network.so.5 => not found
    libQt5Core.so.5 => not found
    libGL.so.1 => not found
    libpthread.so.0 => /gnu/store/m31vlvwm79m89fk3xk0z4h7snk61y510-glibc-2.41/lib/libpthread.so.0 (0x00007fca9eed3000)
    libstdc++.so.6 => not found
    libm.so.6 => /gnu/store/m31vlvwm79m89fk3xk0z4h7snk61y510-glibc-2.41/lib/libm.so.6 (0x00007fca9ede5000)
    libgcc_s.so.1 => not found
    libc.so.6 => /gnu/store/m31vlvwm79m89fk3xk0z4h7snk61y510-glibc-2.41/lib/libc.so.6 (0x00007fca9ebf5000)
    /gnu/store/m31vlvwm79m89fk3xk0z4h7snk61y510-glibc-2.41/lib/ld-linux-x86-64.so.2 (0x00007fca9eee6000)

Remember from the guix shell --container -F command that we can collapse a bunch of libraries along a /usr/lib path so the program can dynamically load shared objects easily. For us to create a package so this becomes a trivial guix install xppen command, we need a definition that would effectively link all the missing shared objects in the package definition. First let's try and at least get the driver tarball into the Guix store.

(define verno "4.0.15")
(package
  (name "xppen")
  (version verno)
  (source
    (origin
      (method url-fetch/tarbomb)
      (uri
        (string-append
          "https://download01.xp-pen.com/file/2026/04/XPPenLinux"
          verno
          "-260422.tar.gz"))
      (sha256
        (base32 
          "19czwmbgqqvlna7k554dvbnl2kk87y724d736aakk7zajvx759r5"))))
  (build-system copy-build-system)
  (native-inputs (list patchelf))
  (inputs
    (list bash coreutils gcc-toolchain glibc icu4c
          openssl zlib libevdev libusb libx11 libxtst
          libxrandr libglvnd glib dbus qtbase-5 qtwayland-5
          usbutils libgudev libappindicator libnotify
          ))
  (home-page "https://www.xp-pen.com/")
  (supported-systems '("x86_64-linux"))
  (synopsis "XP-Pen Driver for the XP-Pen Artist Pro 13.3")
  (description "XP-Pen Driver for the XP-Pen Artist Pro 13.3")
  (license license:expat))

Guix packages work by defining a build-system, and in this case we simply need to copy the contents of the tarball into the Guix store, so we leverage copy-build-system to do exactly that. There's no compilation here whatsoever, but what Guix will do is refer to the inputs as packages it needs to download in order to make this package function. If those packages cannot be built or found, then this package simply cannot work and it will fail.

Another point of how Guix works, since it uses a Git-based process, all these variables refer to the most recent and up-to-date Guix packages available. There may be times where library incompatibilities could happen, and you would need to create package inferiors in order to refer to older packages, but luckily that doesn't seem to be an issue as far as I can see.

After the build, it will appear in a path like /gnu/store/y8w90icad574jan25v0v89vf3xsf2ksr-xppen-4.0.15. But, it's not in a usable state in the slightest. The binary itself since it was not compiled by us has no idea where these shared objects are still. We now need to use a tool called patchelf to alter the binary's ELF references to shared objects.

Now the reason it took me so long to make this post and talk about this is probably due to the patchelf aspect of this all alone. However, I was able to work through the complications of this process and figure out each component. And frankly, having Guix packages written in Guile Scheme makes this much nicer with the added benefit of having a Scheme language being very flexible.

Here we need to create a custom Guix phase. A phase works in that a build system executes a couple of steps to build a program, and sometimes we need to create our own phase to do a little extra work to make a program run. We will need a new phase to link the libraries correctly.

(arguments
  (list
    #:install-plan ''(("XPPenLinux4.0.15-260422" "."))
    #:phases
    #~(modify-phases %standard-phases
      (add-after 'install 'patch-binary
        (lambda* (#:key inputs outputs #:allow-other-keys)
          (let* ((patchelf (assoc-ref inputs "patchelf"))
                  (xppen (assoc-ref outputs "out"))
                  (ld-so (string-append (assoc-ref inputs "glibc")
                                          "/lib/ld-linux-x86-64.so.2"))
                  (pentab (string-append xppen "/App/usr/lib/pentablet/PenTablet")))

          ;; find packages /gnu/store directories based on the inputs
          (define pkg-dirs
            (filter
              (lambda (path)
                (and (file-exists? path)
                     (file-is-directory? path)))
                (map
                  (lambda (pkg)
                    (string-append (assoc-ref inputs pkg) "/lib"))
                inputs)))

          (define rpath (string-join pkg-dirs ":"))
          (invoke "patchelf"
                  "--set-interpreter" ld-so
                  "--set-rpath" rpath
                  pentab)))))))

If that looks long, it's because it is mildly exhausting. What's happening here is that we are binding a new phase to the build system through a lambda* function that is accepting two major variables: the inputs we defined from the inputs field, and a outputs that defines where the package is going (in our case it's a package named xppen).

The easiest way to go about this is taking the entirety of the inputs list and converting each package to a path that resembles /gnu/store/<pkg>/lib. We need to apply an additional filter to assert that the string we made even exists, so we don't bother with inputs that don't provide a shared library. Then, combine all these paths into a big colon-separated string that we can feed into patchelf.

Much to anyone's shock, this still doesn't link everything correctly.

Shared Object Maths

The ldd output now shows a bunch of linked libraries, but also a bunch of libraries still missing. Specifically, zlib is missing at least three times. What the heck is happening and why?

ldd /gnu/store/n0h3nwlgaq5v0y5w09scgwqnd5f3ys8d-xppen-4.0.15/App/usr/lib/pentablet/PenTablet
    ... (redacting text for brevity) ...
    libgobject-2.0.so.0 => not found
    libgthread-2.0.so.0 => not found
    libglib-2.0.so.0 => not found
    libz.so.1 => not found
    libdbus-1.so.3 => not found
    libz.so.1 => not found
    libz.so.1 => not found
    libgthread-2.0.so.0 => not found
    libglib-2.0.so.0 => not found

Now I know that I correctly supplied the libz.so.1 because I personally verified it's existence myself. What's exactly going on here and why? Well, let me revisit something I left out previously.

The PenTablet program includes its own shared libraries meant to be installed on the system. Namely, XP-Pen provides shared libraries mostly because of Qt5. But it's not hard to get Qt5 on most major Linux systems. Why did they include these files?

$ ls XPPEN/App/usr/lib/pentablet/lib
libicudata.so.56    libQt5Core.so.5  libQt5Network.so.5
libQt5XcbQpa.so.5   libicui18n.so.56 libQt5DBus.so.5
libQt5Widgets.so.5  libQt5Xml.so.5   libicuuc.so.56
libQt5Gui.so.5      libQt5X11Extras.so.5

I have to assume only one thing: XP-Pen made modifications to the Qt5 source code and compiled these shared objects to be distributed with the binary. The most important one to assume is the libQt5DBus.so.5 library because that allows Qt to interface with DBus.

My previous iteration of the package layout does take these into account and can easily add them to the main library injection. What doesn't happen is that these libraries do not know where their dependencies are either.

The best way I can describe it is with a little bit of a drawing.

PenTablet needs libraries on the host system as well as libraries inside of the included /lib folder. But /lib also needs libraries on the host system. The way we configured it as is, we never configure the /lib libraries to be contextually aware of the Guix package store, so what we end up with is a gigantic cross-mix of shared object references. The ELF links get flattened when all these links are cross-referenced, so that's why some libraries show up as not found even though we patched in all the library paths correctly.

To rectify this, we need to include the package's shared libraries into the mix as well. Patching each shared object is simpler, all we need to do is make it aware of the Guix store paths, a string we had already made earlier.

(arguments
  (list
    #:install-plan ''(("XPPenLinux4.0.15-260422" "."))
    #:phases
    #~(modify-phases %standard-phases
        (add-after 'install 'patch-binary
          (lambda*
           (#:key inputs outputs #:allow-other-keys)
           (let* ((patchelf (assoc-ref inputs "patchelf"))
                  (xppen (assoc-ref outputs "out"))
                  (ld-so (string-append (assoc-ref inputs "glibc")
                                        "/lib/ld-linux-x86-64.so.2"))
                  (pentab (string-append xppen "/App/usr/lib/pentablet/PenTablet"))
                  (libs (string-append xppen "/App/usr/lib/pentablet/lib")))

             (define pkg-dirs
               (cons libs
                 (filter
                   (lambda (path)
                     (and (file-exists? path)
                          (file-is-directory? path)))
                     (map
                       (lambda (pkg)
                         (string-append (assoc-ref inputs pkg) "/lib"))
                     inputs)))

             (define rpath (string-join pkg-dirs ":"))

             (for-each (lambda (f)
                         (display (format #f "patching ~a ... \n" f))
                         (invoke "patchelf" "--set-rpath" rpath f))
                       (append (find-files libs "\\.so.*")))
             
             (invoke "patchelf"
                     "--set-interpreter" ld-so
                     "--set-rpath" rpath
                     pentab))))))))

After all that exhaustive time, the ldd output finally shows all the libraries are now linked correctly.

ldd /gnu/store/8nx5cm5cb1fwm3rp9cpv2psi62pb4c8q-xppen-4.0.15/App/usr/lib/pentablet/PenTablet
    linux-vdso.so.1 (0x00007fb1620f4000)
    libusb-1.0.so.0 => /gnu/store/93qc1ldl2iwf0fdncmxjbylcdlbg052c-libusb-1.0.25/lib/libusb-1.0.so.0 (0x00007fb1620ce000)
    libX11.so.6 => /gnu/store/i8z1lr827wp475q2pyqgffks0nc3fh6b-libx11-1.8.12/lib/libX11.so.6 (0x00007fb161f89000)
    libXtst.so.6 => /gnu/store/xbb9r684wvnjcj7q8gzpks6b09sgahbs-libxtst-1.2.5/lib/libXtst.so.6 (0x00007fb161f81000)
    libXrandr.so.2 => /gnu/store/9pkarlnf49ly3dvrvj0yk6dx0s9rlp2j-libxrandr-1.5.4/lib/libXrandr.so.2 (0x00007fb161f72000)
    libQt5Widgets.so.5 => /gnu/store/8nx5cm5cb1fwm3rp9cpv2psi62pb4c8q-xppen-4.0.15/App/usr/lib/pentablet/lib/libQt5Widgets.so.5 (0x00007fb161400000)
    libQt5Gui.so.5 => /gnu/store/8nx5cm5cb1fwm3rp9cpv2psi62pb4c8q-xppen-4.0.15/App/usr/lib/pentablet/lib/libQt5Gui.so.5 (0x00007fb160a00000)
    libQt5Xml.so.5 => /gnu/store/8nx5cm5cb1fwm3rp9cpv2psi62pb4c8q-xppen-4.0.15/App/usr/lib/pentablet/lib/libQt5Xml.so.5 (0x00007fb160400000)
    libQt5DBus.so.5 => /gnu/store/8nx5cm5cb1fwm3rp9cpv2psi62pb4c8q-xppen-4.0.15/App/usr/lib/pentablet/lib/libQt5DBus.so.5 (0x00007fb15fe00000)
    libQt5Network.so.5 => /gnu/store/8nx5cm5cb1fwm3rp9cpv2psi62pb4c8q-xppen-4.0.15/App/usr/lib/pentablet/lib/libQt5Network.so.5 (0x00007fb15f800000)
    libQt5Core.so.5 => /gnu/store/8nx5cm5cb1fwm3rp9cpv2psi62pb4c8q-xppen-4.0.15/App/usr/lib/pentablet/lib/libQt5Core.so.5 (0x00007fb15ee00000)
    libGL.so.1 => /gnu/store/2yqqr3b4zx01wlmrjij2b90jz83jwizz-libglvnd-1.7.0/lib/libGL.so.1 (0x00007fb161ee3000)
    libpthread.so.0 => /gnu/store/5sh58d0kdqc8kbq4i3bzvd8zbph2jfnc-gcc-toolchain-14.3.0/lib/libpthread.so.0 (0x00007fb161ede000)
    libstdc++.so.6 => /gnu/store/5sh58d0kdqc8kbq4i3bzvd8zbph2jfnc-gcc-toolchain-14.3.0/lib/libstdc++.so.6 (0x00007fb15ea00000)
    libm.so.6 => /gnu/store/5sh58d0kdqc8kbq4i3bzvd8zbph2jfnc-gcc-toolchain-14.3.0/lib/libm.so.6 (0x00007fb161312000)
    libgcc_s.so.1 => /gnu/store/5sh58d0kdqc8kbq4i3bzvd8zbph2jfnc-gcc-toolchain-14.3.0/lib/libgcc_s.so.1 (0x00007fb161eaf000)
    libc.so.6 => /gnu/store/5sh58d0kdqc8kbq4i3bzvd8zbph2jfnc-gcc-toolchain-14.3.0/lib/libc.so.6 (0x00007fb160812000)
    /gnu/store/9x4178zhvzgpqklyq1k1nxn449f31xhg-glibc-2.41/lib/ld-linux-x86-64.so.2 => /gnu/store/m31vlvwm79m89fk3xk0z4h7snk61y510-glibc-2.41/lib/ld-linux-x86-64.so.2 (0x00007fb1620f6000)
    libxcb.so.1 => /gnu/store/3b8z0yh5irlpkqw646ql4006jrd12qvj-libxcb-1.17.0/lib/libxcb.so.1 (0x00007fb161e86000)
    libXau.so.6 => /gnu/store/f1sc667nw0xrcx8lym2gzzyw255gbxv1-libxau-1.0.12/lib/libXau.so.6 (0x00007fb161e7f000)
    libXdmcp.so.6 => /gnu/store/qchb551c824i7sc0xh56qbg279k7fjh8-libxdmcp-1.1.5/lib/libXdmcp.so.6 (0x00007fb161e77000)
    libXi.so.6 => /gnu/store/qbpzz5lim11cg32344l9spmgskbvw0jw-libxi-1.8.2/lib/libXi.so.6 (0x00007fb1612ff000)
    libXext.so.6 => /gnu/store/h7x83rh0220f5j1vl95v06di01z54s5j-libxext-1.3.6/lib/libXext.so.6 (0x00007fb1612eb000)
    libXrender.so.1 => /gnu/store/sgmk9zbxkrwv3dcxg9p4l10pydxv2rb4-libxrender-0.9.12/lib/libXrender.so.1 (0x00007fb1612dc000)
    libgobject-2.0.so.0 => /gnu/store/25dylanmcbv8jxmljznhjhjra14rz11q-glib-2.86.0/lib/libgobject-2.0.so.0 (0x00007fb161277000)
    libgthread-2.0.so.0 => /gnu/store/25dylanmcbv8jxmljznhjhjra14rz11q-glib-2.86.0/lib/libgthread-2.0.so.0 (0x00007fb161272000)
    librt.so.1 => /gnu/store/5sh58d0kdqc8kbq4i3bzvd8zbph2jfnc-gcc-toolchain-14.3.0/lib/librt.so.1 (0x00007fb16126d000)
    libglib-2.0.so.0 => /gnu/store/25dylanmcbv8jxmljznhjhjra14rz11q-glib-2.86.0/lib/libglib-2.0.so.0 (0x00007fb1602ab000)
    libz.so.1 => /gnu/store/2jzflb96n50lknzn7znfq9ri4qlbifjc-zlib-1.3.1/lib/libz.so.1 (0x00007fb16028c000)
    libdbus-1.so.3 => /gnu/store/bafcxafz3pbcmhjp9w2xs94b3ihxm48g-dbus-1.16.2/lib/libdbus-1.so.3 (0x00007fb160237000)
    libicui18n.so.56 => /gnu/store/8nx5cm5cb1fwm3rp9cpv2psi62pb4c8q-xppen-4.0.15/App/usr/lib/pentablet/lib/libicui18n.so.56 (0x00007fb15e200000)
    libicuuc.so.56 => /gnu/store/8nx5cm5cb1fwm3rp9cpv2psi62pb4c8q-xppen-4.0.15/App/usr/lib/pentablet/lib/libicuuc.so.56 (0x00007fb15dc00000)
    libicudata.so.56 => /gnu/store/8nx5cm5cb1fwm3rp9cpv2psi62pb4c8q-xppen-4.0.15/App/usr/lib/pentablet/lib/libicudata.so.56 (0x00007fb15c000000)
    libdl.so.2 => /gnu/store/5sh58d0kdqc8kbq4i3bzvd8zbph2jfnc-gcc-toolchain-14.3.0/lib/libdl.so.2 (0x00007fb161264000)
    libGLdispatch.so.0 => /gnu/store/2yqqr3b4zx01wlmrjij2b90jz83jwizz-libglvnd-1.7.0/lib/libGLdispatch.so.0 (0x00007fb15fd47000)
    libGLX.so.0 => /gnu/store/2yqqr3b4zx01wlmrjij2b90jz83jwizz-libglvnd-1.7.0/lib/libGLX.so.0 (0x00007fb15fd13000)
    libffi.so.8 => /gnu/store/ij3xh4d12m57dzikkb1mmzbjwjgvf37j-libffi-3.4.6/lib/../lib/libffi.so.8 (0x00007fb160226000)
    libpcre2-8.so.0 => /gnu/store/3my9smd643p4i29yryqha291x3fgw231-pcre2-10.42/lib/libpcre2-8.so.0 (0x00007fb15fc74000)

No longer does it show multiple libz.so.1 entries because the total union of all libraries are now looking at the same places. We can launch the binary and see if we are finally in the money.

{{ img(src="pentablet.png" desc="Well we're still no better off than we started are we") }}

Uh, so nothing got fixed. Great.

Impasse

Several weeks of my spare time, building a package, searching the deepest parts of the internet, lurking the #guix channel, and here I am: no results. I am forced to consider all my options, and I did give weight to abandoning Guix entirely and heading back for Artix Linux.

Despite all this, there's one thing I never talked about: how the tablet will function without any drivers whatsoever. The tablet normally turns on fine, the buttons on it also work fine, and seemingly, the pen input from putting a pen to the display screen also works.

The one question I've never bothered to ask: can I restrict the input of the device to a specific display? That's really what this entire endeavour has been for me. I don't want it to be drawing all across my displays (which is roughly 5,000 pixels wide...), I want it to be constrained to just the display it holds. The only way I did it before was by using the PenTablet binary. I don't care about remapping keys on the tablet, I can do that in software.

Well as it turns out, you can. A number of desktop softwares include the ability to restrict the inputs of a tablet device to a display.

Gnome supports a map to monitor feature
KDE has a pretty large page on supporting tablets
Sway can map inputs to outputs

Lastly, the software I use Niri supports mapping inputs. Has it been in front of me all along and I never noticed?

I append the following to my Niri configuration file.

// input for the XP-Pen 
input {
    tablet {
        // maps to its own display (DP-3)
        map-to-output "DP-3"
    }
}

And... Wow it really was just that simple.

{{ img(src="brush_testing.png" desc="So... I guess I'm done?") }}

Cleaning

While developing the package, I opted not to use guix shell to run it. Simply because I had believed that the shell was restricting some kind of USB access to my driver. That clearly wasn't the case, and despite not knowing the hows or whys of why none of these drivers work on non-FHS systems (well, I think we could slightly attribute it to the lack of FHS), as it turns out, I have a lot of copies of the XP-Pen driver in my /gnu/store now.

$ ls /gnu/store | grep xppen | wc -l
61

The iterative process of me running through this multiple times has blown up my /gnu/store path. So how do I get rid of these manually-installed packages?

The unfortunate side effect of installing a package directly from a file into my live environment created many new generations of profiles in Guix so now that I simply have generations that only show my humble, not-working proprietary software. To start removing all of these stale packages, I need to revert a couple generations.

$ guix package --delete-generations=10..20

I ended up deleting all the package generations I had made since there wasn't a real use for them. I can make the packages later if I removed something I was using, but I'm juggling so many projects I'd rather restart the package generations from scratch.

$ guix package --roll-back

After that I was on generation zero. No overlays made from my baseline. But checking guix gc --referrers I can still see my xppen packages are hanging on for dear life. I've restarted the guix-daemon through herd a few times, closed out lingering shells and programs, and ultimately rebooted my system entirely to get to a clean slate, until finally:

$ guix gc
(... lots of text ...)

$ ls /gnu/store | grep xppen | wc -l
0

Just like that, my old packages were gone. I know now that it was most likely not necessary to directly overlay my custom package into my real system, and ultimately guix shell worked just as fine and leaves your base system more intact and with far less messy generations.

Closing Out

Weeks back when I attempted to solve this problem, I expected some things would happen: that this tablet driver would give me a ton of headaches, and maybe I'd learn something about Guix in the meantime. Both are true. But I'm happy to see that I didn't have to go through all of this madness.

I am greatful for the fact that my tablet can work and function without the need of any weird Chinese-made proprietary drivers. I wish I could use OpenTabletDriver, but I'll have to make do with what I got. Instead of having to downgrade my system to something non-Guix or much less NixOS (ew), I think this is just simpler and far less management. Sure I don't get to configure buttons or pressure maps, but Krita and Gimp also have software for managing that, and have had those features for years now. I was using Wacom tablets with Gimp almost 20 years ago.

Now, is this the best outcome? Probably. Minimal code changes, no need to figure out a closed-source binary, plays well with my setup, and I don't have to do extra work. Am I mad? Oh absolutely. Mostly at myself though for not being able to make it work in the end.

In honor of this entire process, I made a scribble out of pure frustration.

{{ img(src="i_hate_computers.png" desc="Me drawing myself hating computers") }}

Thanks for reading my descent into madness.

Surface Linux

2026-02-12T20:00:00Z

I am coming up on the eight year anniversary of my first-ever used Thinkpad purchase. What makes this so special? Well I guess there's a number of reasons why I'm talking about this huge brick yet again, and what brings me to the point of this post.

Years back, when I first started accumulating disposable income, I wanted to buy a piece of hardware separate from my desktop system. I'm not always at home, and sometimes I'm out and about or at work, a laptop would make sense. I am aware of the fandom surrounding the Thinkpad family of devices, having spent time on places like ThinkWiki. I spent time scouring eBay for a moderately priced device, until I settled on the Thinkpad T440.

Now, is there some level of craziness to buying a piece of hardware this old? Probably. The T440 is a 2013-era laptop system, doesn't come with a whole lot of frills, and certainly doesn't do anything terribly exciting. But, it has been a laptop home of mine for a number of years, a reliable companion. Cafe dates, low-poly or emulated gaming, coding, experimenting with Linux systems, mostly anything I've learned in the last almost-decade I can attribute this laptop to being part of that.

But, perhaps this might be the end of the road for the T440.

Problems

The T440 has been a hardware piece I've been running Guix on for a while. Guix is not the easiest thing to work with, but I got a configuration working and had a pretty happy environment mostly stable and working with Steam. The disk itself had seen quite a number of distros on it over the years, and Guix I felt was the comfortable OS for the rest of its days.

One day, I turned it on, and to my surprise, the Guix system would no longer boot.

In the moment of the computer I had seemingly turned on last month and was using perfectly fine, something was up. Now in all my years of working on computers, maybe my brain didn't quite want to process it, but the CMOS battery warning should have been an obvious one.

The BIOS boot settings were all reset to a factory default, meaning my file partitions EFI files were not being recognized. I ran a live Linux distro to make sure it wasn't a faulty SSD or what have you, since I have quite literally never replaced the SSD this thing came with. It could have been an SSD failure and I wouldn't really know how to recognize it other than connecting it to my main PC.

I ordered a replacement CMOS battery, swapped it in, installed a new OS just to be safe, and I was back in order. I may have to rebuild my Guix OS on it on a different day, but the fear of losing the computer entirely got me thinking, perhaps it was time to retire it as my main device. I wasn't bringing it with me on my trains anymore since it is so particularly heavy and bulky. It's been a number of years, maybe it was time to look for something new.

Replacement

I started a search to find a modest replacement. My life is a little different than what it was eight years ago, and technology has changed a modest amount. We have USB-C charging ports now for mostly everything, bigger space for drives, more RAM, and certainly better screens. The T440 still has it's stock LCD screen because I never wanted to do screen replacement surgery on it. It only has 8GB of RAM, which I was lucky to get for the price since higher RAM models are more rare back then. The battery life isn't totally awesome anymore either.

That being said, new hardware in 2026 is awful. It's all terrible. We have dedicated Copilot AI buttons that can't be re-programmed. RAM in most 2026 models is expected to shrink due to the RAM shortages we're currently experiencing. Not to mention the retail value of mostly all new computers has exploded. Anything that can have a high amount of memory would be overpriced because it's now an "AI developer workstation" or some nonsense, and anything lower than 8GB is unusable because Windows 11 eats up 60% of your RAM doing nothing.

Anyone who spends some time around me knows I try to avoid buying anything on release. I don't buy new smartphones day one, and my phone is currently close to being three years old with only two years left of upstream security support from Google. So a reasonable question to ask: is buying new practical anymore?

Dumpster Diving

I don't want to spend a whole lot of time talking about why I feel computers are in a very strange spot with regards to how major players like Microsoft can choose to make redundant millions of computers at the snap of a finger. My personal belief is that computers can be used and fixed to be made to last longer, and I want to continue sticking to that principle moving forward through 2026. Let's talk about what I'm looking for.

I don't think there's a perfect magic formula one could follow to figure out the hardware requirements you'd need to get by. For the most part, I want something a little bit smaller and a little bit better than what I purchased in 2018. I bought a five-year old device back in 2018, so maybe I could buy a five-year old device now.

By process of elimination, pretty much anything made by Dell, Hewlett-Packard, Acer and ASUS I am immediately disregarding. I have seen and experienced far too many laptops by all of the above, and I'm not happy with any of them anymore. That leaves me with little room to work with, such as Lenovo being one of the few remaining brands.

Lenovo does have potential to be something good. But, I'm not sure if I want to simply buy yet another Thinkpad experience. I have been interested in the X1 Carbon family of Thinkpads for a while, notably a brand of laptops that only came out after I bought my T440 in 2018. They're light-weight, performant, and very popular despite being points of contention in the Thinkpad community.

Sadly, I think this is a time where I'd like to skip on the Thinkpad. My reasoning breaks down into a few points.

Form factor - do I want another boring laptop?
Charging - how many wires do I need to lug with me?
Battery life - is the battery life going to be reasonably decent?
Weight - how much weight would it add to my work bag?

Now I have no doubts about battery life and charging, but weight and form factor leave me skeptical. Do I want yet another clam-fold plastic chassis laptop in my bag? Something about it seems so mundane. I don't want to keep experiencing the same hardware story each time I buy a piece of hardware, it doesn't really lead to new use-cases of experiences in my opinion. Like yes I can carry a laptop everywhere I go, but that doesn't feel particularly very engaging for me.

Reflecting on the other devices I've used throughout my life, I started thinking about the Microsoft Surfaces again.

Surface

Based on the post title, this really shouldn't be a surprise. The Microsoft Surface is a tablet-like computing device geared for business users or people who are really mobile. It first came out when Windows 8 was released, and by some stretch of the imagination, I feel as if we simply weren't ready for what Windows 8 was trying to bring us.

What made them attractive is the direct line from hardware manufacturer to OS provider: Microsoft. No more botched updates from Dell or IBM, the tech blame falls exactly on one gigantic corporation. And of course when these things were first released, Microsoft wanted them in sight of as many people as possible with as much anti-Apple sentiment possible.

There is no specialized OS, it is simply a Windows computer. It's a touch screen, it has a foldable keyboard, it's extremely light-weight, and all that makes it hyper-mobile for the right people. Business-types love the high-tech iPad form factor, creatives get a screen drawing tablet, students get a tiny little thing that they can throw into any backpack.

These devices first originated in 2012, and while I was vaguely interested back then, that opinion quickly changed once I actually had one in my possession.

In 2017, I went to Atlantic City over in New Jersey. I took with me on that trip a Surface 3, the third iteration in the Surface hardware. Why did I have one? It was given to me because a family member simply didn't care for it anymore and got a Surface Book some time later.

I flashed it with Linux Mint at the time, and the experience was... Simply unusable. Wireless networking was hit or miss, graphics and CPU power was not quite there, and I struggled to make it a good daily driver.

Looking back, what went wrong? The Surface 3 utilizes an Intel Atom CPU, which are notably terrible CPUs for anything that isn't a Chromebook. There are most likely other reasons why the hardware wasn't as well supported either like mainline kernel support, but it didn't inspire confidence as making it a daily Linux computer.

Back then, they were horribly overpriced, probably very slow, and had little real-world functionality other than a point-of-sale device. Having experienced this Surface back in 2017, getting the Thinkpad a year later was a boon.

Kernel

Coming back to the hardware acquisition search, I stumbled across a strange video as I was trying to peak my curiosity. How do Surfaces handle Linux in 2026? Thankfully, somebody answered that question by installing Linux on a ton of different Surface models.

The process is as straight-forward as it can be: disable Secure Boot, disable the Windows boot manager, and boot into a live Linux ISO and install. This was a similar process in 2017, and it's largely unchanged. However, this video inspired new-found confidence in the Surfaces I didn't have years ago.

An alternative kernel called linux-surface was born and supported a wide array of different Surface devices through community work. For the most part, everything looked relatively stable, and using this would help me pick a device from the Surface family. With nothing else to go on other than a single GitHub project and a YouTube video, could I make this work and have it become my next travel companion?

Shopping

Browsing through eBay is usually an experience. It's never really known if something will work when you receive it, then you have to go through a two-plus week process to even get refunded by eBay and do the return steps.

I noted earlier that my rule of thumb is picking something a little around five years of age to match my T440 age gap back then. I also want to keep the price fairly low as well, at least not more expensive than what I originally paid for the T440, hence the image at the top. This leaves me with a few options.

The Surface Pro 8 is a desirable pick because of it's 11th generation Intel hardware and reasonable specs on RAM and space. It had released around 2021, fits my five-year age gap estimate, and comes with pretty much everything I would need to get started. The only downside is it's price, which was a little outrageous.

By using eBay's "sold items" filter, the Surface Pro 8s have been selling anywhere from $400 to $700 USD, which is really strange. Are people really buying five-year old hardware for that price? That seems a little ridiculous to me. So I looked a little lower, at the Surface Pro 7 instead.

The Surface Pro 7 rocks Intel 10th generation CPUs, which is a few notches above good enough for TPM 2.0, not that I care about that. It sells anywhere from $200 to $300, which I felt was an agreeable price for what it came with. For only being about two years older, it's not that bad. I found a listing for roughly $200, and it comes with an i7, 16GB of RAM and a 256GB SSD.

After all this time spent analyzing and trying to find the perfect device, I added the Surface Pro 7 to my cart and proceeded to checkout.

Setting Up

A box arrived on my doorstep, and wow, this box weighs nothing at all! I think the box weighed a total of 3lbs when I brought it in. The purchase did not come with a charger, granted who knows how much that would add, but this box was very light.

It came very late in the evening, so I turned it on for a brief moment and fiddled with the supposed window scrobbling problem the listing said it had. Turns out, a laggy window drag in Windows isn't really a hardware or software problem, it's really a Windows 11 problem. I felt fairly confident that this could run Linux with close to zero issues.

The next morning, I hopped on my train geared with the Surface Pro 7 (SP7 here on out) and a flash drive loaded with EndeavourOS. After having sat down, I turned it on, popped into the boot configuration, turned off secure boot and Windows boot manager, and logged into EndeavourOS, and it worked perfectly fine. Then, I started the install, and a little later, I had a clean Linux machine all set up on the train. Perhaps someone behind me watching the entire process had many questions about what they were watching with a giant red lock appearing over the boot screen.

With no internet for the rest of the day, I had to wait until I got home to install the linux-surface kernel. With only base desktop packages installed, it's not like I could do much other than write code via nano or what have you. The kernel install should fix the touch screen at least, although the web camera is currently unsupported, and that is totally fine with me.

After getting home to a plain wifi network, I installed linux-surface per their instructions and set it as my default kernel in the systemd boot configuration. Upon booting into the new kernel, I was playing around with the power button to see the different uses of suspend/hibernate modes available, and... Well after toying around I did hit a kernel panic.

That kernel panic may have just been some rotten luck; I couldn't capture it, otherwise I would have passed it along to the linux-surface project. I need to get better at capturing those problems if I want to be of any use, but that kernel panic never came back after days of use later. That being said, I got things like Python and Rust installed, and now it's basically a normal developer laptop.

Synchronizing

While I primarily operate off my desktop, adding another mobile device with virtually zero internet access most of the day makes it difficult to keep it up-to-date with random things I work on. Let's say I write code on the Surface during my train rides, I explicitly have to remember to upload it once I get home. That's fine, so long as I remember to push to a Git repository and commit my code. But what about other files?

If I want to do a drawing, art files such as Krita are binary, and not terribly suitable for Git differential algorithms. It would not be best to store all my artwork as a Git repo. The cleaner solution is to simply use a network server like a Samba share. However, this isn't a totally automated process. Using a network automatically-mounted share instead would let you treat a network host like a file system instead letting the kernel do normal read/write IO, but just over the network wire instead. But this doesn't technically solve the synchronized issue, just where the data is stored.

What I've found instead is that with rsync, you can create a shell command that lets you push files to a remote host. Punching in a shell function command to sync your computer isn't the nicest option, but there are ways to jerry-rig it such that it can work if you are connected to a network and only execute that maybe say once a day or so. It doesn't even have to be a bi-directional sync if you don't want that either; maybe you only care about a one-way backup option, then rsync is great.

Stealing the name of Dropbox here, but treat it as something similar.

rsync -avr ~/dropbox remotehost:~/dropbox

Doing that pushes files from your ~/dropbox folder and puts it on a remote host on your network in the same name. Reverse the arguments, and you can update your local device with files from the remote host. Want to share books or PDFs or documents you're working on with LibreOffice? Now you can use your network server as a good place to share files. If the remote host has Samba configured, you can access the files via Samba as well.

Bluetooth

The device does technically have Bluetooth on-board, so it can connect to a variety of peripherals. Perfect for those with wireless earbuds and the like, but one thing I never tried using Bluetooth for was file share.

Simply put, if I take a photo on my phone of a certain level of fog of Manhattan one day and I want to edit it on the go, I can certainly connect a USB-C cable to my phone. But I do not want to have to do that; Bluetooth should let me be able to warp files back and forth. My friend sends me a resume to look at and fix, it might be better off fixed while I'm on my Surface instead of fat-thumbing around on whatever editor I have on my phone.

Hotspotting

Before you ask: "can you hotspot?" because I frequently thought of that. There is absolutely every reality where I could use my phone as a Wi-Fi hotspot on my commutes and at least attempt to sync any work I have to an online server. I could easily use OpenCloud and do work that way touching my own private hardware at home.

Hotspotting is usually an on-and-off battle of cellular providers. And to my knowledge, sometimes hotspotting is not always free. However, it does appear that I can use hotspotting on my current plan. But I fear it's a step that may lead to accidental data over-usage on my plan. Not only would it lead to a small increase in WiFi traffic on my Surface, it would also add some strain to my phone's battery as well.

I generally need my phone all day for communication, train tickets, credit cards and all other sorts of things, so using hotspot is at best a small risk for when I want to do some synchronized work on the go. I've yet to really give hotspotting a try since the winter season for commuting has been... Interesting to say the least.

Pics

Nothing much else to talk about other than some action shots of it in the wild.

{{ img(src="coding.jpg" desc="What else do you do on a train with no internet and only left to your thoughts? You start coding up silly things like fraction-based math in Rust.") }}

{{ img(src="master_duel.jpg" desc="I installed Yu-Gi-Oh! Master Duel and decided to see how the touch screen felt. You could get used to playing like this. Granted, you can already do this with mobile devices since it's also on iOS and Android. Swordsoul Tenyi is best deck.") }}

{{ img(src="factorio.jpg" desc="You can very much play Factorio on it. Though I imagine a very large factory will very quickly drain your battery dry.") }}

Thoughts

This was a long and tedious process of trying to find something usable. I have a lot of fond memories of my very first Thinkpad, but I always knew it was going to get aged out in the end. Laptops were not meant to be upgraded (let us ignore Framework in this argument).

Key points that I like about the Surface Pro 7.

Light-weight - it clocks in around 2.5lbs, still much lighter than other alternatives I've seen
Battery life - can easily last the whole day if using a battery saver mode
x86_64 - none of that Snapdragon aarch64 stuff in newer models (see further)
USB-C - another device that easily integrates with my other USB-C charging methods
Smallness - it's a comfortable size, not quite a laptop, not quite an iPad either
Bed time - since it's a tablet, it's easy to relax in bed with and simple to put away

Things I may not like or be a total fan of just yet:

Keyboard - the Surface keyboards have always felt a little weak, feels as if it could be the first to break
High DPI - I can't count out the extra pixels as being a plus, but I will have to use scaling to bring up the pictures to usable viewing resolutions. I can't help but feel like excess pixels leads to worse battery life, but there's not much I can do about that now
Suspend/Hibernate - occasionally the suspend fails, and the login becomes unresponsive? Nothing a reboot can't fix, but I'm not an expert on this yet

Things I'm excited to try out:

Mapping software - downloading an offline copy of something like OpenStreetMap would make it handy for getting around in a pinch
Drawing - perhaps with Krita or Heavypaint it will be another outlet for sketching and painting
Gaming - it could do with an honorary Half-Life playthrough
Coding - another battery-powered device gives reason to want to write more battery-efficient software

I think the most fun part about these new hardware sprints is that you find all sorts of new use-cases you would never likely have expected. Sure I bought it mostly for the trains, long commutes and mostly just to lighten up my bag, but it's become also a new device for being on my couch and multitasking between the TV. I can bring it to bed and finish up a line of code or two. I can bring it to lunch with me if I want to share something I've worked on with friends.

These are the tech upgrades that I think really impact every-day life. Getting smaller computers, simpler computers, or even more mobile versions of Linux is frankly an awesome thing to behold. This Surface Pro 7 is basically 7 years old, but the inspiration and fun it brings is fresh and exciting to have.

Thanks for reading! This one took some weeks to get done just so I could test the Surface. Now onto the next project.

Addendum: Snapdragon

Snapdragon CPUs, the things that drive most of the planet's Android devices, are largely not that compatible with the rest of the technological world. The makers of Snapdragon Qualcomm have done a pretty okay-ish job of getting most things to work, but Snapdragon-based computers are only very recently new to the world with the introduction of Windows for Arm.

Arm-based CPUs using aarch64 aren't terribly compatible with many things like Steam, meaning if you bought a Surface with a Snapdragon chip, chances are you'll likely not be able to play any games with your friends in a meaningful capacity. Games are largely only ever compiled for x86 or amd64 targets, hence the importance of x86_64 Linux for me personally.

If you have a game that was compiled for a different CPU architecture, then you would effectively need a translation layer to convert the instructions that aren't supported into ones that are. This is not a simple 1:1 ratio of instructions, as CPUs are different for what opcodes they have or sizing of bytes and integers. This is solved by using a very recent Windows Prism emulator. However, on Linux platforms this problem isn't entirely solved.

Until very recently, there were only a few options: use something like Box64, or drop down to QEMU. Box64 seems alright, but it's not quite on the same level as Wine and Proton. However, now we have a new tool available called FEX, introduced by Valve and specifically crafted for the Steam Frame that is yet to be released.

Since this is all highly experimental still, I wouldn't call it quite battle-ready for my needs. By design, Arm chips, and soon to be hopefully replaced by RISC-V, are better for energy efficiency and leave behind a lot of the old x86_64 crud designs that prop up much of the chip marketplace. Arm has powered much of the world's devices for this long, I can imagine Arm and RISC-V being the future not too far away. At one point, having a phone would be near-equivalent to having a real desktop PC. Apple has embraced Arm with its M1 chips and is continuing work there as well.

So if you were interested in a Surface with a Snapdragon chip, maybe wait a bit before its considered usable in Linux. If you wanted an Arm-based Linux, consider Raspberry Pi instead!

MIDI Sequencing

2025-04-21T20:00:00Z

I honestly have not been around much as of late. A lot of things have been going on, and my April has been filled with family tragic events, to the point that it had an effect on my mental mood. I wasn't programming or writing much of anything for the last few weeks.

Despite all that, I finished up my most recent project which I will be talking about here. I started it back in March, and it took me until only a few days ago to finally have it uploaded and everything. It sits roughly at around ~500 lines of Rust code, but it'll grow as I add more comment strings.

Primer on MIDI

I have been into music creation for so many years now, but I wouldn't call myself a musician in any sense at all. I like the programming aspect, and the resultant output of programs that create sound. Much of this is achieved via MIDI, and I have many physical MIDI devices I've collected throughout my time.

MIDI is a format by which any and all electronic music devices operate on, and allow software and hardware to play well together. Foundationaly, MIDI when it was first introduced was game-changing, and created a whole musical market for digital devices and software to enable musicians of the next generation.

I'm accustomed to physical hardware MIDI interfaces, and I've written software before in Pure Data to generate MIDI signals to send to hardware interfaces for quite some time now. But this time around, after looking at my Novation Launchpad first generation, I wanted to do something different. It wasn't that long ago that I produced a game on the thing, so I wanted to do something more geared towards musical production.

So I set out to create an arpeggiator, the most basic sequencer program type I can think of. Back then, I did not know it would take me far over fourty-five days finish, however.

Arpeggiation Basics

Arpeggio is the function of creating a melody from a series of notes in a repeated fashion. It's used in many places in music, but I guess the best way to describe it classically would be with some Chopin.

The core idea is that you set a melody, and an arpeggiator will loop through the beats over and over, defining an easy pattern to use as a backing melody in a song. Arpeggiators are not common place anymore I would say, as electronic music production simplifies the process of creating new music. Arpeggiators are fairly common place in something like drum machines rather, because the core ideas are the same; looping over a series of notes in a repeated fashion.

The algorithm, in my head, was as follows:

1. Create a table of notes and a timer set to a BPM speed
2. Iterate over the table of notes one index at a time
3. Flush the note outwards if the note is 'active'
4. Repeat when index hits the end of the table
5. Read user input from the device and adjust notes in the table

Not all steps here are equal in functionality. The reading user input process is much more complicated than simply looping over a note table. But the other components are fairly simple. But in order to keep things relatively in-sync with the time measure, how do I gauge how long to pause actions for, and can I find a way to synchronize it so notes are never played out of sync with the beat?

Scheduling

Gathering user input, sending MIDI messages to the output bus and changing LEDs on the physical device are all things that can take variable amounts of time to execute, so getting these things done while also staying on beat is a challenge to do. Many of these things can happen at different intervals, too. Reading user input can happen at almost any second, while writing MIDI messages only happens every so often. Not to mention the state changes from one table index to the next, things need to happen with a certain level of synchronicity.

So I started writing something I don't ordinarily write very often: a job scheduler. Once upon a time when I once took a course on operating systems development, this sat at the core of the course to make sure things happen autonomously and with a high degree of precision and certainty. It however had been a number of years since, but I had the right idea to do this as it turns out.

First my goal was to break down all the different actions into different categories. There's user input, program state change, and note flushing. These all need to happen within fixed time intervals to ensure not only correctness, but usability. The user input should be slightly near instantaneous, as to not feel sluggish, while the program needs to update it's state and reflect that on the physical device.

Here I got started in Rust, with a simple enumeration representing messages to pass around.

pub enum Message {
    UserInput,
    UpdateState,
    FlushNotes,
}

The variants don't really need to hold additional data. What comes next is a data type to store the message and timing information as a job. I refer to the timing as "ticks" like a tick of a metronome. When a job reaches it's maximum number of ticks, it should be executed by the program.

pub struct Job {
    current_ticks: usize,
    max_ticks: usize,
    msg: Message,
}

Later I realized it didn't make too much sense to write a fixed-type job struct pinned to my message enumeration, so I decided to make it based on a type parameter instead for it to be re-used later in life.

pub struct Job<T> {
    ct: usize,
    mt: usize,
    msg: T,
}

Now for the scheduler's definition that contains all the job information.

pub struct Scheduler<T> {
    tick_duration: std::time::Duration,
    last_time: std::time::Instant,
    jobs: Vec<Job<T>>,
    queue: Vec<T>,
}

From a conceptual standpoint, the scheduler is going to execute at the smallest time interval possible on a loop and update each job to add a new tick onto each one. Once they reach their maximum tick count, they are reset to zero and copied onto the queue. The tick duration and last time fields are there to help us calculate how much time we need to rest our executing thread for.

One thing about vectors is hot allocation when they reach capacity, and I don't want my scheduler to be allocating memory during the program, so I use Vec::with_capacity() to set both vectors to have a size of about a hundred elements each, which is plenty for this type of program. Though hopefully memory allocation never really takes a toll as it's a tightly-bound program. The queue's elements will have to be cleared occasionally, but that's it.

Now for a longer piece of code to show the scheduler's full implementation.

impl<T: Copy> Scheduler<T> {
    pub fn new() -> Scheduler<T> {
        let jobs = Vec::with_capacity(100);
        let queue = Vec::with_capacity(100);
        Scheduler {
            tick_duration: Duration::new(0, 0),
            last_time: Instant::now(),
            jobs: jobs,
            queue: queue,
        }
    }

    pub fn has_events(&self) -> bool {
        self.queue.len() > 0
    }

    pub fn clear_queue(&mut self) {
        // delete all items from queue
        self.queue.clear();
    }

    pub fn interval(&mut self, tick_amt: usize, msg: T) {
        self.jobs.push(Job {
            ct: 0,
            mt: tick_amt,
            msg: msg,
        })
    }

    pub fn set_rate(&mut self, bpm: i32, num_ticks: i32) {
        let ms = 60000000.0 / (bpm * num_ticks) as f64;
        self.tick_duration = Duration::from_micros(ms as u64);
    }

    pub fn update(&mut self) {
        for job in &mut self.jobs {
            job.ct += 1;
            if job.ct == job.mt {
                job.ct = 0;
                self.queue.push(job.msg);
            }
        }
        // trigger a thread sleep HERE
        let new_time = Instant::now();
        let elapsed = new_time.duration_since(self.last_time);
        let delta = self.tick_duration - elapsed;
        thread::sleep(delta);
        self.last_time = Instant::now();
        // end sleep calculation
    }
}

For the most part, it's sort of self explanatory except for the last few methods. The rate we set is a careful f64 calculation dividing 60 seconds as microseconds against our target BPM multiplied by the number of ticks we want the program space to have. The more ticks you have, the smaller the tick duration will become as it tries to divide time further into even slices, but floating-point imprecision can become an issue. My default tick rate is 64, which I don't think causes any floating point problems.

Pushing a job onto the job list using the interval() function will define a job with a specific timing interval based on ticks, not actual time. Because the BPM can be variable, it's better to convert all time measures into ticks, and leave precise time calculations to the update() function.

The update() function will loop through all jobs, bump tick counts, and copy them over to the job queue if a job is ready to be executed. Then, after that, we need to sleep for a precise amount of time and pause the operating system's thread. But the amount of time that has passed between all these actions is variable too, so we need to calculate a difference between our last update call to now, and then take the difference of the time passed against our target tick duration, and sleep that amount. The reliability here depends fully on the operating system's capacity and ability to sleep a correct amount of time, and thankfully on my system, it seems to work fine.

Now with a working scheduler, later in life we'll use it as part of our arpeggiator, but it looks something like this.

sched.set_rate(120, 64); // 120 bpm, 64 ticks
sched.interval(4, Message::UserInput);
sched.interval(32, Message::UpdateState);
sched.interval(32, Message::FlushNotes);

while true {
    sched.update();
}

Device Handling

Using the Portmidi library, based on some old code I had that I was porting forward, it came to be that lifetime markers were now required to represent the connection from software to hardware. Perhaps due to the complexities between Rust objects in program scope and how a physical hardware interface could seemingly fail at every possible step in a program, it was for the best. But it took some effort to learn how to utilize the lifetime annotations.

Portmidi provides a few different data types for working with MIDI interfaces. Namely, everything starts with a Portmidi context type that you can use to initiate virtual device objects. What we need or at least care about, are the ports initiated by Portmidi to create the input and output streams to a physical hardware device. Though I wish I could say it w ere as simple as "take a string and find me a hardware device", it's a bit not like that.

Portmidi's Rust library is a frontend FFI for C code, and the only way to identify a device is to use an integer. We can't pass it the name of a device, and simply put I wouldn't want to have to remember specific integers for different hardware devices. Instead, you can iterate over all known MIDI interfaces in Portmidi and locate your device that way, which I do indeed do.

for dev in ctx.devices().expect("Failed to query devices") {
    println!("Device: {}, id: {}", dev.name(), dev.id());
}

Upon locating the exact integer of your target device, you can then pass that integer ID off to Portmidi to create device streams that you can read and write from/to.

let in_dev = ctx.device(in_id).expect("Failed to get input device");
let out_dev = ctx.device(out_id).expect("Failed to get output device");
let input = ctx.input_port(in_dev, 1024).expect("...");
let output = ctx.output_port(out_dev, 1024).expect("...");

It's tiring, it's explicit, and I still haven't made this necessarily panic-free, but it works well. Now to throw it all together into a very cumbersome-looking constructor.

pub struct Device<'a> {
    pub input: pm::InputPort<'a>,
    pub output: pm::OutputPort<'a>,
}

impl Device<'_> {
    pub fn new<'a>(
        name: &'a str,
        ctx: &'a pm::PortMidi
    ) -> Result<Device<'a>, String> {
        let mut output_id: Option<i32> = None;
        let mut input_id: Option<i32> = None;

        for dev in ctx.devices().expect("Failed to query devices") {
            println!("Device: {}, id: {}", dev.name(), dev.id());
            if dev.name() == name {
                if dev.is_output() {
                    output_id = Some(dev.id());
                }

                if dev.is_input() {
                    input_id = Some(dev.id());
                }
            }
        }

        match (output_id, input_id) {
            (Some(oid), Some(iid)) => {
                let out_port = ctx
                    .device(oid)
                    .expect("Failed to find matching output device");

                let in_port = ctx
                    .device(iid)
                    .expect("Failed to find matching input device");

                Ok(Device {
                    input: ctx
                        .input_port(in_port, 1024)
                        .expect("Failed to open input port"),
                    output: ctx
                        .output_port(out_port, 1024)
                        .expect("Failed to open output port"),
                })
            }
            _ => Err("Failed to create a device context".into()),
        }
    }
}

I add some of my own workarounds here. When we search for matching device integers, I instead choose to store it as an Option<i32> instead of some raw i32 value. It plays better with the match syntax later to match and execute when exactly both integers have Some(x) value instead of if they are non-zero or not (or non-negative?). I think I could do with some improvements, but that might just be for later.

To use this, we pass a string of the name of the device (found by printing the queried devices) and the Portmidi context itself.

let ctx = pm::PortMidi::new()?;
let name: &str = "Launchpad MIDI 1";
let lp = Device::new(&name, &ctx).expect("Failed");

Writing to the device, which we'll do quite a bit, is done as follows.

lp.output.write_message([0x90, 64, 127, 0])?;

Notice the ? at the end? That's because each and every read or write operation can fail, so the result of these operations is typically always a Result<T,E> type where the error value is a specific error labeled ffi::PmError.

To alleviate some typing, I aliased a new type to make code writing aspects a little easier on myself.

pub type MidiRes = Result<(), pm::Error>;

If you know anything about error handling in Rust, custom error types are usually a pain, but here it's been mostly encapsulated by the Rust Portmidi library wrapping errors in their own error enumeration. Otherwise, we would consider using Result<T, dyn Error> to use dynamic dispatch on objects that implement the trait std::error::Error, which may or may not come with issues about some things not having the Sized trait as well.

Anways, moving on.

Arpeggiator

The arpeggiator section of code itself is a bit of an amalgamation of everything mentioned, and I fear I shouldn't even begin to share the struct definition until I at least explain a few different things. There might be a lot going on here, so let me first talk about my application and interface planning.

User Interface

The Novation Launchpad is mainly an eight-by-eight grid of buttons, with additional buttons along the side for various control aspects. Below is a picture of my planned layout.

The goal I have is to have the main 64-button grid act as the note table for the arpeggiation, with each column representing a note to be played every eigth note. I want there to be additional buffers adding up to have a grand total of 32 eighth notes, making up at least four whole bars of music. Most arpeggiators probably don't want or need that many, realistically, but this is isn't exactly some paid professional software I'm engineering.

The additional buttons I have planned are for:

to swap between four different "views" of the note array
play/pause the arpeggiator
swap the scale of the arpeggiator
dedicated quit button
control the octave with the right-hand vertical column

As far as a starter program, I think this is enough functionality to throw at this thing. I don't think it could handle much more, at least not without some advanced button hold-and-release timing mechanisms (one day).

Musical Scales

How many music scales do you know? I only know two, Major and Minor. There's way more than just that though, but the reality is that they just don't work in this 64-button grid context. Seven buttons is however just the perfect amount to use for a musical scale.

Each column will hold a note-like value between 0 and 7, where 0 is indicative of an "off" state for that column. The non-zero 1..7 range will then be translated to a musical scale accordingly via a lookup table array.

pub enum Scale {
    Major,
    Minor,
}

// Major: C D E F G A B
// Minor: C D Ef F G Af Bf
const MAJOR_SCALE: [u8; 8] = [0, 0, 2, 4, 5, 7, 9, 11];
const MINOR_SCALE: [u8; 8] = [0, 0, 2, 3, 5, 7, 8, 10];

fn calc_note(note: u8, scale: &Scale) -> Option<u8> {
    match (note, scale) {
        (1..=7, Scale::Major) => Some(MAJOR_SCALE[note as usize]),
        (1..=7, Scale::Minor) => Some(MINOR_SCALE[note as usize]),
        _ => None,
    }
}

Based on a scale and a u8 value representing a note, we can grab a new base note from the given scale array and return a Some(note) kind of type. If the note and scale don't match, we simply return a None. Why are my scale arrays an extra element long? Mostly because scales have seven notes, but to avoid having to do index subtraction I think it was simply easier to add an extra zero. If I was being snippy about saving two bytes of stack memory, I would change this.

User Inputs

I mentioned earlier that user input processing is a far more annoying task than most others here. If there were no user input, this program wouldn't be very complicated.

First, let's make a function called check_inputs(). The goal of this function is to handle user input from the device, inspect each message, and pass it along to the correct places. MIDI interfaces are simple analog electronics; they are either pressed or they are not pressed, with some devices having sliders, everything is mostly analog. When a button is pressed, the host device receives an interrupt signal and records a timestamp of it and noting it's voltage values representing a few different values. Namely:

the kind of MIDI message it is
the note of the MIDI message
the velocity of the note
the channel it was sent/received to

Many MIDI devices adhere to this basic standard, but some devices have their own interpretations of this standard message interface layout. For our case, our device can only send two kinds of messages, a MIDI message and a "note" message. The MIDI message is more like a control-ish type of message where the note message is more general purpose. When plugging synthesizers and software around, the MIDI kind of messages are not exactly ones you want sent to the synthesizer, but a message you may want to pass along to a digital audio workstation instead for various purposes like beginning or ending recordings, changing instruments, or what have you.

When we receive a message from the device, we will have to inspect it's type of message, and then pass along the remaining values to other parts of the code. In our case, the top row of buttons are MIDI, while the main 64-button grid is a note type. To read from the device stream, we need to poll some data from the analog signal via the Portmidi library, and have Portmidi convert the bytes into a MIDI stream of events.

const MIDI: u8 = 0xB0; // 176
const NOTE: u8 = 0x90; // 144

fn check_inputs(&mut self) -> MidiRes {
    if let Ok(Some(evts)) = self.grid.input.read_n(1024) {
        for evt in evts {
            let status = evt.message.status;
            let note = evt.message.data1;
            let vel = evt.message.data2;
            
            if vel == 0 {
                return Ok(());
            }
            
            match status {
                MIDI => self.top_row_dispatch(note, vel)?,
                NOTE => self.grid_dispatch(note, vel)?,
                _ => {}
            }
        }
    }
    Ok(())
}

Ignoring what the two inner function calls do, the idea is to route the messages to the correct places in the program so that the input can be handled correctly. In the case of the top-row elements, that's more for updating program state and changing the UI LEDs, while the grid is for note processing.

Scheduler, Part II

We made a job scheduler earlier, now we get to use it. Again, ignoring some of the functions we haven't gone over, here is how it is used.

fn update(&mut self) -> MidiRes {
    if self.scheduler.has_events() {
        let mut i = 0;
        while i < self.scheduler.queue.len() {
           match (self.scheduler.queue[i], self.playing) {
               (Msg::Quit, _) => self.quit(),
               (Msg::CheckInputs, _) => self.check_inputs()?,
               (Msg::UpdateState, true) => self.update_state()?,
               (Msg::FlushNotes, true) => self.flush_notes()?,
           }
        }
        self.scheduler.clear_queue();
    }
    Ok(())
}

Using our previous Message enumeration (that I've since renamed, amongst other things), we map out the goals of this music program. We match against anything in the active message queue and execute it. However, only some actions will get executed, because the arpeggiator program may be in a "paused" state, so things like updating internal counters or sending notes outwards will simply not happen unless the device itself is playing.

Each and every action has an associated ? operator to deal with the case where a read/write from the physical device may fail, so we do have to gracefully bubble the errors upwards respectively. Except the quit() function.

Since the scheduler has it's own update() function, we need to run our own internal state along with it. So to wrap it into a nice and easy to use interface, I simply create a run() loop function.

fn run(&mut self) -> MidiRes {
    while self.running {
        self.update()?;
        self.scheduler.update();
    }
    Ok(())
}

Controlled entirely by the self.running Boolean value, this will run until we get a Msg::Quit button somewhere in the scheduler, or if the user input hits the designated quit button.

Now we can focus on the next few aspects of handling input, flushing notes, and closing this project up and moving onto the next thing.

Death

It was around the first week of April that it was announced that one of our family members passed away. They were a close relative and had been almost all my life. It was sudden, unexpected, and truly heart breaking. I didn't really know how to process it at first. They were at my wedding not even all that long ago.

The process for the passing had taken some time, and unfortunately the situation never improved. I don't think I processed it very well, either. Soon we had funeral arrangements made, had family come from all over, and we said goodbye. I was incredibly sad that day, and still feel sad writing this weeks after the fact.

Grief has a strange way of taking a toll on you in ways you don't really expect. You think you're fine, until you try to move on from it one day at a time. I stopped working on the Launchpad arpeggiator once I learned he passed, and spent more time just appreciating life and staying away coding. I spent a little more time outside, spent more time with friends, and it took some time to feel normal again.

Then, once a few days have gone by, we're struck with yet another tragedy: someone on my wife's side passed away, not even a week from the last funeral. It was a weird, cruel joke that I at first didn't even entirely believe. I was already in a slight daze after the first one, that the second one really took both of us by surprise.

So again, more time with family, more focusing on each other, and more trying to repair some holes in our hearts. April wasn't even half way over, yet I already wish it was.

I think it will still take some time to mourn and move on, but I'm just glad I have my friends and family with me in this moment. It's these kinds of times when you have to be appreciative of what you have in the moment, and maybe sometimes just step away from the keyboard every now and again.

Final Touches

I have a lot in place, and the last few sections will touch upon some of the more difficult bits about keeping the system in a relatively sane state.

State

Last I left off, we touched on how the scheduler works, how receiving input works, but we never fully examined what is going on under the hood. As I fleshed this program out over the last month, some things were easier than others to do, like turning on LEDs or testing the scheduler. Yet we never fully touched upon what our internal state quite looks like.

I already talked about our concept of columns, and how the user touches a key on a certain XY position, the program will store the column it touched and the Y-value, and reflect that into the MIDI output later. First, how do we calculate an XY coordinate from our grid?

Each button on the grid is relatively in sequence, and each row is a multiple of 16. Each row starts with a zero, each button then adds one in sequence, until we reach the end of the row and then roll over to the next. This function accordingly will determine if a MIDI value is from the grid or not.

fn find_lp_xy(x: MidiVal) -> Option<(u8, u8)> {
    let nx = match x >= 16 {
        true => x % 16,
        _ => x,
    };
    match nx < 9 {
        true => Some((nx, x / 16)),
        _ => None,
    }
}

Leveraging my favorite type, the Option<T>, it will return a pair of bytes if it's on the grid, and None if not. We can then use this for the grid_dispatch() method from earlier to dictate the column mutation. Simply by taking the resultant x variable tells us our target column, with the respective y value being the "height" by which we want to set it.

First, what does our table of notes look like? Is it an array of pure integers? It very well could be, but I instead opted for an array of structs instead, called ArpCol.

#[derive(Debug, Copy, Clone)]
pub struct ArpCol {
    pub val: u8,
    pub note: u8,
}

impl ArpCol {
    fn new() -> ArpCol {
        ArpCol { val: 0, note: 0 }
    }
}

For something that only has two u8 values, I could compress it into a u16 and do bit shifting, but I was hesitant on that because I am not a bit shift master when it comes to shifting into smaller values (ie. u16 into a u8), and felt it would be far simpler to generalize it as an object instead.

Our arpeggio buffer will then be an array of thirty-two elements of this struct, generated this way.

buffer: [ArpCol::new(); 32],

Which I think looks weird at first, but it is sure simpler than looping over the array myself. Because ArpCol inherits the Copy trait, we can do this and it'll copy it over to every struct in that new array.

Now, to handle messages that were dispatched previously in check_inputs().

fn grid_dispatch(&mut self, note: MidiVal) -> MidiRes {
    if let Some((x, y)) = find_lp_xy(note) {
        if x == 8 {
            self.grid_io.output.write_message([
                NOTE, self.octave_btn[1], 0, 0
            ])?;
            self.octave = 7 - y;
            self.octave_btn[1] = note;
            self.grid.output.write_message(self.octave_btn)?;
            return Ok(());
        }
        let offset = ((self.buffer_index*8) + x) as usize;
        let new_val = 7 - y; // inverting the value

        let column = &mut self.buffer[offset];
        if column.val != new_val {
            // turn off old LED if there was a non-zero value
            if column.val != 0 {
                self.grid.output.write_message([
                    NOTE, column.note, 0, 0
                ])?;
            }

            // and turning on the new LED
            if new_val != 0 {
                self.grid.output.write_message([
                    NOTE, note, 127, 0
                ])?;
            }
            column.val = new_val;
            column.note = note;
        }
    }
    Ok(())
}

There's a lot going on here, so let me clarify.

The Launchpad I use is not really an eight-by-eight grid, but in actuality a nine-by-eight grid with one ninth column. The ninth column is treated sequentially, so our find_lp_xy() function will yield a value of (8,y) for any value that falls on that last column. That column I want to control the octave of the program, acting similar as the other columns, and has it's own associated array of data to use to turn on and off the active LED.

Secondly, if the input falls within the main grid, we need to find an associated column in the array and adjust it's values, as well as turning off the previously held LED if the column was active or not, reason why ArpCol has two integers and not just one. If there was an active LED, we disable it, then enable the new one. If the bottom-most button is pressed, the column is turned off entirely.

Since we can only access eight elements in one view, the self.buffer_index value is controlled by the top-row of buttons instead to swap the view between 0..3. That is handled by top_row_dispatch(), which I'll showcase here.

fn top_row_dispatch(&mut self, note: MidiVal) -> MidiRes {
    if note < 104 {
        return Ok(());
    }
    let idx = note - 104;
    match idx {
        0..=3 => {
            // if the target buffer is different than current,
            // reflash the entire UI and change the buffer index
            // mutate the buffer highlighted button as well
            if idx != self.buffer_index {
                self.buffer_index = idx;
                self.buffer_btn[1] = note;
                self.render_ui()
            } else {
                Ok(())
            }
        }
        4 => self.pause(),
        5 => self.play(),
        6 => self.invert_scale(),
        7 => Ok(self.quit()),
        _ => { Ok(()) }
    }
}

There are eight buttons with their own associated value, and four of those buttons are reserved for buffer swapping. When a user clicks on the buffer buttons, they are shown a view of that section of the arpeggio, which in turn involves reflashing the entire device with correct information. A quick MIDI reset code turns off all LEDs on the device, and we have a special function called render_ui() that renders each and every aspect correctly to the device. Some care is taken to avoid doing a full refresh, like in the case of the user pressing the same buffer button as the active view, which we check for to avoid doing a useless render_ui() call. The other actions in this code are pretty self-explanatory.

Visual Tracking Aid

Apart from listening to audible feedback, there is no visual element to the arpeggio to tell us exactly which note we're on. The best way to do that would be to possibly color the currently active column, but I had a moment of hesitation on that because it would add some more complexity to each ArpCol struct by then having another property to see if it was the active column or not.

You could consider avoiding that by instead having a separate variable outside that tells you what the current active note is, but preferrably I decided to do something else. I instead want a visual tracker at the bottom row of the grid telling me the column that way.

This sent me down a minor rabbit hole of how to implement something like this, and in turn, I went a different direction entirely.

Imagine a regular window-based program. Multiple windows with different functionality to control the program at different aspects. When you click, how do you tell what window event to process? For now I've been doing direct writing to the LEDs for visual UI changes, but what if I had a means of dynamically adding widgets for interactivity?

This method for me didn't work, but I can show you why, and what may change in the future.

Widget Approach

Let's picture we instead want to compose our program out of smaller bits instead of having one gigantic mess of a program. This is where using traits would come in heavily. Instead of hard-coding logic like which LED to write to, we could ideally separate the entire arpeggiator into a list of components that modify a shared state of some kind. It would involve decoupling the state of our program away from our current implementation, but I can show why that might be a good thing.

First a Widget trait. Each widget would need to get updated somehow, so we need an update() function. Perhaps we need a render() function to be able to even draw it to our device.

pub trait Widget {
    fn update(&self, MidiVal, &State) -> MidiRes;
    fn render(&self, &Device) -> MidiRes;
}

Then, when we handle user input, as we receive a MIDI message, we can loop over each widget in a widget container and do some checks somewhere.

for evt in evts {
    for widget in self.widgets {
        widget.update(evt.message.data1, &self.state);
    }
}

Then when rendering the UI, we can iterate over all the widgets and only render the "active" widgets.

for widget in self.widgets {
    widget.render(&self.output);
}

The code is generic enough that specific implementations can code at a high level without having to worry about small component-level details. But my issue with this style of composition is that it's not tightly-bound enough for this domain I want to say. It would also require undoing a lot of my existing code, which wasn't created with this in mind.

First is finding which widgets to interact with. I could add a new method to the Widget trait, something like is_input(), to dictate if a message is within defined bounds of the widget.

for evt in evts {
    for widget in self.widgets {
        if widget.is_input(evt.message.status, evt.message.data1) {
            widget.update(evt.message.data1, &State);
            break;
        }
    }
}

The break is needed to stop iterating over widgets once we found a match. Each widget would have to be coded with some means of dictating if input is bound to it somehow. Ideally with minimal widgets, there wouldn't be too much looping anyway, but audio is a tightly-bound world of programming so excess is really bad for performance.

Now in some cases it would make sense to decouple as many components from one another so things are little tidier and some components have less access to things or something, but I originally wrote this as a program to remain in one file. The component widget style would do better for reusability, but I've yet to develop another application after this one, and frankly I'm not sure what would really be next quite yet. Maybe something more generative and fun than geared towards music creation I guess.

But if I could go back, I would probably try to go for something like a widget composition program instead. That way buttons could be coded as something like:

pub struct ButtonToggle {
    pub button_kind: MidiVal,
    pub button_code: MidiVal,
    pub button_vel: MidiVal,
}

impl ButtonToggle {
    pub fn new(type: MidiVal, code: MidiVal) -> Self {
        ButtonToggle {
            button_kind: type,
            button_code: code,
            button_vel: 0,
        }
    }
    
    pub fn is_input(&self, t: MidiVal, n: MidiVal) -> bool {
        n == self.button_code && t == self.button_kind
    }
    
    pub fn update(&self, v: MidiVal, s: &State) -> MidiRes {
        self.button_vel = v;
        // ...
    }
    
    pub fn render(&self, d: &Device) -> MidiRes {
        d.write_message(...)?;
    }
}

Sorry about that long tangent, now where was I?

Visual Tracker Part II

The visual tracking at the bottom row is something that needed to be handled, and my approach sort of looks like it would belong in the widget family of composition, but not quite. I first started with a struct resembling the following.

pub struct Tracker {
    pub index: u8,
    pub btn: BtnArr, // type alias to [u8; 4]
}

The goal of the tracker is to move the index forward by one tick each time the arpeggio moves a note forward, keeping pace with the current program execution loop. This logic is tightly bound to the arpeggio's update() function and needs some special treatment because of that. Putting it behind a layer of widgets would have added a lot of weird complexity.

The tracker can only render when it's within the current buffer range, so when it's on note index 3, it will draw only when buffer 1 is selected. Meaning the range to check for will be min_index < cur_index < (min_index + 7). I add a few methods to make it slightly easier to manage the visual cue tracker.

impl Tracker {
    fn new() -> Tracker {
        Tracker {
            index: 0,
            btn: [NOTE, 112, 127, 0],
        }
    }

    fn in_range(&self, buffer_index: u8) -> bool {
        let bmin = buffer_index*8;
        return (bmin <= self.index) && (self.index <= (bmin+7));
    }

    fn update(&mut self) {
        self.index += 1;
        if self.index == 32 {
            self.index = 0;
        }
    }

    fn move_right(&mut self) {
        self.btn[1] += 1;
        if self.btn[1] == 120 {
            self.btn[1] = 112;
        }
    }
}

The tracker itself is not responsible for it's rendering, only because (again) my code was not coded around the idea of component-style management. If I had a more decoupled means of sharing the device streams, I would consider re-doing this whole thing, but without direct access, I simply stuck to mutating a small array of integers instead. Mutating a small array is still easy enough, and this implementation alone doesn't have any side effects, so you could argue this code itself might never cause a real problem.

All that's left is really glueing it all together and talking about the update_state() function which sits at the core of everything.

Ticking State

Our arpeggio has an internal counter that we must update, and that counter is used as the index for the note array. Each time a call to update_state() is needed, it moves the counter forwards until it hits its limitation and resets back to zero. On top of that, each time the program is bumped forward, small UI changes will be needed as well, ones that are not glued to any particular process as of yet.

fn update_state(&mut self) -> MidiRes {
    // bump the note index counter
    if self.playing {
        self.index += 1;
        if self.index == 32 {
            self.index = 0;
        }
    }

    // turn off the tracker's previous LED
    // do this before we "move" the button
    if self.tracker.btn[2] != 0 {
        self.grid_io.output.write_message([
            NOTE, self.tracker.btn[1], 0, 0
        ])?;
    }

    self.tracker.update();
    self.tracker.move_right();

    // turn on the tracker's LED if it's "on screen"
    if self.tracker.in_range(self.buffer_index) {
        self.grid_io.output.write_message(
            self.tracker.btn
        )?;
    }
    Ok(())
}

First thing we do is take care of the index. This part can absolutely get re-written into a different struct, since I use this tick counter methodology in a few different places. Not yet though. After that, we need to play with the visual tracker, since that one sits outside of user input and is done autonomously.

Flushing Notes

Almost at the end, a remaining piece of the puzzle is writing MIDI notes outwards to a different port. The thing I wanted to do is be able to have my Rust program serve as the middle man between the Launchpad and any software synthesizer on a computer. It receives all input from the Launchpad, writes MIDI messages to the Launchpad, and writes MIDI notes to a virtual sink elsewhere.

Conceptually this part is pretty simple. We look at our current index, access the note array, and convert it to a musical MIDI note. Only if the column is considered active, however.

fn flush_notes(&mut self) -> MidiRes {
    let col = &self.buffer[self.index];
    if col.val > 0 {
        if let Some(base_note) = calc_note(col.val, &self.scale) {
            self.midi_out.output.write_message([
                NOTE, base_note+(self.octave*12), 127, 1
            ])?;
        }
    }
    Ok(())
}

Using the music scale lookup tables, it converts a column value to note, additionally offset by a defined octave, and writes it to our MIDI output stream (ignoring the weird redundant naming).

Rendering

Rendering is sadly a little uglier looking since there's a bit of duplicate code here from the tracker, but it's necessary for the moment until a new widget composition approach evolves.

fn render_ui(&mut self) -> MidiRes {
    // clear board for a full wipe
    self.clear_board()?;

    // draw UI elements
    self.grid_io.output.write_message(self.buffer_btn)?;
    self.grid_io.output.write_message(self.pp_btn)?;
    self.grid_io.output.write_message(self.scale_btn)?;
    self.grid_io.output.write_message(self.octave_btn)?;

    // draw tracker if it's on screen
    if self.tracker.in_range(self.buffer_index) {
        self.grid_io.output.write_message(self.tracker.btn)?;
    }

    // render all cells
    for c in 0..8 {
        let index = ((self.buffer_index*8) + c) as usize;
        let col = &self.buffer[index];
        if col.val > 0 {
            self.grid_io.output.write_message([
                NOTE, col.note, 127, 0
            ])?;
        }
    }
    Ok(())
}

It's pretty straight-forward. This is full-clear redraw of the whole "scene", and involves clearing the buffer and redrawing every element on a blank device. Because I don't have that fun widget approach, it involves a little manual labor in writing every call to each button specifically, as well as doing the logic for the visual tracker. Otherwise, running over all the columns and converting the value to a color value is simple since we just saw what flushing the notes looked like.

If the self.buffer_index was in a separate object, then we could spin the visual tracker into a widget and combine all the widgets into one list and centralize it all under one widget.render(&device) function, and that's where I think the program would look a lot better. Perhaps in the future programs I make.

Layout

The last thing to share is my struct layout for this program. I saved this for last specifically because it is confusing, and trying to explain it all in the best way involves explaining each individual aspect as I go along, instead of showing one gigantic mess of a definition.

pub struct Arp<'a> {
    pub midi_out: Device<'a>,
    pub grid_io: Device<'a>,
    pub running: bool,
    pub playing: bool,
    pub scheduler: Scheduler<Msg>,
    pub index: usize,
    pub buffer_index: u8,
    pub buffer: [ArpCol; 32],
    pub buffer_btn: BtnArr,
    pub pp_btn: BtnArr,
    pub scale: Scale,
    pub scale_btn: BtnArr,
    pub octave: u8,
    pub octave_btn: BtnArr,
    pub bpm: u8,
    pub tracker: Tracker,
}

The first two items are two different Portmidi devices. One is for the physical device, the other is a virtual output sink for sending our MIDI messages somewhere else in the circuit. Then there's our playing and running Boolean values, doing what they do best. Then our scheduler that holds jobs targeting our message type.

After that, it's a buffer index that we use to keep track of which column we're inspecting. Then the buffer itself, fixed as a 32-wide column buffer. Then it's every single button and button-related integer we need to render all the relevant items on the device. It's definitely hideous to look at, but for now, it does work.

Compiling Pure Data

I need to test this somewhere, and sadly the only method I know best is using Pure Data. Pure Data normally is a great toolkit for coders to throw together audio programs since it can do just about anything you need it to. It however, did not want to play ball at first. And I soon discovered why.

Pure Data is normally a program you can compile to make it work with Portmidi. But soon I realized that my new system of Void Linux, it would not be that way. Looking at the ldd output, you can see that it isn't compiled with Portmidi, due to a lack of a dynamic link to libportmidi.so.

$ ldd $(command -v pd)
linux-vdso.so.1 (0x00007fa0166d0000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007fa016449000)
libasound.so.2 => /usr/lib/libasound.so.2 (0x00007fa01633b000)
libjack.so.0 => /usr/lib/libjack.so.0 (0x00007fa0162e9000)
libfftw3f.so.3 => /usr/lib/libfftw3f.so.3 (0x00007fa016149000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007fa015f5f000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fa0166d2000)
libcelt0.so.2 => /usr/lib/libcelt0.so.2 (0x00007fa015f44000)
libopus.so.0 => /usr/lib/libopus.so.0 (0x00007fa015ee4000)
libdb-5.3.so => /usr/lib/libdb-5.3.so (0x00007fa015d24000)
libsamplerate.so.0 => /usr/lib/libsamplerate.so.0 (0x00007fa015bb6000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fa015800000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fa015b92000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fa015b8b000)

I wondered why this was, so I went over to the void-packages repo for Void Linux build packages, and there it is, no Portmidi support in sight.

// void-packages/srcpkgs/pd/template
pkgname=pd
version=0.55.2
revision=1
_ver=${version%.*}-${version##*.}
build_style=gnu-configure
configure_args="--enable-alsa --enable-jack --disable-portaudio --enable-fftw"
hostmakedepends="automake libtool pkg-config"
makedepends="alsa-lib-devel jack-devel tk-devel fftw-devel"
...

Retrospectively, compared to my old Archlinux system, the build file shows that it does use Portmidi.

// archlinux/packaging/packages/pd/SRCBUILD
...
makedepends=(
  alsa-lib
  fftw
  gendesk
  inkscape
  jack
  portaudio
  portmidi
)
...

It wasn't a simple mistake, the compiler flag and support just isn't there. On top of that, portaudio is also disabled too. That's just plain annoying considering Pure Data's audio is either ALSA or OSS, and both are plain bad.

If I wasn't going to get it from my distro provider, I had two options:

complain on Void packages GitHub repo
just build it myself and move on with my life

...and obviously the second option was much more appealing.

Pure Data for the most part doesn't work all that well without Portaudio, as Portaudio is able to treat Pipewire as an ALSA device. This means I can use other programs while I work within Pure Data, so I don't have to literally turn off all my other audio sources to get anywhere.

On top of needing Portaudio, I'll still need Portmidi anyways, so I need to figure out how to actually compile Pure Data myself.

Following along with the Archlinux SRCBUILD, I am able to build it myself from scratch with all the correct library links in place. Though granted I need to install a few Void packages to get library headers. Eventually I found the correct combination of flags.

./configure --enable-alsa --enable-fftw --enable-jack \
            --enable-libpd --enable-libpd-utils \
            --enable-libpd-instance --enable-libpd-extra \
            --enable-portaudio --enable-portmidi \
            --disable-oss --without-local-portaudio \
            --without-local-portmidi

This generated me a binary with the correct dynamic links.

$ ldd ./bin/pd
linux-vdso.so.1 (0x00007efd10de7000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007efd10b64000)
libasound.so.2 => /usr/lib/libasound.so.2 (0x00007efd10a56000)
libjack.so.0 => /usr/lib/libjack.so.0 (0x00007efd10a04000)
libportaudio.so.2 => /usr/lib/libportaudio.so.2 (0x00007efd109d3000)
libportmidi.so => /usr/lib/libportmidi.so (0x00007efd109c8000)
libfftw3f.so.3 => /usr/lib/libfftw3f.so.3 (0x00007efd10826000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007efd1063c000)
/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007efd10de9000)
libcelt0.so.2 => /usr/lib/libcelt0.so.2 (0x00007efd10621000)
libopus.so.0 => /usr/lib/libopus.so.0 (0x00007efd105c3000)
libdb-5.3.so => /usr/lib/libdb-5.3.so (0x00007efd10403000)
libsamplerate.so.0 => /usr/lib/libsamplerate.so.0 (0x00007efd10295000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007efd0fe00000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007efd1026f000)
librt.so.1 => /usr/lib/librt.so.1 (0x00007efd1026a000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007efd10265000)

Booting it up, I was able to get into a session of Pure Data that could actually receive my software-generated MIDI messages. Hooray!

Testing in Pure Data

This post has gone long enough, so I want to wrap it up and move on.

I threw together a Pure Data patch that would receive a notein input and convert it to an audio sound. I didn't need a fancy synthesizer that would be modulated, just a osc~ audio stream going right to dac~. Also using a vline~ to generate a small decay envelope to make it sound more note-like.

And indeed, it does work. It is able to iterate through all my columns, play some sounds, and I'm able to control it all through the device. Generating sound is (ultimately) not something I'm ready for yet on my own with Rust, but one day, perhaps.

One thing you can do is hook multiple MIDI devices together and use them in conjunction with one another. Here I am using the Launchpad with my laptop and also a Novation Control grid. The only limitation is how much power your USB buses can deliver to each device at this point, before having to rely on external USB power blocks.

Conclusions

I think I had more fun doing this than most projects I've done in the past. It's been long enough since I started that I am frankly glad that I can put this one to bed, finally. There's a lot of work to be done with it, still, to make it a little sturdier. Maybe one day I'll add some additional scheduling tasks to break things up into widgets instead and re-write the shared state aspect of it. But for a 45+ day long project, I think I'm satisfied.

Overall, things I learned:

Scheduling and timing with precision
Maintaining program state across multiple functions without fear
Better understanding of what works in Rust and what doesn't
A little more audio knowledge insight in Linux
Designs I should try next program

Things I should probably do soon:

Re-think how octave is calculated. Not all octaves are useful.
Save compositions to file and be able to load comps from file
Program arguments so device ports aren't hard-coded
Try porting it to a different hardware device

The code is online for anyone who wants to view it in full detail with more comments. I may create more projects in that repo in the near future depending on my creativity. I'm a little exhausted from both this and the other events that happened, so I'm ready for a new project.

Thanks for reading. I think I'll come back in May with something new.

House of WADs

2025-03-08T20:00:00Z

Last month, I finally picked up my copy of House of Leaves and read it fully within a week. The rabbit hole I am soon to describe is slightly related to House of Leaves, though I will not talk about it since I don't think I could really do it justice. Instead, I'd like to talk about a Doom game modification called MyHouse.wad.

I started the recent process of dusting off my old Rust projects and trying to spruce them up, doing various changes or version bumping for the compilers. I found my old project waddler, a Rust project to export Doom game data, and started working on it. And I felt I had the perfect thing to try it out on.

The Beginning

In 2023, a now infamous Doom WAD was uploaded to the internet, and it eventually made internet waves with a video by a YouTuber called "MyHouse.WAD - Inside Doom's Most Terrifying Mod". I recommend watching it if you haven't, it's a fantastic video deep-dive.

The WAD itself is a complex piece of work, that had seemingly been made in secret by many people, and breaks some of the traditions about Doom's original engine. It does this by only being playable in non-traditional Doom source ports, mainly because the map makes use of portals. Because of portals, the map itself seems to have sections of the map on top of one another, which is not possible in classic Doom.

The game itself, without going into too much detail, is a sort of spin on the story of House of Leaves into video game format. There's some story-telling that goes beyond House of Leaves and makes it into it's own tale, but for the most part, anyone familiar with House of Leaves should directly pick up on what is actually happening in MyHouse.wad.

MyHouse.wad has a lot of complicated map design with its use of portals, and no visible automap can be seen in game. When I initially watched the video, picked up MyHouse.wad and played around with, I didn't think too much else beyond how awesome it was that something like this existed and had so much love and passion poured into it. Seeing how it was based on House of Leaves even made me want to read it. I eventually got a physical copy of it, and I waited far too long to start reading it.

Once I picked it up and finished it last month, I still had some House on the brain, and it lead me back to MyHouse.wad. I wanted to go in and extract the maps from MyHouse.wad for fun.

The Code

Going back to waddler, I wanted to debug MyHouse.wad for fun and see if I couldn't export it's map data to share it with friends. Surely it can't be too hard, right?

To get to where I wanted, which was to export maps of levels, there isn't too much involved. The main concept of a WAD is that it is a binary-encoded blob of data that keeps an index table of where things are, called Lumps. A lump is a bookmark type object that tells you where data lives, and where you can find it in a data pool. The WAD, at the top of the file, tells you where to find lumps, and where the data pool is. A WAD has to have some properties to tell you if it's an IWAD (an id Software original WAD), or a PWAD (a player-made WAD). In order for a game to be playable, it has to have an IWAD primarily, so PWADs are made on top of IWADs and can replace levels and assets entirely.

The first step is to create a basic data type to parse header information from the WAD, so I created two struct types to get me going: a Wad and a WadHeader, seen here in action. Upon opening a file as an array of bytes, Rust can select slices of it as in-memory views and hand it off to various functions without ownership changes or hard copying. Since Doom uses a Little Endian format for data, a lot of this code ends up being indexing a u8 array of bytes and doing bit-shifting math to convert it to logical data types, ie. converting four bytes into a DWORD or 32-bit unsigned integer. I have a bunch of functions for it here.

pub struct WadHeader {
    pub is_wad: bool,
    pub wadtype: u32,
    pub numlumps: usize,
    pub lumpaddr: usize,
}

pub struct Wad {
    pub name: String,
    pub header: WadHeader,
    pub lumps: Vec<Lump>,
    pub levels: Vec<Level>,
    pub is_hexen: bool,
}

Once a WadHeader is crafted, you can gather a slice of bytes and convert it to a list of Lumps by doing more byte conversions. Since Doom is heavily documented, it takes little effort to gather struct definitions off the Doom Fan Wiki to see how Doom is made and what drives it. Here's my Lump struct definition, which involves some byte search and checking to see if it is a Level lump or not. The Level type is the next step to finding geometry for each level.

pub struct Lump {
    pub name: String,
    pub posn: usize,
    pub size: usize,
    pub start: usize,
    pub end: usize,
    pub is_level: bool,
}

A Level is a data type that can contain geometry, and is the main thing we need when we want to export map visuals from a WAD. When iterating through all Lumps in a WAD, we collect any and all geometry between one Level lump to the next, as they come typically in sequential fashion. Once you reach the next Level lump, you take whatever you found and bind it together via a struct that can keep it all in memory. For my needs, I only kept Vertex and Linedef objects in memory and ignored other non-geometric data for now.

pub struct Level {
    pub name: String,
    pub vertices: Vec<Vertex>,
    pub linedefs: Vec<LineDef>,
    pub things: Vec<Thing>,
}

At this point, with all Levels accounted for, I can move on to printing out maps to a Scalable Vector Graphics format, meaning anyone can take it and scale it to whatever resolution they see fit, if they wanted to do something like make T-shirts from maps or something of the like. Most raster formats of graphics involve compression, and that's a bit outside the scope of my project for now. To convert a map to SVG, you just have to take each Linedef in a level, take it's two corresponding vertices, and print an XML string to an XML file buffer. I sort of over-engineered this by adding a whole SVG library that wasn't entirely super necessary, but it was a chance for me to play with generic traits. In a nutshell, the map SVG generation sort of looks like this.

pub fn level_to_file(buf: &File) {
    buf.write(svg_header);
    for linedef in &level.linedefs {
        buf.write(linedef.to_svg());
    }
    buf.write(svg_footer);
    buf.to_disk();
}

There's more to it there, but sadly this code is a little dated still. I came back to it, only years later, and had to make a number of changes for it to be compliant with the 2021 Rust compiler. It works fine still. Here is a map of MAP02 Underhalls from the original Doom 2 game.

The House

Now to get started, let's answer the immediate question: does waddler work on MyHouse.wad?

No, no it does not.

MyHouse.wad is actually not the name of the file you can download from the forum post about MyHouse.wad. What you download instead is file called myhouse.pk3, which is a ZIP archive of all the assets used. I did not quite realize this was a ZIP until I tried to use waddler on it, and it failed each and every time.

Upon downloading myhouse.pk3 and unzipping it, you get several files, and it is quite different than what I was expecting for a WAD file. As it turns out, this is an entirely different format for Doom engines to use, and it even dates almost being two decades old now. The format is called Ultimate Doom Map Format.

Now, none of my code works on this format for one simple reason: UDMF is not binary encoded, at least not all of it. A WAD still starts out as a binary blob, but almost everything in the WAD is actually plaintext, meaning a human can visibly read it instead of needing something like Little Endian to convert the bytes. Being plaintext has positives and negatives, but the positive is that it is easier to debug and extends the original engine of Doom to allow for floating-point types. The original Doom code typically only uses 16-bit signed integers, giving maps a max of 65,000 values to work with. Going to 32-bit for integers and floats adds a whole lot more range.

There's also something of a meme about Doomguy, the main character, being able to withstand falls of thousands of meters due to there being no fall damage in the original Doom game. Clearly he'll only get stronger with 32-bit types.

It took some time to sort of put everything together, but I was able to dump maps from this new UDMF format that MyHouse.wad uses. The internet attention on MyHouse.wad has long passed, but my curious nature got the best of me while I was working on waddler.

Upon unzipping myhouse.pk3, the first thing to grab my eyes is a 73MB file called house.wad sitting next to all the other files. Anything that has a *.wad extension is still a Doom WAD, but the contents are wildly different.

...
-rw-r--r-- 1 steve steve 360K Jan  8  2023 hellfish.wad
-rw-r--r-- 1 steve steve 2.8M Dec 12  2022 HF2NME.wad
-rw-r--r-- 1 steve steve  73M Apr 16  2023 house.wad
-rw-r--r-- 1 steve steve  57K Oct  2  2022 kd_smoke.wad
-rw-r--r-- 1 steve steve  814 Sep 11  2022 LANGUAGE
-rw-r--r-- 1 steve steve 2.5K Jan  7  2023 LOCKDEFS
-rw-r--r-- 1 steve steve 7.6K Apr 16  2023 MAPINFO

The only relevant parts from waddler that matter here are only a few: Wad, WadHeader and Lump are the only types needed, everything else is actually not used. Since my old types are fixed on the original Doom engine, I would need new types to account for floating-point types anyway, and I'd sooner not try to make generic type parameters on the original types made for classic Doom files.

I had to make a new program to dump some info about UDMF WADS. For that, I wrote a small program to dump the Lump values from a UDMF wad, and it gave me a better picture about the UDMF format.

fn main() {
    let mut arg_iter = args();
    arg_iter.next();

    for fname in arg_iter {
        let (wad, data) = match parse_into_both(&fname) {
            Ok((w, d)) => (w, d),
            Err(e) => panic!("Welp. {}", e),
        };

        for lump in &wad.lumps {
            println!("{:?}", lump);
            if lump.name == "TEXTMAP".to_string() {
                let txtmap = &data[lump.start..lump.end];
                let mut i = 0;
                while i < txtmap.len() {
                    print!("{}", (txtmap[i]) as char);
                    i += 1;
                }
            }
        }
    }
}

Running this on the house.wad file gave me the following output.

MAP01 <0xC .. 0xC> (0 bytes)
TEXTMAP <0xC .. 0x39E03C9> (60687293 bytes)
BEHAVIOR <0x39E03C9 .. 0x39E2115> (7500 bytes)
ZNODES <0x39E2115 .. 0x485A325> (15172112 bytes)
SCRIPTS <0x485A325 .. 0x485E901> (17884 bytes)
ENDMAP <0x485E901 .. 0x485E901> (0 bytes)

This on a normal WAD would display a large number of Lumps, but because it's a UDMF format, we only have a few. At it's core, a map in UDMF has two markers, MAPXX and ENDMAP to tell you where a map starts and ends, and the TEXTMAP slice itself which occupies most of the data. The other lumps are there for scripting or graphics purposes.

A TEXTMAP slice is, like mentioned, a plaintext representation of a Doom level. The difference here is that instead of using Lumps to store all vertices and lines, it uses a plaintext format almost akin to JSON. Data in TEXTMAP looks vaguely like this.

thing // 1
{
property1 = ...;
property2 = ...;
}

vertex // 2
{
x = 0.0;
y = 0.0;
}

linedef // 3
{
v1 = 2;
v2 = 2;
}

sector // 4 {
...
}

While this is human readable, it's not efficient to try and make a map this way, and is best left to visual editors like SLADE3 or Ultimate Doom Builder. For us to read data like this, we need a bit of a parser to read through all this text and convert it to data types we can use like we did previously.

For my UDMF map minimum viable product, I would need at a minimum all the LINEDEF and VERTEX structs inside TEXTMAP. Extending my first program, when I encounter a TEXTMAP in a WAD, that's when I can start performing data parsing. First I need to map out two new struct types for these UDMF format values.

#[derive(Debug)]
struct UVertex {
    x: f32,
    y: f32,
}
impl UVertex {
    fn new() -> UVertex {
        UVertex { x: 0.0, y: 0.0 }
    }
}

#[derive(Debug)]
struct ULinedef {
    v1: usize,
    v2: usize,
}
impl ULinedef {
    fn new() -> ULinedef {
        ULinedef { v1: 0, v2: 0 }
    }
}

The first is UVertex, a new type for UDMF vertices. Instead of i16, they now use 32-bit floating point values. I'm operating under the assumption that all the values used are 32-bit and not 64-bit. If they were 64-bit, the memory load would be much higher for a map with this much geometry.

ULinedef is different in that it uses usize types to index for two vertices. A usize is like saying a uint32_t or uint64_t, but usize is needed for indexing arrays in Rust for type safety, and may compile to to the largest integer available. I don't think house.wad has over four million vertices, so I think it's fine no matter what.

The text format of TEXTMAP needs a small parser to run through it and collect important properties about each struct type. When we encounter a line that looks like vertex // 123, we start collecting vertex properties until we reach a matching } ending bracket. We need a union data type that can contextually swap between different objects we're collecting values for.

union UParser {
    vertex: UVertex,
    linedef: ULinedef,
    // other types of objects ...
}

A union in any code base is only as good as a tag along side it to help us know what we're working on. We would need an enum type to help us know exactly what we're currently doing.

enum ParseT {
    Vertex,
    Line,
    None,
}

struct ParseD {
    tag: ParseT,
    val: UParser,
}

But here this almost feels... Dated. Why not leverage values inside variants of the enum instead of going the tagged union route?

enum PState {
    Vert(UVertex),
    Line(ULinedef),
    None,
    // others ...
}

Turns out, this is way simpler. Each variant of the PState will keep a value with it's tag, so we can better pattern match against an enum as opposed to all that sort of weird union behavior we'd have to work with.

After a few hours of tinkering, I ended on this, and got steps closer to making maps out of MyHouse.wad.

let line = buffer.next_line();
let mut pstate = PState::None;
let mut pid = 0;
let mut linedefs: HashMap<usize, ULinedef> = HashMap::new();
let mut vertices: HashMap<usize, UVertex> = HashMap::new();
let mut in_struct = false;

if line.starts_with("vertex // ") {
    pstate = PState::Vert(UVertex::new());
    pid = slice_to_usize(&line[10..line.len()]);
} else if line.starts_with("linedef") {
    pstate = PState::Line(ULinedef::new());
    pid = slice_to_usize(&line[11..line.len()]);
} else if line.starts_with("{") {
    in_struct = true;
} else if line.starts_with("}") {
    in_struct = false;
    match pstate {
        PState::Vert(v) => {
            vertices.insert(pid, v);
        }
        PState::Line(l) => {
            linedefs.insert(pid, l);
        }
        _ => {}
    }
    pstate = PState::None;
}

The parsing algorithm here is pretty simple. Our PState is our mutable state, and our scanning triggers are simply <thing> // <id>. If we find an object by it's corresponding label, we mutate pstate to become that object shape and value, then the minute we reach an open bracket, we begin scanning for values. Once we reach a closing bracket, we copy the data off pstate and into our respective data structures for holding geometry.

The next section involves scanning for details, and matches on the current pstate variable.

if in_struct {
    let lend = line.len() - 1;
    match pstate {
        PState::Vert(ref mut v) => {
            if line.starts_with("x = ") {
                let decimal = slice_to_f64(&line[4..lend]);
                if decimal > max_x {
                    max_x = decimal;
                } else if decimal < min_x {
                    min_x = decimal;
                }
                v.x = decimal;
            } else if line.starts_with("y = ") {
                let decimal = slice_to_f64(&line[4..lend]);
                if decimal > max_y {
                    max_y = decimal;
                } else if decimal < min_y {
                    min_y = decimal;
                }
                v.y = decimal;
            }
        }
        PState::Line(ref mut l) => {
            if line.starts_with("v1 = ") {
                let decimal = slice_to_usize(&line[5..lend]);
                l.v1 = decimal;
            } else if line.starts_with("v2 = ") {
                let decimal = slice_to_usize(&line[5..lend]);
                l.v2 = decimal;
            }
        }
        _ => {}
    }
}

It's a little tiring having to do exact byte matching for strings, but it's not the worst thing in the world. For each specific key of a TEXTMAP data type, you would have to run through a list of if/else statements to figure out exactly what you're looking at, and how to map it to which property of a struct, and if it has any exterior logic like with the vertices seeing what the largest X/Y values are for this given map.

With each ULinedef and UVertex made, we will put them into a HashMap collection. The HashMap is driven by a binary tree under the hood, and gives us the best chance at fast access, as opposed to a linear buffer like Vec<T> which may not guarantee the same performance as the buffer grows. I won't spend time benchmarking this, though, just need as fast as possible access to the data.

We're nearly at the finish line. Our original SVG code is not exactly easily adaptable to these new types, and would involve modifying a non-zero amount of code to get it to work, as my SVG code was not made with these new 32-bit types in mind originally. We'll have to create a new SVG printer, but that's okay.

// generalized form of the SVG printer
let buffer = File::new(format!("{}.svg", wad_name))?;
buffer.write(svg_header);

for linedef in &linedefs.items() {
    buffer.write(svg_line(linedef));
}

buffer.write(svg_footer);
buffer.close();

My first pass on this resulted in a 50,000 pixel square map, which was so large, that no program I had would be able to open it. GIMP, despite it's best efforts, would not let me open it at all. When importing it, I scaled it down to half that, around 25,000 pixels squared. Then, I was able to see the full thing in detail. Because it's so large, I can't show it or even host it here. I will share a link so you can see the full thing. Fair warning, it is 10MB in size.

I think I can say I did a good job. It took days to get here, but I'm happy.

Exploration

In my map program, I print out exactly how many vertices and lines we ended scanning from the WAD.

Working on /home/steve/Downloads/MyHouse/house.wad...
Found textmap
Stats:
Number of vertices: 133967
Number of linedefs: 164452
Created map: /home/steve/Downloads/MyHouse/house.wad.svg

With 133,000 vertices and 164,000 lines, that's almost a 3:1 ratio for both vertices and lines per pixel in this map. That's huge. But upon opening the map, it's ... Not very occupied. In fact it's very sparse at points.

I will admit, curiosity got the better of me. Despite having seen the MyHouse.wad video only one or two times, I wanted to re-experience it for myself and perhaps explore a little bit.

To annoyingly say it again, Doom functions as a 2D geometric map projected to create a 3D space experience. However, no rooms can be on top of one another, which was a hard limitation of the Doom engine, and later solved by other FPS games like Duke Nukem with the Build Engine.

All these different map spaces exist because of how the game uses portals to connect one area to another to give off a presentation of a realistic 3D space. There are twenty or so different house because of the progression of MyHouse.wad inter-connected by portals and teleport tricks.

It had been a while since I explored the house, so I wanted to check out all these places for myself via a warp command. The map would give me coordinates that I could use to move around, so I wanted to check out some of the places to see exactly what they were, especially for places I don't remember seeing before.

I warped around a lot just to see what each place was, and discovered a lot of teleport traps that would inevitably teleport you to the near-unescapable Backrooms. The creators most likely made a lot of traps like that to prevent people from genuinely trying to noclip out of bounds.

There were some parts that I was not entirely familiar with. There's almost an entire area dedicated to House of Leaves itself. I'm not particularly certain how to get to these places without warping, and even more uncertain of what their purpose in the game may even serve.

In House of Leaves, there is a circular staircase that is central to the plot that takes characters on a strange and dangerous journey. Perhaps this is what these two staircases are in reference to, but after warping around, I couldn't find them or if they even functioned at all.

A larger room with a staircase-like object sits to the right, with an even bigger empty space equally as dark.

My memory of the video essay may be lacking, I'm certain the author covers this, but if I had to wager a guess, this was simply a nod to the source book itself, nothing more. It may be the closest thing we have to a House of Leaves video game, though.

The Portals

Now that I have geometry all mapped out from UDMF format, there begs the next question that bothered me: would it be possible to trace all the portals used in MyHouse.wad?

Portals are a mechanism used by ZDoom and other ports such as GZDoom to enable non-Euclidean geometric rendering. As the visuals are rendered, if one section is marked as a portal linking it from zone to another, the view will pass through to the connected area via the portal. Very much like how the game Portal works.

In order for portals to work, Linedef objects must be tagged with a special action tag called Line_SetPortal(). A Linedef carries more values than it's vertices, and can have a special tag and up to five argX tags to supply values to the function. In this case, arg0 on the Linedef is the target line it wants to treat as a portal. Time to edit ULinedef again.

#[derive(Debug)]
struct ULinedef {
    v1: usize,
    v2: usize,
    special: usize,
    arg0: i32,
    arg1: i32,
    arg2: i32,
    arg3: i32,
    arg4: i32,
    twosided: bool,
}

Each arg is treated as an i32 since some values can go negative, but the special value itself seems to be positive only. I update the scanning process to account for these new fields, but I won't bore you with six more if statements.

Since we already iterate over all the lines in the map, it would be simple to add another line as well that points to a target portal line if one is found. We simply check if the line has a special code of 156 (the numeric equivalent of Line_SetPortal()) and find it's matching line from arg0.

To draw portal links on the map, I figure it would be easy to simply draw from one line's vertex to the other. I felt it easier to draw from one set of vertices instead of calculating a middle point on both lines. As I iterate over all the lines, I check and see if they have the special tag match the code for Line_SetPortal(), and then draw another line to that target line. By putting lines into a HashMap instead of a Vec, it would hopefully speed up the access speed to find target lines.

if (linedef.special == 156) {
    if (linedef.arg0 != 0) {
        let index = linedef.arg0 as usize;
        let target_line = linedefs
            .get(&index)
            .expect("No matching line");
        let c = vertices
            .get(&target_line.v1)
            .expect("No matching vertex");

        buf.write(svg_line(linedef.v1.line_to(c)));
    }
}

I kept track of how many portals were getting added, and we ended up with a little over 400 different portal lines in total. Wow!

The map is again, very big, so I will share it here. There are red lines going everywhere from the main house, and if it were higher quality, you would be able to trace every single portal line from it's source to the destination. The multiple variations of the house are used here to really showcase how window portals are used.

This doesn't really take into account the full story of the game. There is logic that goes beyond simple line portals. There can be logic bound to sectors in the game, by which if you stood on a particular sector, you might get warped to a completely different spot, and it's hard for me to automatically parse most of this exterior logic without writing a full-blown ACS parser.

The End

I don't quite know what compelled me to go down this route and do all this work. It felt satisfying towards the end. I had spent about a week total doing this digging, and it was fun to share it with my friends and enjoy MyHouse.wad from a different technical perspective.

I made a lot of mistakes along the way, and incidentally driving me House of Leaves crazy in turn. My biggest mistake is what I'll call off-by-twelve, where I spent far too much time trying to understand why I was off by twelve bytes during my read process.

When reading the TEXTMAP, I noticed that the last sector of the blob was short by about twelve bytes. It looked weird, but I actually dug into the GZDoom source code for how UDMF is parsed and read from TEXTMAP. How GZDoom reads TEXTMAP is nearly identical to my Rust version, but their structure for parsing differs from mine by a bit.

In C/C++, you typically will pass an empty struct into a function via pointer to mutate it. Then, once the struct is done being mutated, you store it somewhere and move on. In that same file you can see a function like ParseSector().

void ParseSector(sector_t *sec, int index)
{
    //...

The return type of ParseSector() is void, meaning no data is returned from the function. In reality, it should be a bool, to indicate whether the struct was completely read and mutated without issue. Because my TEXTMAP ended so briefly, that sector was disregarded, whereas with GZDoom, it would not be. Should GZDoom's parser find a random end of file, it would still read and store the incomplete struct it was working on.

However, the issue I soon realized is because my TEXTMAP was off by twelve bytes. The reason why is because I never correctly offset my memory range to account for the twelve bytes from the WadHeader from earlier.

// array of bytes with all data, including WadHeader
let data = &all_bytes;

// fixed array of data (HEADER_W=12)
let data = &all_bytes[HEADER_W..];

I failed to adjust my data range, and I only realized it by the time I was dumping the plain text from the SCRIPTS lump. I was scrolling through pages of GZDoom source code for a good few hours before I moved on and soon realized my own mistake. I had believed that MyHouse.wad might've actually been a broken WAD, which would make no sense considering it was made by a level editor. It drove me a little crazy to say the least.

That doesn't even take into account a problem I had with GZDoom. Since MyHouse.wad's release, the current version of GZDoom (v4.14) doesn't even work with MyHouse.wad anymore. In a GitHub issue ticket, v4.14 was a compatibility-breaking release, and MyHouse.wad would need to be updated for the future. Which, I think I can safely say, will not be happening anytime soon.

I went back through GZDoom releases and grabbed v4.10, which is (to me) the last working version of GZDoom I could grab. OpenGL or Vulkan issues aside, v4.10 gave me the least problems.

Today, days after the 2nd anniversary of MyHouse.wad, I wrap up my little research project here. If you haven't played MyHouse.wad, maybe give it a try. I think I spent enough time in MyHouse.wad, but it was a fun and at times addictive journey to learn more about the Doom format. Maybe one day with my map knowledge, I can create something even more interactive like a full WebGL explorative map of MyHouse.wad, or any WAD for that matter.

Thanks for making it all the way down here!

Data Vault

2025-02-08T20:00:00Z

With an on-going threat of tariffs coming to the United States, and the trade war being as flip-floppy as possible right now, before 2024 ended, I bought about $800 worth of computer parts to work on my next pet project: a data vault.

My wife and I have a lot of photos, and frankly it's upsetting to think that photo, video and other media storage is largely an unsolved space. I run a personal Nextcloud instance, and even that has issues that I'm not totally happy about. The problem with Nextcloud is that the app user experiences are almost always terrible, and I can't go anywhere off WiFi without Nextcloud trying to back up my entire photo album over mobile data. I played with the settings, and couldn't find a happy ground where Nextcloud wouldn't upload gigabytes of photos on mobile data.

Nextcloud would be the most obvious place to start; we have different brands of smartphones, and are unlikely to ever want to pay the premiums for common bigtech cloud storage solutions, like Google or Apple or shudder Dropbox. A year or two ago I compared the different plans that cloud hosting had, and here's an up-to-date look at various cheap plans.

Google Drive - $2/mo for 100GB of storage
Apple iCloud - $0.99/mo for 50GB
Dropbox - $9.99/mo for 2TB of storage

Google and Apple are neck to neck to try and out-do each other, while Dropbox only offers a 2TB plan. Google and Apple also have 2TB plans for the same price, but definitely cater to an audience that may or may not want to commit to a whole two terabytes of data. Apple charging 99 cents I'm sure is what a large percentage of users are already subscribed to and think very little about handing over a dollar a month.

However, compared to just buying an SSD, you get a lot more value. Running your own SSDs for storage comes with complex software problems, which is why almost nobody might ever go down this road in the first place. But I have a renewed interest in using my skills to save money and protect our personal data as a priority. While you get benefits of having cloud data, there are also negatives, like:

Google/Apple/etc using your photos for AI training without your consent
Providers losing your files or downgrading their file quality to lower resolutions
Providers increasing their pricing, giving you zero negotiation power
Providers failing to give you responsible security (Celebrity Leak circa 2014)

Having instantaneous access to your entire collection of memories and files is a boon for many people, but having them always-online comes with certain trade-offs too. Hosting your files locally comes with benefits, and also has trade-offs. There is no scenario where something is just good and has zero downsides here, and that sucks.

I have technical abilities, and an interest in protecting and archiving our data for as long as possible with minimal complexity. Therefore, I bought my pet project that I simply call the Data Vault.

Requirements and Parts

I have built computers, but I've never built tiny computers. The thought of building tiny computers is a freaking nightmare. If you've ever built giant computers, think of how much space you have to wiggle your hands and get into uncomfortable spaces. Isn't it nice having room to be able to fit cables without fear of scraping your hands or having to take everything out and start over?

Well, that's what you don't get with tiny computers.

I have two desktop computers, and I didn't need a third one to take up more space. I wanted to build a Mini-ITX build to take up as little space as possible. One could argue that you can also achieve similar solutions using Raspberry Pi NAS clusters, but in my experience, RPIs are not the greatest when it comes to building something like this. You can do it, but getting the drives to be connected and performant is different, because RPIs natively don't come with PCI-e ports for something like NVMe drives or even SATA ports for 2.5 drives.

Originally, I didn't want to buy the parts for this until a little bit later, but because of the threat of tariffs, I bought hardware before it would get a little overpriced. The tariffs have yet to come into play in the U.S. markets at time of writing, but I think I'll be glad that I bought it sooner rather than later.

Now, I have interest in keeping this server of mine as low power as possible as to not add to the array of devices I have consuming power. I want to keep it as low as possible, within human limits. I would like to get around 1 to 2TB of storage, because storage prices are still a little crazy and I'd rather buy as we grow instead of trying to collectively buy 30TB of unneeded space.

In the end, here's what I went with.

AMD Ryzen 7 8700G 8-Core 4.2 Ghz 65W CPU
Gigabyte A620I AX AM5 Mini-ITX Motherboard
Corsair Vengeance 32GB DDR5 RAM
Crucial P3 Plus 1TB PCI-e NVMe
Silverstone SFX 300W 80+ Bronze PSU
Matching Silverstone Mini-ITX case

In total it cost me roughly $779 to put this all together. I ended up getting an extra SSD from Christmas soon after, because I don't really know what else to ask for in Secret Santas, so I figured why not something useful? I can't keep getting Raspberry Pis for Christmas.

After the Black Friday / Cyber Monday weekend sprint of running numbers and making sure all the parts worked, I clicked my order, and had so many fears of nothing fitting at all. After roughly two weeks, everything came in, and then my work began putting it all together.

Build

Truth be told, the build wasn't without some pains. On the outside, it was a pretty trivial setup. Take out the motherboard, put in the RAM, CPU, NVMe drive, throw on the stock cooler, open up the case, slap in the board and PSU, close it up and turn it on. There was a good enough amount of space for it all to fit, but some components had to be completely removed from the inside. The case came with a top-mounted fan, but it conflicted with the CPU cooler, so I removed that case fan, which thankfully was modular and easy to take out.

However, the wiring and tight spaces soon started being a hassle. My PSU wasn't actually modular, which is my fault, but I was working with tight margins anyway. I didn't buy a video card, and I opted for an on-chip integrated graphics stack with the 8700G, which has honestly pretty good performance (more on that later). The PSU came with extra cabling I didn't terribly need, like molex connectors that just wouldn't be doing anything. So I sort of had to tuck them out of the way as best as possible. And that's already on top of me bending the CPU power connector as best I can and other wires to fit the tiny case design.

Sticking in another hard drive was way too difficult. There's a small box case in the corner meant to hold 2.5" drives, and sticking the extra drive in there and then putting the cabling in from that to the motherboard and the power connector was really tiring. I knew what I was getting into, but I was seriously looking to avoid having to take out the motherboard and start from an empty space.

Once everything was finally in, I was very ready to throw the power switch on and see what it is capa- oh... It doesn't turn on.

I ripped open the case top and started doing electrical debugging where I could. I spent a few hours looking at the manual to be sure I had pinned the power and reset buttons correctly along with the other pins for HDD light (who uses this?) and such. Frustratingly, the power button just wouldn't do anything.

I detached the main CPU power connector and did things the annoying way: putting in a jumper cable into the power pin of the connector itself. This is a method to test if the CPU can be turned on at all by connecting the main power-on pin to a ground and seeing if the power button can close a circuit or not. By routing the pin to ground, it will turn on when you hit the switch on the PSU instead of a power button.

My electrical engineer friend helped me debug the issues. First we wanted to make sure the PSU was working, so we routed the power on pin to the ground and flipped the switch. The PSU fan was indeed working. The next issue would be to figure out why the CPU isn't kicking on to boot. For that, we poked around with a multimeter and figured out why: the power button isn't working.

The wires are glued to the button switches on the front of the case. The reset button was working, but the power button was not. They were seemingly hot-glued in place, but maybe that messed up the connection. I didn't particularly feel like un-gluing it and soldering the circuit back to work, so instead I wired the power on switch to be the reset button instead. I'll probably never notice anyway.

After that, it turns on and boots to BIOS. I want to get testing, so I pick the most generic operating system I can, Linux Mint, to be the test OS. I burned it to a flash drive, plugged it in, installed, and it's fully functional. Hooray no part failures.

Performance

In the first few days with the device, I wanted to make sure everything was pretty battle ready. I wanted to make sure the hard drive could store files, the memory was working, and the CPU was relatively in a good state. To do this, I downloaded video games.

This section will probably be uninteresting for people invested in building data servers for local use, but as a fellow gamer and computer scientist, I was really interested in seeing just how far this computer can go in intensive workloads. This CPU is only rated to draw 65 Watts, so what damage can it really do?

Half-Life 1

The hit shooter from 1998, built upon Gold Engine, obviously runs fine. I would argue that when this game came out, this was the only game that ran flawlessly on widely available hardware. You could probably install this on a work computer in 1998 and play it at a very high FPS. Gone are those days of work computers having Unreal Tournament.

I couldn't get MangoHud to work here or on other Half-Life titles (maybe due to missing lib32 packages), so net_graph 1 it is. For a game this old, it's as cranked out as it can be. I'd be afraid to let it go off the frame limiter.

Quake 1

Quake 1 is a classic boomer shooter alongside Half-Life 1, and between id Software and Valve, they shared many common principles for FPS development at the time. Half-Life 1 is even a port of the the Quake engine, if you can believe it.

Quake 1 got a new rendering engine some years back to use Vulkan, and the game certainly looks a lot better than it's olden days of software rendering. As such, game is flawless and runs perfectly fine.

Half-Life 2

Skipping ahead just a few years, Half-Life 2 shows up in 2004 and barely runs on anyone's computer. I wanted this game day one and it barely ran, but I was also a kid who didn't exactly have money for new hardware. How does Half-Life 2 fare now after it's 20th anniversary and new code patches that Valve deployed?

Playing through the entirety of Route Kanal up to fanboat section, I never noticed a drop. I had the settings on pretty moderately high, removing overkill stuff like antialiasing or motion blur. Love ya Half-Life 2.

Sniper Elite 3

Sniper Elite 3 is a game from 2014, and I remember playing this on the original Xbox One. I absolutely adore the Sniper Elite series, so this was a good game to try out and see what would happen.

In some spots of the game, there's stutter. I tried booting up a random survival mode and had trouble maintaining framerate. In other parts, the game can reach a solid 60 FPS, but I imagine the levels are very large and cause some slight GPU stutter at times.

Dota 2

I haven't turned on Dota 2 in roughly six years. This game was pretty hellish on most PCs during it's early days of 2012, so what's Dota 2 like now, almost 13 years after it's initial beta phase?

I played a bot match in single player by myself. The game was surprisingly short. My bot Earthshaker crushed it and I just farmed. Per MangoHud, the game was hitting highs of 120 frames per second with lows of roughly 80. That, is honestly really damn good.

If I could test it in a higher resolution, I certainly would. I know Dota 2 isn't exactly the fanciest graphics game out there, but I know it as a performance hog, and seeing it play with these frame rates is something else to me.

Black Mesa

Why not play yet another version of Half-Life 1? Black Mesa is a faithful port of the original game to a more modern port of Source 1, re-building up the entire game from scratch and creating new models and assets for just about everything. It first appeared in 2015 and finished it's early access period in 2020.

I don't want to say it's as buttery smooth as past Half-Life games, but I think this was a good push again to see if the hardware could keep up. Turning the frame cap off, in some places the game could hit 300 FPS, while in more open areas, it would hit maybe a rough 90-120 FPS. It would look so much better on 1080p.

Crysis

Couldn't get it to run. 😢

Update in 2026: I did not use the Crysis Remastered edition as I did not own it at the time. However, Crysis Remastered turns out to be very usable on Linux with Proton, even supporting HDR.

Witcher 3: Wild Hunt

Witcher 3 has always been something of a performance benchmark since it's release almost ten years ago. It's a chunky game weighing in at least well over 80GB of disk space, and I can feel the CPU churn as it pre-caches Vulkan shaders. And as I was somewhat expecting, the results were not that great here.

The game itself is an open world style RPG, and requires a ton of graphical calculations. It can hit a okay-ish 60 FPS range if you quite literally turn everything off, and if you do that, the game will honestly look worse than it's 2007 predecessor The Witcher.

For science, I cranked up as many non-raytracing settings and let it rip. It looks... Okay... Not great, and my resolution makes it look terrible. Getting a stable 30-40 at high settings is not the worst for a CPU consuming as much power as it does.

This is my stopping point. A CPU from 2024 playing a 2015 game seems to be the absolute upper-bound limit, and I think that's totally fair for something that only consumes 65 Watts of power, though I wish I had a power meter to actually see how much it uses during these high loads and see if it ever overdraws it's TDP.

Update in 2026: there is a new Witcher 3 Remastered Edition on Steam. No idea how well it will run on lower-end hardware at present.

File Sharing

Back to the story, it's pretty clear that this thing can really rip when it needs to. Video games are not doing the same tasks as basic file storage tasks, but I went into this wanting to make something that'll really stand the test of time. I don't want to upgrade this for at least five years, and if I ever have to, I'd like to re-purpose it later in life at least.

As far as file sharing purposes go, I've decided to settle on Samba. I don't know terribly much about Samba other than it's a file sharing process that was made for Windows fileshare and works across other platforms. I do have a wide array of systems, so the compatibility is nice.

There comes my next point of contention: do I wipe the server and start fresh and make it a GUI-less system? For the testing during this post I was using Linux Mint Desktop edition, and it worked fine with no real hiccups during all that gaming testing. Do I start over?

Based on my work space here, I opted not to. I have a small TV setup here next to my desk that I tend to use for old consoles that don't have HDMI support. This TV is the only thing that has RCA and composite inputs in my house, and it'd be a pain in the butt to find something that might still have those ports.

Right now, how my networked devices all sit, is roughly under my desk on a L-shaped desk setup that I have with my tables from IKEA. The TV sits a bit out of the way, and it is plugged into some consoles and wired into the data vault underneath. I think for now I'd like to keep this setup, because it's pretty fun to have a device that's high-powered and can do many different things. If I need to manage it, I can turn on the TV remotely and do some file work that way, or log in remotely through SSH and do it all via the terminal. It might be burning more CPU than it has to could without a desktop, but it's more about utility than total power saving right now.

Also, if I really need to, I can simply disable the desktop auto service and simply login and create the virtual desktop each time I need to go in. It's not difficult.

The steps to install Samba are just installing samba and turning on the Samba service through systemctl . You will have to edit the root /etc/samba/smbd.conf file that's there, but right now since my drives are currently not unified, I have two folders with one on each drive. I have yet to really figure out if I want to try to set up an LVM or not to treat them both as a unified virtual disk. In concept it might work; in practice it might be annoying. Not sure yet.

Edit your smb.conf file with root permissions and add a section for your shared folder at the bottom.

# /etc/samba/smb.conf
[YourFolderName]
path = /home/your/folder/path
browseable = yes
read only = no
guest ok = yes

Once that's done. Enable and restart the smbd server.

$ systemctl enable smbd.service
$ systemctl restart smbd.service

Once the service is running and your folders are mounted properly through Samba, you should be able to point a file browser at the URI like smb://yourhostname and you should immediately see all folders you mounted. For Windows you need to do something like \\yourhostname, because of course Windows has to be unique. However, you need user permissions to read and write files, so you need to adjust Samba permissions on the server.

$ smbpasswd -a localusername

You can make one global user and have a bunch of computers all use that, or have individual users all with their own passwords to access the drive. For ease, one user is probably fine. There's definitely a way to do completely anonymous access with no login or password, but... I don't think that's a great idea.

Testing

The only way to see if it works is to throw a ton of data at the dang thing. I have a bunch of Linux ISOs in my downloads folder, so I threw them all at it and it went perfectly fine. The Gnome file browser showed about a 90.2 MB/s file speed when transferring a Bazzite Linux over the wire. Not bad. I tried using dd to copy a file from one disk over to the Samba server, and it was way slower, and I'm sure it's a matter of the Samba protocols that dd isn't exactly contextually aware of. It copies at about a speed of 1.8 MB/s. I'm better off using scp at that point.

For reads, I threw some of the digital copies of movies I had and tried streaming them from the file explorer. Again, boring result, but it works perfectly fine. Probably only uses up a few megabytes of bandwidth, which is not very difficult to handle given the hardware.

Overall, I've thrown gigabytes of data at the thing, and it works without any real hitch. It's been super easy to share data across computers on my network now.

Backups

Backups are something I'm generally terrible at. I've always been something of an external hard drive backup guy, and I've already done it a number of times in the last year transferring data off old hard drives into externals. A backup system would be automatic and provide some kind of ability to easily roll back into a previous system state, a la Windows System Restore. However, I'm not really looking for a System Restore clone, more I'm looking for some setup that can help me better secure my files in the long term.

The one problem I see is that if and when my power inevitably goes down due to a storm, I predict that there could be some theoretical data loss or unexpected shutdown to occur. If every device I have is running and all connected and powered, then the loss could potentially impact all devices connected via power, say external USB hard drives.

I don't know if I have the best way to counteract that, other than using off-site backups, which, coming full circle here, can cost money. It's always a good practice to have two copies of everything you have digitally as a backup strategy, and probably even better if they are in two different locations entirely. But as far as saving money is concerned, perhaps there's an alternative.

Let's say I have two hard drives A and B. Hard drive A is plugged in via USB and creates backup jobs automatically once a week or so. Using a basic rsync command would copy anything you want over to hard drive A and can calculate a difference list just as well. If this drive was left plugged in 24/7, it would be subject to the same power loss damage the system could occur if say an electrical surge hit everything on a circuit.

Hard drive B's job would be that of an unconnected device, one that I would have to manually apply an update for by plugging it in and running the rsync operation. Then once that's done, unplug it and leave it somewhere for safe keeping, or just keep it on top of the server box. By being disconnected from the electrical circuit, it would be a degree safer than the always-connected hard drive, with the only disaster scenario being that of a total EMP blowing everything out completely. Or a house fire.

I don't think manual backup strategies are always ideal, but if it's to protect your family's lifetime photos and videos, maybe it's worth doing once or twice a month. Who knows.

Steam Streaming

One benefit of having Steam on the server is that it can technically be used as a proxy video game device for a different computer on the network. Granted, my living room already has a gaming box, so there'd be no need for that to exist. The data vault is technically weaker in the hardware category than all my main devices, Steam Deck included, but it could technically be a means to off-load work or save on storage. Maybe one of my computers is running out of storage and it's not worth the time or effort to reinstall big or small games, so network streaming might be a small advantage.

To test this, I wanted to try both a wired and wireless test. I picked, again, Half-Life 1 since it's the smallest 3D game I know of.

The first test was on LAN, hard wired in, streamed the game through the intro of Blast Pit. There is not much network hopping here, it is as fast as the video encoder can be on the server, and it's not a slouch. It plays pretty okay, lag not noticeable. I cranked it up to 1920x1080, no notable downgrade. The colors felt a little washed, but I expect that's due to stream encoding quality.

The next one is done with Steam Deck wirelessly, and aside from my terrible Steam Deck controls for the classic game, it ran... Fine. The only limitation here felt like how fast it would wirelessly transmit data from the router to the device. If your wireless network is healthy and fast, it's fine. The Steam Deck only has a resolution of 1200x800, and the resolution of the server was scaled down to meet it.

It's hard to capture a screen resolution from the Deck. Since it's being streamed over Steam, the screenshot function on the Deck was bound to the server, not the Deck, so I couldn't capture a picture on the Deck. I would have to go to Desktop mode and apply a desktop screenshot, and I sort of didn't try that hard.

Overall, it's not a terrible thing to keep some games on. If there's a particular game that doesn't support cloud sharing on Steam, but you want to play it from multiple devices, this isn't a bad way to go. Nice little benefit of the APU.

Other Considerations

I'm not a data hoarder pro, and I can't say I know all the ins and outs. I think I know enough to get me started, that's just about it. I've been using Linux computers for years and proper data hoarding has never really been my hobby.

There's other things to take into account here that I may as well list out.

TrueNAS is a common OS I've seen out there that is popular among many data hoarders. It's built on top of FreeBSD, but I can't say I've ever touched a BSD platform in my life. It might be a bit of an adjustment.
OpenZFS (or ZFS) is largely the most popular drive partition type I've ever seen, but I'm not certain how stable it is still on Linux due to how long it took to get ZFS into the main kernel. It's very preferrable over things like btrfs or ext4, but I'll have to do more reading to figure it out.
I'm seeing HexOS as a recommendation for an all-in-one home server, but it looks to be closed source and behind a massive $199 USD license fee. That might be worth it for some people, but I'd rather stick to open systems personally...

Closing Thoughts

I wanted to get this done before the end of last year, but it just took so much effort to get it all done. What should have been a few days has just been a backburner two-month project to get going. Coupled with re-wiring my network stack and cutting ethernet cables, I learned a lot and got things organized much nicer.

The cost of everything combined was pricier than I would have liked, but again, my intention is to make this thing last for as long as possible with as much software and hardware support for as long as possible. I wanted it to be low power consumption and also power efficient to save on power bills where possible. I have an adequate amount of storage for the time being, and will upgrade as we go along.

The one concern I think I have in all this is the rate of solid state drive failure. Some may read this and ask what my fail case is for that scenario, and right now? I don't have one. I have never had a hard drive or an SSD completely fail on me. Based on Backblaze's solid disk and hard drive reporting from being a crazy cloud storage provider, solid state drives are pretty safe overall, and have a lower failure rate than mechanical. Now I know that all SSDs are destined to die, the question is really a matter of when.

I like to estimate that my SSDs may last longer than that of hard core data centers mostly for the reason that my disks are not exactly going to be under full R/W load all the time. It's a networked storage solution that I don't think I could find a way to even put it under load all the time right now. It should hold our media and files safely, and just simply pray for the best. I have thought about RAID setups, but right now I only have two disks, and the redundancy between two disks probably isn't super beneficial. Would need a third disk I think before RAID works well (ie. RAID 5).

Thanks for reading, hopefully I've inspired you to build your own data box of your own.

A Maybe in Python

2024-09-06T20:00:00Z

I had a weird conundrum come up. It comes at a weird time at work, where projects are shifting around and features are growing, but as I was battle-testing my code, I had a realization my code was simply not good enough. For that, I wanted to add a classical Monad from Haskell called Maybe.

The Maybe monad is implemented as follows:

data Maybe a = Just a | Nothing

That's it. It's two different states. One has a value, while the other has nothing. In Rust, this is Option<T> with done with the following:

pub enum Option<T> {
    None,
    Some(T),
}

Both have similar concepts, Haskell enforces a bit more mathematical-ness in how they use monads, whereas Rust wants to add safety and nullable data without all that monad nonsense. This doesn't exist in Python, and I frankly feel like it probably should. I'll explain my thoughts.

The Existence of None

The project I'm on involves parsing data from a source, but the data from JSON can sometimes end up being None. A usage of None would imply that there's nothing there, but something being None can still be a valid state (at least in our application; I'm not fond of it). While parsing objects, I sometimes run into a None value, and it's hard to wrap a program around None as if it were a "good" value.

>>> {}.get("key") # None
>>> {'key': {}}.get("key", {}).get("subkey") # None

This is the ordinary using dictionary's get method. You can add additional default returns when a key is not found. Otherwise, you will get None which implies a failure. Or, it might mean a success?

>>> {'key': None}.get("key") # None; a failure?

Is it dumb? A little bit. I wish my data source did not include a key with None, and I wouldn't have this issue at all, but I'm realizing the problem isn't that the data source uses None, it's more like... None alone doesn't go far enough to include reasoning.

While scanning deeply-nested dictionary objects, I need to evaluate key paths and see what they end up at, whether it's some nested value, or realize the possibility that the key path simply does not exist at all.

def get_nested_attr(full_key, obj, default=None):
    keys = full_key.split('.')
    
    curr_obj = obj
    for key in keys:
        if key in curr_obj:
            curr_obj = curr_obj.get(key)
        else:
            return default
    return curr_obj

This is an example function I use to scan deep-nested objects. It works for dictionaries, but it can be extended to work on plain Python objects as well with getattr() and hasattr(), should you want to go that route. It works by taking a string that looks like "key1.key2", break it apart, and iterate an object until we reach some final value. If we don't find a value, it returns None, implying that the key does not exist. It has a keyword parameter to mutate what the default value should be in case no existence is found, so you can change it how you want.

>>> get_nested_attr("key1.key2", {'key1':{'key2':500}})
500
>>> get_nested_attr("key1.key2", {}) # None

Logically sound, right? There's nothing really wrong with this, at least until you realize your source data happens to use None as a valid data type.

>>> get_nested_attr("key1.key2", {'key1':{'key2':None}})
# None... Is this right though?

After doing all that deep iteration, we end up with None, but it's not clear if it implies a success or failure. And frankly I don't want to start throwing exceptions, because it wouldn't even make sense to do so.

The obvious fix would be to then add a way to provide clarity on whether or not the thing you found was actually what you wanted.

def get_nested_attr(full_key, obj, default=None):
    keys = full_key.split('.')
    
    curr_obj = obj
    for key in keys:
        if key in curr_obj:
            curr_obj = curr_obj.get(key)
        else:
            return (False, default)
    return (True, curr_obj)

This just ends up adding further burden on the end user (me) to solidify code even more with blandness. Say we're checking for two keys in two different JSON blobs, we want to check for two key paths to be equal or not.

data1, data2, = {...}, {...}

found1, value1 = get_nested_attr("key1.key2", data1)
found2, value2 = get_nested_attr("key1.key2", data2)

if found1 and found2:
    if value1 == value2:
        print("Both are equal")
    else:
        print("They are not equal")
else:
    print("Data was not found in both")

I am not a very big fan of this pattern, simply because user burden is now placed on the end user (me, again) to remember that get_nested_attr() returns a tuple of values and not a single thing. It doesn't feel very smart, and honestly that if is really boring.

My head got started down the ol' Haskell brain train yet again and I wondered if encapsulation of data was needed here. Granted, encapsulating data with other data is pretty atypical, and we just did that with Python here, by returning a pair of values, one describing the other's situation. However, the code surrounding the unpacking of the pair is boring, tedious, and would just end up in a ton of boilerplate that I would have to deal with.

The Maybe Class

My frustration got the best of me. I put together a Maybe class, here's how silly it is.

class Maybe(object):
    __slots__ = ['value', 'has_value']
    
    def __init__(self, value):
        self.value = value
        self.has_value = False
    
    def is_some(self):
        return self.has_value
    
    def is_nothing(self):
        return not self.has_value

A class that takes in a value and wraps it into a new object with two methods to dictate if it is something, or if it's just nothing. Is it a weird class? Sure. Monads are not exactly first-class features Python has, and forcing this into a code base would no doubt get some resistance.

The thing to note is that has_value isn't even remotely mutated here. I left that out, because it's not the job of the object itself to mutate that value. I opted to leave that outside, because that's not everything. Let's introduce two helper functions now.

def Some(x):
    m = Maybe(x)
    m.has_value = True
    return m

def Nothing():
    return Maybe(None)

Meet Some(x) and Nothing(); remember these names? It's like I mashed Haskell and Rust together and got these names. I couldn't use None() because that's a reserved keyword in Python, for obvious reasons. If this were Ruby (science project!), we wouldn't need the parentheses for the Nothing() function. Now instead of returning a value from a dictionary search or a pair of values, we can use this instead.

(These are not class methods or static methods because I simply do not want to type Maybe.Some(x) or Maybe.Nothing() each time I invoke it. Too much extra typing.)

def get_nested_attr(full_key, obj):
    keys = full_key.split('.')
    curr_obj = obj
    for key in keys:
        if key in curr_obj:
            curr_obj = curr_obj.get(key)
        else:
            return Nothing() 
    return Some(curr_obj)

When we find something, we can wrap it into a Maybe object that has it's values filled out correctly, whereas if we find nothing, we get a Maybe object with it's fields pretty much left as blank as possible. How does that change our previous code using this function?

data1, data2, = {...}, {...}

lhs = get_nested_attr("key1.key2", data1)
rhs = get_nested_attr("key1.key2", data2)

if lhs.is_some() and rhs.is_some():
    if lhs.value == rhs.value:
        print("Both are equal")
    else:
        print("They are not equal")
else:
    print("Data was not found in both")

Admittedly, not much has changed, but burden has been slightly lifted off our shoulders for having to unpack a pair of values from a return tuple. Rust and Haskell can leverage the Maybe type to better effect with their core functionality, while our syntax does not necessarily allow us to do that nearly as much.

Helper Methods

Since it's Python, we can add some overrides to some methods that might prevent us from having to do so much boilerplate. First one is to add an override for __bool__() to reduce the weight-lifting of if statements comparing Maybe types.

class Maybe(object):
    # ...
    def __bool__(self):
        return self.has_value

The is_some() method and a __bool__() function were effectively the same, so why not add this in? Next is the __eq__() method to compare two Maybe types together and inspect the has_value attribute.

class Maybe(object):
    # ...
    def __eq__(self, o):
        if isinstance(o, Maybe):
            return (self.has_value == o.has_value) and (self.value == o.value)
        return self.has_value and (self.value == o)

# Testing it
>>> Some(5) == Some(5)
True
>>> Some(5) == Nothing()
False
>>> Nothing() == Nothing()
True

data1, data2, = {...}, {...}

lhs = get_nested_attr("key1.key2", data1)
rhs = get_nested_attr("key1.key2", data2)

if lhs and rhs:
    if lhs == rhs:
        print("Both are equal")
    else:
        print("They are not equal")
else:
    print("Data was not found in both")

The genius of short-ciruiting here means that if I'm comparing a Some(x) and a Nothing(), then it will not even proceed to the second argument of the and operation and will bail out. So if you happened to have something very complicated and requires time to compare, it won't execute if it's compared to a Nothing() value. Well, obviously, I guess. It also works on non-Maybe types by checking if the input is a Maybe instance since Python won't exactly do type enforcement for us.

Unpacking the values from a Some(x) value are trivial by simply doing Some(x).value, but this is technically a violation of the safety that Maybe is giving you. Granted it's Python, you can do whatever the hell you want, but it'd be nice if we were able to glue Maybe objects together somehow in a nice and friendly fashion. We would need a syntax like if let Some(x) like Rust, but how can we get close to emulating that?

The sad reality is we can't use something like with Some(x) as x to unpack the data, as that is not how with was designed in mind. Should you want to break out of with, you'd have to use an Exception, which I am not doing. However, nothing's stopping you from using... an Iterator!

class Maybe(object):
    # ...
    def __iter__(self):
        if self.has_value:
            if hasattr(self.value, '__iter__'):
                return self.value
            return iter([self.value])
        return iter([])

Now you can sequence Maybe code together using for as the means to group it all up.

data1, data2, = {...}, {...}

for lhs in get_nested_attr("key1.key2", data1):
    for rhs in get_nested_attr("key1.key2", data2):
        # lhs and rhs are now both raw types here
        if lhs == rhs:
            print("Both are equal")

The difference here is that any code under the first for loop will not be run unless we have a valid Some(x) type being used as if it were the iterator. Meaning any further unpacking is never executed unless the first condition is a success. The unpacked variable is bound for us to use throughout the expression without us having to check if the context is within a Some(x) first either. This is pretty cool.

# unpacking two Some() values
for x in Some(5):
    for y in Some(6):
        print(x + y) # => prints 11

# unpacking with a Nothing()
for x in Some(5):
    for y in Nothing():
        print(x + y) # => never executes!

The tradeoff is that for does not have a meaningful else construct that we could use in the bad case of if the iterator is a Nothing() type. For two-way branching in this case, you're probably better off using an if still, but there could be circumstances where this unpacking and executing abstraction is handy to have. I'm pretty fond of it, even if it abuses normal Python constructs and everyone might call me a sinner.

(There is an else construct for for blocks, but it's terrible and acts more like a goto block; you would have to use break to leave a loop to avoid executing the else statement, which is silly.)

Closing Thoughts

The more I work on my current project, the more I'm frustrated by how lackluster Python is in the world of abstract data types. I might need even more encapsulations like with a State monad just to do some crucial work that involves finer granularity and the ability to pass through certain data to record how a process went. Or hell, I wish I had an Either monad to do railway-oriented programming. Perhaps this is a starting step with the Maybe monad, as the others could be slightly derived from that alone.

I would think I'm not alone in thinking that Python needs to take more inspiration from other languages to help it grow up. I'm probably not the first one to try and implement a baby-sized monad like Maybe and see how it would interact with Python's syntax. If lambdas weren't so shunned by the Python community, I'd use them more here.

The scope which Maybe currently sits in my project is small; I don't know how much it will grow, at least it probably won't grow much out beyond my control, but I think I'm okay with that for right now. I'll eventually flesh out some kind of tooling to help me work towards collecting state and logging data. I do believe railway programming is a good design, even if Python does not immediately agree with that sentiment. Either way, I don't write about Python enough, but I thought this was worth a write.

Cheers and thanks for reading.

Deadlock-Proofing My Computer

2024-08-20T20:00:00Z

Hello again. Apologies for the slow updates here. There's a reason for it, and it's because I commute for work pretty far. I'm not happy about it, but it is what it is. Besides that, I figured I would write about my most recent computer adventure in a bit.

Deadlock

A few weeks ago, I stumbled across an invitation on Steam for a game I completely forgot about after hearing about it initially: Deadlock. Deadlock is a new and in-development game created by Valve, and the initial reactions to the game are pretty mixed, I would have to say.

Deadlock (now that it has a Steam page) is an iteration upon their tried-and-true Dota 2 formula. Dota is a game that I played for many years until I took myself off that addiction. Dota spans many characters, each with interesting abilities and fascinating gameplay that players subject themselves to learning to get better at the game. Dota has (if I was being completely honest) lost a lot of popularity, and the e-sports scene as such taken more interest in games like Counter-Strike 2, another game by Valve. Don't get me wrong, I love both games equally.

Valve was lacking for a number of years in the fun and casual multiplayer game space for a pretty long time. Ever since the likes of Overwatch was released back in 2016, Valve didn't really have a foot in the quick-and-casual game for people on consoles. Their closest was Team Fortress 2, which was an original title on the Orange Box collection years ago, but never made it's way over onto the 8th and 9th generation consoles. It's hard to say if a 17-year old game can stand the test of time, and TF2 itself has been subject to scrutiny from players due to a lack of updates from Valve. Many players feel it is unfair the treatment TF2 gets, and TF2 even has a bot problem that is out of control.

Fast-forward to 2024, beyond TF2 or Dota or CS, we've reached Deadlock. What the heck is Deadlock? If I had to describe it, it's a MOBA like Dota, but in third-person perspective, a multiplayer game that Valve has never done, and has players wondering if it's too many years too late. Smite was already sort of doing what Deadlock is doing now, except now it's under Valve's umbrella, and thousands of players are now playing Deadlock every day.

Deadlock is a third-person "hero shooter" where you (similar to Dota) farm creeps in auto-spawning creep lanes on a very large city map. Together with five other players, you fight against a team of six opposing heroes in a goal to take down the opponents' base before they take down yours. Simple format, very similar to Dota, but is it actually good?

For a shooter, I think it's pretty okay. It's also free! And really rings home a sense of junky shooters where you don't have to be perfect to do well. I think the laning stage where you farm is exciting, and I think the mobilizing your party or friends to do large-scale battles is also pretty fun. The emotional rollercoaster is pretty wild, and there's ups and downs with each game. There's also a lot of imbalance currently amongst the hero roster (thanks Seven).

PC Woes

Playing a new Valve game is not without it's own problems. My hardware is showing it's age.

The first instance of problems showed up in Deadlock when approaching big, large-scale battles with lots of things going on. My frames per second (FPS) would start capping out and hitting the low 20s when these battles were going on. Sometimes even when I get taken out of a fight and forced to respawn, the FPS would still be struggling.

The reason being? My VRAM count sucks on my really old graphics card.

My graphics card is not fancy; it's an RX 480 with 4GB of VRAM. It's cheap and it worked for years, and I never gave thought to upgrading it due to GPU wars during the pandemic. I basically put off for as long as humanly possible because... Well, I didn't really feel compelled to pay more for something I wasn't in total need of.

The crappy old GPU eventually lead to another circumstance of me playing Deadlock and weird things happening. During a real player match, I was forcibly booted from the game because my GPU blacked out and restarted my desktop environment. It wasn't a hard crash since I didn't have to press a power button; the GPU must've crashed from overwork and my computer saw it and likely just crashed the desktop as a safe measure. I was ejected from my voice call and my friends didn't hear from me until I booted back into the game and saw me moving around again.

The other issue is, my RAM seems to also be capping out at times. I have 2x8GB RAM sticks, which only seems to come out to 14.5GB due to what I'll call "RAM Math", and when playing the game, I tend to hit that capacity more often than not, and I leak over into swap memory pretty frequently. This was an issue even before Deadlock, as I can sometimes leak over 1GB of memory into swap when doing random things during the day.

So, both my GPU VRAM and my RAM are issues. I guess it's time to upgrade...

Part Picking

As someone who pays for electricity now as a full-functioning adult, I don't really want to pay more electricity than I have to. I would rather opt and keep my PSU which is a gold-rated 650W power supply. Picking a higher-rated TDP graphics card would most likely mean I need to up my power intake, and I don't really feel like upgrading to something like 750W if I don't have to. I'm pretty minimalist like that, I suppose.

Now, as far as graphics cards go, I would never pick an Nvidia card, so that rules out at least a good chunk of cards on the market.

Doing an initial dive into AMD Radeon cards, the last generation available is the 7000 RX series of cards, so I start my search there. Almost instantly, I'm laser focused on the AMD RX 7600 XT model card which looks to be almost exactly what I would want or need. Decentish performance, lots of memory available, generally an up-to-date lower end spec card which matches my power budget. I think the price is a little higher than it should be, but I think having more VRAM this time around would mean the card should easily last another five years before needing another replacement. Hell, I'd be surprised if I could even find a game that would push that limit.

The RAM, I was able to find two sticks 32GB each to bring my total RAM up to a whopping 64GB of RAM. Reason being is I was jealous of a laptop of mine having 64GB RAM, while my computer had only a quarter of that and I spend far more time on my desktop. It was time to grow up and spend the money.

While I was also at Microcenter, I picked up a new 140mm chassis fan for my top fan to replace. The old fan is starting to go, so I picked up it's replacement that should be much quieter.

(Aside: a Microcenter employee was teaching a young teen how to use PCPartPicker to help him decide his first rig, and I felt it was such a precious moment.)

Installation

The last time I was neck deep in my case was last year when I ripped out my motherboard to swap it for the AM5 compatible board for my Ryzen 5 7600X CPU. It might not be the most popular CPU, but I love it for the value I've gotten out of it. The motherboard replacement was honestly tiring, and I was excited to do an upgrade that wouldn't involve that.

First up, unplugging the graphics card, RAM and chassis fan. All simple tasks. I removed IO shielding locking the card in place, took out the RAM, and unscrewed the fan from the top of the case. Overall, not difficult stuff. Worst bit was finding what the hell I plugged my case fan into, and it was seriously plugged in the wrong place. Oops. Never noticed.

After the fan mistake was corrected, the fan went in with an easy four screws. For comparison, here's a clean fan versus the old one. Oof.

The GPU and RAM are also trivial installs, and I wish I could make it more interesting than simply "I drag and dropped the parts into the PC." I didn't even think to check if I had the correct pins from the PSU, but thankfully there was no change and the cards were equal in input pins.

RAM was easily swapped, nothing much else to do, other than clean up a whole lot of dust that I've ignored.

Post Install

The only thing left to do now is to turn on the damn thing. After changing the video card out, all my video inputs are different. I have one less HDMI port and one more DisplayPort, so one of my three monitors had to get cable swapped. That unfortunately means I have to redo my Sway monitor configuration to correct the output names and get that in order again. I also edit it in Gnome Desktop just to get GDM to play ball.

htop reports a correct amount of memory, my fan is spinning, and the video card appears to be video'ing correctly. Nothing much to report here.

Now let's take it into a bot match on Deadlock. Reason being is that Deadlock multiplayer servers aren't online during the day, typically only during the evening so they can gather data and make corrections the next day. Now, I play on pretty medium tier settings. I don't really enjoy antialiasing as it's usually a performance hog, and I like my textures pretty simple and don't do extraneous effects or screen space ambient occlusion, at least not without some extra testing.

Parts of the game are laggy, like initial loads of objects and textures into memory, but I think it holds up pretty well all things considered it's a game largely in beta. You can see that it's built off some Dota 2 parts, with the interface for settings being nearly identical (and some broken tooltip strings may actually refer to Dota).

Besides funny things like that, the stats in my new build look something like this while running Deadlock in a bot match.

Inspecting Mission Center now, the difference is pretty clear. While I might not totally cleared the CPU usage, I was able to at least give the game some headroom in that regard. The VRAM issue is totally gone, as the game tends to make my system hover around 5GB VRAM usage. So long as I'm not playing more than two instances of Deadlock, I should be fine. Here's my system specs after the upgrades.

Conclusion

Overall, I think I spent close to $600 in parts, and maybe I could have gotten better deals for better price points, but I can't really complain. I didn't have to upgrade the PSU, and I got enough RAM to last me a very long time, at least until AM5 is a thing of the past. Didn't have to get a whole new chassis, and it took me less than an hour to do all the replacement. I kinda like how simple my case is, but I wish my glass panel was less boring and a little quicker to get to the internals. I can dream about another case one day.

I apologize for the short post, but I wrote this almost entirely on a train as my commutes are long and annoying. I am going to try my best and catch up writing using this new phone process of mine, so I hope I start writing more soon enough.

If you like this post, please let me know and I'll try my best to improve. This one has more pictures than I typically do, and I'm trying to get into a better editing groove. One real pain point is GIMP is absolutely terrible on Sway WM, and I imagine it's a wlroots problem and GIMP trying to use Xwayland for everything. But besides that, I'll try my best to keep these posts slightly more entertaining.

Until the next post written on a train!

Much Ado About Tags

2024-06-27T20:00:00Z

And so begins the second week of the new website, and one feature has been missing for a while that I wanted to implement: tags. What is a tag? How are they made? Can we finally "word cloud" like many Blogspot blogs before us? Let's find out.

A tag on blogging platforms is for when you have too many posts and nobody knows where and how they want to start. If they're only interested in certain content, ie. you're posting a tutorial, or a drawing you made, you tag your content with whatever your post can be associated to, and the tags will help you navigate the posts with those matching tags. In essence, each tag generates an index of posts that have a matching tag.

This was a feature I designed Zeus in mind with, so it should be relatively simple to add.

Tag Mapping and Folding

In order to build an index of posts with related tags, we need an associative hash of tags to a list of posts that match said tags.

tags = {
    "life": [post1, ...],
    "tech": [post500, post700, ...],
}

By storying a key/value pair of tags to posts that have said tag, it's relatively simple to do that in any programming language. But... We're in Racket, and we're using immutable hashes. What now? Do we start mutating?

No! First we need a function that can build for us a hash mapping of tags to posts, and we can do it recursively with a function. From our list of posts (metadata only), we can inspect the tagging information and apply some recursive code to build up a hash piece-by-piece.

First, we kind of need a new function to interact with hashes. Something that says "if the key in the hash maps to a list, add to that list, and if it isn't a list, then make it a list, else add the new key and the value as a list". Using hash-update we can do this with the following function.

(define (append H K V)
  (define (updater old-data)
    (if (list? old-data)
        (cons V old-data)
        (list V old-data)))
  (if (hash-has-key? H K)
      (hash-update H K updater)
      (hash-set H K (list V))))

(Note: this overrides the default list append function, but I import this hash library using a prefix ie. (prefix-in Hash: ...))

With this in play, we need an algorithm that matches this functionality:

Posts = GetPosts()
TagMap = MakeMap()
for Post in Posts:
  for Tag in Post.Tags:
    TagMap[Tag] += Post

To keep this a purely functional codebase, we cannot just mutate references casually. Every modification to a structure has to be tied to some function call. Instead of thinking of this in terms of for loops, they should instead be thought of as folds over a list of data with an applicative function. The type signature of foldl (using Haskell) is:

foldl :: Foldable t => (b -> a -> b) -> b -> t a -> b

For a foldl to work (fold-left-associative), we need something that can be "folded" over, and a function to take some value and an accumulative new value we're building. I've talked about foldl so I don't want to dwell on it for too long, but as a refresher, let's visualize a sum function to add all numbers in a given list.

(define (sum lst)
  (define (inner lst acc)
    (if (empty? lst)
        acc
        (inner (cdr lst) (+ acc (car lst)))))
  (inner lst 0))

This recursive pattern of traversing a list is the core of a foldable item, and instead of writing it in a recursive function, we can simply use a call to foldl instead. We can even define product and factorial while we're at it.

(define (sum lst)
  (foldl + 0 lst))

(define (product lst)
  (foldl * 1 lst))

(define (factorial x)
  (foldl * 1 (range 1 (add1 x))))

This foldl method is pretty powerful and can drive a ton of functions and clean up code in many different places. No longer might you suffer from clumsy-looking for loops when you can break a function down into a foldl expression instead. It's an interesting thing to try to reduce your code down into, and in some cases can simplify a lot, however that's not to say it's suitable to replace every for loop in Racket.

In this instance, we need two loops, or two foldl's nested in each other, as weird as that might look. What we end up with is a function that takes a list of metadata from my posts, and we get a hash that maps tags to respective lists of posts with matching tags.

(define (Create-Tag-Map listof-posts)
  (foldl
    (λ (post H1)
     (let ([tags (Page:Data-tags post)])
       (if (empty? tags)
           (Hash:append H1 "untagged" post)
           (foldl (λ (tag H2) (Hash:append H2 tag post))
                  H1 (Page:Data-tags post)))))
   (make-immutable-hash '()) listof-posts))

Does this look bizarre? Yes, yes it does. Can this be replaced with for/hash? No, not really. The take-away here is that we want to gradually add one post at a time to each tag it belongs to, so we iterate over each post's tags, add the post to the matching tag group, then move onto the next tag, until the post is fully exhausted. for/hash collects a linear mapping of values into a hash, whereas this itself can fold over a hash and add data to it. Note how I mark the hash inputs as H1 and H2 to separate the hash reference getting passed around in the folds.

What we end up with is a hash that maps tags to posts, and from here, we can move onto the next part: generating new pages.

Generating Tag Folders

Thankfully this didn't end up being too much work, but it still ended up being around roughly ~50 lines of code added.

Once Zeus is finished building out all the pages from a section, we're at a crossroads where we need to ask ourselves if we want to generate tags to help navigate the website. This is something configurable in Zola, as not everyone may want tags enabled for everything. A setting should be wedged into the section.json of each section of our website to dictate whether we want tags for that section. Simply by adding:

{
  ...
  "tags_enabled": "true",
  ...
}

To our section.json, that should be a good enough user customization and it's now customizable at the section-level instead of a global config-level (like I did on my first go of doing this feature).

However, we need something else: we need a template for these tags! Tags are effectively a new data type being passed into the template rendering system. What does a tag look like on the first sketch? It's a two-value key/map that we'll build out from the Zeus layer.

{
  "name": "tag_name",
  "path": "/<section>/<tag_name>"
}

There might be more later as I build out features like tag clouding but for right now this is fine. We need a template that can iterate over a list of tags and generate some <a> links for users to click on. Let's make a new template called tag_list.html.

{% extends base.html %}

{% block content %}
<h2>All Tags</h2>

<p>{% for tag in tags %}
<a href="{{ tag.path }}">{{ tag.name }}</a>&nbsp;
{% endfor %}</p>
{% endblock %}

Building this feature out, I took some time to remind myself how proud I was for even getting to this point with Zeus and making a working templating system. I didn't really have to stop and think about the intrinsics of it or whether this works; I wrote this template, plugged in the data, and it worked. That was a pretty nice feeling.

We're going to need to plug this template in somewhere at the config-level, so let's add a new key to section.json.

{
  ...
  "tags_enabled": "true",
  "tag_template": "tag_list.html",
  ...
}

Now we can work on generating tags, but wait, we need a template to display posts filtered by tags. How do we do that? Well, thankfully, here we can re-use our original blog.html template that I use to list all my posts. The difference now is that we have tag-relevant information. What if we leveraged the presence of tag data in what we pass to our blog.html, and we can re-use the original template? That's exactly what I'm gonna do.

{% extends base.html %}

{% block content %}
<h2>List of blog pages{% if tag %} tagged {{ tag }}{% endif %}</h2>
<ul>
{% for page in pages %}
<li><a href="/{{ page.url }}">{{ page.title }}</a></li>
{% endfor %}
</ul>
{% endblock %}

If we have a tag key in our context, we add the text "tagged {{ tag }}" into our header to make it look more correct. We don't even need to modify the for page in pages part, because we built a tag map that will pass the relevant pages easily.

Now for the finishing touches, in the following order, we will:

Check if tags is enabled
Generate a /<section>/tags folder and index.html
Pass all tags to render to that index.html
For each tag, build a /<section>/<tag> folder and index.html
Pass posts by tag from the tag map into step 4's index.html

The code ends up looking like this.

(define tags-enabled
  (trim-down (Hash:get-else section-json 'tags_enabled "")))

(when (string=? tags-enabled "true")
  (define tag-map (Create-Tag-Map pages))
  (define tag-tmpl (Hash:get section-json 'tag_template))
  (define tag-root-folder (build-path Output-Directory section "tags"))
  (define tag-root-index (build-path tag-root-folder "index.html"))
  (make-directory* tag-root-folder)

  (define env/alltags
    (Hash:update Env 'tags
                 (map (tag-factory section)
                      (hash-keys tag-map))))
  (Tmpl:Publish Tstore tag-tmpl env/alltags tag-root-index)
  
  (for ([tag (hash-keys tag-map)])
    (define tag-folder (build-path Output-Directory section tag))
    (define tag-file (build-path tag-folder "index.html"))
    (make-directory* tag-folder)

    (define env/tagged-pages
      (Hash:update (Hash:update Env 'tag tag)
       'pages (map Page:Data-meta (Hash:get tag-map tag))))
    
    (Tmpl:Publish Tstore section-template env/tagged-pages tag-file)))

With everything now in place, tags are discovered, posts are grouped, folders are created, and now we have a working tags feature on the website!

Wrapping Up

This was a miniature feature I planned on releasing, and it wasn't overly complicated, and I think it makes it so much nicer to traverse posts. Best part of it all, this is something I put together by hand, and I really enjoyed it.

The overall goal of Zeus has been, and hopefully will continue, to be a simple codebase I can maintain to produce my personal website, and future websites I would like to develop. Writing this tag feature out, I'm starting to see some spots that could do with some improvement. Doing a Hash:update within a Hash:update was not as pretty as I might like it to be, so I think I need to do some tweaking there to make the code a little less crazy. 😅

This new feature will mark me tagging the code as v0.2. Tests are passing currently with 18 unit tests, performance is still in-line with my expectation using a language like Racket, but overall, I'm proud of what I've accomplished so far.

$ make test
18 tests passed
$ time make build
make build  0.76s user 0.04s system 87% cpu 0.918 total
$ cat zeus/*.rkt | wc -l
1373

In the near future, I'd like to start working on automating the deployment process, so I hope to write about that sometime soon.

Thanks for reading and see you around!

Pixel 7 Thoughts

2023-05-10T20:00:00Z

A few weeks ago, I had to make the decision to upgrade from my Pixel 3a XL to a new device; I chose the Pixel 7. In this I will discuss my rationale and how I feel about it so far after a few weeks of use.

The Problems

I have written in the past about wanting to move away from my Pixel 3a XL. I have had my desires to upgrade and get some new performance squeezed out from a hand-held device. Things were running at okay speeds, but it could have been better. The Pixel 3a XL came out only a few years ago, but it came with what I would call a ham-strung processor that is miles worse than it's non-"a" version.

Back in May of 2022, Google stopped giving out security updates to the Qualcomm Snapdragon 600 CPU series for the Pixel 3a line of phones, meaning firmware updates and other security fixes would no longer be provided OTA by Google anymore. The device was effectively dead; no security updates means the device would be subject to random attacks via either application installations or possibly through web-browsing attacks.

When this happened, I installed LineageOS which worked great, and gave the battery a massive battery boost, sometimes lasting well over 24 hours on idle/casual usage. The downside was by rooting the device, certain classes of applications would no longer work due to the presence of rooting, and the strong lack of Google Play Services. This was mostly anything strongly-tightened, like banking apps (which is dumb, because I can still use my mobile browser to use the bank, but the app they provide just "doesn't").

I had no issue originally; I switched myself over to a rather comfy life of using FOSS-only applications while downloading less-FOSS-y stuff via Aurora Store. I was able to keep up a relatively stable digital lifestyle without using Google Play Services entirely. I got along just fine. Notable heroes of my Android success:

Firefox Nightly - web browsing of course
K-9 Mail - e-mail POP3/IMAP client soon to be Thunderbird
OpenCamera - alternative OSS camera
OsmAnd~ - navigation and GPS with OpenStreetMaps
Fedilab - Mastodon and Pixelfed
Element - secure decentralized chatting
Feeder - RSS feeds to remove Reddit
Vanilla Music - a very good music player
KeePassDX - secure password vault
NewPipe - a sane YouTube client

I was able to chat, read my emails, interact socially, read RSS feeds, and do everything I could have done on an official Google OS image. I drove several road trips using OsmAnd~ even. It is very possible to be happy with LineageOS, and I would go back if I could, the battery life was just too much to ignore.

Due to new restrictions, my LineageOS was not cutting it. I needed some specific applications for granting 2FA, as KeePassXC was not something supported at the time. I had to use a very specific 2FA application, and I had no solution in sight for it. After much discussion, I decided it might be best for me to part ways with the 3a XL.

So to sum it all up, there isn't one reason, but a few:

lack of security moving forward; I would still use it if I could
performance lacking, even if it's only a 4-year old CPU chipset
camera quality is crappy (not blaming you entirely OpenCamera)
small storage space available (I only have 64GB!)
can't use a certain class of applications for important life reasons

Picking Pixel 7

I did some modest phone browsing for a few weeks and decided on the Pixel 7. There were some contenders at the time, but I felt Pixel 7 was where the best bang-for-my-buck sat. And looking at the availability of iPhone-level contenders in Android, it was a bit... Sad, I guess.

Over the years, phone brands came and went, and frankly I don't even see Android phones that aren't Google or Samsung anymore in-person. I know a few people who have a OnePlus device, but that's about it. Shopping around I see only Google and Samsung as my main choices for easy access.

LG died years ago on accounts that the LG branded phones failed
HTC was purchased by Google years ago
Nokia exists, but seems to be mostly European and elsewhere
Motorola exists too, but not as heavy-hitting/long-term I feel

The warzone between Samsung and Google is real, but holy cow, have you seen Samsung prices recently? I cannot fathom spending over $1,000 USD on a Galaxy S23. What the heck even is that price point. And about 60GB of that 256GB+ storage will be filled with Samsung-only proprietary garbage? Yuck.

The reality is that, unfortunately, Android is Google's child, and having past experience with a Pixel 3a converting it into a Lineage phone, I felt better just going straight to the source. The price point was more modest in the end.

I had a time-sensitive mission to get this as soon as possible, and unfortunately my choices for an in-person pickup were very low. I ended up using Best Buy as my vendor for this since they had it in stock, versus Target who didn't carry it anywhere. Beats going into the Manhattan Google Store at least.

From Lineage to Google Android

The transition back to a world full of Google Play Services and an almost always-online world felt strange, at first. I was reminded about how I was disgusted about all the weird ways Google tries to farm you for data and information. Then grossed out by all the weird things they try to subscribe you for. Prompt after prompt of saying "no" and unchecking all things I didn't want, I finally made it back to the Home Screen, where I was reminded about the worst part of Google Android: Google Assistant.

The Assistant lurks its way into every snippet of your screen usage, and even on the home screen, if you don't turn it off and swipe left a bit too much, you'll find your "personalized news feed". This disgusts me a bit too much; an automated system that absorbs as much information as possible from you by silently observing everything you type and every interaction you have with anyone around you. I've had this thought for years now, but it's just disgusting to see it literally on your own home screen.

It can be turned off, but I decided that I'd rather not deal with Google on my home screen at all, and moved to OpenLauncher. I used this a bunch on Lineage, and while it's a little old and out of date, it still works. For weather you can replace whatever Google puts on your home screen with wX, which grabs weather station data and is pretty customizable.

So far, the Pixel 7 is not all that much different from my Pixel 3a. The big differences are in performance and camera categories; which I don't need to spend much dwelling on. It's faster, it's better, it's more modern, yada yada yada. Melvor Idle still feels kind of chuggish, but I don't think any phone in the world could make it smoother with all the UI elements it has. I could be wrong though, maybe it's faster on my iPad.

The one upgrade is in the refresh rate department: it can go up to 90Hz on a Pixel 7 (and 120Hz on a Pixel 7 Pro), but this, as I observe, can wreck the battery the fastest. Re-drawing screen elements at higher rates can make your phone burn out in under a day's time. Maybe variable refresh rate will make it so it won't re-draw everything all the time, but I frankly don't really trust it to be battery efficient 100% of the time. It's impossible.

You can turn off the higher refresh rate in the Display settings, which I did. I would much rather enjoy my battery for the full three+ years I'll own the device, thanks. I currently don't see how this could be a useful feature all things considering most content on the web can never go above 60Hz anyways (videos, games, etc).

Here's a list of things I like so far about the Pixel 7 overall.

expanded memory (8GB RAM) helps keeps apps open/in-memory for longer, less need to hard-fetch web pages or re-open games
better camera quality with a dual sensor
full screen bezel-less body with only a front-facing camera punch-out, not super annoying
not the lightest phone, but it's weight is comfortable enough
quick and reactive in almost all basic aspects (WiFi, Bluetooth, web browsing, etc)

The things I probably won't like about the Pixel 7 are going to be...:

security expiring in about two years; I got the phone a year too late, and couldn't wait for the (rumored) Pixel 8 supposedly coming this October. Oh well.
almost every and all Google service heavily integrated into the device
Google Photos is really just a bad photo gallery app
Google Files is also a bad file managing app
not the biggest fan of the under-the-screen fingerprint scanner
auto-screen dimming seems to be all over the place and inconsistent
not a feature that's ever going to be in Android, but the ability to see network up/down and a GPS-in-use indicator was a major plus of Lineage; nothing like that exists in core AOSP
the definition of "privacy" in Google is much different from what I could consider "private", and it's tiresome
anything that uses Chrome
a dearly-departed headphone jack; an important feature for anyone like me with wired headsets who are forced to use the much worse USB-C-to-3.5mm jack adapter (no, I don't want to buy more short-battery-life disposable Bluetooth equipment, thank you)

Conclusions

While I understand some of the choices Google has made, and strongly disagree with others, I feel it's a hard balance to strike in a world where you're viewed as the most used operating system, yet still viewed as an insignificant product in the originating country. The struggle between Android and iPhone has been going on for over a decade now, and I grow more tired of it with each and every passing day. PinePhone's time to shine couldn't come any sooner.

I might continue to be an Android supporter for as long as it is beneficial to me. I am perfectly fine with using a phone that doesn't include Google services (LineageOS), and would be completely okay transforming my life if it meant supporting a truly open-source phone (PinePhone). I am pretty much not okay with my data being harvested for free, nor do I enjoy being served ads everywhere I go. This is where Apple does it right, but they have a bit of a different business model.

It's getting harder and harder to separate the good parts of Android from the bad, and turning off a number of services is a game of whack-a-mole to see what stops working first. Heavy reliance on Google Play Services can completely turn off notifications, and notifications are convenient for some, but might turn you into someone who's terminally online. In my time away from mainstream Android I learned a couple of things about how Android works, and I did find LineageOS to be the best way of servicing my mental health. Google's crappy "well-being" services are not the way; literally just turn off notifications and stop doom-scrolling, you'll see a radical change in days.

Overall, I'm pleased with the phone so far. I have most Googley things disabled, but still rely on things like Android Auto for the sake of my car. I got too accustomed to free software during my LineageOS 3a XL time, so I simply reinstalled F-Droid and went crazy. I like my RSS reader, I enjoy using Fedilab for the Fediverse stuff, and I can still do other basic things. I try to avoid complex things and prefer keeping things simple now; the less push notifications the better.

The goal is still to use this phone as long as I can to avoid having to buy new hardware on repeated cycles. I'm shooting for four years (2027) at least. Moore's Law seems to be dropping off, so I'm hopeful the Pixel 7 will last a good couple of years. By 2025 I can probably drop Lineage onto it just fine and resume my FOSS adventures in Android from there.

Lineage OS

2022-05-06T20:00:00Z

The time has finally arrived, where I install Lineage OS on my Pixel 3a XL. In about four weeks, the Pixel 3a XL will be receiving it's official end of life date. There will be no more software updates, and Android 12 is most likely going to be the last version of Android this device receives.

It is a sad day to think about, considering how good the Pixel 3a XL was of a phone. It had a low price tag, a cheap but decent processor, a good camera (possibly due to software), and overall very good form factor, being large but just the right size.

Sadly, as all good things come to an end, so will the software support for it from Google. It is time to go with a community supported version of Android - namely that of Lineage OS.

The Installation

I have to admit, I have never rooted an Android phone before. The methods always eluded me as to how they worked. I don't have a great understanding of the boot process of an Android phone, but I think I have a vague enough idea of how it works now.

The idea is that, on the phone itself, there are many components in place to act as secure layers, so that no one can simply fiddle with the phone and take/place new data and subject many networks to bad actors. An authority approved by Google has to distribute operating system images and sign them with keys such that the hardware can securely boot the device's operating system.

These measures are more restrictive, such that if you were to buy a phone with a phone plan, you can't unlock it and install an Android version of your own and then never pay for your contract and disappearing off the planet with it. As such, the only thing you can really do with your operating system within the bootloader, is boot it, or use something called a recovery tool.

The recovery tool is a built-in Android utility that allows users to launch a recovery image of Android, for the weird, catastrophic failures that if an update went bad, the Android system itself can roll back updates using the recovery mechanism (if I understand it properly). Once a system has been updated to a fresh number bump of Android, it however, cannot go backwards. You cannot do a recovery boot of Android 11 if your system is currently Android 12.

The recovery tool is where Lineage OS will go it's own way and allow you to side-load and install a new system from this section. You simply tell your Android phone to boot into it's bootloader mode, then activate the recovery tool using the given Lineage OS recovery image (.img file).

# tell adb to boot a recovery image file
$ fastboot flash boot ~/recovery-nightly-bonito.img

From here we move onto the next part. You go into recover mode from your bootloader, to activate this newly installed recovery tool, which allows you to install a new system. Make sure you have the proper .zip file downloaded for your device.

# in recovery, sideload the new system image
$ adb sideload ~/lineage-nightly-bonito.zip

Before I was doing this installation, my bootloader was not unlocked. I was surprised to learn that bootload unlocking actually wipes the device! What a crazy concept. I had moved my files over and everything previously, but in an instant, Android basically factory reset my system all because I asked to unlock the bootloader.

But after a long waiting period of about 20 minutes, Lineage OS was installed, and I was presented with a new bare-bones system, something I hadn't seen in so long on my phone it was... almost breath-taking how empty it was.

Granted, a plain Lineage OS installation does not give you full-blown root access, which seems like it is up to Magisk to be able to solve that. Once I figure out how to get that running then I think I'll be in full control of my phone.

Other Thoughts

I had written a whole lot about my experience so far, but none of it was very interesting in the slightest. But rather, I feel I would prefer to talk about my feelings on technology moving forward when it comes to things like this.

Android has been my phone since 2010. I wanted an Android phone pretty badly when they were first called "Droids" and were only on like... Verizon phones or whatever. My first Android was a HTC Inspire. It's specs are terrible compared to today, but it came with Android 2.1, which I was absolutely in love with and thought how cool it was to have an Android phone.

My second phone was a Samsung S3 Mini. It was tiny, and upgrading to this phone years after the HTC Inspire only costed me about a penny through my carrier at the time. I liked the idea of having a micro phone so it worked well for me. I customized it with themes, a custom home screen, app icons, the works. Eventually it became too slow as Android apps grew into some crazy stuff.

Eventually I moved on and got a Motorola X, which used a leather back and was quite comfortable to hold. It eventually got Android 5 which nearly destroyed my battery because of how bad Lollipop was, but I liked this phone a lot.

Then the phone that I had my biggest hopes for was the LG V20. This phone was touted as one of the first few to come with Android 7 Nougat on release, so when it was available, I did get one. Android 7 was alright, Android 8 was a bit better, but the phone was ... lacking in many categories of performance. Over time it's CPU performance degraded and replacing the thermal paste as advised by the internet, it didn't really give me much of a boost.

For a brief period of time for about six months, I was an iPhone user. I bought an iPhone 8 because I wanted something reliable that wouldn't degrade and had future-proofing with it. Apple makes considerable upgrades to their product line and keeps phones as old as iPhone 6 still chugging along.

(For reference, iPhone 6 is now 8 years old, and there is no Google phone product in the world with that long of a life span)

Eventually I switched back. I was doing okay without a headphone jack and the ability to load my own MP3s without iTunes on a computer, but Apple had made it clear that they were looking to do away with the features I had initially come to iPhone for - I liked Touch ID a lot because I enjoy fingerprint-based authentication.

I do not enjoy Face ID, as I do not want millions of infrared beams being shot at my face. Nor do I want that facial data stored in some cloud or on a device without the ability to really say "hey wait a minute, what are you doing with this data exactly and how do I know you're not sending it off somewhere?"

Handing over sensitive data to any platform is just asking for trouble and EULA updated terms. It doesn't matter if Apple said no to the FBI once, I still wouldn't trust anyone with my facial data whatsoever. Network transmissions of data packets are nigh invisible to users on almost all computer platforms. how do I know if one night, Apple decided to send my face to some 3rd party? I would have zero clue about it until long after the fact. I have no way of knowing where my data will wind up.

Upon seeing the news of Face ID and recognizing this may be how Apple wants to design their products forever, I went back to Android, and got a Pixel 3a XL. It's been reliable and great, and comes with a headphone jack (even if Google has mixed feelings on that).

I should also mention that I do own an Apple iPad Pro 2017 edition. I admire the iPad for it's Apple Pencil greatly as it really adds a great way of drawing on digital devices. I was a Wacom user for so long, but the iPad is great. Google products, however, do not compare.

Next Steps

Lineage OS has been great to me so far, and while getting back and running to where I was previously on a validated, official Google Android image, I'm okay with the jankiness of Lineage, because it's nice having a community of users who hate seeing perfectly good tech get thrown away because of lack of updates from the manufacturer.

I am looking forward to the fun I'll be having with my new (but still the same) toy. Next steps I am looking to take would be to better organize all my electronics and find a way to truly synchronize everything, but that's a pretty long and unspecific subject.

For now, I am simply going to watch Lineage grow and see how long my phone lasts. I am not against buying a new phone if it's required. I am against buying a new phone if I simply don't need to, when my old phone can suffice easily.

Nextcloud

2022-04-04T20:00:00Z

Today I would like to talk about my recent experience creating my own Nextcloud server at home. I have been using it for about a week now to some success that I am happy with, and mostly wanted to share my thoughts on the experience.

Current Cloud Data Market

On my list of current pains in the modern tech world is the idea that we must pay for data ourselves in order to host our insurmountable amount of information in our lives. Gone are the days of VHS collections and photo books, and now is the time of Plex servers and gigantic digital photo albums - the digital era is here.

But the digital era has a cost - tech corporations.

For every need a consumer has, a business exists to satisfy it. If the business doesn't exist, eventually someone will make it. It has been that way for a long time now, and frankly most people probably don't care to remember what the good old days were like. Have a large issue, like data storage and provisioning? Who cares? Pay Google to host your files and move on with your life.

Paying someone else to do something has existed for as long as human history has, but it's time I finally put some effort in escaping a long-winded cycle of destruction. I have a lot of files, but not enough consistency across systems I use. How do I go about changing this? Setting up my own system to help me carry my files across multiple platforms.

The question is, is it worth all the trouble?

Disk Space as a Service

Disk-space renting is such a horrific market I do not want to really consider myself a customer of it anymore. As someone who has paid for storage and used free clients, it's such a dumb market for individual consumers to get trapped in. If you're a big business who needs to move fast and break things, by all means go with a cloud storage provider who can help you elevate your business.

But honestly, if you're a regular person, you really should not ever use a cloud storage subscription. It's a rip-off.

Cloud storage providers are the kings of the castle, and we are the simple peasants working in the fields, watering the bytes. The market is so unfair to users it's not even funny. The rules of data hosting have changed so much in the last five years that it's kind of ridiculous what we're willing to put up with. If we as a working-class society hate landlords for owning all the houses, then we should also hate big tech companies for owning all the hardware and buying everything up.

Things that irk me about cloud providers:

Rules change so frequently that users are told to simply "deal with it". What was once a "free photo storage forever" policy suddenly turned into "okay now your photos are limited, pay up".

Should the cloud provider find that users are breaking terms and service, they will have no choice but to start deleting files off the system. This is pretty typical for DMCA requests, but if your files "happen" to appear similar to other files in the network, what is stopping Google from accidentally deleting your files as well? Maybe you were completely innocent, but Google mistook you as an offender and suspended your account permanently. There goes your photo collection!
Cloud providers expanded so fast and grew so much that they bought all the hardware off the market to stop average joes from starting their own home DIY projects or even their own businesses. The big FAANG companies all have direct contracts with the manufacturing companies, and they get the best deals, while we get the leftovers at more expensive rates.

The computer industry cannot endure this for much longer. Natural resources are drying up, and hardware is getting more expensive. Soon once we run out of minerals, companies will be knocking at your doors every day trying to buy what little silicon may be left in your household as an attempt to gather more computing resources. It's all for the good of corporate growth.

I am for the most part done with these rent-seeking data hosting companies. While I agree there are some things they provide that can be useful, and to which they would have the experience to handle all the downsides of self-hosting, the only way for us to take back privacy is to stop using these big-name providers.

Nextcloud

If you wanted to pay for 2 terabytes of storage, which is about 2,048 gigabytes worth of space, you would pay Dropbox or iCloud about $9.99 a month, or $120 USD a year. Over ten years that's only $1,200 dollars, not bad right?

Or, you could buy something cheaper.

Portable hard drives have been my best friends for as long as I can remember. I use them for games, photos, videos, operating system partitions so I could boot Linux on computers that don't have Linux most of the time, and you could probably do so much more.

While a portable hard drive isn't as convenient, it's a good stepping-off point into taking control of your digital life. The instant convenience of having your photos accessible to you all the time is great, but that's more of a magic trick I don't feel is super important most of the time, and you pay a lot of money a year for a magic trick you might only use every so often.

You could call a plain old portable hard drive like this a "dumb" drive because there is no connectivity to it to the modern internet world. You can't plug a hard drive into your local network and suddenly make it accessible and private (unless you want to re-live the MyBook Live hack).

But fortunately, this is where Nextcloud kicks in - Nextcloud can enable you to power-up your digital life and become a modern-day replacement for any cloud service you keep on your phone. I have been using Nextcloud for work and non-work purposes, and it's been great!

If you are confused as to what the image is above, that is in fact a Samsung laptop that belonged to a family member purchased several years ago. It's not particularly good anymore, but I collect a lot of old hardware as a tech enthusiast. It's sad to see good things go, and this laptop is perfectly good for a project like this.

Oh, yes, I should mention - this laptop is hosting my current Nextcloud setup, and I would say it's usable! The operating system it is running is a server version of Fedora, and because it's not running intensive desktop virtualizations, it can focus on basic things like serving static files or being a basic FTP computer. You can plug as many hard drives as you want into it and expand your storage as you see fit.

Nextcloud comes with (out of the box) the ability to upload files, add users so you can add your friends, and some tools to collaborate with any of your friends on it by creating shared folders. The Nextcloud Android app can synchronize folders automatically if they detect new files. The one thing I am trying to find out of Nextcloud can do is to be able to download files from your Nextcloud server if there are any changes. So far I haven't had luck in this department, but we're getting there.

Now you might ask, "how do you get your web server exposed to the internet?", which is a valid question, because hosting your computers at home and trying to expose them to the public net is no small job. In some instances you often times need to request (and pay more) for a static IP address to your home, which is a sucky process.

I wouldn't be writing about this if I didn't find some cool and slightly easy alternative to it - and it's called Cloudflare Zero-Trust Tunnel.

The goal is to instead of worry about exposing your computer to the world wide web, you expose it to the Cloudflare Argo network instead using a client called cloudflared. The cloudflared daemon will communicate with the Argo network and carry traffic to and from your devices based on the incoming requests. If you configure your DNS through Cloudflare, you can have traffic sent over configured CNAME records and have them reach your target computer easily.

I have been doing this with a few different services and it's worked out great. Cloudflare creates the security certificates and deals with the traffic, and my friends and I enjoy secured, private traffic without needing to expose my home IP address to the web. I consider that a win for the most part.

I do support Cloudflare products, and I enjoy seeing what they make. It's a great service and I am happy to be their customer. Other people's opinions are valid, but it works great for a small-time person like myself trying to create a better web experience.

Next Time

I have set up my home 'server' with Nextcloud, but the next stop is the biggest one - leaving Google Android behind, and moving into the wild world of Google-less Android. My Pixel 3a XL is losing it's software support next month, and it's high time I either try out other Androids, or maybe I just give up and get a new phone? 🤷

Slowly but surely I am trying my best to cut Google out of my life more and more. Thanks for reading!

Beating C++ With Zig

2022-01-17T20:00:00Z

I have started doing more tinkering with Zig, and thought of a case where I might want to try experimenting with Zig; rewriting a Mandelbrot Set program. I have a lot of different projects all related to Mandelbrot computation, so it's something I know nearly off the top of my head without having to look up the basic equation (it's f(z, c) = z*z + c). I understand a lot of it, and I love making fractals from it.

That being said, years ago I had wrote a project which I wanted to make faster by making a C++ version, which I did, named mandelpp. It's a very bare-bones implementation which plain and simple renders a fractal, given a series of information surrounding it. It works, it uses some bare-minimum C++ features, and I had planned on integrating it into one of my larger projects, but never did.

Now I open it up today, and lament at trying to modernize it. GNU Make is simple, but doesn't cover enough bases. GNU Autoconf and Automake is crazy confusing and I couldn't get it to work plainly. CMake is something I thought about, but I'm not crazy familiar on it. Meson and Ninja are still things I thought about but never came up with a solution in my spare time.

Instead, I thought this might be a cool exercise to do in Zig. Where C++ is limited to the double type and the only way to go further is rolling your own types, including new libraries, or maybe using GNU Multi-Precision, Zig has a floating-point type of 128 bits, which is twice as big as the C++ double, so this was a bit appealing to me. If Zig could create code that handled 128-bit floating points, that would be pretty cool.

A Complex Struct

Zig has no notion of classes like C++, but it does have the idea of compile-time macros. The macros in Zig are very powerful, allowing you to do many cool things like create new type definitons at compile-time with a known type. The difference between this:

const Point = struct{
  x: f64,
  y: f64,
};

And this:

pub fn Point(comptime T: type) type {
  return struct {
    x: T,
    y: T,
  };
}

Are huge. One is a fixed-type struct definition where it only holds the f64 type, while the other is what is effectively a compile-time macro-like function that creates a new struct given a type T, and when it's invoked in code will produce the new code definition substituting the given type. You can produce many different size structs with this to meet your needs.

So for us to define a Complex number, which has two components real and imaginary to express the real and imaginary components, we can write a compile-time function to allow for type flexibility.

fn Complex(comptime T: type) type {
  return struct {
    real: T,
    imag: T,
  }
}

Struct Methods

Structs can have methods attached to them, similar to Rust or Golang. We can do this inside the compile-time function as well. We cannot do any type of operator-overload like C++, but we can create basic methods that mimic the functionality of such. The reasons why operator overloading does not exist is that resource management gets tricky when operators get overloaded and create unneeded complexities, which could lead to memory errors all over the place.

So to begin writing some math functions we attach some function signatures. I won't include the low-level math behind imaginary numbers, but we can see how this works. Also to be able to write macro-friendly code, we need to be able to refer to our struct, which we can't do in plain code. We need a compiler macro to generate the name for us, so we will use @This() to create the binding.

fn Complex(comptime T: type) type {
  return struct {
    real: T,
    imag: T,
    const Self = @This();

    fn init(r: T, i: T) Self {
      return Self{ .real = r, .imag = i };
    }
    fn add(self: Self, other: Self) Self { ... }
    fn sub(self: Self, other: Self) Self { ... }
    fn mul(self: Self, other: Self) Self { ... }
    fn div(self: Self, other: Self) Self { ... }
    fn length(self: Self) T { ... }
  };
}

So we create each binary math operation using some (T, T) -> T operations. However, this might not always be ideal. These functions return new structs, which might not be very efficient based on your application. Then there's the notion of pass-by-value or pass-by-pointer. For simple programs like the one we are writing, pass-by-value is completely okay, because our underlying struct only contains two variables, so it's very trivial data.

However, with larger structures, and sometimes with things like file locks or allocated memory, you are best off using pass-by-pointer because you do not wish to copy whole values over in function calls. But should we still copy a struct over or return a new struct entirely? That would involve copying information over, which we do not want to do. Instead it might be more reasonable to mutate in-place, so I'll make functions that take pointers to values instead.

fn Complex(comptime T: type) type {
  return struct {
    real: T,
    imag: T,
    const Self = @This();

    fn init(r: T, i: T) Self {
      return Self{ .real = r, .imag = i };
    }
    fn add(self: *Self, other: *Self) void { ... }
    fn sub(self: *Self, other: *Self) void { ... }
    fn mul(self: *Self, other: *Self) void { ... }
    fn div(self: *Self, other: *Self) void { ... }
    fn length(self: *Self) T { ... }
  };
}

Now instead of operating on raw values, we operate only with pointers, and instead of returning values, we simply return void, meaning we want to mutate our data in-place. This will avoid having to do so many raw copies of structs between addresses.

The Mandelbrot Recursion

How a Mandelbrot Set is calculated makes it beautiful when visualized in a 2D format. The idea behind a Mandelbrot set is no different than that of visualizing any other function of two variables; you plot an X and a Y axis, and you go through every point and calculate the function f with f(x, y).

For complex numbers we don't often say X or Y axes, but we talk about real and imaginary components. So instead of X we have the real axis, and instead of Y it's the imaginary axis, and instead of f, it's Z, which is often used when making complex functions.

We must now explore what the Mandelbrot Set equation is.

Zn+1 = Zn^2 + C   sqrt(|Zn+1|) < 2

There are two variables here in this equation: Z and C. Also note that when we notate n and n+1, we can see that this is a recursive relationship, the value of this relies on the value before it.

Z and C are two different complex numbers here, so there are a total of four values in play, each being their respective real and imaginary values. Remember when I said we are doing a two-dimensional plot of a two-variable function? Well, why are there four here?

One of these values here, will actually always start at zero. What we're changing when we do our plot is changing the values of C. So when Z is set to zero, it gets the initial C value added to it, and it recurses until the new Zn+1 value's magnitude exceeds a value of 4. Why 4? The initial equation compares the square root of the magnitude of Z against the number 2, but if we multiply both sides by the inverse of the square root function, aka the the x**2 function, we get |Zn+1| < 4, removing a nasty square root from the equation.

So while we have two different variables here, only the C is ever changing. So let's write our function that will recurse for us up until a maximum.

pub fn get_numcycles(Z: *Cbig, C: *Cbig) u8 {
    var i: u8 = 0;
    while (i < 255 and (Z.length() < 4.0)) : (i += 1) {
        Z.muls(Z);
        Z.adds(C);
    }
    return i;
}

Meet the get_numcycles function, where we count how long it takes for our initial C value to escape the |Z|<4 boundary, or diverge, however you want to call it. When drawing fractals, we care about how many steps it took for it it to escape. This number is hard-capped at 255 right now, or enough to fit into a u8 eight-bit value. Which conveniently also describes a RGB color format, interesting right?

So the inner while loop simply iterates on two conditions, whether i is under our iteration limit, and whether or not our Z value exceeds it's boundary. The other part of that while line is the simple i increment, which we can put into it's own little block as part of the while syntax. Then we simply do Z.mul(Z) and Z.add(C), until we are done. Pretty easy to remember, the hard part is crafting the C variable itself.

Rendering the Plot

Next step, we need to actually plot. For every pixel in our target image, we need to convert it to a valid C variable for us to render it, then we draw that pixel with a color. For simplicity sake, we will not be using any kind of fancy image saving formats, and will use the Netpbm format for images, which involves raw values and no compression. It's much easier to use something like ImageMagick to convert images to your desired format afterwards (which I have done for this blog post).

Let's define a basic loop implementation here.

var x : usize = 0;
var y : usize = 0;
const width : usize = 640;
const height: usize = 480;

var Z = Complex.init(0, 0);
var C = Complex.init(0, 0);

var i : u8 = 0;

while (y < height) : (y += 1) {
  while (x < width) : (x += 1) {
    Z.real = 0;
    Z.imag = 0;
    C.real = @intToFloat(f64, x);
    C.imag = @intToFloat(f64, y);
    i = get_numcycles(&Z, &C);
    // draw?
  }
  x = 0;
}

This is a naive implementation with no real draw function yet, and it will render across the entire spectrum between (0, 0) and (640, 480).

But... The Mandelbrot set is not really... visible in this range. It's really only good for about (-2,-2) to (2, 2). So we could render a 4x4 pixel image, but that isn't exactly useful...

We need to stretch the domain in which the Mandelbrot set is actually visible in by scaling up our drawing pixel coordinates a bit. A 640x480 is too large, and a 4x4 image is too small, so we must scale large coordinates down to smaller ones. There's two approaches we can take here, but I am going to stick with one I believe is better for performance, which I'll call complex incrementing.

In complex incrementing, the idea is we calculate the exact offset we need to apply to C each step of the way before we begin the main loop. That way, instead of setting C.real = x, we can instead do C.add(&re_increment) instead, where re_increment is the tiny bump needed to move C just a tiny bit. We do the same using im_increment, which does the imaginary component.

(The alternative would be linear interpolating our (x, y) co-ordinate values each loop cycle and setting C to be "lerped" values. This is a more demanding approach because of the added stress of needing to do two lerp calculations each frame, which involves a lot of floating point multiplication.)

So first, we must define some boundaries for which we can create a Mandelbrot Set fractal. Let us pick (-1.67, -1.12) and (0.8, 1.12), as this is the boundary which gives me the best results.

// our domain and range of Real/Imaginary
const r_min: f64 = -1.67;
const r_max: f64 = 0.8;
const i_min: f64 = -1.12;
const i_max: f64 = 1.12;

// our starting values
const Z = Complex.init(0, 0);
const C = Complex.init(r_min, i_min);

Now we can create an increment to bump our variables for every loop. If we are doing NxM number of pixels over an area, we must divide the total space of our fractal by however many pixels we wish to iterate. That is to say we must do |min|+|max| / num_pixels for both real and imaginary. We'll need to import std.math for the fabs function. We can also wrap both the increment values into a Complex type so we don't have to do the addition part ourselves. But we do also need to convert our width and heights to friendly numbers for division, so I will cast them to floats with @intToFloat.

const width_f: f64 = @intToFloat(f64, width);
const height_f: f64 = @intToFloat(f64, height);
var r_increment = Complex.init((math.fabs(r_min)+math.fabs(r_max))/width_f, 0); 
var r_increment = Complex.init(0, (math.fabs(i_min)+math.fabs(i_max))/height_f);

Now we get very tiny numbers we can bump C by every loop. Or very large numbers, depending on your domain. Who knows. We have to take the absolute value of both the min and max values, because if they were opposites of eachother and completely cancelled out, then it would result in zero. Taking the absolute value of both ensures we can have a correct calculation of space.

Now our iteration will look something like this (drawing is still left out right now).

// main loop for rendering
while (y < height) : (y += 1) {
    while (x < width) : (x += 1) {
        Z.real = 0;
        Z.imag = 0;
        i = get_numcycles(&Z, &C);
        C.add(&r_increment);
        // draw(x, y);
    }
    x = 0;
    // punch it back like a typewriter
    C.real = r_min;
    C.add(&i_increment);
}

File Writing

File writing is... Well, boring. Zig has a lot of shortcuts for handling errors, because you can return errors with Zig, which allow for added safety and compile-time gotchas. If you can't open a resource, the Zig program should be able to produce a panic, and by default provides error handling for those cases. You can add fallback strategies in place, or you can default to basic panics given by Zig. This approach is pretty similar to Rust's try! macro, where it tries to force the result of a error-inducing function.

For file writing, we technically don't have to write to a file for this type of program. It's possible to run the program and dump the standard output text to a file via shell redirection, like ./mandelbrot > my_image.ppm. But that's an extra step required for the user, so let's just open a file and give it a shot.

File opening in Zig is tricky, but the way I have had most success with is using a method called createFile. It is based off the std.fs.cwd() method to give you a Dir object relative to your environment, but createFile serves as a function to both create new files, and also open files for writing. Unlike openFile, which only opens files, createFile handles the case where there is no files, removing an annoying check you would have to do.

So when writing the header of our image, we would do something like this.

// create a file and open up a writer stream
var fi = try fs.cwd().createFile("test.ppm", .{});
var writer = fi.writer();
defer fi.close();

// write the headers
try writer.print("P6\n", .{});
try writer.print("#Real <>, Imag <>\n", .{});
try writer.print("640 480\n", .{});
try writer.print("255\n", .{});

Using defer we can have the file be closed at the end of the scope. A defer statement helps us so we don't forget, as Zig does not include fancy features like RAII for automatic memory freeing. This Writer will help us also render our pixels during the iteration count loop part.

while (y < height) : (y += 1) {
  while (x < width) : (x += 1) {
    Z.real = 0;
    Z.imag = 0;
    i = iter(&Z, &C);
    C.adds(&r_increment);
    // draw the pixel
    writer.print("{c}{c}{c}", .{i, i, i});
  }
  x = 0;
  C.real = r_min;
  C.add(&i_increment);
}

The Netpbm format requires us to render out pixels in top-left to bottom-right order, so the way we iterate from 0 < y < height and 0 < x < width is the proper way of going about it. But at each step of the way, we print three numbers of 8-bits each representing an RGB tuple. In this sample program, the number of iteration cycles is how dark or how bright our pixel will be. It can generate the following picture now.

Speeding up Writing

Every cycle of iteration we do a writer.print call, which effectively slows down the hot loop by a large margin. The rendering phase has to stop each time to run a call to the file writer, then after writing, it gives back control to the loop to continue to the next cycle. This can be avoided if we were to somehow buffer our output data into some storage, then write the buffer to the file every so often instead.

Before we begin the rendering steps, we are going to set up a fixed-sized array buffer and use that for our rendering stage.

var bi: usize = 0;
const bufsize: usize = 900;
var buffer: [bufsize]u8 = undefined;

The amount of the buffer size can be anything you want, but I picked a nice odd number as we have to store three numbers at a time for each pixel. So be sure to pick a multiple of three that's a decently good size.

The next step is to write what's essentially a buffer store and a buffer flush. We store to the buffer each loop, but the minute our buffer index bi exceeds the buffer capacity, we flush out to the file via writer.print.

First let's store to the buffer in the inner drawing loop.

//... other while loop code {
i = get_numcycles(&Z, &C);

// store it in buffer
buffer[bi] = i;
buffer[bi + 1] = i;
buffer[bi + 2] = i;
bi += 3;

// check if we filled the buffer
if (bi >= bufsize) {
    try writer.print("{s}", .{buffer[0..bufsize]});
    bi = 0;
}
// end while loop }

That will store all values into a temporary buffer, then write to file once the buffer is full, and then pick up where it left off. We don't have to fill the array with zeroes or anything, as we can reset the buffer index to zero for all the data to be overridden in the next loops.

But uhoh, there's an issue. If our image isn't an exact multiple of the size of the buffer, values get left behind. This is where we will do a post-render flush just in case there's data left in the buffer. Using another Zig slicer (ie: buffer[0..bi]) we can tell Zig the exact size of the array we wish to write to the file.

// after the two while loops - do this
if (bi > 0) {
    try writer.print("{s}", .{buffer[0..bi]});
}

Now we have a properly buffered Mandelbrot set renderer, and this performance trick will save us on speed, at the cost of making the program space a little larger due to the buffer (it's negligible though, 900 bytes is not that much).

Analysis

I will label my results with the following: Zig (f64), Zig (f128), and C++. I do not want to say my C++ version is perfect, and I think it could have improvements in it, but here were my results from testing.

Zig (f64)  - Average Time: ~84ms
Zig (f128) - Average Time: ~3.2s
C++        - Average Time: ~210ms

All binaries built with release optimizations

At first sight, seeing anything below my C++ version was incredible! The 64-bit floating point is notably better by almost a 2x factor, is written very similar, and means I probably don't have to write C++ anytime soon.

The 128-bit floating point version is where I was scratching my head for a while, because I didn't understand why it was that much slower. I started with a 128-bit program, but then converted it to 64-bit once I saw it was that much slower than the C++ one. Upon defaulting to 64-bit, speeds improved. But why?

My only conclusion I can come to is that despite the type being available in the language, it does not mean processors are going to natively support a 128-bit floating point type. What most likely happens is Zig generates some code in the background to target our host architecture, then exports code that would allow us to use 128-bit types, but that doesn't mean they are actually part of the original architecture. There is no way a 128-bit floating point type is native to my CPU, so it most likely has to do a lot of instructions every time a multiplication is actually involved.

So by turning my program into an only-64-bit program, my runtime is much quicker than expected. But if I wanted the added safety of a lot of bits of precision, then I can turn on 128-bit, wait a little bit longer, and get more accurate results in the long run. But this wouldn't be possible inside of C++ at least, because there is no 128-bit type available. I would have to import a library somewhere. So I consider this a cool Zig feature that we can plug in and make trade-offs with easily.

Now, what the hell is wrong with my C++ version? I suspect it's because I don't use a proper buffer. Each cycle is written to the file stream as-is and I make zero attempts to buffer it in any way with a fixed-size array. I don't know if I will ever add this feature, but I suspect my C++ version could go low, if not a little bit lower, than my basic Zig implementation here.

Conclusion

I'm pretty happy with Zig so far. I need to flesh this program out some more before I finally can use it for some practical use-cases, but Zig has been great so far. The lack of resources on the topic is what hurts Zig the most right now, but the language is very usable to get started with. It can interact with C libraries, it can even compile C/C++ code. Zig is pretty darn cool, and I can't wait to see what else Zig does in the future.

Also check out this blooper image I rendered when developing the program. This was rendered incorrectly when I used a bad domain, and also because I had written the colors incorrectly. But it looks pretty cool for some reason, so I'm not ashamed of my mistake.

If you liked this post, you can check out my live Mandelbrot Set generator. It's a lot like this, but it does it slowly and one section at a time until a final image is reached. Thanks for reading!

Designing A Work CLI

2022-01-06T20:00:00Z

I have been playing around with the idea of creating a work interface of some sorts for various purposes. My job consists of doing a lot of database look-ups, checking different websites, juggling tabs, and overall mostly just web navigation and data managing.

But I thought I might take a stab at creating a kind of REPL environment for my job, so here we go.

How to Use

The first big step is - how do I use this CLI? What's my user interface?

Recently I made the switch over to a Gnome environment, so I have been using fairly stock software with no real frills. But if I was asked to do a database lookup by a specific identifier, I have to go to my browser, load the web page of our in-house tools, then punch in the identifier, then navigate several pages to find the details I most likely need.

This isn't ideal for me anymore. I want to punch in a string and be taken to where I want to be, and to do this, I need some form of quick URL navigation combined with a fast way of opening web pages, most likely by invoking the browser directly or using xdg-open. I have a bad relationship with xdg-open, so I am simply going to pass arguments to the browser binary instead.

I need a way of accessing a REPL text environment quickly, so I installed Guake. Guake is a cool little terminal similar to the days of old where games had consoles you can drop into and do cool tricks with. Guake has a fast key shortcut to get to a terminal, so I gave this a whirl. Guake starts on login (if you set it), and you can toggle the terminal's visibility with guake-toggle. Then you customize it to your heart's content.

This is where I wanted my programs to be invoked from, almost as if I were playing Quake/Doom again.

Now all that's left is to design a REPL environment that works, so.... Let's use Racket

REPL with Racket

Racket, when you start it from the command line, drops you into a REPL automatically. The next step would be to then find a way to create a program that doesn't get interpreted as if it were a program, but instead evaluates and lets you use the bindings in a REPL shell.

In order to do this, we must pass Racket some information, but we can create a REPL program with this command.

$ racket -l racket/base -if WorkCLI.rkt

The three arguments here are -l, meaning library load a language. We want to use the racket/base language as it loads the bare minimum Racket environment, cutting down on memory overhead. The other two flags joined into one are -i and -f, one meaning to start an interpreter environment, and the other is to load a specific file.

When doing this, in our WorkCLI.rkt program, we must not define a language at the top level - that job is handled by the -l racket/base part. If we define a language at the top level, the function is entirely different - it gets treated as a module space with a language definition instead, which we do not want. Instead, we simply declare the language with the -l flag, then we import definitions using the -f flag.

(You can also choose to not use the -l flag, but it will default to the basic racket language module, which imports a lot more than the racket/base module, causing startup times to be slower and will use more memory.)

Here we can start creating definitions and actual Racket code and injecting it into the Racket REPL. So let's start writing.

Web Browser

Imagine we're in Python - we want to open a web page, how do we do it?

import webbrowser
webbrowser.open("https://gitlab.com/")

That's pretty convenient! How does it work, magical Python bindings?

Er... No, not really. But it is pretty widely compatible with multiple platforms, it just uses common grounds between all browsers. Since we're on Linux, our desktop refers to the XDG media query system for opening certain files or paths, and if it sees a URL looking item, it defers it to some web browser on the system.

Unfortunately - we are not using Python. We are in Racket. So we must do this ourselves. If we want to make it cross-platform, we best also check for multiple browsers as well.

So let's say I started off with my naive approach - I had a few browsers in mind, but as the browser list expanded, so did my code...

; shortcut name
(define fep find-executable-path)

(define *browser*
 (let ([ff (fep "firefox")])
  (if (not (false? ff))
   ff
   (let ([chrome1 (fep "chrome")])
    (if (not (false? chrome1))
      chrome1
      (let ([chrome2 (fep "chromium")])
      (if (not (false? chrome2))
       chrome2
       (let ([edge (fep "MicrosoftEdge")])
        (if (not (false? edge))
         edge
         (error "Where is your web browser?!??"))))))))))

On another kind of day, I'd let this sit, because I think it looks downright hilarious and represents the grind of dealing with browser executable location. There are many other browsers in the world, but I'll focus on these four for now. Sorry IceCat, Konqueror, SeaMonkey, Lynx, Wget, Nyxt, Emacs, and all the other homies.

I decided to re-write this anyway since it resembles a very basic pattern anyway.

(define (find-bowser-pls listof-bowsers)
  (define (iter xs acc)
    (if (not (false? acc))
        acc
        (if (empty? xs)
            (error "Where is your browser?!?")
            (iter (cdr xs)
                  (find-executable-path (car xs))))))
  (iter listof-bowsers #f))

; Top-level browser definition
(define *browser*
  (find-bowser-pls
   `("firefox" "chrome" "chromium" "MicrosoftEdge")))

Now you can support all browsers much easier without adding several let/if scopes. Cool, right?

Now we can use a subprocess to open up our target browser with a URL each time we want to quickly access a web page with some primary key search or whatever you can rig it to. Let's write the Racket version of webbrowser.open.

; webbrowser.open/1 equivalent
(define (webpage-open URL)
  (define-values (s i o e)
    (subprocess #f #f 'stdout *browser* URL))
  (subprocess-wait s))

A subprocess is just a way of invoking other programs on the host system, and linking the main Racket process as the parent. The parent can run, monitor, suspend and kill child tasks all with relative ease, and that goes for browser programs too like Firefox.

Firefox isn't like other programs, it's more like a running server, so invoking firefox under Racket doesn't always mean the whole Firefox program is going to live as a child of the Racket process. Rather, Firefox starts a daemon process and receives inputs instead. This is similar to how you can run Emacs as a server, so Emacs never dies as a process, but it can host multiple clients at once.

So by directly calling the browser binary with an argument input, it does indeed open up our web page. Next up is simple string formatting and creating easy and simple to use functions that do what we need.

Creating Tool Functions

The primitive way of creating a REPL environment is to simply define functions no different than what we're used to. If I want to search a term on Amazon, I simply string format the search term into a URL with a pattern variable.

; add on amazon
(define (amzn-search identifier)
  (webpage-open
    (format "https://.amazon.com/s?k=~a" identifier)))

This is decent for one-off instances, but if endpoints or URLs ever change, we will unfortunately have to stroll back through some code and figure out what changed where and how to fix. Maybe not so bad in the long run, especially if it saves you a lot of time before changes occur.

If we were to wrap this around some kind of API-like web system, where the functions are pieced together like "webpage.com/catalog/add-product", we can encapsulate it using a bit better using constructive functions that wrap around URLs to create easy wrappers.

; build a functione dependent on *url* to be set
(define (builder func-path)
 (lambda (identifier)
   (format (format "~a~a" *url* func-path) identifier)))

; create a catalog browsing shortcut
(define catalog-browse (builder "/catalog/browser?ID=~a"))

; call it now
(catalog-browse "01118778735")

Using a double-format call allows us to replace a variable in a template string with one that contains another variable. This allows you to chain templates together for further text replacement.

You can build up the logic however you wish. Whether it be with clever function currying like the above, but the unfortunate part is how we deal with the next issue - telling the user what commands are actually available in the REPL.

A Help Function

Normally when you're in some kind of REPL-like environment, or playing a text-based adventure game in the year 2022 and onwards, you normally rely on a function that spits out all possible commands you are legally allowed to enter into it. Racket does not provide this, but, unironically, Python does it a lot better. Using the dir() function in Python, you can see almost everything in the runtime.

>>> dir()
['__annotations__', '__builtins__', '__doc__', ...]
>>> dir(__builtins__)
['ArithmeticError', 'AssertionError', 'AttributeError', ...]

This is actually how I discovered some crazy things Python has, like the built-in Ellipsis type, which is almost the same size as None. Why it exists? Honestly, I don't really know...

>>> x = Ellipsis
>>> print(x)
Ellipsis
>>> x.__sizeof__()
16
>>> y = None
>>> y.__sizeof__()
16   # hmmm 🤔

But, all that aside, the dir() function is very cool and lets you explore modules all from inside the Python runtime. Sadly, I don't think this exists in Racket. That's not a bad thing, but ... if it existed, I just don't know about it. If there was a way to expose the bindings defined in the namespace, maybe? I'm still not sure.

We need a help function for the user, in case they forget our weird functions. Or in most cases, if I were to forget. We can use a hash type to associate function names with help messages of some sort, but unlike Python, we don't have help strings available. Instead we must do this afterwards.

; creating a help me screen of info
(define *helplist*
  (make-parameter (make-immutable-hash)))

(define (update-help k v)
  (unless (hash-has-key? (*helplist*) k)
    (*helplist* (hash-set (*helplist*) k v))))

(define (some-function x)
  (displayln "Done"))

(update-help 'some-function "Does nothing and prints done")

Now we have a parameterized hash which relates names to help messages. But it's kind of annoying that for each function we would have to update the hash manually like that, but none the less, we can print out our help screen with:

; print out the help hash
(define (help)
  (hash-for-each (*helplist*)
    (lambda (k v)
      (displayln (format "~a - ~a" k v)))))

I kind of admire the docstring format of Python a little bit more, a way of nudging a small string into the function declaration, so that way coders who dive into your code don't have to feel so alone. Granted I'm the only one using this, but those docstrings could be repurposed to match the help screen hash needs.

Problem is, the define syntax doesn't fit this solution, so we might need to think of something else..... Or, we could use a macro?

Help Strings and a Macro

Let's say for goodness sake that instead of using a macro, we just went with the normal define syntax. I don't need to do the update-hash function outside of the function, I can do it from within the function entirely, and that still works.

; docstrings, but as a function
(define (do-nothing)
  (update-hash 'do-nothing "Does nothing, don't use")
  (displayln "Done"))

> (help)
do-nothing - Does nothing, don't use

Completely plausible to do this, but we don't want to register the hash each time we call the function, we only want to register it once. So even though update-hash checks if the function has already been registered, we still don't need to make the call at all. Not necessarily a performance bump, but still not needed.

The other issue is that we have to re-declare the title of the function so update-hash knows the name. Doesn't this seem kind of silly? We're inside the function, do we really need to spell it out?

I wrote a little macro for this that we could make everyone happy, and get to practice writing macros again. Let's take a look.

; The Action! macro, join help messages with code
(define-syntax (Action! stx)
 (syntax-case stx ()
  [(_ nameof helpmsg code)
   (if (not (list? (syntax->datum #'nameof)))
       #`(begin
          (update-help (syntax->datum #'nameof) helpmsg)
          (define nameof code))
       (with-syntax
        ([siglist (syntax->datum #'nameof)])
         (begin
          (unless (andmap identifier? (syntax->list #'siglist))
           (raise-syntax-error #f "Invalid identifiers given" stx))
          (unless (string? (syntax->datum #'helpmsg))
           (raise-syntax-error #f "Given helpmsg not a string" stx))
          #`(begin
             (update-help (syntax->datum (car (syntax->list #'siglist)))
                          helpmsg)
             (define nameof code)))))]))

This might be one of the least complicated macros you'll ever see, and it wasn't that hard to put together. But the concept is we want to register a hash with a symbol pointing to a help message, then we do the literal function definition right after. The complicated unless part is where we do the hash update, and then after that, the define line is pretty simple.

Using this looks pretty basic in action.

; defining an action, which has a help string
(Action! (say-hello name)
  "Says hello to whatever name is fed"
  (displayln (format "Hello ~a!" name)))

; use it
> (say-hello "Steven")
Hello Steven!

But more importantly, it registers in the hash we defined earlier too.

> (help)
say-hello - Says hello to whatever name is fed

Now all the functions defined with the Action! macro are registered to the help hash. No frills added, but Action! may make your life more annoying for your text editing plugin depending on how you want to format it.

However, note that we have an if statement very early asking if the nameof variable is a list or not. Why is that? Because we have a bit of a branch to check for when writing code.

The Racket macro compiler sees these two pieces of code as matching the macro rule.

(Action! single-var "A single variable" 0)
(Action! (a-func x) "Do something" (displayln X))

Here, the nameof variables are completely different - one is a single symbol value, while the other is a list. So to return different results, we need to see if the nameof type looks something like a list or not, thus resulting in us creating different code paths.

The former rule here allows us to implicitly create functions, while the latter explicitly creates functions. Kind of neat difference, and separates writing styles. I lean more towards implicit in some cases where I can, because it reduces code strain, but there's nothing wrong with explicit either.

(define (func-maker X)
 (lambda (Y)
   (+ X Y)))

; implicitly declare
(Action! add5 
  "Adds 5 to whatever"
  (func-maker 5))

; or explicitly
(Action! (add6 X)
  "Adds 6 to whatever"
  (+ 6 X))

Now with that all the way, let's move onto the last bit.

Lastly, Helping When Needed

Say I don't know anything, I don't even know help is a function. I should be printing a welcome message that says "hey, use the help function if you actually need help", but a few lines down and some forgotten days, maybe that's not clear unless you reboot the program.

Let's pretend in some world we just make a mistake, and we need help. How do we go about that? What happens if I type in a function wrong?

> (do-somethinG)  ; accidental capital letter
; do-somethinG: undefined;
;  cannot reference an identifier before its definition
;   in module: top-level
; [,bt for context]

When we try to reference a binding, all we get are exceptions that tell us our issue literally, from the perspective of the runtime. Alone, it won't tell us that we were off by one letter, or we typo'd and put a capital letter in some place that didn't need it. There's no error detection like that to really assist the user.

But also, I don't plan on adding that feature either currently, because it would involve a lot of debugging I don't know if I want to support right now. Maybe in the future I'll think about how to handle this, but for now I want to be able to show the help message when an error occurs. How do we go about this?

My familiarity with errors, or exceptions, is simple - you can run code, but even if you account for all possible return paths and conditions, something can always break, even without you account for it. Out-of-memory, network disconnections, process gets killed by the admin, or your computer turns off. Exceptions are designed as fall-backs for when certain errors occur that might not necessarily be the fault of the user.

So how does one handle exceptions normally? By using with-handlers

; handle some code gracefully with another function
(with-handlers
  ([exn:fail? (lambda (e) (displayln "Error caught"))])
  (error "I am raising an error!!"))

; output
Error caught

We went from verbose error message with details, to a much more simple string output displaying the worst error information imaginable. But hey, this kind of works for us.

The problem is there is no way we can perform with-handlers in a REPL environment, because there's no body for which we can bind this macro to. When the REPL runs, it's not like we can bind it inside with-handlers. Instead, we need to defer to some special parameters that exist, namely uncaught-exception-handler.

; Help the poor user
(uncaught-exception-handler
 (λ (x)
   (begin
     (displayln "Invalid pattern or syntax")
     (displayln "Try one of the following commands")
     (help)
     ((error-escape-handler)))))

uncaught-exception-handler is a specialized parameter called when exceptions are raised in the Racket runtime, but don't have handlers to deal with them. This default parameter function is pretty commonly used, because it's not like there's better functions to handle it.

In our REPL environment, this is our one and only way of converting all sorts of errors, into something that can assist the user in doing things properly and getting what they need done. By overwriting the default handler, you can create fun REPL applications this way, you could even make games. It's still a Racket shell, so valid Racket code will always work, but I thought this was a fun thing to try and do.

Wrapping Up

So after some hours of experimenting, getting Guake to run properly on keybind, learning about how keybinds work in Wayland, and writing this entire program up, I now have a working CLI for job oriented purposes. 😄

I can pop down my Guake shell, pop in a command, and it will either direct me to the web page I need, or perform other various office-related duties for me. It's kind of cool, and I hope I can make some co-workers jealous with my Linux desktop.

The better part will be how I connect it to my other Racket programs, but I'll have to work on that part. Should make for some interesting developments.

Alternatives I have tried:

Making a home page for the office - I can't configure browsers to change their new tab page contents without some extensive plugins. I tried creating something that resembled an Opera browser "speed dial", but it became more burdensome figuring out how to make it work in multiple browsers.
Using a program launcher like xfce4-appfinder or something to run the programs - creating multiple scripts linked to some random binaries or whatever didn't seem very ideal. I at first thought about using rofi to do what I'm doing now, but I figured Guake was more practical.
Bookmarks and other things - browsing the web normally is a pain in the butt, especially when I can craft URLs with identifiers much easier in code. I am trying to do away with having to rely on bookmarks for everything, and want to make my CLI capable of doing a lot.

That is it for now, thank you for reading. 🦖

A Macro Journey

2021-12-24T20:00:00Z

A while back I decided to write about a story of me at work, developing tools for wrangling spreadsheet data, namely CSV sheets. I was using the Racket programming language, which is a language I've really come to enjoy working within. It's simple, easy, and the functionality and flexibility is pretty incredible. It's multiple paradigms at once, can do many things, can compile it's code to contain miniature VMs for multiple platforms, and overall proves to be useful to me over and over.

But this story didn't always start out pretty. I was tasked with handling a lot of random spreadsheet data and trying to create logic to interweave different data sets. I didn't have much in the way of project support, and I didn't want to integrate parts that would make it more complicated. As the sole programmer I could pick whatever environment I would want to work with, but finding a language that dealt with CSV sheets gracefully was a draining process.

This story will be about how I approached this problem using native Racket components, and why I decided to go the way I went. And eventually how I ported this code to my open source CSV package ez-csv.

Structs

A struct can mean different things in many different languages, but in reality, a struct in Racket is a lot like a list that has named accessor methods. The basic gist of a struct looks like the following:

; defining the name of the struct, and it's fields
(struct point (x y z))

; make a struct
(define p1 (point 3 4 5))

; access the Z field
(displayln (point-z p1)

; predicate to determine if a type is a point
(displayln (point? p1))

The struct is effectively a macro that defines many different functions for us. From that one line definition it creates a constructor method, the accessors to all the fields, and a predicate function. There are other struct mechanisms that exist like some basic inheritance and data copying, but I won't talk about those quite yet.

So naturally, a struct seems like a decent starting out point to read in some CSV data. A CSV record is a string separated by commas that represent N-number of fields. The first line of a CSV file tells us the headers while the following lines are all the records. We can create a naive implementation of a CSV reader in a few lines using some basic string functionality.

(define (read-csv csv-file-path)
  (call-with-input-file csv-file-path
    (lambda (input)
      (map
        (lambda (line)
          (string-split line ","))
      (port->lines input)))))

> (read-csv "example.csv")
'(("name", "id_number")
  ("steven", "id123")
  ("thomas", "id457")
  ...

This implements a basic kind of parser. You can take the head of the list and use it as header information, and the rest of the list as the data. You can encapsulate the CSV using structs to create easy accessors instead of depending on traversal like car, cdr and cadr.

; define an Employee struct
(struct Employee (name id))

; define a mapper tool to iterate over a list of data
(define (lines->Employees listof-records)
  (map (lambda (listof-data) (apply Employee listof-data))
       listof-records))

; create an easy way to convert file to structs
; uses a cdr call to skip the first header row
(define file->Employees
  (compose lines->Employees cdr read-csv))

Now we can read a file and turn it into a struct by using apply. But however, the reverse operation is less clear, because the headers were contained inside the read data. By throwing it away, we've failed to remember it for when we want to write records back out. This may or may not matter depending on the application, but keeping track of data is now an extra chore to keep in mind.

(define (read-header-and-records fpath)
  (call-with-input-file)
    (lambda (in-file)
      (define all-rows
        (map (lambda (line)
          (apply Employee (string-split line ",")))
          (port->lines in-file)))
      (values (car all-rows)
              (cdr all-rows))))

(define-values (header records)
  (read-header-and-records "my_employees.csv")

You can do it by using values to return multiple values each time, but now the header information is still taken apart from the original record data. To write file out, we must remember to include it with generic functions, as well as always remembering to use define-values instead of regular define, which could be an extra mental step.

Updating records to keep a more up-to-date file, or maybe to do some simple operations on data, are easy to do using struct-copy, which allows you to copy a struct and return a new one, leaving the original completely unmodified.

; use struct-copy to copy the original struct and apply changes
(define (update-record rec)
  (struct-copy Employee rec
               [name (string-upcase (Employee-name rec))]))

; now all employees will have names with uppercase letters only
(define fixed-data (map update-record all-employees))

Now we come to the last part of the problem, where we must now write the records to a CSV file. We have to pass in the headers externally if they are kept the same.

; we need to convert a record to a string
(define (Employee->string rec)
  (string-join (list (Employee-name rec)
                     (Employee-id rec)) ","))

; iterate over all records and write to file
(define (write-csv out-path header records)
  (call-with-output-file #:exists 'replace
    (lambda (out-port)
      (parameterize ([current-output-port out-port])
        (displayln (Employee->string header))
        (for-each
          (lambda (record)
            (displayln (Employee->string record)))
           records)))))

; write the data to file using the above call
(write-csv "new_data.csv" header fixed-data)

We now have a full successful means of operating over records. However, there's so many issues I have with all this code that it drives me up the wall. Yes, we can work with this code, and yes, I have been doing it in similar ways using these functions for months now. But unfortuatenly, it makes me go crazy.

Since header information is part of the spreadsheet, it must be kept either alongside the data, or inside some data type that can hold both. It is significantly easier to not include headers into the native struct itself, so you don't always have to include a header field each time you declare a new struct definition.
Structs make it convenient, but it cannot be written to a file directly without some sort of serialization that displayln can understand. We need it in a string format, meaning we need to also keep track of a delimiter each time. Additional information overhead, and not exactly something we also want to have to create an accessor field for in a struct.
Defining all the intermediary functions like struct->string is kind of boring. This part can be done automatically in some sense, but the struct macro will not provide it. You can provide a function by setting prop:custom-write, but again, that's additional overhead you will have to tack on yourself.

I have been writing CSV parsing code for a while now, and this is the main three problems I had. struct got me close, but not close enough to not be tiresome in the long run. I had a few choices to look into.

Classes and Objects
Third party libraries
A DIY Racket Macro

Classes and Objects

The reason why I don't like the class idea is because it doesn't resonate well with me. Racket is largely a functional Lisp-family language, and I use it primarily because I enjoy the functional programming aspect. Going back to an object-oriented style of coding would be a detriment to me, as it means I have to interject OOP-like closures and new methods and retain more mental gymnastics.

While I think a class/object relationship approach would be nice, there are some ideas that mesh rather well. Sometimes I have data in one sheet that needs to be extended and carry over information. Classes can define that relationship well, by having parents and children classes that extend fields.

The downside is to interact with objects you largely have to use a function called send. send will quite literally send a message to an object to execute some code that is bound to that instance of data. It could be mutable, or it could be immutable. Hard to tell from that angle, as you may or may not get return values based on the code itself.

It seemed both better and worse in some aspects. I could get OOP relationships in data, but functionality would be all over the place and (probably) broken up in different classes that wouldn't need classes. I would have to write new methods that can interact with the send function, and then I have to remember key identifiers that match up to the bound functions. Maybe some functions aren't bound to certain data types, and I forget that, and lose an hour trying to figure out where I messed up if the stack trace errors aren't clear enough. It seemed like a net loss more than a net gain.

Third Party Libraries

My other solution would be to use other people's approaches to CSV files. From what I have seen, there aren't many projects out there dedicated to this, or maybe some of them are simply too old for me to want to run the risk of using. Here's a sample of libraries I've looked at.

csv - last updated August 2017
csv-reading - how do I even contribute to this??
csv-writing - last updated November 2019
simple-csv - fails to build

I need to write reliable code now, that I can possibly share with others. CSV files aren't that complex, but if I have shortcomings with someone else's code, I need to get things working now, not later. I have time restrictions that simply make it not possible for me to make open source contributions that could take who knows how long to get merged.

Therefore, I have published ez-csv, as a means of approaching this weird CSV space that doesn't get enough attention. I am sharing my code that reads and writes CSV files to be helpful and give back to the Racket community.

A DIY Racket Macro

I am vaguely familiar with the notions behind a Racket macro. A Racket macro is essentially a type of Racket function that operates on a code as if it were a data type. Instead of using something like a C preprocessor, which comes with a slew of developer woes, Racket uses it's own language as the preprocessor.

The chain of operations looks something like this:

<Racket Macro Code> ; (define x 3)
 |
 -> [Racket Macro Expansion]
    |
    -> <Racket Code> ; (define-values (x) (quote 3))
       |
       > [Racket Interpreter]
         |
         > <Results> ; > 3

So writing macro code looks really no different than that of regular Racket code. In fact, you are allowed to write macro and non-macro code in the same space. The difference is that the data we operate on is a little bit different - it's mostly symbols and lists.

So no longer are you working with numbers or strings, you're working purely with lists and creating new lists. You take in a list of symbols and inner lists, and output a new list of symbols/lists. Let's observe some simple macros we'll call syntax rules.

; a simple identity rule - return whatever it gives
(define-syntax-rule (id X) X)

; null - return nothing given something
(define-syntax-rule (null X)
  (void))
  
; duplicate - duplicate whatever is given
(define-syntax-rule (duplicate X)
 `(,X ,X))


> (id 3) ; => 3
> (null 3) ; => <returns void, no value>
> (duplicate 3)
'(3 3)
> (duplicate (displayln "hi"))
hi
hi
'(#<void> #<void>)

The duplicate function is slightly interesting. Notice how it evaluates the effects of displayln, and then captures the void values of the displayln call? It's because we evaluated a single piece of code twice! That's the core of all programming right there - doing more with less.

This is the key difference in how macros work versus how ordinary functions work. In a normal evaluation, the displayln call would have been evaluated, "hi" would have been printed, and we would see two void values. Instead in a macro environment, we took displayln as if it were a normal list of symbols, then returned the expansion of it being duplicated.

Though if we were to want to avoid this by using a macro, we need to evaluate the code first, store that result, then duplicate the values afterwards. We could write this as an ordinary function, but it's a neat exercise in understanding macro code expansion.

(define-syntax-rule (run-once-dupe X)
  (let ([runs-once X])
    `(,runs-once ,runs-once)))

> (run-once-dupe (displayln "hi"))
hi
'(#<void> #<void>)

At a glance, you think this might not work at all. There's no real difference between doing this as a macro versus an ordinary function. You could be right - there isn't any. Some people might think it should print twice, and you are correct in that line of thinking. But in macro land, things are different.

In that let binding, it appears as if all we're doing is binding the variable runs-once to the input argument of X, which would be a list of some Racket code. The let variable should then be evaluated twice as per the quasi-quoted list, but at this point, we're past that point. We've already expanded the input argument and turned it into a regular let expression. The value is evaluated at the let expression, even with a binding as simple as the one pictured. The real expansion looks something like this now.

(let ([runs-once (displayln "hi")])
  `(,runs-once ,runs-once))

The input code was transformed into a let block, and now there's no multiple displayln expressions. Therefore, displayln only prints once, but we duplicate the #<void> value like before.

This isn't a fancy macro at all, but it's an interesting dive into how macro transformation works. You have to take code literally and visualize almost like a copy-and-paste transform of code when doing macro expansion. There's no real tricks to it, it's very simple.

One of the special things with macros we can do is the ability to have code defined for us ahead of time using some magic macro trickery. We have to use some more advanced macro tooling, but it's not too hard to follow. Namely we're going to use syntax-case and with-syntax as special ways of creating new code.

(define-syntax (make-names stx)
  (syntax-case stx ()
    [(_ first last)
     (with-syntax
      ([first-name (format-id #'first "first-name")]
       [last-name (format-id #'last "last-name")]
       [full-name (format-id #'first "full-name")])
     #'(begin
        (define first-name first)
        (define last-name last)
        (define full-name (string-append first " " last))))]))

> (make-names "Steven" "Leibrock")
> full-name
"Steven Leibrock"

syntax-case and with-syntax are special functions designed to work with syntax templates and patterns, otherwise we would be left with destructuring syntax lists ourselves, and that's really not much fun. The final result is pretty cool, it is able to create three variable bindings on our behalf from only two strings, and now we get bindings to first, last and the full name on top of it by only doing one line of code. Pretty neat!

This effectively serves as the basis of my new CSV binding code. Using syntax-case and with-syntax, I am able to create a pretty feature-complete portable CSV library that allows users to define their own CSV record type. The code is far too long to actually paste here, but you can see the full thing over at ez-csv.

The #'(begin ...) pattern is the syntax for defining a new code block, which the macro will return for later evaluation. It's the same thing as a regular Racket list. You can use cons to attach something to the front, you can iterate over it, and do many cool things within macros. My CSV library is really only scratching the surface of what one can do with some simple macros. In some cases of my programming, basic macros are simply hygienic and prevent me from doing some insane code re-use.

The case for with-syntax is that we cannot directly refer to template variables from syntax-case directly. They instead have to be cast into something we can use, which is where with-syntax comes into play. In fact, with-syntax is a specialized form of syntax-case, but it's written a bit differently. format-id is used to create identifiers that we can reference to in code, and it can inject symbols into format patterns. It does a bit more than simple string formatting, such as generating proper identifiers, but that one's above my head.

You can also use things like identifier? or free-identifier? to determine if symbols are actually identifiers (or free to reserve in the runtime), from which you can say "hey, let's not bother continuing, these variables are already in use" by throwing a syntax error. Or if you would like to mutate the already in-place variable using set!, you can do that too!

Macros are pretty funny when you start to get into them, and it's powerful what you can do. Here's a sample snippet of how I use my defrec macro, a macro that auto-creates a ton of CSV handling code for me.

; define a CSV record, which populates many functions for me
(defrec Contact ["Name" "Number"] [name number] ",")

; import my phonebook, each as a Contact record
(define my-contacts (file->Contacts "phonebook.csv")

; map a displayln over all records using a Contact->string proc
(for-each (compose displayln Contact->string) my-contacts)

There's some more hidden functions, but again, check out the repository to get a better picture of things.

Conclusion

I have been working on this macro for maybe what seems like two weeks now, and I am pleased with it's progress in my real-life work scenarios. I don't think the other libraries do what this macro is doing currently. I'm excited and also scared at the idea of other people using this and submitting feedback to me. I'm also scared of not doing macros the right way, so I think I need to seek feedback from more veteran Racket users still.

This has been a fun journey and I do love using Racket. It's been a blast, and since I can't share it with most of my real-life friends, I figured I would do my best to share my experience with the world. Thank you for reading!

CSS Practices

2021-09-22T20:00:00Z

CSS, while not my strongest suit, is something I'm trying to actively get better with. I am by no means a "designer" or "visual artist", but I have developed some senses over the years that lend to my ability to know good from bad.

CSS is pretty much the opposite of coding in the sense that you create logic to automate and do away with the nastiness of doing things yourself. But it's not too different from a form of programming called Composition over Inheritance, where you describe properties for objects via traits or interfaces, instead of inheriting from a parent object.

The key groupings in CSS are based on Elements, Identifiers, and Classes. Element refers to an HTML element in the page, Identifier refers to a singular object in the page, while Class refers to many elements in a page.

-- CSS, but if it were Haskell
data Rule = Rule String String
data CSS = Element [Rule]
         | Identifier Element
         | Class [Element]

Recently I had a large HTML/CSS project, and it wasn't something I was totally amazing at, but I tried my best to learn some tricks along the way. Here I will share my insight into CSS that I have developed over the course of a few years.

Divide Your Screen Real Estate

Pretend you're designing a GUI program in C++ and Qt. How would you do it? You have an 800 pixel wide, 600 pixel high screen space. What do you do? You probably start by saying the first 50 pixels of your box are reserved for the File menu bar at the top. Then you start coming up with how you want to present your program. Data lists, buttons, interactions, a canvas for drawing displays, all that fun stuff.

This applies in CSS. If you're designing a web page, think about how you set up your screen space. If it's intended to be a three-column website, think of how evenly or unevenly you want the columns to be. 20/60/20? 10/70/20? Or maybe it's not even multiple columns at all, it's a single-column website.

The main thing to start getting used to in CSS is the idea of breaking things up into groupings of div elements, and using that to create your visual space. It's a lot easier to break things down into rows than it is to break things into columns. Doing the reverse requires a bit of CSS manipulation to get it to display correctly.

Example: I use a mono-column website, as do many blogs. I prefer my navigation to stay at the top so it doesn't get in the way of the user. This is especially helpful when it comes to mobile responsiveness. Phone screens are a lot smaller, so multi-column layout websites become more complex when displayed on mobile devices. But if I keep my site mono-column, then it doesn't get in the way nearly as much.

Inheritance of Space

The big thing that is hard to keep track of is how much space an element will actually have. Despite not being an object-oriented programming language with inheritance properties, CSS actually does pass information along from a parent element to the child. If the parent div has a 300px wide screen space, then it's children know how long they can run until they hit that wall.

Let's start with the width property, the one property everyone will use. It dictates how much space an element can run left and right on a screen. When this gets smaller, text and other elements will start to run out of space. The children of an element with a fixed width property cannot adjust it, or exceed it, but text can always be rendered in weird ways when approaching these limits.

Let's examine how we can make a mono-column website with only two CSS rules.

body {
    width: 100%;
}
#column {
    width: 50%;
}

<div id="page">
  <div id="column">
    <h1>My webpage</h1>
    <p>Welcome</p>
  </div>
</div>

The first rule for the body takes on the inherited width from the browser window, which can be be variable depending on the dimensions of the browser. It can be 1000px or 100px, but the body knows to use 100% of that space for the rest of the document. The #column identifier says to use 50% width of it's parent, meaning it can compute on the fly how much space it has to work with. You can stretch the browser window to see this in action.

Sometimes that isn't going to cut it, because if you shrunk the window to 100px, only 50px would be available for rendering text. Instead we need to introduce more rules, like min-width.

#column {
    width: 50%;
    min-width: 700px;
}

Now the #column identifier knows it can allocate a minimum of 700 pixels for it's space. It will use the width: 50% rule up until the space is too small, then defer to min-width: 700px.

The space using percentages instead of flat pixel amounts can help a ton to create responsive websites, and also set up for when you want to build complex spacing. Pretend we did want to do a three-column layout, we can write that using a bit of percentages.

.multi-col {
    width: %33;
    display: inline-block;
}

<body>
  <div class="multi-col">A</div>
  <div class="multi-col">B</div>
  <div class="multi-col">C</div>
</body>

Because there are multiple elements, we switch from an Identifer to a Class for CSS rules. This replaces the need to write three individual identifiers, cutting work by 2/3rds. We use a property for display called inline-block, which is able to display elements in the same group in an "inline" style instead of "block". Block typically means newlines are auto-applied when rendering, whereas inline will try to render all in the same line.

Composable Classes

This is a lesson I had to learn when I needed it the most: combining multiple classes for sanity.

A lot of the times when I was doing text decoration, it's a lot easier to put rules into a class and call it a day, but the more properties a class has, the more complex and harder it is to share with other elements. Let's say you made a rule that large red text, but later on you had elements that you wanted red text, but not as large. What's the best way to go about that?

/* How do we cut down on rule clutter? */
.large_red {
    color: red;
    font-size: 24px;
}
.medium_red {
    color: red;
    font-size: 18px;
}

There are two mindsets we can see this issue as:

it's okay to be explicit and have verbose rules in large classes
it's better to have multiple classes of lesser rules and to combine them later

If we were to use multiple classes, all the properties get merged in together and will create a new invisible class ruleset. But the behavior here is where it gets unreliable. It isn't clear which class will have it's properties preserved when you merge classes, so to avoid rule collision, this behavior should not be counted on when creating web pages. If you used both large_red and medium_red, what will be the value of font-size? It's not exactly clear.

Instead, it's easier to combine multiple classes that are good at doing easy tasks instead of doing large different roles in a class. Think about text coloring and decorations; you wouldn't want to define font size, color and decorations all inside one class. It would be more ideal to create separate classes that are easy to memorize while developing a page, then join those classes together.

.red_text { color: red; }
.yellow_text { color: yellow; }
.large_text { font-size: 24px; }
.sub_text { font-size: 10px; }
.underline { text-decoration: underline; }

While at first I think it seems silly that I'm using one-rule classes, it actually makes a difference in how I write HTML. I'm not constantly switching between <i><u><b><span style="color:x"> and instead using <span class="class1 class2"> instead. If I have a link I want to style, it's no different than how I style my actual text elements.

Add Conditional Rules

This one is going to be the last area I touch upon. In my case, I couldn't simply use an external library, but if I could, I might be keen to use Bootstrap or Tailwind instead. Responsive CSS designing is so dull and drab that I wouldn't wish it upon anyone realistically.

Media queries, the one way to check for certain conditions in CSS, is absolutely needed when it comes to making mobile-responsive elements. After all the careful consideration you may do for when picking out margin sizes, padding and indentation, goes out the window the minute your page is put on a phone. Suddenly you need to crank out all your elements to use the maximum space possible. You have to redeclare everything.

For instance, this will probably happen to you in the near future.

/* your carefully curated page, margins/padding and all */
#page {
    margin: 0 auto;
    padding: 10px;
    width: 50%;
    min-width: 600px;
}

/* now do it again but with way less */
@media only screen and (max-width: 600px) {
    #page {
    	margin: 0 auto;
    	width: 100%;
    }
}

The first part is a regular declaration block, while the second part is a conditional query to the media device to ask about certain properties. If the screen has a maximum width of 600, the new rules will be put in place instead, overriding previous definitions. So instead of including padding and a width of half the space, we instead tell our page to expand to the entire width, making sure to use as much screen space as possible.

This is the pain in the butt part because you have to carefully curate your elements and go through your entire web page a second time to ensure everything works properly. If you had ten elements for your page, you're going to have to create another ten rules to make them responsive, which means double the amount of effort you put in.

The only way you could try to avoid this is by using an additional overhead class to tag an element as a "mobile-friendly" kind of element. This is why composability is kind of cool sometimes.

/* empty because we don't want it to do anything normally */
.mobile100 {}

/* now write it's mobile query */
@media only screen and (max-width: 600px) {
    .mobile100 {
    	width: 100% !important;
    }
}

The !important tag can be used in this way to override all other definitions in case of problems with multiple rule definitions. This is how Bootstrap and other mobile-responsive frameworks out there kind of operate, sometimes with more intelligence through the use of JavaScript APIs, but primarily through these kinds of tricks. You can also make some other mobile helpers for various widths as well.

Optional: Use a CSS Compiler

This one is entirely optional depending on your project requirements, but a CSS compiler can go a long way in correcting mistakes for you, while also being able to macro-create some rules on your behalf. SASS, what I use on this website, can do a lot of heavy-lifting on your behalf if you let it, and comes with a wide array of extra functions you can utilize. Most importantly, it comes with variables, a huge upgrade.

Overall, responsive CSS is a hell of a journey to chug through, and these are some lessons I picked up along the way. Thanks for reading.

Ste5e's Site

Purging CA Certs

Updating Guix

Podman / Docker

Conclusions

New Looks

What's New

Reasons for Haunt

What's Next

Where Next

Further Reading

Linear Algebra

Infrastructure

Music and Code

Blog Changes

Road Ahead

Podman

What's In A Pod

Guix and Podman

Converting Services

Website

Pi-Hole

Immich

Home Assistant

Combining and Autostarting

Final Thoughts

Addendum: Hardware Changes

Netbook Guix

Junk Shopping

Revisiting Guix

Minimum Bootable System

Trimming Guix

Generating the Image

Deploying Remotely

Guix Keyring

Guix Deployment

Closing Out

Tablet Tale

Bootstrapping XP-Pen

Stracing

OpenTabletDriver

Packaging XP-Pen

Shared Object Maths

Impasse

Cleaning

Closing Out

Surface Linux

Problems

Replacement

Dumpster Diving

Surface

Kernel

Shopping

Setting Up

Synchronizing

Bluetooth

Hotspotting

Pics

Thoughts

Addendum: Snapdragon

MIDI Sequencing

Primer on MIDI

Arpeggiation Basics

Scheduling

Device Handling

Arpeggiator

User Interface

Musical Scales

User Inputs

Scheduler, Part II

Death

Final Touches

State

Visual Tracking Aid

Widget Approach

Visual Tracker Part II

Ticking State

Flushing Notes

Rendering

Layout