There are plenty of introductions to the Guix package manager out there, so this is none of them. I rather wanted to share two ideas I use in my own current setup that made Guix useful for me without touching the operating system underneath. The first one is a way to structure the machines user level configuration as a whole in small features that I define using scheme lisp. They can be combined in different ways for different machines. The second one is to define different project workspaces that only have exactly the tooling that they need. This will become more clear in just a minute.

For all that I run Guix Home on top of my regular linux distribution, currently Ubuntu 24.04 on my work laptop and Debian unstable on my personal MNT Reform¹. Which distribution it is does not matter much. This would be applicable to any linux system you have. The distribution keeps managing the base system: The kernel, drivers, the display stack and audio system. In addition to that I use the guix package manager to manage and provision everything in my home directory. That would be my user’s programs, dotfiles containing my programms configuration and in addition to that user level background services that run under my personal user’s permissions.

Okay wow, but I can handle my system without learning lisp, Ben!

Sure, but let me show you what got me excited to try this. First you can view my configuration some sort of a “backup” that can be applied to set up a new system. Say, when I set up a new machine I install guix, clone the repository and bootstrap the configuration from there. It installs all programms I need, my shell, my editor, my window manager config and my music daemon and all configurations for those. It’s configuration as code for my workstations.

Second, Guix is also a nice place to put small scripts, configs and tools. A package definition for a script or a tool of my own is only a few lines, and it installs like any other package. I can define the dependencies and have everything installed. Potentially if somebody else would be interested, I could provide packages to others through my guix channel, a special kind of git repository, as well. That makes writing own tools way more applicable than just having everything laying around.

Defining Features

Let’s dive a bit deeper. I split my whole configuration into what I call features. How does that look like? A feature is just a Guile module defined in lisp that exports two lists: the package names it needs and the services it sets up. Once they are defined, a host file, so the one that describes what goes onto a Machine (host), just defines the features that machine should have.

This is how the files are laid out in my repository, which lives on the guix config directory:

~/.config/guix/
└── modules/bbr/
    ├── hosts/
    │   ├── lt1550.scm     picks the features for my laptop
    │   └── reform.scm     picks the features for the Reform
    └── features/
        ├── base.scm
        ├── desktop.scm
        ├── music.scm      exports music-packages and music-services
        ├── music/
        │   └── mpd.conf   a plain config file, deployed as it is
        └── radio.scm

At evaluation time, guile finds a module by its name: (bbr features music) for example corresponds to the file bbr/features/music.scm. Configuration file templates for programs like mpd.conf are not modules, they just sit in a directory next to the feature that uses them.

As an example here is the first part of features/music.scm. It is a feature for my music player setup, and the package names defined in music-packages are the same ones you would pass to guix install:

(define-module (bbr features music)
  #:use-module (gnu home services)
  #:use-module (gnu home services shepherd)
  #:use-module (gnu packages mpd)
  #:use-module (guix gexp)
  #:export (music-packages music-services))

(define music-packages
  '("ncmpc" "mpd" "mpd-mpc"))

This defines my music player setup. mpd is the music player daemon, ncmpc a terminal client for it and mpd-mpc a small command line tool to control it. Importing and using the packages like this is enough to get them installed. In the file there are also definitions for how to run mpd as a shepherd service, as well as how and where do deploy mpd’s configuration to:

(define mpd-shepherd-service
  (shepherd-service
    (documentation "Run the Music Player Daemon.")
    (provision '(mpd))
    (modules '((shepherd support)))
    (start #~(make-forkexec-constructor
              (list #$(file-append mpd "/bin/mpd") "--no-daemon" "--stderr"
                    (string-append (getenv "HOME") "/.config/mpd/mpd.conf"))
              #:log-file (string-append %user-log-dir "/mpd.log")))
    (stop #~(make-kill-destructor))))

(define music-services
  (list (simple-service 'music-shepherd home-shepherd-service-type
                        (list mpd-shepherd-service))
        (simple-service 'music-files home-files-service-type
                        `((".config/mpd/mpd.conf" ,(local-file "music/mpd.conf"))))))

I’ve defined several other features like this. For example my emacs config, or my work workplaces. Next, how do I assign them to machines now?

Assigning Features to Hosts

A host can bundle the features together and later deploy them towards a linux machine. The Reform has fewer features than my laptop because I do not need work related stuff there, so it’s host file is shorter. Here you can see the main part of hosts/reform.scm, which lists the features of this machine:

(home-environment
  (packages (specifications->packages
              (append base-packages
                      emacs-packages
                      desktop-packages
                      music-packages
                      radio-packages)))
  (services (append base-services
                    music-services
                    desktop-services
                    emacs-services)))

A feature that needs a background process or a dotfile adds that by using a service extension. The music feature for example, runs the Music Player Daemon under Shepherd, the user-level service manager of Guix Home, and deploys its configuration file next to it. That is the what you saw in the second part of features/music.scm.

When I want my music setup on a new machine, I add music-packages and music-services to its host file. When I do not want it anymore, I remove those two names.

The deploy command points Guix at the directories with my modules, and the host file of the current machine is the one to apply. The second -L is the directory with the package definition of one of my own tools, more on that below:

guix home -L ~/.config/guix/modules -L ~/dev/archiving/packaging \
  reconfigure ~/.config/guix/modules/bbr/hosts/reform.scm

Packages that are not in Guix yet

A feature can also use packages that do not exist in Guix at all. For example my radio feature does that. It installs two small command line tools I have written and packaged myself.

• azi-qso-map generates azimuthal maps from the log files of my amateur radio activations, which I described in a previous post.

The package definition of azi-qso-map is published in my own Guix channel, a git repository that Guix pulls package definitions from. That can be wrapped in it’s own feature definition like this:

(define-module (bbr features radio)
  #:use-module (asciijungle packages azi-qso-map)
  #:export (radio-packages))

(define radio-packages
  '("azi-qso-map"))

For the host it makes no difference whether a package comes from Guix itself or from my own definitions. That concludes what I wanted to show regarding the feature composition.

Workspaces: one directory per customer

Another interesting part for my daily work is how I handle projects workspaces in this setup. Each client at work gets its own directory below ~/dev, for example ~/dev/client-a and ~/dev/client-b. All the repositories of that customer live inside. A single work feature deploys what each workspace needs, and a few things happen automatically as soon as I enter the directory.

The right git identity

Every customer has its own email address, SSH key and commit signing rules. Git can switch these based on the location of the repository:

[includeIf "gitdir:~/dev/client-a/**"]
    path = ~/dev/client-a/client-a.gitconfig

[includeIf "gitdir:~/dev/client-b/**"]
    path = ~/dev/client-b/client-b.gitconfig

Each included file sets user.email, the SSH key to use and whether commits are signed. This git config is managed by my work feature in guix.

Installing different Software Tools per Workspace

Customers do not agree on tooling. One project needs a certain Java and Python, another one needs a database client library, Node.js and Ansible on top. I do not want any of this installed globally, and I do not want to remember which version belongs where.

Each workspace gets a manifest, which is just a list of Guix packages:

(specifications->manifest
  '("openjdk@25.0.2:jdk"
    "python"))

Next to it Guix Home deploys an .envrc for direnv²:

use guix -m manifest.scm

JAVA_HOME="$(dirname "$(dirname "$(readlink -f "$(which java)")")")"
export JAVA_HOME

PATH_add bin

This enables that when I cd into ~/dev/client-a with a shell, direnv builds an environment from the manifest and puts those packages on my PATH. JAVA_HOME is derived from the Java that is now first on the PATH. When I leave the directory, everything is gone again. java in ~/dev/client-b is a different one than in ~/dev/client-a, and in my home directory there is none at all. A project that still needs an older Java only changes one line in the manifest of its workspace.

Installing small helper scripts

The last line of the .envrc, PATH_add bin, makes a bin directory of the workspace available. Home deploys small scripts there that only make sense for that customer. Two I use all the time are called vpn and db in almost every workspace, so these work the same way for each customer³.

The vpn script opens the tunnel to the customer’s network, simplified like this:

#!/bin/sh
cloudflared access tcp --hostname https://vpn.example.com --url localhost:6432

The db script then connects to the test database through that tunnel. It reads the credentials from the cluster and starts psql, either interactively or with a query given as an argument:

db "select count(*) from accounts"

In another workspace, db talks to a completely different kind of database with different tools, but I type the same thing. The knowledge about how to reach test data is part of the configuration instead of living only in my head or in an outdated wiki page.

Wiring it all up

All of this is declared in the work feature. It is one list of files to deploy:

(define work-services
  (list (simple-service 'work-files home-files-service-type
                        `(("dev/client-a/.envrc" ,(local-file "work/client-a.envrc"))
                          ("dev/client-a/manifest.scm" ,(local-file "work/client-a.manifest"))
                          ("dev/client-a/bin/vpn" ,(local-file "work/client-a-vpn" #:recursive? #t))
                          ("dev/client-a/bin/db" ,(local-file "work/client-a-db" #:recursive? #t))
                          ("dev/client-b/.envrc" ,(local-file "work/client-b.envrc"))
                          ("dev/client-b/manifest.scm" ,(local-file "work/client-b.manifest"))))))

The work feature is only part of my laptop’s host file. The Reform has no customer directories at all and therefore does not get any of this.

I really like this way of setting things up. It allows me to keep things separated and orientate myself quickly. It would even be possible to share configurations accross the team if more people want to use guix for their config setup.

Limitations and Challenges I’ve encountered

While setting this up and using it for more and more aspects of my system I came across a few challenges and also limitations of Guix.

First of all the performance of Guix is not blazingly fast. The first apply takes a long time. That is because there are several parts of the system that are compiled and all the dependencies have to be fetched onto the machine. On the MNT Reform it ran for several hours until everything was in place. Later changes are fast, because only what changed gets rebuilt but setting up initially just takes time.

Another thing is that I do not manage everything on my machine using Guix. Some tools I still manage from the distribution, those are for example parts of the desktop, the sound system, system tools etc. The distribution stays in charge of the base system. Moving to Guix OS is an option for the future but honestly the system works like this too. Of course there are also propietery tools that are not offered via guix. AWS cli, intellij and google chrome for example. Those have to still be installed and updated by hand.

The first rollout to another machine also lead to small surprises. A different version of a program in another distribution can mean a differently named setting in a config file for the desktop for example. That is why it makes sense to keep the system configurations alike as much as possible. My host files handle those differences with a parameter, but I only find them by trial and error. That being said it is awesome that guix can manage this after all. The two laptops that I provision run on different processor architectures (AMD 64 vs aarch64) and I was able to set everything up with only minor changes.

Conclusion

For me, the combination of features and workspaces is what makes Guix Home worth it. Features make my configuration modular with a machine definition that is just a short list of identifiers. Workspaces make my customer work tidy with the right git identity, the correct tools and the command line helpers that show up when I enter a disoctory and disappear when I leave again. To me that is a nice benefit which got me enjoying this setup at the moment. To be fair this can also be accomplished by just using direnv but without guix I find this tedious to pull off.

And I am aware that you can accomplish similar things with different tools. If you have used Nix, devenv or plain version managers for similar things, I’d be curious to learn about your setup and how your solution compares to this. And if you’d like to see more details about a part of this setup just let me know. As always I’m happy to get in touch. Cheers and thanks for taking the time.

Footnotes

I’ve been using tiling window managers for way over a decade now. As a long time user of Xmonad on Xorg I switched to sway to be able to use the more modern Wayland based Desktop. On Xmonad I loved redshift, a program that adjusts the color temperature of the display in order to have a warmer display with less blue light during at night time.

On sway there is an alternative: wlsunset. It works basically the same. It’s a background program that I started in my sway config on startup of the desktop session.

But I noticed an issue with this setup: when I lock the screen before sunset and the monitors get switched off while I’m away, the screens are still cold when I wake up my monitors. The wlsunset process is running, and its log even says setting temperature to 4000 K. But the tint is just not being visible.

Cause of the Problem

wlsunset only talks to the compositor when it wants to change the temperature. During sunset that is every few seconds but after dusk the warmest value is reached and no changes are applied anymore. If the outputs are powered off while the last steps happen, nothing ever applies the final value.

The fix I found

Restarting wlsunset applies the current temperature right away. So I wake it up whenever the screens come back on. I put the restart into a small script:

#!/bin/sh
pkill -x wlsunset
while pgrep -x wlsunset >/dev/null; do sleep 0.1; done
setsid -f wlsunset -T 5500 -t 4000 -l 53.5 -L 9.9

The loop waits until the old process is gone. Then I start a new instance of wlsunset with the desired config¹. If you’d want to use this for your setup just replace the coordinates with your own.

Killing a process and starting it again feels a bit dirty, but it is the best fix I found. If you can think of a better way I’d love to hear about that!

Then the script runs in two places: At startup it replaces the plain exec wlsunset line. Also it is called from the resume command of swayidle². This is what the part of my sway config looks like:

exec swayidle -w timeout 300 'swaymsg "output * dpms off"' resume 'swaymsg "output * dpms on"; ~/.config/sway/wlsunset-restart'
exec ~/.config/sway/wlsunset-restart

After 300 seconds without input the screens go off. On the next key press or mouse movement they come back on and wlsunset is restarted.

Bonus: Lock and switch off the screens with one key

While I was at it I tried to find a way to switch the screens off right after locking, because sometimes I know I’ll be away from the computer for longer and I don’t want to wait for the screen to switch off after the timeout.

swayidle enters its idle state immediately when it receives the SIGUSR1 signal. That results in the same dpms off command being issued as caused by the timeout. The resume command takes care of waking everything up again in any case. The trick here is that sway lets you bind a key differently depending on whether a screen locker is running or not. That way a binding with the --locked flag³ can be used while swaylock is active and the one without it in the other case. In the sway configuration file this looks like that:

set $locker 'swaylock -f -c 000000 --inside-color 000000b3 -K'
bindsym $mod+Shift+z exec $locker
bindsym --locked $mod+Shift+z exec pkill -USR1 -x swayidle

So what does that do? I can simply press $mod+Shift+z once to lock my screen. The Displays stay powered and would switch off after the 300 second timeout. But when I press it again the screens go off immediately. Any key press wakes them up with the tint intact. This is only one combination and I can even press it with one hand on my split keyboard. Neat!

Conclusion

wlsunset itself does nothing wrong here. It just has no way to find out that the screens were switched off in between. Restarting it on resume is a cheap workaround that works for me. Maybe you are struggling with the same configuration issues.

Do you have a different solution for it? How does your sway configuration look like? I’d love to hear about it.

Footnotes

As an amateur radio operator I like to do “parks on the air” activations and operate out in the wild. Setting up the portable radio station in a park and operating using my voice for single side band or morse code for cw operation is just a real joy for up to a few hours.

After some activations I post about the activation over on mastodon. For that I would snap one or two pictures that show the setup and the location. To give a deeper insight into the particular activation I wanted to top that up with a summary of the contacts. At first, I just included a screenshot of the map view of my logging program (Wavelog¹), like in this early post. That way people can see at a glance how many contacts I had, how far I reached and which countries I got in contact with.

That already gave an impression, but I wanted to convey even more information. The reader should get a better feeling on how far the contacts really were apart. They should get an idea of how many contacts were done on which band. To bring it even further, I wanted to show how my QSO² rates were on certain transmission modes or bands. All in all that should really give a rich picture how the radio conditions were on that day.

So I came up with the idea of creating my own unique map design for sharing pictures of maps for amateur radio activations, like this one.

I was looking for libraries that I could use to accomplish what I wanted and settled for the Python ecosystem. The cartopy library provides an easy interface to render maps using different projections. It also comes with cartopy.crs.AzimuthalEquidistant, a map projection other than the Mercator projection that we all are used to from for example online map services.

Why Azimuthal Equidistant?

It works like this: Imagine you’re standing in the middle of the map at your location, looking down. You flatten everything you see into a circle around you. On the map you are at the center. Distances in every direction from you are the same. If a contact is 2,000 km away in real life, it’s exactly twice as far from the center as one at 1,000 km. Lastly, if someone is northeast of you in reality, they’re northeast on the map too. That way you exactly see the contact on a line at the correct degree from your antenna. This is great for directional antennas as you will be able to see, for example, if all of your contacts are in a certain direction. It is pretty much what you would see on a radar screen.

There are some downsides as well. Shapes and areas get increasingly distorted the further you go from the center. Continents near the edges look stretched. The resulting map might look unfamiliar. For what I want to show in my map that projection is really nice though.

Cartopy supports taking the continent outlines from a public domain dataset and map it towards that projection. Those outlines are vector graphics so it is possible to export an .svg file in the end which scales nicely.

Putting Contacts on the Map

Next I wanted to add the stations as colored dots onto the map. For that I take each contact’s maidenhead locator³ from the ADIF⁴ logfile of the activation. That gets converted to geographic coordinates which then can be put on the map by applying the projection. I can add additional information from the contact like for example the band that contact has been logged on and, depending on that, color the dot on the map accordingly.

Graphing QSO Rates

To be able to see how the activation progressed I added a graph showing logged contacts per minute in 5 minute windows. It highlights the different bands and modes for easy comparison. The plot is done with matplotlib, a Python library that allows to plot different kind of data graphs.

The graph together with the map really tells the story of the activation. In the example shown above you’ll be able to analyze it like this. First you take a look at the map. You see contacts across Europe and some on the west coast of the US. That is not bad for a park activation so you dive deeper. Taking a look at the QSO rate you’ll see that all of the contacts were worked in CW, which is Morse code. There are three bands used in this activation. 17m, 15m and finally 10m. You see that the band changes occurred each time after a drop in the QSO rate. The activator worked each band until there were no more stations calling them, then switched to another band. You see they operated for about 1 and a half hours from 15:45 until about 17:30. See what I mean? A powerful visualization conveying a lot of information.

Try it yourself

I have received requests from multiple people asking how I created my shared maps so I have published my map tool as azi-qso-map on codeberg. Finally you’ll be able to just apply it directly to the .adif files of your last activation! The code is available under the MIT license. If you have feedback or want to report issues you can do so on codeberg. Installation and usage is described in the README.md on the project page. I would love to hear from you if you try it out. You can reach out via Mastodon.

Footnotes

When I started my amateur radio classes in 2022 and started to hang out with fellow radio operators, it did not take long before I heard people talk about using telegraphy with Morse code to communicate via amateur radio. At first I wondered, why would people invest a substantial amount of time learning to send and decode Morse signals without any aid of technical devices?

In fact, there are several reasons why people still use this first of all modes. One reason is that apparently the signal to noise ratio is higher when compared to a voice signal in single sideband (SSB). That means your signal can reach stations further away when sent with the same power using Morse code. Quite a good reason. Secondly, the mode is very flexible. You can have very short contest style QSOs¹ but also use it for conversational contacts.

The Course

There are even more reasons to learn and use it, so I was interested in learning CW² and participated in my local club’s CW course in 2024. The teacher was Edmund Ramm, DJ6UX, a very good operator with many years of operation as well as teaching practice. He has taught many operators over the years and his course evolved from in person to online in the recent years.

We used Mumble³ to receive and send the code along with voice over the internet. We would meet every Friday evening online for two hours and receive groups of five characters out of the already learned characters. We had to copy them down to paper and read them back. If we made a mistake we would get the same group of five characters sent again until we got it right. If the group was correct we would receive an `r` and the next student continued with the next group of five.

Every week there were new characters joining in, so we had to prepare by practicing daily for at least 15 minutes. This is the list of new characters in each week:

KW40: A,Q,S
KW41: T,2
KW42: C,O
KW43: D,5,/
KW44: I,R,9
KW45: G,X
KW46: F,4
KW47: repeat
KW48: N,U,7
KW49: H,,
KW50: =,.
KW51: B,K
KW52: P,3
KW01: M,Y
KW02: J,W,Z
KW03: 1,6
KW04: 8,?

Only once we had learned to receive all the signs we would start sending them. Every participant had a Junker Straight key and would send groups of 5 chars off of a sheet of paper to the others just like Eddi had done before. A pretty straightforward and effective method. But one that requires patience and persistence. Forget or neglect your practice in one week and you’ll have a hard time in the upcoming weeks. By learning in a group I felt a motivation that helped staying on board. And after some time I really wanted to make it and be able to use CW for radio contacts as I had already invested quite some time. In total, the course took a whole year. After we were done learning to send and receive we invested the remaining weeks to practice gaining more speed (up to 15wpm) and to acquire rudimentary head copy⁴ skills. A year might seem very long but all this takes a lot of time and practice.

The most challenging part for me was to allocate time for the course on Friday evenings. I was used to using that time to socialize, meet friends, go out and so on. It was very challenging to always stay strong and prioritize the cw course above all else. Some weeks I struggled. It felt like with every new character I had to also relearn the others to some degree. But slowly I gained more and more confidence. To me it felt most effective to spread the daily practice time across the day. Having short sessions in the morning, lunch time, and evening yielded better results than having one longer session.

Making First Contact

By the end of January 2025, a few others from the course and I tried to see if we could receive each others signals on the 2m band within Hamburg. What started as a test became my first Morse code radio contact, my first CW QSO. Many people have very vivid memory of their first CW QSO. To me I was a bit surprised but I knew what to expect so it felt more like the natural next step than a big revolution. It was not long before I started to also have my first QSOs on the short wave frequencies.

First Park Activation

Two weeks after that, I went for my first Parks on the Air⁵ activation in CW. 11 QSOs with a relative straightforward exchange, however when calling CQ you have to be able to directly respond to an unknown callsign which was quite challenging at first. You do not have to be perfect though, just get the first few letters and add a question mark and they’ll send it again for you. To me this first small POTA pileup in CW and handling it was the real tipping point. In the first QSO I knew the other operator and it was kind of like online but having natural static and fading on the signals on the short wave really was something totally new. From this moment on I just wanted to work more stations from various countries. The time invested had paid off.

Field Day

We continued with the course and during the first weekend in June a few operators from the local club and I participated in the IARU Region 1 field day in CW as DL0AT/P. It was quite hard for me to copy at the higher speeds. The weather was cold and rainy. We had issues with too much wind on our antenna. We organized in teams of two operators and took shifts of one to three hours each. It was good to have another operator listening next to me when I was having issues with the call signs. Together as a team we managed to get a few hundred QSOs into the log.

The Exam

In the summer on August 11th I took my official Morse code exam at the Bundesnetzagentur⁶. This is an optional added exam that officially certifies that the operator is able to send and receive Morse code at 12 words per minute. An operator that passed the exam would get a remark in his official license document. To officially prove that you are capable of Morse code operation is required to operate in some countries. However, I took that exam mainly just for me. Having an official verification to be a Morse code operator is a nice feeling.

Contest Operation from Home

In October that year I then worked my first contest from home. The Worked All Germany Contest wich yielded quite some QSOs despite the local noise at my station. For this I had to set up some contest logging software and configure the interface with my transceiver so the logbook could pull the frequency and band information from there. My strategy back then: I only worked search and pounce⁷ style. That way I had enough time to copy down the callsign of the station running on the frequency and also the sequential number they were giving out. That was challenging because of heavy QRM⁸ and the higher speeds I was not yet used to at the time.

Since then, I have activated many parks, taken part in multiple contests and had a lot of conversational contacts using Morse code. My speed improved to about 18 words per minute. And not long ago I got confirmation of my 100th DXCC⁹ entry worked in CW according to Logbook of The World¹⁰.

Start your own CW Endeavor

If you want to get into CW using Morse code and wonder if it is worth the effort I’d encourage you it is really fun and nerdy and lets you experience the early days of radio communication. You can build your own CW transceiver, get nice results using only a few watts of transmission power. You can reach more amateur radio operators as there are a few that only ever work in cw.

If you cannot or do not want to participate in a course like mine, the method that I used to learn the code can also be followed all by yourself. There are awesome tools like the website Learn CW Online that are available for free. For me however it was crucial to have a group to learn with. The regular classes at a certain point in time made it important to practice every day and in the end kept me going. If you can I’d recommend you also join a group or course to keep up the motivation over several months. In the end it will be worth it. My Morse buddy Peter (DL6PL) who participated in the same course as me also really motivated me to keep at it. The community and the shared experiences are what amateur radio is all about. It feels nice to share the challenge of learning Morse code with others. CW operators are forming another layer of community inside of the hobby. All of them have invested the time to get to the point where they can use the skill to have contacts with others. The conversation partners value that which makes it much more valuable than contacts in many other operation modes.

Looking back, now I truly understand why it is worth spending a substantial amount of time learning how to use Morse code for amateur radio contacts. Now go and start your CW endeavor.

Footnotes

After participating in several amateur radio contests in telegraphy last year, I wanted to also take part in a phone contest. The amateur radio operators call that SSB which stands for single sideband. It is the most common analogue voice transmission mode. I still consider myself to be a contest rookie - this was only my sixth contest and the first time operating SSB during a major contest.

For this contest I was able to use the big station of my local amateur radio club. That enabled me to work in the High Power category and use the three element yagi for the 20, 15 and 10 meter bands. For me it is an exciting station as I cannot really raise antennas at home.

My Goals for the Contest

Before the contest I sat down and thought about my goals for the contest. I wanted to only work a couple of hours but make the best out of it. For the Two Modes Award from the DIG I was lacking SSB contacts with south america and oceania that are confirmed by QSL Cards¹ or via Logbook of the World². So I set it as my goal to get at least one confirmation from both of these continents. Confirmations from stations in countries I have not yet worked, so called DXCCs³, would also be nice but were not a primary goal. As for my placement in the contest the goal is to not be the last station in my category.

Making a Plan for the Weekend

Having these goals in mind I planned my contest weekend. I’d start on Saturday after noon, take the bus to the club station and setup my Linux logging PC. During the afternoon I would try to work south America. In the evening I would try working stations on the low bands, knowing that the low band antennas are not the strength of the station. Then I’d call it a day go home and get some sleep.

On Sunday I would get up early to start operation around 6:00 UTC and try to catch Oceania and some asia. Around noon I’d call it a day and have the rest of the day to recover.

Report from Saturday

I arrived as planned and had the station ready to log the first QSO⁴ at 12:23 UTC⁵. After checking the first QSOs on 40m I jumped onto the 15m band turned the beam to south America and directly worked P45A in Aruba.

At 12:57 UTC, I switched to 10m, turned the beam to 240°, and immediately heard PY2KQ calling. That was the moment - South America was wide open. Over the following 36 minutes, the band delivered: PY2KQ, PY2MP, PY2WB, and PY2UD from Brazil, then LU2FE from Argentina. I was surprised by how good the propagation on 10m was. With a solar flux index of over 160 after it had been below 150 for weeks the conditions were awesome. On 10m there was barely any QRM noticeable. Times like these are really the fun of being an amateur radio operator.

After I had worked the bands front to back and had my goal for south America already achieved, I took a look to the cluster. I tried to spot some stations from new to me DXCCs. Cape Verde was exceptionally active with three different D4 stations worked. Indonesia (YB) gave me a backup for Oceania, just in case I would not be able to work Australia on Sunday.

As the afternoon wore on, the 20m band gave me multiple north American stations but also Europe and even China and Indonesia. As I was sure at least one of the Indonesian stations would confirm the contact with QSL I had my goals already achieved on Saturday evening!

As darkness fell, I moved down to 40m and then 80m. This turned out to be the fastest part of the contest - 80m was absolutely packed with European stations, and I logged 33 contacts in 40 minutes. There was much QRM⁶ on 40 and 80 though which made it really tough to hear the stations.

With all my goals achieved and a bunch of new DXCCs in the log, I packed up and headed home at 22:09 UTC for a short night’s sleep. The clock would jump forward an hour as we slid into summer time and that would cost me another hour of sleep.

Early Start on Sunday

On Sunday I jumped on the early bus and logged my first QSO at 05:48 UTC. That day started much slower as there were only a few stations calling in the morning. I checked 15m but went back to 20 as there was not much going on at that early time. After some time I was able to work south Korea, Kyrgyzstan, Kazakhstan, and Japan.

The Highlight of the Contest

Still no Australia so I had the idea to try 20m on the long path. My beam was pointed west at 290° and heard VK6NC loud and clear. I called, and they came back!

This was my moment of the weekend. I finally had Australia in the log. It was my 234th QSO of the contest, and probably the most satisfying. Taking advantage of them being in the grayline during their sun set, the contact was made on the long path - roughly 26,000 kilometers going west through the Atlantic and Americas, rather than the shorter 14,000 km path through Asia.

After this contact I had to take a short break. There are not many moments like this.

For the final stretch, I returned to 10m to sweep up any remaining remote stations. The last 40 minutes yielded some fantastic catches: Namibia, Bangladesh, and HC8NA from the Galapagos Islands!

After that I was extremely exhausted and called it a day at 10:37 UTC.

Conclusion

Next time I would probably not start this early on Sunday as the first hours were a bit of a slow start. I could have instead used some more night time on the low bands.

I would not have thought that this contest would be that successful that made me really happy. Not only did I work the desired contacts with south America and Oceania, but I also managed to work 27 new DXCCs in SSB and 13 new ones regarding all modes. As we are on the decline of Solar Cycle #25 it was awesome to get to work a contest with stellar conditions once more. The 20 hours of operation time during the contest were more than expected. They yielded 256 QSOs in total. And I can tell that by now I have gotten the missing QSLs so I’ll get my “Two Modes Award”! In the end I’m really thankful for the experience of working an SSB contest at a very capable station.

Footnotes

Everyone has personal data that is worth archiving. Data that does not change, that is not accessed all the time but data that you want to access in the future. There are different solutions to this problem. Here I’ll show you how I archive my photo collection.

Over the years I have collected quite a number of personal photos. I wanted a way to store them safely to have them available for many years to come. Most people use a photo service that promises to store pictures for a monthly fee in their cloud. At first their prices are quite appealing but over the time as you use more and more storage and as you have to pay each month it becomes more and more unattractive. That is the reason I want to store the data on physical media in my control.

Physical Storage Options

There are different options for storing larger amounts of Data on physical media. For me it is important to store the data for a long time without having to copy it, refresh it or renew the physical media every now and then. I want the data to be stored offline, that means I do not want to use for example an network attached storage device that has to be plugged in. The last criterion of importance to me was a format that has sufficient relevance, so there is a high likelihood that devices capable of reading it will be available in the future.

M-Disk Standard

For now I went with M-Disk. That is a disk technology compatible with the DVD or Blu-ray standard. They use a glassy carbon material instead of the organic material that is commonly used in DVDs and Blu-rays. That’s why they claim they have a longer lifetime of several hundred years. Of course there is no way to find out how long they live since they only have been around for a few years by now. However there have been several tests by different entities that suggest a longer lifetime than regular blu-rays.

M-Disks exist in different storage sizes: DVD with 4,7 GB, Blu-ray with 25 GB and Blu-ray XL with up to 128 GB of storage. Depending on the amount of data to archive that may be a lot of disks. I went with the regular Blu-ray ones since it appears to be a good tradeoff between large enough storage and a widespread format that can be read by a lot of drives. For the drives I won’t be recommending any devices. I’ve got an external USB Blu-ray writer that is capable of writing M-Disks.

A Thought about Encryption

The content on a disc could be encrypted using different methods. I thought about this but decided against it. My personal photo collection is insensitive enough to be readable by anyone who can get their hands on the disks. Access Keys also need to be securely stored in a way that is different than the disks. If the keys are lost there is no way to restore the archive. That’s why for now I decided against encrypting the data on my disks.

What is on a Disk

My disks currently follow a straight forward folder structure.

2026_PHOTOS_2022_0812
   ├── README.md
   ├── VERIFICATION.md
   ├── manifest.sha256
   ├── redundancy/
   └── CONTENT/

The readme file contains information about the disk. This is primarily for myself if I’m accessing one of the discs in the future. Each disk gets a disk id that I assign to it which also goes into the readme. Here is an example of a readme file in one of my discs:

# Archive Disk
Disc ID: 2026_PHOTOS_2022_0812

This data belongs to Benjamin Brunzel
E-Mail: b***@***.de
*Address goes here*

## Metadata
Created: 2026-01-09
Category: Photos
Content:
 - Photos from 2022 August to December
 - Includes hamburg, radtour
Size: 20GB

## Privacy / Usage Notice
These data contain personal and private content (photos, videos, documents).
They are intended for personal use only.

**It is strictly prohibited to copy, distribute, publish, or otherwise use the data without the explicit permission of the owners.**

The verification file describes the verification process that can be used to check the integrity of the disk. This process will be described later.

The manifest file contains sha256 checksums for every file in the CONTENT folder. They allow for checking if a file can be correctly read from the archive. This is checked during verification.

The redundancy folder contains redundant parity data per top level folder in CONTENT This allows to reconstruct the data in case of losing data within the archive due to unreadable sectors. In my configuration I use 15% of redundancy. This means I can recover from up to 15% lost sectors. So if a disk only has 86% of readable sectors I’m theoretically able to restore the files without errors.

Creating a Disk

To create a disk I’m selecting which data goes onto it. I copy it to a directory on my linux machine that I want to use to burn the data to the disk. In that directory I create the target file structure as it should be found on the resulting disk. I have to make sure not to exceed 20GB of file size in CONTENT because I will add the 15% of redundancy that has to be accounted for.

After I’m fine with the contents that should go on my disk I create the sha256 checksums with the following command on the shell:

find ./CONTENT -type f -print0 | sort -z | xargs -0 sha256sum > manifest.sha256

Next, I’m generating the redundancy using the par2 program and move those files into the redundancy directory. It will generate multiple files for each folder in the top level of the CONTENT directory.

mkdir redundancy

for dir in CONTENT/*/; do
  if [ -d "$dir" ]; then
    par2 create -r15 -s4194304 "$(basename "$dir").par2" $(find "$dir" -type f)
  fi
done

mv *.par2 redundancy

Now the directory is ready to generate the universal disk format file system image that can be used to burn the resulting disk. We use this command:

mkisofs -udf -r -V 2026_PHOTOS_2022_0812 -o ../2026_PHOTOS_2022_0812.iso .

You should check the filesize of the resulting iso file. It must not exceed 25 GB to fit onto a blu-ray. Finally, I write the image onto a M-Disk using the growisofs command. Make sure you have your drive attached and an empty disk inserted. We will limit the writing speed to 4x which is reported to work better when writing M-Disk.

growisofs -dvd-compat -speed=4 -Z /dev/sr0=2026_PHOTOS_2022_0812.iso

Once this succeeds the data has been written to the disk. Now it’s time to verify the integrity of the disk we just created.

Verification of a Disk

This procedure will be conducted directly after creation of a new disk to make sure that the data on it can be relied on. After that it is recommended to verify stored disks every so often to ensure data remains to be readable. If you experience unreadable sectors you should still be able to restore all the data using the par2 parity data.

There are multiple steps to verification. First we’ll read all sectors of the disk printing an error in case there are any broken sectors.

dd if=/dev/sr0 of=/dev/null bs=2048 conv=noerror,sync status=progress

In case of an error it would look like this: dd: error reading '/dev/sr0': Input/output error Note that we are using the block size of 2048 as this is the standard block size for optical media.

If that command runs without any issues we continue with checking the checksums from the manifest file we’ve created earlier.

sudo mount /dev/sr0 /mnt
cd /mnt
sha256sum -c manifest.sha256 2>&1 | grep -v ': OK$'

This command checks all the files for their checksum in the manifest file. Only files that are not OK will be printed. If there is no output from this command everything is fine and the verification was successful. This process should be done every few years. I write down the verification result with date in the booklet of the disks enclosure. I’ve created over twenty disks by now and so far none of them had any issues during creation and verification.

Storage of the Disks

The Disks should be stored in a dark and dry environment in vertical orientation. I write each iso twice and store the disks in two different locations.

Let’s see how long this archive will hold up for me. So far all disks were without any errors. It is worth noting that writing disks and also the verification takes quite some time. If you have a lot of disks this becomes unbearable. Probably it would be an option to look into Blu-ray XL disks for larger data sets. For my personal archive I’ve not had any issue yet.

Are you archiving data? What is your approach? I’d like to hear about it.

The web browser is one of the most important programs in my day-to-day work on the computer. I want a browser setup that addresses a couple of goals. First I’d like to be independent of large corporations that try to observe my browsing behaviour for their profit. That is why I’d rather use an open source implementation instead of the various chrome based browsers. Since the servo browser¹ is not there yet², a mozilla firefox based browser is the most viable option as of today. Because Mozilla has started to put more and more questionable features into the firefox main release line, I was looking to try one of the open source forks promising more privacy. I need a reasonable tradeoff between privacy and ease of use. That is why I didn’t went with the tor browser providing the best privacy of the alternatives I looked at. I settled with the LibreWolf browser.

The tiling window manager on my linux distribution can be completely managed from the keyboard without having to use the mouse. That is why I’d like to have improved keybindings for my browser as well. Since I’m using vim bindings for many of my other programs as well a vi plugin serves me very well in this regard. I added the firefox extension Tridactyl to LibreWolf.

Finally I also wanted to do some changes to the UI of the browser so the space used for the actual content of a webpage is maximized and it is not displaying active UI elements that I do not use.

The result is a quite clean web browsing experience as can be seen on the screenshot above. It works nicely on large screens as well as smaller tiled areas as can be seen in the example.

How to set this up

I’m describing how to install the setup here so I can remember when moving to a different installation. First you have to install the LibreWolf browser on your linux machine. The complete setup should in my eyes also work on MacOS or even Windows machines even though I have never tested it myself. The LibreWolf website describes the installation for the various operating systems.

After that you have to install the Tridactyl browser extension through the extension management in about:addons. Open that page in your browser, search for the plugin and apply it. Once that is done I have opted for the default dark theme. This can be selected in about:preferences. Next I wanted to hide the huge tab bar on the top of the browser. Tridactyle offers a tab list by pressing b, but I found that I do not use nearly as many open tabs at a time now that I use this extension³. This can be achieved by using a userChrome.css file⁴. That file has to go into your browser profile directory which is located in your home config directory.

/* asciijungles's userChrome.css
 * write to ~/.config/librewolf/librewolf/{yourProfile}/chrome/userChrome.css
 */

/* hide header tab bar */
#TabsToolbar {
    visibility: collapse !important;
}
/* hide huge sidebar header*/
#sidebar-header, #search-box {
    display: none!important;
}
#sidebar-header {
    display: none !important;
}

/* minimize sidebar splitter */
#sidebar-splitter {
    background-color: black!important;
    width: 1px!important;
    border: 0px!important;
}

After that you have to enable usage of the userChrome file in about:config. On that page you are able to set values for configuration properties. Serch for the following config and toggle its value to be true.

toolkit.legacyUserProfileCustomizations.stylesheets = true

The changes apply after a restart of the program.

Last quick thing: I like to cycle through the open tabs with Ctrl + Tab. I use the setting “Activate Ctrl+Tab cycles through tabs in recently used order” that can be set in about:preferences. That way I can always jump back to the previous tab instead of tabbing through the whole list. Try it, it’s nice.

That is all you have to do to clone my setup. Enjoy.

What I wish for

I’m very happy with the setup as is at the moment. But there is one thing that I would really love to have. The browser supports multiple profiles with separate histories and shared sessions between tabs. In order to change profiles you have to start a different instance of the browser program. At work I have different profiles for different customers I work for and can be logged in to several web applications using OpenID Connect. I have to run two browsers and switch between them on different workspaces. It works but, I think it would be awesome to be able to use different profiles per tab in the same session. I imagine to be able to have a keybinding “open in profile 2” that opens in a profile other than my default.

What do you think? How are you using your browser? I’d love to hear about it. Please get in touch with me on the fediverse.

Footnotes

The CCC’s Chaos Communication Congress of is an event I’m always looking forward to. It’s the annual conference of the German and international sub culture around computers and the culture surrounding it. I’ve attended all four days and want to share my key takeaways as well as what I observed to be the major topics of 2025

Digital sovereignty

The global political balance is shifting. Not just in the United States, populist movements are gaining traction. Relying on big tech vendors and trusting them with our data and using their infrastructure for critical communication has become a liability.

On the congress there were multiple talks but also self organized sessions around what options we have, and how to move away from IT services under direct control of other parties. Using selfhosted open source alternatives run on hardware run and controlled by someone you trust is the proposed solution.

The community proclaimed the Digital Independence Day, a distributed event where people get help with moving their digital infrastructure towards self hosted alternatives. The talk Die Känguru-Rebellion: Digital Independence Day introduced the idea and gave a motivation on why it is important.

In addition to that vendors are selling devices that come with operating systems in their control and it is getting harder to write and distribute programs for them that are not approved by them. This has been discussed in the talk A post-American, enshittification-resistant internet.

European citizens are customers and give a lot of power to the vendors and operators of critical digital infrastructure. It is on us to minimize risks of being held to ransom.

Gen- and Agentic AI

The rise of Artificial Intelligence powered features in software has also been addressed by a couple of talks and sessions. In AI Agent, AI Spy the people behind the signal messenger talked about a new feature within the windows operating system currently in beta testing. Windows recall is a function that is taking a screenshot of the whole desktop every couple of seconds. These screenshots are then run through a text detection LLM run locally to extract information from the text displayed on the screen and store them in a local file for later usage by copilot.

According to security researchers this is extremely dangerous because it is creating a honeypot of very sensible data that also includes information that has been end to end encrypted because that encryption is terminated on the end device.

Features like that put users at extreme risk. In addition to that such features could be used to find out extreme sensitive information for example about employees.

Another talk went though the possible ways to attack software engineers using coding agent tools by doing prompt injection attacks: Agentic ProbLLMs: Exploiting AI Computer-Use and Coding Agents.

Because todays Large Language Models (LLMs) do not have a separation of input data into context and commands, these issues will not be fixed but only mitigations will make it harder to exploit these design flaws. Currently, they cannot be prevented completely.

Hardware Security

Over the years security researchers have demonstrated various attacks on the hardware of computer systems like the CPU or the RAM. This years congress demonstrated that even though vendors invested in mitigations various of these attacks were proven to be exploitable in real world scenarios. The talk Rowhammer in the Wild: Large-Scale Insights from FlippyR.AM evaluated a field study were various people ran tests on their system. These studies showed that a lot of systems can be targeted by rowhammer attacks. Also Spectre in the real world: Leaking your private data from the cloud with CPU vulnerabilities showed that it was possible to extract information from other customers CPU processes on a public cloud provider. In addition to that the talk Bluetooth Headphone Jacking: A Key to Your Phone showed that insecure bluetooth devices that can act as an input device can be used to attack a computer paired to them.

Rise of far right politics

The last topic that I want to point out is the rise of far right political actors in a lot of western democracies. I especially liked the talk Gegenmacht - Best of Informationsfreiheit which pointed out the importance of the Informationsfreiheitsgesetz, Germanys Freedom of Information Act, which allows civil society actors to use the courts to have information made publicly available by the government. This is effective to control a government and fight corruption.

Call to action

The chaos communication congress is a very important event with talks of outstanding quality. They point out techological and social hotspots. It is on us to have a look, discuss and address them. The event shows that technology always has a social impact on all of us. As people working with software it is our responsibility to keep that in mind when working on software and digital infrastructure. All of the talks from 39C3 are being published on the media.ccc.de portal. I encourage you to take a peek into the recordings.

Hexagonal Architecture and GenAI: A good Combination for Modern Software Development

Introduction

Some time ago, I had the pleasure of leading a workshop on “Hexagonal Architecture and AI Tools for Modern Software Development.” For those who couldn’t attend, I’d like to summarize the key insights here – particularly why Hexagonal Architecture harmonizes so well with GenAI tooling.

Hexagonal Architecture, also known as Ports-and-Adapters Architecture, is an architectural pattern that emphasizes separation of concerns. Without going too deep into details: it’s about isolating core business logic (the “Domain”) from external systems and technical details by using clearly defined interfaces (“Ports”) and interchangeable implementations (“Adapters”).

But why is this architecture also particularly advantageous when working with GenAI tools like rooCode? That’s what I’d like to explain in this post.

Benefits of Hexagonal Architecture for GenAI Development

Clear Separation of Concerns

Like human developers, AI also benefits from clear context:

• The distinct structure shows where specific code can be found. This is ideally already evident from the directory listing with descriptive filenames.
• The business logic is well-described in the domain with strong types
• AI can focus on specific parts without needing to understand the entire system. That keeps the context small.
• Changes to one part don’t affect others (reducing the risk of AI-introduced errors)

Directory Structure

Let’s have a look at what an example of the directory structure might look like:

src/
├── domain/
│   ├── model/     # Domain entities and value objects
│   ├── service/   # Domain business logic
│   └── ports/
│       ├── in/    # Input ports (application APIs)
│       └── out/   # Output ports (repositories, etc.)
├── adapters/
│   ├── in/        # Input adapters (controllers, UI, etc.)
│   └── out/       # Output adapters (database, external services)

That is a standard directory structure of a ports and adapters architecture that is also known to most coding models. That way the agent natuarly knows where to look for files. File names are also very important they should use the same terms that are used in the business logic.

Well-defined Interfaces (Ports)

Ports are clearly defined interfaces that help AI, and also humans, understand expected behavior. Type safety guides AI to generate compatible code. That’s why I encourage to use a strongly typed Language and use expressive types like Domain Entities or Enums over basic types like e.g. String.

Interface types serve as documentation for AI. They define the building blocks that are supposed to be used to build new functionality. When using Ports it’s easy to break development tasks down into manageable tasks. You can first write the business logic of a new feature against new or existing ports, and then generate the adapter implementations in a second step.

Here’s an example from our workshop example codebase of how a port definition might look like:

// Example: Database interface
export interface Database extends Adapter {
    findById(id: MMSI): Vessel | undefined
    saveVessel(vessel: Vessel): void
    findAllVessels(): Vessel[];
    savePosition(id: MMSI, position: Position): void
    findHistory(id: MMSI): VesselHistory | undefined
}

With this clear interface definition, AI knows exactly which operations are available, which parameters are expected, and which return types to expect. This makes it much easier for AI to generate correct code.

Swappable and Mockable Implementations (Adapters)

Adapters are implementations of ports that connect the domain with external systems. They may be swapped with other implementations or mocks in tests. This interchangeability offers several advantages for working with GenAI tools:

AI-generated code can be tested in isolationk. New adapters can be developed in parallel with existing ones to ensure backward compatibility. So if you ask AI to implement a new adapter, you can test it without affecting the rest of the system. If it doesn’t work as expected, you can simply revert to the previous adapter. In our definition functional external dependencies may only be used in Adapter implementations. That way we ensure that our business logic stays pure. This can also be enforced by architecture tests.

Practical Example: Vessel Tracking Application

In our workshop, we worked with an application for tracking vessel positions. This application tracks ships via AIS (Automatic Identification System), stores position data and vessel information, and displays current positions and history on a map.

In this application:

• The Domain contains the core business logic and entities such as Vessel, Position, and VesselHistory
• The Ports define interfaces such as DataAccessPort, AisPort, and Database
• The Adapters implement these interfaces to connect the domain with external systems, such as WebAppAdapter, MqttAdapter, and InMemoryDatabase

This clear structure makes it easier for GenAI tools to understand and extend the application.

Practical Workflow with GenAI Tools

During the workshop, we developed a practical workflow for working with GenAI tools in a hexagonal architecture. First, create an implementation plan, then implement this plan step by step and Continuously test and refine the outcome.

Division of Labor Between Developers and AI

You can take over yourself whenever you see fit. One strategy might be:

1. You as a developer define the domain and ports
2. AI implements business functions on the defined types
3. You define testcases and let the test code generate
4. AI creates adapter implementations

That way you stay in control and there is at now point in time too much code generated so that it would be too much to review.

Test-Driven Development

Hexagonal architecture nicely supports a test-driven approach. You can define test cases for the expected functionality in prosa text. Then you let the AI implement the code that fulfills these tests. Because of the Architecture you can always test implementations in isolation before integrating them into the overall system. You can mock your Ports for extensive coverage of your business logic. That way you’ll get a good confidence that the resulting system does what it should. RooCode can run unit and architecture tests independently and thus correct errors directly from within the chat.

Conclusion

In my eyes Hexagonal Architecture provides an ideal foundation for working with GenAI tools like rooCode:

• Clear separation of concerns allows AI to understand and modify specific parts of the code without affecting others.
• Well-defined interfaces make it easier for AI to understand expected behavior and generate compatible code.
• Swappable implementations enable testing AI-generated code in isolation before it’s integrated.

By applying this architecture, development teams can use GenAI tools more effectively to accelerate development, reduce errors, and improve code quality. The overhead of implementing abstraction layers is not such a burden when using GenAI tools. The clear boundaries and contracts provided by the architecture help AI generate correct and compatible code, while the isolation of components reduces the risk of AI-introduced errors affecting the entire system. Thus your code base becomes manageable even over lots of iterations.

I’d like to hear about your expeciences. Did you try something similar?

When working with containerized applications in AWS ECS, you might encounter file permission issues, especially when implementing security best practices. Recently, I had to debug file permissions in an AWS ECS task with the following requirements:

• The process within the container must run as an unprivileged user
• The root file system must be mounted as read-only
• The process must be able to write to the /tmp directory
• The contents of the /tmp directory should be cleared on container start
• The container runs in AWS ECS with Fargate backend

Me and the team tried multiple approaches. Involving EFS and S3 as volumes meant we had to implement the clearing the /tmp directory on container start. Defining and mounting an ephemeral storage volume caused trouble because it was mounted as the root user so it cannot be used by the unprivileged user. Then I stumbled accross a solution that was posted in a subreddit¹. They suggested to define a Docker volume in the Dockerfile and have ECS create an ephemeral storage mount automatically.

VOLUME /tmp

In this post, I’ll share my debugging approach used to validate this solution. The debugging approach is not limited to the use of AWS ECS. It can be of value for all container environments such as kubernetes or plain docker.

Debugging with an overriden Container Entrypoint

When troubleshooting container permission issues, it’s helpful to use simple CLI tools to check the current user and directory permissions. I modified my container’s entry point in the ECS task definition to run a series of diagnostic commands:

resource "aws_ecs_task_definition" "my-task" {
  family                   = "service"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "256"
  memory                   = "512"

  execution_role_arn = aws_iam_role.ecs-task-execution-role.arn
  task_role_arn = aws_iam_role.ecs-task-execution-role.arn

  container_definitions = jsonencode([
    {
      name                     = "my-service"
      image                    = local.image_uri
      cpu                      = 256
      memory                   = 512
      essential                = true
      readonlyRootFilesystem   = true
      user                     = "185"
      portMappings = [
        {
          containerPort = 9090
          hostPort      = 9090
        }
      ]
      entryPoint = [
        "sh",
        "-c"
      ]
      command = [
         "set -x; whoami; ls -la /tmp; touch /tmp/itWorks; ls -la /tmp; cat /proc/mounts; df -h"
      ]
    }
  ])
}

Let’s break down these commands and understand what each one does:

Command Chaining with Semicolons

In the shell, the semicolon `;` allows you to run multiple commands sequentially on a single line. Each command runs independently, regardless of whether the previous command succeeded or failed. This is different from using `&&` which only runs the next command if the previous command succeeded, or `||` which only runs the next command if the previous one failed. In this case failing means returning a non zero returncode.

set -x

The `set -x` command enables debug mode in the shell. When this mode is active, the shell prints each command before executing it, prefixed with a + sign. This is useful for debugging scripts as it shows what commands are being executed. Without this it can be difficult to see which command yielded which output.

whoami

The `whoami` command prints the current user’s username. In our case, it showed that the container was running as the “jboss” user, confirming that we were indeed running as an unprivileged user defined in the Dockerfile of the base image.

ls -la /tmp

The `ls -la` command lists all files in a directory showing permissions, ownership, size, and modification time. This helped me see the current state of the /tmp directory and its permissions.

touch /tmp/itWorks

The `touch` command creates an empty file. I used it to test if the current user had write permissions in the /tmp directory. If this command succeeds, it confirms that the current user can write to the specified location.

cat /proc/mounts

The `cat /proc/mounts` command displays all mounted filesystems in the container. This is useful for seeing how filesystems are mounted. In this case, it showed that the root filesystem was indeed mounted as read-only (ro), while /tmp had its own mount point.

df -h

The `df -h` command shows disk space usage. This helped confirm which filesystems were available and how much space was allocated to them.

Analyzing the Output

Let’s run the container and have a look at the output of these commands:

+ whoami
jboss
+ ls -la /tmp
total 16
drwxrwxrwt 2 root root 4096 Apr 24 11:55 .
drwxr-xr-x 1 root root 4096 Apr 24 11:55 ..
-rwx------ 1 root root 291 Mar 13 11:05 ks-script-fb_e6y2x
-rwx------ 1 root root 701 Mar 13 11:05 ks-script-s7hte0c5
+ touch /tmp/itWorks
+ ls -la /tmp
total 16
drwxrwxrwt 2 root root 4096 Apr 24 11:55 .
drwxr-xr-x 1 root root 4096 Apr 24 11:55 ..
-rw-r--r-- 1 jboss root 0 Apr 24 11:55 itWorks
-rwx------ 1 root root 291 Mar 13 11:05 ks-script-fb_e6y2x
-rwx------ 1 root root 701 Mar 13 11:05 ks-script-s7hte0c5

From this output, I could see:

1. The container was running as the “jboss” user (unprivileged) otherwise it would have stated `root`
2. The /tmp directory belongs to root bu it has the sticky bit set (t in drwxrwxrwt), which allows any user to create files but only the owner can delete them. This is common for the /tmp directory.
3. The “jboss” user was able to successfully create a file in /tmp

The mount information was particularly revealing:

+ cat /proc/mounts
overlay / 
overlay ro,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/103/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/102/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/101/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/100/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/99/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/93/fs,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/107/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/107/work 0 0
/dev/nvme1n1 /tmp ext4 rw,relatime 0 0

This showed that:

1. The root filesystem (/) was mounted as read-only (ro)
2. The /tmp directory was mounted as a separate filesystem with read-write permissions (rw)

The disk usage information confirmed this setup:

+ df -h
Filesystem Size Used Avail Use% Mounted on
overlay 30G 11G 18G 37% /
tmpfs 64M 0 64M 0% /dev
shm 464M 0 464M 0% /dev/shm
tmpfs 464M 0 464M 0% /sys/fs/cgroup
/dev/nvme1n1 30G 11G 18G 37% /tmp
tmpfs 464M 0 464M 0% /proc/acpi

So in the end the definition of a docker volume for /tmp did the trick. When you define a volume in your Dockerfile, ECS with Fargate automatically creates an ephemeral storage mount with the correct permissions for the user specified in your container. This eliminates the need for manual permission adjustments. The storage used is AWS’s ephemeral storage, meaning it is temporary and will be removed when the container stops. This is ideal for temporary data storage needs.

Additional Considerations

ECS on Fargate automatically handles permissions for volumes. It will create the montpoint folder owned by the user that is specified in your container (e.g., user 185 or jboss). You can define multiple volumes if your application needs to write to different directories. If you need persistent storage instead of ephemeral, you would need to use EFS or other persistent storage options with ECS. This solution will not work for you in that case. I have not found any resource documenting this, admittedly useful, implementation of ECS. I don’t know but this behaviour might change in the future. There is an open Github Issue with a discussion regarding this though².

Conclusion

Debugging file permissions in containerized environments can be challenging, but using simple CLI tools can provide valuable insights into what’s happening inside your container. Hope that helps you out.

Footnotes