wolk-jjba-1bc2d24e-ba37

job succeeded

starting new job...


→ running command: cd "/etc/byggsteg/job-clone//wolk-jjba/trunk" && git reset --hard && git pull

HEAD is now at 8164fda fix: Name the secret that is missing when config.scm cannot be loaded
From https://codeberg.org/jjba23/wolk-jjba
   8164fda..54d96de  trunk      -> origin/trunk
 * [new tag]         v1.19.0    -> v1.19.0
Updating 8164fda..54d96de
Fast-forward
 README.org                                         | 293 +++++++++++++++++++--
 config.scm                                         |  11 +
 src/wolk-jjba/iptables.scm                         |  53 ++++
 src/wolk-jjba/monitoring.scm                       | 201 ++++++++++++--
 src/wolk-jjba/monitoring/alloy.alloy               |  40 ++-
 .../monitoring/dashboards/storage-retention.json   |   4 +-
 src/wolk-jjba/monitoring/loki.yml                  |  60 ++++-
 src/wolk-jjba/monitoring/prometheus.yml            |  42 ++-
 src/wolk-jjba/nginx.scm                            |  98 ++++++-
 src/wolk-jjba/sysctl.scm                           | 144 ++++++++++
 test/veritas/unit/iptables-spec.scm                |  72 ++++-
 test/veritas/unit/nginx-spec.scm                   |  60 +++++
 test/veritas/unit/sysctl-spec.scm                  | 153 +++++++++++
 13 files changed, 1139 insertions(+), 92 deletions(-)
 create mode 100644 src/wolk-jjba/sysctl.scm
 create mode 100644 test/veritas/unit/sysctl-spec.scm

→ running command: cd /etc/byggsteg/job-clone/wolk-jjba/trunk && uname -a

Linux wolk-jjba 7.1.5-gnu #1 SMP PREEMPT_DYNAMIC 1 x86_64 GNU/Linux

→ running command: cd /etc/byggsteg/job-clone/wolk-jjba/trunk && git log --name-status HEAD^..HEAD

commit 54d96defdb044b95731d7d6afbb7ddb2caea42c1
Author: Josep Bigorra <jjbigorra@gmail.com>
Date:   Fri Oct 23 07:49:24 2026 +0200

    feat: Add hardened monitor

M       README.org
M       config.scm
M       src/wolk-jjba/iptables.scm
M       src/wolk-jjba/monitoring.scm
M       src/wolk-jjba/monitoring/alloy.alloy
M       src/wolk-jjba/monitoring/dashboards/storage-retention.json
M       src/wolk-jjba/monitoring/loki.yml
M       src/wolk-jjba/monitoring/prometheus.yml
M       src/wolk-jjba/nginx.scm
A       src/wolk-jjba/sysctl.scm
M       test/veritas/unit/iptables-spec.scm
M       test/veritas/unit/nginx-spec.scm
A       test/veritas/unit/sysctl-spec.scm

→ running command: cd /etc/byggsteg/job-clone/wolk-jjba/trunk && maak test


scanning directories for unit tests...
scanning present working directory for unit tests...

shuffled the files with seed 441518405 (replay with VERITAS_SEED=441518405)

found unit tests: (logging-spec.scm prelude-spec.scm gfs-spec.scm sysctl-spec.scm maintenance-spec.scm jobs-spec.scm certbot-spec.scm radio-spec.scm operating-system-spec.scm nginx-spec.scm iptables-spec.scm)


running unit tests from file: logging-spec.scm, entrypoint: spec


ℹ️ info - shuffled with seed 441518405 (replay with VERITAS_SEED=441518405)
+----------------------------------------------+
[📑 suite] Logging Suite
  [🧪 test] Everything else lands in the catch-all
    [⚙️ assert] There is a catch-all file
    ✅ got: #t
    [⚙️ assert] Private authorization messages are routed too
    ✅ got: #t
    [⚙️ assert] Facilities are read through the Shepherd
    ✅ got: #t
  [🧪 test] AUTH, mail and cron have their own files
    [⚙️ assert] Authentication goes to auth.log
    ✅ got: #t
    [⚙️ assert] Mail goes to mail.log
    ✅ got: #t
    [⚙️ assert] Scheduled jobs go to cron.log
    ✅ got: #t
    [⚙️ assert] Kernel messages go to kern.log
    ✅ got: #t
    [⚙️ assert] Daemon messages go to daemon.log
    ✅ got: #t
  [🧪 test] The capability replaces the Shepherd's destination
    [⚙️ assert] The system log service survives the rewrite
    ✅ got: #t
    [⚙️ assert] The configured destination is this module's procedure
    ✅ got: #t
    [⚙️ assert] It is not the default catch-all destination
    ✅ got: #f
  [🧪 test] The capability is a rewriting function, not a service
    [⚙️ assert] The service list it is given comes back
    ✅ got: 1
  [🧪 test] Urgent messages reach the console
    [⚙️ assert] emergency and alert are written to /dev/console
    ✅ got: #t
    [⚙️ assert] The emergency priority is named in the table
    ✅ got: #t
    [⚙️ assert] The alert priority is named in the table
    ✅ got: #t
    [⚙️ assert] Priorities are read through the Shepherd
    ✅ got: #t
ℹ️ info - ran 1 tasks in: 0 seconds, 448252 microseconds

running unit tests from file: prelude-spec.scm, entrypoint: spec


ℹ️ info - shuffled with seed 441518405 (replay with VERITAS_SEED=441518405)
+----------------------------------------------+
[📑 suite] Prelude Suite
  [🧪 test] Every factory call yields an independent service type
    [⚙️ assert] Two daemons are two service types
    ✅ got: #f
  [🧪 test] The daemon environment carries a UTF-8 locale
    [⚙️ assert] LANG is set for every application daemon
    ✅ got: #t
    [⚙️ assert] Nothing forces the C locale
    ✅ got: #f
    [⚙️ assert] Guile finds the system's modules
    ✅ got: #t
    [⚙️ assert] The environment is the declared eleven
    ✅ got: 11
    [⚙️ assert] A PATH is passed, so make and guile resolve
    ✅ got: #t
  [🧪 test] The daemon logs to its own file, not the system log
    [⚙️ assert] The log file is named after the daemon
    ✅ got: #t
    [⚙️ assert] The log directory is created before exec
    ✅ got: #t
    [⚙️ assert] The file is handed to #:log-file
    ✅ got: #t
  [🧪 test] The daemon runs maak in the project directory
    [⚙️ assert] The default command is make production-server
    ✅ got: #t
    [⚙️ assert] The command is maak, run through make
    ✅ got: #t
    [⚙️ assert] The project directory is the working one
    ✅ got: #t
    [⚙️ assert] A working directory reaches the constructor
    ✅ got: #t
  [🧪 test] Per-daemon logs share one directory
    [⚙️ assert] The log directory is the one Alloy tails
    ✅ got: /var/log/wolk-jjba
  [🧪 test] A daemon is a respawning, auto-started shepherd service
    [⚙️ assert] The provision name is the daemon name
    ✅ got: (probe-daemon)
    [⚙️ assert] A crashed daemon must be respawned by the Shepherd
    ✅ got: #t
    [⚙️ assert] A daemon starts with the system
    ✅ got: #t
    [⚙️ assert] The factory imposes no ordering of its own
    ✅ got: ()
  [🧪 test] The log file can be overridden
    [⚙️ assert] The explicit path wins over the convention
    ✅ got: #t
    [⚙️ assert] The conventional path is not also passed
    ✅ got: #f
  [🧪 test] The factory names its own service type
    [⚙️ assert] The service type is named after the daemon
    ✅ got: named-daemon
    [⚙️ assert] The description is the documentation
    ✅ got: documented
    [⚙️ assert] The service carries no value of its own
    ✅ got: ()
  [🧪 test] Managed projects live under one checkout tree
    [⚙️ assert] The default branch is trunk
    ✅ got: /etc/byggsteg/job-clone/lucidplan/trunk
    [⚙️ assert] A non-default branch is honoured
    ✅ got: /etc/byggsteg/job-clone/wikimusic/experiment
    [⚙️ assert] The name may be any symbol
    ✅ got: /etc/byggsteg/job-clone/byggsteg/trunk
  [🧪 test] Shell helpers return what the command printed
    ✅ got: hello

ℹ️ info - ran 1 tasks in: 0 seconds, 507379 microseconds

running unit tests from file: gfs-spec.scm, entrypoint: spec


ℹ️ info - shuffled with seed 441518405 (replay with VERITAS_SEED=441518405)
+----------------------------------------------+
[📑 suite] GFS Retention Suite
  [🧪 test] Nothing older than a year survives, and a year is honoured
    [⚙️ assert] 364 days old is still kept
    ✅ got: (364)
    [⚙️ assert] 365 days and older are deleted (same month)
    ✅ got: (365 366 400)
    [⚙️ assert] A year-old archive in another month is kept
    ✅ got: (300)
  [🧪 test] Every archive is kept or deleted, never both
    [⚙️ assert] The two lists partition the directory
    ✅ got: 40
    [⚙️ assert] No archive appears in both
    ✅ got: ()
    [⚙️ assert] Nothing kept is over a year old
    ✅ got: #t
  [🧪 test] Past a week, the week's representative survives
    [⚙️ assert] No week is represented twice
    ✅ got: 5
    [⚙️ assert] A deleted archive's week is kept by another
    ✅ got: #t
    [⚙️ assert] Those survivors are father archives
    ✅ got: #t
  [🧪 test] A dry run reports without touching the box
    [⚙️ assert] No copy in a dry run
    ✅ got: ()
    [⚙️ assert] No deletion in a dry run
    ✅ got: ()
  [🧪 test] Parsing sorts newest first and drops everything else
    [⚙️ assert] Newest first, strays ignored
    ✅ got: (300 200 100)
    [⚙️ assert] An empty directory parses to nothing
    ✅ got: ()
  [🧪 test] A run twice in a row changes nothing the second time
    [⚙️ assert] The archive tree did not grow
    ✅ got: 2
    [⚙️ assert] The second run copied nothing
    ✅ got: 2
  [🧪 test] Past a month, the month's representative survives
    [⚙️ assert] No month is represented twice
    ✅ got: 7
    [⚙️ assert] A deleted archive's month is kept by another
    ✅ got: #t
    [⚙️ assert] Those survivors are grandfather archives
    ✅ got: #t
  [🧪 test] Running a command returns its lines and its exit status
    [⚙️ assert] The output is read line by line
    ✅ got: (one two)
    [⚙️ assert] The exit status is not swallowed
    ✅ got: 3
  [🧪 test] Promotion is once per week and once per month
    [⚙️ assert] An empty archive tree takes the newest
    ✅ got: ((1792584000 . proj-backup-1792584000.tar.gz.enc))
    [⚙️ assert] A week that is already represented needs no copy
    ✅ got: ()
    [⚙️ assert] A month that is already represented still takes a new week
    ✅ got: ((1792584000 . proj-backup-1792584000.tar.gz.enc))
  [🧪 test] A run promotes, prunes and reports
    [⚙️ assert] One long-term archive per project
    ✅ got: ((proj . 1))
    [⚙️ assert] The newest mirror archive was promoted
    ✅ got: #t
    [⚙️ assert] A stray file in the archive tree survives
    ✅ got: #t
    [⚙️ assert] The expired archive was deleted
    ✅ got: #f
    [⚙️ assert] Nothing outside the archive tree is deleted
    ✅ got: #t
    [⚙️ assert] The promoted archive is not then deleted
    ✅ got: #t
  [🧪 test] Shell quoting survives an apostrophe
    [⚙️ assert] A quote is escaped the POSIX way
    ✅ got: 'it'\''s'
    [⚙️ assert] Spaces stay inside one word
    ✅ got: 'ls -1 /a b'
  [🧪 test] An unreachable box is a failure, not an empty directory
    [⚙️ assert] No project is reported as counted
    ✅ got: ()
    [⚙️ assert] Every unreachable project is reported
    ✅ got: (proj other)
    [⚙️ assert] A silent empty directory would leave this empty
    ✅ got: #t
  [🧪 test] Archive names identify an archive; sidecars and strays do not
    [⚙️ assert] A tar archive carries its timestamp
    ✅ got: 1700000000
    [⚙️ assert] So does an SQL dump
    ✅ got: 1700000000
    [⚙️ assert] The checksum sidecar is not an archive
    ✅ got: #f
    [⚙️ assert] A stray file is not an archive
    ✅ got: #f
    [⚙️ assert] An unencrypted tarball is not an archive
    ✅ got: #f
  [🧪 test] Every archive up to a week old survives
    [⚙️ assert] Ages 0 to 7 are all kept
    ✅ got: (0 1 2 3 4 5 6 7)
    [⚙️ assert] Nothing in the son window is deleted
    ✅ got: ()
    [⚙️ assert] All of them are kept as son
    ✅ got: #t
ℹ️ info - ran 1 tasks in: 5 seconds, -688251 microsecondsgfs-backup: proj failed: misc-error (#f "cannot reach the Storage Box over ssh" () #f)
gfs-backup: other failed: misc-error (#f "cannot reach the Storage Box over ssh" () #f)


running unit tests from file: sysctl-spec.scm, entrypoint: spec


ℹ️ info - shuffled with seed 441518405 (replay with VERITAS_SEED=441518405)
+----------------------------------------------+
[📑 suite] Sysctl Suite
  [🧪 test] The host keeps its routing and addressing role
    [⚙️ assert] Packet forwarding is left to Podman
    ✅ got: #f
    [⚙️ assert] IPv6 forwarding is left alone
    ✅ got: #f
    [⚙️ assert] Router advertisements stay with NetworkManager
    ✅ got: #f
    [⚙️ assert] IPv6 is not switched off
    ✅ got: #f
  [🧪 test] The kernel survives a SYN flood
    [⚙️ assert] Syncookies are on
    ✅ got: 1
    [⚙️ assert] The SYN backlog is larger than the default
    ✅ got: #t
    [⚙️ assert] The accept queue is larger than the default
    ✅ got: #t
  [🧪 test] Kernel internals are not readable by ordinary users
    [⚙️ assert] Kernel pointers are hidden
    ✅ got: 2
    [⚙️ assert] Only root reads the ring buffer
    ✅ got: 1
    [⚙️ assert] Unprivileged BPF loading is off
    ✅ got: 1
  [🧪 test] Spoofed and redirected traffic is not honoured
    [⚙️ assert] Reverse-path filtering is strict
    ✅ got: 1
    [⚙️ assert] and applies to interfaces created later
    ✅ got: 1
    [⚙️ assert] IPv4 redirects are refused
    ✅ got: 0
    [⚙️ assert] IPv6 redirects are refused
    ✅ got: 0
    [⚙️ assert] This host sends no redirects
    ✅ got: 0
    [⚙️ assert] Source-routed IPv4 is refused
    ✅ got: 0
  [🧪 test] Every setting is a well-formed key/value pair
    [⚙️ assert] Each entry is a dotted key and a string
    ✅ got: #t
    [⚙️ assert] No key is set twice
    ✅ got: 30
  [🧪 test] The settings extend Guix's defaults instead of replacing them
    [⚙️ assert] Guix's own defaults survive
    ✅ got: #t
    [⚙️ assert] and this module's settings are added
    ✅ got: #t
    [⚙️ assert] Nothing is set twice or dropped
    ✅ got: 32
  [🧪 test] ICMP is bounded
    [⚙️ assert] Broadcast echo is ignored
    ✅ got: 1
    [⚙️ assert] Echo replies are rate limited
    ✅ got: #t
ℹ️ info - ran 1 tasks in: 0 seconds, 408771 microseconds;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-auto-compile argument to disable.
;;; compiling ./test/veritas/unit/sysctl-spec.scm
;;; compiled /root/.cache/guile/ccache/3.0-LE-8-4.7/etc/byggsteg/job-clone/wolk-jjba/trunk/test/veritas/unit/sysctl-spec.scm.go


running unit tests from file: maintenance-spec.scm, entrypoint: spec


ℹ️ info - shuffled with seed 441518405 (replay with VERITAS_SEED=441518405)
+----------------------------------------------+
[📑 suite] Maintenance Suite
  [🧪 test] Textfiles are replaced atomically
    [⚙️ assert] The new file is written under a temporary name
    ✅ got: #t
    [⚙️ assert] The temporary file is renamed into place
    ✅ got: #t
    [⚙️ assert] It lands in node exporter's textfile directory
    ✅ got: #t
    [⚙️ assert] And never as a partial .prom file
    ✅ got: #t
  [🧪 test] The Guix collection frees generations before the store
    [⚙️ assert] Old system generations go first
    ✅ got: #t
    [⚙️ assert] joe's home generations go too
    ✅ got: #t
    [⚙️ assert] The home profile is pruned as its owner
    ✅ got: #t
    [⚙️ assert] The store is collected afterwards
    ✅ got: #t
    [⚙️ assert] The size of the store is published
    ✅ got: #t
  [🧪 test] The backup verification actually reads the archive
    [⚙️ assert] The checksum sidecar is checked
    ✅ got: #t
    [⚙️ assert] A tar archive is listed through a pipe
    ✅ got: #t
    [⚙️ assert] An SQL dump is tested through a pipe
    ✅ got: #t
    [⚙️ assert] A failed verification fails the job
    ✅ got: #t
    [⚙️ assert] A success is published, so a stale check is visible
    ✅ got: #t
    [⚙️ assert] Nothing is decrypted to a temporary file
    ✅ got: #t
  [🧪 test] Pruning podman state leaves volumes alone
    [⚙️ assert] Unused containers, images and build cache go
    ✅ got: #t
    [⚙️ assert] Only things untouched for a week
    ✅ got: #t
    [⚙️ assert] Volumes are never pruned: a container may own data there
    ✅ got: #t
    [⚙️ assert] The CI runner's images are pruned too
    ✅ got: #t
  [🧪 test] The temporary-file cleanup spares sockets
    [⚙️ assert] A long-running daemon keeps its socket
    ✅ got: #t
    [⚙️ assert] Only week-old files are candidates
    ✅ got: #t
    [⚙️ assert] Both temporary directories are covered
    ✅ got: #t
  [🧪 test] Every job logs to a rotated file
    [⚙️ assert] The log files are registered for rotation
    ✅ got: (/var/log/wolk-jjba/disk-report.log /var/log/wolk-jjba/guix-gc.log /var/log/wolk-jjba/nix-gc.log /var/log/wolk-jjba/verify-backups.log /var/log/wolk-jjba/gfs-backup.log /var/log/wolk-jjba/podman-prune.log /var/log/wolk-jjba/cache-clean.log /var/log/wolk-jjba/tmp-clean.log)
    [⚙️ assert] The garbage collection log is kept
    ✅ got: #t
  [🧪 test] The long-term retention job runs the Scheme program, unprivileged
    [⚙️ assert] It runs every night at 02:45
    ✅ got: #t
    [⚙️ assert] It runs the gfs-backup program, not a shell script
    ✅ got: #t
    [⚙️ assert] It runs as the user that owns the Storage Box key
    ✅ got: #t
    [⚙️ assert] With that user's HOME, so ssh finds the alias
    ✅ got: #t
    [⚙️ assert] No shell is involved
    ✅ got: #t
  [🧪 test] The caches that are deleted are download caches
    [⚙️ assert] The npm cache goes
    ✅ got: #t
    [⚙️ assert] The pip cache goes
    ✅ got: #t
    [⚙️ assert] The cache is removed, not truncated
    ✅ got: #t
  [🧪 test] The disk report publishes what the alerts need
    [⚙️ assert] One series per project, with a project label
    ✅ got: #t
    [⚙️ assert] The newest archive of each project is dated
    ✅ got: #t
    [⚙️ assert] A project without archives reports 0, not nothing
    ✅ got: #t
    [⚙️ assert] The report dates itself, so a stale report is visible
    ✅ got: #t
    [⚙️ assert] The backup tree is the thing being reported
    ✅ got: #t
  [🧪 test] The disk report publishes the long-term retention state
    [⚙️ assert] The last successful GFS run is a metric
    ✅ got: #t
    [⚙️ assert] And so is the archive count per project
    ✅ got: #t
    [⚙️ assert] It reads the marker the job leaves behind
    ✅ got: #t
  [🧪 test] The service creates the metrics directory it writes into
    [⚙️ assert] There is an activation extension
    ✅ got: #t
    [⚙️ assert] The activation creates node exporter's textfile directory
    ✅ got: #t
  [🧪 test] Every maintenance job is scheduled exactly once
    [⚙️ assert] The eight jobs are the documented ones
    ✅ got: (disk-report guix-gc nix-gc verify-backups gfs-backup podman-prune cache-clean tmp-clean)
    [⚙️ assert] No job is scheduled twice
    ✅ got: 8
    [⚙️ assert] Every job is respawned
    ✅ got: #t
ℹ️ info - ran 1 tasks in: 0 seconds, 72167 microseconds

running unit tests from file: jobs-spec.scm, entrypoint: spec


ℹ️ info - shuffled with seed 441518405 (replay with VERITAS_SEED=441518405)
+----------------------------------------------+
[📑 suite] Jobs Suite
  [🧪 test] PostgreSQL is dumped, not tarred, and as the postgres role
    [⚙️ assert] The dump is restorable into a live cluster
    ✅ got: #t
    [⚙️ assert] peer authentication is satisfied by running as postgres
    ✅ got: #t
  [🧪 test] A database backup is one transaction
    [⚙️ assert] A failing step fails the job
    ✅ got: #t
    [⚙️ assert] The archive is made in the project directory
    ✅ got: #t
    [⚙️ assert] The archive name carries a timestamp
    ✅ got: #t
    [⚙️ assert] A trap removes the plaintext and partial files
    ✅ got: #t
    [⚙️ assert] The trap names the archive it cleans up
    ✅ got: #t
    [⚙️ assert] The archive is encrypted on the way out
    ✅ got: #t
    [⚙️ assert] The passphrase is read from a file, not the script
    ✅ got: #t
    [⚙️ assert] The ciphertext is staged and renamed into place
    ✅ got: #t
    [⚙️ assert] A checksum sidecar is written next to the archive
    ✅ got: #t
    [⚙️ assert] The archive and its checksum are moved into place
    ✅ got: #t
    [⚙️ assert] The destination is created if missing
    ✅ got: #t
  [🧪 test] The schedules are the ones the README documents
    [⚙️ assert] wikimusic is archived twice a day
    ✅ got: #t
    [⚙️ assert] Minecraft is pruned hourly
    ✅ got: #t
    [⚙️ assert] PostgreSQL is dumped twice a day
    ✅ got: #t
    [⚙️ assert] The mirror runs just before the hour
    ✅ got: #t
  [🧪 test] Each job logs to its own file, which is rotated
    [⚙️ assert] The backup log is named after the job
    ✅ got: #t
    [⚙️ assert] Every job log is registered for rotation
    ✅ got: 18
    [⚙️ assert] The Storage Box mirror log is rotated too
    ✅ got: #t
    [⚙️ assert] The PostgreSQL dump log is rotated too
    ✅ got: #t
  [🧪 test] Pruning can also bound an archive by age
    [⚙️ assert] An archive older than the age limit goes
    ✅ got: #t
    [⚙️ assert] No age limit means count only
    ✅ got: #t
  [🧪 test] The weekly upgrade stays outside the Shepherd
    [⚙️ assert] The reconfigure job extends mcron, not shepherd
    ✅ got: (mcron)
  [🧪 test] Applications are restarted hourly
    [⚙️ assert] lucidplan is restarted by the timer
    ✅ got: #t
    [⚙️ assert] byggsteg is restarted by the timer
    ✅ got: #t
    [⚙️ assert] torikomi is restarted by the timer
    ✅ got: #t
    [⚙️ assert] symbanta is restarted by the timer
    ✅ got: #t
  [🧪 test] Every job is a timer with a log and a trigger
    [⚙️ assert] Eighteen jobs are scheduled
    ✅ got: 18
    [⚙️ assert] No job is scheduled twice
    ✅ got: 18
    [⚙️ assert] Every timer is respawned
    ✅ got: #t
    [⚙️ assert] A job can be run by hand with herd trigger
    ✅ got: (trigger)
    [⚙️ assert] The timers come from the Shepherd's timer module
    ✅ got: #t
  [🧪 test] A dump backup refuses an empty dump
    [⚙️ assert] The command writes the archive
    ✅ got: #t
    [⚙️ assert] An empty dump fails the job instead of being stored
    ✅ got: #t
    [⚙️ assert] The dump keeps its own extension
    ✅ got: #t
  [🧪 test] Pruning deletes exactly the archives it should
    [⚙️ assert] Only regular files matching the archive pattern
    ✅ got: #t
    [⚙️ assert] Keeping three means dropping from the fourth
    ✅ got: #t
    [⚙️ assert] Keeping one means dropping from the second
    ✅ got: #t
    [⚙️ assert] The checksum goes with its archive
    ✅ got: #t
    [⚙️ assert] The prune no longer deletes whatever sorts last
    ✅ got: #t
    [⚙️ assert] Leftovers of an interrupted run are removed later
    ✅ got: #t
  [🧪 test] A file backup still tars what it is given
    [⚙️ assert] The include is tarred as it is
    ✅ got: #t
    [⚙️ assert] No database is snapshotted for a file backup
    ✅ got: #t
  [🧪 test] A database backup snapshots instead of copying live files
    [⚙️ assert] The database is snapshotted with sqlite3 .backup
    ✅ got: #t
    [⚙️ assert] The snapshot is checked before it is stored
    ✅ got: #t
    [⚙️ assert] The archive holds the snapshot, not the live file
    ✅ got: #t
    [⚙️ assert] The -wal and -shm files are no longer tarred
    ✅ got: #t
  [🧪 test] The mirror is a delete-mirror run as an unprivileged user
    [⚙️ assert] Deleted backups disappear from the Storage Box
    ✅ got: #t
    [⚙️ assert] The Storage Box is the destination
    ✅ got: #t
    [⚙️ assert] The mirror does not run as root
    ✅ got: #t
    [⚙️ assert] The long-term archive is not mirrored over
    ✅ got: #t
ℹ️ info - ran 1 tasks in: 1 seconds, -870191 microseconds

running unit tests from file: certbot-spec.scm, entrypoint: spec


ℹ️ info - shuffled with seed 441518405 (replay with VERITAS_SEED=441518405)
+----------------------------------------------+
[📑 suite] Certbot Suite
  [🧪 test] The capability keeps the documented account and webroot
    [⚙️ assert] The ACME account address is set
    ✅ got: jjbigorra@gmail.com
    [⚙️ assert] HTTP-01 challenges are answered from /srv/http
    ✅ got: /srv/http
  [🧪 test] Fifteen certificates are requested
    [⚙️ assert] The count matches the list in the module
    ✅ got: 15
    [⚙️ assert] No domain is requested twice
    ✅ got: 28
  [🧪 test] Each vhost has a certificate
    [⚙️ assert] Every TLS vhost name is covered by a certificate
    ✅ got: ()
  [🧪 test] Every certificate reloads nginx
    [⚙️ assert] A deploy hook is attached to each request
    ✅ got: #t
    [⚙️ assert] All of them are the one nginx reload hook
    ✅ got: #t
    [⚙️ assert] The hook reads nginx's pid file
    ✅ got: #t
    [⚙️ assert] The hook reloads nginx with SIGHUP
    ✅ got: #t
  [🧪 test] A certificate request covers the apex and its www
    [⚙️ assert] The www alias is requested by default
    ✅ got: (example.org www.example.org)
    [⚙️ assert] with-www? #f requests the apex alone
    ✅ got: (naked.example)
  [🧪 test] Each certificate has a vhost
    [⚙️ assert] No certificate is requested for an unserved name
    ✅ got: ()
ℹ️ info - ran 1 tasks in: 0 seconds, 33807 microseconds

running unit tests from file: radio-spec.scm, entrypoint: spec


ℹ️ info - shuffled with seed 441518405 (replay with VERITAS_SEED=441518405)
+----------------------------------------------+
[📑 suite] Radio Suite
  [🧪 test] The scan job skips a library that is not there
    [⚙️ assert] It asks whether the mount is up
    ✅ got: #t
    [⚙️ assert] And says so instead of scanning
    ✅ got: #t
    [⚙️ assert] Otherwise it refreshes the database
    ✅ got: #t
    [⚙️ assert] It authenticates like every other mpc call
    ✅ got: #t
  [🧪 test] MPD clients must authenticate, and the two passwords differ
    [⚙️ assert] An admin and a read-only password
    ✅ got: (admin-pw@read,add,control,admin read-pw@read,add,control)
  [🧪 test] The service provides the Icecast account
    [⚙️ assert] The unprivileged user is part of the capability
    ✅ got: #t
  [🧪 test] The activation creates what the Shepherd does not
    [⚙️ assert] MPD's playlist directory
    ✅ got: #t
    [⚙️ assert] The failure counter's directory
    ✅ got: #t
  [🧪 test] Icecast is configured from the secrets, at the sizes a phone needs
    [⚙️ assert] The source password comes from the secret
    ✅ got: #t
    [⚙️ assert] And so does the admin password
    ✅ got: #t
    [⚙️ assert] The burst is 128 KiB, not Icecast's 64 KiB default
    ✅ got: #t
    [⚙️ assert] The queue holds two bursts
    ✅ got: #t
    [⚙️ assert] There is room for every mount plus one
    ✅ got: #t
    [⚙️ assert] It listens on 6690
    ✅ got: #t
    [⚙️ assert] / is the status page
    ✅ got: #t
    [⚙️ assert] The web root is a placeholder for the store path
    ✅ got: #t
  [🧪 test] Both timers are scheduled and rotated
    [⚙️ assert] Two radio timers
    ✅ got: (radio-health mpd-scan)
    [⚙️ assert] The health check runs every five minutes
    ✅ got: #t
    [⚙️ assert] The library scan runs nightly
    ✅ got: #t
    [⚙️ assert] Icecast's access log is rotated
    ✅ got: #t
    [⚙️ assert] Its error log is rotated too
    ✅ got: #t
    [⚙️ assert] And the job log
    ✅ got: #t
  [🧪 test] MPD pushes the same audio into three mounts
    [⚙️ assert] Three streams
    ✅ got: 3
    [⚙️ assert] The mounts are the documented ones
    ✅ got: (/radio.opus /radio-low.opus /radio.mp3)
    [⚙️ assert] Two Opus and one MP3 for reach
    ✅ got: (opus opus lame)
    [⚙️ assert] 128k, 64k for mobile data, 96k MP3
    ✅ got: (128000 64000 96000)
    [⚙️ assert] Every output stays connected between tracks
    ✅ got: #t
    [⚙️ assert] Every output is an Icecast client
    ✅ got: #t
    [⚙️ assert] Every output authenticates with the source password
    ✅ got: #t
  [🧪 test] MPD keeps its index on local disk, not on the mount
    [⚙️ assert] The database is local
    ✅ got: /var/lib/mpd/tag_cache
    [⚙️ assert] So is the state file that survives a restart
    ✅ got: /var/lib/mpd/state
    [⚙️ assert] And the playlist directory
    ✅ got: /var/lib/mpd/playlists
    [⚙️ assert] The library is the CIFS mount
    ✅ got: #t
    [⚙️ assert] Eight megabytes of decode buffer absorb slow reads
    ✅ got: 8192
  [🧪 test] The live vhost carries the stream
    [⚙️ assert] https://jointhefreeworld.org/radio/ is served
    ✅ got: #t
  [🧪 test] Passwords are escaped before they reach the XML
    [⚙️ assert] All five metacharacters are escaped
    ✅ got: #t
    [⚙️ assert] Nothing raw is left behind
    ✅ got: #t
  [🧪 test] No stream password is committed to the repository
    [⚙️ assert] The old literal passwords are gone
    ✅ got: #t
    [⚙️ assert] And the secret file is used instead
    ✅ got: #t
  [🧪 test] nginx proxies the stream without buffering it
    [⚙️ assert] A ^~ location wins over the static-asset regex
    ✅ got: ^~ /radio/
    [⚙️ assert] Audio is forwarded, not buffered
    ✅ got: #t
    [⚙️ assert] A stream is one request that never ends
    ✅ got: #t
    [⚙️ assert] The upstream speaks HTTP/1.1
    ✅ got: #t
    [⚙️ assert] Web players on another origin are allowed
    ✅ got: #t
    [⚙️ assert] The stream is not cacheable
    ✅ got: #t
    [⚙️ assert] The location repeats the security headers
    ✅ got: #t
  [🧪 test] The health job authenticates and watches every mount
    [⚙️ assert] The main mount is checked
    ✅ got: #t
    [⚙️ assert] The mobile mount is checked
    ✅ got: #t
    [⚙️ assert] The MP3 mount is checked
    ✅ got: #t
    [⚙️ assert] MPD liveness is a TCP connect
    ✅ got: #t
    [⚙️ assert] The library mount is checked
    ✅ got: #t
    [⚙️ assert] The player state comes from mpc's format string
    ✅ got: #t
    [⚙️ assert] mpc authenticates with the password from the secret
    ✅ got: #t
    [⚙️ assert] The secrets directory is the real one
    ✅ got: #t
    [⚙️ assert] The password is read at run time
    ✅ got: #t
    [⚙️ assert] No password is baked into the job script
    ✅ got: #t
    [⚙️ assert] The song count is read, for the empty-database case
    ✅ got: #t
    [⚙️ assert] An empty database is rescanned, with its output off stdout
    ✅ got: #t
  [🧪 test] The health job only intervenes after repeated failures
    [⚙️ assert] Failures are counted across runs
    ✅ got: #t
    [⚙️ assert] Three bad checks in a row are needed
    ✅ got: #t
    [⚙️ assert] Icecast is restarted first
    ✅ got: #t
    [⚙️ assert] Then MPD, which would otherwise reconnect to nothing
    ✅ got: #t
ℹ️ info - ran 1 tasks in: 0 seconds, 141205 microseconds

running unit tests from file: operating-system-spec.scm, entrypoint: spec

Backtrace:
In ice-9/boot-9.scm:
  1784:12  9 (with-exception-handler _ _ #:unwind? _ #:unwind-for-type _)
In unknown file:
           8 (apply-smob/0 #<thunk 7fbc84716340>)
In ice-9/boot-9.scm:
    733:2  7 (call-with-prompt _ _ #<procedure default-prompt-handler (k proc)>)
In ice-9/eval.scm:
    619:8  6 (_ #(#(#<directory (guile-user) 7fbc84719c80>)))
In ice-9/command-line.scm:
   187:19  5 (_ #<input: custom-port 7fbc847003f0>)
In unknown file:
           4 (eval (load "/etc/byggsteg/job-clone/wolk-jjba/trunk/config.scm") #<directory (guile-user) 7fbc84719c80>)
In ice-9/boot-9.scm:
   2890:4  3 (save-module-excursion _)
  4440:12  2 (_)
In /etc/byggsteg/job-clone/wolk-jjba/trunk/config.scm:
    243:2  1 (_)
    233:8  0 (read-secret "mpd-admin")

/etc/byggsteg/job-clone/wolk-jjba/trunk/config.scm:233:8: In procedure read-secret:
missing secret: /tmp/wolk-jjba-spec-no-such-directory/mpd-admin
The radio reads four secrets from /tmp/wolk-jjba-spec-no-such-directory: mpd-admin, mpd-read, icecast-source and icecast-admin.
Create them as described in "Create the secrets" in README.org, e.g.
  openssl rand -base64 24 | sudo tee /tmp/wolk-jjba-spec-no-such-directory/mpd-admin >/dev/null
  sudo chmod 600 /tmp/wolk-jjba-spec-no-such-directory/mpd-admin

ℹ️ info - shuffled with seed 441518405 (replay with VERITAS_SEED=441518405)
+----------------------------------------------+
[📑 suite] Operating System Suite
  [🧪 test] The languages a visitor may ask for are compiled
    [⚙️ assert] Japanese, Greek, Chinese (both scripts), Arabic and Russian
    ✅ got: (#t #t #t #t #t #t)
  [🧪 test] The machine still has its services
    [⚙️ assert] The service list is the whole machine
    ✅ got: #t
  [🧪 test] A missing secrets directory fails the evaluation
    [⚙️ assert] config.scm cannot silently ship without secrets
    ✅ got: #t
  [🧪 test] The system configuration evaluates as an ordinary user
    [⚙️ assert] config.scm evaluates without root
    ✅ got: 0
    [⚙️ assert] The host keeps its name
    ✅ got: wolk-jjba
    [⚙️ assert] The result is an operating-system
    ✅ got: #t
  [🧪 test] The passwords come from the secrets directory the environment names
    [⚙️ assert] The admin password is the fixture's
    ✅ got: fixture-admin-password
    [⚙️ assert] The read password is the fixture's
    ✅ got: fixture-read-password
ℹ️ info - ran 1 tasks in: 0 seconds, 16253 microseconds

running unit tests from file: nginx-spec.scm, entrypoint: spec


ℹ️ info - shuffled with seed 441518405 (replay with VERITAS_SEED=441518405)
+----------------------------------------------+
[📑 suite] Nginx Suite
  [🧪 test] Every vhost denies dotfiles
    [⚙️ assert] The dotfile rule is the first location
    ✅ got: ~ /\.(?!well-known)
    [⚙️ assert] and it denies everything
    ✅ got: (deny all;)
  [🧪 test] Every upstream a vhost names is declared
    [⚙️ assert] Ten upstreams are declared
    ✅ got: 10
    [⚙️ assert] Each proxy_pass names a declared upstream
    ✅ got: ()
  [🧪 test] The ACME vhost on port 80 is rate limited
    [⚙️ assert] Plain HTTP gets a request bucket
    ✅ got: #t
    [⚙️ assert] and a connection cap
    ✅ got: #t
  [🧪 test] A reverse proxy passes the client's view upstream
    [⚙️ assert] The domain defaults to jointhefreeworld.org
    ✅ got: (probe.jointhefreeworld.org www.probe.jointhefreeworld.org)
    [⚙️ assert] The vhost proxies to its own upstream
    ✅ got: #t
    [⚙️ assert] The upstream learns the client's scheme
    ✅ got: #t
    [⚙️ assert] WebSocket upgrades survive the proxy
    ✅ got: #t
    [⚙️ assert] Static assets are cached upstream too
    ✅ got: #t
    [⚙️ assert] with-gzip? #f disables upstream gzip
    ✅ got: #t
    [⚙️ assert] with-cors? adds the CORS header
    ✅ got: #t
  [🧪 test] The ACME vhost answers on plain HTTP and redirects
    [⚙️ assert] The ACME vhost listens on port 80
    ✅ got: #t
    [⚙️ assert] Minimal mode emits no TLS vhost
    ✅ got: #f
    [⚙️ assert] Plain HTTP is redirected to HTTPS
    ✅ got: #t
    [⚙️ assert] The ACME block answers for the Grafana name
    ✅ got: #t
  [🧪 test] The administrative vhosts are locked down
    [⚙️ assert] Grafana uses the strict request bucket
    ✅ got: #t
    [⚙️ assert] Prometheus uses the strict request bucket
    ✅ got: #t
    [⚙️ assert] The node exporter uses the strict request bucket
    ✅ got: #t
    [⚙️ assert] Grafana caps connections per address
    ✅ got: #t
    [⚙️ assert] and not with the application bucket
    ✅ got: #f
    [⚙️ assert] The node exporter is no longer public
    ✅ got: #t
    [⚙️ assert] Grafana has a lock in front of its login
    ✅ got: #t
    [⚙️ assert] An admin vhost takes no large uploads
    ✅ got: #t
  [🧪 test] The global block carries the tuning and the zones
    [⚙️ assert] sendfile is on
    ✅ got: #t
    [⚙️ assert] The proxy rate-limit zone exists
    ✅ got: #t
    [⚙️ assert] The static rate-limit zone exists
    ✅ got: #t
    [⚙️ assert] The WebSocket map is defined
    ✅ got: #t
    [⚙️ assert] The nginx version is hidden
    ✅ got: #t
  [🧪 test] A static vhost applies the shared security baseline
    [⚙️ assert] HSTS is sent
    ✅ got: #t
    [⚙️ assert] A CSP is sent
    ✅ got: #t
    [⚙️ assert] nosniff is sent
    ✅ got: #t
    [⚙️ assert] HTTP/2 is enabled
    ✅ got: #t
    [⚙️ assert] Per-IP connections are capped
    ✅ got: #t
    [⚙️ assert] with-listdir? turns the index on
    ✅ got: #t
    [⚙️ assert] with-rate-limit? #f removes the request cap
    ✅ got: #f
  [🧪 test] The static sites serve from /srv/http
    [⚙️ assert] A static site is rooted at its own name
    ✅ got: /srv/http/jointhefreeworld.org
    [⚙️ assert] A site may point somewhere else
    ✅ got: /srv/symbanta.jointhefreeworld.org
  [🧪 test] Restricted vhosts carry basic authentication
    [⚙️ assert] torikomi is password protected
    ✅ got: #t
  [🧪 test] The access log is nginx's logfmt format
    [⚙️ assert] The capability selects its own log format
    ✅ got: wolk
    [⚙️ assert] Request time is logged
    ✅ got: #t
    [⚙️ assert] Upstream response time is logged
    ✅ got: #t
    [⚙️ assert] The status code is logged
    ✅ got: #t
  [🧪 test] Minimal mode keeps only the ACME and metrics vhosts
    [⚙️ assert] A new machine gets two blocks
    ✅ got: 2
    [⚙️ assert] A configured machine serves seventeen
    ✅ got: 17
  [🧪 test] Upstreams are on loopback with keepalive
    [⚙️ assert] No upstream leaves the machine
    ✅ got: #t
    [⚙️ assert] Every request/response upstream keeps connections alive
    ✅ got: #t
    [⚙️ assert] The streaming upstream is not pooled
    ✅ got: #t
    [⚙️ assert] byggsteg listens where its daemon does
    ✅ got: (127.0.0.1:50002)
  [🧪 test] Every vhost has a server block
    [⚙️ assert] The count is the one the README documents
    ✅ got: 17
    [⚙️ assert] jointhefreeworld.org is served
    ✅ got: #t
    [⚙️ assert] byggsteg is served
    ✅ got: #t
    [⚙️ assert] torikomi is served
    ✅ got: #t
    [⚙️ assert] symbanta is served
    ✅ got: #t
  [🧪 test] A static vhost serves its own directory
    [⚙️ assert] with-www? #f leaves the bare name alone
    ✅ got: (example.test)
    [⚙️ assert] The root is the site directory under /srv/http
    ✅ got: /srv/http/example.test
    [⚙️ assert] The certificate follows the name
    ✅ got: /etc/letsencrypt/live/example.test/fullchain.pem
    [⚙️ assert] TLS and QUIC are both served
    ✅ got: (443 ssl 443 quic)
ℹ️ info - ran 1 tasks in: 0 seconds, 40936 microseconds;;; note: source file ./test/veritas/unit/nginx-spec.scm
;;;       newer than compiled /root/.cache/guile/ccache/3.0-LE-8-4.7/etc/byggsteg/job-clone/wolk-jjba/trunk/test/veritas/unit/nginx-spec.scm.go
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-auto-compile argument to disable.
;;; compiling ./test/veritas/unit/nginx-spec.scm
;;; compiled /root/.cache/guile/ccache/3.0-LE-8-4.7/etc/byggsteg/job-clone/wolk-jjba/trunk/test/veritas/unit/nginx-spec.scm.go


running unit tests from file: iptables-spec.scm, entrypoint: spec


ℹ️ info - shuffled with seed 441518405 (replay with VERITAS_SEED=441518405)
+----------------------------------------------+
[📑 suite] Firewall Suite
  [🧪 test] IPv6 mirrors IPv4
    [⚙️ assert] IPv6 serves HTTPS
    ✅ got: #t
    [⚙️ assert] IPv6 serves SSH
    ✅ got: #t
    [⚙️ assert] IPv6 accepts ICMPv6
    ✅ got: #t
    [⚙️ assert] IPv6 does not expose PostgreSQL
    ✅ got: #f
  [🧪 test] The served ports are open
    [⚙️ assert] SSH is on 2323
    ✅ got: #t
    [⚙️ assert] HTTP is open for ACME challenges
    ✅ got: #t
    [⚙️ assert] HTTPS is open
    ✅ got: #t
    [⚙️ assert] HTTP/3 is open
    ✅ got: #t
    [⚙️ assert] Icecast is open
    ✅ got: #t
    [⚙️ assert] Icecast sees UDP sources
    ✅ got: #t
    [⚙️ assert] Minecraft is open
    ✅ got: #t
    [⚙️ assert] Minecraft sees UDP
    ✅ got: #t
  [🧪 test] Private services stay closed
    [⚙️ assert] PostgreSQL is not exposed
    ✅ got: #f
    [⚙️ assert] Grafana is reached through nginx only
    ✅ got: #f
    [⚙️ assert] Prometheus is reached through nginx only
    ✅ got: #f
    [⚙️ assert] The lucidplan upstream is not exposed
    ✅ got: #f
  [🧪 test] Both rule sets exist and are balanced
    [⚙️ assert] The IPv4 rule set is not empty
    ✅ got: #t
    [⚙️ assert] The IPv6 rule set is not empty
    ✅ got: #t
    [⚙️ assert] IPv4 commits exactly once
    ✅ got: 1
    [⚙️ assert] IPv6 commits exactly once
    ✅ got: 1
  [🧪 test] The rules are a default-deny set
    [⚙️ assert] The loopback is trusted
    ✅ got: #t
    [⚙️ assert] Replies to outbound connections are allowed
    ✅ got: #t
    [⚙️ assert] Everything else is rejected
    ✅ got: #t
    [⚙️ assert] IPv6 rejects with the IPv6 ICMP type
    ✅ got: #t
  [🧪 test] Hostile traffic is bounded before it reaches a service
    [⚙️ assert] Malformed packets are dropped, not parsed
    ✅ got: #t
    [⚙️ assert] New SSH connections are rate limited per source
    ✅ got: #t
    [⚙️ assert] Every served port has a per-source connection cap
    ✅ got: #t
    [⚙️ assert] ICMP echo is rate limited before it is accepted
    ✅ got: #t
    [⚙️ assert] and the excess is dropped, not queued
    ✅ got: #t
    [⚙️ assert] IPv6 drops malformed packets too
    ✅ got: #t
    [⚙️ assert] IPv6 caps connections per source
    ✅ got: #t
ℹ️ info - ran 1 tasks in: 0 seconds, 77535 microseconds;;; note: source file ./test/veritas/unit/iptables-spec.scm
;;;       newer than compiled /root/.cache/guile/ccache/3.0-LE-8-4.7/etc/byggsteg/job-clone/wolk-jjba/trunk/test/veritas/unit/iptables-spec.scm.go
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-auto-compile argument to disable.
;;; compiling ./test/veritas/unit/iptables-spec.scm
;;; compiled /root/.cache/guile/ccache/3.0-LE-8-4.7/etc/byggsteg/job-clone/wolk-jjba/trunk/test/veritas/unit/iptables-spec.scm.go


11 files in 4 worker processes: 11 passed, 0 failed in 322.62 seconds

finished running tests


--------------------------------------------------------------

Maak: the infinitely extensible command runner

Loading tasks from Maak file: /etc/byggsteg/job-clone/wolk-jjba/trunk/maak.scm

Executing Maak tasks: test

--------------------------------------------------------------

=> Running Maak task: test

  Run the Veritas unit-test suite, one process per test file.

Runs inside the manifest shell like the other Scheme tasks: the suite
imports (guix gexp) and (gnu services ...) and needs veritas, none of
which are in the machine's system profile, so a bare `guile' fails with
`no code for module (veritas runner)' -- which is what the Byggsteg CI
task `maak-test' used to hit.
inspect the job's lambda expression
(lambda()
  `((task . "maak-test")
    (project . "wolk-jjba")
    (branch-name . "trunk")
    (clone-url . "https://codeberg.org/jjba23/wolk-jjba.git")))