Starting on July 18 (eval [759526](https://cuirass.genenetwork.org/eval/759526)) cuirass.genenetwork.org started experiencing odd build failures.

[rekahsoft@fosstodon.org]

Starting on July 18 (eval [759526](https://cuirass.genenetwork.org/eval/759526)) cuirass.genenetwork.org started experiencing odd build failures. The first was the test suite of coreutils. This is explained (and resolved - for some value of resolved, by https://issues.guix.gnu.org/74203)

However, since working around this issue, there have been other test suites that have unexpectedly failed (namely `automake` and `go`), which prevent the system from being updated or new changes to be built

Investigation is ongoing

#guix

[rekahsoft@fosstodon.org]

I'm happy to report that as of 2025-08-21, cuirass.genenetwork.org is again building substitutes!

That being said, this was achieved with a workaround (using an automake substitute from ci.guix.gnu.org instead of building it locally - which currently unexpectedly fails).

I hope to submit a bug report with further details of the issue later in the week.

Thanks for your patience!

#guix

[rekahsoft@fosstodon.org]

cuiass.genenetwork.org is experiencing an issue, and is now down for maintenance.

I'll provide an update later today, but a quick summary of the latest issue is that btrfs metadata was exhausted, and is now being rebalanced. There may also be an issue introduced by the last update of the system, but its hard to tell without resolving the btrfs metadata out of space problem. Sorry for any inconveniences caused!

#guix

[rekahsoft@fosstodon.org]

Update on the status of cuirass.genenetwork.org - the system remains down, and will be for another few days

Why? - the btrfs root filesystem is in a degraded state preventing boot.

Prognosis - Luckily "btrfs Not tainted" is seen in logs, so a full recovery of the system is expected

What's next? - The instance of debian that was used to bootstrap guix will be used to repair the btrfs root; reboot, back online!

Thanks for your ongoing patience and understanding

#guix

[rekahsoft@fosstodon.org]

Update on the status of cuirass.genenetwork.org - the system is still down, but is expected to be back online in the next 24-48h!

After spending quiet some time attempting to recover from a ENOSPC btrfs issue, further complicated by a stuck/failed balance, I have opted to do a partial recovery to get the system back online. Luckily, the guix-publish and nginx caches were recovered, along with cuirass database state, guix-daemon signing keys, ssh host keys, etc..

[rekahsoft@fosstodon.org]

Update on the status of cuirass.genenetwork.org - system remains down, but is expected to be back online in the next 24-48h!

After spending quiet some time attempting to recover from a ENOSPC btrfs issue, further complicated by a stuck/failed balance, I have opted to do a partial recovery to get the system back online. Luckily, the guix-publish and nginx caches were recovered, along with cuirass database state, guix-daemon signing keys, ssh host keys, etc..

Thanks for your ongoing patience!

[rekahsoft@fosstodon.org]

Update on the status of cuirass.genenetwork.org - system remains down, but is expected to be back online in the next 24-48h!

After spending quiet some time attempting to recover from a ENOSPC btrfs issue, further complicated by a stuck/failed balance, I have opted to do a partial recovery to get the system back online. Luckily, the guix-publish and nginx caches were recovered, along with cuirass database state, guix signing keys, ssh host keys, ...

Thanks for your ongoing patience

#guix