A Fork, a Fire, and a Failure

The LoongFire team maintains LoongFire, a fork of the IPFire firewall distribution retargeted for the Loongson LoongArch64 architecture. It is a labor of love: a full Linux distribution built from scratch for a niche but growing CPU family. Everything was working—until one day it wasn't.

The trouble began after the LoongFire team upgraded the glibc in their toolchain from version 2.43 to 2.44. They did a clean rebuild of the entire toolchain, then a clean rebuild of the ISO. The resulting installer booted fine, but when it came time to find the installation medium—the USB drive the ISO had been written to—it simply couldn't. Instead of proceeding to install, the installer fell back to asking the user to "Download installation image."

The only thing that had changed was glibc. And bizarrely, the same toolchain with glibc 2.44 built an x86_64 ISO that worked perfectly. This led the LoongFire team to initially suspect the issue was loongarch64-specific—a reasonable assumption given that the failure appeared only on their target architecture while x86_64 builds were unaffected.

Filing the Report

On October 23, 2026, the LoongFire team filed Bug 34861 against glibc 2.44. They included the relevant IPFire installer code: hw_find_source_medium() walks block devices via libudev, skipping loop, fd, ram, and md devices, and testing each remaining candidate with hw_test_source_medium(). They explained what they knew: the only variable was glibc 2.43 → 2.44, the problem appeared to be architecture-specific, and reverting the glibc commit made it disappear.

The glibc maintainers—Adhemerval Zanella and Carlos O'Donell—responded promptly but cautiously. They agreed the report needed more detail: what exactly was failing, what symbols were involved, and where the behavior diverged. They suggested strace, gdb, valgrind, and—crucially—bisecting glibc between the 2.43 and 2.44 tags. Carlos noted there were 550 commits between the two releases, meaning roughly 9–10 build-and-test cycles to narrow it down.

The Bisect

The LoongFire team got to work. Each cycle was time-consuming: rebuild glibc, rebuild the toolchain, rebuild the ISO, write it to USB, boot a Loongson machine, and see whether the installer found the medium. But they persisted, and by October 25 they had it: commit 99303f3871, "io: Use gnulib fts implementation (BZ 22944, BZ 20331)."

The bisect log told a clear story. ISOs built before that commit were 515 MB and worked. ISOs built at or after it were 487 MB and failed. Something in the new fts implementation was causing dracut—which LoongFire, like IPFire, uses to build its initramfs—to silently drop files.

The LoongFire team connected the dots: LoongFire's ISO build invokes dracut --install "/usr/sbin/chroot" --force -a "installer" --strip ..., and they suspected this was related to Bug 34078, which involved dracut-install. They tested a simple dracut-ng fix from PR #2394 referencing that bug—and it resolved the issue.

Adhemerval confirmed the connection, and Bug 34861 was marked a duplicate of Bug 34078.

The Root Cause

The technical story is elegant and nasty in equal measure. glibc 2.44 switched its fts (file tree walk) implementation to the one from gnulib. The gnulib implementation honors the FTS_NOSTAT flag more strictly than the old one. dracut-install walks kernel module directories using FTS_NOSTAT, which tells fts not to bother calling stat() on entries. Under the old implementation, regular files were still reported as FTS_F. Under the new one, they come back as FTS_NSOK—"no stat information available"—and dracut-install's logic, which was looking for FTS_F, silently ignored every single one.

The result: dracut could still find the directories (e.g., drivers/net/ethernet or drivers/md), but it skipped every module file inside them. Only modules named explicitly—like e1000 or virtio_net—made it into the initramfs.

IPFire Picks Up the Thread

The LoongFire team had also informed the IPFire team, and on October 26, Michael Tremer filed IPFire Bug 14666. His analysis was thorough and alarming. IPFire ships dracut-ng 108, which lacks the upstream fix. And Core Update 205 was about to ship glibc 2.44.

The impact was severe:

• System initramfs would contain only 69 modules instead of hundreds. Missing would be MegaRAID (megaraid_sas), LSI/Broadcom SAS (mpt3sas), HPE Smart Array (hpsa), Adaptec (aacraid), Microsemi (smartpqi), Hyper-V storage (hv_storvsc), VMware PVSCSI, Xen block devices, software RAID (raid1, raid10, raid456), and SD/MMC. Any system whose root filesystem needed one of these might not boot after its initramfs was regenerated.
• The installer would contain only six network drivers (the ones named explicitly by the qemu-net module). Common NICs like e1000e, igb, ixgbe, i40e, ice, igc, and r8169 would be missing—meaning network installation would fail on most real hardware.

Michael provided a crisp reproducer inside the build chroot:

mkdir /tmp/x
/usr/lib/dracut/dracut-install -D /tmp/x --kerneldir /lib/modules/$(KVER) -m =drivers/net/ethernet
find /tmp/x -name "*.ko*" | wc -l

It returned 0 without the fix and 191 with it.

He initially prepared a backport of the upstream dracut-ng fix (commit 40e7af014b47, "fix(dracut-install): remove FTS_NOSTAT in install_modules() fts traversal"). But within half an hour, a better path emerged: Adolf had submitted an update to dracut 112 (patchwork 10498 and 10497), which already contains the upstream fix. No custom backport needed.

The Fix Lands

By the afternoon of October 26, Michael had applied Adolf's two patches ("dracut: Update to version 112 and move from dracut-ng to dracut" and "core206: Ship dracut"), plus removal of /usr/lib/dracut/dracut-init.sh and /usr/lib/dracut/dracut-version.sh, which dracut 112 no longer ships.

A full clean build confirmed success:

• System initramfs: 588 kernel modules (up from 69), now 54 MB instead of 44 MB. MegaRAID, mpt3sas, hv_storvsc, vmw_pvscsi, raid1, xen-blkfront, and the common network drivers were all back.
• Installer: all network and storage drivers restored.

But the fix revealed a secondary problem: with all modules found again, the installer's dependency on the drm dracut module pulled in every GPU driver and its firmware—NVIDIA, AMDGPU, i915—ballooning the installer from 56 MB to 192 MB. Since the installer only needs a text console (the kernel provides that with built-in simpledrm/efifb/vesafb), Michael added "installer: Don't add any graphics drivers," bringing it back to 55 MB with all drivers intact.

The Scariest Part

Perhaps the most important correction came in Michael's fifth comment. He had initially written that existing systems updated through Pakfire would still have an older glibc and thus be safe. He corrected himself: Core Update 205 ships glibc 2.44, and its update.sh runs dracut --regenerate-all. Every system that installed Core Update 205 with dracut-ng 108 would have received an initramfs missing most storage drivers. Systems whose root filesystem needed MegaRAID, MPT3SAS, HPE Smart Array, Adaptec, Microsemi, Hyper-V, VMware PVSCSI, Xen, or software RAID might not have booted afterward.

The fix was pushed to both origin/master (Core Update 205 now ships dracut 112) and origin/next, with the master branch build to follow.

What This Story Shows

A single upstream library change—an internal switch to a more standards-compliant fts implementation—propagated through a toolchain, into an initramfs builder, and produced a distribution that could install on some hardware but not others, and might have bricked bootable systems had it shipped. It was caught because the maintainers of a niche architecture fork noticed their installer couldn't find its own USB stick, filed a careful bug report, and spent days bisecting 550 commits.

The LoongFire team's initial suspicion that the bug was loongarch64-specific turned out to be wrong—but that suspicion was a natural starting point, and the bisect process is what proved the bug was architecture-independent. The glibc maintainers were right to ask for more data; the LoongFire team was right to provide it. And the IPFire team's rapid response—identifying the blast radius, preparing a backport, then pivoting to a cleaner upstream update—turned a potential disaster into a footnote.

一个分支、一团火焰、一次故障

LoongFire 团队维护着 LoongFire，这是 IPFire 防火墙发行版的一个分支，被移植到龙芯 LoongArch64 架构上。这是一个倾注心血的项目：一个为小众但不断成长的 CPU 家族从零构建的完整 Linux 发行版。一切本来运行良好——直到某一天不再如此。

问题始于 LoongFire 团队将工具链中的 glibc 从 2.43 升级到 2.44 之后。他们完整地重新构建了整个工具链，然后干净地重新构建了 ISO。生成的安装程序启动正常，但当需要查找安装介质——即写入 ISO 的 U 盘——时，它却找不到。安装程序没有继续安装，而是退回到提示用户"下载安装镜像"。

唯一改变的就是 glibc。而奇怪的是，同样的工具链配合 glibc 2.44 构建出的 x86_64 ISO 工作完全正常。这使 LoongFire 团队最初怀疑该问题是 loongarch64 特有的——考虑到故障只出现在他们的目标架构上而 x86_64 构建不受影响，这是一个合理的假设。

提交报告

2026 年 10 月 23 日，LoongFire 团队针对 glibc 2.44 提交了 Bug 34861。他们附上了 IPFire 安装程序的相关代码：hw_find_source_medium() 通过 libudev 遍历块设备，跳过 loop、fd、ram 和 md 设备，并用 hw_test_source_medium() 测试每个剩余候选设备。他们说明了自己掌握的情况：唯一的变量是 glibc 2.43 → 2.44，问题看起来具有架构特异性，回退该 glibc 提交后问题消失。

glibc 维护者——Adhemerval Zanella 和 Carlos O'Donell——迅速但谨慎地回应。他们一致认为报告需要更多细节：究竟什么失败了、涉及哪些符号、行为在哪里出现分歧。他们建议使用 strace、gdb、valgrind，以及关键的一点——在 2.43 和 2.44 标签之间对 glibc 进行二分查找。Carlos 指出两个版本之间有 550 个提交，意味着大约需要 9 到 10 轮构建与测试才能缩小范围。

二分查找

LoongFire 团队开始工作。每一轮都很耗时：重新构建 glibc、重新构建工具链、重新构建 ISO、写入 U 盘、启动龙芯机器，然后看安装程序是否能找到介质。但他们坚持了下来，到 10 月 25 日，他们找到了：提交 99303f3871，"io: Use gnulib fts implementation (BZ 22944, BZ 20331)"。

二分日志清晰地讲述了故事。在该提交之前构建的 ISO 为 515 MB 且工作正常。在该提交及之后构建的 ISO 为 487 MB 且失败。新的 fts 实现中的某些东西导致 dracut——LoongFire 与 IPFire 一样用它来构建 initramfs——静默丢弃了文件。

LoongFire 团队将线索串联起来：LoongFire 的 ISO 构建调用了 dracut --install "/usr/sbin/chroot" --force -a "installer" --strip ...，他们怀疑这与涉及 dracut-install 的 Bug 34078 有关。他们测试了引用该 bug 的 PR #2394 中的一个简单 dracut-ng 修复——问题解决了。

Adhemerval 确认了这一关联，Bug 34861 被标记为 Bug 34078 的重复项。

根本原因

这个技术故事既优雅又棘手。glibc 2.44 将其 fts（文件树遍历）实现切换为来自 gnulib 的版本。gnulib 实现比旧实现更严格地遵守 FTS_NOSTAT 标志。dracut-install 使用 FTS_NOSTAT 遍历内核模块目录，该标志告诉 fts 不必对条目调用 stat()。在旧实现下，常规文件仍被报告为 FTS_F。在新实现下，它们返回 FTS_NSOK——"无 stat 信息可用"——而 dracut-install 中寻找 FTS_F 的逻辑静默忽略了每一个文件。

结果：dracut 仍能找到目录（例如 drivers/net/ethernet 或 drivers/md），但跳过了其中的每个模块文件。只有显式命名的模块——如 e1000 或 virtio_net——才进入 initramfs。

IPFire 接棒

LoongFire 团队也通知了 IPFire 团队，10 月 26 日，Michael Tremer 提交了 IPFire Bug 14666。他的分析详尽且令人警觉。IPFire 发布的是 dracut-ng 108，缺少上游修复。而 Core Update 205 即将发布 glibc 2.44。

影响十分严重：

• 系统 initramfs 将只包含 69 个模块，而非数百个。缺失的将包括 MegaRAID（megaraid_sas）、LSI/Broadcom SAS（mpt3sas）、HPE Smart Array（hpsa）、Adaptec（aacraid）、Microsemi（smartpqi）、Hyper-V 存储（hv_storvsc）、VMware PVSCSI、Xen 块设备、软件 RAID（raid1、raid10、raid456）以及 SD/MMC。任何根文件系统需要其中之一的系统，在 initramfs 重新生成后可能无法启动。
• 安装程序将只包含六个网络驱动（由 qemu-net 模块显式命名的那些）。常见网卡如 e1000e、igb、ixgbe、i40e、ice、igc 和 r8169 将缺失——意味着网络安装在大多数真实硬件上会失败。

Michael 在构建 chroot 内提供了一个简洁的复现方法：

mkdir /tmp/x
/usr/lib/dracut/dracut-install -D /tmp/x --kerneldir /lib/modules/$(KVER) -m =drivers/net/ethernet
find /tmp/x -name "*.ko*" | wc -l

没有修复时返回 0，有修复时返回 191。

他最初准备了一个上游 dracut-ng 修复的回移（提交 40e7af014b47，"fix(dracut-install): remove FTS_NOSTAT in install_modules() fts traversal"）。但半小时内，出现了更好的路径：Adolf 提交了 dracut 112 的更新（patchwork 10498 和 10497），其中已包含上游修复。无需自定义回移。

修复落地

到 10 月 26 日下午，Michael 已应用 Adolf 的两个补丁（"dracut: Update to version 112 and move from dracut-ng to dracut" 和 "core206: Ship dracut"），并移除了 dracut 112 不再提供的 /usr/lib/dracut/dracut-init.sh 和 /usr/lib/dracut/dracut-version.sh。

一次完整的干净构建确认成功：

• 系统 initramfs：588 个内核模块（从 69 个增加），现在为 54 MB 而非 44 MB。MegaRAID、mpt3sas、hv_storvsc、vmw_pvscsi、raid1、xen-blkfront 以及常见网络驱动全部恢复。
• 安装程序：所有网络和存储驱动恢复。

但修复揭示了一个次要问题：由于所有模块再次被找到，安装程序对 drm dracut 模块的依赖拉入了所有 GPU 驱动及其固件——NVIDIA、AMDGPU、i915——使安装程序从 56 MB 膨胀到 192 MB。由于安装程序只需要文本控制台（内核通过内置的 simpledrm/efifb/vesafb 提供），Michael 添加了 "installer: Don't add any graphics drivers"，使其在包含所有驱动的情况下回到 55 MB。

最惊险的部分

或许最重要的更正出现在 Michael 的第五条评论中。他最初写道，通过 Pakfire 更新的现有系统仍将拥有较旧的 glibc，因此是安全的。他更正了自己：Core Update 205 发布 glibc 2.44，其 update.sh 运行 dracut --regenerate-all。每个安装了 Core Update 205 且使用 dracut-ng 108 的系统，都将收到缺少大多数存储驱动的 initramfs。根文件系统需要 MegaRAID、MPT3SAS、HPE Smart Array、Adaptec、Microsemi、Hyper-V、VMware PVSCSI、Xen 或软件 RAID 的系统，之后可能无法启动。

修复已推送到 origin/master（Core Update 205 现在发布 dracut 112）和 origin/next，master 分支的构建随后进行。

这个故事说明了什么

一个上游库的单一变更——内部切换到一个更符合标准的 fts 实现——通过工具链传播，进入 initramfs 构建器，产生了一个能在某些硬件上安装而在另一些硬件上不能的发行版，而且如果发布，可能会让可启动系统变砖。它之所以被捕获，是因为一个小众架构分支的维护者注意到他们的安装程序找不到自己的 U 盘，提交了一份细致的 bug 报告，并花了数天对 550 个提交进行二分查找。

LoongFire 团队最初怀疑 bug 是 loongarch64 特有的，这一怀疑最终被证明是错误的——但这个怀疑是自然的起点，而二分查找过程正是证明该 bug 与架构无关的关键。glibc 维护者要求更多数据是正确的；LoongFire 团队提供这些数据也是正确的。而 IPFire 团队的快速响应——识别影响范围、准备回移、然后转向更干净的上游更新——将一场潜在的灾难变成了一段脚注。