Open Firmware NetBoot with a third-party Ethernet card
Introduction
Have you ever wondered how an (old) Mac boots? Or have you wondered about Open Firmware, similar in purpose to the PC BIOS (though arguably far more elegant and refined), and the mysterious “ok prompt” command-line interface it provides, typically seen when something goes wrong during boot? Have you ever slogged through installing device drivers after adding hardware to your computer, wishing there was an easier way? In this post we’ll dive deep on these things and more as we explore the process of building an Open Firmware driver enabling a PowerPC Mac to boot over the network using a non-Apple network card powered by the RTL8139 chip.
The system we’ll be using for our exploration is a 1999 “Sawtooth” Power Mac G4, which I acquired for $25 from a government surplus auction site a while ago. As usual for government surplus, they removed the hard drive, so I had two choices to get this machine “working”: be practical and buy an adapter to get a hard drive or CF card to work, or be impractical and find something fun to do with this machine that doesn’t require buying a storage device. Impractical personal projects are often more fun for me, so I chose the latter.
NetBoot with a third-party NIC: Challenge accepted
Many computers are able to boot over a network connection without requiring a local storage device such as a hard drive, DVD, or USB flash drive. This is especially useful in enterprise environments where machines need to be able to boot and reboot without human hands touching them. In the PC world, this is typically called PXE (“pixie”) boot, where PXE stands for Preboot eXecution Environment, the name of a standard for this published by Intel with application to the PC platform. While modern (Apple Silicon) Macs can no longer boot over the network, Intel and PowerPC Macs were able to do so, and this was referred to as NetBoot, a brand name for the Apple-specific vendor extensions to industry standards.
The network side of these network booting protocols (as opposed to the platform side) often has the same basic process. The machine’s boot firmware will acquire its IP address and other connection settings via BOOTP or DHCP; the information received will also include the address of a TFTP server and a file path for an executable to load. The machine will download this executable over TFTP and launch it. Typically this is the initial bootloader, which will perform basic hardware configuration, load the next bootloader or the operating system kernel (perhaps in several pieces, and/or over a different protocol than TFTP like NFS, HTTP, etc), and transfer control to that to continue the boot process.
Of course, the various implementations of network booting all have something else in common: they need to be able to send and receive packets on the network. The boot firmware which runs on initial power-up is usually fairly simple and doesn’t by itself know how to control all the devices in the system. This means that it must be provided with drivers that tell it how to talk to the various devices and find one to boot from. This applies to all categories of devices – it needs drivers to control hard drives, USB mass storage devices, network cards, graphics cards, and any other devices the user wants to be able to boot from or use during boot. Because the boot firmware is its own environment and is quite stripped-down compared to a fully-fledged operating system, regular operating system device drivers can’t just be reused. Instead, firmware-only drivers are required, and this means that typically, only devices which reasonably need to be available during boot get these drivers written for them.
Unfortunately, it seems that no third-party PCI Ethernet NICs had Open Firmware drivers written for them that could be used to boot a Mac. The NetBSD diskless installation instructions state:
All NetBSD-supported Power Macs can boot over their built-in network interface. There are no reported cases of people being able to netboot over PCI, Cardbus, SCSI, or Airport network interfaces.
This makes practical sense, because who would bother trying to boot over a secondary network interface when the Power Macs already had built-in Gigabit Ethernet interfaces? Even if you wanted to have multiple interfaces in your Mac, it’s a perfectly practical restriction to only be able to boot from the built-in one. But as you may recall, I wanted this to be an impractical project, so I took a “challenge accepted” attitude and decided to figure out how to be perhaps the first person to boot a Power Mac over a network interface that wasn’t built-in.
PCI option ROMs
The general problem of users wanting to boot computers from non-built-in devices, or use such devices during boot, has existed for a while (for example, hard drive controllers or video cards), and the dominant solution at the time the Power Macs were introduced was the use of option ROMs (note: linked Wikipedia article is very PC-centric and doesn’t consider other firmware like Open Firmware, but the general concepts are similar). An option ROM is a memory chip included on the expansion card that contains, potentially among other things, the appropriate drivers for the host system’s firmware to use the device to boot. In the context of network cards, this is sometimes also called a “PXE ROM”. Option ROMs have been around for a while, dating back at least to the Video BIOS on the original IBM PC. There are specifications for the use of option ROMs in ISA and PCI(e) cards, and modern variants are still supported today by computers that have UEFI firmware.
Unfortunately, due to the low-level nature of boot firmware, the code contained on the option ROM must be specific to the host system’s firmware. For example, an option ROM designed to be used with a PC BIOS will contain native x86 real-mode code that hooks interrupts to provide BIOS services. An option ROM for UEFI will have a UEFI driver on it, most likely in native x86 code but very rarely in architecture-agnostic EFI bytecode (EBC). Neither of these types of option ROM will work in a system that uses Open Firmware, which expects architecture-agnostic Forth bytecode (FCode). This is the reason why there were historically “Mac” and “PC” versions of expansion cards – the main difference was in the contents of the option ROM because the systems used different firmware at boot. This is also why it wasn’t possible to boot a PowerPC Mac from a third-party network card – manufacturers simply didn’t bother writing option ROMs for Open Firmware because the market share was so small. Although such cards couldn’t be used for boot, they were often perfectly functional once the OS had booted with the installation of a driver for Mac OS 9 or a kernel extension (kext) for Mac OS X.
Open Firmware
Open Firmware started at Sun Microsystems in the early 1990s, where it was called OpenBoot and used as the boot firmware for SPARC workstations and servers. Sun decided to make it an open standard and it was standardized as IEEE 1275 in 1994. Open Firmware was used in systems from Sun, Apple, and IBM. I’ll be describing the implementation of Open Firmware used in so-called “New World” Macs, which are those with PCI buses as opposed to the prior NuBus. Previous PowerPC Macs did include a version of Open Firmware, but its role was different (in particular, it was not involved in choosing a startup device). “New World” Macs are implemented in near, but not complete, compliance with the PowerPC Common Hardware Reference Platform (CHRP), which dictates the expanded use of Open Firmware.
There are two major Open Firmware concepts relevant to this project: the device tree and the use of Forth bytecode (FCode).
The device tree
The device tree is a hierarchical description of the hardware installed in a particular system. It is generated by the firmware at boot time as the system probes the various expansion buses (PCI, ISA, USB, etc) for devices. The device tree also contains nodes that do not represent physical hardware but rather represent a grouping of information or behavior provided by the firmware (more on that later).
Here is a textual representation of the device tree from my Power Mac G4:
ff83b2e8: /cpus
ff83b518: /PowerPC,G4@0
ff83b890: /l2-cache
ff83c3b0: /chosen
ff83c540: /memory@0
ff83c758: /openprom
ff83c888: /client-services
ff83da28: /rom@ff800000
ff83dbb0: /boot-rom@fff00000
ff83dd30: /macos
ff83ddb0: /options
ff83de30: /aliases
ff83e848: /packages
ff83e8b0: /deblocker
ff83f1d0: /disk-label
ff83fbd0: /obp-tftp
ff846d00: /telnet
ff847580: /mac-parts
ff848658: /mac-files
ff84b468: /hfs-plus-files
ff8501d8: /fat-files
ff851f08: /iso-9660-files
ff852b10: /bootinfo-loader
ff8547b0: /xcoff-loader
ff8551c8: /pe-loader
ff855ba0: /elf-loader
ff8571d0: /usb-hid-class
ff8594e8: /usb-ms-class
ff85b6e0: /sbp2-disk
ff85d8d0: /ata-disk
ff85eb30: /atapi-disk
ff860208: /bootpath-search
ff866af8: /terminal-emulator
ff866b90: /firewire-disk-mode
ff884370: /psuedo-hid
ff8843f8: /keyboard
ff884a78: /mouse
ff884f90: /multiboot
ff891470: /diagnostics
ff8914d8: /tools-node
ff893138: /rtas
ff893338: /nvram@fff04000
ff893e00: /uni-n@f8000000
ff894048: /i2c@f8001000
ff894898: /cereal
ff894f48: /pci@f0000000
ff8f2500: /uni-north-agp@b
ff8f2770: /ATY,Rage128Ps@10
ff895ff0: /pci@f2000000
ff897060: /pci-bridge@d
ff8990f0: /mac-io@7
ff89a130: /interrupt-controller@40000
ff89a300: /gpio@50
ff89a3e8: /extint-gpio1
ff89a580: /programmer-switch
ff89a6b8: /escc-legacy@12000
ff89a8b0: /ch-a@12004
ff89aa30: /ch-b@12000
ff89abb0: /escc@13000
ff89adb8: /ch-a@13020
ff89b760: /ch-b@13000
ff89c078: /davbus@14000
ff89c2f8: /sound
ff89c9f8: /timer@15000
ff89cb88: /via-pmu@16000
ff89fcd8: /rtc
ff8a03c8: /power-mgt
ff91afa8: /usb-power-mgt
ff8a0630: /i2c@18000
ff8a0ec0: /cereal
ff8a1588: /ata-4@1f000
ff8a3300: /disk
ff8a3a08: /ata-3@20000
ff8a5780: /disk
ff8a5e88: /ata-3@21000
ff8a7c00: /disk
ff8a96f0: /ADPT,2930CU@3
ff8dd488: /disk
ff8de350: /tape
ff8df4a0: /pci10ec,8139@4
ff8df770: /usb@8
ff919f38: /hub@1
ff91a0c8: /keyboard@1
ff8e4e30: /usb@9
ff8ea4f0: /firewire@a
ff898070: /pci@f4000000
ff916268: /ethernet@f
The tree starts with nodes for the CPU, memory, and ROM. There are a bunch of nodes not representing physical hardware, including /chosen, /openprom, /options, /aliases, /packages (more on that in a moment), etc. Then there are nodes for the hardware installed in the computer, including the PCI buses and devices connected to them, devices connected via a PCI bridge, and more. A device tree node can be referred to using a hierarchical path from the root. For example, we might refer to the keyboard as /pci@f2000000/@d/usb@8/@1/keyboard@1, or equivalently /pci1/@d/@8/@1/@1. Each node has a set of properties that describe more information about that node.
For example, here are the properties for the built-in Ethernet adapter (/pci@f4000000/ethernet is one way to refer to this device):
vendor-id 0000106b
device-id 00000021
revision-id 00000000
class-code 00020000
interrupts 00000001
min-grant 00000040
max-latency 00000040
devsel-speed 00000002
fast-back-to-back
name ethernet
device_type network
network-type ethernet
removable network
category 6e657400
compatible gmac
built-in
address-bits 00000030
max-frame-size 000005ee
reg 00007800 00000000 00000000 00000000 00000000
02007810 00000000 00000000 00000000 00020000
02007830 00000000 00000000 00000000 00010000
stats 000000dd 00000000 00000000
local-mac-address 00306552 464c
assigned-addresses 82007810 00000000 f5200000 00000000 00200000
82007830 00000000 f5000000 00000000 00100000
The properties encompass identifying information, like PCI vendor and device ID, as well as information specific to the type of device it is (in this example, what kind of network card it is, and what its MAC address is). It also includes information about the memory ranges requested by and associated with the device (reg and assigned-addresses). These properties provide the information necessary to match drivers with the device and to set up the memory regions required to interface with it.
In Open Firmware, device tree nodes can also have code associated with them, called “methods” (similar to the object-oriented sense of the term). Methods are written in Forth bytecode (FCode), the subject of the next section.
Open Firmware uses the device tree to locate a boot device and load the bootloader from it. The bootloader generally prepares a “flattened” binary representation of the device tree (after adding additional properties like boot args) and hands it to the operating system kernel as it boots. The kernel then uses the device tree to inform its initial configuration of device drivers. The hierarchical nature of the device tree and its use for the firmware to describe hardware to the OS is similar in concept to ACPI in the x86 PC world, though the implementations are vastly different.
While the use of Open Firmware to boot computers has more or less ceased these days, the device tree lives on because it’s a sensible, straightforward, and flexible way to describe hardware. The modern Devicetree specification incorporates by reference the hardware-description parts of the IEEE 1275 device tree (nodes, children, properties, etc) but leaves out the associated Forth methods and the rest of the IEEE 1275 spec as it relates to Open Firmware. Devicetree is commonly used for booting embedded systems and it’s found wide use in the ARM ecosystem, where it’s one of the more standardized pieces in an otherwise fairly fragmented boot firmware landscape. Devicetree is used on modern smartphones and a variant of it is even used by Apple today for booting Macs with Apple Silicon chips.
Forth bytecode
As previously mentioned, device tree nodes in Open Firmware can have methods associated with them. These methods, implemented in Forth and compiled to a bytecode representation called FCode, can perform functions related to the device they are associated with. As you might expect from the hierarchical structure, methods can be inherited from parent nodes, and parent node implementations can be overridden or used as part of a child’s implementation in much the same way you’d override a method from a superclass while implementing a subclass. Indeed, many of the nodes in the tree above exist only for the purpose of organizing methods, such as /packages, which contains nodes for various boot services that Open Firmware provides (block device and filesystem support, executable loaders, TFTP-based network boot). Child nodes of /packages, such as /packages/obp-tftp, contain methods used to implement the functionality of that package, which can be delegated to from methods of other device tree nodes.
For example, the Open Firmware Recommended Practice: TFTP Booting Extension (a supplement to the main IEEE 1275 standard) prescribes open, close, and load methods for the /packages/obp-tftp node, with high-level behavior specified for each. The load method, for example, will do BOOTP/DHCP to get an IP address and also expects the BOOTP server to include information in the response indicating from where the bootloader can be loaded. load will then load the bootloader over TFTP into memory.
Device drivers
In order to boot the computer from a particular device, two relevant conditions must be met. First, Open Firmware must know how to talk to that device enough to load the bootloader and OS kernel (typically the bootloader makes use of the services that firmware provides for this, though not always). Second, the target operating system must know how to talk to the device enough to successfully load the root filesystem, which means that the OS kernel cannot rely on loading the device driver for the boot device from the root filesystem. It has to be either built in to the kernel or provided another way.
As discussed above in the “PCI option ROMs” section, the first condition (firmware must be able to talk to the device) is typically met using option ROMs or a similar mechanism, where devices can provide code that the firmware knows how to read and execute in a small ROM. Open Firmware adopts this mechanism, defining a standard for devices to bundle FCode (Forth) code in their option ROMs. Because FCode is an architecture-independent bytecode and Open Firmware FCode drivers are defined to use a platform-independent API, an FCode driver should work in any Open Firmware-compliant system, no matter its processor architecture or platform vendor. Open Firmware will find the FCode during the process of probing the PCI bus for devices, and will execute any code it finds before moving on to probe the next device. The code typically defines new methods on the device tree node being probed. These methods typically implement a device driver for the hardware, with all the necessary code to initialize the device and perform its basic functions. Once Open Firmware has chosen a device to boot from, it will use these methods (open, load, etc) to interface with the device and load the bootloader.
For example, here is the load method for the built-in Ethernet adapter whose properties were listed above (retrieved on my G4 by doing dev /pci@f4000000/@f to browse to the Ethernet adapter in the device tree and see load to decompile the load method). It’s very short; all it does is print some debug output before calling the load method from an instantiation of the obp-tftp package, which is a shared implementation of the desired network booting behavior.
: load
" enet: Load" 1 .enet-debug " load" obp-tftp $call-method
;
The second condition (target OS kernel must be able to interface with the device to load the root filesystem and continue boot) can be trickier to meet and different operating systems satisfy it in different ways. For Linux, you can compile all the required drivers directly into the kernel, or (more commonly), if using a generic kernel with loadable kernel modules for device drivers, require the bootloader to load an intermediate filesystem into RAM (initrd/initramfs). Windows and (at least older versions of) Mac OS X instead have the bootloader load both the kernel and any drivers that are specifically marked as required at boot time before beginning execution of the kernel. With Mac OS X, these essential drivers (kernel extensions or kexts) are bundled in an archive with extension .mkext, which is a so-called “kext cache” of multiple kexts. The bootloader then loads two files into RAM, the kernel itself and the .mkext kext cache.
While both of these methods can be implemented with Open Firmware, they have their downsides. Perhaps the largest is that, for a third-party device to be bootable, you first need a working OS installation to install the drivers and include them in the initramfs or .mkext cache so that the driver is available to use during the next boot. What if the device could “just work” right out of the box, without the user having to install any drivers? This is a level of “magic” seamlessness that we hardly see today, but it was very possible in the Open Firmware era.
Open Firmware defines a mechanism for devices to provide in their option ROMs not only an FCode driver to use while booting, but also one or more native code drivers, compiled for a specific host operating system. For example, a device could include an FCode driver for boot, a Solaris driver with native SPARC code, a Mac OS 9 driver with PowerPC code, and a PowerPC Mac OS X kernel extension in its option ROM (assuming there is space). This would enable the device to work immediately upon installation with any of those operating systems, with no additional driver installation required by the user!
As you might expect, the drivers are represented simply as additional properties on the device tree node, with the names signifying which OS they are for. For example, the Mac OS 9 driver property would be named driver,AAPL,MacOS,PowerPC, while the OS X driver would be named driver,AAPL,MacOSX,PowerPC. The value of each would be binary data containing the native driver code for that architecture and operating system combination. This approach is elegantly simple – the OS kernel just has to walk the device tree on startup (which it’s already doing) and load any drivers it finds. Devices can be truly self-contained and work without any user configuration required.
In my opinion, this is one of the features that sets Open Firmware apart from other boot firmware and makes it feel light-years ahead of what was available on other platforms at the time. There are certain downsides to this – for example, what if the OS changes driver formats and now you have to re-flash your card with a new driver? Broader industry changes, such as the adoption of standards like AHCI or NVMe which make device-specific drivers for certain classes of devices largely obsolete, also mean this isn’t really necessary these days. At the time, however, it represented a real innovation in the user experience.
My project: Roadblocks and solutions
As discussed above, I set myself the task of getting my Power Mac G4 to boot from a third-party network card, something not supported out of the box because my card did not have its own FCode option ROM with drivers. This meant I would need to write my own FCode driver for the RTL8139 chip. For the most part, the actual development of the driver methods proceeded much like writing a regular operating system device driver. Because of this and because the RTL8139 is relatively well-documented and many drivers already exist, I will spare the details of the driver development process in this post. Instead, I’ll focus on some of the unique aspects of Open Firmware and Forth development and share a few roadblocks I encountered and how I solved them.
RTL8139 configuration
The fact that the RTL8139 chip is so widely used, and therefore has several open-source drivers available (independent drivers in the BSDs and Linux, plus some other hobbyist drivers), was both a blessing and a curse for this project. Between the official documentation (which is easily obtainable on the web), the FreeBSD driver, the Linux driver, and hobbyist sources like the OSDev wiki, there were several different approaches laid out to correctly program the RTL8139. As a challenge, I wanted to write the driver logic myself rather than trying to directly port the logic from a well-known driver or some example code. My Open Firmware driver was also a bit different than the other drivers because Open Firmware doesn’t support interrupts – you must poll devices to check their status. The RTL8139 also has different options, like whether the ring buffer for packet reception should wrap around immediately (potentially splitting a received packet across two non-contiguous memory regions) or use some preallocated “overflow” space to allow the packet to go past the nominal end of the ring buffer and have the wrap to the beginning happen with the next received packet. I chose different options than the mainstream drivers because I thought it would make writing my driver in Forth more convenient, but this meant I was working without known-good examples. There are also quirks of the device, like the fact that certain values related to the ring buffer acknowledgments need to be written back with a value 16 bytes smaller than the value should intuitively be and other things like that.
All these quirks led to a bunch of bugs in my driver. I spent by far the longest time troubleshooting persistent issues with the receive side, where the ring buffer would become “de-synced” with my bookkeeping due to subtle timing issues as well as logic bugs. Sometimes log prints were load-bearing, while sometimes their presence broke things. Sometimes the buffer would fail to advance and the transfer would stall. Often these bugs happened after the transfer of a set number of packets or number of bytes. I worked my way through all this mostly by adding (or removing…) logging (I had to film the screen in slow motion with my iPhone camera to catch it all) and by taking and analyzing packet captures of the Ethernet traffic with tcpdump and Wireshark to understand what Open Firmware was sending and (supposedly) receiving on the wire.
For the sake of quick iteration, I sometimes wanted to start the logging only when I hit a certain TFTP block number in the transfer. This didn’t always happen after the same number of packets received due to variations in how many ARP packets got sent in the beginning and how many TFTP retries were going on. Inspecting the packets themselves wasn’t adequate because I wanted to track down issues potentially related to incorrect memory handling that meant received packets might not be making it to the Open Firmware-provided obp-tftp code. What I really wanted was a way to peer inside the Open Firmware-provided obp-tftp package’s internal state and get the last TFTP block number that it had processed.
This was one of the first situations where I began to truly appreciate the dynamism provided by Forth and the Open Firmware environment. Using completely supported operations, I was able to introspect the obp-tftp package and access its internal variables to ultimately get the memory address where it stored the last TFTP block number it had processed. I could then read from that address (with little performance impact) anytime I needed to check whether it was time to start logging. This is the sort of trick that’s only possible to do in a supported way with a dynamic language that has a separate runtime – something like this would be a lot more brittle and potentially undefined behavior, if it was even possible, in C. The incredibly dynamic nature of Forth and Open Firmware lends itself easily to rapid prototyping and debugging – you really can just reach into the runtime and read or even change stuff at will. It even invites comparisons to another famously dynamic part of the Mac ecosystem – the Objective-C language and runtime, where developers can swap out method implementations on objects they don’t control (method swizzling) and do all sorts of other nasty runtime tricks. The level of dynamism, while maintaining low-level control, that Open Firmware allowed was at the time and remains practically unheard-of in system firmware.
Starting with Mac OS 9 and pivoting
I originally wanted to boot Mac OS 9 on my Power Mac G4. The G4 is somewhat unique in that it’s the last Mac that can boot both Mac OS 9 and Mac OS X. I had never played with “Classic” Mac OS before, so I thought it would be fun to boot Mac OS 9. The target operating system was largely irrelevant to the bulk of my Open Firmware driver development and I figured I would basically be able to boot whatever I wanted once I had Open Firmware talking to the network card.
Unfortunately, this did not turn out to be the case with Mac OS 9. Once I got my Open Firmware driver to load the Mac OS ROM (the first code required for boot) over the RTL8139’s Ethernet connection, Mac OS booted to a black and white screen with a folder and flashing question mark, a sure sign that it couldn’t locate the device to boot from. This meant that I wasn’t meeting the second condition I discussed above for an operating system to boot from a particular device – the OS must know how to interface with the device so that it can load the root filesystem and continue boot.
I had obtained a Mac OS 9 driver for the RTL8139, and verified in a QEMU VM that it worked. Based on my reading of Apple’s official technical documentation from the time, Designing PCI Cards and Drivers for Power Macintosh Computers, it seemed like I just had to put this driver into a driver,AAPL,MacOS,PowerPC property on the Ethernet card’s device tree node, and Mac OS 9 should load it. Indeed, this was exactly how the Adaptec SCSI card that’s also installed in my G4 was set up – it had a property with that name that contained bytes for a PEF-format executable, the same thing I was putting in my property. I spent a while trying to determine if maybe Mac OS was failing to match the driver I was providing with the actual device tree node, perhaps due to some missing or incorrect property values on the node. I tried patching the driver I had to add some flags that indicated it was part of the boot process. I even extracted the compiled bootloader from the Mac OS ROM file and decompiled it in Ghidra, looking for any clues about what could be going wrong (this wasn’t very fruitful because Ghidra wasn’t able to identify where any strings were being used). I did find some debug flags, activated by setting an undocumented AAPL,debug property on the root device tree node, but none of the flags were especially helpful in understanding what was going wrong.
As a last-ditch attempt, I ended up poking through the Mac OS ROM some more, looking to see how the driver for the built-in Ethernet controller was exposed (the Mac OS ROM contains the drivers for some built-in hardware). To my surprise, while I did find some driver,AAPL,MacOS,PowerPC entries for other devices, I did not find one for the built-in gmac Ethernet controller! Instead, I found a similar-looking device tree property name, lanLib,AAPL,MacOS,PowerPC – lanLib instead of driver. After some research, it turns out that this lanLib is required to use network cards early in Mac OS boot, before the Open Transport subsystem is fully available. Unfortunately, no documentation I could find mentioned the existence of this property or how one might create a lanLib, and the examples in Designing PCI Cards and Drivers for Power Macintosh Computers explicitly mentioned network cards as something that could participate in boot and benefit from a driver,AAPL,MacOS,PowerPC property, so I was kind of stuck.
Though the need for a lanLib meant that the idea of booting Mac OS 9 was pretty much doomed, I did find something else while I was searching for answers related to this: evidence that Mac OS X did, at one point, support loading device drivers from the device tree, via a driver,AAPL,MacOSX,PowerPC (note the MacOSX) property. Since I was out of options to boot Mac OS 9, I decided to pivot to Mac OS X, which I figured would be easier because more of it is open source and I’d be able to see what was going on.
BootX null pointer crash
After obtaining an installer for Mac OS X Server 10.2 (Panther), I set up a quick NetBoot 2.0 server on my Xserve running Mac OS X Server 10.6 (Snow Leopard). This took a little bit of hacking because Snow Leopard doesn’t let you make a PowerPC image with the GUI, but I found a helpful blog post that laid out the steps to create a valid image manually. The first part of network booting Mac OS X is loading the bootloader (boringly named “BootX”) over TFTP, and then BootX will use the services provided by Open Firmware (including any Forth device drivers) to load the Mac OS X kernel and kext cache over TFTP as well. I got BootX to load and run fairly easily, but it failed pretty early on with only a cryptic message logged to the OF console:
DEFAULT CATCH!, code=300 at %SRR0: 01c02708 %SRR1: 00003030
This is basically Open Firmware-speak for “segmentation fault”: accessing unmapped memory. DEFAULT CATCH is what the top-level error handling code in Open Firmware prints when it gets an error not caught anywhere else. Code 300 is an issue related to memory addressing or access, and %SRR0 and %SRR1 are PowerPC architectural registers (Save/Restore Register 0 and 1). SRR0 contains the address of the instruction that caused the fault and SRR1 stores the processor’s Machine State Register (containing things like interrupt status, big/little endian execution mode, etc). These registers are used to restore the processor state after the exception is handled (the return-from-interrupt instruction automatically restores their contents back to the appropriate registers).
Unfortunately, no text output was printed from BootX before it crashed and burned, so it was pretty difficult to tell what the problem was. Thankfully, Apple has released BootX as open source and the code is surprisingly easy to follow. I began tracing the boot logic, looking for any operations that could cause a failure or a null pointer dereference. Unfortunately, there are a lot of things that need to work correctly before BootX can print to the screen, and it seemed like one of those was going wrong. One nice thing that made this easier to debug was the fact that Open Firmware’s “default catch” logic cleans up and drops you back at the Open Firmware prompt so you can inspect the state of the machine and device tree. I was able to use this to somewhat narrow down where in BootX it was failing, based on the presence of certain device tree properties that BootX adds as it progresses.
Analysis of the code for the steps after the last known-working spot wasn’t proving very fruitful. I was only able to establish for sure that it was getting through basics like allocating some chunks of memory. I decided to try a different approach. The BootX binary is in XCOFF format, so I decoded the XCOFF header from the binary to find the address where the .text section is loaded into memory (0x01c00000, which is 0x2708 bytes before the instruction that faulted). From there, I used a Ghidra extension to open the binary and find out exactly which instruction would be in memory at 0x01c02708, the address of the fault. It turns out that this is in the middle of the GetBootPaths function (source), which copies the boot device path from the /chosen device tree node and parses it to find the : character, which separates the device path from any arguments (like file path) coming after it. It goes in a loop, each time reading one more byte from the path string, until it finds the : character. It became immediately obvious that the reason this was crashing was that my device path didn’t actually have a : character in it, so the loop eventually started reading unmapped memory and crashed. Open Firmware was fine without the : but it seems that BootX assumed it would always be there. Correcting my boot device path from /pci@f2000000/@d/@4 to /pci@f2000000/@d/@4: solved the issue.
Again, the elegant design of Open Firmware saved the day here: it’s running with virtual memory mappings already enabled and is high-level enough to have a default exception handler installed and drop you back to an interactive prompt after helpfully printing the exact place the fault occurred. On an x86 PC from the same time with its BIOS and bootloader running in real mode, this would just loop forever, overflowing the loop counter, or worse, find a random 0x3a (:) byte somewhere far away in memory and end up with an extraordinarily large string, which would likely cause memory corruption when copied somewhere else. Without Open Firmware telling me exactly which instruction faulted, it would have taken me much, much longer to figure out what was going wrong.
Monkey-patching the memory map
Once I had BootX working, I got it to load the OS X kernel properly. However, it failed loading the .mkext kext cache file with a very interesting failure pattern. It seemed that no matter what I did, I could only transfer about 4 MiB over TFTP before the transfer failed. This limit was shared across BootX, the Mac OS X kernel, and the mkext – during a normal boot, after BootX and the kernel had been loaded, I could only load about 300-350 KiB of the mkext before it failed. However, if I had the BOOTP server tell Open Firmware to load the mkext first (this wouldn’t work for booting, I just did it to test), I could load closer to 4 MiB of the file. I spent a lot of time trying to find problems with my driver and changed some things that improved its operation so that I got the TFTP transfer to consistently fail at the same point every time. It was always failing on block 8193, which is one more than a power of 2 (8192) and because TFTP has 512-byte blocks, it was always failing exactly at 4 MiB, indicating some kind of limit was being hit.
At this point, I once again took advantage of the dynamism of Open Firmware. Because the user-facing command prompt uses Forth just like the FCode drivers, I was able to use the evaluate word to run user commands from inside my driver. While the IEEE 1275 spec does not technically require this to be supported, and there is probably a supported way to do this, it does work on the Apple Open Firmware so I left it as-is. I changed my driver to print the properties of the memory device (" dev /memory .properties" evaluate) right after downloading block 8192 (before it would fail on 8193) so that I could see what was allocated. Sure enough, the available property read:
0x3000 0x1ffd000 0x2400000 0x4da00000
This means there’s 0x1ffd000 bytes free starting at 0x3000 and 0x4da00000 bytes free starting at 0x240000. In other words, 0x20000000 to 0x2400000 was allocated, a span of 4 MiB. 0x2000000 was the value I’d set for load-base, the location where the load command will place whatever binary it loads, so it was clear that only 4 MiB had been allocated for loading (the obp-tftp package’s load method eventually ends up calling the main load word, which is responsible for allocating this space). I’m not entirely sure what the built-in drivers are doing to get around this limitation – perhaps something like the solution I came up with or perhaps they’re doing their own load implementation entirely. After finding the problem, the solution was simple: just do some monkey-patching to allocate more space after the 4 MiB. I added the following code to my own driver’s load method:
\ Claim 8 MiB of physical memory at 4 MiB after load-base (total 12 MiB)
" dev /memory load-base 400000 + 800000 0 claim" evaluate drop
\ Map this memory 1:1 to a virtual address (so it's both physically and virtually contiguous with the default 4 MiB)
" dev /cpus/@0 load-base 400000 + dup 800000 10 map" evaluate
Once again, there’s probably a supported way to do this using only commands that are supposed to be available to device drivers, not user interface commands, but this works fine. This allocates an additional 8 MiB of space contiguous with the 4 MiB allocated before, for a total of 12 MiB. This allocation is in physical address space. Then it creates a 1:1 virtual:physical mapping to make the allocated memory available in virtual address space as well, so that the executing code sees one contiguous 12 MiB region and the load doesn’t try to write into unmapped memory and fail early. After this, it would happily load BootX, the kernel, and the mkext cache in their entirety, and Mac OS X began to boot for the first time.
It didn’t get very far, though, because while the device tree told it that the chosen boot device was the RTL8139 card, it didn’t have a driver for the card so couldn’t read from it. The solution should be simple, right? I figured I would just allocate some memory (<driver size> alloc-mem) and then use encode-bytes to put the .kext file I had downloaded into the driver,AAPL,MacOSX,PowerPC property on the device tree node so Mac OS X would find it. Unfortunately, that won’t work for two reasons: first, it turns out that since a .kext is a directory, there’s no way to fit the entire kext into the property, so Mac OS X is expecting a .mkext file in the property, which is a single compressed archive. This was an easy fix – I found docs on the .mkext format from the Snow Leopard 10.6 kernel and wrote a Python script to turn my .kext into a .mkext (version 1 format). The second issue is that alloc-mem has a limit that’s smaller than the size of the .mkext file, so we can’t use that to allocate memory. The solution to this one is also simple: I did a similar claim and map process to map myself some memory directly for the driver.
This was the last roadblock in the project. Once I had the space allocated for the kext and the property in the device tree, Mac OS X found and loaded it and was able to use the RTL8139 to load the rest of the drivers from the NetBoot server over NFS and boot to the Mac OS X GUI.
Conclusion
This was a fun, impractical, and rewarding project. It’s the only example I can find of a third-party network card being used to boot a PowerPC Mac with complete “out of the box” driver support, and it feels cool to have done something that few to no people have done before, and (almost) certainly that nobody has publicly documented. Like with most of my projects, I learned so much with this one: a deep dive into an elegantly designed firmware system that was miles ahead of what the PC had at the time; a basic network card driver; details on the Ethernet, IP, BOOTP, DHCP, TFTP, and ARP protocols; and the guts of a bootloader, among many other things.
It was all great fun to learn and tinker with. This is one of my favorite things to do with my software skills: use them to take apart the layers of abstraction and find out how something works. Computers can be understood, and all it takes to understand how these wonderfully complex machines work are some software and logical reasoning skills, sufficient curiosity and motivation, and a whole lot of free time. I think that’s really cool.
On the off-chance that you’d like to look at or play around with my driver, it’s available on GitHub.