%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Copyright (c) 2013, ETH Zurich. % All rights reserved. % % This file is distributed under the terms in the attached LICENSE file. % If you do not find this file, copies can be found by writing to: % ETH Zurich D-INFK, Universitaetstrasse 6, CH-8092 Zurich. Attn: Systems Group. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \documentclass[a4paper,twoside]{report} \usepackage{bftn} \usepackage{booktabs} \usepackage{hyperref} \usepackage{hyphenat} \usepackage{listings} \usepackage{makeidx} \usepackage{natbib} \usepackage{xspace} \def\chapterautorefname{Chapter} \def\sectionautorefname{Section} \def\subsectionautorefname{Section} \def\subsubsectionautorefname{Section} \def\tableautorefname{Table} \def\qemu{QEMU\xspace} \lstdefinelanguage{armasm}{ numbers=left, numberstyle=\tiny, numbersep=5pt, basicstyle=\ttfamily\small } \lstset{language=armasm} \title{Barrelfish on ARMv8} \author{David Cock} \tnnumber{022} % give the number of the tech report \tnkey{ARMv8} % Short title, will appear in footer \begin{document} \maketitle \begin{versionhistory} \vhEntry{1.0}{11.04.2016}{DC}{Initial version} \end{versionhistory} \chapter{Summary} Barrelfish now supports ARMv7 and ARMv8 as primary platforms, and we have discontinued support for all older architecture revisions (ARMv5, ARMv6). The current Barrelfish release contains a port to a simulated ARMv8 environment, derived from the existing ARMv7 codebase and running under GEM5, with generous contributions from HP Research. Simultaneously, we are undertaking a clean-slate redesign of the CPU driver for ARMv8, as it presents a number of novel features, and greatly improved platform standardisation (\autoref{s:sbsa}), that should allow for a much cleaner and simpler implementation. This redesigned CPU driver will for the basis for ongoing research into large-scale non-cache-coherent systems using ARMv8 cores. This document presents the new CPU driver design (\autoref{c:design}), briefly covering those features of ARMv8 of greatest relevance (\autoref{c:background}), and discusses a number of technical challenges presented by the new architecture (\autoref{c:tech}). \chapter{Background}\label{c:background} The Barrelfish research operating system is a vehicle for research into software support for likely future architectures, where large numbers of non-coherent (or weakly-coherent) heterogeneous processor cores are assembled into a single large-scale system. As such, support for a common non-x86 architecture has always been part of the project, beginning with the ARMv5 (XScale) port, which permitted the embedded processor on a network interface card to be integrated as a first-class part of the system, with its own CPU driver. We have also actively maintained an ARMv7 port, to the OMAP4460 processor on the Pandaboard ES, which we use as a teaching platform in the Advanced Operating Systems course at ETH Z\"urich. These ports are described more fully in the accompanying technical report, \citet{btn017-arm}. \section{The ARMv8 Architecture} The ARMv8 architecture is quite a radical departure from previous versions, and represents the culmination of a trend that has been developing for quite some time. While the first wave of ARM-based microservers, based on the 32-bit ARMv7 architecture, was largely a commercial failure, it's clear that ARM is now actively targeting the server market, where Intel currently has near-total dominance. ARMv8 discards some long-standing features of the ARM instruction set: universal conditional execution, multiple loads/stores, and the program counter as a general-purpose register. These most likely caused difficulty in scaling the processor pipeline to high clock rates, and we present some consequences of their loss in \autoref{s:threads} and \autoref{s:traps}. The instruction-set changes, however challenging to the systems programmer, are ultimately of little consequence compared to the consolidation of the ARM ecosystem into a serious server platform. The two features of most interest at this stage in the design process are the standardisation of hardware features and memory maps, and of the boot process. \subsection{ARM Server Base System Architecture}\label{s:sbsa} ARM has long been criticised by systems programmers for its highly fragmented, and non-uniform programming interface. Linux, in particular, has struggled for years with supporting the great multiplicity of ARM platforms. The principal reason for this is the lack of any concept of a \emph{platform}: a set of assumptions (available hardware, memory map, etc.), that programmers can rely on when initialising and managing a system. The Linux source tree famously contained a vastly greater amount of code in the ARM platform support subtrees, than that for x86. The relative standardisation of the x86 platform is largely a historical accident, due to the rapid proliferation of PC/AT clones in the early 1980s. The x86 platform thus contains layers of ossified legacy interfaces, necessary to ensure broad cross and backward compatibility. ARM's business model, on the contrary, has long emphasised the specialisation of implementations: an ARM licensee would take their ARM-designed CPU core, and integrate it themselves in to a complex SoC (system on a chip), with their own specialised, proprietary interfaces. The upsides of this were the possibility to highly optimise a particular design, and no requirement on ARM itself to maintain a coherent platform. While ARM's customisable platform worked well for embedded devices, and scaled reasonably well to relatively powerful smartphones, it's a disaster for producing high-quality systems software, able to execute on a broad range of hardware from competing vendors: exactly what a competitive server platform requires. ARM clearly know this, and since 2014 have published the Server Base System Architecture \citep{arm:sbsa}. To the extent that manufacturers adhere to these guidelines, our job as systems programmers is significantly simpler: it should be possible to write a single set of initialisation and configuration code for ARMv8, that will run on any SBSA-compliant system, much as we already do for x86-64. \begin{table} \begin{center} \begin{tabular}{lllp{6cm}} \toprule Supplier & Processor & Name & \\ \midrule APM & APM883208 & Mustang & 1P 8-core X-Gene 1 with serial trace. \\ \addlinespace[2pt] & APM883408-X2 & X-C2 & 1P 8-core X-Gene 2. \\ \addlinespace[2pt] Cavium & CN8890 & StratusX & 1P 48-core ThunderX. \\ \addlinespace[2pt] & & Cirrus & 2P 48-core ThunderX. \\ \addlinespace[2pt] ARM & AEM & Fixed Virtual Platform & The \emph{architectural envelope model} covers the range of behaviour permitted by ARMv8. Bare-metal debug. \\ \addlinespace[2pt] & & Foundation Platform & Freely available, compatible with FVP. \\ \bottomrule \end{tabular} \end{center} \caption{ARMv8 platforms of interest}\label{t:platforms} \end{table} Our target platforms listed in \autoref{t:platforms} all support SBSA to some extent, and absent any compelling reason, we will only support SBSA-compliant platforms. \subsection{UEFI}\label{s:uefi} One aspect of the SBSA which eases portability is the specification, for the first time, of a boot process for ARM systems. ARM has specified that SBSA systems must support UEFI \citep{uefi} (the unified extensible firmware interface). UEFI is a descendant of the EFI specification, developed by Intel for the Itanium project. While Itanium is no longer a platform of any great commercial interest, UEFI support is now widespread in the x86-64 market. UEFI, in turn, specifies the use of ACPI \citep{acpi} (the advanced configuration and power interface) for platform discovery and control. Supporting ACPI and UEFI requires a one-off investment of effort to design a new boot and configuration subsystem, but should pay off in the long term, as ports to new ARM boards will no longer require extensive manual configuration. The code should also be largely reusable for x86 UEFI systems. Our new UEFI bootloader is described in \autoref{s:hagfish}. \section{A Direct Port from ARMv7} As already described, the Barrelfish release current at time of writing includes an initial ARMv8 port to the GEM5 simulator. This port contains code generously contributed by HP Research. Being developed from the existing codebase, this ARMv8 port follows the structure of the existing ARMv7 code closely. While it is highly useful to have a running port, we are nevertheless continuing with a significant redesign of the CPU driver, as significant improvements and simplifications will be possible, once we no longer need to follow the existing structure, originally developed for a significantly different platform. The GEM5 simulator's model of an ARMv8 platform is relatively primitive, and does not conform to modern platform conventions, for example placing RAM at address \texttt{0}, rather than \texttt{0x80000000} as mandated by the SBSA. For this reason, in addition to better integration with ARM debugging tools, we have switched to the ARM Fixed Virtual Platform as our default simulation environment, with the Foundation Platform supported as a freely available simulator. \section{Registers} \subsection{General-purpose Registers} In total there are 31+1 general purpose registers (\texttt{r0-r30}) of size 64bits(\autoref{tab:registers}). They are usually referred to by the names \texttt{x0-x30}. The 32-bit content of the registers are referred to as \texttt{w0-w30}. The additional stack pointer \texttt{SP} register can be accessed with a restricted number of instructions. \begin{table}[!h] \begin{center} \begin{tabular}{lll} \textbf{Register} & \textbf{Special} & \textbf{Description} \\ \hline \texttt{X0-X7} & Caller-save & function call arguments and return value \\ \texttt{X8} & & indirect result e.g. location of large return value (struct) \\ \texttt{X9-X15} & Caller-save& temporary registers \\ \texttt{X16} & IP0 & The first intra-procedure-call scratch register\footnote{can be used by call veneers and PLT code; at other times may be used as a temporary register. Same for X17}\\ \texttt{X17} & IP1 & The second intra-procedure-call temporary register \\ \texttt{X18} & & The Platform Register (TLS), if needed; otherwise a temporary register. \\ \texttt{X19-X28} & Callee-save & need to be preseved and restored when modified\\ \texttt{X29} & FP & frame pointer \\ \texttt{X30} & LR & link register \\ \texttt{SP} & & stack pointer (XZR) \\ \end{tabular} \caption{ARMv8 General purpose Registers} \label{tab:registers} \end{center} \end{table} \paragraph{Procedure call} \begin{itemize} \item The registers \texttt{x19-28} and \texttt{SP} are callee-saved and hence must be preserved by the called subroutine. All 64 bits have to be preserved even when executing in the 32-bit mode. \item The registers \texttt{x0-x7} and \texttt{x9-x15} are caller saved. \item During procedure calls the registers \texttt{x16}, \texttt{x17}, \texttt{x29} and \texttt{x30} have special roles i.e.\ they store relevant addresses such as the return address. \item Arguments for calls are passed in the registers \texttt{x0-x7}, \texttt{v0-v7} for floats/SIMD and on the stack \end{itemize} \paragraph{Indirect result} This register is used when returning a large value such as declared by this function: \texttt{struct mystruct foo(int arg);}. \paragraph{Platform specific} The use of register \texttt{x18} is platform specific and needs to be defined by the platform ABI. This register can hold inter-procedural state such as the thread context. \paragraph{Linker} The registers \texttt{IP0} and \texttt{IP1} can be used by the linker as a scratch register or to hold intermediate values between subroutine calls. \subsection{SIMD and Floating point} There are 32 registers to be used by floating point and SIMD operations. The name of those registers will change, depending on the size of the operation. \begin{table}[!h] \begin{center} \begin{tabular}{ll} \textbf{Register} & \textbf{Description} \\ \hline \texttt{v0-v7} & function call arguments, intermediate values and return value, caller save registers \\ \texttt{v8-v15} & Callee-save registers. They need to be preserved\\ \texttt{v16-v31} & Caller-save registers \end{tabular} \caption{ARMv8 General purpose Registers} \label{tab:registers} \end{center} \end{table} \chapter{Design and Implementation}\label{c:design} \section{Redesigning the CPU Driver} Given that ARMv8 is a significantly different platform to ARMv7, and that the ARMv7 codebase carries a significant legacy, reaching right back to ARMv5, we are pursuing substantial redesign of the CPU driver. Taking advantage of the standardisation of the hardware platform mandated by the SBSA (\autoref{s:sbsa}), and the facilities provided by UEFI (\autoref{s:uefi}), in addition to a relatively unrestricted virtual address space, we are able to significantly reduce the complexity of the CPU driver. In this section we describe the updated design, and our progress on its implementation, while the UEFI interface (Hagfish) is described separately, in \autoref{s:hagfish}. \paragraph{Terminology} In the interest of clarity, in the discussion that follows, we use a few terms with precise intent: \begin{description} \item[shall] indicates features or characteristics of the design to which the Barrelfish implementation must conform. \item[should] indicates features which should be supported if at all possible. \item[initially] indicates features which will be provided from the outset in the Barrelfish implementation. \item[eventually] indicates features which will be provided later in the Barrelfish implementation, and which the initial design will aim to facilitate. \end{description} \subsection{Goals} Our goal is to provide a reference design for the CPU driver and user-space execution environment for Barrelfish on an ARMv8 core, in order to understand both positive and negative implications of the architecture for a multikernel system. The design \textbf{should} be applicable to any ARMv8 with virtualisation (\texttt{EL2}) support. \textbf{Initially}, our hardware development platform is the APM X-Gene 1, using the Mustang Development Board. We are using the Mustang principally as it was relatively easily available, as well as being a comparatively complex and powerful CPU. The ThunderX platform from Cavium is very interesting for Barrelfish, as it ties a large number (48) of less-powerful (2-issue) cores. We do not have the resources to develop for two platforms simultaneously, but we hope to \textbf{eventually} add support for the ThunderX. Our target simulation environment is the ARM Fixed Virtual Platform, and the Foundation Platform. These models are supplied by ARM. The Foundation Platform is freely available, and will be the default supported simulation platform for the public Barrelfish tree, while we will use the FVP internally to allow bare-metal debugging. Future support for \qemu is desirable, to the extent that it models a compatible system --- GEM5, which the ARMv7 port targets, currently does not. \textbf{Initially}, the design will support running both the CPU driver and user-space processes in AArch64 mode without support for virtualisation. \textbf{Eventually} the design will support running the CPU driver in AArch64 mode, and user-space processes in both AArch64 and AArch32 modes without virtualisation, and virtual machines in AArch64 mode. We will only support virtualisation on ARMv8.1 or later platforms, that support the VHE extensions, as described in \autoref{s:layout}. \subsection{Processor Modes and Virtualisation} Where possible, we will keep the virtualisation model similar to that on Barrelfish/x86. In particular, it \textbf{should} be possible to implement native applications, fully virtualised (e.g. Linux) VMs, and VM-level applications e.g. Arrakis \citep{peter:osdi14}. ARMv8 has a somewhat different virtualisation model to x86, and different again from the ARMv7 virtualisation extensions. Rather than having exception levels (rings) duplicated between guest and host, ARMv8 provides 4 exception levels (ELs): \begin{itemize} \item \texttt{EL0} is unprivileged --- user applications. \item \texttt{EL1} is privileged --- OS kernel. \item \texttt{EL2} is hypervisor state. \item \texttt{EL3} is for switching between secure and non-secure (TrustZone) modes. The X-Gene 1 does not implement \texttt{EL3}, and it is currently not of interest for Barrelfish. \end{itemize} Explicit traps (syscalls/hypercalls) target only the next level up: \texttt{EL0} can call \texttt{EL1} using \texttt{svc} (syscall), and \texttt{EL1} can call \texttt{EL2} using \texttt{hvc} (hypercall), but \texttt{EL0} cannot directly call \texttt{EL2}, unless \texttt{EL1} is completely disabled. Exceptions return to the caller's exception level. ELs \textbf{shall} be distributed as follows: The CPU driver \textbf{shall} exist at both \texttt{EL1} and \texttt{EL2}, and take both syscalls (\texttt{svc}, from \texttt{EL0} applications) and hypercalls (\texttt{hvc}, from \texttt{EL1} applications). The system \textbf{shall} support applications both at \texttt{EL0}, and at \texttt{EL1} (e.g. Arrakis, VMs). Most code paths \textbf{should} be identical, as most CPU driver operations do not depend on \texttt{EL2} privileges. Hypercalls from \texttt{EL0} \textbf{shall} be chained via \texttt{EL1} (with appropriate permission checks). \texttt{EL1} apps such as Arrakis, and paravirtualised VMs using hypercalls know that they are being virtualised, and will use \texttt{hvc} explicitly. Fully-virtualised \texttt{EL1} VMs do not make hypercalls. ARMv8 implements two-level address translation: VA (virtual address) to IPA (intermediate physical address), and IPA to PA (physical address). \texttt{EL1} guests \textbf{shall} be isolated at the L1 translation layer, and by trapping all accesses to system control registers. \subsection{Virtual Address Space Layout}\label{s:layout} ARMv8 has an effective 48-bit virtual address space. At the lowest execution levels (0 --- BF user \& 1 --- BF CPU driver), the hardware supports two (up to) 48-bit (256TB) 'windows' in a 64-bit space: one at the bottom, and one at the top. Each region has its own translation table base register (\texttt{TTBR0} \& \texttt{TTBR1}). \texttt{TTBR0} is used at \texttt{EL0}, and \texttt{TTBR1} at \texttt{EL1}. In the initial ARMv8 specification, this split address space was not implemented at \texttt{EL2}, which would require a separate CPU driver instance for virtualisation, and hypercalls (e.g. for Arrakis). ARMv8.1 introduced the virtualisation host extensions (VHE) which, among other things, extends the split address space to \texttt{EL2}. As this provides a cleaner implementation model, and to avoid having to support a now-deprecated interface, virtualisation will \textbf{only} be supported on ARMv8.1 and later. This means that we will not support virtualisation on the X-Gene 1. Both the simulation environment (FVP/FP) and, seemingly, the ThunderX chips, support VHE. The CPU driver \textbf{shall} use \texttt{TTBR1} to provide a complete physical window. The ARMv8 CPU driver \textbf{shall not} dynamically map device memory into its own window (as the ARMv7 CPU driver does) --- the few memory-mapped devices required will be statically mapped on boot, with appropriate memory attributes. All physical addresses, RAM and device, \textbf{shall} be accessible at a static, standard offset (the base of the \texttt{TTBR1} region). User-level page tables will \textbf{initially} be limited to a 4k translation granularity. \textbf{Eventually} user-level page tables \textbf{should} have access to all page-table formats and page sizes, as is the case in the current Barrelfish x86 implementation. \subsection{Address Space, Context, and Thread Identifiers} ARMv8 also provides address-space identifiers (ASIDs) in the TLB to avoid flushing the translation cache on a context switch. ARMv8 ASIDs (referred to in ARM documentation as context IDs) are architecturally allowed to be either 8 or 16 bits, although the SBSA specifies that they must be at least 16. Relying on the SBSA platform will allow us to avoid multiplexing IDs among active processes, on any reasonably-sized system. Managing the reuse of context IDs can be left to user-level code, and does not need to be on the critical path of a context switch. The CPU driver need only ensure that every allocated dispatcher has a unique ASID, which is loaded into the \texttt{ContextID} register on dispatch. The value in the \texttt{ContextID} register is also checked against the hardware breakpoint and watchpoint registers, in generating debug exceptions. Therefore, it \texttt{shall} be possible for authorised user-level code to load the Context ID for a given dispatcher into a breakpoint register --- this \texttt{may} be an invocation on the dispatcher capability. \begin{table} \begin{center} \begin{tabular}{ll} \texttt{tpidrro\_el0} & EL0 Read-Only Software Thread ID Register \\ \texttt{tpidr\_el0} & EL0 Read/Write Software Thread ID Register \\ \texttt{tpidr\_el1} & EL1 Read/Write Software Thread ID Register \\ \texttt{tpidr\_el2} & EL2 Read/Write Software Thread ID Register \\ \texttt{tpidr\_el3} & EL3 Read/Write Software Thread ID Register \\ \end{tabular} \end{center} \caption{Thread ID registers in ARMv8} \label{t:threadid} \end{table} In addition to the \texttt{ContextID} register, used to tag TLB entries, ARMv8 also provides a set of thread ID registers with no architecturally-defined semantics, as listed in \autoref{t:threadid}. The client-writeable \texttt{tpidr\_el0} and \texttt{tpidr\_el1} \textbf{shall} have no CPU driver-defined purpose, but \textbf{shall} be saved and restored in a dispatcher's trap frame, to allow their use as thread-local storage (TLS). Recall that the Barrelfish CPU driver has no awareness of threads, which are implemented purely at user level. To implement the upcall/dispatch mechanism of Barrelfish, the CPU driver and the user-level dispatcher need to share a certain amount of state --- the user-visible portion of the dispatcher control block, which contains the trap frames, and the disabled flag (used to achieve atomic dispatch). The address of this structure needs to be known to both the CPU driver, and to user-level code, and moreover be efficiently-accessible, as the CPU driver needs to find the trap frame on the critical path of system calls and exceptions. This pointer also needs to be trustworthy, from the CPU driver's perspective, and thus cannot be directly modifiable by user-level code. The x86-32, x86-64, and ARMv7 CPU drivers all store the address of the running dispatcher's shared segment at a fixed known address, \texttt{dcb\_current}, which is loaded by the trap handler. At user level, on x86 this address is held in a \emph{segment register} (\texttt{fs} on x86-64, and \texttt{gs} on x86-32), while on ARMv7 we sacrifice a general-purpose register (\texttt{r9}) for this purpose. Using the \texttt{tpidrro\_el0} register to hold the address of the current dispatcher structure will allow us to avoid both a memory load on the fast path, and sacrificing a register in user-level code, thus \texttt{tpidrro\_el0} \textbf{shall} hold the address of the currently-running dispatcher. \subsection{Instruction Sets} ARMv8 supports both AArch64, and legacy ARM/Thumb (renamed AArch32). Switching execution mode is only possible when switching execution level i.e. on a trap or return, and can only be changed while at the higher execution level. Thus, \texttt{EL2} can set execution mode for \texttt{EL1}, and \texttt{EL1} for \texttt{EL0}. There is no way for a program to change its own execution mode. If \texttt{ELn} is in AArch64, then \texttt{EL(n-1)} can be in either AArch64 or AArch32. If \texttt{ELn} is in AArch32, all lower ELs must also be AArch32. The CPU driver \textbf{shall} execute in AArch64. \textbf{Initially}, the CPU driver will enforce that all directly-scheduled threads also use AArch64, by controlling all downward EL transitions. An \texttt{EL1} client (such as Arrakis or a full virtual machine) may execute its own \texttt{EL0} clients in AArch32 (and there is no way to prevent this). However, all transitions into the CPU driver (\texttt{svc}, \texttt{hvc} or exception) must come from a direct client of the CPU driver, and thus from AArch64. The syscall ABI \textbf{shall} be AArch64. \textbf{Eventually}, Barrelfish \textbf{should} also support the execution of AArch32 dispatcher processes, by marking each dispatcher with a flag indicating the instruction set to be used (much as is already done with VM/non-VM mode in the Arrakis CPU driver). \subsection{User-Space Access to Architectural Functions} Generally, anything that can be safely exported, \textbf{should} be made available outside of the CPU driver, preferable as a memory-mapped interface, at 4kiB granularity. The SBSA mandates that devices be present at addresses that can be individually mapped, thus this should not be a problem. \subsection{Cache Management} ARMv8 has moved most cache and TLB management from the system control coprocessor (cp15), into the core ISA. Several cache operations (invalidate/clean by VA) are executable at \texttt{EL0}, and thus no kernel interface is required. The system must take into account that user-directed flushes may have occurred, or may occur concurrently with any memory operation. \subsection{Performance Monitors} Performance monitors \textbf{should} be exposed, if it can be done safely. \subsection{Debugging} Self-hosted debug \textbf{should} be exposed, if it can be done safely. This is under active development. \subsection{Booting} Platform support i.e.~a standard set of peripherals, and a defined boot process, has improved dramatically on ARM, as it has been repositioned as a server platform. UEFI and ACPI support are widespread, including on the Mustang development board. We will assume support for UEFI booting, make use of ACPI data, where available. The Barrelfish CPU driver and initial image \textbf{shall} be loaded and executed by a UEFI shim, which will pass through all UEFI-supplied information, such as ACPI tables, and be able to interpret a Barrelfish Multiboot image. This shim, or second-stage bootloader, is called Hagfish, and is described in \autoref{s:hagfish}. \subsection{Interrupts} ARMv8 interrupt handling is not substantially different from the existing architectures and platforms supported by Barrelfish. While a redesign of the Barrelfish interrupt system is under way (to use capabilities to grant access to receive interrupts), we do not anticipate ARMv8 to impose any particular challenges. The ARMv8 systems we \textbf{initially} target all use minor variations on the ARM Generic Interrupt Controller (GIC) design, already supported in Barrelfish. We currently have support for version 2 of the GIC, with which later implementations are backward-compatible. We will \textbf{eventually} support GICv3, the current specification at time of writing. \subsection{Inter-Domain Communication} User-level communication between cache-coherent cores in Barrelfish for ARMv8 is likely to the same as with ARMv7 and x86, and we expect the existing User-level Message-Passing over Cache-Coherence (UMP-CC) interconnect driver to work unmodified. Between dispatchers on the same core, however, the different register set on the ARMv8 is likely to result in a very different Local Message Passing (LMP) interconnect driver---this is always an architecture-specific part of the CPU driver. In practice, its design will be closely tied to the context switch and upcall dispatch code. \chapter{Booting}\label{c:booting} Booting ARM systems has always been difficult to do in a standard way, and ARMv8 systems are no exception. Barrelfish uses one of two methods of booting an initial ARMv8 core, depending on whether the hardware platform supports UEFI~\cite{uefi} or U-Boot. If a platform supports neither, more work will be required to boot the board. If a board has full support for UEFI (such as TianoCore), you can use Hagfish~\ref{s:hagfish} to individually load the modules needed to boot Barrelfish and set up the initial CPU/MMU environment before entering the CPU driver proper. Note that U-Boot also claims to support UEFI. However, in practice it supports a small subset of UEFI functionality sufficient to boot \texttt{grub} or the Linux kernel as an EFI binary. If your board boots via U-Boot, you should use the minimal EFI bootloader~\ref{s:uboot} which loads a single multiboot image into memory and sets up the environment similar to Hagfish. \section{Hagfish}\label{s:hagfish} The Barrelfish/ARMv8 UEFI loader prototype is called Hagfish\footnote{A hagfish is a basal chordate i.e. something like the ancestor of all fishes.}. Hagfish is a second-stage bootloader for Barrelfish on UEFI platforms, initially the ARMv8 server platform. Hagfish is loaded as a UEFI application, and uses the large set of supplied services to do as much of the one-time (boot core) setup that the CPU driver needs as is reasonably possible. More specifically, Hagfish: \begin{itemize} \item Is loaded over BOOTP/PXE. \item Reuses the PXE environment to load a menu.lst-style configuration. \item Loads the kernel image and the initial applications, as directed, and builds a Multiboot image. \item Allocates and builds the CPU driver's page tables. \item Activates the initial page table, and allocates a stack. \end{itemize} \subsection{Why Another Bootloader?} The ARMv8 machines that we're porting to are different to both existing ARM boards, and to x86. They have a full pre-boot environment, unlike most embedded boards, but it's not a PC-style BIOS. The ARM Server Base Boot Requirements specify UEFI. Moreover, there is no mainline support from GNU GRUB for the ARMv8 architecture, so no matter what, we need some amount of fresh code. Given that we had to write at least a shim loader, and keeping in mind that UEFI is multi-platform (and becoming more and more common in the x86 world), we're taking the opportunity to simplify the initial boot process within the CPU driver by moving the once-only initialisation into the bootloader. In particular, while running under UEFI boot services, we have memory allocation available for free, e.g. for the initial page tables. By moving ELF loading and relocation code into the bootloader, we can eliminate the need to relocate running code, and can cut down (hopefully eliminate) special-case code for booting the initial core. Subsequent cores can rely on user-level Coreboot code to relocate them, and to construct their page tables. \subsection{Assumptions and Requirements} Hagfish is (initially at least) intended to support development work on AArch64 server-style hardware and, as such, makes the following assumptions: \begin{itemize} \item 64-bit architecture, using ELF binaries. Porting to 32-bit architectures wouldn't be hard, if it were ever necessary (probably not). \item PXE/BOOTP/TFTP available for booting. Hagfish expects to load its configuration, and any binaries needed, using the same PXE context with which it was booted. Changing this to boot from a local device (e.g. HDD) wouldn't be hard, as the UEFI \texttt{LoadFile} interface abstracts from the hardware. \end{itemize} \subsection{Boot Process} In detail, Hagfish currently boots as follows: \begin{enumerate} \item \texttt{Hagfish.efi} is loaded over PXE by UEFI, and is executed at a runtime-allocated address, with translation (MMU) and caching enabled. \item Hagfish queries EFI for the PXE protocol instance used to load it, and squirrels away the current network configuration. \item Hagfish loads the file \texttt{hagfish.A.B.C.D.cfg} from the TFTP server root (where \texttt{A.B.C.D} is the IP address on the interface that ran PXE). \item Hagfish parses its configuration, which is essentially a GRUB menu.lst, and loads the kernel image and any additional modules specified therein. All ELF images are loaded into page-aligned regions of type \texttt{EfiBarrelfishELFData}. \item Hagfish queries UEFI for the system memory map, then allocates and initialises the inital page tables for the CPU driver (mapping all occupied physical addresses, within the \texttt{TTBR1} window, see \autoref{s:layout}). The frames holding these tables are marked with the EFI memory type\\ \texttt{EfiBarrelfishBootPagetable}, allocated from the OS-specific range (\texttt{0x80000000}--\texttt{0x8fffffff}). All memory allocated by Hagfish on behalf of the CPU driver is page-aligned, and tagged with an OS-specific type, to allow EFI and Hagfish regions to be safely reclaimed. \item Hagfish builds a Multiboot 2 information structure, containing as much information as it can get from EFI, including: \begin{itemize} \item ACPI 1.0 and 2.0 tables. \item The EFI memory map (including Hagfish's custom-tagged regions). \item Network configuration (the saved DHCP ack packet). \item The kernel command line. \item All loaded modules. \item The kernel's ELF section headers. \end{itemize} \item Hagfish allocates a page-aligned kernel stack (type \texttt{EfiBarrelfishCPUDriverStack}), of the size specified in the configuration. \item Hagfish terminates EFI boot services (calls \texttt{ExitBootServices}), activates the CPU driver page table, switches to the kernel stack, and jumps into the relocated CPU driver image. \end{enumerate} \subsection{Post-Boot state} When the CPU driver on the boot core begins executing, it can assume the following: \begin{itemize} \item The MMU is configured with all RAM and I/O regions mapped via \texttt{TTBR1}. \item The CPU driver's code and data are both fully relocated into one or more distinct 4kiB-aligned regions. \item The stack pointer is at the top of a distinct 4kiB-aligned region of at least the requested size. \item The first argument register holds the Multiboot 2 magic value. \item The second holds a pointer to a Multiboot 2 information structure, in its own distinct 4kiB-aligned region. \item The console device is configured. \item Only one core is enabled. \item The Multiboot structure contains at least: \begin{itemize} \item The final EFI memory map, with all areas allocated by Hagfish to hold data passed to the CPU driver marked with OS-specific types, all of which refer to non-overlapping 4k-aligned regions: \begin{description} \item[\ttfamily EfiBarrelfishCPUDriver] The currently-executing CPU driver's text and data segments. \item[\ttfamily EfiBarrelfishCPUDriverStack] The CPU driver's stack. \item[\ttfamily EfiBarrelfishMultibootData] The Multiboot structure. \item[\ttfamily EfiBarrelfishELFData] The unrelocated ELF image for a boot-time module (including that for the CPU driver itself), as loaded over TFTP. \item[\ttfamily EfiBarrelfishBootPageTable] The currently-active page tables. \end{description} \item The CPU driver (kernel) command line. \item A copy of the last DHCP Ack packet. \item A copy of the section headers from the CPU driver's ELF image. \item Module descriptions for the CPU driver and all other boot modules. \item If UEFI provided an ACPI root table, the Multiboot structure contains a pointer to it. \end{itemize} \end{itemize} \subsection{Configuration} Hagfish configures itself by loading a file whose path is generated from its assigned IP address. Thus if your development machine receives the address 192.168.1.100, Hagfish will load the file\\ \texttt{hagfish.192.168.1.100.cfg} from the same TFTP server used to load it. The format is intended to be as close as practical to that of an old-style GRUB menu.lst file. The example configuration in \autoref{f:hag_config} loads \texttt{/armv8/sbin/cpu\_apm88xxxx} as the CPU driver, with arguments \texttt{loglevel=3}, and an 8192B (2-page) stack. \begin{figure}[htb] \begin{center} \begin{lstlisting} kernel /armv8/sbin/cpu_apm88xxxx loglevel=3 stack 8192 module /armv8/sbin/cpu_apm88xxxx module /armv8/sbin/init # Domains spawned by init module /armv8/sbin/mem_serv module /armv8/sbin/monitor # Special boot time domains spawned by monitor module /armv8/sbin/chips boot module /armv8/sbin/ramfsd boot module /armv8/sbin/skb boot module /armv8/sbin/kaluga boot module /armv8/sbin/spawnd boot bootarm=0 module /armv8/sbin/startd boot # General user domains module /armv8/sbin/serial auto portbase=2 module /armv8/sbin/fish nospawn module /armv8/sbin/angler serial0.terminal xterm module /armv8/sbin/memtest module /armv8/sbin/corectrl auto module /armv8/sbin/usb_manager auto module /armv8/sbin/usb_keyboard auto module /armv8/sbin/sdma auto \end{lstlisting} \end{center} \caption{Hagfish configuration file} \label{f:hag_config} \end{figure} \subsection{Booting with Hagfish in \qemu}\label{c:qemu} When booting a \qemu image for 64-bit ARM, a number of options are available (see \texttt{make help-boot}). Building a boot image for \qemu with ARMv8 will typically result in a file in the build directory called \texttt{armv8__qemu_image}. This is a disk image which can be read by Hagfish through EFI calls. Booting this with a boot target from \texttt{make} will run the following: \begin{lstlisting} srcdir/tools/qemu-wrapper.sh \\ --image armv8__qemu_image \\ --arch armv8 \\ --bios ../git/barrelfish/tools/hagfish/QEMU_EFI.fd \end{lstlisting} This wrapper script is complex, but reasonably well documented (use '\texttt{--help}'). It will invoke \qemu as follows: \begin{lstlisting} qemu-system-aarch64 \\ -m 1024 \\ -cpu cortex-a57 \\ -M virt \\ -d guest_errors \\ -M gic_version=3 \\ -smp 1 \\ -bios ../git/barrelfish/tools/hagfish/QEMU_EFI.fd \\ -device virtio-blk-device,drive=image \\ -drive if=none,id=image,file=armv8__qemu_image,format=raw \\ -nographic \end{lstlisting} Note that for this script to work, you need to have \texttt{mtools} (the MS-DOS file system manipulation tools) installed, since they are used to prepare the \texttt{armv8__qemu_image} file. More specifically, the \texttt{armv8__qemu_image} file is generated by \texttt{tools/harness/efiimage.py}. This creates an EFI file system image out of the plain Barrelfish binaries built in \texttt{\textit{builddir}/armv8/sbin}, plus the Hagfish EFI image we regularly use for real hardware. The \texttt{QEMU\_EFI.fd} file is the UEFI runtime built for \qemu. \section{Booting from U-Boot}\label{s:uboot} Where a full UEFI environment is not available, it is possible to boot Barrelfish from U-Boot~\cite{uboot}. We boot Barrelfish from U-Boot using U-Boot's limited EFI support: a build-time tool (\texttt{armv8\_bootimage} builds a single binary which only requires the minimal EFI environment provided by U-Boot. This binary contains a loader (\texttt{efi\_loader}) which sets up the rest of the image as a multiboot image in memory before starting the CPU driver. \subsection{Booting in \qemu with U-Boot} A ``platform'' target like \texttt{QEMU\_UBoot} which build such an image for \qemu, and the \texttt{qemu-wrapper.sh} script can be invoked to use U-Boot instead of Hagfish: \begin{lstlisting} srcdir/tools/qemu-wrapper.sh \\ --image armv8_a57_qemu_image.efi \\ --arch armv8 \\ --uboot-img srcdir/tools/qemu-armv8-uboot.bin \end{lstlisting} This invoked \qemu as follows: \begin{lstlisting} qemu-system-aarch64 \\ -m 1024 \\ -cpu cortex-a57 \\ -M virt \\ -d guest_errors \\ -M gic_version=3 \\ -smp 1 \\ -bios srcdir/tools/qemu-armv8-uboot.bin \\ -device loader,addr=0x50000000,file=armv8_a57_qemu_image.efi \\ -nographic \end{lstlisting} As you can see, the UBoot binary is given as the BIOS, and the minimal EFI image with the complete set of multiboot modules compiled in is pre-loaded into memory when \qemu starts. \chapter{Technical Observations}\label{c:tech} \section{User-Space Threading}\label{s:threads} \begin{figure}[htb] \begin{center} \begin{minipage}[t]{0.3\textwidth} \begin{lstlisting} clrex /* Restore CPSR */ ldr r0, [r1], #4 msr cpsr, r0 /* Restore registers */ ldmia r1, {r0-r15} \end{lstlisting} \end{minipage} \hspace{2cm} \begin{minipage}[t]{0.5\textwidth} \begin{lstlisting} /* Restore PSTATE, load resume * address into x18 */ ldp x18, x2, [x1, #(PC_REG * 8)] /* Set only NZCV. */ and x2, x2, #0xf0000000 msr nzcv, x2 /* Restore the stack pointer and x30. */ ldp x30, x2, [x1, #(30 * 8)] mov sp, x2 /* Restore everything else. */ ldp x28, x29, [x1, #(28 * 8)] ldp x26, x27, [x1, #(26 * 8)] ldp x24, x25, [x1, #(24 * 8)] ldp x22, x23, [x1, #(22 * 8)] ldp x20, x21, [x1, #(20 * 8)] /* n.b. don't reload x18 */ ldr x19, [x1, #(19 * 8)] ldp x16, x17, [x1, #(16 * 8)] ldp x14, x15, [x1, #(14 * 8)] ldp x12, x13, [x1, #(12 * 8)] ldp x10, x11, [x1, #(10 * 8)] ldp x8, x9, [x1, #( 8 * 8)] ldp x6, x7, [x1, #( 6 * 8)] ldp x4, x5, [x1, #( 4 * 8)] ldp x2, x3, [x1, #( 2 * 8)] /* n.b. this clobbers x0&x1 */ ldp x0, x1, [x1, #( 0 * 8)] /* Return to the thread. */ br x18 \end{lstlisting} \end{minipage} \end{center} \caption{\texttt{disp\_resume\_context} on ARMv7 (left) and ARMv8 (right)} \label{f:disp_resume} \end{figure} The ARMv8 architecture is in some ways an improvement, and in other ways problematic, for the sort of user-level threading implemented in Barrelfish, via \emph{scheduler activations}. Under this scheme, the kernel (in Barrelfish terms, the \emph{CPU driver}), does not schedule threads directly, but instead exposes all scheduling-relevant events via \emph{upcalls} to predefined user-level handlers (in Barrelfish, the \emph{dispatcher}), which then implements thread scheduling (or something else entirely), as it sees fit. This differs from the behaviour of a system such as UNIX, which only ever restores a user-level execution context simultaneously with dropping from a privileged to an unprivileged execution level. Processor architectures are, understandably, designed with common software in mind. Thus, the primitives available for restoring an execution context i.e. register state are often tied closely to those for changing privilege level. A common design (which ARMv8 also implements) is the \emph{exception return}, where privileged code can atomically drop its privilege, and jump to a user-level execution address. In ARMv8, the \texttt{eret} instruction atomically updates the program state (PSTATE, most importantly the privilege level bits), and branches to the address held in the \emph{exception link register}, \texttt{elr}. In implementing user-level threading, we're not concerned with privilege levels, but the lack of some equivalent of \texttt{elr} is frustrating. Not only does \texttt{eret} provide an atomic update of the program counter and the program state, it does so without modifying any general-purpose register. Replicating this behaviour at \texttt{EL0}, where \texttt{eret} is unavailable is problematic. ARMv8 differs from ARMv7, in that the program counter can no longer be the target of a load instruction, but can only be loaded via a general-purpose register. Specifically, the only PC-modifying instructions (other than \texttt{eret}) are PC-relative branches (which are useless in this scenario) and branch-to-register (of which \texttt{br}, \texttt{blr} and \texttt{ret} are all special encodings). Since ARMv8 has also removed the \texttt{ldm} (load multiple) instruction, there is no way to load the program counter with an arbitrary value (the thread's restart address), without overwriting one of the general-purpose registers. We cannot restore the thread's register value \emph{before} we branch to it, as we'd overwrite the return address, and we obviously can't do so afterwards, as the thread likely has no idea that it's been interrupted. The only alternative is to trampoline through kernel mode in order to use \texttt{eret} (which would eliminate the speed benefit of user-level threading), or to reserve a general-purpose register for use by the dispatcher. Neither option is appealing, but we went with the second option, reserving \texttt{x18}, reasoning that with 31 general-purpose registers available, the loss of one isn't a huge penalty. Register \texttt{x18} is explicitly marked as the \emph{platform register} in the AArch64 ABI \citep{arm:aa64pcs}, for such a purpose. Future revisions of the ARM architecture could prevent this issue in a number of ways: allowing the use of \texttt{eret} at \texttt{EL0} or providing an equivalent functionality (specifically a non-general-purpose register such as \texttt{elr}, that doesn't need to be restored); or alternatively, adding indirect jumps (load to PC) back to the instruction set. \autoref{f:disp_resume} compares the user-level thread resume code for the Barrelfish dispatcher (function \texttt{disp\_resume}) for ARMv7 and ARMv8 side-by-side. The effect of removing the load-multiple instructions, and direct-to-SP loads, on code density is clearly visible: everything on lines 8--29 for ARMv8 corresponds to the single \texttt{ldmia} instruction on lines 9 for ARMv7 --- one instruction is now 18, on the thread-switch critical path! Note also, on line 17, that the ARMv8 code does not restore the thread's \texttt{r18}, but instead uses it to hold the branch address for use on line 29. The only improvement on ARMv8 is that the \texttt{clrex} (clear exclusive monitor) instruction is no longer required, as the monitor is cleared on returning from the kernel. Note also that the usual method to efficiently load multiple registers, using 16-word SIMD (NEON) loads, isn't available, as there's no guarantee that the SIMD extensions are enabled on this dispatcher, and we cannot handle a fault in this code. \section{Trap Handling}\label{s:traps} \begin{figure} \begin{lstlisting} el0_aarch64_sync: msr daifset, #3 /* IRQ and FIQ masked, Debug and Abort enabled. */ stp x11, x12, [sp, #-(2 * 8)]! stp x9, x10, [sp, #-(2 * 8)]! mrs x10, tpidr_el1 mrs x9, elr_el1 ldp x11, x12, [x10, #OFFSETOF_DISP_CRIT_PC_LOW] cmp x11, x9 ccmp x12, x9, #0, ls ldr w11, [x10, #OFFSETOF_DISP_DISABLED] ccmp x11, xzr, #0, ls /* NE <-> (low <= PC && PC < high) || disabled != 0 */ mrs x11, esr_el1 /* Exception Syndrome Register */ lsr x11, x11, #26 /* Exception Class field is bits [31:26] */ b.ne el0_sync_disabled add x10, x10, #OFFSETOF_DISP_ENABLED_AREA save_syscall_context: str x7, [x10, #(7 * 8)] stp x19, x20, [x10, #(19 * 8)] stp x21, x22, [x10, #(21 * 8)] stp x23, x24, [x10, #(23 * 8)] stp x25, x26, [x10, #(25 * 8)] stp x27, x28, [x10, #(27 * 8)] stp x29, x30, [x10, #(29 * 8)] /* FP & LR */ mrs x20, sp_el0 stp x20, x9, [x10, #(31 * 8)] mrs x19, spsr_el1 str x19, [x10, #(33 * 8)] cmp x11, #0x15 /* SVC or HVC from AArch64 EL0 */ b.ne el0_abort_enabled add sp, sp, #(4 * 8) mov x7, x10 b sys_syscall \end{lstlisting} \caption{BF/ARMv8 synchronous exception handler} \label{f:sync_el0} \end{figure} \autoref{f:sync_el0} shows the CPU driver exception stub, for a synchronous abort from \texttt{EL0}. This exception class includes system calls, breakpoints, and page faults on both code and data. The effect of the loss of store multiple instructions is again visible, for example on lines 27--32. Although not as severe as in the case of the user-level thread restore in \autoref{s:threads}, the extra instructions required do constrain us somewhat, as each trap handler is constrained to 128 bytes, or 32 instructions, before branching to another code block. We were able to squeeze the necessary code into the space available, including the optimised test for a disabled dispatcher at lines 10--14, but only by splitting the page fault handler (\texttt{el0\_abort\_enabled}) into a separate subroutine, incurring an unnecessary branch. A more significant annoyance is that system calls (\texttt{svc} and \texttt{hvc}) are routed to the same exception vector as page faults (aborts). The effect of this is that we are forced to spill registers to the stack (\texttt{x9}--\texttt{x12} on lines 4--5), even on the system call fast path, as we need at least one register to check the exception syndrome (\texttt{esr\_el1}) to distinguish aborts (where we must preserve all registers) from system calls (where we could immediately begin using the caller-saved registers). Note that the code on lines 27--32 only needs to stack the callee-saved registers, and leaves the system call arguments in \texttt{x0}--\texttt{x7}, to be read as required by \texttt{sys\_syscall} (in C). This sort of mismatch between the exception-handling interface of the CPU architecture, and what is required for really high-performance systems code is unfortunately extremely common. Unnecessary overheads, such as the additional stacked registers here hurt the performance of highly-componentised systems, such as Barrelfish, which rely on frequently crossing protection domains. The relatively well-compressed boolean arithmetic on lines 10--14 demonstrates that, even with the loss of ARM's fully-conditional instructions, the conditional compares which remain are still relatively powerful. \section{Cache Coherence} One aspect of the ARM architecture that is of particular interest for the Barrelfish project, but which we have not yet explored in depth, is the configurable cache coherency and fine-grained cache management operations available. Any virtual mapping on a recent ARM architecture, including both ARMv7 and ARMv8, can be tagged with various cacheability properties: inner (L1), outer (L2+, usually), write-back or write-through. Combined with the explicit flush operations at cache-line granularity, able to target either PoU (point of unification, where data and instruction caches merge) or PoC (point of coherency, typically RAM), a multi-core, multi-socket ARMv8 system would make a very interesting testbed for investigating efficient cache management and communication primitives for future partially-coherent architectures. Indeed, the latest revision of the ARMv8 specification, ARMv8.2, introduced flush to PoP, or \emph{point of persistence} --- perhaps in response to interest from well-known systems integration firms investigating large persistent memories. The design presented in this report is intended to expose as much control over the caching hierarchy as possible to user-level code, to provide a platform for future research. \bibliographystyle{plainnat} \bibliography{defs,barrelfish} \end{document}