\documentclass[a4paper,11pt,twoside]{report} \usepackage{bftn} \usepackage{calc} \usepackage{verbatim} \usepackage{xspace} \usepackage{pifont} \usepackage{pxfonts} \usepackage{textcomp} \usepackage{amsmath} \usepackage{multirow} \usepackage{listings} \usepackage[framemethod=default]{mdframed} \usepackage[shortlabels]{enumitem} \usepackage{parskip} \usepackage{xparse} \newcommand{\todo}[1]{[\textcolor{red}{\emph{#1}}]} \title{Coreboot in Barrelfish} \author{Barrelfish project} \tnnumber{23} \tnkey{Coreboot} \begin{document} \maketitle % Uncomment for final draft \begin{versionhistory} \vhEntry{0.1}{28.02.2017}{GZ}{Initial Version} \vhEntry{0.1}{02.03.2017}{RA}{Adding basic structure} \end{versionhistory} % \intro{Abstract} % Insert abstract here % \intro{Acknowledgements} % Uncomment (if needed) for acknowledgements \tableofcontents % Uncomment (if needed) for final draft % \listoffigures % Uncomment (if needed) for final draft % \listoftables % Uncomment (if needed) for final draft \cleardoublepage \setcounter{secnumdepth}{2} \newcommand{\fnname}[1]{\textit{\texttt{#1}}}% \newcommand{\datatype}[1]{\textit{\texttt{#1}}}% \newcommand{\varname}[1]{\texttt{#1}}% \newcommand{\keywname}[1]{\textbf{\texttt{#1}}}% \newcommand{\pathname}[1]{\texttt{#1}}% \newcommand{\tabindent}{\hspace*{3ex}}% \newcommand{\sockeye}{\lstinline[language=sockeye]} \newcommand{\ccode}{\lstinline[language=C]} \lstset{ language=C, basicstyle=\ttfamily \small, keywordstyle=\bfseries, flexiblecolumns=false, basewidth={0.5em,0.45em}, boxpos=t, captionpos=b } \chapter{Introduction} \label{chap:introduction} This document describes the way Barrelfish boots cores on supported architectures. We first explain the terminology used throughout this document. Then, we will give an overview of supported operations and finally, explain in detail how the implementation of these operations on differ for various supported architectures. \section{Boot drivers} Barrelfish uses \textit{boot drivers}, which is a piece of code running on a ``home core'' and manages a ``target core''. It encapsulates the hardware functionality to boot, suspend, resume, and power-down the latter. Boot drivers run as processes, but closely resemble device drivers and could equally run as software objects within another process. The boot driver abstraction treats CPU cores much like peripheral devices, and allows us to reuse the OS's existing device and hot-plug management infrastructure to handle new cores and select drivers and kernels for them. The code for the Barrelfish boot driver can be found in \texttt{usr/drivers/cpuboot/}. From the source folder, a binary called ``corectrl'' is generated at build time and can be used to do target core management. The ``corectrl'' program is a command line tool that supports different operations as described below. \section{CPU driver} Is the binary that runs on a core and executes in privileged mode. CPU drivers are the Barrelfish equivalent of a kernel, except that there is one per core, and they share no state or synchronization. CPU drivers are typically non-preemptible, single-threaded, and mostly stateless. The CPU driver binary is selected and loaded by a \textit{boot driver}, which among other things, is responsible of making sure the CPU driver is executed on a core. TN-021 contains a more detailed explanation of CPU drivers and the provided functionality. The code for the Barrelfish CPU drivers can be found in \texttt{kernel/}. \section{Kernel Control Block (KCB)} The Kernel Control Block (or KCB) is a memory region that can only be accessed by kernel-space. However, a program, especially boot drivers, can create and reference KCBs in user-space through the capability system. The Kernel Control Block's purpose is to hold or keep pointers to all per-core state. All per-KCB state maintained by a kernel must be reachable from the KCB itself or reconstructible by reading the KCB. Usually, a KCB is created for each core we start and passed to the new core on start-up. On boot-up, the kernel checks if the KCB it received is initialized. In that case, it will use the KCB and start to dispatch the applications it contains. If the KCB is uninitialized, the kernel sets all the entries of the KCB to its default values and continues bootstrapping the core (and the KCB block) by starting the monitor application, which is critical for communication across cores in user-space. Every kernel has the possibility to maintain multiple KCBs. The kernel maintains a ring structure of KCBs it owns. A CPU driver supports switching to another KCB on demand. In case a CPU driver is running multiple KCBs it divides the time-slices evenly among different KCBs and switches KCBs after a given amount of time has passed. Currently, the CPU driver uses round-robin scheduling for having multiple KCBs on one core but other models are possible as well. \chapter{Generic Operations} \label{chap:generic} The boot driver (currently called ``corectrl'') has support for a range of operations, which we describe in more detail in this chapter. As a summary the operations are listed here: \begin{itemize} \item boot \textless target coreid\textgreater: Boots a new core with a KCB. \item stop \textless target coreid\textgreater: Stop execution on an existing core. \item update \textless target coreid\textgreater: Update the CPU driver on an existing core. \item give \textless from kcbid\textgreater \textless to kcbid\textgreater: Give kcb from one core to another. \item rmkcb \textless kcbid\textgreater: Remove a running KCB from a core. %\item park \textless kcbid to stop\textgreater \textless recv kcbid\textgreater: Stop execution on an existing core and park KCB on another core. %\item unpark \textless kcbid to unpark\textgreater: Reestablish parked KCB on its original core. \item lscpu: List current status of all cores. \item lskcb: List current KCBs. \end{itemize} Note that the implementation of ``corectrl'' in it's current from is essentially a command line tool that must be invoked for every operation. This means that the boot driver is currently state-less which is fine for just executing the operations since all information required can be reconstructed from the SKB. There are a few optional flags that can be passed to the different operations: \begin{itemize} \item -d, --debug: Print debug information \item -k, --kernel \textless binary\textgreater: Overwrite default kernel binary. \item -x, --monitor \textless binary\textgreater: Overwrite default monitor binary. \item -a, --kargs \textless args\textgreater: Overwrite default kernel command line arguments. \item -n, --newkcb: Create a new KCB even if there is already one for that core. \item -m, --nomsg: Don't wait for a monitor message. \end{itemize} \section{boot \textless target coreid\textgreater} Starts a CPU driver on a core. If the new core that has never been started before, it is brought online as follows: \begin{penumerate} \item The new core is detected by some platform-specific mechanism (e.g., ACPI) and its appearance registered with the device management subsystem. This is done by adding an octopus record to the SKB. The record name is of the format \texttt{hw.processor.ID} where ID is an identifier guaranteed to be unique by the SKB/Octopus. The record has the following mandatory, architecture independent fields: \begin{itemize} \item \texttt{enabled}: A boolean field that says if the core is usable or not (for example, on x86 if hyper-threads are disabled, this flag would be set to false for hyper-threads -- the resulting core would not be booted by default). \item \texttt{barrelfish\_id}: A unique integer identifier used by Barrelfish to refer to this core. \item \texttt{hw\_id}: An identifier assigned by the hardware/platform to this core. \item \texttt{type}: A number that corresponds to that cores architecture as described in \texttt{enum cpu\_type} (defined in barrelfish\_kpi/cpu.h). \end{itemize} \item Barrelfish selects and starts an appropriate boot driver for the new core. This is done by propagating the added octopus record to kaluga which then invokes ``corectrl'' with the boot command. \item The boot driver selects a kernel binary and arguments for the new core, and directs the boot driver to boot the kernel on the core. By default ``corectrl'' uses the \texttt{type} field in the octopus record to decide which binary to spawn on the target core. However, this can be overridden by manually selecting a kernel with the --kernel flag. Alternatively, the default parameters passed to the kernel can be overridden with the --kargs option. \item The boot driver loads and relocates the kernel, and executes the hardware protocol to start the new core. This is done by using the file system to load the binaries and additionally a series of system calls to invoke protected operations (starting a core, sending an IPI etc.). In this process, also a new KCB is created for the core and initialized with default arguments. \item The new kernel initializes and uses existing Barrelfish protocols for integrating into the running OS. This involves spawning the first user-space program (the monitor) on the new core. The default choice for the first user-space program loaded on the new core can be overridden by using the --monitor flag. \item The monitor will inform the other boot driver about its existence by sending a message back to it. Then, the boot driver will tell its local monitor about the newly available and initialized core. In some cases (for example if something else is spawned than the default monitor binary) we may not want to wait for such a message. If the --nomsg flag is passed to ``corectrl'' this step is skipped. \item The SKB is updated with information about the new core: The core is marked online and booted. The KCB that was created during the boot process by ``corectrl'' is stored in the capability storage in case it needs to be retrieved at a different point in time. \end{penumerate} At least three special cases need to be considered: \begin{itemize} \item The core id is invalid: If we don't find the core in the SKB, the operation is aborted. \item The core is already running: In that case we abort with an error. \item The core has been booted before: By default ``corectrl'' will check if an existing KCB for the given core already exists. If yes, the capability for the KCB will be retrieved from the SKB and passed to the new core. This behavior can be overridden by passing the --newkcb flag which makes sure that an existing KCB is not reused. \end{itemize} \section{stop \textless target coreid\textgreater} Stops a CPU driver on a core. If the core is already stopped, nothing is done. The KCB running on this core will stop dispatching until the core is either restarted, or the KCB is given to another core. In general the operations executed is follow: \begin{itemize} \item Invoke hardware specific protocol to stop the core. \item Optional: Update meta-data in SKB to mark core as stopped. \end{itemize} This operations does not take any optional flags or additional arguments. Note that while this is deliberately very simple and low-overhead, just having the KCB stop to execute is generally a bad idea as there may be messages waiting to be received on that core etc. which can cause the whole system to lock-up. It makes sense for example to park the KCB after stopping (see the give operation later). \section{update \textless target coreid\textgreater} Update is implemented as a sequence of executing the \texttt{stop} and \texttt{start} command. It takes the same arguments and flags as described in the start section. \section{give \textless target kcbid\textgreater \textless destination kcbid\textgreater} Removes a KCB on the core it is currently executing and transfers it to the core that is currently dispatching the destination KCB. This command involves several steps: \begin{penumerate} \item The boot driver instructs its local monitor to send a message to the monitor of the target KCB. The message instructs the receiving monitor to inform its local CPU driver to stop dispatching the KCB (or OSNode). \item The boot driver sends message to the monitor of the destination KCB, passing along the cap reference of the target KCB. Upon receipt, the destination monitor will hand the reference to its CPU driver. \item The destination CPU driver will do some minimal initialization, resetting timers and scheduling state, check for registered interrupts and up-call drivers to re-register them, and then start dispatching the target KCB. \item The boot driver will update the SKB to change the core assigned to the target kcbid. \end{penumerate} The following special cases need to be considered: \begin{itemize} \item The target or destination KCB ID is invalid: The operation is aborted. \item The destination KCB is not currently assigned a core: The operation is aborted. \item The target KCB is not currently running on a core: The step where we first send a message to the target KCB is skipped. \end{itemize} \section{rmkcb \textless kcbid\textgreater} \begin{penumerate} \item The boot driver instructs its local monitor to send a message to the monitor of the target KCB. \item Upon receipt, the monitor of the target KCB will instruct the CPU driver to remove the KCB from the scheduling queue. If queue is now empty, the CPU driver will stop itself, otherwise it will switch to the next KCB in the list. \end{penumerate} The following special cases need to be considered: \begin{itemize} \item The KCB id is invalid: If we don't find the KCB capability in the SKB, the operation is aborted. \item The KCB is not dispatched on any core: If the KCB is not currently associated with any core, it can't be removed and therefore the operation is a NOP. \end{itemize} \todo{Don't remember exactly if we need to send this to KCBID we want to remove or the core/KCB that is currently hosting us...} \chapter{Booting x86 Cores} \label{chap:x86} \section{Core discovery and identification} \section{Booting a new core} \chapter{Booting ARMv7 Cores} \label{chap:armv7} \todo{explain ARMv7 specific boot arguements / protocols} \section{Core discovery and identification} \section{Booting a new core} \chapter{Booting ARMv8 Cores} \label{chap:armv8} \todo{explain ARMv8 specific boot arguements / protocols} \section{Core discovery and identification} \section{Booting a new core} % %http://infocenter.arm.com/help/topic/com.arm.doc.den0044b/DEN0044B_Server_Base_Boot_Requirements.pdf \subsection{Power State Coordination Interface (PSCI)} % %http://infocenter.arm.com/help/topic/com.arm.doc.den0022c/DEN0022C_Power_State_Coordination_Interface.pdf \subsection{Parking Protocol} %https://acpica.org/sites/acpica/files/MP%20Startup%20for%20ARM%20platforms.doc \subsection{All at once} Raspberry Pi %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \bibliographystyle{abbrv} \bibliography{barrelfish} \end{document}