386 lines
15 KiB
TeX
386 lines
15 KiB
TeX
\documentclass[a4paper,11pt,twoside]{report}
|
|
\usepackage{bftn}
|
|
\usepackage{calc}
|
|
\usepackage{verbatim}
|
|
\usepackage{xspace}
|
|
\usepackage{pifont}
|
|
\usepackage{pxfonts}
|
|
\usepackage{textcomp}
|
|
\usepackage{amsmath}
|
|
\usepackage{multirow}
|
|
\usepackage{listings}
|
|
\usepackage[framemethod=default]{mdframed}
|
|
\usepackage[shortlabels]{enumitem}
|
|
\usepackage{parskip}
|
|
\usepackage{xparse}
|
|
|
|
\newcommand{\todo}[1]{[\textcolor{red}{\emph{#1}}]}
|
|
|
|
\title{Coreboot in Barrelfish}
|
|
\author{Barrelfish project}
|
|
\tnnumber{23}
|
|
\tnkey{Coreboot}
|
|
|
|
\begin{document}
|
|
\maketitle % Uncomment for final draft
|
|
|
|
\begin{versionhistory}
|
|
\vhEntry{0.1}{28.02.2017}{GZ}{Initial Version}
|
|
\vhEntry{0.1}{02.03.2017}{RA}{Adding basic structure}
|
|
\end{versionhistory}
|
|
|
|
% \intro{Abstract} % Insert abstract here
|
|
% \intro{Acknowledgements} % Uncomment (if needed) for acknowledgements
|
|
\tableofcontents % Uncomment (if needed) for final draft
|
|
% \listoffigures % Uncomment (if needed) for final draft
|
|
% \listoftables % Uncomment (if needed) for final draft
|
|
\cleardoublepage
|
|
\setcounter{secnumdepth}{2}
|
|
|
|
\newcommand{\fnname}[1]{\textit{\texttt{#1}}}%
|
|
\newcommand{\datatype}[1]{\textit{\texttt{#1}}}%
|
|
\newcommand{\varname}[1]{\texttt{#1}}%
|
|
\newcommand{\keywname}[1]{\textbf{\texttt{#1}}}%
|
|
\newcommand{\pathname}[1]{\texttt{#1}}%
|
|
\newcommand{\tabindent}{\hspace*{3ex}}%
|
|
\newcommand{\sockeye}{\lstinline[language=sockeye]}
|
|
\newcommand{\ccode}{\lstinline[language=C]}
|
|
|
|
\lstset{
|
|
language=C,
|
|
basicstyle=\ttfamily \small,
|
|
keywordstyle=\bfseries,
|
|
flexiblecolumns=false,
|
|
basewidth={0.5em,0.45em},
|
|
boxpos=t,
|
|
captionpos=b
|
|
}
|
|
|
|
\chapter{Introduction}
|
|
\label{chap:introduction}
|
|
|
|
This document describes the way Barrelfish boots cores on supported
|
|
architectures. We first explain the terminology used throughout this document.
|
|
Then, we will give an overview of supported operations and finally, explain in
|
|
detail how the implementation of these operations on differ for various
|
|
supported architectures.
|
|
|
|
\section{Boot drivers}
|
|
|
|
Barrelfish uses \textit{boot drivers}, which is a piece of code running on a
|
|
``home core'' and manages a ``target core''. It encapsulates the hardware
|
|
functionality to boot, suspend, resume, and power-down the latter. Boot drivers
|
|
run as processes, but closely resemble device drivers and could equally run as
|
|
software objects within another process. The boot driver abstraction treats CPU
|
|
cores much like peripheral devices, and allows us to reuse the OS's existing
|
|
device and hot-plug management infrastructure to handle new cores and select
|
|
drivers and kernels for them. The code for the Barrelfish boot driver can be
|
|
found in \texttt{usr/drivers/cpuboot/}. From the source folder, a binary called
|
|
``corectrl'' is generated at build time and can be used to do target core
|
|
management. The ``corectrl'' program is a command line tool that supports
|
|
different operations as described below.
|
|
|
|
\section{CPU driver}
|
|
|
|
Is the binary that runs on a core and executes in privileged mode. CPU drivers
|
|
are the Barrelfish equivalent of a kernel, except that there is one per core,
|
|
and they share no state or synchronization. CPU drivers are typically
|
|
non-preemptible, single-threaded, and mostly stateless. The CPU driver binary is
|
|
selected and loaded by a \textit{boot driver}, which among other things, is
|
|
responsible of making sure the CPU driver is executed on a core. TN-021 contains
|
|
a more detailed explanation of CPU drivers and the provided functionality.
|
|
|
|
The code for the Barrelfish CPU drivers can be found in \texttt{kernel/}.
|
|
|
|
\section{Kernel Control Block (KCB)}
|
|
|
|
The Kernel Control Block (or KCB) is a memory region that can only be accessed
|
|
by kernel-space. However, a program, especially boot drivers, can create and
|
|
reference KCBs in user-space through the capability system. The Kernel Control
|
|
Block's purpose is to hold or keep pointers to all per-core state. All per-KCB
|
|
state maintained by a kernel must be reachable from the KCB itself or
|
|
reconstructible by reading the KCB.
|
|
|
|
Usually, a KCB is created for each core we start and passed to the new core on
|
|
start-up. On boot-up, the kernel checks if the KCB it received is initialized.
|
|
In that case, it will use the KCB and start to dispatch the applications it
|
|
contains. If the KCB is uninitialized, the kernel sets all the entries of the
|
|
KCB to its default values and continues bootstrapping the core (and the KCB
|
|
block) by starting the monitor application, which is critical for communication
|
|
across cores in user-space.
|
|
|
|
Every kernel has the possibility to maintain multiple KCBs. The kernel maintains
|
|
a ring structure of KCBs it owns. A CPU driver supports switching to another KCB
|
|
on demand. In case a CPU driver is running multiple KCBs it divides the
|
|
time-slices evenly among different KCBs and switches KCBs after a given amount
|
|
of time has passed. Currently, the CPU driver uses round-robin scheduling for
|
|
having multiple KCBs on one core but other models are possible as well.
|
|
|
|
|
|
\chapter{Generic Operations}
|
|
\label{chap:generic}
|
|
|
|
The boot driver (currently called ``corectrl'') has support for a range of
|
|
operations, which we describe in more detail in this chapter. As a summary the
|
|
operations are listed here:
|
|
|
|
\begin{itemize}
|
|
\item boot \textless target coreid\textgreater: Boots a new core with a KCB.
|
|
\item stop \textless target coreid\textgreater: Stop execution on an existing core.
|
|
\item update \textless target coreid\textgreater: Update the CPU driver on an existing core.
|
|
\item give \textless from kcbid\textgreater \textless to kcbid\textgreater: Give kcb from one core to another.
|
|
\item rmkcb \textless kcbid\textgreater: Remove a running KCB from a core.
|
|
%\item park \textless kcbid to stop\textgreater \textless recv kcbid\textgreater: Stop execution on an existing core and park KCB on another core.
|
|
%\item unpark \textless kcbid to unpark\textgreater: Reestablish parked KCB on its original core.
|
|
\item lscpu: List current status of all cores.
|
|
\item lskcb: List current KCBs.
|
|
\end{itemize}
|
|
|
|
Note that the implementation of ``corectrl'' in it's current from is essentially
|
|
a command line tool that must be invoked for every operation. This means that
|
|
the boot driver is currently state-less which is fine for just executing the
|
|
operations since all information required can be reconstructed from the SKB.
|
|
|
|
There are a few optional flags that can be passed to the different operations:
|
|
\begin{itemize}
|
|
\item -d, --debug: Print debug information
|
|
\item -k, --kernel \textless binary\textgreater: Overwrite default kernel binary.
|
|
\item -x, --monitor \textless binary\textgreater: Overwrite default monitor binary.
|
|
\item -a, --kargs \textless args\textgreater: Overwrite default kernel command line arguments.
|
|
\item -n, --newkcb: Create a new KCB even if there is already one for that core.
|
|
\item -m, --nomsg: Don't wait for a monitor message.
|
|
\end{itemize}
|
|
|
|
|
|
\section{boot \textless target coreid\textgreater}
|
|
|
|
Starts a CPU driver on a core. If the new core that has never been started
|
|
before, it is brought online as follows:
|
|
|
|
\begin{penumerate}
|
|
|
|
\item The new core is detected by some platform-specific mechanism (e.g., ACPI)
|
|
and its appearance registered with the device management subsystem. This is done
|
|
by adding an octopus record to the SKB. The record name is of the format
|
|
\texttt{hw.processor.ID} where ID is an identifier guaranteed to be unique by the
|
|
SKB/Octopus. The record has the following mandatory, architecture independent
|
|
fields:
|
|
|
|
\begin{itemize}
|
|
|
|
\item \texttt{enabled}: A boolean field that says if the core is usable or not
|
|
(for example, on x86 if hyper-threads are disabled, this flag would be set to
|
|
false for hyper-threads -- the resulting core would not be booted by default).
|
|
|
|
\item \texttt{barrelfish\_id}: A unique integer identifier used by Barrelfish to
|
|
refer to this core.
|
|
|
|
\item \texttt{hw\_id}: An identifier assigned by the hardware/platform to this
|
|
core.
|
|
|
|
\item \texttt{type}: A number that corresponds to that cores architecture as
|
|
described in \texttt{enum cpu\_type} (defined in barrelfish\_kpi/cpu.h).
|
|
|
|
\end{itemize}
|
|
|
|
\item Barrelfish selects and starts an appropriate boot driver for the new core.
|
|
This is done by propagating the added octopus record to kaluga which then
|
|
invokes ``corectrl'' with the boot command.
|
|
|
|
\item The boot driver selects a kernel binary and arguments for the new core,
|
|
and directs the boot driver to boot the kernel on the core. By default
|
|
``corectrl'' uses the \texttt{type} field in the octopus record to decide which
|
|
binary to spawn on the target core. However, this can be overridden by manually
|
|
selecting a kernel with the --kernel flag. Alternatively, the default
|
|
parameters passed to the kernel can be overridden with the --kargs option.
|
|
|
|
\item The boot driver loads and relocates the kernel, and executes the hardware
|
|
protocol to start the new core. This is done by using the file system to load
|
|
the binaries and additionally a series of system calls to invoke protected
|
|
operations (starting a core, sending an IPI etc.). In this process, also a new
|
|
KCB is created for the core and initialized with default arguments.
|
|
|
|
\item The new kernel initializes and uses existing Barrelfish protocols for
|
|
integrating into the running OS. This involves spawning the first user-space
|
|
program (the monitor) on the new core. The default choice
|
|
for the first user-space program loaded on the new core can be overridden
|
|
by using the --monitor flag.
|
|
|
|
\item The monitor will inform the other boot driver about its existence by
|
|
sending a message back to it. Then, the boot driver will tell its local monitor
|
|
about the newly available and initialized core. In some cases (for example if
|
|
something else is spawned than the default monitor binary) we may not want to
|
|
wait for such a message. If the --nomsg flag is passed to ``corectrl'' this
|
|
step is skipped.
|
|
|
|
\item The SKB is updated with information about the new core: The core is marked
|
|
online and booted. The KCB that was created during the boot process by
|
|
``corectrl'' is stored in the capability storage in case it needs to be
|
|
retrieved at a different point in time.
|
|
|
|
\end{penumerate}
|
|
|
|
At least three special cases need to be considered:
|
|
|
|
\begin{itemize}
|
|
|
|
\item The core id is invalid: If we don't find the core in the SKB,
|
|
the operation is aborted.
|
|
|
|
\item The core is already running: In that case we abort with an error.
|
|
|
|
\item The core has been booted before: By default ``corectrl'' will check if
|
|
an existing KCB for the given core already exists. If yes, the capability
|
|
for the KCB will be retrieved from the SKB and passed to the new core. This
|
|
behavior can be overridden by passing the --newkcb flag which makes sure
|
|
that an existing KCB is not reused.
|
|
|
|
\end{itemize}
|
|
|
|
|
|
\section{stop \textless target coreid\textgreater}
|
|
|
|
Stops a CPU driver on a core. If the core is already stopped, nothing is done.
|
|
The KCB running on this core will stop dispatching until the core is either
|
|
restarted, or the KCB is given to another core.
|
|
|
|
In general the operations executed is follow:
|
|
|
|
\begin{itemize}
|
|
|
|
\item Invoke hardware specific protocol to stop the core.
|
|
|
|
\item Optional: Update meta-data in SKB to mark core as stopped.
|
|
|
|
\end{itemize}
|
|
|
|
This operations does not take any optional flags or additional arguments.
|
|
|
|
Note that while this is deliberately very simple and low-overhead, just having
|
|
the KCB stop to execute is generally a bad idea as there may be messages waiting
|
|
to be received on that core etc. which can cause the whole system to lock-up. It
|
|
makes sense for example to park the KCB after stopping (see the give operation
|
|
later).
|
|
|
|
\section{update \textless target coreid\textgreater}
|
|
|
|
Update is implemented as a sequence of executing the \texttt{stop} and
|
|
\texttt{start} command. It takes the same arguments and flags as described in
|
|
the start section.
|
|
|
|
\section{give \textless target kcbid\textgreater \textless destination kcbid\textgreater}
|
|
|
|
Removes a KCB on the core it is currently executing and transfers it to
|
|
the core that is currently dispatching the destination KCB.
|
|
This command involves several steps:
|
|
|
|
\begin{penumerate}
|
|
|
|
\item The boot driver instructs its local monitor to send a message to the
|
|
monitor of the target KCB. The message instructs the receiving monitor to inform
|
|
its local CPU driver to stop dispatching the KCB (or OSNode).
|
|
|
|
\item The boot driver sends message to the monitor of the destination KCB,
|
|
passing along the cap reference of the target KCB. Upon receipt, the destination
|
|
monitor will hand the reference to its CPU driver.
|
|
|
|
\item The destination CPU driver will do some minimal initialization, resetting
|
|
timers and scheduling state, check for registered interrupts and up-call drivers
|
|
to re-register them, and then start dispatching the target KCB.
|
|
|
|
\item The boot driver will update the SKB to change the core assigned to the
|
|
target kcbid.
|
|
|
|
\end{penumerate}
|
|
|
|
|
|
The following special cases need to be considered:
|
|
|
|
\begin{itemize}
|
|
|
|
\item The target or destination KCB ID is invalid: The operation is aborted.
|
|
|
|
\item The destination KCB is not currently assigned a core: The operation is
|
|
aborted.
|
|
|
|
\item The target KCB is not currently running on a core: The step where we
|
|
first send a message to the target KCB is skipped.
|
|
|
|
\end{itemize}
|
|
|
|
|
|
\section{rmkcb \textless kcbid\textgreater}
|
|
|
|
\begin{penumerate}
|
|
|
|
\item The boot driver instructs its local monitor to send a message to the
|
|
monitor of the target KCB.
|
|
|
|
\item Upon receipt, the monitor of the target KCB will instruct the CPU driver
|
|
to remove the KCB from the scheduling queue. If queue is now empty, the CPU
|
|
driver will stop itself, otherwise it will switch to the next KCB in the list.
|
|
|
|
\end{penumerate}
|
|
|
|
The following special cases need to be considered:
|
|
|
|
\begin{itemize}
|
|
|
|
\item The KCB id is invalid: If we don't find the KCB capability in the SKB,
|
|
the operation is aborted.
|
|
|
|
\item The KCB is not dispatched on any core: If the KCB is not currently
|
|
associated with any core, it can't be removed and therefore the operation
|
|
is a NOP.
|
|
|
|
\end{itemize}
|
|
|
|
\todo{Don't remember exactly if we need to send this to KCBID we want to
|
|
remove or the core/KCB that is currently hosting us...}
|
|
|
|
\chapter{Booting x86 Cores}
|
|
\label{chap:x86}
|
|
|
|
\section{Core discovery and identification}
|
|
|
|
\section{Booting a new core}
|
|
|
|
|
|
|
|
\chapter{Booting ARMv7 Cores}
|
|
\label{chap:armv7}
|
|
\todo{explain ARMv7 specific boot arguements / protocols}
|
|
|
|
\section{Core discovery and identification}
|
|
|
|
\section{Booting a new core}
|
|
|
|
\chapter{Booting ARMv8 Cores}
|
|
\label{chap:armv8}
|
|
|
|
\todo{explain ARMv8 specific boot arguements / protocols}
|
|
|
|
\section{Core discovery and identification}
|
|
|
|
\section{Booting a new core}
|
|
%
|
|
%http://infocenter.arm.com/help/topic/com.arm.doc.den0044b/DEN0044B_Server_Base_Boot_Requirements.pdf
|
|
|
|
\subsection{Power State Coordination Interface (PSCI)}
|
|
|
|
%
|
|
%http://infocenter.arm.com/help/topic/com.arm.doc.den0022c/DEN0022C_Power_State_Coordination_Interface.pdf
|
|
|
|
\subsection{Parking Protocol}
|
|
%https://acpica.org/sites/acpica/files/MP%20Startup%20for%20ARM%20platforms.doc
|
|
|
|
\subsection{All at once}
|
|
Raspberry Pi
|
|
|
|
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
\bibliographystyle{abbrv}
|
|
\bibliography{barrelfish}
|
|
|
|
\end{document}
|