%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Copyright (c) 2015, ETH Zurich.
% All rights reserved.
%
% This file is distributed under the terms in the attached LICENSE file.
% If you do not find this file, copies can be found by writing to:
% ETH Zurich D-INFK, Universitaetstr 6, CH-8092 Zurich. Attn: Systems Group.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass[a4paper,11pt,twoside]{report}
\usepackage{bftn}
\usepackage{calc}
\usepackage{verbatim}
\usepackage{xspace}
\usepackage{pifont}
\usepackage{pxfonts}
\usepackage{textcomp}
\usepackage{amsmath}
\usepackage{multirow}
\usepackage{listings}
\usepackage[framemethod=default]{mdframed}
\usepackage[shortlabels]{enumitem}
\usepackage{parskip}
\usepackage{xparse}

\newcommand{\todo}[1]{[\textcolor{red}{\emph{#1}}]}

\title{CPU drivers in Barrelfish}
\author{Barrelfish project}
\tnnumber{21}
\tnkey{CPU drivers}

\begin{document}
\maketitle			% Uncomment for final draft

\begin{versionhistory}
\vhEntry{0.1}{01.12.2015}{GZ}{Initial Version}
\end{versionhistory}

% \intro{Abstract}		% Insert abstract here
% \intro{Acknowledgements}	% Uncomment (if needed) for acknowledgements
\tableofcontents		% Uncomment (if needed) for final draft
% \listoffigures		% Uncomment (if needed) for final draft
% \listoftables			% Uncomment (if needed) for final draft
\cleardoublepage
\setcounter{secnumdepth}{2}

\newcommand{\fnname}[1]{\textit{\texttt{#1}}}%
\newcommand{\datatype}[1]{\textit{\texttt{#1}}}%
\newcommand{\varname}[1]{\texttt{#1}}%
\newcommand{\keywname}[1]{\textbf{\texttt{#1}}}%
\newcommand{\pathname}[1]{\texttt{#1}}%
\newcommand{\tabindent}{\hspace*{3ex}}%
\newcommand{\sockeye}{\lstinline[language=sockeye]}
\newcommand{\ccode}{\lstinline[language=C]}

\lstset{
  language=C,
  basicstyle=\ttfamily \small,
  keywordstyle=\bfseries,
  flexiblecolumns=false,
  basewidth={0.5em,0.45em},
  boxpos=t,
  captionpos=b
}

\chapter{Introduction}
\label{chap:introduction}

This document describes the CPU driver, the part of Barrelfish that typically
runs in privileged mode (also known as kernel) on our supported architectures.

Barrelfish currently supports the following CPU drivers for
different CPU architectures and platforms:
\begin{itemize}
    \item x86-32
    \item x86-64
    \item k1om
    \item ARMv7
    \item ...
    \item ARMv8
\end{itemize}

\section{General design decisions}
\begin{itemize}
\item No dynamic memory allocation
\item No preemption
\item ...
\end{itemize}

\section{Code file structure and layout}
TODO: Should explain things such as naming, where goes architecture dependent, platform specific code?
What libraries we use in the kernel? Where is the shared code between libbarrelfish and a cpudriver?

\chapter{x86-64}

The x86-64 implementation of Barrelfish is specific to the AMD64
and Intel 64 architectures. This text will refer to features of
those architectures. Those and further features can be found in
\cite{intelsa} and \cite{amdsa} for the Intel 64 and AMD64
architectures, respectively.

\section{Boot process}

We first describe the boot process for the initial BSP core, followed by
the boot process of an APP core.

\subsection{BSP Core}

Barrelfish relies on a multiboot v1~\cite{multiboot1} compliant boot-loader to
load the initial kernel on the BSP core. In our current set-up we use GRUB as
our boot-loader which contains an implementation of the multiboot standard.

On start-up, GRUB will search the supplied kernel module (on x86-64 this is the
binary called elver in \pathname{tools/elver/}) for a magic byte sequence
(defined by multiboot) and begin execution just after that sequence appeared
(see \pathname{tools/elver/boot.S}).

\pathname{boot.S} in elver will set-up an preliminary GDT, an IA32-e page-table,
and stack for execution. \pathname{elver.c} will then search for a
binary called \keywname{kernel} or \keywname{cpu} in all the multiboot
modules, relocate that module and then jump to the relocated kernel module. At
this point, we have set-up a 1 GiB identity mapping of the physical address
space using 2 MiB  pages in order to address everything we need initially.

Note that the reason elver exists is because multiboot v1 does not support
ELF64 images (or setting up long-mode). If we use a bootloader that supports
loading relocatable ELF64 images into 64-bit mode, elver would be redundant.

After \keywname{elver} is done, execution in the proper BSP kernel program
begins in \pathname{kernel/arch/x86\_64/boot.S} which then calls
\fnname{arch\_init}, the first kernel C entry point.

\subsection{APP Core}

APP cores are booted using the coreboot infrastructure in Barrelfish. The
logic that boots APP cores resides in \pathname{usr/drivers/cpuboot}.

The source code responsible for booting a new core on x86 is found in
\pathname{usr/drivers/cpuboot/x86boot.c}, specifically in the function called
\fnname{spawn\_xcore\_monitor}. \fnname{spawn\_xcore\_monitor} will load the
\keywname{kernel} and \keywname{monitor} binary, and relocate the kernel. The
function called \fnname{start\_aps\_x86\_64\_start} will afterwards map in the
bootstrap code (which is defined in \pathname{init\_ap\_x86\_64.S}) for booting the
APP core. One complication for this code is that it has to resides below 1 MiB
in physical memory since the new APP core starts in protected mode and
therefore can not address anything above that limit in the beginning. Once the
mapping is initiated, the entry point address for the new APP kernel will be
written into this memory region. Finally, a set of system calls are invoked
in order to send the necessary IPIs to bootstrap the new processor.

\section{Virtual Address Space}

The page table is constructed by copying VNode capabilities into VNodes to
link intermediate page tables, and minting Frame / DeviceFrame capabilities
into leaf VNodes to perform mappings.

When minting a frame capability to a VNode, the frame must be at least as
large as the smallest page size. The type-specific parameters are:

\begin{enumerate}
  \item \textbf{Access flags:}
    The permissible set of flags is PTABLE\_GLOBAL\_PAGE
    | PTABLE\_ATTR\_INDEX | PTABLE\_CACHE\_DISABLED |
    PTABLE\_WRITE\_THROUGH. Access flags are set from frame capability
    access flags. All other flags are not settable from user-space (like
    PRESENT and SUPERVISOR).

  \item \textbf{Number of base-page-sized pages to map:} If non-zero, this
    parameter allows the caller to prevent the entire frame capability from
    being mapped, by specifying the number of base-page-sized pages
    of the region (starting from offset zero) to map.
\end{enumerate}

\todo{address space layout after initialization is done}

\section{IO capabilities}

IO capabilities provide kernel-mediated access to the legacy IO space of
the processor. Each IO capability allows access only to a specific range of
ports.

The Mint invocation (see \autoref{sec:mint}) allows the permissible
port range to be reduced (with the lower limit in the first
type-specific parameter, and the upper limit in the second type-specific
parameter).

At boot, an IO capability for the entire port space is passed to the
initial user domain. Aside from being copied or minted, IO capabilities may not
be created.

\section{Global Descriptor Table (GDT)}

The GDT table is loaded by the \fnname{gdt\_reset} function during start-up and statically defined.

The table contains the following entries:

\begin{tabular}{c|l}
    Index & Description \\ \hline
    0 & NULL segment  \\
    1 & Kernel code segment  \\
    2 & Kernel stack segment  \\
    3 & User stack segment  \\
    4 & User code segment  \\
    5 & Task state segment  \\
    6 & Task state segment (cont.)  \\
    7 & Local descriptor table \\
    8 & Local descriptor table (cont.) \\
\end{tabular}

\section{Interrupts and Exceptions}

The initial (Interrupt Descriptor Table) IDT is set-up by
\fnname{setup\_default\_idt} in \pathname{irq.c}. The number of entries in the
IDT is set to  256 entries which are initialized in the following way:

\begin{tabular}{c|l}
    Index & Description \\ \hline
    0  &  Divide Error \\
    1  &  Debug \\
    2  &  Nonmaskable External Interrupt \\
    3  &  Breakpoint \\
    4  &  Overflow \\
    5  &  Bound Range Exceeded \\
    6  &  Undefined/Invalid Opcode \\
    7  &  No Math Coprocessor \\
    8  &  Double Fault \\
    9  &  Coprocessor Segment Overrun \\
    10 &  Invalid TSS \\
    11 &  Segment Not Present \\
    12 &  Stack Segment Fault \\
    13 &  General Protection Fault \\
    14 &  Page Fault \\
    15 &  Unused \\
    16 &  FPU Floating-Point Error \\
    17 &  Alignment Check \\
    18 &  Machine Check \\
    19 &  SIMD Floating-Point Exception \\
    \hline
    32 & \multirow{3}{*}{PIC Interrupts} \\
    \vdots{} & \\
    47 & \\
    \hline
    48 & \multirow{3}{*}{Generic Interrupts} \\
    \vdots{} & \\
    61 & \\
    \hline
    62 & Tracing IPI \\
    63 & Tracing IPI \\
    \hline
    64 & \multirow{3}{*}{Unused} \\
    \vdots{} & \\
    247 & \\
    \hline
    248 & Halt IPI (Stopping a core) \\
    249 & Inter core vector (IPI notifications) \\
    250 & APIC Timer \\
    251 & APIC Thermal \\
    252 & APIC Performance monitoring interrupt \\
    253 & APIC Error \\
    254 & APIC Spurious interrupt \\
    255 & Unused \\
\end{tabular}

The lower 32 interrupts are reserved as CPU exceptions. Except for a
double fault exception, which is always handled by the kernel
directly, an exception is forwarded to the dispatcher handling the
domain on the CPU on which it appeared.

Page faults (interrupt 14) are dispatched to the `pagefault` entry
point of the dispatcher. All other exceptions are dispatched to the
`trap` entry point of the dispatcher.

There are 224 hardware interrupts, ranging from IRQ number 32 to 255.
The kernel delivers an interrupt that is not an exception and not
the local APIC timer interrupt to user-space. The local APIC timer
interrupt is used by the kernel for preemptive scheduling and not
delivered to user-space.

Unused entries will be initialized by a special handler function. The slots
reserved for generic interrupts can be allocated by user-space applications.

\section{Local Descriptor Table (LDT)}

The local descriptor table segment in the GDT will
initially point to NULL as no LDT is installed.

User-space applications can install their own LDT table
which is loaded on context-switching using the
\fnname{maybe\_reload\_ldt} function.


\section{Registers}

\paragraph{Segment registers}

Segment registers are initialized by the \fnname{gdt\_reset} function during start-up and each of them points to a GDT entry (index of the GDT table slot for each segment is given in brackets).

\begin{itemize}
\item[cs] Kernel code segment (1)
\item[ds] NULL segment (0)
\item[es] NULL segment (0)
\item[fs] NULL segment (0)
\item[gs] NULL segment (0)
\item[ss] Kernel stack segment (2)
\end{itemize}

We also note that the \keywname{fs} and \keywname{gs} segment registers are
preserved and restored across context switches.

\paragraph{General purpose registers}
\begin{itemize}
\item \keywname{rcx} contains the start address when running a dispatcher
for the first time.
\end{itemize}

\todo{Floating point / SIMD}
\todo{Machine specific registers (MSR)}

\section{Hardware devices}

\subsection{Serial port}
On x86, the serial device (a PC16550 compatible controller) is initialized for the first time by the BSP core on boot-up.

By default serial port \varname{0x3f8} will be used, but the port can be changed by
using a command line argument supplied to the kernel.

Notable settings for the serial driver are:
\begin{itemize}
    \item Interrupts are disabled.
    \item FIFOs are enabled.
    \item No stop bit.
    \item 8 data bits.
    \item No parity bit.
    \item BAUD rate is 115200.
\end{itemize}

The serial device is later re-initialized into a different state once the
serial driver takes over the device. For example, interrupts will then be
enabled and handled by the driver.

\subsection{PIC -- Programmable Interrupt Controller}
\todo{describe}

\subsection{xAPIC -- Advanced Programmable Interrupt Controller}
\todo{describe}

\subsection{System call API}
This section describe the architectural system calls that are not
common with other architectures.

\begin{itemize}
\item[7] SYSCALL\_X86\_FPU\_TRAP\_ON: Turn FPU trap on (x86)
\item[8] SYSCALL\_X86\_RELOAD\_LDT: Reload the LDT register (x86\_64)
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\bibliographystyle{abbrv}
\bibliography{barrelfish}

\end{document}
