%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Copyright (c) 2015, ETH Zurich. % All rights reserved. % % This file is distributed under the terms in the attached LICENSE file. % If you do not find this file, copies can be found by writing to: % ETH Zurich D-INFK, Universitaetstr 6, CH-8092 Zurich. Attn: Systems Group. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \documentclass[a4paper,11pt,twoside]{report} \usepackage{bftn} \usepackage{calc} \usepackage{verbatim} \usepackage{xspace} \usepackage{pifont} \usepackage{pxfonts} \usepackage{textcomp} \usepackage{amsmath} \usepackage{multirow} \usepackage{listings} \usepackage[framemethod=default]{mdframed} \usepackage[shortlabels]{enumitem} \usepackage{parskip} \usepackage{xparse} \newcommand{\todo}[1]{[\textcolor{red}{\emph{#1}}]} \title{CPU drivers in Barrelfish} \author{Barrelfish project} \tnnumber{21} \tnkey{CPU drivers} \begin{document} \maketitle % Uncomment for final draft \begin{versionhistory} \vhEntry{0.1}{01.12.2015}{GZ}{Initial Version} \end{versionhistory} % \intro{Abstract} % Insert abstract here % \intro{Acknowledgements} % Uncomment (if needed) for acknowledgements \tableofcontents % Uncomment (if needed) for final draft % \listoffigures % Uncomment (if needed) for final draft % \listoftables % Uncomment (if needed) for final draft \cleardoublepage \setcounter{secnumdepth}{2} \newcommand{\fnname}[1]{\textit{\texttt{#1}}}% \newcommand{\datatype}[1]{\textit{\texttt{#1}}}% \newcommand{\varname}[1]{\texttt{#1}}% \newcommand{\keywname}[1]{\textbf{\texttt{#1}}}% \newcommand{\pathname}[1]{\texttt{#1}}% \newcommand{\tabindent}{\hspace*{3ex}}% \newcommand{\sockeye}{\lstinline[language=sockeye]} \newcommand{\ccode}{\lstinline[language=C]} \lstset{ language=C, basicstyle=\ttfamily \small, keywordstyle=\bfseries, flexiblecolumns=false, basewidth={0.5em,0.45em}, boxpos=t, captionpos=b } \chapter{Introduction} \label{chap:introduction} This document describes the CPU driver, the part of Barrelfish that typically runs in privileged mode (also known as kernel) on our supported architectures. Barrelfish currently supports the following CPU drivers for different CPU architectures and platforms: \begin{itemize} \item x86-32 \item x86-64 \item k1om \item ARMv7 \item ... \item ARMv8 \end{itemize} \section{General design decisions} \begin{itemize} \item No dynamic memory allocation \item No preemption \item ... \end{itemize} \section{Code file structure and layout} TODO: Should explain things such as naming, where goes architecture dependent, platform specific code? What libraries we use in the kernel? Where is the shared code between libbarrelfish and a cpudriver? \chapter{x86-64} The x86-64 implementation of Barrelfish is specific to the AMD64 and Intel 64 architectures. This text will refer to features of those architectures. Those and further features can be found in \cite{intelsa} and \cite{amdsa} for the Intel 64 and AMD64 architectures, respectively. \section{Boot process} We first describe the boot process for the initial BSP core, followed by the boot process of an APP core. \subsection{BSP Core} Barrelfish relies on a multiboot v1~\cite{multiboot1} compliant boot-loader to load the initial kernel on the BSP core. In our current set-up we use GRUB as our boot-loader which contains an implementation of the multiboot standard. On start-up, GRUB will search the supplied kernel module (on x86-64 this is the binary called elver in \pathname{tools/elver/}) for a magic byte sequence (defined by multiboot) and begin execution just after that sequence appeared (see \pathname{tools/elver/boot.S}). \pathname{boot.S} in elver will set-up an preliminary GDT, an IA32-e page-table, and stack for execution. \pathname{elver.c} will then search for a binary called \keywname{kernel} or \keywname{cpu} in all the multiboot modules, relocate that module and then jump to the relocated kernel module. At this point, we have set-up a 1 GiB identity mapping of the physical address space using 2 MiB pages in order to address everything we need initially. Note that the reason elver exists is because multiboot v1 does not support ELF64 images (or setting up long-mode). If we use a bootloader that supports loading relocatable ELF64 images into 64-bit mode, elver would be redundant. After \keywname{elver} is done, execution in the proper BSP kernel program begins in \pathname{kernel/arch/x86\_64/boot.S} which then calls \fnname{arch\_init}, the first kernel C entry point. \subsection{APP Core} APP cores are booted using the coreboot infrastructure in Barrelfish. The logic that boots APP cores resides in \pathname{usr/drivers/cpuboot}. The source code responsible for booting a new core on x86 is found in \pathname{usr/drivers/cpuboot/x86boot.c}, specifically in the function called \fnname{spawn\_xcore\_monitor}. \fnname{spawn\_xcore\_monitor} will load the \keywname{kernel} and \keywname{monitor} binary, and relocate the kernel. The function called \fnname{start\_aps\_x86\_64\_start} will afterwards map in the bootstrap code (which is defined in \pathname{init\_ap\_x86\_64.S}) for booting the APP core. One complication for this code is that it has to resides below 1 MiB in physical memory since the new APP core starts in protected mode and therefore can not address anything above that limit in the beginning. Once the mapping is initiated, the entry point address for the new APP kernel will be written into this memory region. Finally, a set of system calls are invoked in order to send the necessary IPIs to bootstrap the new processor. \section{Virtual Address Space} The page table is constructed by copying VNode capabilities into VNodes to link intermediate page tables, and minting Frame / DeviceFrame capabilities into leaf VNodes to perform mappings. When minting a frame capability to a VNode, the frame must be at least as large as the smallest page size. The type-specific parameters are: \begin{enumerate} \item \textbf{Access flags:} The permissible set of flags is PTABLE\_GLOBAL\_PAGE | PTABLE\_ATTR\_INDEX | PTABLE\_CACHE\_DISABLED | PTABLE\_WRITE\_THROUGH. Access flags are set from frame capability access flags. All other flags are not settable from user-space (like PRESENT and SUPERVISOR). \item \textbf{Number of base-page-sized pages to map:} If non-zero, this parameter allows the caller to prevent the entire frame capability from being mapped, by specifying the number of base-page-sized pages of the region (starting from offset zero) to map. \end{enumerate} \todo{address space layout after initialization is done} \section{IO capabilities} IO capabilities provide kernel-mediated access to the legacy IO space of the processor. Each IO capability allows access only to a specific range of ports. The Mint invocation (see \autoref{sec:mint}) allows the permissible port range to be reduced (with the lower limit in the first type-specific parameter, and the upper limit in the second type-specific parameter). At boot, an IO capability for the entire port space is passed to the initial user domain. Aside from being copied or minted, IO capabilities may not be created. \section{Global Descriptor Table (GDT)} The GDT table is loaded by the \fnname{gdt\_reset} function during start-up and statically defined. The table contains the following entries: \begin{tabular}{c|l} Index & Description \\ \hline 0 & NULL segment \\ 1 & Kernel code segment \\ 2 & Kernel stack segment \\ 3 & User stack segment \\ 4 & User code segment \\ 5 & Task state segment \\ 6 & Task state segment (cont.) \\ 7 & Local descriptor table \\ 8 & Local descriptor table (cont.) \\ \end{tabular} \section{Interrupts and Exceptions} The initial (Interrupt Descriptor Table) IDT is set-up by \fnname{setup\_default\_idt} in \pathname{irq.c}. The number of entries in the IDT is set to 256 entries which are initialized in the following way: \begin{tabular}{c|l} Index & Description \\ \hline 0 & Divide Error \\ 1 & Debug \\ 2 & Nonmaskable External Interrupt \\ 3 & Breakpoint \\ 4 & Overflow \\ 5 & Bound Range Exceeded \\ 6 & Undefined/Invalid Opcode \\ 7 & No Math Coprocessor \\ 8 & Double Fault \\ 9 & Coprocessor Segment Overrun \\ 10 & Invalid TSS \\ 11 & Segment Not Present \\ 12 & Stack Segment Fault \\ 13 & General Protection Fault \\ 14 & Page Fault \\ 15 & Unused \\ 16 & FPU Floating-Point Error \\ 17 & Alignment Check \\ 18 & Machine Check \\ 19 & SIMD Floating-Point Exception \\ \hline 32 & \multirow{3}{*}{PIC Interrupts} \\ \vdots{} & \\ 47 & \\ \hline 48 & \multirow{3}{*}{Generic Interrupts} \\ \vdots{} & \\ 61 & \\ \hline 62 & Tracing IPI \\ 63 & Tracing IPI \\ \hline 64 & \multirow{3}{*}{Unused} \\ \vdots{} & \\ 247 & \\ \hline 248 & Halt IPI (Stopping a core) \\ 249 & Inter core vector (IPI notifications) \\ 250 & APIC Timer \\ 251 & APIC Thermal \\ 252 & APIC Performance monitoring interrupt \\ 253 & APIC Error \\ 254 & APIC Spurious interrupt \\ 255 & Unused \\ \end{tabular} The lower 32 interrupts are reserved as CPU exceptions. Except for a double fault exception, which is always handled by the kernel directly, an exception is forwarded to the dispatcher handling the domain on the CPU on which it appeared. Page faults (interrupt 14) are dispatched to the `pagefault` entry point of the dispatcher. All other exceptions are dispatched to the `trap` entry point of the dispatcher. There are 224 hardware interrupts, ranging from IRQ number 32 to 255. The kernel delivers an interrupt that is not an exception and not the local APIC timer interrupt to user-space. The local APIC timer interrupt is used by the kernel for preemptive scheduling and not delivered to user-space. Unused entries will be initialized by a special handler function. The slots reserved for generic interrupts can be allocated by user-space applications. \section{Local Descriptor Table (LDT)} The local descriptor table segment in the GDT will initially point to NULL as no LDT is installed. User-space applications can install their own LDT table which is loaded on context-switching using the \fnname{maybe\_reload\_ldt} function. \section{Registers} \paragraph{Segment registers} Segment registers are initialized by the \fnname{gdt\_reset} function during start-up and each of them points to a GDT entry (index of the GDT table slot for each segment is given in brackets). \begin{itemize} \item[cs] Kernel code segment (1) \item[ds] NULL segment (0) \item[es] NULL segment (0) \item[fs] NULL segment (0) \item[gs] NULL segment (0) \item[ss] Kernel stack segment (2) \end{itemize} We also note that the \keywname{fs} and \keywname{gs} segment registers are preserved and restored across context switches. \paragraph{General purpose registers} \begin{itemize} \item \keywname{rcx} contains the start address when running a dispatcher for the first time. \end{itemize} \todo{Floating point / SIMD} \todo{Machine specific registers (MSR)} \section{Hardware devices} \subsection{Serial port} On x86, the serial device (a PC16550 compatible controller) is initialized for the first time by the BSP core on boot-up. By default serial port \varname{0x3f8} will be used, but the port can be changed by using a command line argument supplied to the kernel. Notable settings for the serial driver are: \begin{itemize} \item Interrupts are disabled. \item FIFOs are enabled. \item No stop bit. \item 8 data bits. \item No parity bit. \item BAUD rate is 115200. \end{itemize} The serial device is later re-initialized into a different state once the serial driver takes over the device. For example, interrupts will then be enabled and handled by the driver. \subsection{PIC -- Programmable Interrupt Controller} \todo{describe} \subsection{xAPIC -- Advanced Programmable Interrupt Controller} \todo{describe} \subsection{System call API} This section describe the architectural system calls that are not common with other architectures. \begin{itemize} \item[7] SYSCALL\_X86\_FPU\_TRAP\_ON: Turn FPU trap on (x86) \item[8] SYSCALL\_X86\_RELOAD\_LDT: Reload the LDT register (x86\_64) \end{itemize} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \bibliographystyle{abbrv} \bibliography{barrelfish} \end{document}