906 lines
35 KiB
TeX
906 lines
35 KiB
TeX
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
% Copyright (c) 2013-2016, ETH Zurich.
|
||
% All rights reserved.
|
||
%
|
||
% This file is distributed under the terms in the attached LICENSE file.
|
||
% If you do not find this file, copies can be found by writing to:
|
||
% ETH Zurich D-INFK, Universitaetstrasse 6, CH-8092 Zurich. Attn: Systems Group.
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
|
||
\documentclass[a4paper,twoside]{report} % for a report (default)
|
||
|
||
\usepackage{bftn} % You need this
|
||
\usepackage{multirow}
|
||
\usepackage{listings}
|
||
\usepackage{color}
|
||
\usepackage{xspace}
|
||
|
||
\title{Barrelfish on ARMv7-A} % title of report
|
||
\author{Simon Gerber \and Stefan Kaestle \and Timothy Roscoe \and
|
||
Pravin Shinde \and Gerd Zellweger}
|
||
\tnnumber{017} % give the number of the tech report
|
||
\tnkey{ARMv7-A} % Short title, will appear in footer
|
||
|
||
% \date{Month Year} % Not needed - will be taken from version history
|
||
|
||
\newcommand{\todo}[1]{\note{\textbf{TODO:} #1}}
|
||
|
||
\begin{document}
|
||
\maketitle
|
||
|
||
\newcommand{\code}[1]{{\lstinline!#1!}}
|
||
\newcommand{\file}[1]{{\lstinline!#1!}}
|
||
\newcommand{\mode}[1]{\texttt{#1} mode\xspace}
|
||
|
||
%configure listings properly
|
||
\lstset{%
|
||
basicstyle=\small\ttfamily,
|
||
escapechar=@
|
||
}
|
||
|
||
%
|
||
% Include version history first
|
||
%
|
||
\begin{versionhistory}
|
||
\vhEntry{0.1}{05.12.2013}{SK}{Initial version}
|
||
\vhEntry{0.2}{08.12.2015}{TR}{Rewritten for new ARMv7 code}
|
||
\vhEntry{1.0}{31.05.2016}{TR}{Newly-factored ARMv7 platform support}
|
||
\end{versionhistory}
|
||
|
||
% \intro{Abstract} % Insert abstract here
|
||
% \intro{Acknowledgements} % Uncomment (if needed) for acknowledgements
|
||
\tableofcontents % Uncomment (if needed) for final draft
|
||
% \listoffigures % Uncomment (if needed) for final draft
|
||
% \listoftables % Uncomment (if needed) for final draft
|
||
|
||
\lstset{
|
||
language=C,
|
||
basicstyle=\ttfamily \small,
|
||
flexiblecolumns=false,
|
||
basewidth={0.5em,0.45em},
|
||
boxpos=t,
|
||
}
|
||
|
||
\newcommand{\eclipse}{ECL\textsuperscript{i}PS\textsuperscript{e}\xspace}
|
||
\newcommand{\codesize}{\scriptsize}
|
||
\newcommand{\note}[1]{[\textcolor{red}{\emph{#1}}]}
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\chapter{Introduction}
|
||
|
||
This document describes the state of support for ARMv7-A processors in
|
||
Barrelfish.
|
||
|
||
ARM hardware is highly diverse, and has evolved over time. As a
|
||
research OS, Barrelfish focusses ARM support on a small number of
|
||
platforms based on wide availability, ease of maintenance, and
|
||
research interest. However, since management of hardware complexity
|
||
and diversity is also a research goal of the Barrelfish project, we
|
||
aim to make it easy to add new ARM-based platforms with a mixture of
|
||
traditional and non-traditional engineering techniques.
|
||
|
||
The principal processors with 32-bit ARM support in Barrelfish at present are
|
||
ARMv7-A (Cortex A-series), in particular the Cortex A9.
|
||
|
||
Past support for older ARM 32-bit architectures in Barrelfish included:
|
||
\begin{itemize}
|
||
\item ARMv7m (Cortex M-series), in particular the Cortex M3.
|
||
\item ARMv5 processors, in particular the Intel iXP2800 network
|
||
processor (which uses an XScale core).
|
||
\item ARMv6 (ARM11MP) processors running under simulation in
|
||
\file{qemu}.
|
||
\end{itemize}
|
||
|
||
The main 32-bit ARM-based systems we target at present are:
|
||
\begin{itemize}
|
||
\item The Texas Instruments OMAP4460 SoC used in the Pandaboard ES
|
||
platform.
|
||
\item The ARM VExpress\_EMM board, under emulation in the GEM5
|
||
simulator.
|
||
\end{itemize}
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\chapter{Compilation}
|
||
\label{sec:armcompile}
|
||
|
||
Building Barrelfish with ARMv7 is straightforward; detailed
|
||
requirements for packages are described in the latest README file.
|
||
|
||
Compiling ARM support in Barrelfish requires a cross-compilation
|
||
toolchain on the programmers \code{PATH}. For ARMv7 support we
|
||
track the GNU toolchain shipped with Ubuntu LTS (14.04.3 at time of
|
||
writing).
|
||
|
||
Once you have the right tools, run hake with the correct options,
|
||
e.g.:
|
||
\begin{lstlisting}
|
||
$ cd /build/barrelfish
|
||
$ /git/barrelfish/hake/hake.sh -a armv7 -s /git/barrelfish
|
||
...
|
||
$
|
||
\end{lstlisting}
|
||
|
||
After running \code{hake} with appropriate architecture support
|
||
(i.e. use \code{-a armv7}), you can ask the Makefile what platforms it
|
||
supports:
|
||
|
||
\begin{lstlisting}
|
||
$ make help-platforms
|
||
------------------------------------------------------------------
|
||
Platforms supported by this Makefile. Use 'make <platform name>':
|
||
(these are the platforms available with your architecture choices)
|
||
|
||
Documentation:
|
||
Documentation for Barrelfish
|
||
PandaboardES:
|
||
Standard Pandaboard ES build image and modules
|
||
ARMv7-GEM5:
|
||
GEM5 emulator for ARM Cortex-A series multicore processors
|
||
------------------------------------------------------------------
|
||
$
|
||
\end{lstlisting}
|
||
|
||
Then build:
|
||
|
||
\begin{lstlisting}
|
||
$ make -j 8 PandaboardES
|
||
\end{lstlisting}
|
||
|
||
\section{Building for GEM5}
|
||
|
||
To boot Barrelfish in GEM5, in addition to the previous steps you
|
||
will need a supported version of GEM5. The GEM5 website
|
||
(\url{gem5.org}) has comprehensive information.
|
||
|
||
Unfortunately, different
|
||
versions of GEM5 manifest different subtle bugs when emulating ARM
|
||
systems. We recommend revision 0fea324c832c of GEM5 at present;
|
||
please let us know if you find a more recent version that works well.
|
||
|
||
To fetch and build GEM5 on Ubuntu LTS:
|
||
|
||
\begin{lstlisting}
|
||
$ sudo apt-get install scons swig python-dev libgoogle-perftools-dev m4 protobuf-compiler libprotobuf-dev
|
||
$ hg clone http://repo.gem5.org/gem5 -r 0fea324c832c gem5
|
||
adding changesets
|
||
adding manifests
|
||
adding file changes
|
||
added 9356 changesets with 53499 changes to 6576 files
|
||
updating to branch default
|
||
3269 files updated, 0 files merged, 0 files removed, 0 files unresolved
|
||
$ cd ./gem5
|
||
$ scons build/ARM/gem5.fast
|
||
...
|
||
|
||
$
|
||
\end{lstlisting}
|
||
|
||
GEM5 is a large system and may take some time to build. In addition,
|
||
you may have to install minor fixes to ensure compilation (I had to
|
||
add some initializers to \file{mem/ruby/network/orion/Wire.cc}, for
|
||
example).
|
||
|
||
After the compilation of GEM5 is finished, add the binary to your PATH.
|
||
|
||
Now, build Barrelfish like this:
|
||
\begin{lstlisting}
|
||
$ make -j 8 ARMv7-GEM5
|
||
\end{lstlisting}
|
||
|
||
It's a good idea to set \code{armv7_platform} in
|
||
\file{<build_dir>/hake/Config.hs} to \texttt{gem5} in order to enable
|
||
the cache quirk workarounds for GEM5 and proper offsets for the
|
||
platform simulated by GEM5.
|
||
|
||
You can also build Barrelfish and boot inside GEM5 in a single step:
|
||
|
||
\begin{lstlisting}
|
||
$ make help-boot
|
||
------------------------------------------------------------------
|
||
Boot instructions supported by this Makefile. Use 'make <boot name>':
|
||
(these are the targets available with your architecture choices)
|
||
|
||
gem5_armv7:
|
||
Boot an ARMv7a multicore image in GEM5
|
||
gem5_armv7_detailed:
|
||
Boot an ARMv7a multicore image in GEM5 using a detailed CPU model
|
||
$ make gem5_armv7
|
||
...
|
||
\end{lstlisting}
|
||
|
||
To get the output of Barrelfish you should:
|
||
\begin{lstlisting}
|
||
$ telnet localhost 3456
|
||
\end{lstlisting}
|
||
|
||
GEM5 is a highly configurable simulator. You can print the supported
|
||
options of the GEM5 script as follows:
|
||
|
||
\begin{lstlisting}
|
||
$ gem5.fast gem5/gem5script.py -h
|
||
\end{lstlisting}
|
||
|
||
Note that if you boot using \code{make arm_gem5_detailed} rather than
|
||
\code{make arm_gem5}, the simulation takes a long time (depending on
|
||
your machine up to an hour just to boot Barrelfish).
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\chapter{Hardware assumptions and limitations}
|
||
|
||
The current state of ARMv7 support in Barrelfish makes a number of
|
||
assumptions about the underlying hardware platform, and also imposes
|
||
some limitations. We discuss these here.
|
||
|
||
\section{No support for Large Physical Address Extensions}
|
||
|
||
The current Barrelfish design does not support LPAE for 32-bit ARM
|
||
processors. Instead, it assumes a 32-bit physical address space.
|
||
Supporting LPAE would require changes to the paging code, but would
|
||
also require a mechanism to address user memory from the kernel
|
||
effectively (see below).
|
||
|
||
\section{Physical RAM starts at 2GB}
|
||
|
||
Within the 32-bit physical address space, RAM is assumed to start at
|
||
the 2GB boundary (i.e. \code{0x80000000}). This is the
|
||
architectural recommendation for Cortex-A series processors, and we
|
||
have yet to encounter non-LPAE ARMv7-A hardware which does not do
|
||
this. Changing this assumption in the code should be possible, but in
|
||
practice is likely to be dominated by the other limitations mentioned
|
||
here.
|
||
|
||
\section{Physical RAM is limited to 1GB}
|
||
|
||
The Barrelfish ARMv7 CPU drivers can handle up to 1GB RAM,
|
||
contiguously situated in the physical address space starting at 2GB.
|
||
This limit could be raised by half a Gigabyte or so, at the cost of
|
||
space for mapping kernel devices. In practice, the CPU does not need
|
||
to map many kernel devices since most drivers run in user space on
|
||
Barrelfish. Consequently, the allocation of the top 2GB of the
|
||
virtual address space betwen 1-1 mapped RAM and kernel hardware
|
||
devices could easily be moved.
|
||
|
||
However, it remains that the total RAM visible to the CPU \emph{plus}
|
||
the mappings for any devices needed by the CPU driver must fit into
|
||
the top 2GB of the address space (mapped by the TTBR1 register).
|
||
|
||
In particular, the CPU driver assumes that all physical RAM is mapped
|
||
1-1, and relies on this when performing capability invocations. If
|
||
the system had more RAM that could be mapped 1-1 into kernel virtual
|
||
address space, we would need a method for the CPU driver to quickly
|
||
access arbitrary physical addresses, entailing some kind of paging
|
||
system.
|
||
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\chapter{Organization of the address space}
|
||
|
||
Like many other popular operating systems, Barrelfish employs a memory
|
||
split. The idea behind a memory split is to separate kernel code from
|
||
user space code in the virtual address space. This allows the kernel
|
||
to be mapped in every virtual address space of each user space
|
||
program, which is necessary to allow user space code to access kernel
|
||
features through the system call interface. If the kernel was not
|
||
mapped into the virtual address space of each program, it would be
|
||
impossible to jump to kernel code without switching the virtual
|
||
address space.
|
||
|
||
\begin{figure}[htb]
|
||
\centering
|
||
\includegraphics[width=8cm]{figures/virtual_addressing.pdf}
|
||
\caption{Barrelfish virtual address space layout for ARMv7-A}
|
||
\label{fig:memory_layout}
|
||
\end{figure}
|
||
|
||
Additionally ARMv7-A provides two translation table
|
||
base registers, TTBR0 and TTBR1. We configure the system to use
|
||
TTBR0 for address translations of virtual addresses below 2GB and
|
||
TTBR1 for virtual address above 2GB. This saves us the explicit
|
||
mapping of the kernel pages into every L1 page table of each process.
|
||
Even though the kernel is mapped to each virtual address space, it is
|
||
invisible for the user space program. Accessing memory, which belongs
|
||
to the kernel, leads to a pagefault. Since many mappings can point to
|
||
the same physical memory, memory usage is not increased by this
|
||
technique.
|
||
|
||
Figure~\ref{fig:memory_layout} shows the memory layout of the complete
|
||
virtual address space of a single ARMv7-A core running Barrelfish.
|
||
|
||
We have a memory split at 2GB, where everything upwards is only
|
||
accessible in privileged modes and the lower 2GB of memory is
|
||
accessible for user space programs.
|
||
|
||
The kernel runs out of the kernel virtual address space where system
|
||
RAM is mapped 1-1; in the region between \texttt{0x80000000} and
|
||
\texttt{0xC0000000} RAM is mapped directly physical-to-virtual.
|
||
|
||
The L1 page table of the kernel address space is located inside the
|
||
data segment of the kernel right after the
|
||
kernel and naturally aligned to 16KB.
|
||
|
||
We map the whole available physical memory into the kernel’s virtual
|
||
address space using ``sections'' (1MB large pages), obviating the need
|
||
for a kernel L2 page table.
|
||
|
||
Above \texttt{0xC0000000}, the CPU driver maps regions of physical
|
||
memory corresponding to hardware devices it needs to directly access
|
||
(typically the UARTs, interrupt controller, timers, Snoop Control
|
||
Unit, and a few others). These are also mapped using sections.
|
||
Virtual address regions are allocated in 1MB increments (the size of a
|
||
section mapping) working down from the top section, which is used to
|
||
map the area of RAM containing the CPU driver's exception vectors.
|
||
|
||
Below the \texttt{0x80000000}, all mappings are handled by TTBR0 and
|
||
changed on every context switch. At startup, the kernel uses another
|
||
page table (also 16kB-aligned and located inside its data segment) to
|
||
map low memory virtual-to-physical as well, as a way to access
|
||
hardware devices in this region before the rest of the system has come
|
||
up. However, after the early stages of bootstrap this table is no
|
||
longer used.
|
||
|
||
Instead, TTBR0 is always loaded with the address of a user domain's
|
||
hardware page table and changes on a context switch. TTBR1 does not
|
||
change, ensuring the kernel mappings are static after boot.
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\chapter{Boot sequence}
|
||
|
||
\section{BSP (initial) core}
|
||
|
||
\begin{enumerate}
|
||
\item \file{boot.S:start} is called by the bootloader. It
|
||
sets the processor \mode{System}, sets up the (single) kernel
|
||
stack, the global object table pointer, and jumps to
|
||
\file{arch_init}.
|
||
\item \file{init.c:arch_init} is called with a single
|
||
argument: the address of the multiboot info block. It first
|
||
initializes the serial console \file{serial_early_init} and
|
||
checks to see if this is the BSP. If so, it calls
|
||
\file{bsp_init}.
|
||
\item \file{init.c:bsp_init} reads information from the multiboot
|
||
info into the global data structure, initializing it. It also
|
||
resets global spinlocks, and sizes RAM (though this information is
|
||
not yet used). It returns.
|
||
\item \file{init.c:arch_init} continues by initialzing paging,
|
||
calling:
|
||
\item \file{paging.c:paging_init} populates the two initial page
|
||
tables (one for each base register). The kernel (upper) page table
|
||
is initialized to map 1GB of RAM at 0x80000000, and the exception
|
||
vectors at the top of memory. The initial user (lower) page table
|
||
is set to map the lower 2GB of the physical address space 1-1 to
|
||
enable early device access. The MMU is then enabled.
|
||
\item \file{init.c:arch_init} continues with the MMU enabled by
|
||
jumping at:
|
||
\item \file{init.c:arch_init_2} which initializes exceptions,
|
||
relocating the current KCB, parses the command line arguments, and
|
||
re-initializes the serial ports so that the UART hardware is now
|
||
mapped correctly into kernel address space with a section mapping.
|
||
|
||
It then initializes the GIC, the Snoop Control Unit, the Global
|
||
Timer, and the Time Slice Counter. Cycle counter access from
|
||
\mode{User} is enabled, and the coreboot spawn handler set up. It
|
||
then calls:
|
||
|
||
\item \file{startup_arch.c:arm_kernel_startup} which initializes
|
||
a simple memory allocator from the global structure, allocates the
|
||
a new KCB, and calls:
|
||
|
||
\item \file{startup_arch.c:spawn_bsp_init} which creates the
|
||
initial kernel data structures for spawning the init process. It
|
||
also creates the initial capabilities for init to use to allocate
|
||
memory, and returns.
|
||
|
||
\item \file{startup_arch.c:arm_kernel_startup} continues
|
||
but calling \code{dispatch} on the init DCB, and we are now up and
|
||
running.
|
||
\end{enumerate}
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\chapter{Exception code paths}
|
||
|
||
ARMv7-A exceptions are initialized in
|
||
\file{exceptions.S:exceptions_init}, which for some reason is
|
||
written in assembly. It assumes the core is running in \mode{System}.
|
||
|
||
There is a 256-byte statically-allocated stack for each exception
|
||
mode, and an 8kB stack used for subsequently calling into C in
|
||
\mode{System}, all defined in \file{exceptions.S}.
|
||
|
||
Most exception handlers in the vector table start by checking whether
|
||
the processor was in \mode{User} or not when the trap happened. In most
|
||
cases, if the processor was not in \mode{User}, the result is that
|
||
\mode{System} is entered and the processor jumps to
|
||
\file{exn.c:fatal_kernel_fault}, which panics. Exceptions to this
|
||
rule are noted below.
|
||
|
||
For exceptions taken while the processor is in \mode{User}, the
|
||
address of the current (user space) dispatcher is loaded (macro \\
|
||
\code{get_dispatcher_shared_arm}), and a check is made to see if the
|
||
dispatcher is ``enabled'' (in other words, whether the dispatcher
|
||
should be upcalled when next dispatched).
|
||
|
||
This latter check is performed by the macro \code{disp_is_disabled},
|
||
and returns non-zero if:
|
||
\begin{enumerate}
|
||
\item The \code{disabled} value in the dispatcher (at offset
|
||
\code{OFFSET_OF_DISP_DISABLED}) is non-zero, \emph{or}
|
||
\item The PC lies between the two values in the dispatcher with
|
||
offsets \code{OFFSETOF_DISP_CRIT_PC_LOW} and
|
||
\code{OFFSETOF_DISP_CRIT_PC_HIGH}\footnote{A trick suggested by
|
||
Justin Cappos to allow an atomic resume of a user-level thread
|
||
without entering the kernel}.
|
||
\end{enumerate}
|
||
|
||
Depending on this, context is saved in a different area of the
|
||
dispatcher, \mode{System} is entered, and a call is made to C code as
|
||
noted below.
|
||
|
||
Taking each exception in turn:
|
||
|
||
\section{Reset exception}
|
||
|
||
This is vector 0x00, and is not used.
|
||
|
||
\section{Undefined Instruction exception}
|
||
|
||
This is vector offset 0x04, and is referred to as
|
||
\code{ARM_EVECTOR_UNDEF} in the source.
|
||
The processor enters \code{undef_handler} in \mode{Undefined}.
|
||
Context is saved in either the \texttt{ENABLED} or \texttt{TRAP} area.
|
||
C is entered at \code{exn.c:handle_user_undef}.
|
||
|
||
\section{Supervisor call (software interrupt)}
|
||
|
||
This is vector offset 0x08, and referred to as
|
||
\code{ARM_EVECTOR_SWI} in the source.
|
||
The processor enters \code{swi_handler} in \mode{Supervisor}.
|
||
|
||
If the syscall was issued from user space, context is saved in either
|
||
the \code{ENABLED} or \code{DISABLED} area. C is entered at
|
||
\code{syscall.c:sys_syscall}.
|
||
|
||
If the syscall was issued from kernel space, no context is saved and
|
||
C is entered at \code{syscall.c:sys_syscall_kernel}.
|
||
|
||
\section{Prefetch Abort exception}
|
||
|
||
This is vector offset 0x0C, and referred to as
|
||
\code{ARM_EVECTOR_PABT} in the source.
|
||
The processor enters \code{pabt_handler} in \mode{Abort}.
|
||
|
||
Context is saved in either the \texttt{ENABLED} or \texttt{TRAP} area.
|
||
C is entered at \code{exn.c:handle_user_page_fault}.
|
||
|
||
\section{Data Abort exception}
|
||
|
||
This is vector offset 0x10, and referred to as
|
||
\code{ARM_EVECTOR_DABT} in the source.
|
||
The processor enters \code{dabt_handler} in \mode{Abort}.
|
||
|
||
Context is saved in either the \texttt{ENABLED} or \texttt{TRAP} area.
|
||
C is entered at \code{exn.c:handle_user_page_fault} with the faulting
|
||
address in \code{r0}.
|
||
|
||
\section{Hyp Trap, or Hyp mode entry}
|
||
|
||
This is vector offset 0x14, and is not used in Barrelfish.
|
||
|
||
\section{IRQ interrupt}
|
||
|
||
This is vector offset 0x18, and referred to as
|
||
\code{ARM_EVECTOR_IRQ} in the source.
|
||
The processor enters \code{irq_handler} in \mode{IRQ}.
|
||
|
||
If the syscall was issued from user space, context is saved in either
|
||
the \code{ENABLED} or \code{DISABLED} area. C is entered at
|
||
\code{exn.c:handle_irq}.
|
||
|
||
If the syscall was issued from kernel space, context is saved in
|
||
\code{irq_save_area}, \mode{System} is entered, and C is called at
|
||
\code{exn.c:handle_irq}.
|
||
|
||
\section{Fast interrupt}
|
||
|
||
This is vector offset 0x1C, and referred to as
|
||
\code{ARM_EVECTOR_FIQ} in the source.
|
||
The processor enters \code{fiq_handler} in \mode{FIQ}.
|
||
|
||
If the syscall was issued from user space, context is saved in either
|
||
the \code{ENABLED} or \code{DISABLED} area. C is entered at
|
||
\code{exn.c:handle_irq} (as for IRQ).
|
||
|
||
If the syscall was issued from kernel space, context is saved in
|
||
\code{irq_save_area}, \mode{System} is entered, and C is called at
|
||
\code{exn.c:handle_irq} (as for IRQ).
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\chapter{The Dispatch mechanism}
|
||
|
||
Each time a CPU driver decides to switch to running a domain, it
|
||
dispatches the domain in one of two ways:
|
||
|
||
\begin{description}
|
||
\item[RESUME], also known as ``disabled'': in this mode, the domain is
|
||
resumed exactly where it was preempted before, much as in operating
|
||
systems like Unix.
|
||
\item[UPCALL], also known as ``enabled'': as with Scheduler
|
||
Activations, the domain is upcalled at a fixed address with a new
|
||
context on a small, dedicated stack. The context of the
|
||
previously-running thread in teh domain is available to be resumed
|
||
in user space, if the user-level scheduler (also known as the
|
||
activation handler) decides to.
|
||
\end{description}
|
||
|
||
Which one of these happens depends on the state of the domain.
|
||
|
||
When a domain is running in user space (i.e. the kernel is \emph{not}
|
||
executing) the domain is in one of two states, indicated by a
|
||
combination of:
|
||
\begin{itemize}
|
||
\item the \code{disabled} field of the \code{struct
|
||
dispatcher_shared_generic} structure,
|
||
\item the current program counter,
|
||
\item the \code{crit_pc_low} and \code{crit_pc_high} fields of the \code{struct
|
||
dispatcher_shared_generic} structure.
|
||
\end{itemize}
|
||
|
||
Note that all of these values can be written by the user program.
|
||
|
||
Specifically, the domain is in \code{RESUME} state \emph{iff}:
|
||
\begin{enumerate}
|
||
\item \code{disabled} is \code{true}, \emph{or}
|
||
\item the current program counter lies between \code{crit_pc_low} and
|
||
\code{crit_pc_high}
|
||
\end{enumerate}
|
||
|
||
Otherwise, it is in state \code{UPCALL}.
|
||
|
||
Once the kernel is entered, the \code{disabled} flag of the domain's
|
||
\code{struct dcb} structure (as opposed to the \code{struct
|
||
dispatcher_shared_generic}) is updated to reflect the state of the
|
||
preempted domain.
|
||
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\chapter{Key data structures}
|
||
|
||
\begin{itemize}
|
||
\item \code{struct dcb}: in \code{kernel/include/dispatch.h};
|
||
the main domain control block.
|
||
|
||
\code{dcb_current} is a global pointer in the CPU driver that points
|
||
to the current DCB.
|
||
|
||
If \code{dp} is of type \core{struct dcb *}, then
|
||
\code{dp->disabled} is a flag which is 1 if the current DCB has
|
||
activations disabled (i.e. it should be resumed when next scheduled
|
||
to run) and 0 otherwise (in which case it should be upcalled) - the
|
||
analogy is with enabling and disabling interrupts. The flag is set
|
||
on entry to the kernel.
|
||
|
||
\item \code{struct dispatcher_shared_generic}: in
|
||
\code{include/barrelfish_kpi/dispatcher_shared.h}: the
|
||
architecture-independent part of the a dispatcher, the user-space
|
||
datastructure corresponding to a DCB. This is the first struct in
|
||
architecture-dependent variants, such as \code{struct
|
||
dispatcher_shared_arm}.
|
||
|
||
If \code{dp} is of type \core{struct dispatcher_shared_generic *}, then
|
||
\code{dp->disabled} is a flag which is 1 if the current DCB has
|
||
|
||
\end{itemize}
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\chapter{Hardware abstraction layers}
|
||
|
||
Barrelfish distinguishes between:
|
||
\begin{itemize}
|
||
\item General code
|
||
\item Architecture-specific code (e.g. ARMv7-A code)
|
||
\item Platform-specific code (e.g. code for the OMAP4460 SoC)
|
||
\end{itemize}
|
||
|
||
Since most Barrelfish device drivers run in userspace, the difference
|
||
between ``platform'' as a chip (such as the OMAP4460) and ``platform''
|
||
as a board or complete machine (such as the PandaBoard ES) are
|
||
relatively unimportant inside the CPU driver, since most of the
|
||
platform-specific CPU driver code is actually specific to a chip or
|
||
SoC.
|
||
|
||
Barrelfish CPU driver source code for ARMv7-A systems therefore
|
||
consists of the following categories:
|
||
\begin{itemize}
|
||
\item Portable, architecture-independent code.
|
||
\item ARMv7-A-specific code which common to all ARMv7-A platforms
|
||
\item Code for particular devices or macrocells which are only used on
|
||
ARMv7-A, but might appear on multiple ARMv7-A platforms.
|
||
\item Platform-specific code.
|
||
\end{itemize}
|
||
|
||
We restrict platform-specific code to a single source file, which
|
||
roughly corresponds to ARM's concept of an ``integrator'', and acts as
|
||
a compilation-time indirection layer between commmon ARMv7-A-specific
|
||
code and individual device and macrocell drivers.
|
||
|
||
\section{The ARMv7-A HAL}
|
||
|
||
Platform code for a Barrelfish ARMv7-A CPU driver must implement the
|
||
following interfaces:
|
||
|
||
\begin{description}
|
||
\item[serial.h]: Low-level drivers for a multiple UART devices.
|
||
\item[spinlock.h]: Some number of static spinlocks, used for
|
||
coordinating access to e.g. serial devices between CPU drivers on
|
||
different cores.
|
||
\end{description}
|
||
|
||
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\chapter{Code organization}
|
||
|
||
The variety of ARM platforms make organizing source trees to maximise
|
||
code reuse across different platforms a challenge.
|
||
|
||
Barrelfish distinguishes between \emph{Architectures}, which are
|
||
typically processor architectures like ``ARMv7-A'', and \emph{Platforms},
|
||
which are complete system targets, like ``PandaBoard-ES''.
|
||
|
||
Code and headers specific to a particular architecture are found in
|
||
the source tree is various subdirectories of the form
|
||
\file{../arch/armv7/}.
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\chapter{Versatile Express platform}
|
||
|
||
%--------------------------------------------------
|
||
\chapter{GEM5 specifics}
|
||
|
||
The GEM5~\cite{gem5:sigarch11} simulator combines the best aspects of
|
||
the M5~\cite{m5:micro06} and GEMS~\cite{gems:sigarch05}
|
||
simulators. With its flexible and highly modular design, GEM5 allows
|
||
the simulation of a wide range of systems. GEM5 supports a wide range
|
||
of ISAs like x86, SPARC, Alpha and, in our case most importantly,
|
||
ARM. In the following we will list some features of GEM5.
|
||
|
||
GEM5 supports four different CPU models: AtomicSimple, TimingSimple,
|
||
In-Order and O3. The first two are simple one-cycle-per-instruction
|
||
CPU models. The difference between the two lies in the way they handle
|
||
memory accesses. The AtomicSimple model completes all memory accesses
|
||
immediately, whereas the TimingSimple CPU models the timing of memory
|
||
accesses. Due to their simplicity, the simulation speed is far above
|
||
the other two models. The InOrder CPU models an in-order pipeline and
|
||
focuses on timing and simulation accuracy. The pipeline can be
|
||
configured to model different numbers of stages and hardware threads.
|
||
The O3 CPU models a pipelined, out-of-order and possibly superscalar
|
||
CPU model. It simulates dependencies between instructions, memory
|
||
accesses, pipeline stages and functional units. With a load/store
|
||
queue and reorder buffer its possible to simulate superscalar
|
||
architectures as well as multiple hardware threads.
|
||
|
||
The GEM5 simulator provides a tight integration of Python into the
|
||
simulator. Python is mainly used for system configuration. Every
|
||
simulated building block of a system is implemented in C++ but are
|
||
also reflected as a Python class and derive from a single superclass
|
||
SimObject. This provides a very flexible way of system construction
|
||
and allows to tailor nearly every aspect of the system to our needs.
|
||
Python is also used to control the simulation, taking and restoring
|
||
snapshots as well as all the command line processing.
|
||
|
||
We use a VExpress\_EMM based system to run Barrelfish. The number of
|
||
cores can be passed as an argument to the GEM5 script. Cores are
|
||
clocked at 1 GHz and main memory is 64 MB starting at 2 GB.
|
||
|
||
\section{Boot process: first (bootstrap) core}
|
||
|
||
% Source: Samuel's thesis, 4.1.1
|
||
|
||
This section gives a high-level overview of the boot up process of the
|
||
Barrelfish
|
||
kernel on ARMv7-a. In subsequent sections we will go more into details
|
||
involved
|
||
in the single steps.
|
||
\begin{enumerate}
|
||
\item Setup kernel stack and ensure privileged mode
|
||
\item Allocate L1 page table for kernel
|
||
\item Create necessary mappings for address translation
|
||
\item Set translation table base register (TTBR) and domain
|
||
permissions
|
||
\item Activate MMU, relocate program counter and stack pointer
|
||
\item Invalidate TLB, setup arguments for first C-function arch init
|
||
\item Setup exception handling
|
||
\item Map the available physical memory in the kernel L1 page table
|
||
\item Parse command line and set corresponding variables
|
||
\item Initialize devices
|
||
\item Create a physical memory map for the available memory
|
||
\item Check ramdisk for errors
|
||
\item Initialize and switch to init’s address space
|
||
\item Load init image from ramdisk into memory
|
||
\item Load and create capabilities for modules defined by menu.lst
|
||
\item Start timer for scheduling
|
||
\item Schedule init and switch to user space
|
||
\item init brings up the monitor and mem serv
|
||
\item monitor spawns ramfsd, skb and all the other modules
|
||
\end{enumerate}
|
||
|
||
\section{Boot process: subsequent cores}
|
||
|
||
% Source: Samuel, 4.2.2
|
||
|
||
The boot up protocol for the multi-core port differs in various ways
|
||
from the boot up procedure of our previous single-core port. We
|
||
therefore include this revised overview here. The first core is called
|
||
the bootstrap processor and every subsequent core is called an
|
||
application processor On bootstrap processor:
|
||
|
||
\begin{enumerate}
|
||
\item Pass argument from bootloader to first C-function arch
|
||
init 18
|
||
\item Make multiboot information passed by bootloader globally
|
||
available
|
||
\item Create 1:1 mapping of address space and alias the same region at
|
||
high memory
|
||
\item Configure and activate MMU
|
||
\item Relocate kernel image to high memory
|
||
\item Reset mapping, only map in the physical memory aliased at high
|
||
memory
|
||
\item Parse command line and set corresponding variables
|
||
\item Initialize devices
|
||
\item Initialize and switch to init’s address space
|
||
\item Load init image into memory
|
||
\item Create capabilities for modules defined by the multiboot info
|
||
\item Schedule init and switch to user space
|
||
\item init brings up the monitor and mem serv
|
||
\item monitor spawns ramfsd, skb and all the other modules
|
||
\item spawnd parses its cmd line and tells the monitor to bring up a
|
||
new core
|
||
\item monitor setups inter-monitor communication channel
|
||
\item monitor allocates memory for new kernel and remote monitor
|
||
\item monitor loads kernel image and relocates it to destination
|
||
address
|
||
\item monitor setups boot information for new kernel
|
||
\item spawnd issues syscall to start new core
|
||
\item Kernel writes entry address for new core into SYSFLAG registers
|
||
\item Kernel raises software interrupt to start new core
|
||
\item Kernel spins on pseudo-lock until other kernel releases it
|
||
\item repeat steps 15 to 23 for each application processor
|
||
\end{enumerate}
|
||
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\chapter{OMAP44xx platform}
|
||
|
||
% Source: Claudio 3.1
|
||
|
||
The OMAP4460 is a system on a chip (SoC) by Texas Instruments,
|
||
intended for use in consumer devices like smartphones and tablet
|
||
computers. It contains:
|
||
|
||
\begin{itemize}
|
||
\item A dual core ARM Cortex-A9 processor
|
||
\item Two ARM Cortex-M3 processors
|
||
\item A hardware spinlock module
|
||
\item A mailbox module
|
||
\item Many devices to process media input and output
|
||
\end{itemize}
|
||
|
||
The intention is that the Cortex-A9 will be running a general purpose
|
||
operating system, while the Cortex-M3 processors will only be running
|
||
a real-time operating system to control the imaging subsystem.
|
||
|
||
The processor configuration in the OMAP4460 is somewhat
|
||
unconventional; for example, the Cortex-M3 processors share a
|
||
custom MMU with page faults handled by code running on the Cortex-A9
|
||
processors and hence are constrained to run in the same virtual
|
||
address at all times. They are also not cache-coherent with the
|
||
Cortex-A9 cores.
|
||
|
||
\section{Compiling and booting}
|
||
|
||
To compile Barrelfish for the Pandaboard, first configure your
|
||
toolchain as described in Section~\ref{sec:armcompile}. Then execute:
|
||
|
||
\begin{lstlisting}
|
||
cd @\shell@SRC
|
||
mkdir build
|
||
cd build
|
||
../hake/hake.sh -a armv7 -s ../
|
||
make pandaboard_image
|
||
\end{lstlisting}
|
||
|
||
The resulting image can be booted on the Pandaboard over the USB OTG
|
||
connector using the standard \texttt{usbboot} utility. It will
|
||
generate console output on the Pandaboard's serial connector.
|
||
|
||
\section{Booting the second OMAP A9 core}
|
||
|
||
% source: AOS m6
|
||
|
||
Here is a brief overview of how the bootstrapping process for the second core
|
||
works: it waits for a signal from the BSP core (an interrupt), and when this
|
||
signal is received, the application core will read an address from a well-
|
||
defined register and start executing the code from this address.
|
||
|
||
To boot the second core, one can write the address of
|
||
a function to the register and send the inter-processor
|
||
interrupt. Following are some pointers to the documentation to help
|
||
understand the bootstrapping process in more detail:
|
||
|
||
\begin{itemize}
|
||
\item Section 27.4.4 in the OMAP44xx manual talks about the boot process for
|
||
application cores.
|
||
\item Pages 1144 \textit{ff.} in the OMAP44xx manual have the register
|
||
layout for the registers that are used in the boot process of the
|
||
second core.
|
||
\end{itemize}
|
||
|
||
Note that the Barrelfish codebase distinguishes between the BSP (bootstrap)
|
||
processor and APP (application) processors. This distinction and naming
|
||
originates from Intel x86 support where the BIOS will choose a
|
||
distinguished BSP processor at start-up and the OS
|
||
is responsible for starting the rest of the processors (the APP
|
||
processors). Although it works somewhat differently on
|
||
ARM, the naming convention is applicable here as well.
|
||
|
||
Note also that the second core will start working with the MMU
|
||
disabled, so is running in physical address space. The bootstrapping
|
||
code sets up a stack, initial page tables and an initial Barrelfish
|
||
dispatcher.
|
||
|
||
\section{Physical address space}
|
||
|
||
At present, a temporary limitation in the core boot protocol means
|
||
that running Barrelfish on both A9 cores requires static partitioning of
|
||
the available RAM into two halves, with an independent memory server
|
||
running on each core. This is will fixed in a subsequent release.
|
||
|
||
\section{Interconnect driver}\label{sec:interconnect}
|
||
|
||
Communication between A9 cores on the OMAP processor is performed
|
||
using a variant of the CC-UMP interconnect driver, modified for the
|
||
32-byte cache line size of the ARMv7 architecture. A notification
|
||
driver for inter-processor interrupts exists.
|
||
|
||
The OMAP4460 also has mailbox hardware which can be used by both the
|
||
A9 and M3 cores. Barrelfish support for this hardware is in
|
||
progress.
|
||
|
||
\section{M3 cores}
|
||
|
||
Barrelfish also has rudimentary support for running on both the A9 and
|
||
M3 cores. This is limited by the requirement that the M3 cores must
|
||
run in the same virtual address space, and do not have a way to
|
||
automatically change address space on a kernel trap. For this reason,
|
||
we only execute on a single M3 core at present.
|
||
|
||
Before the Cortex-M3 can start executing code, the following steps
|
||
have to be taken by the Cortex-A9:
|
||
|
||
\begin{enumerate}
|
||
\item Power on the Cortex-M3 subsystem
|
||
\item Activate the Cortex-M3 subsystem clock
|
||
\item Load the image to be executed into memory
|
||
\item Enable the L2 MMU
|
||
\item Set up mappings for the loaded image in the L2 MMU (can be
|
||
written directly into the TLB)
|
||
\item Write the first two entries of the vectortable (initial sp and
|
||
reset vector)
|
||
\item Take the Cortex-M3 out of reset
|
||
\end{enumerate}
|
||
|
||
It is important to note that the Cortex-M3 is in a virtual address
|
||
space from the very beginning, reading the vector table at virtual
|
||
address 0. Inserting a 1:1 mapping for the kernel image greatly
|
||
simplifies the bootstrapping of memory management on the Cortex-M3
|
||
once it is running, because it needs to know the physical address of
|
||
the page tables it sets up.
|
||
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\bibliographystyle{abbrv}
|
||
\bibliography{defs,barrelfish}
|
||
|
||
\end{document}
|
||
|
||
\end{document}
|