258 lines
12 KiB
TeX
258 lines
12 KiB
TeX
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
% Copyright (c) 2011, ETH Zurich.
|
|
% All rights reserved.
|
|
%
|
|
% This file is distributed under the terms in the attached LICENSE file.
|
|
% If you do not find this file, copies can be found by writing to:
|
|
% ETH Zurich D-INFK, Universitaetstrasse 6, CH-8092 Zurich. Attn: Systems Group.
|
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
|
|
|
\documentclass[a4paper,11pt,twoside]{report}
|
|
\usepackage{bftn}
|
|
\usepackage{calc}
|
|
\usepackage{verbatim}
|
|
\usepackage{xspace}
|
|
\usepackage{pifont}
|
|
\usepackage{textcomp}
|
|
\usepackage{amsmath}
|
|
|
|
\title{Message Notifications}
|
|
\author{Barrelfish project}
|
|
% \date{\today} % Uncomment (if needed) - date is automatic
|
|
\tnnumber{9}
|
|
\tnkey{Notifications}
|
|
|
|
\begin{document}
|
|
\maketitle % Uncomment for final draft
|
|
|
|
\begin{versionhistory}
|
|
\vhEntry{1.0}{16.06.2010}{RI}{Initial version}
|
|
\end{versionhistory}
|
|
|
|
% \intro{Abstract} % Insert abstract here
|
|
% \intro{Acknowledgements} % Uncomment (if needed) for acknowledgements
|
|
% \tableofcontents % Uncomment (if needed) for final draft
|
|
% \listoffigures % Uncomment (if needed) for final draft
|
|
% \listoftables % Uncomment (if needed) for final draft
|
|
|
|
|
|
\chapter{Overview}
|
|
|
|
\section{Introduction}
|
|
Inter-core messaging on Barrelfish (UMP) is currently based on shared
|
|
memory circular buffers and a polling mechanism which is designed to
|
|
work efficiently given the cache-coherence protocols of a typical NUMA
|
|
multiprocessor system. Communication latency can vary by many orders
|
|
of magnitude depending on how frequently the receiving process polls
|
|
each channel. This document describes the design and implementation of
|
|
a new kernel notification primitive for Barrelfish.
|
|
|
|
The reason I believe we need an IDC notification path can be seen in
|
|
most of the traces I took of Tim's IDC and THC test program (see
|
|
email of 21/5/10).
|
|
|
|
screenshot goes here
|
|
|
|
If you look on core 0 you see 3 domains polling for incoming URPC
|
|
messages. Each polls for a while and then yields, and with 3 domains
|
|
it takes up to 10000 cycles to notice a message, and obviously the
|
|
current mechanism will scale with the number of domains on the
|
|
destination core. (always >= 2!). This could be reduced by moving
|
|
polling into the kernel, but if any domain is running we have no way
|
|
to pre-empt it until the next timer interrupt (about 18 million
|
|
cycles!).
|
|
|
|
\subsection{Polling and cache coherence}
|
|
Sending a message involves the sender modifying a single cache line
|
|
which (in the expected case) the receiver is actively polling. The
|
|
cache line starts in shared (S) mode in the cache of both sender and
|
|
receiver cores. When the sender writes to the cache line this causes a
|
|
transition to the owned (O) state and an invalidation of the copy in
|
|
the receiver's cache. On most of our NUMA systems, be they
|
|
Hypertransport, QPI or shared bus, this is effectively a system-wide
|
|
broadcast. Newer AMD Istanbul processors have a directory-based cache
|
|
coherency protocol which avoids the broadcast. The receiver then
|
|
pulls the modified cache line from the sender's cache resulting in the
|
|
cache line being in both caches in the shared (S) state.
|
|
|
|
\subsection{Message Latency}
|
|
When sender and receiver threads are the only things running on each
|
|
core this can be extremely low latency (~600 cycles). When the
|
|
destination core is shared by multiple threads, or even multiple
|
|
domains, the message latency is determined by kernel- and user-mode
|
|
scheduling policies and is typically a function of the kernel clock
|
|
interrupt rate and the number of domains (and channels) in the system.
|
|
Even in simple cases the message latency will usually increase to at
|
|
least one timer tick (at least 1ms, probably 10ms - i.e.~millions of
|
|
cycles!)
|
|
|
|
Several Barrelfish papers have talked about sechemes where receiver
|
|
domains initially poll for messages, but eventually back off to a more
|
|
heavyweight blocking mechanism. In the current tree this involves
|
|
domains eventually ``handing off'' the polling of message channels to
|
|
their local monitor process via a (blocking) local IPC. The monitor
|
|
polls URPC channels for \emph{all} blocked domains an when it finds a
|
|
message it sends an IPC to the receiver process causing it to wake.
|
|
Since all these cache lines will be ``hot'' in the cache, this is not
|
|
as expensive as it might appear, but still does not allow for
|
|
preemption of a running thread before the next clock interrupt. It
|
|
also potentially captures kernel scheduling policy.
|
|
|
|
\subsection{Scalability}
|
|
Barrelfish does not currently multiplex URPC channels in any way, so
|
|
it is common to see $O(N)$ and even $O(N^2)$ URPC channels between
|
|
services which run on each core (e.g.~monitors), or between the
|
|
dispatchers of a domain which ``spans'' multiple cores. Though the
|
|
memory consumption is not a huge problem (a URPC channel is a handful
|
|
of cache lines), the number of channels can grow rapidly and this will
|
|
have an effect on polling costs and message latency.
|
|
|
|
Domains do not have any efficient way to identify the (probably) small
|
|
set of URPC channels which currently have pending messages. In Nemesis
|
|
this was achieved by each domain having a ``hint FIFO'' which
|
|
contained a list of channels with \emph{new} incoming messages
|
|
(identifying new messages required an explicit \emph{acknowlegement
|
|
count} in each channel). However, when a domain was activated it
|
|
could efficiently dispatch new messages, and if the FIFO overflowed
|
|
then the domain resorted to polling all channels.
|
|
|
|
In a many-core environment, a single hint FIFO would potentitally be
|
|
an expensive bottleneck due to shared-write cache lines accessed by
|
|
many cores. Having a FIFO for each potential sending core requires
|
|
$O(N)$ space, but would avoid contention.
|
|
|
|
\subsection{Primitives}
|
|
|
|
Ideally we need primitives which allow fast URPC-style messaging when
|
|
a receiver is known to be polling, but which also allow us to control
|
|
sending timely notifications to a remote kernel, domain and thread.
|
|
|
|
\chapter{Design}
|
|
|
|
In the global kernel data page have an array of pointers to per-core
|
|
notification (PCN) pages. (by the way, why is struct global still
|
|
declared in kputchar.h! :-) Each per-core notification page is divided
|
|
into cacheline-sized slots. (i.e. 1 page has 64 slots of 64-bytes)
|
|
There is one slot for each sending core. Each slot is treated as a
|
|
shared FIFO with some agreed number of entries. Each entry can
|
|
contain a short channel ID or zero. Each kernel keeps private arrays
|
|
of head pointers and tail pointers (need only 1 byte per entry x
|
|
num cpus).
|
|
|
|
On the sending side I have a new system call:
|
|
|
|
sys$\_$notify(dest$\_$core, dest$\_$chanid);
|
|
|
|
This looks up the PCN page of the destination core in the kernel globals page.
|
|
It then indexes into the PCN page using its own core id to locate the pairwise notification FIFO.
|
|
It looks up the FIFO head pointer using the dest$\_$core id.
|
|
It then looks at that entry of the FIFO... if it is non-zero then the fifo is full.
|
|
If the entry is zero then it writes the dest$\_$chanid and increments the private head pointer.
|
|
|
|
|
|
On the receiving side, the destination core keeps a private index into
|
|
each of its incoming fifos. These tells it which entry it needs to
|
|
look at next. It could therefore poll each of the FIFOs waiting for a
|
|
non-zero channelID value...
|
|
|
|
The mechanism so far results in a single FIFO cacheline toggling
|
|
between Shared and Modified state on both sender and receiver. I
|
|
timed 10000 invocations of the above sys$\_$notify() call (with the
|
|
receiver core in a tight polling loop) with a cost of 350 cycles per
|
|
notification (for a shared L3) and 450 cycles cross-package. Note
|
|
that the extra cache traffic of this design is probably not optimal,
|
|
but it's in the noise compared to our current IDC costs when we have
|
|
$>$1 domain on a core (i.e. always!)
|
|
|
|
Obviously a tight polling loop on the receiver is not ideal... we
|
|
aren't always polling, and in any event this would scale as O(N
|
|
cores).
|
|
|
|
One solution is for sys$\_$notify() to send an IPI to dest$\_$core
|
|
whenever the FIFO goes non-empty, or at the request of the receiver
|
|
(e.g. if it wrote a special 'request IPI' value into the next empty
|
|
slot, rather than zero).
|
|
|
|
Given the number of CPUs in our current ccNUMA machines, we could
|
|
easily afford to use a separate interrupt vector for each sending
|
|
core. This would identify to the destination core which FIFO to look
|
|
at, with no polling overhead. We could use a single IPI vector, but
|
|
this would need some hierarchical shared datastructure to efficiently
|
|
identify which fifos to poll. (the PCN entry for src$\_$core ==
|
|
dest$\_$core is unused and could be treated as 512 flag-bits c/f Simons
|
|
RCK code)
|
|
|
|
Sending the IPI within sys$\_$notify() would take a few hundred extra
|
|
cycles (but may overlap with the cache coherence messages?) Taking
|
|
the IRQ and acking it on the receiver is probably between 500 and 1000
|
|
cycles depending on which ring the destination core is executing in.
|
|
(I tried Richard's HLT in Ring0 with interrupts disabled trick and it
|
|
does not work any more!).
|
|
|
|
One interesting trick might be to deliver notifications to a
|
|
hyperthread so that interrupt latency and polling costs were
|
|
interleaved with normal processing...and only interrupt the
|
|
'application hyperthread' if a reschedule is necessary.
|
|
|
|
In all of the above cases, I would imagine that notifications would
|
|
not necessarily cause the running domain to be pre-empted. However a
|
|
scheduler activation for incoming IDC would be possible.
|
|
|
|
Given that true polling URPC *could* cost only a few hundred cycles if
|
|
both domains are in a tight loop, this notification mechanism is not
|
|
something I would imagine using on each message. Instead I would
|
|
suggest having a 'PUSH' flag you can pass on urpc$\_$send(), or a
|
|
b->push() method on the flounder binding so the programmer can decide
|
|
to expedite message delivery a suitable points.
|
|
|
|
\chapter{Implementation}
|
|
|
|
I just finished doing a more complete version of the UMP Notification
|
|
mechanism. It now uses a UmpNotify capability which you get by
|
|
retyping your Dispatcher cap. The monitor's UMP binding mechanism
|
|
already propagates a notification cap between client and server. I
|
|
hand edited the bench.if stubs to allocate the caps and invoke the
|
|
notification when doing a message send.
|
|
|
|
The cap$\_$invoke handler puts the receivers's DCB pointer into the
|
|
destination core's incoming notification FIFO and sends an IPI with a
|
|
vector identifying the sender core (allowing demux without polling).
|
|
The receiving core has an notification IPI handler which drains the
|
|
notification fifo and does a cswitch to the most recently notified
|
|
domain. The domain will get an activation and poll its URPC channels
|
|
(eventually).
|
|
|
|
After thinking a bit more about the costs of notification, it seems
|
|
that 2800 cycles is quite a lot to pay in the default path to send a
|
|
notification. Quite a bit of this is the high cost of cap$\_$invoke on
|
|
x86$\_$64 (2K cycles) compared to the hacky syscall I was using last week
|
|
(~800), but in general we don't want to pay much for notification
|
|
unless it's necessary.
|
|
|
|
I added code in the domain dispatch path to publish the identity of
|
|
the currently running DCB, and ideally the time at which it will next
|
|
be preempted. The notification kernel code on the sender side can
|
|
therefore tell if it's worth sending an IPI and return immediately if
|
|
the domain is already running. This leads to the behaviour below
|
|
where, just before T=70000 core 1 sends a message to core 2, notices
|
|
it isn't the currently running domain and so sends a notification IPI.
|
|
The monitor on core2 is preempted and the receiver domain gets to run.
|
|
Activation code takes about 2000 cycles but the message gets there
|
|
pretty quickly.
|
|
|
|
This allows me to increase MAX$\_$POLLS$\_$PER$\_$TIMESLICE to 1000
|
|
without excessive penalty, which in turn allows the client and server
|
|
to remain in the polling loop and notice messages before they yield to
|
|
other domains. Net result is that the common case RPC cost is about
|
|
4x faster. The worst case is hopefully bounded by the cost of
|
|
(cap$\_$invoke + notifiy IPI + domain activation + ump$\_$poll)... about
|
|
4000 cycles. I ought to check this by running some ``while(1)''
|
|
domains on each core... but I'm fairly (naively?) optimistic.
|
|
|
|
|
|
\section{Performance}
|
|
|
|
\chapter{Testing and Debugging}
|
|
|
|
\end{document}
|