%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Copyright (c) 2011, ETH Zurich. % All rights reserved. % % This file is distributed under the terms in the attached LICENSE file. % If you do not find this file, copies can be found by writing to: % ETH Zurich D-INFK, Universitaetstrasse 6, CH-8092 Zurich. Attn: Systems Group. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \documentclass[a4paper,twoside]{report} % for a report (default) \usepackage{bftn} % You need this \usepackage{hyperref} % You need this \title{Bulk Transfer} % title of report \author{Pravin Shinde} % author \tnnumber{014} % give the number of the tech report \tnkey{Bulk Transport: Cross-domain transfer facility for large data} % Short title, will appear in footer % \date{Month Year} % Not needed - will be taken from version history \begin{document} \maketitle % % Include version history first % \begin{versionhistory} \vhEntry{1.0}{11.08.2011}{TR}{Initial version} \end{versionhistory} % \intro{Abstract} % Insert abstract here % \intro{Acknowledgements} % Uncomment (if needed) for acknowledgements \tableofcontents % Uncomment (if needed) for final draft % \listoffigures % Uncomment (if needed) for final draft % \listoftables % Uncomment (if needed) for final draft \chapter{Introduction} \label{chap:intro} This technical note is aimed to describe the design of suitable cross domain bulk transport for Barrelfish. This chapter gives a motivation for this design by providing the details about existing implementations of bulk-transport in Barrelfish and reasoning about why those are not good enough. The \autoref{chap:design} contains the actual details about the design. So, if you are already convinced of the need for better bulk-transport in Barrelfish, then you can directly jump to the \autoref{chap:design} \section{Existing implementation in Barrelfish} There are two bulk transport mechanisms that exist in the current(as of August 2011) implementation of Barrelfish. \texttt{$lib/barrelfish/bulk\_transfer.c$} and \textit{pbufs} used in the network stack. We will discuss them briefly in following two sub-sections. \subsection{$bulk\_transfer.c/h$} This is the official bulk transfer method supported by Barrelfish. In brief, this facility works by sharing a continuous piece of memory (in form of capability) between two domains. These domains then map this physical memory into their virtual address-space. The virtual address where this shared memory is loaded can be different. So, participating processes can't exchange the virtual addresses directly. Bulk transfer mechanism works by dividing the entire memory area into the blocks of fixed size. The domain which initiated the bulk transfer is responsible for managing these blocks. This responsibility includes following. \begin{enumerate} \item Allocation and deallocation of the blocks. \item Maintaining the information about free and used blocks. \end{enumerate} So, this bulk transfer facility works in master-slave setup. The master allocates the blocks and passes the index of the block to slave. Once the block is passed to the slave, master should not touch it. The slave can locate the block by adding the \texttt{(index*block-size)} to the virtual address pointing to the beginning of the shared memory area. Now slave can read/modify the contents in the memory area of this block. Once the slave is done with accessing the block, it can pass the block-index back to the master. Master can either access the block or release it for future use. \subsubsection{Limitations} \paragraph{Security} This transfer mechanism can only be used between domains which are co-operative and willing to follow the protocol set for using the bulk transfer mechanism. Malicious domain can corrupt the data in shared memory by writing at random locations in it or can refuse to return the blocks once passed to it. \paragraph{More than two domains} The design of this bulk transport does not stop you from using it with more than two domains as long as the protocol is followed. But current implementation does not track which domain is holding the buffer. So, one can use the current implementation with multiple domains as long as the domains are co-operative and applications are willing to deal with added complexity of tracking which domains hold which buffers. \paragraph{Current usage} Due to the relatively simple nature of the implementation, its use is limited only to few places. And hence it is not throughly tested in various scenarios. \subsection{pbufs} The network stack uses its own custom bulk transport mechanisms. Lets call them pbufs. These pbufs work in similar way to above mechanism, but are more flexible. The application shares some piece of physical memory with network driver which is used as shared memory. Then application creates a list of pbuf structures, each one of them holding an offset within shared memory, length, \textit{pbuf-id} and the shared memory id to which they belong. The key difference here is that the these pbufs may not hold consecutive memory locations even when \textit{pbuf-id}'s are consecutive. In contrast with Barrelfish bulk-transfer where buffer-id value is enough to find the location of buffer within shared memory, pbuf needs to store the offset separately in another list. So, pbufs provide another layer of indirection to allow more flexible use of memory. The way this mechanism currently works is that, application creates pool of initial pbufs and registers them with network driver. Both, application and network driver maintain the list of pbufs in their private memory. This list is kept in synchronization by sending explicit messages. Now each pbuf can in principle point to any buffer of any size, located anywhere in the shared memory. This flexibility is used by application when it receives a data from the network driver. Application creates a new pbuf structure with same pbuf-id but pointing at new buffer location and send it back to the driver as new free pbuf to use. And the location of previous buffer is used for processing the data. This way, application can return the pbufs back to driver ASAP without getting affected by how long does the data processing takes. When application is done with processing the data in that buffer, it releases that buffer. This released buffer is then used to create new pbuf which will be registered again with driver in future. \subsubsection{Limitations} \paragraph{Needs more memory} The shared memory needs to be bigger than the memory shared with network driver in form of registered pbufs. This is because, at any given point in time, some pbufs will be in application data processing phase and hence can-not be used by driver to receive the new data. \paragraph{Data corruption} This bulk-transfer mechanism assumes co-operative domains. Both domains have read/write access to the shared memory at all the time. This means that if they do not follow the protocol correctly, they may end up writing at same physical location, leading to data corruption. \paragraph{Complicated memory reclamation} The current implementation of pbufs assumes that application can have multiple threads and all of them can access the data in pbufs which are delivered to the application. This complicates the problem of detecting when exactly all threads are done with accessing the particular pbuf. The current implementation uses a reference count mechanism for this detection. Even though this memory reclamation is functional, it is complicated and one can easily get it wrong, leading to memory leaks. \paragraph{Supporting more than two domains} In theory, this design can be used with more than two domains which are co-operating with each other, but implementation is not designed with such a case in mind. The problematic issues will be tracking which memory buffer is with which domain, and when the memory buffer can be reused. \subsubsection{More issues with current network stack implementation} Following issues are not exactly due to the bulk transfer mechanisms, but are mainly the implementation issues of network stack. I am documenting them here for sake of completeness. Also, knowing the issues in current stack will help in understanding the design decisions made. \paragraph{Favors the setup with driver on different core} It is optimized for the case where application and network driver are running on separate core. It is reasonable to prefer such cases for multicore architectures where presence of large number of cores is assumed. This preference has resulted on overly dependence on efficiency of UMP messages. Current implementation sends around 2 messages (send-packet, tx-done) for transmitting single packet assuming that packet fits into one pbuf. Similarly on packet receiving side, each received packet involves 2 messages (register-pbuf, packet-received). The performance impact of these messages is not strongly visible when application and driver are on different core. But there are cases like SCC where you may end up running the driver on same core as an application. In such scenarios LMP's are used for communication which involves doing context-switch in sending every message. This leads to lot of performance in-efficiencies when current network stack implementation is used in LMP contexts. Specially on those hardware where cost of context-switch is relatively high. %I have tried to measure the performance difference on \texttt{nos4-6} %boxes with webserver cache loading test, but the performance %difference was not significant. I have not investigated the reasons %behind them yet, but my preliminary guess was that context switches on %these machines are quite efficient or webserver cache loading test %does not stress the network stack enough to see any significant %difference. %\section{Questions to be asked about bulk transfer mechanisms} %\begin{enumerate} % \item Does it uses memory efficiently? % \item Can it be used with mutually non-trusting domains? % \item Can it be used across more than two domains? % \item Can consumer/producer put a back-pressure on other side? % \item Can it support multiple producer and multiple consumers? % \item Can it allow implementation of \textit{True Zero Copy} % sending and receiving? % \item Can it lead to deadlock if one domain is only sending without % checking for arriving messages? %\end{enumerate} \section{Related work} The issues of bulk-transport has been investigated quite a few times in the past by work like \textit{F-bufs}, \textit{R-bufs} and \textit{Beltway buffers} so the solution of this problem is known to the large extent. What this document is aiming is to choose right approach and right solution which will suit the requirements of the Barrelfish. %The work which has particularly influenced this design are %\begin{enumerate} % \item F-bufs: Helped in clearly understanding the problems in % designing cross-domain transfer facility. % \item R-bufs: Helped in understanding how to implement % scatter-gather packet sending and receiving. % \item Beltway buffers: Gave clear idea of how to design a % system with multiple ring-buffer to get lot of flexibility. % The problem with beltway buffers is that they are % designed to be POSIX compatible and due to that, the whole % functionality of these buffers is pushed into trusted OS code-base. % The applications are unaware of these buffers. Also, these buffers % can be used mostly to short-circuit the data path between different % trusted kernel sub-systems. If applications are going to modify the % data then beltway buffers don't help much. %\end{enumerate} \section{Requirements of Barrelfish} Following are the ideal requirements from the bulk-transport mechanism. \begin{enumerate} \item Avoid data copy as much as possible, if can't avoid, then try to push it into user-space/user-core. \item Ability to batch the notifications. \item Should work with more than two domains. \item Should work with multiple producers and multiple consumers. \item True zero copy capability(scatter-gather packet sending/receiving). \end{enumerate} The solution that we are trying to design tried to satisfy as many of these requirements as possible. \chapter{Design} \label{chap:design} This chapter discusses the design of bulk-transport mechanism that will be implemented in future for Barrelfish network stack. \section{Bird's eye view of design} We are aiming to design a cross domain bulk transport mechanism, which supports following features. \begin{enumerate} \item Should reduce the data copy as much as possible. \item Should exploit the fact that complete data-isolation is not always needed. \item More than two separate domains should be able to share the data without copying it. \item Number of explicit notifications needed should be low. \item It should work in \textit{single producer, single consumer} and \textit{single producer, multiple consumers} \end{enumerate} \section{Terminology used} This section briefly explains the term used in the design description. This should help in the understanding by reducing the ambiguity of the terms. \begin{enumerate} \item \textbf{Generator:} An entity which is generating the data. It can be software or hardware entity. For example, Network Interface Card(NIC) is a generator, or the application code which is generating the packets to send over the NIC. \item \textbf{data-element:} The data-element is what generator is generating. In case of NIC device, a packet will be a data-element. \item \textbf{Producer:} An entity which is managing the generated data. This management contains providing access mitigation, notifying interested consumers, reclaiming the memory when data-element is consumed. The example of producer will be \textit{NIC device driver}. The distinction between generator and producer is rather fine. They are intentionally kept separate because generators can have hardware constraints which does not allow it to provide all functionalities. \item \textbf{Consumer:} An entity which is consuming the data-elements. \item \textbf{Slot:} The contiguous piece of memory where data-element is entirely or partially stored. A data-element can span over more than one slots. \item \textbf{Shared-pool:} A contiguous piece of memory accessible in read/write mode to producer and accessible in read only mode to consumers. Shared pools are divided into above slots. \item \textbf{Production-pool:} It is a collection of all shared-pools used by the particular producer. The initial shared-pool is added by the producer, and later on every consumer which joins the system contributes a new shared-pool to the production-pool. \item \textbf{Free slot:} A slot which is available, and can be given to generator. \item \textbf{In-generation slot:} A slot is given to generator. Only the generator should read/write this slot now. This is the only mode when data can be written in the slot. \item \textbf{In-consumption slot:} One or more consumers are currently consuming this slot. In this mode, slot is strictly read-only no one should modify the data in slot in this mode. \item \textbf{Slot-pointer:} A data-structure which holds enough information to locate particular slot present in any shared-pool with producer. These are typically used in cross domain communication to identify the exact slot. A typical slot-pointer should have a shared-pool-id and offset within shared-pool to find the particular slot. \item \textbf{Classification:} This term refers to the classification of data-elements between consumers. In other words, it is a process of deciding which consumers should receive this particular data-element. In network stack parlance, this classification will map to deciding which process should get the received packet. \end{enumerate} \section{The Producer} This section describes the producer and generator together with their responsibilities. Even though producer and generator can be a different entities internally, the consumers will only see the producer interface. \subsection{Generator} Lets make a rough sketch of typical generator. Here we are aiming for the NIC devices. Typically, these devices will have an internal queue (RX-queue) of slots where the packet received from wire will be copied. These generators are also capable of generating a notification in form of interrupt. Now the way RX-queues work on different devices differ in some way. Few devices expect contiguous memory in the slots whereas other devices are capable of DMAing the packet in non-contiguous memory as well. The software generators like application logic which creates new packet to send out is much simpler than NIC devices but can have quirky behavior like above described NIC devices. From bulk-transport point-of-view we will be treating the generator as a part of producer. \subsection{The Producer abstraction} The producer abstraction is nothing but the combination of generator and remaining producer functionalities. The producer abstraction is what consumer sees and interacts with. It is responsible for managing the data produced by generator and using shared-pools to get that data till consumers. It is also responsible for managing the shared-pool memory and co-ordinating the access to this shared resource. \subsection{Shared-pool} Shared-pool is the area where producer will generate the data and consumers will read it from. The producer breaks this shared-pool into \textit{slots}. The size of slot is based on the capability of generator and should be multiple of cache-line size. The details of how to choose the slot size and the side-effects of small or large size are discussed in \autoref{sec:slotsize}. The producer is responsible for managing these shared-pools, and it does by maintaining the list of shared-pools. At the time of producer initialization there will be only one shared-pool which is added by the producer itself. As new consumers join the system, each one of them will contribute an additional shared-pool. \subsubsection{Releasing the shared-pool} If any consumer decides to leave the system, then producer stops using the shared-pool provided by him. The producer will wait till all the slots in that shared-pool are free and then it will ask all consumers who have mapped this shared-pool to release the pool. Once all consumers release the pool then producer can also release the pool and inform the consumer who contributed the shared-pool about completion of the process. This is rather long process and depends on all the consumers for it's completion. But this process is not part of the critical path, it is only part of the tear-down process so some inefficiency can be tolerated here. %The virtual address where the shared-pool is mapped need not be same for %the producer and consumers. \subsubsection{Ideal size of production-pool} The production-pool is just a collection of all valid shared-pools available at given time. Ideally, production-pool should be big enough to accommodate the queue of the generator (ie. All the descriptors in RX-queue of NIC device) and all the data-elements which are with consumers and are not released back (in-consumption state). So, the total memory size should be proportional to the capacity of the generator(ie. NIC device) and number of consumers. So, one way to look at this is that, producer will provide shared-pool which can satisfy the needs of generator and each consumer will contribute a shared-pool to allow in-consumption data-elements. Each consumer is allowed to keep only fixed amount of in-consumption data-elements based on the size of shared-pool it contributed. This way, fixing the size of allowed pending in-consumption data-elements limits the ability to consumers to over-consume the slots from production-pool. \subsection{A meta-slot structure} This structure hold the additional information about each slot in a shared-pool. It can be seen as index table on shared-pool. This structure is private to producer and is used to manage the slots within shared-pool. Following are the key elements of this structure. \begin{enumerate} \item \textbf{slot-id:} The slot identifier. \item \textbf{offset:} The location of slot within the shared pool. \item \textbf{data-len:} Size of valid data in the slot. \item \textbf{state:} The state of slot (Free, in-Generation, in-Consumption) \item \textbf{consumer-list:} Which consumers are accessing it? \item \textbf{Next, prev:} Used to maintain the list (eg. Free slots) \end{enumerate} The producer maintains a separate meta-slot list for very shared-pool available with the producer. This setup enables producer to track every slot and its state. Other than this, the producer also maintains the list of consumers who have registered. \subsection{shared and private data-structure} This section briefly describes which of above data-structures are private and which of them are shared with other entities. \subsubsection{Private data-structures} The producer maintains lot of state in private data-structures. It includes following. \begin{enumerate} \item List of consumers connected. This list also maintains the state of each consumer with respect to the communication with producer. \item List of shared-pool-meta-data. This meta-data includes following information. \begin{enumerate} \item Consumers having access to this shared-pool. \item meta-slot structure containing private information about state of each slot in this shared-pool. \item List of free slots in this shared-pool. \item The generator-queue which is used by generator to create data-elements is shared with generator. But as we treat generator as internal entity, we mark this as private data instead of shared data. \end{enumerate} \end{enumerate} \subsubsection{Shared data-structures} The producer shares following data-structures with different entities. \begin{enumerate} \item Shared-pools can be shared with any consumer. \item For every consumer, a consumer-queue (see \ref{sec:consumer-queue}) is pairwise shared with that consumer. \end{enumerate} \section{Consumer} This section describes the internals of the consumer. Consumer mainly consist of a consumer-queue data-structure which allows sharing of slots between producer and consumer. Consumer also has read-only access to the shared-pools. \subsection{Consumer-queue} \label{sec:consumer-queue} This section describes the consumer-queue in details. The consumer-queue is a shared data-structure between consumer and producer and it is exclusive between each pair of consumer-producer. This data-structure is also maintained on contiguous shared memory which is read/writable by both consumer and producer. This data-structure is shared ring-buffer with few virtual-registers which are used to manage the access the ring-buffer. \begin{enumerate} \item \textbf{write-register:} This virtual register is the first element in the consumer-queue. This element should be of size cache-line to avoid any cache-conflicts. This element can be modified only by the producer and consumer can only read it. The value in this virtual-register contains the write-index for consumer-queue. \textit{The write-index points to the slot-pointer within consumer-queue that producer will provide next.} \item \textbf{read-register:} This virtual register is the second element in the consumer-queue. This element should be of size cache-line to avoid any cache-conflicts. This element can be modified only by the consumer and producer can only read it. The value in this virtual-register contains the read-index for consumer-queue. \textit{The read-index points to the slot-pointer within consumer-queue that consumer will consume next.} \item \textbf{queue-size-register:} This virtual register is the third element in the consumer-queue. This element can be modified by producer and consumer can only read it. The value in this virtual-register indicate the number of slot-pointers which are valid in the consumer-queue. This register allows dynamic adjustment queue-size based on the current load. Similar feature is implemented in the \textit{beltway buffers} with the claim that keeping the size of queue as small as possible helps in cache friendliness. This feature is actually optional and will be added in later stages. To implement this feature, one also need to made decision like when the queue size should be reduced and when it should be increased again. Producer can make these decisions based on how much preference it wants to give to this particular consumer over others. \item \textbf{Slot-pointers:} After above three registers, the rest of the space in consumer-queue is used for storing slot-pointers. The slot pointers are used to point a particular slot in one of the shared-pools within production-pool of the producer. Slot-pointers have following information. \begin{enumerate} \item \texttt{shared-pool-id:} Id of the shared-pool which is holding this particular slot. These id's are given by the producer and will be unique within that particular producer for given shared-pool. As the shared-pools within particular producer may increase/decrease over time, the consumer may receive a slot-pointer with shared-pool-id which it has not mapped yet. In such a case, it should send a message to producer asking for read-only access to this new shared-pool and then map it into the virtual address-space. Once the shared memory frame associated with shared-pool memory is mapped, consumer can continue to access the slot using slot-pointer. \item \texttt{slot-index:} The index of slot within that shared-buffer. This value is only useful for producer as this slot-index maps into the meta-slot structure which is private to the producer. Consumer should not alter this value. In case malicious consumer alters this value, the producer will be able to detect it as meta-slot structure maintains the information about which all consumers are currently consuming the slot and whenever the producer reclaims the slots freed by consumer, it validates if this consumer was consuming the slot being released. The reason for maintaining this information in slot-index even when it is not useful to consumer is because it speeds up the producer in relocating the slot within shared-pool when it is freed by the consumer. \item \texttt{Offset:} The start of the slot within shared-buffer. In case of fixed size slot, offset can be calculated by slot-index. \item \texttt{More:} As the data-element can span more than one slots, this flag tells us if there are more slot-pointers following which belongs to same data-element. This flag is equivalent to the \textbf{More Fragments} bit in the \textbf{IP protocol}. In contrast to IP protocol there is no fragment identification number, but the slot-pointers belonging to same data-elements are assumed to follow each other. So, the order in which the slot-pointers are added to the consumer-queue is important. As we have only one producer which is adding the slot-pointers in \textit{available to consume} section and also this producer is dealing with one data-element at one time, maintaining this order is fairly simple. \end{enumerate} \end{enumerate} \subsubsection{Conditions on consumer-queue registers} \begin{enumerate} \item Queue empty condition: read-index == write-index \item Queue full condition: ((write-index + 1) \% size) == read-index \item Elements available to consume \newline (assuming queue not empty):\newline if (write-index $>$ read-index) \newline then \{[read-index, (write-index - 1)]\} \newline else \{[read-index, (size - 1)], [0, (write-index - 1)]\} \newline \item Elements which are already consumed and are now free. \newline (assuming queue not empty): \newline if (write-index $>$ read-index) \newline then \{[write-index, (size - 1)], [0, (read-index - 1)]\} \newline else \{[write-index, (read-index - 1)]\} \newline \end{enumerate} \subsection{shared and private data-structure} This section briefly describes which of above data-structures are private and which of them are shared with other entities. \subsubsection{Private data-structures} The consumer does not have to maintain lot of private state based on which model of slot-consumption is used (refer \ref{sec:memory-management}). When doing out-of-order consumption it will have to maintain some state about which slots are in consumption and which slots are free and ready to go back to producer. Every consumer will also need some private state to remember the state of the producer and the consumer-queue status. If this consumer is directly communicating with other consumers then it will also need to maintain some private state about that communication (refer \ref{slot-forward}). \subsubsection{Shared data-structures} The consumer shares following data-structures with producer and other consumers. \begin{enumerate} \item The shared-pool that it has contributed to producer and potentially shared with all consumers. \item The consumer-queue is exclusively shared with producer. \end{enumerate} \section{Events/notification/communication between consumer and producer} This section describes the communication between the consumer and the producer. \subsection{From Producer to Consumer} This paragraph describes the events sent by producer to consumers. \begin{enumerate} \item \textbf{More data arrived:} This notification is sent to consumer whenever the consumer-queue is empty and new data-element arrived. This callback is not triggered for every arrival of data, but only when consumer is not explicitly polling for data. Whenever new data arrives on the empty queue, producer can assume that consumer is not polling the channel as queue is empty, and hence producer should send this notification to consumer to wake it up. In other case where queue is non-empty, consumer is already aware of the presence of data-elements there. Consumers are expected to deal with dynamically growing of queue-size and hence adding more elements to non-empty queue without sending explicit notifications should not break the consumer-logic. \item \textbf{Consumer-queue full:} This notification is sent to consumer whenever consumer-queue is full and producer is not able to add new data-elements to the consumer-queue. Triggering of this notification means that consumer is slow in consuming the data from the consumer-queue. And result of this producer is going to drop the data-elements which were aimed for this consumer till there is more free space with this consumer. This message also means that producer is not going to actively check if consumer-queue has a free space or not. It is a responsibility of consumer to send a notification to the producer whenever it is ready to receive more data. \item \textbf{Consumer-queue almost full:} This is an optional notification and is designed as refinement of the basic approach. This additional event can be called when the number of free slot-pointers in the queue drops bellow certain threshold. This notification can be used as a warning sign to the consumer that either it takes some action to free up more slot-pointers in consumer-queue, or producer will soon start dropping the data-elements addressed to this consumer. \item \textbf{Error event:} This notification is sent when something unexpectedly goes wrong. The information and severity of the error will inform the consumer that if the problem is transient or permanent. And based on the problem type, consumer can take corrective actions. Few samples of these errors are corruption of consumer-queue, fetal error in producer, etc. \item \textbf{Add shared-pool:} This is an optional notification which allow producer to push the notifications about newly added shared-pools. This message is optional because consumers can lazily ask for these new shared-pools whenever they encounter them as part of slot-pointer while consuming data-elements. This notification is not considered as part of critical path or part of the data-flow. This notification will be generated only on the arrival of new consumer or if producer decides to increase the available shared-pools. This can be classified as setup or maintenance path. \item \textbf{Remove shared-pool:} This notification is opposite of above \textit{add shared-pool} call. It tells consumer is that producer will not be using a particular shared-pool from now onwards for whatever reason (eg. the consumer which gave that shared-pool frame has terminated the connection with producer). So, every consumer which receives this message should remove the mapping for shared-pool from it's virtual memory. This notification is also relatively rare and will be generated only when one of the consumer decides to quit or if producer decides to reduce the number of available shared-pool. So, this notification should not be considered as part of critical-path or data-flow. It can be classified into setup or maintenance path. \end{enumerate} \subsection{From Consumer to Producer} This paragraph describes the notifications sent by consumer to the producer. \begin{enumerate} \item \textbf{Consumer-queue space available:} This notification is sent to the producer when consumer-queue is full and new free slot-pointer is added creating new space. Once the queue is full, producer is not expected to check the queue status until it receives this message saying that now application has more space to receive packet. As an optimization, consumer might wait till some more free-space is accumulated before informing the producer to re-start the flow. This way the inefficiency similar to \textit{silly window behavior} in TCP flow control can be avoided. \item \textbf{Get frame for shared-pool-id:} This notification is sent to the producer when consumer receives a slot-pointer with shared-pool-id which is not mapped by this consumer yet. By sending this notification, consumer is asking for the read-only access to this shared-pool. The producer should respond to this message with either with valid capability or with error. \item \textbf{Forwarding slot-pointer to other consumer:} \label{slot-forward} This message is related to the ability of consumers to forward slot-pointers within each other. There are two cases here. In first case, the consumer forwards the slot-pointer to other consumer and then also collects it back before declaring it as free. In this case, producer does not need to be informed at-all about this sharing as consumer is taking responsibility of properly freeing the slot. But in case where consumer just wants to forward the slot-pointer and continue on it's own working without bothering about when and how other consumer is going to stop accessing that slot, then it should inform the producer that it has forwarded that slot to some other consumer. It can be done with this message. The reason we need this separate message is that, because of sharing there will be two consumers which will be releasing the slots and producer should be aware of them. The information from this message is used to update the meta-slot structure inside producer for given slot with one more consumer. This message contains the slot identification information and consumer identification information. The other consumer can now release the slot whenever it is done using it's consumer-queue, and producer can validate and account that operation correctly. This forwarding has some more implications which are not fully explored in this document. Here are few leads on that. Firstly, this path allows consumer to receive the slots without being initiated by producer and this slot will not be a part of consumer-queue. So, the producer need to somehow account for these slots so that it can maintain the fairness in the number of outstanding slots per consumer. This mechanism also needs some way by which two consumers can talk with each other and exchange the information like consumer-id's, producer-id and slot-pointers. Also, consumers should be careful in dealing this information as consumer-id and slot-pointers are only valid in context of particular producer. \end{enumerate} \subsection{The communication between producer and generator} The communication between producer and generator is very hardware dependent, and I am not sure if it should be part of this document. Following are the generic notifications which are exchanged between the producer and generator. These notifications are aimed to maintain the generator-queue. \begin{enumerate} \item Add empty slot (from producer to generator): This message will add a new free slot into generator-queue. \item Data-element generated (from generator to producer): This message informs the producer that there is new data-element in the generator queue that should be handled. It is equivalent of \textit{packet received interrupt} from NIC hardware to the device driver. \item Generator-queue full (from generator to producer): This message tells the producer that there is no space left for generator to create more data-elements. \end{enumerate} \subsection{The communication between consumers} TBD: The communication between consumers is needed to share the slot-pointers between them. \section{Initialization} \label{sec:initialization} This section briefly provides the steps involved in initialization of producer and consumers. \subsection{Producer initialization} At producer initialization, following steps will happen. \begin{enumerate} \item Create and initialize the empty list of shared-pools and meta-slots. \item Create and add the initial shared-pool big enough to satisfy generator needs. \item Divide the shared-pool into slots, create a meta-slots list for this shared-pool and add it to meta-slots list. \item Create an empty list of consumers. \item Initialize the generator (if needed). This might involve populating the generator-queue with free slots so that generator can start producing data. \item Loop: wait for next event/message and process it accordingly. \end{enumerate} \subsection{Consumer registration} As part of initialization process, consumer should register itself with the producer. This initialization process includes following steps. \begin{enumerate} \item Create a new contiguous memory frame and provide it to producer as new shared-pool frame. \item Get the list of registered shared-pools from producer and map it as read-only in the virtual address-space of consumer. \item Create a new contiguous memory block to maintain a consumer-queue. \item Initialize this memory block with consumer-queue data-structure. \item Share this memory block with producer. \item Send a notification \textit{consumer-queue space available} to the producer so that it will start sending data-elements. \item Loop: wait for next event/message and process it accordingly. \end{enumerate} \section{Memory management (slot management)} \label{sec:memory-management} The slots are most important aspect of this bulk transport mechanism. They hold the data which will be sent across the domains. Now lets see how exactly these slots work. \subsection{Slot state machine} Slots are created by the producer in the memory provided by shared-pools. Once created, slot remains valid until and unless the shared-pool holding the slot needs to be freed. In the lifetime of the slot, it goes through following states. \begin{enumerate} \item \textbf{Free:} This is the initial state, in this state slot is free and ready to be used. The producer maintains a list of free slots so that it can quickly find them whenever they are needed. \item \textbf{In-generation:} This is the second stage of slot. In this state, the slot is owned by the generator and no one should access it(not even in read state!). This is the only state in which the data will be written into the slot. \item \textbf{In-classification:} This is the third and transient state. This is the state when generator returns the slot after it has generated the data. The producer will use the classification to figure out which consumers this data-element should go. The producer is free to use any type of classification that is needed. The description of how that classification should happen is not in the scope of this document. Once classified, the producer will update the consumer-queues of each selected consumer with slot-pointer and if needed, send them the notifications. Once the slot-pointers are added into consumer-queue, the state of slot is updated to the \textbf{in-consumption}. If no consumer is interested, then the slot is marked as free and added back to the list of free slots. This is the stage where \textbf{zero copy data access} is achieved. As instead of actually copying the data from the slot, slot-pointers are provided to the consumers. These slot-pointers provide zero-copy access and data-sharing capabilities. \item \textbf{In-consumption:} This is the state in which data is available to the consumers. Here, consumer has multiple options about what to do with this slot. \begin{itemize} \item \textbf{In-order consumption:} Consume the slot as quickly as possible, and once consumed add it to freed slot by moving the read-index. As consumer is dealing with one data-element at a time, there is no need for complex state management and free-slot management inside consumer-queue is sufficient enough without needing any additional state machinery. \item \textbf{Out-of-order consumption:} Saves the newly arrived slot-pointer and replace it with one of the other slot-pointers which has been previously consumed and then add this replaced slot to freed slot by moving the read index. This method allows you to consume the slots at leisure and you can have more than one data-elements in consumption at any given moment of time. It also allows you to free up the slots which don't hold useful data quickly and release the slots with important data later on. This method gives much more flexibility but at cost of added free-slot management within consumer. As consumer is releasing the slot in out-of-order of their consumption, it has to maintain some state about which slots are released and which are still in consumption. \item \textbf{Private-mutable consumption:} If the application needs a private access to the slots where it can modify it then it should make copy of this slot in it's private memory and release the actual slot. This way, application is free to do anything it wants with the private slot. This solution scales well with large number of private-mutable consumers because the data-copy operation is performed by consumers and not by producer. As data-copy operation is typically most expensive, we want to push it out from producer into consumers. If consumers are running on separate cores then the data-copy operation of one consumer will not affect other consumers. \end{itemize} Once the consumer is done with consumption, the slot will be returned to the producer. And when all consumers who are sharing the slot report that they are done with consumption then producer can mark the slot as free and add it to the list of free slots for future reuse. \end{enumerate} \subsection{Slot management inside producer} The producer maintains a private meta-slot data-structure for each slot to track these slots. It also maintains the list of free, in-generation and in consumption slots. \subsection{Slot management inside consumer} The slot management is not needed in consumer if they are following in-order or private-mutable consumption. In these two cases, the consumer-queue implicitly does the slot management on behalf of consumers. In case of out-of-order consumption, the consumer will have to maintain the list of all slot-pointers which are currently accessible and also the list of slot-pointers which are done with the use and can be released as free slots. \subsection{Freeing up the slots} Whenever consumer is done with accessing/consuming the data-element, it should be added back to the free-slot-pointers area of the consumer queue. It is important to note that the order in which slots will be freed by consumer need not match the order in which it consumed the slots. This flexibility allows consumer to take longer time on certain slots while quickly returning the slots which have arrived afterwards. Consumer can hold back certain number of slots without releasing them, but this number depends on the length of the consumer queue. As the producer controls the consumer-queue length it can give some consumers more slots than other based on the configurable policies. The policy that initial implementation of this buffer will be using is that, the size of consumer-queue will be directly proportional to the size of shared-pool that particular consumer has contributed. \subsection{Slot size} \label{sec:slotsize} The size of the slot is greatly dictated by the capabilities of the generator. If generator needs continuous memory to produce data-element then the size of slot will be the size of biggest data-element. If the generator can DMA the data-element into multiple non-contiguous memory locations then, the slot size should be based on the average data-element size. In any case, the slot size must alway be a multiple of cache size and every slot should be cache aligned. This is mostly due to the fact that each data element might get consumed by different consumer running on different cores, and we do not want cache conflicts when they are accessing different data-elements. The dis-advantage of having slots of size of largest data-element is that it leads to internal fragmentation and waste of memory when packets are small. On other hand, when slot size is small, we reduce the capacity of NIC hardware as typically RX-queue will have limited number of entries, and by using small sized slots, we will fill up the entries quickly. \section{Security and trust model} \label{sec:security-trust} This section discusses the trust model assumed by this bulk transport design and it's security implications. \subsection{Slot access security} \label{sec:slot-security} This bulk transport mechanism works by sharing the memory and it needs security against invalid memory accesses. This security is provided by the slot-pointers. Slot-pointers are designed to be valid across the virtual address-spaces of all consumers. Slot-pointers contain information like shared-pool-id which can be used to find the starting address and length of the pool. When consumer is calculating actual pointer, it can always validate that the pointer lies within shared-pool by making sure that offset is always smaller than shared-pool size. \subsection{Security against memory invalidation} \label{sec:memory-security} The more problematic case is when one of the consumer dies or invalidate the memory given to producer without informing first. This case can lead to invalid memory access in producer, generator and other consumers as one of the shared-pool is not accessible any longer. The protection against such a case should be provided by operating system by not removing the memory frame as long as it is mapped by at-least one domain/process. If operating system does not provide such an assurance and if the consumers are non-trustworthy then producer should somehow take the complete ownership of the memory provided by the consumer, or provide all the memory itself. From above discussion, it can be seen that the trust mainly lies with producer and not the consumer. Consumers can't do much damage to producer as all the actions of consumers can be validated by the producer. For example, when any consumer tries to free up the slot, the producer can verify if the consumer really holds that slot or not. \subsection{Security against data sniffing} \label{sec:security-sniffing} The only damage that consumer can do is to try and read the data-elements which are not destined to it. Consumers can do this as they have read-only access to entire production-pool. Performing this type of snooping is not trivial as all the meta-data about the slots is maintained in the private memory by the producer, so consumer can only guess about location and type of data inside the other visible slots. But nevertheless unauthorized data-sniffing between consumers is theoretically possible in this design. This shortcoming can be overcome by sharing the shared-pool pairwise between consumer and producer and then by either adding the overhead of performing data-copy, or by using generator capability to perform early classification of data elements. The \ref{sec:privacy} will give more details about these stricter privacy models. \section{Privacy model} \label{sec:privacy} This bulk transport is designed to work in relaxed privacy model with possibility of tightening the privacy model by either using early classification capabilities of generator or at added cost of data-copy. This section describes all of these modes. \subsection{Relaxed privacy model} \label{sec:relaxed-privacy} The bulk-transport design described most of the above document is based the relaxed privacy model. The reason for describing this model in detail is that other models which gives better privacy (described in \ref{sec:data-copy-privacy}, \ref{sec:generator-classification-privacy}) are sub-set of this model hence they are easy to understand once this model is clear. In this model, all consumers have read-only access to entire production-pool. And hence in theory malicious consumers can sniff the data-elements addressed to other consumers. The \ref{sec:security-sniffing} gives some description of this issue from the security perspective. \subsection{Strict privacy model with additional data-copy} \label{sec:data-copy-privacy} In this model, shared-pools are not shared with all consumers, but every shared-pool is pairwise shared between producer and with only one consumer (one who has contributed that shared-pool). The initial shared-pool contributed by the producer itself is shared between producer and generator. This shared-pool will not be directly accessible to any consumer. All the data generated in this pool which is private to producer and generator and then based on the classification of the data-element, it is explicitly copied into the shared-pool of the selected consumers. This way, other consumers can never see the data-elements which are not destined to them. This stricter privacy policy comes with the cost of additional data-copy. Also, as producer is performing this data-copy, this can adversely affect the throughput of the whole system. \subsection{Strict privacy model with early classification support from generator} \label{sec:generator-classification-privacy} In this model as well, shared-pools are not shared with all consumers, but every shared-pool is shared between producer and with only one consumer. This model depends on the capabilities of generator to provide stricter privacy model. If the generator is smart enough and can classify the data-elements before they are created, then these data-elements can be directly created into the shared-pool of selected consumer. This avoids additional copy from the producers pool to the shared-pool of the consumer. This is an ideal model which provides strict privacy without compromising on the performance. But it needs smarter generators. The recent NIC hardwares like \textbf{Solarflair} and \textbf{e10000} are capable of performing such a classification. \section{Adaptability with different hardware} This section discusses the ability of this design to adapt with different designs and features of NIC hardwares. We believe that the interface provided by the producer is generic enough to cover most of the functionalities provided by the smarter generators. And as we can control the integration between producer and generator without affecting the consumers, we can easily adapt to exploit the different hardware features provided by the generators. One of the example of such an adaptation can be seen in \ref{sec:generator-classification-privacy}, the design discussed here easily adapted to exploit the early classification capabilities of the generator to give stricter privacy based zero copy bulk transport. \section{Limitations} This design still misses some good features in favor of keeping the complexity reasonable. These features include support for multiple-producer, multiple-consumer setup and ability for consumer to specify where data-element should be generated (true zero copy receive). \section{Conclusion} This chapter provides the details about the new cross-domain transfer facility design. This facility allows zero-copy network implementation which can support sharing between more than two domains as well as multi-level sharing (sharing by consumers between consumers). This solution also reduce the number of messages exchanged by sending them only when it is absolutely necessary. \end{document} %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%