\label{api_streaming_md_doc_streaming}%
\Hypertarget{api_streaming_md_doc_streaming}%
 A key design requirement for librsync is that it should handle data as and when the hosting application requires it. librsync can be used inside applications that do non-\/blocking IO or filtering of network streams, because it never does IO directly, or needs to block waiting for data.

Arbitrary-\/length input and output buffers are passed to the library by the application, through an instance of \mbox{\hyperlink{librsync_8h_abf9f543dbfe5c1e60c8ed1c93d087767}{rs\+\_\+buffers\+\_\+t}}. The library proceeds as far as it can, and returns an \mbox{\hyperlink{librsync_8h_adec85b529224f0240ae1afccff827462}{rs\+\_\+result}} value indicating whether it needs more data or space.

All the state needed by the library to resume processing when more data is available is kept in a small opaque \mbox{\hyperlink{librsync_8h_add6622b38e3fa557301a190876d5ed4a}{rs\+\_\+job\+\_\+t}} structure. After creation of a job, repeated calls to \mbox{\hyperlink{librsync_8h_aede8e0f42424b9aa29093f94b59ea029}{rs\+\_\+job\+\_\+iter()}} in between filling and emptying the buffers keeps data flowing through the stream. The \mbox{\hyperlink{librsync_8h_adec85b529224f0240ae1afccff827462}{rs\+\_\+result}} values returned may indicate


\begin{DoxyItemize}
\item \mbox{\hyperlink{librsync_8h_a7feb858ceba3b8f3cf048d49be108253a739063053a289b5c3393d78cc77b41b2}{RS\+\_\+\+DONE}}\+: processing is complete
\item \mbox{\hyperlink{librsync_8h_a7feb858ceba3b8f3cf048d49be108253af2d289bbc65678b4b00f56a2e6632957}{RS\+\_\+\+BLOCKED}}\+: processing has blocked pending more data
\item one of various possible errors in processing (see \mbox{\hyperlink{librsync_8h_adec85b529224f0240ae1afccff827462}{rs\+\_\+result}}.)
\end{DoxyItemize}

These can be converted to a human-\/readable string by \mbox{\hyperlink{librsync_8h_a53c3ba4c320a497218a86936e8ee802c}{rs\+\_\+strerror()}}.

\begin{DoxyNote}{Note}
Smaller buffers have high relative handling costs. Application performance will be improved by using buffers of at least 32kb or so on each call.
\end{DoxyNote}
\begin{DoxySeeAlso}{See also}
\mbox{\hyperlink{api_whole}{Whole-\/file API}} -\/ Simpler but more limited interface than the streaming interface.

\mbox{\hyperlink{api_pull}{Pull API}} -\/ Intermediate-\/complexity callback interface.

\mbox{\hyperlink{api_callbacks}{IO callbacks}} -\/ for reading from the basis file when doing a \char`\"{}patch\char`\"{} operation.
\end{DoxySeeAlso}
\hypertarget{api_streaming_autotoc_md80}{}\doxysubsection{Creating Jobs}\label{api_streaming_autotoc_md80}
All streaming librsync jobs are initiated using a {\ttfamily \+\_\+begin} function to create a \mbox{\hyperlink{librsync_8h_add6622b38e3fa557301a190876d5ed4a}{rs\+\_\+job\+\_\+t}} object, passing in any necessary initialization parameters. The various jobs available are\+:


\begin{DoxyItemize}
\item \mbox{\hyperlink{librsync_8h_a226f57577fcad2e4c97f3e4d612650d3}{rs\+\_\+sig\+\_\+begin()}}\+: Calculate the signature of a file.
\item \mbox{\hyperlink{librsync_8h_a5898e9ed00b5cdfe5fa58bfa0f25821c}{rs\+\_\+loadsig\+\_\+begin()}}\+: Load a signature into memory.
\item \mbox{\hyperlink{delta_8c_a59fe8536218632417097d084606bb675}{rs\+\_\+delta\+\_\+begin()}}\+: Calculate the delta between a signature and a new file.
\item \mbox{\hyperlink{librsync_8h_a15efa6e180d239fee2c195874618b0ea}{rs\+\_\+patch\+\_\+begin()}}\+: Apply a delta to a basis to recreate the new file.
\end{DoxyItemize}

Additionally, the following helper functions can be used to get the recommended signature arguments from the input file\textquotesingle{}s size.


\begin{DoxyItemize}
\item \mbox{\hyperlink{librsync_8h_a3c4c3fcea2814610109536f3ae029c5e}{rs\+\_\+sig\+\_\+args()}}\+: Get the recommended sigature arguments from the file size.
\end{DoxyItemize}

After a signature has been loaded, before it can be used to calculate a delta, the hashtable needs to be initialized by calling


\begin{DoxyItemize}
\item \mbox{\hyperlink{librsync_8h_a1c11ff785ebcd7210f5485d0c97fc812}{rs\+\_\+build\+\_\+hash\+\_\+table()}}\+: Initialized the signature hashtable.
\end{DoxyItemize}

The patch job accepts the patch as input, and uses a callback to look up blocks within the basis file.

You must configure read, write and basis callbacks after creating the job but before it is run.\hypertarget{api_streaming_autotoc_md81}{}\doxysubsection{Running Jobs}\label{api_streaming_autotoc_md81}
The work of the operation is done when the application calls \mbox{\hyperlink{librsync_8h_aede8e0f42424b9aa29093f94b59ea029}{rs\+\_\+job\+\_\+iter()}}. This includes reading from input files via the callback, running the rsync algorithms, and writing output.

The IO callbacks are only called from inside \mbox{\hyperlink{librsync_8h_aede8e0f42424b9aa29093f94b59ea029}{rs\+\_\+job\+\_\+iter()}}. If any of them return an error, \mbox{\hyperlink{librsync_8h_aede8e0f42424b9aa29093f94b59ea029}{rs\+\_\+job\+\_\+iter()}} will generally return the same error.

When librsync needs to do input or output, it calls one of the callback functions. \mbox{\hyperlink{librsync_8h_aede8e0f42424b9aa29093f94b59ea029}{rs\+\_\+job\+\_\+iter()}} returns when the operation has completed or failed, or when one of the IO callbacks has blocked.

\mbox{\hyperlink{librsync_8h_aede8e0f42424b9aa29093f94b59ea029}{rs\+\_\+job\+\_\+iter()}} will usually be called in a loop, perhaps alternating librsync processing with other application functions.\hypertarget{api_streaming_autotoc_md82}{}\doxysubsection{Deleting Jobs}\label{api_streaming_autotoc_md82}
A job is deleted and its memory freed up using \mbox{\hyperlink{librsync_8h_a297b5522990715aff982f8556052c273}{rs\+\_\+job\+\_\+free()}}.

This is typically called when the job has completed or failed. It can be called earlier if the application decides it wants to cancel processing.

\mbox{\hyperlink{librsync_8h_a297b5522990715aff982f8556052c273}{rs\+\_\+job\+\_\+free()}} does not delete the output of the job, such as the sumset loaded into memory. It does delete the job\textquotesingle{}s statistics.\hypertarget{api_streaming_autotoc_md83}{}\doxysubsection{State Machine Internals}\label{api_streaming_autotoc_md83}
Internally, the operations are implemented as state machines that move through various states as input and output buffers become available.

All computers and programs are state machines. So why is the representation as a state machine a little more explicit (and perhaps verbose) in librsync than other places? Because we need to be able to let the real computer go off and do something else like waiting for network traffic, while still remembering where it was in the librsync state machine.

librsync will never block waiting for IO, unless the callbacks do that.

The current state is represented by the private field \mbox{\hyperlink{structrs__job_aa789990d61f5eadc6a26aeedf4bec765}{rs\+\_\+job\+\_\+t\+::statefn}}, which points to a function with a name like {\ttfamily rs\+\_\+\+OPERATION\+\_\+s\+\_\+\+STATE}. Every time librsync tries to make progress, it will call this function.

The state function returns one of the \mbox{\hyperlink{librsync_8h_adec85b529224f0240ae1afccff827462}{rs\+\_\+result}} values. The most important values are


\begin{DoxyItemize}
\item \mbox{\hyperlink{librsync_8h_a7feb858ceba3b8f3cf048d49be108253a739063053a289b5c3393d78cc77b41b2}{RS\+\_\+\+DONE}}\+: Completed successfully.
\item \mbox{\hyperlink{librsync_8h_a7feb858ceba3b8f3cf048d49be108253af2d289bbc65678b4b00f56a2e6632957}{RS\+\_\+\+BLOCKED}}\+: Cannot make further progress at this point.
\item \mbox{\hyperlink{librsync_8h_a7feb858ceba3b8f3cf048d49be108253a5366e53561c2ec9a897470535ea0139c}{RS\+\_\+\+RUNNING}}\+: The state function has neither completed nor blocked but wants to be called again. {\bfseries{XXX}}\+: Perhaps this should be removed?
\end{DoxyItemize}

States need to correspond to suspension points. The only place the job can resume after blocking is at the entry to a state function.

Therefore states must be \char`\"{}all or nothing\char`\"{} in that they can either complete, or restart without losing information.

Basically every state needs to work from one input buffer to one output buffer.

States should never generally return \mbox{\hyperlink{librsync_8h_a7feb858ceba3b8f3cf048d49be108253a739063053a289b5c3393d78cc77b41b2}{RS\+\_\+\+DONE}} directly. Instead, they should call rs\+\_\+\+\_\+job\+\_\+done(), which sets the state function to rs\+\_\+\+\_\+s\+\_\+done(). This makes sure that any pending output is flushed out before \mbox{\hyperlink{librsync_8h_a7feb858ceba3b8f3cf048d49be108253a739063053a289b5c3393d78cc77b41b2}{RS\+\_\+\+DONE}} is returned to the application. 