[Qemu-devel] [PATCHv2] Improve documentation of FUA and FLUSH

[Alex Bligh]

Improve the documentation of NBD_CMD_FLUSH and NBD_CMD_FLAG_FUA. Specifically
the latter may be set on any command, and its semantics on commands other
than NBD_CMD_WRITE need explaining. Further, explain how these relate to
reordering of commands.

Signed-off-by: Alex Bligh <address@hidden>
---
 doc/proto.md | 83 +++++++++++++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 68 insertions(+), 15 deletions(-)

Changes since v1:

* Rebased on master

* Eric's modifications

* Included 2 options as to how we might handle FUA on non-write commands.
  Let's establish whether either the kernel or Qemu have meaningful uses for
  read+FUA.

diff --git a/doc/proto.md b/doc/proto.md
index 8c83382..389b186 100644
--- a/doc/proto.md
+++ b/doc/proto.md
@@ -213,6 +213,47 @@ handle as was sent by the client in the corresponding 
request.  In
 this way, the client can correlate which request is receiving a
 response.
 
+#### Ordering of messages and writes
+
+The server MAY process commands out of order, and MAY reply out of
+order, save that:
+
+* All write commands (that includes both `NBD_CMD_WRITE` and
+  `NBD_CMD_TRIM`) that the server completes (i.e. replies to)
+  prior to processing to a `NBD_CMD_FLUSH` MUST be written to non-volatile
+  storage prior to replying to that `NBD_CMD_FLUSH`. The server SHOULD ensure
+  that all write command received prior to processing the `NBD_CMD_FLUSH`
+  (whether they are replied to or not) are written to non-volatile
+  storage prior to processing an `NBD_CMD_FLUSH`; note this is a
+  stronger condition than the previous 'MUST' condition. This
+  paragraph only applies if `NBD_FLAG_SEND_FLUSH` is set within
+  the transmission flags, as otherwise `NBD_CMD_FLUSH` will never
+  be sent by the client to the server.
+
+* A server MUST NOT reply to a command that has `NBD_CMD_FLAG_FUA` set
+
+[Option #C1]
+
+  in its command flags until the data area referred to by that command
+
+[Option #C1]
+
+  in its command flags until the data (if any) written by that command
+
+[All options]
+
+  is persisted to non-volatile storage. This only applies if
+  `NBD_FLAG_SEND_FLUSH` is set within the transmission flags, as otherwise
+  `NBD_CMD_FLAG_FUA` will not be set on any commands sent to the server
+  by the client.
+
+`NBD_CMD_FLUSH` is modelled on the Linux kernel empty bio with
+`REQ_FLUSH` set. `NBD_CMD_FLAG_FUA` is modelled on the Linux
+kernel bio with `REQ_FUA` set. In case of ambiguity in this
+specification, the
+[kernel 
documentation](https://www.kernel.org/doc/Documentation/block/writeback_cache_control.txt)
+may be useful.
+
 #### Request message
 
 The request message, sent by the client, looks as follows:
@@ -480,10 +521,33 @@ affects a particular command.  Clients MUST NOT set a 
command flag bit
 that is not documented for the particular command; and whether a flag is
 valid may depend on negotiation during the handshake phase.
 
-- bit 0, `NBD_CMD_FLAG_FUA`; valid during `NBD_CMD_WRITE` and
-  `NBD_CMD_WRITE_ZEROES` commands.  SHOULD be set to 1 if the client requires
-  "Force Unit Access" mode of operation.  MUST NOT be set unless transmission
-  flags included `NBD_FLAG_SEND_FUA`.
+- bit 0, `NBD_CMD_FLAG_FUA`. This flag is valid for all commands. This bit
+  MUST be set to 0 unless the `NBD_FLAG_SEND_FUA` flag ("Force Unit Access")
+  was set in the transmission flags field. If the `NBD_FLAG_SEND_FUA`
+  is set in the transmission flags field, the client MAY set
+  `NBD_CMD_FLAG_FUA` in any request. If this bit is set, the server
+
+[Option #C1]
+
+  MUST NOT send a reply until it has ensured that any data referred to
+  by this request (i.e. written data on a `NBD_CMD_WRITE` or
+  `NBD_CMD_WRITE_ZEROES`, read data on an `NBD_CMD_READ`) has reached
+  permanent storage. There will be certain commands
+  (e.g. `NBD_CMD_DISC`) for which this flag will thus not alter behaviour
+  (as the command does not refer to any data), in which case the server
+  MUST ignore this bit.
+
+[Option #C2]
+
+  MUST NOT send a reply until it has ensured that the data (if any) written
+  by this request (e.g. by `NBD_CMD_WRITE` or `NBD_CMD_WRITE_ZEROES`) has
+  reached permanent storage. There will be certain commands (e.g. 
`NBD_CMD_READ`
+  and `NBD_CMD_DISC`) for which this flag will thus not alter behaviour
+  (as the command does not write any data), in which case the server
+  MUST ignore this bit.
+
+[All options]
+
 - bit 1, `NBD_CMD_MAY_TRIM`; defined by the experimental `WRITE_ZEROES`
   extension; see below.
 - bit 2, `NBD_CMD_FLAG_DF`; defined by the experimental `STRUCTURED_REPLY`
@@ -532,12 +596,6 @@ The following request types exist:
     message. The server MAY send the reply message before the data has
     reached permanent storage.
 
-    If the `NBD_FLAG_SEND_FUA` flag ("Force Unit Access") was set in the
-    transmission flags field, the client MAY set the flag `NBD_CMD_FLAG_FUA` in
-    the command flags field. If this flag was set, the server MUST NOT send
-    the reply until it has ensured that the newly-written data has reached
-    permanent storage.
-
     If an error occurs, the server SHOULD set the appropriate error code
     in the error field. The server MAY then close the connection.
 
@@ -749,11 +807,6 @@ one new command and one new command flag.
     A client MUST NOT send a write zeroes request unless
     `NBD_FLAG_SEND_WRITE_ZEROES` was set in the transmission flags field.
 
-    If the `NBD_FLAG_SEND_FUA` flag was set in the transmission flags field,
-    the client MAY set the flag `NBD_CMD_FLAG_FUA` in the command flags field.
-    If this flag was set, the server MUST NOT send the reply until it has
-    ensured that the newly-zeroed data has reached permanent storage.
-
     If the flag `NBD_CMD_FLAG_MAY_TRIM` was set by the client in the command
     flags field, the server MAY use trimming to zero out the area, but it
     MUST ensure that the data reads back as zero.
-- 
1.9.1