No validator in 200 response for conditional update

Discussion:

Vladimir Lashchev

2018-10-26 22:52:21 UTC

Hello,

RFC7232 in https://tools.ietf.org/html/rfc7232#section-3.1 has the following clause:
```
In the latter case, the origin server MUST NOT send a validator header field in the response unless it can verify that the request is a duplicate of an immediately prior change made by the same user agent
```
It doesn't really explain what security or performance considerations are leading to such a requirement and seems to favor idempotent updates coming from the same user agent.
Sending validator (ETag) to all requestors seems to be a simpler and better choice.
Could somebody please clarify why we need to this as suggested in RFC?

Thanks,
Vladimir Lashchev

Julian Reschke

2018-10-27 09:38:38 UTC

Permalink

Post by Vladimir Lashchev
Hello,
```
In the latter case, the origin server MUST NOT send a validator header field in the response unless it can verify that the request is a duplicate of an immediately prior change made by the same user agent
```
It doesn't really explain what security or performance considerations are leading to such a requirement and seems to favor idempotent updates coming from the same user agent.
Sending validator (ETag) to all requestors seems to be a simpler and better choice.
Could somebody please clarify why we need to this as suggested in RFC?
Thanks,
Vladimir Lashchev
b) one of the 2xx (Successful) status codes if the origin server
has verified that a state change is being requested and the final
state is already reflected in the current state of the target
resource (i.e., the change requested by the user agent has already
succeeded, but the user agent might not be aware of it, perhaps
because the prior response was lost or a compatible change was made
by some other user agent)

...so that's an edge case already.

Best regards, Julian

Yishuai Li

2018-10-27 06:04:34 UTC

Permalink

I am also confused by this clause. Consider the following scenario (for
simplicity, the entity-tag is identical to entity, and the all requests
have the same target):

(1) A --> S : PUT "a" (unconditional)
(2) S --> A : 204 No Content, ETag: "a"
(3) A --> S : PUT "a", If-Match: "b" (not satisfied)
(4) S --> A : 204 No Content
(5) A --> S : PUT "a", If-Match: "b" (not satisfied)
(6) S --> A : 204 No Content, ETag "a"

Did I understand the specification correctly that: S MUST NOT send a
validator in (4), but MAY do so in (6), because (5) is a duplicate of (3),
but (3) is not a duplicate of (1)?

Thanks,
Yishuai Li

Julian Reschke

2018-10-29 13:05:19 UTC

Permalink

Why would step (4) result in a 204? Should be Precondition Failed, no?

Best regards, Julian

Julian Reschke

2018-10-29 17:06:07 UTC

Permalink

Post by Yishuai Li
I am also confused by this clause. Consider the following scenario (for
simplicity, the entity-tag is identical to entity, and the all requests
(1) A --> S : PUT "a" (unconditional)
(2) S --> A : 204 No Content, ETag: "a"
(3) A --> S : PUT "a", If-Match: "b" (not satisfied)
(4) S --> A : 204 No Content
(5) A --> S : PUT "a", If-Match: "b" (not satisfied)
(6) S --> A : 204 No Content, ETag "a"
Did I understand the specification correctly that: S MUST NOT send a
validator in (4), but MAY do so in (6), because (5) is a duplicate of
(3), but (3) is not a duplicate of (1)?
Thanks,
Yishuai Li
...

Ok, now I understand the question. Ignore my previous reply.

So that's an edge case of an edge case, right?

Post by Yishuai Li
An origin server MUST NOT perform the requested method if a received
If-Match condition evaluates to false; instead, the origin server
MUST respond with either a) the 412 (Precondition Failed) status code
or b) one of the 2xx (Successful) status codes if the origin server
has verified that a state change is being requested and the final
state is already reflected in the current state of the target
resource (i.e., the change requested by the user agent has already
succeeded, but the user agent might not be aware of it, perhaps
because the prior response was lost or a compatible change was made
by some other user agent). In the latter case, the origin server
MUST NOT send a validator header field in the response unless it can
verify that the request is a duplicate of an immediately prior change
made by the same user agent.

Note the "immediately prior change" - so no (4) did not result in a
change, so the rule does not apply and no etag should be sent.

Best regards, Julian

Vladimir Lashchev

2018-11-06 22:35:28 UTC

Permalink

Thanks for your replies!

In general, since the spec says that server "MUST NOT" send an ETag in some case so in order to comply with the spec we must implement that logic and it does not matter if it is a corner case or if it may happen only once in our API. In reality it is actually not an edge case and happens quite often (up to single digit % of all requests in high volume concurrent scenarios).

That's why I am trying to understand the reason because that clause calls for a lot of development.

Why "a lot" of development?
Because the clause "... unless it can verify that the request is a duplicate of an immediately prior change made by the same user agent ..." requires us to persistently track previous requests from all clients which is a huge infra requirement for stateless API server farms (which is a typical design for servers with high performance requirements).

Actually, scenario of that 2xx clause (as I read it) is a bit more complicated - it applies in the case when 2 agents (A1 and A2) are talking to the sever and trying to apply semantically same changes or same agent asks for the same change in non-consecutive requests. Here is my understanding:
1) "state change is being requested and the final state is already reflected in the current state" ==> means that requests may not be identical, as long as they ask for the same end-result state.
2) "server MUST NOT send a validator header field in the response unless it can verify that the request is a duplicate of an immediately prior change made by the same user agent " ==> means that:
a) server MUST NOT send ETag to A2 if state change was done by A1, but similar change is requested by A2
b) server MUST NOT send ETag to A1 if state change was done by A1 but there was some other request (even GET without any side-effects) between that original change request and a new, semantically-identical change request.

So expanding on Yishuai's example to include all cases it would be something like:

(step 1) A1 --> S : PUT "a" (unconditional)
(step 2) S --> A1 : 204 No Content, ETag: "a"
(step 3) A1 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 4) S --> A1 : 204 No Content, ETag: "a" (same change-originating agent, same end-result requested in the immediate prior request)
(step 5) A2 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 6) S --> A1 : 204 No Content (No ETag) (not a change-originating agent, same end-result requested)
(step 7) A1 --> S : GET ... (unconditional, no side-effects)
(step 8) S-> A1 (doesn't matter what is replied here)
(step 9) A1 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 10) S --> A1 : 204 No Content (No ETag) (same change-originating agent, same end-result requested, but not in the immediate prior request)
(step 11) A3 --> S : PUT "zzz", If-Match: "b" (not satisfied)
(step 12) S --> A1 : 412 No Content (No ETag) (any agent, different end-result requested)

My question is about (step 6) and (step 10) where server MUST NOT send a ETag value back.
Above is quite complicated logic required without explanation and it is not clear of how it is better/safer than simple and stateless:
(step 1) A1 --> S : PUT "a" (unconditional)
(step 2) S --> A1 : 204 No Content, ETag: "a"
(step 3) A1 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 4) S --> A1 : 204 No Content, ETag: "a" (any agent, same end-result requested)
(step 11) A3 --> S : PUT "zzz", If-Match: "b" (not satisfied)
(step 12) S --> A1 : 412 No Content (No ETag) (any agent, different end-result requested)

Thanks,
Vladimir

-----Original Message-----
From: Julian Reschke <***@gmx.de>
Sent: Monday, October 29, 2018 10:06 AM
To: Yishuai Li <***@cis.upenn.edu>; Vladimir Lashchev <***@oracle.com>
Cc: ietf-http-***@w3.org
Subject: Re: No validator in 200 response for conditional update

Post by Yishuai Li
I am also confused by this clause. Consider the following scenario
(for simplicity, the entity-tag is identical to entity, and the all
(1) A --> S : PUT "a" (unconditional)
(2) S --> A : 204 No Content, ETag: "a"
(3) A --> S : PUT "a", If-Match: "b" (not satisfied)
(4) S --> A : 204 No Content
(5) A --> S : PUT "a", If-Match: "b" (not satisfied)
(6) S --> A : 204 No Content, ETag "a"
Did I understand the specification correctly that: S MUST NOT send a
validator in (4), but MAY do so in (6), because (5) is a duplicate of
(3), but (3) is not a duplicate of (1)?
Thanks,
Yishuai Li
...

Ok, now I understand the question. Ignore my previous reply.

So that's an edge case of an edge case, right?

Note the "immediately prior change" - so no (4) did not result in a change, so the rule does not apply and no etag should be sent.

Best regards, Julian

Amos Jeffries

2018-11-07 01:03:07 UTC

Permalink

Post by Vladimir Lashchev
Thanks for your replies!
In general, since the spec says that server "MUST NOT" send an ETag in some case so in order to comply with the spec we must implement that logic and it does not matter if it is a corner case or if it may happen only once in our API. In reality it is actually not an edge case and happens quite often (up to single digit % of all requests in high volume concurrent scenarios).

On a side note;

My understanding of the expected behaviour is that agents doing a lot of
repeats are broken in some way. Probably their author misunderstood that
412 is *not* an error.

A 4xx is just a notice from the server to the agent that the requested
change/fetch was not enacted. Not really errors in the way we tend to
think of errors in software.

So agents receiving back a 412 with ETag matching the one they tried to
set the resource to - should usually treat that as as much a success as
any 2xx they get. It is only when the 412 ETag is different that the
agent needs to figure whether to try again, and HTTP cannot help by
specifying anything about that decision.

I fail to see how there is any difference in overhead for a large farm
than for a single server. All that needs to be tracked is what the
current ETag validator is and which agent made it exist. Usually those
would exist anyway in server state as author and version IDs.

In a large farm each stateless server may be tracking a different
version of the object (or not) with its own ETag + agent labels. That is
a given implied by them being stateless. The condition rules from the
request are applied by the individual server against its own object(s)
copy regardless of what other objects might be elsewhere or what might
have happened in old history for the relevant local object(s).

You _can_ do sharing of updates + info between the servers. But that is
up to the implementation and kind of disolves the claim of being stateless.

That said, I don't recall what was behind this RFC wording so am
watching to see what others reply to your specific questions.

HTH
Amos

Amos Jeffries

2018-11-14 05:35:05 UTC

Permalink

From: Amos Jeffries
Sent: Tuesday, November 6, 2018 5:03 PM
On a side note;
My understanding of the expected behaviour is that agents doing a lot of
repeats are broken in some way. Probably their author misunderstood that
412 is *not* an error.
A 4xx is just a notice from the server to the agent that the requested
change/fetch was not enacted. Not really errors in the way we tend to think
of errors in software.
So agents receiving back a 412 with ETag matching the one they tried to set
the resource to - should usually treat that as as much a success as any 2xx
they get. It is only when the 412 ETag is different that the agent needs to
figure whether to try again, and HTTP cannot help by specifying anything
about that decision.

Actually 412 is a proper error as I see it.
Server denies modification request and indicates that request was not successfully processed (desired resulting state was not achieved).

You are assuming "state" here. Which is wrong and leading you to a wrong
conclusion.

412 states that the _preconditions_ in that individual request were not
obtainable. By itself it does not say anything about any state beyond
the individual request message being responded to.

The ETag validator is the only bit involving any higher-level state and
that has a defined meaning agnostic of the status code.

I don't see a scenario where client can treat it as a success. It is a failed update in an optimistic concurrency flow.

When the precondition fails because the Etag has already been changed to
a) the desired one, or b) something even newer than the desired one.

The exact edge-edge-case under discussion here is one situation where
the desired state *was* achieved without the preconditions being true.
It just happened to be achieved previously by an earlier change request.

Anyway, my question is actually about 2xx response and not 412 :-)

And the answer perhaps is "why 2xx when 412 conveys the desired meaning".

an immediately prior change made by the same user agent ..." requires us to
persistently track previous requests from all clients which is a huge infra
requirement for stateless API server farms (which is a typical design for
servers with high performance requirements).
I fail to see how there is any difference in overhead for a large farm than for
a single server. All that needs to be tracked is what the current ETag validator
is and which agent made it exist. Usually those would exist anyway in server
state as author and version IDs.

Request #2 can come to any server in the farm and spec calls that all servers should know about previous request from that agent so they can decide whether to send ETag back or not.
It calls for state sharing among all the servers and thus they become not stateless (which we do not want to have).

That is not my interpretation.

The spec is written in terms of individual request-response
transactions. Not in terms of client-service transactions - which may
span multiple request-response transactions in a stateful way.

In a large farm each stateless server may be tracking a different version of
the object (or not) with its own ETag + agent labels. That is a given implied by
them being stateless. The condition rules from the request are applied by
the individual server against its own object(s) copy regardless of what other
objects might be elsewhere or what might have happened in old history for
the relevant local object(s).

Spec is defining how a particular server in a farm should behave - it defines API endpoint contract.

Agreed.

Endpoint is deployed at reverse proxy that is fanning out requests to a farm of identical servers.

"server" does not mean "server state". The underlying HTTP protocol /
contract is stateless. The service layer above usually has state, but
may not.

We don't have sticky session, so modification request #2 can be processed on a different server from where request #1 was processed.
Spec defines how client will see the response regardless of how many servers we have, so it should not matter on what worker server it was processed.

Here you assume a relationship between end-servers which is beyond
anything HTTP covers.

The spec / API contract outlines the set of valid responses to any given
request. The meanings for various parts of the message is relative to
the status and the request type.

The spec does *not* dictate statefulness between servers which would
lead to identical behaviour.

Yes there is a design problem inherent in that, but by remaining
strictly stateless HTTP pushes it up to the layer above. There are many
possible solutions to that and IIRC most if not all of them, are
actively used somewhere.

You _can_ do sharing of updates + info between the servers. But that is up to
the implementation and kind of disolves the claim of being stateless.

Yes, and we don't want state sharing.

Then you need the client endpoints to be robust against servers being at
different update levels with their internal state.

Such as by using the spec interpretation roughly along the lines I used
earlier to guide design decisions in the client software:

* 5xx are service errors, the client unlikely able do anything to fix
(some exceptions where it can retry).

* 4xx notices that the request did not go exactly as client intended,
with hints to the client about why. Leaving the client to decide if
retry is worth doing based on those hints.

* 2xx everything went exactly as planned. Nothing further for the client
to consider.

If you try to go beyond that to push multiple meaning into the 2xx
status is when you hit problems with whether ETag is possible to send at
all, or what value to send when the server is *more* up to date than the
client.
Both semantically have the same "I could not do this because I have
object "a"' meaning. The client is the agent which knows best whether
that situation ignorable or needs handling.

AYJ

Vladimir Lashchev

2018-11-29 23:33:58 UTC

Permalink

Sent: Tuesday, November 13, 2018 9:35 PM

Actually 412 is a proper error as I see it.
Server denies modification request and indicates that request was not
successfully processed (desired resulting state was not achieved).

You are assuming "state" here. Which is wrong and leading you to a wrong
conclusion.
412 states that the _preconditions_ in that individual request were not
obtainable. By itself it does not say anything about any state beyond the
individual request message being responded to.
The ETag validator is the only bit involving any higher-level state and that has a
defined meaning agnostic of the status code.

I mean "state" as in the paragraph in question (rfc7232#section-3.1):
"... a) the 412 (Precondition Failed) status code
or b) one of the 2xx (Successful) status codes if the origin server
has verified that a state change is being requested and the final
state is already reflected in the current state of the target
resource ..."

I don't see a scenario where client can treat it as a success. It is a failed update
in an optimistic concurrency flow.

When the precondition fails because the Etag has already been changed to
a) the desired one, or b) something even newer than the desired one.

Case a) is explicitly covered with 2xx return code in rfc7232#section-3.1 in question
Not sure about case b) because ETag in general cannot be compared with "newer", it is _typically_ a black box like hash or GUID.

The exact edge-edge-case under discussion here is one situation where the
desired state *was* achieved without the preconditions being true.
It just happened to be achieved previously by an earlier change request.

Yes.

To be precise - it applies only if desired state was achieved by immediate previous request from the same exact agent.
It actually doesn't matter if it is edge case or not - server that conforms with RFC MUST implement it (that what RFC says).

Anyway, my question is actually about 2xx response and not 412 :-)

And the answer perhaps is "why 2xx when 412 conveys the desired meaning".

Responding with 2xx provides an optimization by converting some error flows into positive flows.

Request #2 can come to any server in the farm and spec calls that all servers

should know about previous request from that agent so they can decide whether
to send ETag back or not.

It calls for state sharing among all the servers and thus they become not

stateless (which we do not want to have).
That is not my interpretation.
The spec is written in terms of individual request-response transactions. Not in
terms of client-service transactions - which may span multiple request-response
transactions in a stateful way.

Could you please help me understand how this sentence from RFC can be applied to a single request-response:
"unless [server] can verify that the request is a duplicate of an immediately prior change
made by the same user agent"
To apply such logic IMHO server needs to remember prior changes made by each agent.
And remembering prior changes implies state management on the server farm.

The spec / API contract outlines the set of valid responses to any given request.
The meanings for various parts of the message is relative to the status and the
request type.
The spec does *not* dictate statefulness between servers which would lead to
identical behaviour.

Are we still talking about rfc7232#section-3.1?
How the check required by that section can be achieved without statefulness on the server?

Such as by using the spec interpretation roughly along the lines I used earlier to
...
If you try to go beyond that to push multiple meaning into the 2xx status is when
you hit problems with whether ETag is possible to send at all, or what value to
send when the server is *more* up to date than the client.
Both semantically have the same "I could not do this because I have object "a"'
meaning. The client is the agent which knows best whether that situation
ignorable or needs handling.

Going back to the question at hand - what would be your proposal on how to change rfc7232#section-3.1, part with 412 and 2xx + (conditional) ETag responses?

Mine was to remove this sentence:
" In the latter case, the origin server
MUST NOT send a validator header field in the response unless it can
verify that the request is a duplicate of an immediately prior change
made by the same user agent"

VL

Amos Jeffries

2018-11-30 11:57:19 UTC

Permalink

Post by Vladimir Lashchev

From: Amos Jeffries
Sent: Tuesday, November 13, 2018 9:35 PM

Actually 412 is a proper error as I see it.
Server denies modification request and indicates that request was not
successfully processed (desired resulting state was not achieved).

You are assuming "state" here. Which is wrong and leading you to a wrong
conclusion.
412 states that the _preconditions_ in that individual request were not
obtainable. By itself it does not say anything about any state beyond the
individual request message being responded to.
The ETag validator is the only bit involving any higher-level state and that has a
defined meaning agnostic of the status code.

"... a) the 412 (Precondition Failed) status code
or b) one of the 2xx (Successful) status codes if the origin server
has verified that a state change is being requested and the final
state is already reflected in the current state of the target
resource ..."

The "state" being mentioned in the spec is the exact octets of the
resource itself. The value / octets which the request is attempting to
change.

The way you have been using "state" was to mean the entire history
information of the resource. Which is very a different type of "state".

The difference between changing a document, and changing the changelog
of a document. The former can be done with some flexibility in the
response payload, the latter cannot.

Post by Vladimir Lashchev

I don't see a scenario where client can treat it as a success. It is a failed update
in an optimistic concurrency flow.

When the precondition fails because the Etag has already been changed to
a) the desired one, or b) something even newer than the desired one.

Yes.
To be precise - it applies only if desired state was achieved by immediate previous request from the same exact agent.
It actually doesn't matter if it is edge case or not - server that conforms with RFC MUST implement it (that what RFC says).

Anyway, my question is actually about 2xx response and not 412 :-)

And the answer perhaps is "why 2xx when 412 conveys the desired meaning".

Responding with 2xx provides an optimization by converting some error flows into positive flows.

Request #2 can come to any server in the farm and spec calls that all servers

should know about previous request from that agent so they can decide whether
to send ETag back or not.

It calls for state sharing among all the servers and thus they become not

"unless [server] can verify that the request is a duplicate of an immediately prior change
made by the same user agent"
To apply such logic IMHO server needs to remember prior changes made by each agent.
And remembering prior changes implies state management on the server farm.

It does require the "immediately prior" state be remembered. Not the
entire history of state, nor state from other servers.

The resource may be sensitive to distributed changes. But that is a
separate issue imposed by the nature of the resource not this RFC. Any
other protocol would face the same issue.

Post by Vladimir Lashchev

Are we still talking about rfc7232#section-3.1?
How the check required by that section can be achieved without statefulness on the server?

The resource gets saved to a file (or equivalent). Changes to it get
labeled with modification time and some label to identify the client
doing the modification. That is all that is needed IMO to meet the spec
requirement.

The server receiving an conditional change request checks it against
those two details - according to the section-3.1 criteria.

Then (always) updates its stored label and mtime values so any followup
request gets based on the new details. Some nil value for the client
label on duplicates so these repeats get detected easily as a mismatch
on the repeats not "immediately prior".

The server can then respond correctly correctly to any clients while it
asynchronously handles any sync with a master copy.

Post by Vladimir Lashchev

Going back to the question at hand - what would be your proposal on how to change rfc7232#section-3.1, part with 412 and 2xx + (conditional) ETag responses?
" In the latter case, the origin server
MUST NOT send a validator header field in the response unless it can
verify that the request is a duplicate of an immediately prior change
made by the same user agent"

Unless someone else can recall why the text is the way it is now. My
preference would also be to go with an Eratta for making that change.

Amos

Julian Reschke

2018-11-07 07:51:36 UTC

Permalink

Are we talking about the same thing?

1) It needs to be a state-modifying request,

2) That comes with a validator,

3) The validator doesn't match, but the current state of the resource does.

So this only happens when multiple clients do concurrent updates of a
resource, and write the same content.

It would be interesting to understand why this happens so frequently -
can you provide more information about the type of the server/service?

Post by Vladimir Lashchev
That's why I am trying to understand the reason because that clause calls for a lot of development.
Why "a lot" of development?
Because the clause "... unless it can verify that the request is a duplicate of an immediately prior change made by the same user agent ..." requires us to persistently track previous requests from all clients which is a huge infra requirement for stateless API server farms (which is a typical design for servers with high performance requirements).
1) "state change is being requested and the final state is already reflected in the current state" ==> means that requests may not be identical, as long as they ask for the same end-result state.
a) server MUST NOT send ETag to A2 if state change was done by A1, but similar change is requested by A2
b) server MUST NOT send ETag to A1 if state change was done by A1 but there was some other request (even GET without any side-effects) between that original change request and a new, semantically-identical change request.

In-between GET requests may be covered by "duplicate of an immediately
prior change made by the same user agent" - but maybe this could be
made more clearer.

Post by Vladimir Lashchev
(step 1) A1 --> S : PUT "a" (unconditional)
(step 2) S --> A1 : 204 No Content, ETag: "a"
(step 3) A1 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 4) S --> A1 : 204 No Content, ETag: "a" (same change-originating agent, same end-result requested in the immediate prior request)
(step 5) A2 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 6) S --> A1 : 204 No Content (No ETag) (not a change-originating agent, same end-result requested)
(step 7) A1 --> S : GET ... (unconditional, no side-effects)
(step 8) S-> A1 (doesn't matter what is replied here)
(step 9) A1 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 10) S --> A1 : 204 No Content (No ETag) (same change-originating agent, same end-result requested, but not in the immediate prior request)
(step 11) A3 --> S : PUT "zzz", If-Match: "b" (not satisfied)
(step 12) S --> A1 : 412 No Content (No ETag) (any agent, different end-result requested)
My question is about (step 6) and (step 10) where server MUST NOT send a ETag value back.
(step 1) A1 --> S : PUT "a" (unconditional)
(step 2) S --> A1 : 204 No Content, ETag: "a"
(step 3) A1 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 4) S --> A1 : 204 No Content, ETag: "a" (any agent, same end-result requested)
(step 11) A3 --> S : PUT "zzz", If-Match: "b" (not satisfied)
(step 12) S --> A1 : 412 No Content (No ETag) (any agent, different end-result requested)
...

You always have the choice of not replying with 204 but with 412, in
which case the server logic would be simpler, but clients would need to
do more work.

Which brings me back to my question about what kind of server this is
that shows this type of traffic pattern.

Best regards, Julian

Yishuai Li

2018-11-07 12:53:45 UTC

Permalink

As you have said, we are talking about an "edge case of an edge case":
1. If precondition is satisfied, then process; otherwise
2.1. Respond with 412 Precondition Failed, or
2.2. If the proposed change has been reflected, MAY respond with 2xx
(Success), and
2.2.1. If current request is a duplicate of an immediately prior change
made by the same user agent, MAY send etag, otherwise
2.2.2. MUST NOT include etag in the response.

Why do we specify 2.2.1 and 2.2.2? I believe Vladimir has some traffic
pattern that match 2.2, where 2xx Success is more informative than 412
Precondition failed, but determining whether to send an etag or not leads
to extra development effort.

Yishuai

Post by Vladimir Lashchev

Post by Vladimir Lashchev
Thanks for your replies!
In general, since the spec says that server "MUST NOT" send an ETag in

some case so in order to comply with the spec we must implement that logic
and it does not matter if it is a corner case or if it may happen only once
in our API. In reality it is actually not an edge case and happens quite
often (up to single digit % of all requests in high volume concurrent
scenarios).
Are we talking about the same thing?
1) It needs to be a state-modifying request,
2) That comes with a validator,
3) The validator doesn't match, but the current state of the resource does.
So this only happens when multiple clients do concurrent updates of a
resource, and write the same content.
It would be interesting to understand why this happens so frequently -
can you provide more information about the type of the server/service?

Post by Vladimir Lashchev
That's why I am trying to understand the reason because that clause

calls for a lot of development.

Post by Vladimir Lashchev
Why "a lot" of development?
Because the clause "... unless it can verify that the request is a

duplicate of an immediately prior change made by the same user agent ..."
requires us to persistently track previous requests from all clients which
is a huge infra requirement for stateless API server farms (which is a
typical design for servers with high performance requirements).

Post by Vladimir Lashchev
Actually, scenario of that 2xx clause (as I read it) is a bit more

complicated - it applies in the case when 2 agents (A1 and A2) are talking
to the sever and trying to apply semantically same changes or same agent
asks for the same change in non-consecutive requests. Here is my

Post by Vladimir Lashchev
1) "state change is being requested and the final state is already

reflected in the current state" ==> means that requests may not be
identical, as long as they ask for the same end-result state.

Post by Vladimir Lashchev
2) "server MUST NOT send a validator header field in the response unless

it can verify that the request is a duplicate of an immediately prior

Post by Vladimir Lashchev
a) server MUST NOT send ETag to A2 if state change was done by A1, but

similar change is requested by A2

Post by Vladimir Lashchev
b) server MUST NOT send ETag to A1 if state change was done by A1 but

there was some other request (even GET without any side-effects) between
that original change request and a new, semantically-identical change
request.
In-between GET requests may be covered by "duplicate of an immediately
prior change made by the same user agent" - but maybe this could be
made more clearer.

Post by Vladimir Lashchev
So expanding on Yishuai's example to include all cases it would be
(step 1) A1 --> S : PUT "a" (unconditional)
(step 2) S --> A1 : 204 No Content, ETag: "a"
(step 3) A1 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 4) S --> A1 : 204 No Content, ETag: "a" (same change-originating

agent, same end-result requested in the immediate prior request)

Post by Vladimir Lashchev
(step 5) A2 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 6) S --> A1 : 204 No Content (No ETag) (not a change-originating

agent, same end-result requested)

Post by Vladimir Lashchev
(step 7) A1 --> S : GET ... (unconditional, no side-effects)
(step 8) S-> A1 (doesn't matter what is replied here)
(step 9) A1 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 10) S --> A1 : 204 No Content (No ETag) (same change-originating

agent, same end-result requested, but not in the immediate prior request)

Post by Vladimir Lashchev
(step 11) A3 --> S : PUT "zzz", If-Match: "b" (not satisfied)
(step 12) S --> A1 : 412 No Content (No ETag) (any agent, different

end-result requested)

Post by Vladimir Lashchev
My question is about (step 6) and (step 10) where server MUST NOT send a

ETag value back.

Post by Vladimir Lashchev
Above is quite complicated logic required without explanation and it is
(step 1) A1 --> S : PUT "a" (unconditional)
(step 2) S --> A1 : 204 No Content, ETag: "a"
(step 3) A1 --> S : PUT "a", If-Match: "b" (not satisfied)
(step 4) S --> A1 : 204 No Content, ETag: "a" (any agent, same

end-result requested)

Post by Vladimir Lashchev
(step 11) A3 --> S : PUT "zzz", If-Match: "b" (not satisfied)
(step 12) S --> A1 : 412 No Content (No ETag) (any agent, different

end-result requested)

Post by Vladimir Lashchev
...

You always have the choice of not replying with 204 but with 412, in
which case the server logic would be simpler, but clients would need to
do more work.
Which brings me back to my question about what kind of server this is
that shows this type of traffic pattern.
Best regards, Julian

Julian Reschke

2018-11-07 13:01:50 UTC

Permalink

Post by Yishuai Li
1. If precondition is satisfied, then process; otherwise
2.1. Respond with 412 Precondition Failed, or
2.2. If the proposed change has been reflected, MAY respond with 2xx
(Success), and
2.2.1. If current request is a duplicate of an immediately prior change
made by the same user agent, MAY send etag, otherwise
2.2.2. MUST NOT include etag in the response.
Why do we specify 2.2.1 and 2.2.2? I believe Vladimir has some traffic
pattern that match 2.2, where 2xx Success is more informative than 412
Precondition failed, but determining whether to send an etag or not
leads to extra development effort.
Yishuai
...

Well, sending the etag saves work on the client (because it doesn't have
to refetch the etag if it's interested in it).

On the other hand it's more work on the server. The server implementer
can choose what's better...

Best regards, Julian

Yishuai Li

2018-11-07 13:19:28 UTC

Permalink

I think that sending etag is not a huge work itself, if the server is to
implement etag at all. The difficulty is determining whether to send it or
not, because sending etag in 2.2.2 is violating the specification. Is there
a specific reason for this restriction? If not, shall we remove such
constraint?

Yishuai

Post by Julian Reschke

Well, sending the etag saves work on the client (because it doesn't have
to refetch the etag if it's interested in it).
On the other hand it's more work on the server. The server implementer
can choose what's better...
Best regards, Julian

Vladimir Lashchev

2018-11-13 22:52:09 UTC

Permalink

Yishuai summarized the issue correctly.

I think that sending etag is not a huge work itself, if the server is to implement etag at all.
The difficulty is determining whether to send it or not, because sending etag in 2.2.2 is violating the specification. Is there a specific reason for this restriction? If not, shall we remove such constraint?

Exactly.
Sending ETag back all the time is a no-brainer.
There is no additional work here at all since we do send ETag with regular 2xx responses already in state update happy path.
But deciding when to send it and when not to send calls for quite a lot of dev work and previous request tracking infra (ongoing runtime cost).
And we really-really don't want to do this.
But spec says we MUST do this and doesn't explain/hint why.
May be the reason is valid (like security), but we need to know it so we can justify our investment.

Well, sending the etag saves work on the client (because it doesn't have
to refetch the etag if it's interested in it).
On the other hand it's more work on the server. The server implementer
can choose what's better...
Best regards, Julian

Vladimir Lashchev

2018-11-13 22:39:04 UTC

Permalink

Post by Vladimir Lashchev

Post by Vladimir Lashchev
In general, since the spec says that server "MUST NOT" send an ETag in

some case so in order to comply with the spec we must implement that logic
and it does not matter if it is a corner case or if it may happen only once in our
API. In reality it is actually not an edge case and happens quite often (up to
single digit % of all requests in high volume concurrent scenarios).
Are we talking about the same thing?
1) It needs to be a state-modifying request,
2) That comes with a validator,
3) The validator doesn't match, but the current state of the resource does.
So this only happens when multiple clients do concurrent updates of a
resource, and write the same content.

Correct.
Duplicated request from other clients for desired end-state that is already achieved.

Post by Vladimir Lashchev
It would be interesting to understand why this happens so frequently - can
you provide more information about the type of the server/service?

It happens in a typical message-based architecture where components exchange messages and some of them are state changes.
All our state change requests carry If-Match for validation purposes.

Why duplicated updates?
Because components are not just ours, some of them are 3rd party, some are behind bad/geo-long networks.
So, messages in order to be guaranteed, can be delivered one or more times.
Network slowness and errors result in connection terminations which lead to retries and retries lead to state change request duplicates.
This happens pretty often (monthly or so).
But during such network congestions it can be pretty high number of message duplicates.

Post by Vladimir Lashchev
You always have the choice of not replying with 204 but with 412, in which
case the server logic would be simpler, but clients would need to do more
work.

Sending back 200 from the state management endpoint makes sense since desired state is already achieved and it will reduce infra pressure because clients will move forward with such success updates and also less retries/error handling will be happening.

Thanks, Vladimir