Add conditional writes

[willglynn]

Amazon S3 now supports conditional writes (blog post). Docs indicate that s3:PutObject and s3:CompleteMultipartUpload now support the request header If-None-Match: *, in which case S3 will:

1. return a 200 if it created the object did not exist and was created as requested,
2. return a 412 Precondition Failed if the object already exists, or
3. return a 409 Conflict if the object is deleted before creation completes

This upstream change updates #218.

[willglynn]

S3 expanded conditional writes to include compare-and-swap (blog post). Docs describe both conditionals together. Conditional overwrites are implemented by If-Match: <expected ETag>, e.g. on s3:PutObject.

[chailuecha]

Just would like to confirm. The ticket is open. So CEPH doesn't support conditional put via S3 API?
Let me know if CEPH support conditional put in other form/mechanism.

[willglynn]

Ceph supports conditional puts via If-Match: and If-None-Match:: https://tracker.ceph.com/issues/8562

This functionality isn't quite tested here. There's no If-None-Match: test of any kind, and while there is an If-Match: test exercising conditional writes, it's annotated with @pytest.mark.fails_on_aws. I opened this issue to point out that S3's upstream behavior has changed.

[chailuecha]

Thanks a ton for the explanation. I tried header f-None-Match with aws-sdk-1.11 (yep, its very old) to my CEPH installation. It works great.

[willglynn]

PRs #663, #673, #682 will add relevant test coverage, though I haven't reviewed them closely enough to decide whether coverage is total relative to S3.

[scheuref]

So Ceph RGW S3 is supporting the request headers If-Match and If-None-Match for GetObject
(cf. https://docs.ceph.com/en/latest/radosgw/s3/objectops/#get-object)
and apparently also for PutObject (but this is missing in documentation),
unsure if CompleteMultipartUpload and CopyObject also supports it.
And Ceph also supports the request headers x-amz-copy-if-match and x-amz-copy-if-none-match for CopyObject
(cf. https://docs.ceph.com/en/latest/radosgw/s3/objectops/#copy-object).
AFAIK none of these headers can be enforced in the bucket policy.

In contrast, AWS S3 supports If-Match and If-None-Match for the following operations: PutObject, CompleteMultipartUpload and CopyObject
(cf. https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes.html#conditional-write-key-names).
AWS also supports advanced bucket policy conditions for enforcing these headers, such as:
"Condition": {"Null": {"s3:if-none-match": "false"}, "StringEquals": {"s3:if-none-match": "*"} }
"Condition": {"Bool": {"s3:ObjectCreationOperation": "false"} }
These can be very useful for preventing overwrites and achieving immutability (similar to WORM buckets).
(cf. https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-writes-enforce.html#conditional-writes-enforce-ex1).

It would be great if Ceph could achieve the same.