get()/put() serialization

[Divneil Rai Wadhawan (divneil-intel)]

Based on the understanding from discussion #3926, I am looking to confirm the following as I got into more doubts.

Let's say I have the following:
o Key1 as K1 in KVStore1 as KV1
o Key2 as K2 in KVStore1 as KV1

Case 1

So, seems like following code will break the serialization if getting executed in parallel as the root of the transaction is same for both and one of them will get rejected.

EDIT1: Seems like it won't break serialization as the Keys are different

Thread1 (Key1)

kv_handle = ctx.tx.rw(KV1)
record = kv_handle->get_globally_committed(K1) (<-- must be replaced by get())
value = record.value()

// Update the value
kv_handle->put(K1, value)

Thread2 (Key2)

kv_handle = ctx.tx.rw(KV1)
record = kv_handle->get_globally_committed(K2) (<-- must be replaced by get())
value = record.value()

// Update the value
kv_handle->put(K2, value)

Case 2

Same result as above, as there's no surety that consensus has committed the transaction

EDIT1: Not same as above, but will result in a bad state of KV

for (some condition)
{
  kv_handle = ctx.tx.rw(KV1)
  record = kv_handle->get_globally_committed(K1) (<-- must be replaced by get())
  value = record.value()

  // Update the value
  kv_handle->put(K1, value)
}

[Eddy Ashton (eddyashton)]

Divneil Rai Wadhawan (@divneil-intel) Yes, as Amaury Chamayou (@achamayou) said in #3926:

Note that using get_globally_committed() in the context of a write transaction is dangerous and will potentially lead to a serializability break.

If you use get() and put() in the same transaction, then CCF will detect any conflicts with concurrently executing transactions (ie - those that have completed in the gap between this transaction selecting a read version and it completing), and automatically rerun the transaction (reading newer state). get_globally_committed() can be used to deliberately circumvent this flow to report only committed state, but reads of that committed state will not be treated as conflicts, so if they influence writes you may get surprising results.

When a transaction begins, it is assigned 2 distinct versions to read from: the first is the most recent state, that all get()s will operate over, and the second is the current committed state, which may be arbitrarily stale, but is guaranteed to be persisted. I think the simplest example is 2 transaction like the following:

op_write = {
  kv.rw("table1").put("foo", params["value"]);
}

op_copy = {
  v = kv.rw("table1").get("foo");
  kv.rw("table2").put("previous_foo", v);
}

The intent is that you could run a sequence of operations like op_write("bar"), op_copy(), op_write("baz"), op_copy(), and be guaranteed that if you looked at the resulting sequence of writes, each change to table2.previous_foo was the most recent value from table1.foo. With get(), that is the case. If an op_copy is executing and reading version 5, at which table1.foo is "bar", and during that execution another op_write executes and completes at version 6, then when the op_copy completes (trying to write the 'wrong' value to table2.previous_foo) CCF will detect a conflict and re-execute the op_copy. If however the get in op_copy is replaced with get_globally_committed, this does not hold! It is possible the commit is still at version 2, missing multiple intervening op_write values. These semantics are well-defined, but probably not what is intended in this case.

[Divneil Rai Wadhawan (divneil-intel)]

Thanks Eddy Ashton (@eddyashton) for the detailed reply.
As I understand now, get()s/put()s in multiple transactions generating the conflicts are managed by rebasing the offending transaction.

How does it gets managed when the primary node forwards the transaction to the other node? Is there a handling to be done in application or the transaction results will start failing indicating consensus failed.

[Eddy Ashton (eddyashton)]

Divneil Rai Wadhawan (@divneil-intel) Transactions are not forwarded from the primary to other node, rather they are forwarded the other way round from backups to the primary if they contain writes. Regardless there shouldn't need to be any handling of this in the application. If you submit a series of transactions over the same session and then wait for the final one to be committed, that implies the commit of all earlier transactions, regardless of whether they were executed on the node you first spoke to or forwarded to the primary. For more precise control, or to detect progress when these transactions do not reach consensus (ie - they are accepted by one node, but that is partitioned and unable to commit the transaction), client applications can poll for status of individual TxIDs.