Impact of Ozone Manager HA on performance

[myskov]

Hi devs,
Our team is investigating Ozone performance improvements, and we've identified OM HA as a potential bottleneck.
Here are the results from our performance testing:
Writing small files per second with OM HA (~3000 files per second, single bucket):

Writing small files per second with OM HA (~3000 files per second, multiple buckets):

Writing small files per second without OM HA (~6000 files per second, single bucket):

Questions:

1. Do you see similar performance impacts in your test runs?
2. Do you have any suggestions for overcoming this bottleneck? Would tweaking Ratis parameters help?

[kerneltime]

@myskov When HA is turned on, the data is replicated across leaders and followers, and this entails additional round trips on the network, which adds to latency per request and, consequently, a drop in performance. This is a laws of physics limitation..

[kerneltime]

@myskov When HA is turned on, the data is replicated across leaders and followers, and this entails additional round trips on the network, which adds to latency per request and, consequently, a drop in performance. This is a laws of physics limitation..

When HA is off the data is only present on a single node. HA stands for high availability which comes from using Ratis to replicate data.

[szetszwo]

@myskov , It is great that you have benchmarked the performance difference between HA and non-HA! As @kerneltime mentioned, it is expected to have a performance drop. The benchmark is very useful for maintaining the performance and seeking new improvements.

[myskov]

@szetszwo @kerneltime
Yes, it's expected to have a performance drop when using HA mode. I was wondering if you have any recommended Ratis configurations (based on internal testing or customer deployments) that could help minimize this drop?

[myskov]

I updated the original post with one more case - writing small files into many buckets (the performance is the same as with single bucket).

[ChenSammi]

As there is only one raft group for OM HA, it's an expected behavior that write to multiple buckets/volumes will have the same performance as single bucket, but read will have performance advantage when multiple buckets are used.

My colleague Sumit is working on this new flow of OM HA, #7583, which will have better OM read/write IOPS.