Use a rwlock to have MP (multiple producers) access to a Ring?

The use-case is multiple threads trying to submit operations to an io_uring SQ ring, using a rwlock, a local sq_tail to be synchronized with the tail shared with the kernel (sq_ktail).

Maybe we can submit to the SQ ring from multiple threads with reduced contentions using a rwlock instead of a mutex. The read lock would protect incrementing the local sq_tail (updated with a CAS) and preparing the SQE, while the write lock would protect updates to the sq_ktail shared with the kernel (updated with a STORE). That would allow threads to prepare SQE in the SQ ring with lazy submissions, which should happen at worst when the SQ ring is full (in which case it creates a contention point).

loop do
  @rwlock.lock_read
  tail = @sq_tail.get(:relaxed)
  head = @sq_khead.get(:relaxed)
  size = tail &- head

  if (count < tail &- head)
    if @sq_tail.cas(tail, tail + count, :acquire_release, :relaxed)
      yield SubRing.new(@sqes, @sq_mask.value, tail, count)
      @rwlock.unlock_read
      @rwlock.try_lock_write(lazy: true) { submit } # <= may submit
      return
    end
    @rwlock.unlock_read
  else
    @rwlock.unlock_read
    @rwlock.try_lock_write { submit(wait: true) } # <= must submit
  end
end

NOTES:

The above algorithm may not be correct (no proof) 🤡
The may submit can abort early if there is any locked reader because that reader will try to lock write after it unlocks read and thus be the one to submit; the must submit however can only abort if another thread has the write lock, and that thread must wait for all readers to unlock.
We could only lock read before the CAS, but then threads would busy loop when the ring is full, instead of efficiently waiting on the lock.
On low activity, a few parallel submissions should process smoothly and never get blocked, granted that there is enough space in the rings to fit at least max_sqe_chain_size * num_threads * arbitrary_multiple SQEs.
On high activity, parallel submissions should start filling the rings that would only be submitted when that happens, creating delays and a contention point because we need the write lock to sync sq_tail to sq_ktail, and SQPOLL wouldn't even help 😞
We don't have to wait for the queue to be filled to force a submit, we could "must submit" at 25%... but that would still prevent other threads from adding entries to the SQ ring (must lock write).
Maybe a write biaised rwlock (instead of the usual bias on read) might be a better choice, but it would stil have an impact on submissions, blocking other threads from submitting...

Conclusion

The idea ain't bad, but it poorly applies to a case where we want nearly as many writes as reads.

ysbaddaden/rwlock_ring.md

Use a rwlock to have MP (multiple producers) access to a Ring?