-----
This means that the test executed a write at LBA 0x94fa and, after
confirming that the write was completed, issue 2 reads in the same LBA to
assert the written contents and found out a mismatch.
I've tested all sort of configurations between disk vs LUN, cache modes and
AIO. My findings are:
- using device='lun' instead of device='disk', I can't reproduce the issue
doesn't matter what other configurations are;
- using device='disk' but with cache='writethrough', issue doesn't happen
(haven't checked other cache modes);
- using device='disk', cache='none' and io='native', issue doesn't happen.
The issue seems to be tied with the combination device=disk + cache=none +
io=threads. I've started digging into the SCSI layer all the way down to the
block backend. With a shameful amount of logs I've discovered that, in the
write that the test finds a miscompare, in block/file-posix.c:
- when doing the write, handle_aiocb_rw_vector() returns success, pwritev()
reports that all bytes were written
- in both reads after the write, handle_aiocb_rw_vector returns success, all
bytes read by preadv(). In both reads, the data read is different from the
data written by the pwritev() that happened before
In the discussions at [1], Fam Zheng suggested a test in which we would take
down the number of threads created in the POSIX thread pool from 64 to 1.
The idea is to ensure that we're using the same thread to write and read.
There was a suspicion that the kernel can't guarantee data coherency between
different threads, even if using the same fd, when using pwritev() and
preadv(). This would explain why the following reads in the same fd would
fail to retrieve the same data that was written before. After doing this
modification, the miscompare didn't reproduce.
After reverting the thread pool number change, I've made a couple of
attempts trying to flush before read() and flushing after write(). Both
attempts failed - the miscompare appears in both scenarios. This enforces
the suspicion we have above - if data coherency can't be granted between
different threads, flushing in different threads wouldn't make a difference
too. I've also tested a suggestion from Fam where I started the disks with
"cache.direct=on,cache.no-flush=off" - bug still reproduces.
This is the current status of this investigation. I decided to start a
discussion here, see if someone can point me something that I overlooked or
got it wrong, before I started changing the POSIX thread pool behavior to
see if I can enforce one specific POSIX thread to do a read() if we had a
write() done in the same fd. Any suggestions?
ps: it is worth mentioning that I was able to reproduce this same bug in a
POWER8 system running Ubuntu 18.04. Given that the code we're dealing with
doesn't have any arch-specific behavior I wouldn't be surprised if this bug
is also reproducible in other archs like x86.
Thanks,
Daniel
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1561017
[2] https://github.com/open-power/HTX