Re: [Qemu-devel] Re: [PATCH][v2] Align file accesses with cache=off (O_DIRECT)

[Avi Kivity]

Anthony Liguori wrote:
> Avi Kivity wrote:
> > Anthony Liguori wrote:
> > > "cached" is not a terribly accurate term.  O_DIRECT avoids the host 
> > > page cache but it doesn't guarantee that the disk is using 
> > > write-through.  For that, you need to use hdparm.
> > >
> > > O_SYNC basically turns the host page cache into a write-through 
> > > cache.  In terms of data integrity, the only question that matters 
> > > is whether you're misleading the guest into thinking data is on the 
> > > disk when it isn't.  Both O_DIRECT and O_SYNC accomplish this.
> > >
> > > If you just are concerned with data integrity, O_SYNC is probably 
> > > better because you get the benefits of host caching.  O_DIRECT is 
> > > really for circumstances where you know that using the host page 
> > > cache is going to reduce performance.
> >
> > In one specific circumstance O_SYNC has data integrity problems: 
> > shared disks with guests running on different hosts (or even a guest 
> > on one host, sharing a disk with another host).  In these cases, two 
> > reads can return different values without an intervening write.
>
> Are you assuming the underlying disk sharing protocol does not keep 
> the page cache coherent?

I was assuming access to a raw partition.  But yes, with a cluster file 
system this objection goes away.

At a significant cost, though.  You're now running two cache coherency 
protocols on top of each other.

> > In the general case, O_DIRECT gives better performance.  It avoids 
> > copying from the host pagecache to guest memory, and if you have 
> > spare memory to benefit from caching, give it to the guest; the 
> > nearer to the data consumer the cache is, the faster it performs.
>
> This assumes the only thing you're running on the machine is VMs.  If 
> you're just running one VM on your desktop, it is probably preferable 
> to go through the host page cache.  In particular, the host page cache 
> can be automatically aged and adjusted whereas, in general, you cannot 
> reduce the guests page cache size.

Agreed.  For casual uses, O_DIRECT is overkill.  It does get rid of the 
data copies, though.

> > In one specific case O_SYNC (or regular read/write cached operation) 
> > is better, rebooting the same guest over and over, as the guest cache 
> > is flushed on reboot.  Not a very interesting case, but unfortunately 
> > one that is very visible.
> >
> > (if you have a backing file for a COW disk, then opening the backing 
> > file without O_DIRECT may be a good idea too, as the file can be 
> > shared among many guests).
>
> FWIW, we really only need to use O_SYNC when the guest has disabled 
> write-back.  I think we should do that unconditionally too as it's an 
> issue of correctness. 

If the guest is used for non critical applications (like testing distro 
installers), then it's just a slowdown.

Even if the guest did not disable disk writeback, pagecache and disk 
write caches have vastly different characteristics, so I think we should 
set O_SYNC there as well.

Here's a summary of the use cases I saw so far:

- casual use, no critical data: write back cache

- backing file shared among many guests: read-only, cached

- desktop system, but don't lose my data: O_SYNC
(significant resources on the host)

- dedicated virtualization engine: O_DIRECT
(most host resources assigned to guests)

--
error compiling committee.c: too many arguments to function