On Mon, Jun 13, 2022 at 03:28:34PM +0530, manish.mishra wrote:
On 26/05/22 8:21 am, Jason Wang wrote:
On Wed, May 25, 2022 at 11:56 PM Peter Xu <peterx@redhat.com> wrote:
On Wed, May 25, 2022 at 11:38:26PM +0800, Hyman Huang wrote:
2. Also this algorithm only control or limits dirty rate by guest
writes. There can be some memory dirtying done by virtio based devices
which is accounted only at qemu level so may not be accounted through
dirty rings so do we have plan for that in future? Those are not issue
for auto-converge as it slows full VM but dirty rate limit only slows
guest writes.
From the migration point of view, time spent on migrating memory is far
greater than migrating devices emulated by qemu. I think we can do that when
migrating device costs the same magnitude time as migrating memory.
As to auto-converge, it throttle vcpu by kicking it and force it to sleep
periodically. The two seems has no much difference from the perspective of
internal method but the auto-converge is kind of "offensive" when doing
restraint. I'll read the auto-converge implementation code and figure out
the problem you point out.
This seems to be not virtio-specific, but can be applied to any device DMA
writting to guest mem (if not including vfio). But indeed virtio can be
normally faster.
I'm also curious how fast a device DMA could dirty memories. This could be
a question to answer to all vcpu-based throttling approaches (including the
quota based approach that was proposed on KVM list). Maybe for kernel
virtio drivers we can have some easier estimation?
As you said below, it really depends on the speed of the backend.
My guess is it'll be
much harder for DPDK-in-guest (aka userspace drivers) because IIUC that
could use a large chunk of guest mem.
Probably, for vhost-user backend, it could be ~20Mpps or even higher.
Sorry for late response on this. We did experiment with IO on virtio-scsi based
disk.
Thanks for trying this and sharing it out.
We could see dirty rate of ~500MBps on my system and most of that was not
tracked
as kvm_dirty_log. Also for reference i am attaching test we used to avoid
tacking
in KVM. (as attached file).
The number looks sane as it seems to be the sequential bandwidth for a
disk, though I'm not 100% sure it'll work as expected since you mmap()ed
the region with private pages rather than shared, so after you did I'm
wondering whether below will happen (also based on the fact that you mapped
twice the size of guest mem as you mentioned in the comment):
(1) Swap out will start to trigger after you read a lot of data into the
mem already, then old-read pages will be swapped out to disk (and
hopefully the swap device does not reside on the same virtio-scsi
disk or it'll be even more complicated scenario of mixture IOs..),
meanwhile when you finish reading a round and start to read from
offset 0 swap-in will start to happen too. Swapping can slow down
things already, and I'm wondering whether the 500MB/s was really
caused by the swapout rather than backend disk reads. More below.
(2) Another attribute of private pages AFAICT is after you read it once
it does not need to be read again from the virtio-scsi disks. In
other words, I'm thinking whether starting from the 2nd iteration
your program won't trigger any DMA at all but purely torturing the
swap device.
Maybe changing MAP_PRIVATE to MAP_SHARED can emulate better on what we want
to measure, but I'm also not 100% sure on whether it could be accurate..
Thanks,
Thanks
[copy Jason too]
--
Peter Xu
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/time.h>
#include <time.h>
#include <unistd.h>
#define PAGE_SIZE 4096
#define GB (1024 * 1024 * 1024)
int main()
{
char *buff;
size_t size;
struct stat stat;
// Take file of size atleast double of RAM size to
// achieve max dirty rate possible.
const char * file_name = "file_10_gb";
int fd;
size_t i = 0, count = 0;
struct timespec ts1, ts0;
double time_diff;
fd = open(file_name, O_RDONLY);
if (fd == -1) {
perror("Error opening file");
exit(1);
}
fstat (fd, &stat);
size = stat.st_size;
printf("File size %ld\n", (long)size);
buff = (char *)mmap(0, size, PROT_READ, MAP_PRIVATE, fd, 0);
if (buff == MAP_FAILED) {
perror("Mmap Error");
exit(1);
}
(void)clock_gettime(CLOCK_MONOTONIC, &ts0);
while(1) {
char c;
i = (i + PAGE_SIZE) % size;
c = buff[i];
count++;
// Check on every 10K pages for rate.
if (count % 10000 == 0) {
(void)clock_gettime(CLOCK_MONOTONIC, &ts1);
time_diff = ((double)ts1.tv_sec + ts1.tv_nsec * 1.0e-9)
-((double)ts0.tv_sec + ts0.tv_nsec * 1.0e-9);
printf("Expected Dirty rate %f\n", (10000.0 * PAGE_SIZE) / GB /
time_diff);
ts0 = ts1;
}
}
close(fd);
return 0;
}