qemu-devel
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH 00/14] rust: example of bindings code for Rust in QEMU


From: Pierrick Bouvier
Subject: Re: [PATCH 00/14] rust: example of bindings code for Rust in QEMU
Date: Thu, 4 Jul 2024 12:26:33 -0700
User-agent: Mozilla Thunderbird

Hi Paolo,

thanks for this series!
Some comments below.

On 7/1/24 07:58, Paolo Bonzini wrote:
Hi all,

this is an example of what some bindings code for QEMU would look like.
Note that some parts of this code are barely expected to compile, and
are probably full of bugs, but still should look like finished code
(in fact, because they compile, the type system parts should be okay;
though a conditional "should" is required).

This code is not integrated in the QEMU source tree, because again it
is just a example of what kind of Rust code would exist to handle the
C<->Rust FFI.  The translation of a handful of C structs and function
prototypes is done by hand rather than with bindgen, in particular.

The patches are organized as follows:

Patches 1-2 introduce the skeleton for the rest of the code and are
not particularly interesting, since that skeleton would be provided
by the patches that introduce Rust usage in QEMU.


Patches 3-4 define common code to handle conversion of data structures
between Rust and C.  I couldn't find an existing crate to do this,
though there are similar concepts in glib-rs.  The crate defines
variants of Clone, From and Into that convert a Rust struct to a
C pointer, for example:

    let s = "abc".clone_to_foreign();        // mallocs a NULL-terminated copy
    let p = s.as_ptr();
    drop(s);                                 // p is freed now

or the other way round:

    let s = String::cloned_from_foreign(p);  // p is a *const c_char

    let t: String = p.into_native();         // p is a *mut c_char and is 
free()d

The second patch defines what you need to convert strings from and to
C.  It's used by tests but also by a couple of QOM functions implemented
below; it lets you forget the differences between String, &str, CString,
&CStr and Cow<'_, str>.

This is the only complete part of the skeleton (in fact I wrote it
a couple years ago), and it comes with testcases that pass in both
"#[test]" and "doctest" (i.e. snippets integrated in the documentation
comments) styles.


Patch 5 (mostly complete except for &error_abort support and missing
tests) allows using the above functionality for the QEMU Error*.


Patches 6-8 provide an example of how to use QOM from Rust.  They
define all the functionality to:

- handle reference counting

- handle typesafe casting (i.e. code doesn't compile if an upcast
   is invalid, or if a downcast cannot possibly succeed).

- faking inheritance since Rust doesn't support it.

While I started with some implementation ideas in glib-rs, this has
evolved quite a bit and doesn't really look much like glib-rs anymore.
I provided a handful of functions to wrap C APIs.  For example:

    object_property_add_child(obj, "foo", object_new(TYPE_BLAH))

becomes

    obj.property_add_child("foo", Blah::new());

... except that the former leaks a reference and the latter does not
(gotcha :)).  The idea is that these functions would be written, in
general, as soon as Rust code uses them, but I included a few as an
example of using the various abstractions for typecasting, reference
counting, and converting from/to C data strcutures.

A large part of this code is of the "it compiles and it looks nice, it
must be perfect" kind.  Rust is _very_ picky on type safety, in case you
didn't know.  One day we're all going to be programming in Haskell,
without even noticing it.


Patches 9-10 deal with how to define new subclasses in Rust.  They are
a lot less polished and less ready.  There is probably a lot of polish
that could be applied to make the code look nicer, but I guess there is
always time to make the code look nicer until the details are sorted out.

The things that I considered here are:

- splitting configuration from runtime state.  Configuration is
   immutable throughout the lifetime of the object, and holds the
   value of user-configured properties.  State will typically be a
   Mutex<> or RefCell<> since the QOM bindings make wide use of interior
   mutability---almost all functions are declared as &self---following
   the lead of glib-rs.

- automatic generation of instance_init and instance_finalize.  For
   this I opted to introduce a new initialization step that is tailored to
   Rust, called instance_mem_init(), that is executed before the default
   value of properties is set.  This makes sure that user code only ever
   sees valid values for the whole struct, including after a downcast;
   it matters because some Rust types (notably references) cannot be
   initialized to a zero-bytes pattern.  The default implementation of
   instance_mem_init is simply a memset(), since the callback replaces the

     memset(obj, 0, type->instance_size);

   line in object_initialize_with_type().  I have prototyped this change
   in QEMU already.

- generation of C vtables from safe code that is written in Rust. I
   chose a trait that only contains associated constants as a way to
   access the vtables generically.  For example:

     impl ObjectImpl for TestDevice {
         const UNPARENT: Option<fn(&TestDevice)> = Some(TestDevice::unparent);
     }

     impl DeviceImpl for TestDevice {
         const REALIZE: Option<fn(&TestDevice) -> Result<()>> = 
Some(TestDevice::realize);
     }

   This works and it seems like a style that (in the future) we could apply
   macro or even procedural macro magic to.

- generation of qdev property tables.  While only boolean properties are
   implemented here, one idea that I experimented with, is that the
   default value of properties is derived from the ConstDefault trait.
   (ConstDefault is provided by the const_default external crate).  Again,
   this is material for future conversion to procedural macros.

I absolutely didn't look at vmstate, but it shouldn't be too different
from properties, at least for the common cases.


Patches 11-14 finally are an example of the changes that are needed
to respect a minimum supported Rust version consistent with what is in
Debian Bullseye.  It's not too bad, especially since the current version
of the QOM bindings does not require generic associated types anymore.


Why am I posting this?  Because this kind of glue code is the ultimate
source of technical debt.  It is the thing that we should be scared of
when introducing a new language in QEMU.  It makes it harder to change C
code, and it is hard to change once Rust code becomes more widespread.
If we think a C API is not fully baked, we probably shouldn't write
Rust code that uses it (including bindings code).  If we think a Rust
API is not fully baked, we probably shouldn't add too much Rust code
that uses it.


Shouldn't we focus more on how we want to write a QOM device in Rust instead, by making abstraction of existing C implementation first? Once we have satisfying idea, we could evaluate how it maps to existing implementation, and which compromise should be made for efficiency.

It seems that the current approach is to stick to existing model, and derive Rust bindings from this.

I agree this glue can be a source of technical debt, but on the other side, it should be easy to refactor it, if we decided first what the "clean and idiomatic" Rust API should look like.

A compromise where you have to manually translate some structs, or copy memory to traverse languages borders at some point, could be worth the safety and expressiveness.

We should have an idea of what this glue code looks like, in order to make
an informed choice.  If we think we're not comfortable with reviewing it,
well, we should be ready to say so and stick with C until we are.


While it is important that this glue code is maintainable and looks great, I don't think it should be the main reason to accept usage of a new language.

The alternative could be to use Rust without this kind of binding.  I
think it's a bad idea.  It removes many of the advantages of Rust
(which are exemplified by the above object_property_add_child one-liner),
and it also introduces _new_ kinds of memory errors, since Rust has
its own undefined behavior conditions that are not there in C/C++.
For example:

     impl Struct {
         pub fn f(&self) {
             call_some_c_function(Self::g, self as *const Self as *mut _);
         }

         fn do_g(&mut self) {
             ...
         }

         extern "C" fn g(ptr: *mut Self) {
             unsafe { &mut *ptr }.do_g();
         }
     }

is invalid because a &mut reference (exclusive) is alive at the same time
as a & reference (shared).  It is left as an exercise to the reader to
figure out all the possible ways in which we can shoot our own feet,
considering the pervasive use of callbacks in QEMU.


I agree with this point that you made on the original RFC series from Manos. Just calling qemu C API through unsafe code is not making the best possible usage of Rust.


With respect to callbacks, that's something that is missing in this
prototype.  Fortunately, that's also something that will be tackled very
soon if the PL011 example is merged, because memory regions and character
devices both introduce them.  Also, as I understand it, Rust code using
callbacks is not particularly nice anyway, though it is of course doable.
Instead, this exercise are about being able to write *nice* Rust code,
with all the advantages provided by the language, and the cost of
writing/maintaining the glue code that makes it possible.  I expect
that we'll use a technique similar to the extern_c crate (it's 20 lines of
code; https://docs.rs/crate/extern-c/) to convert something that implements
Fn(), including a member function, into an extern "C" function.


It could be an interesting thing to explore. Is the best solution to use specific traits (no callbacks involved from Rust code), or should we have Rust closures that mimic this behavior.

We could have a specific trait with functions to handle various kind of events. And the glue code could be responsible to translate this to callbacks for the C side (and calling Rust code accordingly, eventually serializing this on a single thread to avoid any race issues).


Anyhow: I think we can do it, otherwise I would not have written 2000 lines
of code (some of it two or three times).  But if people are now scared and
think we shouldn't, well, that's also a success of its own kind.

Paolo


Paolo Bonzini (14):
   add skeleton
   set expectations
   rust: define traits and pointer wrappers to convert from/to C
     representations
   rust: add tests for util::foreign
   rust: define wrappers for Error
   rust: define wrappers for basic QOM concepts
   rust: define wrappers for methods of the QOM Object class
   rust: define wrappers for methods of the QOM Device class
   rust: add idiomatic bindings to define Object subclasses
   rust: add idiomatic bindings to define Device subclasses
   rust: replace std::ffi::c_char with libc::c_char
   rust: replace c"" literals with cstr crate
   rust: introduce alternative to offset_of!
   rust: use version of toml_edit that does not require new Rust




reply via email to

[Prev in Thread] Current Thread [Next in Thread]