[PATCH 00/14] rust: example of bindings code for Rust in QEMU

[Paolo Bonzini]

Hi all,

this is an example of what some bindings code for QEMU would look like.
Note that some parts of this code are barely expected to compile, and
are probably full of bugs, but still should look like finished code
(in fact, because they compile, the type system parts should be okay;
though a conditional "should" is required).

This code is not integrated in the QEMU source tree, because again it
is just a example of what kind of Rust code would exist to handle the
C<->Rust FFI.  The translation of a handful of C structs and function
prototypes is done by hand rather than with bindgen, in particular.

The patches are organized as follows:

Patches 1-2 introduce the skeleton for the rest of the code and are
not particularly interesting, since that skeleton would be provided
by the patches that introduce Rust usage in QEMU.


Patches 3-4 define common code to handle conversion of data structures
between Rust and C.  I couldn't find an existing crate to do this,
though there are similar concepts in glib-rs.  The crate defines
variants of Clone, From and Into that convert a Rust struct to a
C pointer, for example:

   let s = "abc".clone_to_foreign();        // mallocs a NULL-terminated copy
   let p = s.as_ptr();
   drop(s);                                 // p is freed now

or the other way round:

   let s = String::cloned_from_foreign(p);  // p is a *const c_char

   let t: String = p.into_native();         // p is a *mut c_char and is free()d

The second patch defines what you need to convert strings from and to
C.  It's used by tests but also by a couple of QOM functions implemented
below; it lets you forget the differences between String, &str, CString,
&CStr and Cow<'_, str>.

This is the only complete part of the skeleton (in fact I wrote it
a couple years ago), and it comes with testcases that pass in both
"#[test]" and "doctest" (i.e. snippets integrated in the documentation
comments) styles.


Patch 5 (mostly complete except for &error_abort support and missing
tests) allows using the above functionality for the QEMU Error*.


Patches 6-8 provide an example of how to use QOM from Rust.  They
define all the functionality to:

- handle reference counting

- handle typesafe casting (i.e. code doesn't compile if an upcast
  is invalid, or if a downcast cannot possibly succeed).

- faking inheritance since Rust doesn't support it.

While I started with some implementation ideas in glib-rs, this has
evolved quite a bit and doesn't really look much like glib-rs anymore.
I provided a handful of functions to wrap C APIs.  For example:

   object_property_add_child(obj, "foo", object_new(TYPE_BLAH))

becomes

   obj.property_add_child("foo", Blah::new());

... except that the former leaks a reference and the latter does not
(gotcha :)).  The idea is that these functions would be written, in
general, as soon as Rust code uses them, but I included a few as an
example of using the various abstractions for typecasting, reference
counting, and converting from/to C data strcutures.

A large part of this code is of the "it compiles and it looks nice, it
must be perfect" kind.  Rust is _very_ picky on type safety, in case you
didn't know.  One day we're all going to be programming in Haskell,
without even noticing it.


Patches 9-10 deal with how to define new subclasses in Rust.  They are
a lot less polished and less ready.  There is probably a lot of polish
that could be applied to make the code look nicer, but I guess there is
always time to make the code look nicer until the details are sorted out.

The things that I considered here are:

- splitting configuration from runtime state.  Configuration is
  immutable throughout the lifetime of the object, and holds the
  value of user-configured properties.  State will typically be a
  Mutex<> or RefCell<> since the QOM bindings make wide use of interior
  mutability---almost all functions are declared as &self---following
  the lead of glib-rs.

- automatic generation of instance_init and instance_finalize.  For
  this I opted to introduce a new initialization step that is tailored to
  Rust, called instance_mem_init(), that is executed before the default
  value of properties is set.  This makes sure that user code only ever
  sees valid values for the whole struct, including after a downcast;
  it matters because some Rust types (notably references) cannot be
  initialized to a zero-bytes pattern.  The default implementation of
  instance_mem_init is simply a memset(), since the callback replaces the

    memset(obj, 0, type->instance_size);

  line in object_initialize_with_type().  I have prototyped this change
  in QEMU already.

- generation of C vtables from safe code that is written in Rust. I
  chose a trait that only contains associated constants as a way to
  access the vtables generically.  For example:

    impl ObjectImpl for TestDevice {
        const UNPARENT: Option<fn(&TestDevice)> = Some(TestDevice::unparent);
    }

    impl DeviceImpl for TestDevice {
        const REALIZE: Option<fn(&TestDevice) -> Result<()>> = 
Some(TestDevice::realize);
    }

  This works and it seems like a style that (in the future) we could apply
  macro or even procedural macro magic to.

- generation of qdev property tables.  While only boolean properties are
  implemented here, one idea that I experimented with, is that the
  default value of properties is derived from the ConstDefault trait.
  (ConstDefault is provided by the const_default external crate).  Again,
  this is material for future conversion to procedural macros.

I absolutely didn't look at vmstate, but it shouldn't be too different
from properties, at least for the common cases.


Patches 11-14 finally are an example of the changes that are needed
to respect a minimum supported Rust version consistent with what is in
Debian Bullseye.  It's not too bad, especially since the current version
of the QOM bindings does not require generic associated types anymore.


Why am I posting this?  Because this kind of glue code is the ultimate
source of technical debt.  It is the thing that we should be scared of
when introducing a new language in QEMU.  It makes it harder to change C
code, and it is hard to change once Rust code becomes more widespread.
If we think a C API is not fully baked, we probably shouldn't write
Rust code that uses it (including bindings code).  If we think a Rust
API is not fully baked, we probably shouldn't add too much Rust code
that uses it.

We should have an idea of what this glue code looks like, in order to make
an informed choice.  If we think we're not comfortable with reviewing it,
well, we should be ready to say so and stick with C until we are.

The alternative could be to use Rust without this kind of binding.  I
think it's a bad idea.  It removes many of the advantages of Rust 
(which are exemplified by the above object_property_add_child one-liner),
and it also introduces _new_ kinds of memory errors, since Rust has
its own undefined behavior conditions that are not there in C/C++.
For example:

    impl Struct {
        pub fn f(&self) {
            call_some_c_function(Self::g, self as *const Self as *mut _);
        }

        fn do_g(&mut self) {
            ...
        }

        extern "C" fn g(ptr: *mut Self) {
            unsafe { &mut *ptr }.do_g();
        }
    }

is invalid because a &mut reference (exclusive) is alive at the same time
as a & reference (shared).  It is left as an exercise to the reader to
figure out all the possible ways in which we can shoot our own feet,
considering the pervasive use of callbacks in QEMU.


With respect to callbacks, that's something that is missing in this
prototype.  Fortunately, that's also something that will be tackled very
soon if the PL011 example is merged, because memory regions and character
devices both introduce them.  Also, as I understand it, Rust code using
callbacks is not particularly nice anyway, though it is of course doable.
Instead, this exercise are about being able to write *nice* Rust code,
with all the advantages provided by the language, and the cost of
writing/maintaining the glue code that makes it possible.  I expect
that we'll use a technique similar to the extern_c crate (it's 20 lines of
code; https://docs.rs/crate/extern-c/) to convert something that implements
Fn(), including a member function, into an extern "C" function.


Anyhow: I think we can do it, otherwise I would not have written 2000 lines
of code (some of it two or three times).  But if people are now scared and
think we shouldn't, well, that's also a success of its own kind.

Paolo


Paolo Bonzini (14):
  add skeleton
  set expectations
  rust: define traits and pointer wrappers to convert from/to C
    representations
  rust: add tests for util::foreign
  rust: define wrappers for Error
  rust: define wrappers for basic QOM concepts
  rust: define wrappers for methods of the QOM Object class
  rust: define wrappers for methods of the QOM Device class
  rust: add idiomatic bindings to define Object subclasses
  rust: add idiomatic bindings to define Device subclasses
  rust: replace std::ffi::c_char with libc::c_char
  rust: replace c"" literals with cstr crate
  rust: introduce alternative to offset_of!
  rust: use version of toml_edit that does not require new Rust

-- 
2.45.2