[Qemu-devel] Re: Release of COREMU, a scalable and portable full-system emulator

[Jan Kiszka]

Chen Yufei wrote:
> On 2010-7-22, at 上午1:04, Stefan Weil wrote:
> 
>> Am 21.07.2010 09:03, schrieb Chen Yufei:
>>> On 2010-7-21, at 上午5:43, Blue Swirl wrote:
>>>
>>>   
>>>> On Sat, Jul 17, 2010 at 10:27 AM, Chen Yufei<address@hidden>  wrote:
>>>>     
>>>>> We are pleased to announce COREMU, which is a "multicore-on-multicore" 
>>>>> full-system emulator built on Qemu. (Simply speaking, we made Qemu 
>>>>> parallel.)
>>>>>
>>>>> The project web page is located at:
>>>>> http://ppi.fudan.edu.cn/coremu
>>>>>
>>>>> You can also download the source code, images for playing on sourceforge
>>>>> http://sf.net/p/coremu
>>>>>
>>>>> COREMU is composed of
>>>>> 1. a parallel emulation library
>>>>> 2. a set of patches to qemu
>>>>> (We worked on the master branch, commit 
>>>>> 54d7cf136f040713095cbc064f62d753bff6f9d2)
>>>>>
>>>>> It currently supports full-system emulation of x64 and ARM MPcore 
>>>>> platforms.
>>>>>
>>>>> By leveraging the underlying multicore resources, it can emulate up to 
>>>>> 255 cores running commodity operating systems (even on a 4-core machine).
>>>>>
>>>>> Enjoy,
>>>>>       
>>>> Nice work. Do you plan to submit the improvements back to upstream QEMU?
>>>>     
>>> It would be great if we can submit our code to QEMU, but we do not know the 
>>> process.
>>> Would you please give us some instructions?
>>>
>>> --
>>> Best regards,
>>> Chen Yufei
>>>   
>> Some hints can be found here:
>> http://wiki.qemu.org/Contribute/StartHere
>>
>> Kind regards,
>> Stefan Weil
> 
> The patch is in the attachment, produced with command
> git diff 54d7cf136f040713095cbc064f62d753bff6f9d2
> 
> In order to separate what need to be done to make QEMU parallel, we created a 
> separate library, and the patched QEMU need to be compiled and linked with 
> that library. To submit our enhancement to QEMU, maybe we need to incorporate 
> this library into QEMU. I don't know what would be the best solution.

For upstream QEMU, the goal should be to integrate your modifications
and enhancements into the existing architecture in a mostly seamless
way. The library approach may help maintaining your changes out of tree,
but it likely cannot contribute any benefit to an in-tree extension of
QEMU for parallel TCG VCPUs.

> 
> Our approach to make QEMU parallel can be found at 
> http://ppi.fudan.edu.cn/coremu
> 
> I will give a short summary here:
> 
> 1. Each emulated core thread runs a separate binary translator engine and has 
> private code cache. We marked some variables in TCG as thread local. We also 
> modified the TB invalidation mechanism.
> 
> 2. Each core has a queue holding pending interrupts. The COREMU library 
> provides this queue, and interrupt notification is done by sending realtime 
> signals to the emulated core thread.
> 
> 3. Atomic instruction emulation has to be modified for parallel emulation. We 
> use lightweight memory transaction which requires only compare-and-swap 
> instruction to emulate atomic instruction.
> 
> 4. Some code in the original QEMU may cause data race bug after we make it 
> parallel. We fixed these problems.
> 

Upstream integration requires such iterative steps as well - in form of
ideally small, focused patches that finally convert QEMU into a parallel
emulator.

Also note that upstream already supports threaded VCPUs - in KVM mode.
You obviously have resolved the major blocking points to apply this on
TCG mode as well. But I don't see yet why we may need a new VCPU
threading infrastructure for this. Rather only small tuning of what KVM
already uses should suffice - if that's required at all.

To give it a start, you could identify some more trivial changes in your
patches, split them out and rebase them over latest qemu.git, then post
them as a patch series for inclusion (see the mailing list for various
examples). Make sure to describe the reason for your changes as clear as
possible, specifically if they are not (yet) obvious in the absence of
COREMU features in upstream QEMU.

Be prepared that merging your code can be a lengthy process with quite a
few discussions about why and how things are done, likely also with
requests to change your current solution in some aspects. However, the
result should be an optimal solution for the overall goal, parallel VCPU
emulation - and no longer any need to maintain your private set of
patches against quickly evolving QEMU.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux