Re: [Libreboot-dev] Suggestions and proposal for easier Libreboot development

[Francis Rowe]

On 20/04/16 23:21, Paul Kocialkowski wrote:
> Hi,
>
> Le mercredi 20 avril 2016 à 21:55 +0100, Ministry of Freedom a écrit :
>> Op 20/04/16 om 17:19 schreef Paul Kocialkowski:
>>> After spending some hours updating C201 support in Libreboot, here
>>> are some suggestions I'd like to submit to make Libreboot
>>> development easier and more comfortable.
>>>
>>> First, there were talks about moving the build system to something
>>> other than scripts. Is it still planned? What is the new
>>> alternative going to be? I would strongly urge you to keep a
>>> script-based approach. It's a perfect fit for such a project and a
>>> Makefile-based approach should like a train-wreck for this task, if
>>> doable at all.
>> the current build system is fine and will be kept, but we want to
>> start using Makefiles around that. see libreboot.org/gnu
> I could start looking into something Makefile-based, too. It would be nice for
> handling dependencies but might also cause a lot of duplication, where scripts
> are much more flexible. I'm not saying Makefiles are necessarily a bad idea, 
> but
> it'll be much trickier to get it right.
>
>>> Regarding the current script-based build system, I noticed how
>>> multiple versions of coreboot are handled, to have a different
>>> revision per payload-device association. I think it's a very nice
>>> thing to have, but the way it's done current seems quite painful to
>>> me. Instead of duplicating the tree, I'd suggest using git and
>>> branches, for each devices (even ones that don't need it).
>>>
>> impossible, because of blobs in coreboot git history. this is why we
>> create directories for each revision of coreboot used, and then run
>> git-init again and apply patches for each board, making a branch per
>> board per revision, with patches applied per branch(board)
> I don't see what the problem is with blobs in the history. As long as blobs 
> are
> removed from the tree and the git history is not shipped with source code
> releases, there should be no problem.

There is a problem when the release archives include coreboot already,
and need to be able to be built without any .git directories in the
release archives.

>>> Also, the various scripts would be much easier to handle with some
>>> included library, instead of duplicating lots of aspects in each
>>> script, making it hard to maintain each and creating disparity
>>> between projects.
>>>
>>> For instance, a library that handles generic project-related
>>> aspects for all the other scripts would be very welcome. This would
>>> also make Libreboot more consistent. Such functions could handle: *
>>> Cloning a particular project from given URLs (with a primary URL
>>> and mirrors)
>> This would be nice
>>
>>> * Creating a branch for each payload-device pair and reverting it
>>> to the appropriate revision. Similarly, having a branch for
>>> "modules", used for building host tools (crossgcc, cbfstool in
>>> coreboot, flashrom, bucts, etc). * Patching each branch with the
>>> appropriate patches, both with patches common to the payload and
>>> specific to the device. * Releasing each project's source code for
>>> each payload-device pair, making a copy for each and removing the
>>> git history.
>>>
>> This needs further thought.
>>
>> I agree with the overall premise; the current build system is very
>> inflexible.
>>
>>> This way, all downloaded projects would only have one copy and a
>>> branch for each needed source tree. For coreboot, it would imply
>>> keeping the git history. Removing it is very inconvenient when
>>> developing and there is about no need to do it prior to releasing.
>>>
>>> In order to make everything even more simply, I think it would be
>>> best to switch to a project-based structure, where each project
>>> simply defines a few functions with given names, such as
>>> "download", "build", with relevant parameters. The global scripts
>>> would simply include the appropriate scripts and call the functions
>>> with the right arguments. This would severely reduce the
>>> dissipation of files for a particular project. Each file would
>>> actually be quite simple thanks to the use of a library for common
>>> operations.
>> Could you whip up a basic proof of concept for all of this? Even just
>> diagrams/concepts are fine, I find it hard to understand text because
>> we're talking about massive changes to the build system.
> Of course, I'll let you know when it does something significant. I'll target
> building images for veyron_speedy with it.

I suggest holding off for now. The priority is a stable release. We need
to also finish the talks listed on https://libreboot.org/gnu/

Can you work on that?

>>> Projects could either be tools (flashrom, bucts), the bootup
>>> software (coreboot) or a payload (GRUB, depthcharge). Since one
>>> might surely depend on the other, each project's script would have
>>> a "depends" function calling other scripts and building the
>>> required components.
>>
>>
>>> Regarding the naming of projects, I find some to be quite
>>> confusing. Libreboot sometimes refers to the whole build system or
>>> to the deblobbed coreboot, which is sometimes also called
>>> "libreboot-libre". This kind of inconsistency makes it hard to
>>> figure out exactly which component does what.
>>>
>>> Overall, here is a suggested directory structure example matching
>>> these changes: ressources/projects/coreboot/libreboot (script with
>>> the functions) 
>>> ressources/projects/coreboot/config/depthcharge/veyron_speedy/config
>>> (coreboot configuration) 
>>> ressources/projects/coreboot/config/depthcharge/veyron_speedy/revision
>>> (coreboot revision) 
>>> ressources/projects/coreboot/patches/depthcharge/0001-foo.patch 
>>> ressources/projects/coreboot/patches/depthcharge/veyron_speedy/0001-ba
>> r.patch
>>>
>> ressources/projects/vboot/libreboot
>>> ressources/projects/vboot/config/depthcharge/veyron_speedy/revision
>>>
>>>
>> ressources/libs/git (git-related library)
>>> ressources/libs/common (common operations library)
>>>
>> I think rather, we just need better documentation for maintenance.
>> Having a more logical structure makes more sense. I've considered
>> something like what you're proposing.
>>
>> We're focussing on a release at the moment. Organisational/structural
>> changes to the project are not on the table at the moment, but we'll
>> revisit this discussion thread later.
> Of course, I'm not asking that you or anyone else spend their time on it. I'd 
> be
> happy to contribute to it myself to the extent I can. I really care about
> Libreboot and see the current build system as somewhat inconvenient. Thus the
> naturel thing to do from my perspective is to try and improve it. Hopefully,
> others will also agree that it's an improvement.
>
>>> Also, it would be best to build each project out-of-tree, or to
>>> ensure that the tree is cleaned between different builds (something
>>> that could be done in a clean() step, either called before each
>>> build() for in-tree build or skipped for out-of tree build).
>>>
>>> This is really just a mockup at this point, but if you don't hate
>>> the idea or see a reason not to do this, I'll probably be spending
>>> some time working in that direction, because I find the Libreboot
>>> build system quite painful to deal with as it is now and it won't
>>> scale well for more projects anyway.
>>>
>>> Cheers,
>>>
>> See above. You hit the nail on the head; it's a mockup. Let's get a
>> new stable release done first, before thinking about changing the
>> entire structure of the libreboot project.
> Well, things are ready on my side for the release (veyron_speedy-wise), so 
> I'll
> work on that build system change on my own for now and ping you back about it
> when the release is out of the way.
>
>> (this response is hastily written, because paulk pinged me on IRC and
>> told me to respond quickly)
> Thanks for sharing your thoughts under such short notice, it is greatly
> appreciated.
>