Re: [RFC PATCH 2/2] GitLab CI: crude mapping of PMM's scripts to jobs

[Thomas Huth]

On 06/02/2020 14.54, Philippe Mathieu-Daudé wrote:
> On 2/6/20 2:52 PM, Wainer dos Santos Moschetta wrote:
>> On 2/6/20 11:03 AM, Philippe Mathieu-Daudé wrote:
>>> On 2/3/20 4:23 AM, Cleber Rosa wrote:
>>>> This is a crude and straightforward mapping of Peter's
>>>> "remake-merge-builds" and "pull-buildtest" scripts.
>>>>
>>>> Some characteristics were removed for simplicity sake (but eventually
>>>> will), including:
>>>>   * number of simultaneous make jobs
>>>>   * make's synchronous output, not needed because of previous point
>>>>   * out-of-tree builds
>>>>
>>>> This covers the "x86-64 Linux with a variety of different build
>>>> configs"[1].  I've personally tested all of them, and only had
>>>> issues with the "notcg" job[2], but it seems to be a test specific
>>>> issue with the nested KVM I was using.
>>>>
>>>> [1] - https://wiki.qemu.org/Requirements/GatingCI#Current_Tests
>>>> [2] - https://paste.centos.org/view/1dd43a1c
>>>>
>>>> Signed-off-by: Cleber Rosa <address@hidden>
>>>> ---
>>>>   .gitlab-ci.yml | 116
>>>> +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>   1 file changed, 116 insertions(+)
>>>>
>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>>> index d2c7d2198e..eb4077e2ab 100644
>>>> --- a/.gitlab-ci.yml
>>>> +++ b/.gitlab-ci.yml
>>>> @@ -2,6 +2,8 @@ include:
>>>>     - local: '/.gitlab-ci-edk2.yml'
>>>>     build-system1:
>>>> + rules:
>>>> + - if: '$CI_COMMIT_REF_NAME != "staging"'
>>>>    before_script: &before_scr_apt
>>>>    - apt-get update -qq
>>>>    - apt-get install -y -qq flex bison libglib2.0-dev
>>>> libpixman-1-dev genisoimage
>>>> @@ -17,6 +19,8 @@ build-system1:
>>>>    - make -j2 check
>>>>     build-system2:
>>>> + rules:
>>>> + - if: '$CI_COMMIT_REF_NAME != "staging"'
>>>>    before_script:
>>>>     *before_scr_apt
>>>>    script:
>>>> @@ -31,6 +35,8 @@ build-system2:
>>>>    - make -j2 check
>>>>     build-disabled:
>>>> + rules:
>>>> + - if: '$CI_COMMIT_REF_NAME != "staging"'
>>>>    before_script:
>>>>     *before_scr_apt
>>>>    script:
>>>> @@ -47,6 +53,8 @@ build-disabled:
>>>>    - make -j2 check-qtest SPEED=slow
>>>>     build-tcg-disabled:
>>>> + rules:
>>>> + - if: '$CI_COMMIT_REF_NAME != "staging"'
>>>>    before_script:
>>>>     *before_scr_apt
>>>>    script:
>>>> @@ -67,6 +75,8 @@ build-tcg-disabled:
>>>>               248 250 254 255 256
>>>>     build-user:
>>>> + rules:
>>>> + - if: '$CI_COMMIT_REF_NAME != "staging"'
>>>>    before_script:
>>>>     *before_scr_apt
>>>>    script:
>>>> @@ -78,6 +88,8 @@ build-user:
>>>>    - make run-tcg-tests-i386-linux-user run-tcg-tests-x86_64-linux-user
>>>>     build-clang:
>>>> + rules:
>>>> + - if: '$CI_COMMIT_REF_NAME != "staging"'
>>>>    before_script:
>>>>     *before_scr_apt
>>>>    script:
>>>> @@ -92,6 +104,8 @@ build-clang:
>>>>    - make -j2 check
>>>>     build-tci:
>>>> + rules:
>>>> + - if: '$CI_COMMIT_REF_NAME != "staging"'
>>>>    before_script:
>>>>     *before_scr_apt
>>>>    script:
>>>> @@ -111,3 +125,105 @@ build-tci:
>>>>    - QTEST_QEMU_BINARY="x86_64-softmmu/qemu-system-x86_64"
>>>> ./tests/qtest/pxe-test
>>>>    - QTEST_QEMU_BINARY="s390x-softmmu/qemu-system-s390x"
>>>>      ./tests/qtest/pxe-test -m slow
>>>> +
>>>> +ubuntu-18.04.3-x86_64-notools:
>>>> + tags:
>>>> + - ubuntu_18.04.3
>>>> + - x86_64
>>>> + rules:
>>>> + - if: '$CI_COMMIT_REF_NAME == "staging"'
>>>> + script:
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/remake-merge-builds#n22
>>>>
>>>> + - ./configure --target-list=arm-softmmu --disable-tools
>>>> --disable-libssh
>>>> + # There is no make / make check in the "pull-buildtest" script for
>>>> this.
>>>> + # Question: should it at least be built? Or dropped?
>>>> + - make
>>>> +
>>>> +ubuntu-18.04.3-x86_64-all-linux-static:
>>>> + tags:
>>>> + - ubuntu_18.04.3
>>>> + - x86_64
>>>> + rules:
>>>> + - if: '$CI_COMMIT_REF_NAME == "staging"'
>>>> + script:
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/remake-merge-builds#n25
>>>>
>>>> + - ./configure --enable-debug --static --disable-system
>>>> --disable-glusterfs --disable-libssh
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n36
>>>>
>>>> + - make
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n45
>>>>
>>>> + - make check V=1
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n48
>>>>
>>>> + - make check-tcg V=1
>>>> +
>>>> +ubuntu-18.04.3-x86_64-all:
>>>> + tags:
>>>> + - ubuntu_18.04.3
>>>> + - x86_64
>>>> + rules:
>>>> + - if: '$CI_COMMIT_REF_NAME == "staging"'
>>>> + script:
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/remake-merge-builds#n26
>>>>
>>>> + - ./configure --disable-libssh
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n28
>>>>
>>>> + - make
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n37
>>>>
>>>> + - make check V=1
>>>> +
>>>> +ubuntu-18.04.3-x86_64-alldbg:
>>>> + tags:
>>>> + - ubuntu_18.04.3
>>>> + - x86_64
>>>> + rules:
>>>> + - if: '$CI_COMMIT_REF_NAME == "staging"'
>>>> + script:
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/remake-merge-builds#n27
>>>>
>>>> + - ./configure --disable-libssh
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n27
>>>>
>>>> + - make clean
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n29
>>>>
>>>> + - make
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n37
>>>>
>>>> + - make check V=1
>>>> +
>>>> +ubuntu-18.04.3-x86_64-clang:
>>>> + tags:
>>>> + - ubuntu_18.04.3
>>>> + - x86_64
>>>> + rules:
>>>> + - if: '$CI_COMMIT_REF_NAME == "staging"'
>>>> + script:
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/remake-merge-builds#n31
>>>>
>>>> + - ./configure --disable-libssh --cc=clang --cxx=clang++
>>>> --enable-gtk --extra-cflags='-fsanitize=undefined
>>>> -fno-sanitize=shift-base -Werror'
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n33
>>>>
>>>> + - make
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n39
>>>>
>>>> + - make check V=1
>>>> +
>>>> +ubuntu-18.04.3-x86_64-tci:
>>>> + tags:
>>>> + - ubuntu_18.04.3
>>>> + - x86_64
>>>> + rules:
>>>> + - if: '$CI_COMMIT_REF_NAME == "staging"'
>>>> + script:
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/remake-merge-builds#n33
>>>>
>>>> + - ./configure --disable-libssh --enable-debug
>>>> --enable-tcg-interpreter
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n34
>>>>
>>>> + - make
>>>> +
>>>> +ubuntu-18.04.3-x86_64-notcg:
>>>> + tags:
>>>> + - ubuntu_18.04.3
>>>> + - x86_64
>>>> + rules:
>>>> + - if: '$CI_COMMIT_REF_NAME == "staging"'
>>>> + script:
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/remake-merge-builds#n35
>>>>
>>>> + - ./configure --disable-libssh --enable-debug --disable-tcg
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n35
>>>>
>>>> + - make
>>>> + #
>>>> https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n39
>>>>
>>>> + # Question: check is disabled on the original script, because the
>>>> machine
>>>> + # is said to be running VirtualBox.  Should this be dropped or
>>>> should the
>>>> + # machine be tweaked or substituted?
>>>> + - make check V=1
>>>
>>> On another thread Thomas said he doesn't want to be responsible of
>>> other tests:
>>> https://www.mail-archive.com/address@hidden/msg670132.html
>>>
>>> We could move the set of tests converted from Peter personal
>>> testsuite into a separate YAML, so the interested (you Cleber, and
>>> Peter) can have a F: entry in MAINTAINERS for it:
>>
>>
>> I like that idea.
>>
>>
>>>
>>>
>>> include:
>>>   - local: '/.gitlab-ci-pm215.yml'
>>
>>
>> If so, I suggest moving the file to the .gitlab-ci.d directory
>> instead. Likewise the .gitlab-ci-edk2.yml file. This way the sources
>> root directory does not get infested of GitLab files.
> 
> Yes, +1. I'm pretty sure Thomas will like that too.

Sure. But I'd be also fine sharing or even completely handing over the
entry in MAINTAINERS for .gitlab-ci.yml to someone else. I'm only listed
there since I was initially the only one who was using the gitlab CI.
Now that there is interest for using gitlab CI as our gating CI, I think
it would be good to have some other people listed here, too...

 Thomas