Re: [Qemu-devel] Yet another git submodule rant

[Alexey Kardashevskiy]

On 11/11/17 12:10, Alexey Kardashevskiy wrote:
> On 11/11/17 01:22, Daniel P. Berrange wrote:
>> On Sat, Nov 11, 2017 at 12:46:36AM +1100, Alexey Kardashevskiy wrote:
>>> On 10/11/17 21:41, Daniel P. Berrange wrote:
>>>> On Fri, Nov 10, 2017 at 09:35:54PM +1100, Alexey Kardashevskiy wrote:
>>>>> On 09/11/17 00:01, Daniel P. Berrange wrote:
>>>>>> On Wed, Nov 08, 2017 at 09:26:01AM -0300, Philippe Mathieu-Daudé wrote:
>>>>>>> On 11/08/2017 06:57 AM, Thomas Huth wrote:
>>>>>>>>
>>>>>>>> That automatic git submodule stuff now broke my workflow again. I
>>>>>>>> usually keep the git repository on my laptop and then simply rsync the
>>>>>>>> sources (without .git directories) to my target machine to compile it
>>>>>>>> there. Used to work great for years. Now it's broken, the build process
>>>>>>>> complains:
>>>>>>>>
>>>>>>>> GIT submodule checkout is out of date. Please run
>>>>>>>>   scripts/git-submodule.sh update
>>>>>>>> from the source directory checkout /home/thuth/devel/qemu
>>>>>>>>
>>>>>>>> Running "scripts/git-submodule.sh update" did not fix the issue at all 
>>>>>>>> -
>>>>>>>> I first had to tinker with it for a while to find out that I simply 
>>>>>>>> have
>>>>>>>> to delete ".git-submodule-status" in my git tree to fix the issue.
>>>>>>>>
>>>>>>>> I've got the feeling that all this submodule crap is constantly causing
>>>>>>>> pain ... do we really need this? Can't we find another solution 
>>>>>>>> instead?
>>>>>>>> Or at least stop modifying files automatically in the $SRC_PATH ?
>>>>>>>
>>>>>>> Also yesterday on IRC:
>>>>>>>
>>>>>>> <RaV3N> [...] I downloaded the qemu source from git and tried to compile
>>>>>>> it. I am getting this:
>>>>>>>
>>>>>>> ./configure --static && make && sudo make install
>>>>>>>  CC      ui/input-keymap.o
>>>>>>> ui/input-keymap.c:8:10: fatal error: ui/input-keymap-linux-to-qcode.c:
>>>>>>> No such file or directory
>>>>>>
>>>>>> I had a pull request merged yesterday later afternoon which possibly
>>>>>> would address that problem, though hard hard to say for certain.
>>>>>
>>>>> wow, already? :(
>>>>>
>>>>> I still wonder why do not we checkout submodules into the build directory
>>>>> and why .git-submodule-status is not there too...
>>>>
>>>> That simply isn't the way submodules work, they are inherently part of
>>>> the source tree, and the status file reflects that too.
>>>
>>> Sorry, I am missing the point here. What precisely does prevent us from
>>> checking out the required modules to the build directory and build them
>>> there? git provides a submodule repository url and sha1 for the current
>>> qemu branch.
>>
>> The build directory should never contain any of your version controlled
>> source, as that will get irretrievably lost when the build dir is purged.
> 
> I am not suggesting editing submodule sources from the build directory, I
> am suggesting not building them from the source directory, in order to keep
> them in sync, Makefile can rsync or "git push" them to the build directory.
> Yeah, fairly ugly but still more correct than the current solution.
> 
> And again, .git-submodule-status is not a source file, what is it doing in
> $SRC_PATH?
> 
> Now I cannot compile qemu pretty much 50% of time because of that
> enhancement. In the today's episode:
> 
> ssh aikhostos2 cd /home/aik/pbuild/qemu-aikhostos2-ppc64/ ;
> /home/aik/p/qemu/configure --target-list=ppc64-softmmu
> --source-path=/home/aik/p/qemu/ --enable-debug --enable-debug-info
> --disable-werror --disable-git-update --enable-trace-backend=log
> 
> All good. One detail:
> capstone          git
> 
> ssh aikhostos2 make -C /home/aik/pbuild/qemu-aikhostos2-ppc64/ -j24
> 
> make: Entering directory `/home/aik/pbuild/qemu-aikhostos2-ppc64'
>   GEN     ppc64-softmmu/config-devices.mak.tmp
> mkdir -p dtc/libfdt
> mkdir -p dtc/tests
>   GEN     config-host.h
>   GEN     qemu-options.def
> make[1]: *** No rule to make target
> `/home/aik/pbuild/qemu-aikhostos2-ppc64/capstone/libcapstone.a'.  Stop.
> make: *** [subdir-capstone] Error 2
> make: *** Waiting for unfinished jobs....
>   GEN     ppc64-softmmu/config-devices.mak
> 
> 
> How, why, where are all these nice warning messages? Ah, there they are
> (added 'set -x' to the script):
> 
> + substat=.git-submodule-status
> + command=status
> + shift
> + maybe_modules='ui/keycodemapdb dtc capstone'
> + test -z git
> + modules=
> + for m in '$maybe_modules'
> + git submodule status ui/keycodemapdb
> + test 128 = 0
> + echo 'warn: ignoring non-existent submodule ui/keycodemapdb'
> + for m in '$maybe_modules'
> + git submodule status dtc
> + test 128 = 0:m
> + echo 'warn: ignoring non-existent submodule dtc'
> + for m in '$maybe_modules'
> + git submodule status capstone
> + test 128 = 0
> + echo 'warn: ignoring non-existent submodule capstone'
> + test -n 'ui/keycodemapdb dtc capstone'
> + test -e .git
> + case "$command" in
> + test -z 'ui/keycodemapdb dtc capstone'
> + test -f .git-submodule-status
> + exit 1
> 
> 
> I assume that 'warn' is not printed because of Makefile's 'quiet-command'.
> 
> And this fails because on the server the source directory is created with
> 'git worktree' which points to bare git repo outside of the source tree and
> the git on that build machine has no idea about worktrees (the repo is
> visible to the build machine).
> 
> [vpl1 ~]$ ssh -x aikhostos2 'cd p/qemu && pwd && git --version && ls -d
> ../qemu.git && git submodule status'
> /home/aik/p/qemu
> git version 1.8.3.1
> ../qemu.git
> fatal: Not a git repository: /home/aik/p/qemu.git/worktrees/qemu
> 
> 
> Again - ok, when I know what is the problem, I know how to fix it - simply
> move .git away when running ./configure and make (as it also runs
> ./configure some time) - but the messages... Firstly ./configure decides
> that 'capstone' comes from 'git', then it simply fails with no indication
> why it went wrong.
> 
> Ha. I just found another workaround - I can do ./configure --with-git=false
> and silently disable git updates. Pretty much pointing a gun to my leg,
> ready to shoot.
> 
> We could just check git submodule status every build and print a warning if
> it is out of sync but continue. This way I would look at the submodules
> list and decide what to do.


I assume that no response to this means my proposal is somehow worse that
what we got now, correct? Thanks.



-- 
Alexey