Re: [Qemu-devel] Yet another git submodule rant

[Alexey Kardashevskiy]

On 11/11/17 01:22, Daniel P. Berrange wrote:
> On Sat, Nov 11, 2017 at 12:46:36AM +1100, Alexey Kardashevskiy wrote:
>> On 10/11/17 21:41, Daniel P. Berrange wrote:
>>> On Fri, Nov 10, 2017 at 09:35:54PM +1100, Alexey Kardashevskiy wrote:
>>>> On 09/11/17 00:01, Daniel P. Berrange wrote:
>>>>> On Wed, Nov 08, 2017 at 09:26:01AM -0300, Philippe Mathieu-Daudé wrote:
>>>>>> On 11/08/2017 06:57 AM, Thomas Huth wrote:
>>>>>>>
>>>>>>> That automatic git submodule stuff now broke my workflow again. I
>>>>>>> usually keep the git repository on my laptop and then simply rsync the
>>>>>>> sources (without .git directories) to my target machine to compile it
>>>>>>> there. Used to work great for years. Now it's broken, the build process
>>>>>>> complains:
>>>>>>>
>>>>>>> GIT submodule checkout is out of date. Please run
>>>>>>>   scripts/git-submodule.sh update
>>>>>>> from the source directory checkout /home/thuth/devel/qemu
>>>>>>>
>>>>>>> Running "scripts/git-submodule.sh update" did not fix the issue at all -
>>>>>>> I first had to tinker with it for a while to find out that I simply have
>>>>>>> to delete ".git-submodule-status" in my git tree to fix the issue.
>>>>>>>
>>>>>>> I've got the feeling that all this submodule crap is constantly causing
>>>>>>> pain ... do we really need this? Can't we find another solution instead?
>>>>>>> Or at least stop modifying files automatically in the $SRC_PATH ?
>>>>>>
>>>>>> Also yesterday on IRC:
>>>>>>
>>>>>> <RaV3N> [...] I downloaded the qemu source from git and tried to compile
>>>>>> it. I am getting this:
>>>>>>
>>>>>> ./configure --static && make && sudo make install
>>>>>>  CC      ui/input-keymap.o
>>>>>> ui/input-keymap.c:8:10: fatal error: ui/input-keymap-linux-to-qcode.c:
>>>>>> No such file or directory
>>>>>
>>>>> I had a pull request merged yesterday later afternoon which possibly
>>>>> would address that problem, though hard hard to say for certain.
>>>>
>>>> wow, already? :(
>>>>
>>>> I still wonder why do not we checkout submodules into the build directory
>>>> and why .git-submodule-status is not there too...
>>>
>>> That simply isn't the way submodules work, they are inherently part of
>>> the source tree, and the status file reflects that too.
>>
>> Sorry, I am missing the point here. What precisely does prevent us from
>> checking out the required modules to the build directory and build them
>> there? git provides a submodule repository url and sha1 for the current
>> qemu branch.
> 
> The build directory should never contain any of your version controlled
> source, as that will get irretrievably lost when the build dir is purged.

I am not suggesting editing submodule sources from the build directory, I
am suggesting not building them from the source directory, in order to keep
them in sync, Makefile can rsync or "git push" them to the build directory.
Yeah, fairly ugly but still more correct than the current solution.

And again, .git-submodule-status is not a source file, what is it doing in
$SRC_PATH?

Now I cannot compile qemu pretty much 50% of time because of that
enhancement. In the today's episode:

ssh aikhostos2 cd /home/aik/pbuild/qemu-aikhostos2-ppc64/ ;
/home/aik/p/qemu/configure --target-list=ppc64-softmmu
--source-path=/home/aik/p/qemu/ --enable-debug --enable-debug-info
--disable-werror --disable-git-update --enable-trace-backend=log

All good. One detail:
capstone          git

ssh aikhostos2 make -C /home/aik/pbuild/qemu-aikhostos2-ppc64/ -j24

make: Entering directory `/home/aik/pbuild/qemu-aikhostos2-ppc64'
  GEN     ppc64-softmmu/config-devices.mak.tmp
mkdir -p dtc/libfdt
mkdir -p dtc/tests
  GEN     config-host.h
  GEN     qemu-options.def
make[1]: *** No rule to make target
`/home/aik/pbuild/qemu-aikhostos2-ppc64/capstone/libcapstone.a'.  Stop.
make: *** [subdir-capstone] Error 2
make: *** Waiting for unfinished jobs....
  GEN     ppc64-softmmu/config-devices.mak


How, why, where are all these nice warning messages? Ah, there they are
(added 'set -x' to the script):

+ substat=.git-submodule-status
+ command=status
+ shift
+ maybe_modules='ui/keycodemapdb dtc capstone'
+ test -z git
+ modules=
+ for m in '$maybe_modules'
+ git submodule status ui/keycodemapdb
+ test 128 = 0
+ echo 'warn: ignoring non-existent submodule ui/keycodemapdb'
+ for m in '$maybe_modules'
+ git submodule status dtc
+ test 128 = 0:m
+ echo 'warn: ignoring non-existent submodule dtc'
+ for m in '$maybe_modules'
+ git submodule status capstone
+ test 128 = 0
+ echo 'warn: ignoring non-existent submodule capstone'
+ test -n 'ui/keycodemapdb dtc capstone'
+ test -e .git
+ case "$command" in
+ test -z 'ui/keycodemapdb dtc capstone'
+ test -f .git-submodule-status
+ exit 1


I assume that 'warn' is not printed because of Makefile's 'quiet-command'.

And this fails because on the server the source directory is created with
'git worktree' which points to bare git repo outside of the source tree and
the git on that build machine has no idea about worktrees (the repo is
visible to the build machine).

[vpl1 ~]$ ssh -x aikhostos2 'cd p/qemu && pwd && git --version && ls -d
../qemu.git && git submodule status'
/home/aik/p/qemu
git version 1.8.3.1
../qemu.git
fatal: Not a git repository: /home/aik/p/qemu.git/worktrees/qemu


Again - ok, when I know what is the problem, I know how to fix it - simply
move .git away when running ./configure and make (as it also runs
./configure some time) - but the messages... Firstly ./configure decides
that 'capstone' comes from 'git', then it simply fails with no indication
why it went wrong.

Ha. I just found another workaround - I can do ./configure --with-git=false
and silently disable git updates. Pretty much pointing a gun to my leg,
ready to shoot.

We could just check git submodule status every build and print a warning if
it is out of sync but continue. This way I would look at the submodules
list and decide what to do.


-- 
Alexey