Re: [Ltib] Special tag localdir_nobuild causes LTIB to go to the config screen.

[aaron]

Hi Stuart.

I admit that I hadn't thought it out all the way through since I was
thinking about the meaning of LEAVESRC and that it probably should be
EXPANDSRC, but I forgot that a kernel module package will need a couple of
more things than just an expanded source tree.  In general, in the kernel
tree, you need a .config, a make oldconfig, and a make modules.  You don't
need the whole kernel built, but I suppose there's also a LTIB dependency
on the time stamp of the kernel tree relative to the time stamp of a
binary RPM.  Maybe there could be a shortcut here, but it might be
difficult to implement.

In terms of keeping it simple, it would be nice if the configuration used
to build a BSP were preserved in the BSP and in place at the time of the
first installation.  Also, it would be cleaner if having a kernel module
package in a BSP with binary target RPM files did not automatically cause
a rebuild of the kernel package.

Perhaps it's possible to restrict the clearing of transient flags in some
select cases, consistent with what happens in the logic of Ltibrelease.pm?

Aaron

> Hi Aaron,
>
> As I explained before, this flag must be transient.  Take this use case:
>
> * user installed BSP with no builds occurring.
> * user selects a module that has not been previously built
>
> In this case you need the setting of LEAVESRC trigged by the module to
> be different from the previous state so you can trigger a kernel
> unpack/build.  If this flag were left already set (in the ISO preset)
> then you'd never get a change of state.
>
> The bug is as you point out that the kernel should not build on
> installation.  This needs investigation.
>
> Regards, Stuart
>
>
> address@hidden wrote:
>> Speaking honestly, for me the way this transient *LEAVESRC configuration
>> variable is utilized right now is counter-intuitive, inflexible, and
>> prone
>> to unintended consequences.  In terms of being counter-intuitive, I
>> think
>> that the purpose of this variable is not clear because its transient
>> nature is obscured, and it cost me time to figure out what was going on
>> and why I wasn't able to do what I thought I should be able to do
>> setting
>> up my BSP.  In terms of being inflexible, I had to try harder than usual
>> to put in place what I wanted despite the functionality of *LEAVESRC.
>> In
>> terms of being prone to unintended consequences, it can lead to things
>> like unnecessary kernel builds, and many iterations of BSP generation
>> and
>> testing, which take a long time.
>>
>> I don't think there is any convenience in having this as a one-shot or
>> transient variable.  If the source of a package is expanded as a result
>> of
>> setting this variable then I think both the source can stay expanded and
>> the variable can stay set.  I think if its origin traces back to trying
>> to
>> detect a state change in a dependency list, then it makes more sense,
>> but
>> I still feel it's wrong, because you don't need state information.  In
>> the
>> case of the kernel, if CONFIG_PKG_KERNEL_LEAVESRC=y, and there is no
>> directory rpm/BUILD/linux, then the kernel should be expanded with -m
>> prep, but not built purely as a result of *LEAVESRC, which might be set
>> by
>> a kernel module that only needs the kernel's kbuild infrastructure.  If
>> the kernel package is already built in the BSP, then that's a time-saver
>> for a developer who might only be working with a module, or a useful
>> logical partition for a developer who might want to prove that only the
>> module is affected, and nothing in the kernel tree.  In fact, maybe it
>> should be renamed to EXPANDSRC, or something similar, and not treated as
>> transient.  Any time this flag is set for any package, and the source
>> has
>> not been expanded in the build directory, a -m prep takes place, and the
>> variable stays set.  If the user wants, the package source directory can
>> be removed from the build directory, and the variable cleared in the
>> menu
>> configuration.
>>
>> I think in terms of the BSP feature, the more flexibility you can offer
>> the better.  The user should be allowed some freedom to package their
>> board support software in the way that they want.  Even if they wanted
>> to
>> drop the user to a ltib/kernel/busybox configuration menu right away
>> (*WANT_CF), or force a complete target image rebuild (localdir_nobuild),
>> then that should be possible.  This might seem goofy the way one
>> developer
>> thinks about it, but it might be useful to someone else.  Another thing
>> I
>> would suggest is that if you're going to have a stage directory and
>> build
>> a BSP in that stage directory (which implicitly proves that the software
>> of the BSP can be successfully built), then that same software and
>> configuration should be preserved in the BSP and presented to the BSP
>> installer, and not something slightly different.  Even if the chance is
>> small, if something is slightly different the software might not build.
>> I
>> ran into this.
>>
>> This is mostly interpretation stuff.  The only "bug" case at this point
>> is:
>>
>> * Constructing a BSP with any tag other than localdir_nobuild that has a
>> package that selects PKG_KERNEL_LEAVESRC.  The installer of the BSP will
>> incur an unnecessary kernel build.
>>
>> Even this isn't really a bug, but things can work better, and they can
>> work worse.  Of course I'm not maintaining the LTIB, but that is from my
>> perspective as a user.
>>
>> Aaron
>>
>>> Hi Aaron/Olivia,
>>>
>>> I've been thinking about this a little more (you have similar related
>>> issues I think).
>>>
>>> On common question that arises is whether or not the
>>> PKG_KERNEL_LEAVESRC
>>> should be transient.
>>>
>>> I believe it should because the principle is that when you install an
>>> ISO based LTIB image it should not build anything for the target image
>>> on the first ./ltib run, all it should do is re-populate the rootfs
>>> image based on the pre-built binary rpms.
>>>
>>> Given this, if you then for example enable a kernel module from the
>>> config system, this will set the transient flag: PKG_KERNEL_LEAVESRC
>>> This change of state will be picked up by LTIB and cause:
>>>
>>> * the kernel to unpack, build and keep the built source directory
>>> * the module itself to build
>>> * on exit the transient flag PKG_KERNEL_LEAVESRC
>>>
>>> On the other hand, if you were to not clear the  PKG_KERNEL_LEAVESRC in
>>> the .config for the ISO image, and you then enabled a module in the
>>> config system, this would set the already set PKG_KERNEL_LEAVESRC.  As
>>> there is no change of state, ltib would not know to unpack the kernel
>>> sources.
>>>
>>> Notwithstanding this, there may be other bugs in this area, but to
>>> investigate I need a very simple (bullet points) test case I can
>>> understand.
>>>
>>> Regards, Stuart
>>>
>>> FYI, here's a run I did where I enabled the helloword module on a
>>> system
>>> that had already built the kernel but had no source code unpacked:
>>>
>>> Processing platform: A&MLtd Adder MPC875 PowerPC board
>>> ========================================================
>>> using config/platform/qs875s/.config
>>> PKG_KERNEL_LEAVESRC has changed, kernel-2.6.16-875 needs a rebuild
>>>
>>> Processing: fake-provides
>>> ===========================
>>>
>>> Processing: kernel-2.6.16-875
>>> ===============================
>>> Build path taken because: spec file newer than rpm,
>>>
>>>
>>>
>>> Stuart Hughes wrote:
>>>> Hi Aaron,
>>>>
>>>> Thanks, I'll take a look and think about this one and the
>>>> implications.
>>>>
>>>> Regards, Stuart
>>>>
>>>> address@hidden wrote:
>>>>> Hi Stuart.
>>>>>
>>>>> I think there's a comfort that the configuration used to create the
>>>>> BSP is
>>>>> the same configuration that is presented to the user when installing
>>>>> the
>>>>> BSP.  If the user wants to do a "./ltib --force" right away then it
>>>>> should
>>>>> be the same type of configuration and build that created the BSP.
>>>>> That's
>>>>> why I think the transient flags should not be affected in the release
>>>>> process.  I'm not a Perl programmer, so for sure this is not an
>>>>> elegant
>>>>> approach, but this patch is simple and works for me with both
>>>>> localdir
>>>>> and
>>>>> localdir_nobuild special tags.  Aside from preserving the
>>>>> configuration,
>>>>> it ensures that the kernel is not rebuilt in the case we discussed.
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Aaron
>>>>>
>>>>>> Hi Aaron,
>>>>>>
>>>>>> You have some points, however transient flags should not be relied
>>>>>> upon
>>>>>> for installs.  The most important IMHO you make is the unnecessary
>>>>>> kernel build on install, this is not meant to happen.  However your
>>>>>> concern about rebuilding the kernel module should not be an issue as
>>>>>> if
>>>>>> the module builds, it will should unpack and build the kernel.
>>>>>>
>>>>>> This a real corner case and I doubt I will have time to look into it
>>>>>> in
>>>>>> the immediate future.  Could you try to come up with a patch that
>>>>>> implement what you need (with the minimum change of behaviour).
>>>>>>
>>>>>> Regards, Stuart
>>>>>>
>>>>>> address@hidden wrote:
>>>>>>> Hi Stuart.  I know this thread is from a while ago, and that this
>>>>>>> is
>>>>>>> probably a minor issue.  I just thought I'd mention that I think it
>>>>>>> might
>>>>>>> be more correct to avoid clearing these transient configuration
>>>>>>> flags
>>>>>>> (*WANT_CF and *LEAVESRC) when constructing a release or test
>>>>>>> release
>>>>>>> BSP.
>>>>>>>
>>>>>>> The thing is that even if I have a "select PKG_KERNEL_LEAVESRC"
>>>>>>> line
>>>>>>> somewhere, or otherwise have it configured, it will get cleared
>>>>>>> with
>>>>>>> the
>>>>>>> "./ltib -m clean" line in bin/Ltibrelease.pm when building a
>>>>>>> release.
>>>>>>> Now
>>>>>>> that the fix we worked on is present, the localdir_nobuild tag
>>>>>>> happily
>>>>>>> does not clear the transient flags automatically, but other tags
>>>>>>> such as
>>>>>>> localdir will.  This means that the configuration used to construct
>>>>>>> the
>>>>>>> BSP will differ from the configuration of the BSP when it's being
>>>>>>> installed from the iso (if any of the *WANT_CF or *LEAVESRC flags
>>>>>>> are
>>>>>>> set
>>>>>>> to 'y' in the default configuration) for all release tags except
>>>>>>> localdir_nobuild, and I think this is a little inconsistent.  I
>>>>>>> believe
>>>>>>> that the configuration that is present at the time the iso BSP is
>>>>>>> being
>>>>>>> built should be the equivalent configuration when the iso BSP is
>>>>>>> being
>>>>>>> installed.
>>>>>>>
>>>>>>> Particularly bad is the way the localdir special tag is working
>>>>>>> right
>>>>>>> now.
>>>>>>>  Since I have a kernel module that depends on the kernel sources
>>>>>>> being
>>>>>>> unpacked, I have my select PKG_KERNEL_LEAVESRC line in the
>>>>>>> package.lkc.
>>>>>>> I
>>>>>>> go to make a test release, and this builds the kernel binary RPM as
>>>>>>> expected, but after it's built, the LTIB script clears
>>>>>>> CONFIG_PKG_KERNEL_LEAVESRC in the .config file that is distributed
>>>>>>> with
>>>>>>> the BSP.  When the user goes to install the BSP from the iso with
>>>>>>> "cd
>>>>>>> non-*; ./ltib", the kernel package is actually automatically
>>>>>>> rebuilt
>>>>>>> because there is a file .config.old that has
>>>>>>> CONFIG_PKG_KERNEL_LEAVESRC=y,
>>>>>>> and this differs with the .config file.  So, in effect, it rebuilds
>>>>>>> the
>>>>>>> kernel binary RPM unnecessarily, but on top of that, it clears the
>>>>>>> kernel
>>>>>>> source when it's finished, because in the new .config file
>>>>>>> CONFIG_PKG_KERNEL_LEAVESRC is not set.  Supposing a user wants to
>>>>>>> force
>>>>>>> a
>>>>>>> rebuild of the kernel module at this point, the kernel source will
>>>>>>> not
>>>>>>> be
>>>>>>> available in the BUILD directory.
>>>>>>>
>>>>>>> Perhaps it might work for only the "./ltib -m clean" operation to
>>>>>>> avoid
>>>>>>> clearing transient flags, and then these flags will persist through
>>>>>>> this
>>>>>>> call, and be present in the same state in the BSP as they were to
>>>>>>> build
>>>>>>> the BSP.
>>>>>>>
>>>>>>> What do you think?
>>>>>>>
>>>>>>> Aaron
>>>>>>>
>>>>>>>> Hi Aaron,
>>>>>>>>
>>>>>>>> PKG_KERNEL_LEAVESRC is meant to be a one-shot as once unpacked the
>>>>>>>> source will remain and so there's no need to keep the flag
>>>>>>>> enabled.
>>>>>>>>
>>>>>>>> If you package needs the kernel source unpacked it should be
>>>>>>>> selecting
>>>>>>>> PKG_KERNEL_LEAVESRC rather than relying on manually setting this
>>>>>>>> in
>>>>>>>> the
>>>>>>>> ./ltib -m config session.  Take a look at
>>>>>>>> config/userspace/package.lkc,
>>>>>>>> for example:
>>>>>>>>
>>>>>>>> config PKG_HELLOWORLD_MOD
>>>>>>>>      depends CAP_HAS_MMU
>>>>>>>>      select PKG_KERNEL_LEAVESRC
>>>>>>>>      bool "hello world module example"
>>>>>>>>      help
>>>>>>>>          simple hello world kernel modules example
>>>>>>>>
>>>>>>>> Regards, Stuart
>>>>>>>>
>>>>>>>>
>>>>>>>> address@hidden wrote:
>>>>>>>>> Hi Peter and Stuart.  Working with my kernel module is precisely
>>>>>>>>> where
>>>>>>>>> this gave me a little trouble.  I understand that you need the
>>>>>>>>> kernel
>>>>>>>>> source unpacked for a kernel module package.  My question was why
>>>>>>>>> this
>>>>>>>>> is
>>>>>>>>> a one-shot or transient variable.  The area that it was giving me
>>>>>>>>> a
>>>>>>>>> few
>>>>>>>>> problems was after creating an iso release, the build and
>>>>>>>>> installation
>>>>>>>>> of
>>>>>>>>> the BSP onto another machine.  Using `./ltib` I would attempt to
>>>>>>>>> build
>>>>>>>>> the
>>>>>>>>> whole project (host and target packages), and I would get stuck
>>>>>>>>> on
>>>>>>>>> the
>>>>>>>>> kernel module package because my kernel source would not remain
>>>>>>>>> in
>>>>>>>>> the
>>>>>>>>> BUILD directory.  This was because the PKG_KERNEL_LEAVESRC=y that
>>>>>>>>> I
>>>>>>>>> had
>>>>>>>>> put in place earlier was getting cleared automatically by a call
>>>>>>>>> to
>>>>>>>>> `./ltib` when building the iso.  Since I was running into this
>>>>>>>>> sort of
>>>>>>>>> problem, I was wondering if there is any beneficial reason for
>>>>>>>>> having
>>>>>>>>> the
>>>>>>>>> LEAVESRC variable as a one-shot or transient variable.  It seems
>>>>>>>>> like
>>>>>>>>> if
>>>>>>>>> you have the source out for a particular package, and then you
>>>>>>>>> wanted
>>>>>>>>> to
>>>>>>>>> get rid of it, you can just remove it, and change the
>>>>>>>>> configuration to
>>>>>>>>> comment out any variables like PKG_KERNEL_LEAVESRC if necessary.
>>>>>>>>> Updating
>>>>>>>>> the date/time stamp on my config/platform/${board}/.config file
>>>>>>>>> does
>>>>>>>>> not
>>>>>>>>> force a rebuild of the kernel for me.
>>>>>>>>>
>>>>>>>>>> One to thik about is a seperate package that builds a kernel
>>>>>>>>>> module
>>>>>>>>>> (like PKG_HELLO_WORLD_MOD).  It needs the kernel source to
>>>>>>>>>> remain
>>>>>>>>>> so
>>>>>>>>>> it
>>>>>>>>>> can use the kernel build structure/headers to create a module
>>>>>>>>>> that is
>>>>>>>>>> loadable by the kernel....
>>>>>>>>>> Hi Aaron,
>>>>>>>>>>
>>>>>>>>>> Once source code is blown out and left (with LEAVESRC) it will
>>>>>>>>>> stay
>>>>>>>>>> there. nothing will remove it until you do manually, so the flag
>>>>>>>>>> only
>>>>>>>>>> needs to be transient.  The need for this is as Peter says, some
>>>>>>>>>> packages (like kernel modules) need access to the the real built
>>>>>>>>>> kernel
>>>>>>>>>> sources.
>>>>>>>>>>
>>>>>>>>>> --leavesrc --force should leave all the sources unpacked.  This
>>>>>>>>>> is
>>>>>>>>>> generally speaking a bad idea.
>>>>>>>>>>
>>>>>>>>>> Thanks for the patch.
>>>>>>>>>>
>>>>>>>>>> Regards, Stuart
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> LTIB home page: http://ltib.org
>>>>>>>>>
>>>>>>>>> Ltib mailing list
>>>>>>>>> address@hidden
>>>>>>>>> http://lists.nongnu.org/mailman/listinfo/ltib
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> LTIB home page: http://ltib.org
>>>>>>>
>>>>>>> Ltib mailing list
>>>>>>> address@hidden
>>>>>>> http://lists.nongnu.org/mailman/listinfo/ltib
>>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>> _______________________________________________
>>>>>> LTIB home page: http://ltib.org
>>>>>>
>>>>>> Ltib mailing list
>>>>>> address@hidden
>>>>>> http://lists.nongnu.org/mailman/listinfo/ltib
>>>>
>>>> _______________________________________________
>>>> LTIB home page: http://ltib.org
>>>>
>>>> Ltib mailing list
>>>> address@hidden
>>>> http://lists.nongnu.org/mailman/listinfo/ltib
>>>>
>>
>>
>>
>