Re: [Ltib] Need to remove package build source if that package .spec changed

[Peter Barada]

On 01/31/2011 12:52 PM, Stuart Hughes wrote:
> Hi Peter,
>
> I started implementing this and again I found myself asking; what if
> someone has edits they want in that directory and they get clobbered?
> Whether tarball or svn (git, cvs etc) based packages, no one will thank
> you if their precious work gets zapped.
>
> The only safe way to do this would be to ensure prior to source removal
> ensure that the sources have not been changed.  The problem with this is
> that:
>
> * This can take a long time as it needs a full unpack of the reference
> and then a diff -q, which I guess could be okay/acceptable
>
> * This can only work for tarball based package which is okay/acceptable
>
> * This is not reliable unless the clean/distclean for the package
> actually works. Which may/may not be acceptable
>
> Let me know if:
>
> a) You agree with my reasoning
>
> b) You still think this is worth trying to do (it's more complicated
> than the patch you sent)
I believe its still worth doing for the tarball/patch case to make LTIB more 
usable in an automated build system as part of continuous integration 
development where user interaction is impractical.

"--clobber" is not a switch for the feint of heart; it specifically allows LTIB 
to remove something that could possibly contain changes you want to keep, and 
should only be specified by those that fully understand the risks of doing so.

W/o digging into RPM deeper, is there any way RPM can determine if a .spec 
files contains %source/%patch changes (from the databse LTIB keeps) that 
require re-executing %prep to bring the source into sync?

> Regards, Stuart
>
> On 31/01/11 16:21, Stuart Hughes wrote:
>> Hi Peter,
>>
>> Okay, I think I understand the requirement now.  I won't rely on the svn
>> tag in the name, I think that's too weak/restrictive (in case someone
>> wants to use git or something else).  Instead I think the absence of a
>> SOURCE: tag should be the clue to LTIB that this is a directory build
>> (svn, cvs, git etc) and so clobbering is not allowed.
>>
>> Would you like me to send the patch to the list for you to look at/try
>> out before checking in?
>>
>> Regards, Stuart
>>
>> On 31/01/11 15:22, Peter Barada wrote:
>>> On 01/31/2011 10:10 AM, Stuart Hughes wrote:
>>>> Hi Peter,
>>>>
>>>> Okay, there are 2 separate use cases here;
>>>>
>>>> 1/ The tarball+patches builds
>>>>
>>>> 2/ The SVN (SCM) based directory builds.
>>>>
>>>> In the case of 2/ all that I've said before stands (e.g. clobber is a
>>>> very bad idea).
>>>>
>>>> In the case of 1/ your patch would be reasonable, but I would also add a
>>>> prompt to the user to make sure.  This would be bypassed if the --batch
>>>> option was in force (e.g. for auto-builder).  My preference would be to
>>>> limit this to only tarball based builds.  The detection I'd propose is
>>>> checking for the existence of a source tag being present.  Does this
>>>> sounds reasonable?
>>> That sounds perfectly reasonable.
>>>
>>> However in the case of #2, --clobber can't happen for a SCM build since
>>> in this case the patch has:
>>>
>>>     # If its a SCM (source code management) package, force prep
>>>     my $svn_bld = 0;
>>>     if ($tok->{version} eq 'svn') {
>>>     $svn_bld = 1;
>>>     *$dir_bld = 0*;
>>>     }
>>>
>>> And later:
>>>  
>>>        if ($cf->{clobber} && *$dir_bld* && $unpack eq 'yes' && $spec_upd) {
>>>         print "Clobber forces removal of
>>> $cf->{rpmdir}/BUILD/$tok->{pkg_dir_name}\n";
>>>         system_nb("rm -rf $cf->{rpmdir}/BUILD/$tok->{pkg_dir_name}");
>>>         # force prep
>>>         $dir_bld = 0;
>>>         }
>>>
>>>
>>>> Regards, Stuart
>>>>
>>>> On 31/01/11 14:23, Peter Barada wrote:
>>>>> On 01/31/2011 05:13 AM, Stuart Hughes wrote:
>>>>>> Hi Peter,
>>>>>>
>>>>>> I'm reluctant to add this as it breaks a golden rule that LTIB promises
>>>>>> not to delete source code once unpacked.  What if some developer had put
>>>>>> in some edits and forgotten and had the --clobber in a script?
>>>>> Then they shoot themselves in the foot.  You can't make things completely 
>>>>> foolproof as God will come along and create a more determined fool...
>>>>>
>>>>> Perhaps renaming the option to "---clobber" or "--KlObBeR" or even 
>>>>> "--Remove-prepped-source-if-spec-file-updates" to make it stand out even 
>>>>> more would help...
>>>>>> Reading this, and your other email about SVN support, maybe there's
>>>>>> another way.  Rather than removing the source, why not try to update it;
>>>>>> this would prevent accidental clobbering and also allow updates.  This
>>>>>> way different developers can pull in unrelated updates and stay in sync.
>>>>> That is in the case where a packages is not kept as a tarball+patches but 
>>>>> instead as a directory under some SCM.  In the case where developer A 
>>>>> checks in a patch (and correspodning .spec file change to add the 
>>>>> %patch), and devloper B updates his LTIB source, everything works fine 
>>>>> unless the patch checked in is an update to a package that LTIB leaves 
>>>>> checked out due to another package needing the source.
>>>>>
>>>>> As an example if the LTIB project developer A and B are working on has 
>>>>> helloworld_mod enabled, then the kernel source will always be left 
>>>>> prepped.  If Developer A adds a patch to the kernel (and updates the 
>>>>> kernel .spec file), and developer B then updates his LTIB with the 
>>>>> changes from developer A and builds, he will not build the change 
>>>>> developer A checked in since the kernel source is already %prepped.   
>>>>> This can lead to confusion...
>>>>>
>>>>>> This can be done using this directory build mechanism.  Below is an
>>>>>> example of what I mean, stripped back as a guide. There are related
>>>>>> examples in dtc-dir-build.spec.in,
>>>>>> dist/lfs-5.1/kernel/kernel26-dir-build.spec.in &
>>>>>> dist/lfs-5.1/u-boot/u-boot-dir-build.spec.in ):
>>>>>>
>>>>>> Maybe this could do what you need?
>>>>>>
>>>>>> Regards, Stuart
>>>>>>
>>>>>> --- a spec file that manages it's own SCM commands ---
>>>>>>
>>>>>> %define pfx /opt/freescale/rootfs/%{_target_cpu}
>>>>>> %define buildsubdir %{name}-%{version}
>>>>>> ....
>>>>>> Name            : some_name
>>>>>> Version         : 1.1
>>>>>> Release         : 1
>>>>>> ....
>>>>>> %Prep
>>>>>> # only do the code checkout if the target directory doesn't exist
>>>>>> if [ -e "%{buildsubdir}" ]
>>>>>> then
>>>>>>     exit 0
>>>>>> fi
>>>>>> mkdir %{buildsubdir}
>>>>>> cd %{buildsubdir}
>>>>>> git-clone git://some_url/some_project
>>>>>> git checkout some_branch
>>>>>>
>>>>>> %build
>>>>>> cd some_project
>>>>>> git pull
>>>>>> make
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 28/01/11 16:09, Peter Barada wrote:
>>>>>>> Stuart,
>>>>>>>
>>>>>>> As discussed previously, if I have a build that leaves source
>>>>>>> unpacked
>>>>>> (because its used by another package as in the kernel source used by
>>>>>> helloworld_mod), I ran into the problem that if I updated my LTIB tree
>>>>>> from SVN (and another developer added a patch to the kernel), the
>>>>>> current version of LTIB wouldn't clobber the kernel source from
>>>>>> rpm/BUILD. This can lead to serious confusion as Developer A says "Hey,
>>>>>> I checked in a patch to fix that bug in package X" and developer B pulls
>>>>>> updates LTIB, does "./ltib" and responds "no you didn't"....
>>>>>>> The attached patch adds "--clobber" which removes a packages source
>>>>>> and then runs the %prep step if it detects that the spec file has been
>>>>>> updated.
>>>>>>
>>>
>>> -- 
>>> Peter Barada
>>> address@hidden
>>>
>> _______________________________________________
>> LTIB home page: http://ltib.org
>>
>> Ltib mailing list
>> address@hidden
>> http://lists.nongnu.org/mailman/listinfo/ltib
>>
> _______________________________________________
> LTIB home page: http://ltib.org
>
> Ltib mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/ltib


-- 
Peter Barada
address@hidden