| Age | Commit message (Collapse) | Author | Files |
|---|
|
This patch enables multiconfig dependencies (mcdepends) to be used on
recipes using the following format:
task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on"
For the sake of simplicity consider the following example:
Assuming we have set up multiconfig builds, one for qemux86 and one for
qemuarm, named x86 and arm respectively.
Adding the following line to an image recipe (core-image-sato):
do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"
Would state that core-image-sato:do_image from x86 will depend on
core-image-minimal:do_rootfs from arm so it can be executed.
This patch makes modifications to bitbake.conf to enable mcdepends, and
to sstatesig and staging.bbclass to avoid conflicts between packages from
different multiconfigs.
[YOCTO #10681]
Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we're in a multilib context already and want a non-multilib context
this function returned incorrect values.
Try and retain optimisations for the common case not needing to request
a datastore but allow the different multilib/non-multilib combinations
to work too.
This fixes bugs where rootfs generation of a multilib image would
write into incorrect locations, or be unable to find sstate manifest
files due to incorrect data stores being used to expand data.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Now that SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS supports the wildcard syntax,
these exclusions no longer need to be hardcoded
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The special string "*" on the left hand side of the dependency
specification matches all recipes except the one on the right hand side.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Changes to the icecc-create-env recipe should not cause all recipes to
rebuild just because the have inherited icecc.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Create a common function for locating task manifest files rather than
several implementations with missing pieces.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With a '*' as a wildcard for the signature here we can also match a
portion of the task name with the result that we may match a sigdata
file for the wrong task. Luckily the signature is always the same
length - 32 characters - so we can simply use 32 '?' characters instead.
(A regex would have been another alternative, but the wildcard should be
effective and I felt like a regex would complicate the code more than
this solution).
Fixes [YOCTO #11763].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Enabling ccache should not cause a complete rebuild, so filter out ccache-native
from the dependencies the same way we do for quilt-native (so the world doesn't
repatch if quilt changes).
This doesn't effect the actual dependencies, just the dependencies that impact
the hash.
[ YOCTO #11417 ]
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Adding or removing archiver.bbclass from a build configuration causes
rebuilds of linux-yocto-based kernels because of the
do_kernel_configme->do_unpack_and_patch task dependency.
This particular dependency can be ignored for the do_kernel_configme
sstate signature calculcation. Idea for the fix from Richard Purdie.
Note that building the kernel and adding archiver.bbclass later to
archive sources leads to do_unpack_and_patch running after
do_kernel_configme (because that already ran in the first build),
which might be problematic. This is independent of the change here.
The use case in YOCTO #11441 is to removed archiver.bbclass between a
production build with archiving enabled and builds via oe-selftests
without archiving. That direction is fine.
Fixes: YOCTO #11441
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Having added writing out of the task signature list to buildhistory
(when BUILDHISTORY_FEATURES includes "task"), we now need a way to
compare the list. This just shows which tasks have been added / changed
signature / removed.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we want to determine what changed since the last build, one angle
from which to look at it is to check the signatures. However, if we
don't actually have the signatures from the last build we don't have
anywhere to start. Save the signatures on each build in order to give us
the starting point.
NOTE: you need to set your BUILDHISTORY_FEATURES value to include
"task" to enable collection of these signatures as it is is disabled by
default.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In find_siginfo(), which is used by bitbake-diffsigs among other things,
avoid adding a siginfo file from the sstate-cache where we've already
collected a sigdata file from the stamps directory with the same hash.
This avoids the possibility that the top two files (as picked by default
using the bitbake-diffsigs -t option) are for the same signature and
thus the tool would report no differences. In order to do that, just use
the hashfiles dict that we already have - we just need to change the
code to populate that even if we're collecting matching files without
looking for a fixed set of hashes (i.e. taskhashlist isn't set).
This replaces previous code in bitbake-diffsigs that attempted to filter
these out with limited success.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When comparing signatures with bitbake-diffsigs -t or bitbake -S
printdiff, we use this find_siginfo() function implemented in this
module to find the siginfo/sigdata files corresponding to the tasks
we're looking for. However, native sstate files go into a
NATIVELSBSTRING subdirectory and there was no handling for this when
asking about native recipes.
I'm not even sure why we were walking SSTATE_DIR in order to find
this - we don't need to, we just need to run glob.glob() on the filespec
we calculate, which should be a little bit more efficient.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Sort keys of dict 'types' prior to dumping, in order to have
identical output every time. This could make it a little easier
to diff these human-readable dumps.
Signed-off-by: Jianxun Zhang <jianxun.zhang@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
getVarFlag() now defaults to expanding by default, thus remove the
True option from getVarFlag() calls with a regex search and
replace.
Search made with the following regex:
getVarFlag ?\(( ?[^,()]*, ?[^,()]*), True\)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
If we're building the extensible SDK we don't need to see the "Writing
locked sigs" message; it's only necessary when the user explicitly runs
bitbake -S none <target>.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Iterators now return views, not lists in python3. Where we need
lists, handle this explicitly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
python3 standardises its use of iteration operations. Update
the code to match the for python3 requires.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In order to support workflows using devtool where a user might want to
modify tasks that exist in locked-sigs.inc, there must be a way to unlock
recipes.
This patch adds that support by allowing the user to add recipes to
SIGGEN_UNLOCKED_RECIPES. Recipes that exist in that variable will have
all their tasks unlocked, as well as any tasks that depend on that
recipe.
For example if foo->bar->baz, if you unlock baz, it will also unlock bar
so that foo can be rebuilt without explicitly specifying bar as being
unlocked.
[YOCTO #9195]
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The previous message when signatures didn't match between the metadata
and the locked signatures file, the message output was a bit confusing.
Now the message should be of the form:
The zlib-native:do_install sig is computed to be
53531910a2a7848432da89def942a91a, but the sig is locked to
d25ba9035f7ccb308e51bbe1066e8d27 in SIGGEN_LOCKEDSIGS_t-x86-64
which will hopefully be more useful in understanding the problem.
[YOCTO #9195]
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add the SIGGEN_LOCKEDSIGS_TASKSIG_CHECK and
SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK variables to replace
SIGGEN_LOCKEDSIGS_CHECK_LEVEL.
SIGGEN_LOCKEDSIGS_TASKSIG_CHECK will no control whether there is a
warning or error if a task's hash in the locked signature file doesn't match
the computed hash from the current metadata.
SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK will control whther there is a
warning or error if a task that supports sstate is in the locked
signature file, but no sstate exists for the task.
Previously you could only have warning/errors for both controlled by
SIGGEN_LOCKEDSIGS_CHECK_LEVEL. This was an issue in the extensible sdk,
because we know sstate won't exist for certain items in the reverse
dependencies list for tasks. However, we still want to error if task
signatures don't match.
[YOCTO #9195]
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Without this, do_package_write_rpm doesn't depend on rpm-native which
it really should since that is needed to build rpms.
[YOCTO #8047]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
do_shared_workdir is not a proper sstate task, it always reruns if
needed, so special-case it in warnings when checking locked sigs.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Often, bitbake -S printdiff would show that there was a checksum not found
which would turn out to be from gcc-source. This is due to it being a
shared-workdir recipe.
For now, hardcode the special case into the sstatesig code to stop people
(including me) puzzling over this.
If/as/when we add any other shared workdir recipes, we'll need to rethink
this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In SIGGEN_EXCLUDERECIPES_ABISAFE and SIGGEN_EXCLUDE_SAFE_RECIPE_DEP
we really need to be able to use virtual/xxx namespaces but this
currently doesn't work.
To make this work, we need to translate them into the resolved
providers. After such a hook was added to bitbake, we can add
this translation here.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It is useful in a few different contexts to see which files have been
written out by an sstate task; break out a function that lets us get the
path to the manifest file easily.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The logic tries to remove the -native suffix from pn to handle this (though it
doesn't succeed, as it doesn't assign the new pn to the variable), but we need
to do more for the swspec tasks than just not set the extrapath, we also need
to change from SSTATE_PKGSPEC to SSTATE_SWSPEC. Alter to correct the spec for
these cases, and also add preconfigure to align with the current logic in
sstate.bbclass, which includes that task as well in the list of tasks to
adjust to use swspec.
[YOCTO #7563]
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Require a sig file which SIGGEN_LOCKEDSIGS_i586 is not null, but
the actual SIGGEN_LOCKEDSIGS_i586 is null.
Invoking 'bitbake -e' and we got:
...
3935 # $SIGGEN_LOCKEDSIGS_i586 [2 operations]
3936 # set /path/to/locked-sigs.inc:8576
3938 # del data_smart.py:406 [finalize]
3939 # ""
...
It was caused by the following commit:
...
(Bitbake rev: 899d45b90061eb3cf3e71029072eee42cd80930c)
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Tue May 31 23:52:50 2011 +0100
bitbake/data_smart: Change overrides behaviour to remove expanded
variables from the datastore
...
We add prefix 't-' to type to workaround the overrides behaviour.
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
|
|
Using underscores in the "types" parts of the variable names can cause
unexpected issues with overrides. For example, if you have both
SSTATE_LOCKEDSIGS_x86_64 and SSTATE_LOCKEDSIGS_x86_64_i586, and i586
is in OVERRIDES, then you lose all of the contents in
SSTATE_LOCKEDSIGS_x86_64 and thus don't get some of the locked sstate.
Using '-'s in the variable names instead, eliminates these issues.
(From OE-Core rev: 6662c412a949a9f6b602c848e6303b19db7e5272)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add code in the sstate hash validation code to ensure it really did
install these from sstate since if it didn't should to warn/abort
the build. The judgment condition is:
1) If a build is replaced by locked sstate-cache, it will triger a
warn/error;
2) If objects are not used from the locked cache, it will triger a
warn/error;
3) Use SIGGEN_LOCKEDSIGS_CHECK_LEVEL variable controls whether this
is just a warning or a fatal error or nothing to report.
[YOCTO #6639]
(From OE-Core rev: 305912dce61c4fed0cbf631aa98a9e6f29db88e4)
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Instead of a single monolithic SIGGEN_LOCKEDSIGS, split this into
separate variables, one per sstate package architecture. Add in
a new SIGGEN_LOCKEDSIGS_TYPES variable which lists the package
architectures to load in.
SIGGEN_LOCKEDSIGS_TYPES is made machine specific using overrides.
Also sort the hashes in the lists by PN to make diffing them easier.
(From OE-Core rev: d8b0ce35981931a39e7db9d8e78de6e009b34688)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
I've been giving things some thought, specifically why sstate doesn't
get used more and why we have people requesting external toolchains. I'm
guessing the issue is that people don't like how often sstate can change
and the lack of an easy way to lock it down.
Locking it down is actually quite easy so patch implements some basics
of how you can do this (for example to a specific toolchain). With an
addition like this to local.conf (or wherever):
SIGGEN_LOCKEDSIGS = "\
gcc-cross:do_populate_sysroot:a8d91b35b98e1494957a2ddaf4598956 \
eglibc:do_populate_sysroot:13e8c68553dc61f9d67564f13b9b2d67 \
eglibc:do_packagedata:bfca0db1782c719d373f8636282596ee \
gcc-cross:do_packagedata:4b601ff4f67601395ee49c46701122f6 \
"
the code at the end of the email will force the hashes to those values
for the recipes mentioned. The system would then find and use those
specific objects from the sstate cache instead of trying to build
anything.
Obviously this is a little simplistic, you might need to put an override
against this to only apply those revisions for a specific architecture
for example. You'd also probably want to put code in the sstate hash
validation code to ensure it really did install these from sstate since
if it didn't you'd want to abort the build.
This patch also implements support to add to bitbake -S which dumps the
locked sstate checksums for each task into a ready prepared include file
locked-sigs.inc (currently placed into cwd). There is a function,
bb.parse.siggen.dump_lockedsigs() which can be called to trigger the
same functionality from task space.
A warning is added to sstate.bbclass through a call back into the siggen
class to warn if objects are not used from the locked cache. The
SIGGEN_ENFORCE_LOCKEDSIGS variable controls whether this is just a warning
or a fatal error.
A script is provided to generate sstate directory from a locked-sigs file.
(From OE-Core rev: 7e14784f2493a19c6bfe3ec3f05a5cf9797a2f22)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The idea of squashing packagegroup dependencies was to avoid allarch
packages rebuilding upon tune/arch changes. Now that the allarch class
inclusion is conditional, we can narrow down the packagegroup squashing
to be specifically applied to allarch recipes.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Its possible sstate symlinks to other sstate mirrors which then my get
removed/cleaned. If we find invalid symlinks, skip over them rather
than error with a backtrace.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We want to use the saferecipedeps handling code to allow gcc-cross-* to
work on multiple different tunes. Its currently in target only code
so it needs to be earlier to allow it to work on native-> target
dependencies.
This change has no effect on existing uses but makes gcc-cross become
shared as desired.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With the renaming of the cross packages, its no longer possible to use
endswith("-cross") and similar to detect cross packages. Replace these
references with other techniques.
This resolves certain build from sstate failures which were due to the
system believing cross packages were target packages and therefore
dependency handling was altered.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
There was a nasty sstate hash corruption issue occurring where the
fact the testimage bbclass was inherited meant that the checksum
changed due to testimage.bbclass being confused with image.bbclass.
This patch anchors the bbclass names to avoid this confusion.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* I don't have any real evidence or good statistics for this, but when
comparing signature dumps from my big bitbake world builds I usually
see a lot of rebuilds caused by changes in .bbclasses and only very
rare would be the case where oe-core upgrade brings changes in -native
recipes and no change in .bbclasses used from target recipes
* changing the default to include them shouldn't cause significant
increase in rebuilds and sstate reuse a bit safer
* people working on toolchain (e.g. using gcc from AUTOREV) can easily
extend sstate_rundepfilter to ignore them again (it's easier than
removing existing filter), example how add own signature handler in
your layer is here:
https://github.com/openwebos/meta-webos/commit/9ac3a7c803e7793b3274e4998f167b6278db8042
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Like fetch, unpack and patch, populate_lic doesn't vary between different
archs so we should mark it as such. This means better sstate cache reuse
with fewer duplicate files as well as less confusing sstate debugging.
sstatesig also needs to account for the fact BPN is used for sstate files
in these cases.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
By resetting filesdates at this point, we lose matches from stamps which
may not have been in sstatedir. When we don't have hashes specicifed,
its better to return all matches and have the caller decide which are
relavent and which are not since this function has no ability to
decide. There will almost always be one match from stamps we need
to keep and refer to.
(From OE-Core rev: f4c1c9ad2c7e944d4926d0629611da97f9df6a9a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
We've dropped sstate-name so we can remove this code. The fallback was
incorrect since we use taskname without the do_ prefix so this patch
updates to account for that too.
(From OE-Core rev: 72ff58124081333d46d37f31f2d1bf40d715e3bd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When the hashes to find isn't specified we need to return matches from both
the sstate cache and the local stamps directory regardless of how many we've
found so far. If we don't do this, we can miss stamps and the comparison is
less accurate/incorrect.
(From OE-Core rev: 08a074e11e2d517b81ca71fd9bda65297bb015a7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
With the "ABI safe" recipes, we've been excluding those from signatures. This
is fine in the general case but in the specific case of image recipes it breaks.
A good test case is the interfaces file. Editting this causes init-ifupdown
to rebuild but not an image containing it (e.g. core-image-minimal).
We need to ensure the checksums are added to the image recipes and this change
does that.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
packagegroups are allarch and shouldn't change depending on the target
or machine selected. In general they should have good stable namespaces
for their dependencies. As such we can exclude them from rebuilding when
dependency checksums change.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
There are some left over tab characters in the python functions. This
removes them and resolves python 3 errors.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Fixes "bitbake-diffsigs -t" for changes to the stamp directory layout,
and this time uses the actual value of STAMP to get the location of
sigdata files in the stamp directory rather than trying to do it
manually, which should be a little more robust.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add a function that can be used from BitBake code which will find
signature data (sigdata/siginfo) files based on specified criteria, and
hook it into BitBake as bb.siggen.find_siginfo.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
