| Age | Commit message (Collapse) | Author | Files |
|---|
|
The syntax for octal values changed in python3, adapt to it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Previously the sstate was all downloaded to the same directory and then
symlinks were added in the directories that pointed to the siginfo and
sstate in the parent directory.
This change makes it so that now the files are just downloaded to the
correct location without the need for symlinks.
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This avoids 'basehash changed' errors with python 3 but could break
build determinism in general.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add the SIGGEN_LOCKEDSIGS_TASKSIG_CHECK and
SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK variables to replace
SIGGEN_LOCKEDSIGS_CHECK_LEVEL.
SIGGEN_LOCKEDSIGS_TASKSIG_CHECK will no control whether there is a
warning or error if a task's hash in the locked signature file doesn't match
the computed hash from the current metadata.
SIGGEN_LOCKEDSIGS_SSTATE_EXISTS_CHECK will control whther there is a
warning or error if a task that supports sstate is in the locked
signature file, but no sstate exists for the task.
Previously you could only have warning/errors for both controlled by
SIGGEN_LOCKEDSIGS_CHECK_LEVEL. This was an issue in the extensible sdk,
because we know sstate won't exist for certain items in the reverse
dependencies list for tasks. However, we still want to error if task
signatures don't match.
[YOCTO #9195]
Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
At present, if a recipe is built which creates users/groups via
useradd.bbclass, those users/groups are not removed from sysroot
when the recipe/package is cleaned using clean/cleansstate/cleanall
or when a recipe is rebuild and 'unstaged' from the the sysroot.
The "userdel_sysroot_sstate()" provides that functionality.
[YOCTO #9262]
Signed-off-by: Maxin B. John <maxin.john@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
uninative needs to adjust NATIVELSBSTRING fairly late in the
configuration parsing process but the sstate code encodes it into
variables. Since this string doesn't vary on a per recipe basis, we
defer its expansion until usage time.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
[YOCTO #9006]
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
do_cleansstate wasn't cleaning do_populate_lic sstate objects in the
native/cross case since the wildcard path entry wasn't being cleared
at the same time as the path extra prefix. Fix by clearing it at the
same time.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This variable is only used by populate_lic tasks (gcc-source overrides it)
and refers to BPN. In recipes like gcc, where there are multiple variants,
it is resulting in sstate objects which encode PN (they install into a PN subdir)
but the sstate object reflects BPN. This leads to corruption between then and
eventually, warnings from image builds like:
WARNING: The license listed GPLv3 was not in the licenses collected for recipe gcc
WARNING: The license listed GPL-3.0-with-GCC-exception was not in the licenses collected for recipe gcc
WARNING: The license listed GPLv3 was not in the licenses collected for recipe gcc
WARNING: The license listed NCSA was not in the licenses collected for recipe gcc-sanitizers
WARNING: The license listed MIT was not in the licenses collected for recipe gcc-sanitizers
WARNING: The license listed GPL-3.0-with-GCC-exception was not in the licenses collected for recipe gcc
WARNING: The license listed GPLv3 was not in the licenses collected for recipe gcc
WARNING: The license listed NCSA was not in the licenses collected for recipe gcc-sanitizers
By referring to PN, as used by license.bbclass, this issue is resolved.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=2d89cff42af2bb0049224bfaaebaa2b21966169f
we added a workaround for dealing with lack of time sync between build
machines and their users.
This has turned out to cause problems for people who rely on timestamps
being preserved in sstate output.
Since our autobuilders are all in time sync with ntp, revert the commit.
[YOCTO #8996]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
useradd has sstate [depends] for both do_package and do_populate_sysroot
yet the dependency validation code only covers do_package.
Add coverage of populate_sysroot, else the order inversion that [depends]
creates means unexpected installation of users of useradd.bbclass (e.g.
avahi do_populate_sysroot) in cases where it shouldn't be (e.g.
libnss-mdns -c packagedata).
The code needs to move above the other populate_sysroot intercept code
since there are specific cases we need to cover before that code.
The result of this change is more optimal installation of sstate objects
in common usage scenarios.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
When cleaning old builds from the sysroots, also print the sysroot architecture.
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
If there are lots of changes between the previous build and the build about to
start bitbake will potentially print pages of:
DEBUG: Stamp $BUILD/stamps/corei7-64-poky-linux/libdrm/2.4.66-r0 is not reachable, removing related manifests
Instead of spamming the console with this list, write the list of manifests only
to the debug log and simply write a count to the console. This way the user
doesn't get spammed but still knows what is happening if bitbake appears to
stall with heavy I/O.
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
If we "bitbake X -c packagedata" and the packagedata comes from sstate, we
don't need any of the tasks dependencies. This is similar to the
populate_lic case, we only care about the end result.
Therefore short circuit the dependencies so packagedata doesn't pull in
any other dependencies.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It was observed that do_fetch dependencies (e.g. subversion-native of tremor)
were being installed even when sstate was available and hence no fetch was
needed. This turned out to be due to the recursive nature of the rootfs
dependencies which include populate_lic.
We can explicitly whitelist these dependencies as being ones we don't
need to follow when installing sstate (the code defaults to being paranoid).
[YOCTO #8746]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Maintaining an sstate mirror is complicated at the moment as only the main
sstate tgz files have their timestamp updated when they're used.
This causes problems as the siginfo files can be removed from the server
due to inactivity but the sstate fetch code tries to fetch them leading
to confusing warnings and reduced debug capability.
This change ensures we touch any sig/siginfo files present and should
help ageing of the objects on sstate mirrors. It doesn't handle the intermediate
task siginfo files but those are never fetched by the current sstate code
so are an order of magnitude less problematic.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The package feed signing code supports the user providing the path to the gpg
binary and an alternative gpg 'home' (usually ~/.gnupg), which are useful for
both deployment and QA purposes.
Factor out the gpg command line construction to a function which can fetch both
of these variables, and also use pipes.quote() to sanitise the arguments when
used in a shell context.
[ YOCTO #8559 ]
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
To avoid races between the sstate tasks/hooks using ${B} as the cwd, and other
tasks such as cmake_do_configure which deletes and re-creates ${B}, ensure that
all sstate hooks are run in the right directory, and run the prefunc/postfunc in WORKDIR.
(From OE-Core rev: 1d3bde02641f4b40030cf7e305ee3d7c2faabe29)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
It is useful in a few different contexts to see which files have been
written out by an sstate task; break out a function that lets us get the
path to the manifest file easily.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Only fetch remote signatures if verification has been enabled, as otherwise the
fetcher throws errors that sstate.bbclass can't ignore.
[ YOCTO #8265 ]
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
To provide some element of integrity to sstate archives, allow sstate archives
to be GPG signed with a specified key (detached signature to a sidecar .sig
file), and verify the signatures when sstate archives are unpacked.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
* Rename from gtk-update-icon-cache to gtk-icon-utils to better
reflect the content. Fix references in other recipes and classes
* Upgrade to GTK+ 3.16.4
* Add gtk-encode-symbolic-svg binary: it is used by icon themes
(e.g. Adwaita) to generate png versions of svg icons.
* Depend on librsvg-native for gtk-encode-symbolic-svg
* Add a patch that removes Gdk dependency from gtk-encode-symbolic-svg:
this way the native build stays slim.
Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Update sstate_create_package and sstate_unpack_package to remove
redundant initial mkdir and cd commands. The working directory is
now setup correctly before the shell functions are called.
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Use FetcherConnectionCache to improve times when do checkstatus over
sstate resources.
[YOCTO #7796]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Add init/end helper functions for ThreadWorker also pass ThreadWorker
as first argument to init/end/func functions this enables per-thread
storage handling.
classes/sstate.bbclass: Add thread_worker argument to checkstatus
function.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In order to fix Thread leakage caused by not call join() in Threads,
Pass num_tasks in ThreadPool for add all the tasks into a Queue this
enable catch of Queue.Empty exception and exit the threads.
classes/sstate.bbclass: Change checkstatus function to match new
ThreadPool operation.
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Rather than just use d.getVar(X), use the more explict d.getVar(X, False)
since at some point in the future, having the default of expansion would
be nice. This is the first step towards that.
This patch was mostly made using the command:
sed -e 's:\(getVar([^,()]*\)\s*):\1, False):g' -i `grep -ril getVar *`
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
"Incremental builds do not work well when renaming recipes or changing
architecture" is a long standing issue which causes people considerable
pain. We've struggled for a long time to come up with a way to
generically address the problem.
There are additional issues where removal of a layer caused data to
continue to exist and additionally, changing DISTRO_FEATURES also caused
problems in an existing TMPDIR.
This patch attempts to address this by adding a mapping between stamp
files and manifests. After parsing we can easily tell which stamp files
are still reachable, if any manifest has a stamp that can no longer be
reached, we can remove it. Since this code ties this to the sstate
architecture list, it will not remove data from other than the current
MACHINE (and its active architectures). It does not clean the sstate
cache so if another build activates something which was cleaned, it
should reinstall from sstate.
We can also go one step further, depending on the setting of
SSTATE_PRUNE_OBSOLETEWORKDIR, workdirs which are no longer active can
also be removed. This avoids the buildup of many old copies of data in
WORKDIR for example when versions are upgraded.
The one thing which may surprise people with this change is if you
remove a layer, data added by that layer will be "uninstalled" before
the next build continues. I believe this is a feature and a good thing
to do though.
This code is safe with existing builds. If something isn't in the new
index it simply isn't removed. Since changes to the sstate code trigger
a rebuild, after this merges, we can assume the code will start to
detect changes from that point onwards.
[YOCTO #4102]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The build can pause whilst remote sstate mirrors are checked for
sstate objects. Inform the user this is happening.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently the urls are checked serially which is a performance bottleneck when
looking at http:// urls in particular. This adds code to check the url status in
parallel, mirroring the way we do this elsewhere.
We need the datastore for the fetcher so we use threads, not multiprocess.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Changing TMPDIR and rebuilding an image was resulting in rebuilds of
kernels due to dependencies on the shared_workdir task. If installed
from sstate, nothing needs this task so add it to the whitelisted
task patterns.
After this change, the kernel does not rebuild when a new TMPDIR and
hot sstate cache is used.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In some cases we want to test the availability of siginfo files, in some
cases we do not and really want the .tgz files (which may or may
not be present too). This makes adds a parameter to the function to allow
this.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
FILESPATH was only being overridden in one fetch location, it should be
equally handled in both.
Also use SSTATE_DIR as FILESPATH so that mirror urls which do remapping
can search the local SSTATE_DIR for other paths.
Also ensure that MIRRORS is removed in both locations, previously
it was only unset in one but both codepaths should be consistent.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Without this patch the source archiver class is not allowed to
archive the same source archive for different builds.
Signed-off-by: Adrian <adrian.freihofer@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Commit e9672387 split one long line into a multi-line string, but in
the process white space between words was lost. This results in badly
formatted output when this message is printed.
Signed-off-by: Peter Urbanec <openembedded-devel@urbanec.net>
|
|
For some recipes that inhrient cmake, the ${B} may be removed by
cmake_do_configure() while sstate_hardcode_path() running, this
causes build errors:
Exception: OSError: [Errno 2] No such file or directory: \
'/path/to/build'
The function sstate_hardcode_path() called command:
$SSTATE_SCAN_CMD which extended as "find ${SSTATE_BUILDDIR} ..."
So the proper function dirs could be ${SSTATE_BUILDDIR}.
Signed-off-by: Wenzong Fan <wenzong.fan@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The too long line would cause "git send-email" report errors:
patch contains a line longer than 998 characters
Though we can use "--no-validate" to force the send.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
The code:
bb.fatal("foo1")
bb.fatal("foo2")
Would make the second one not work, use bb.error for first one to fix
the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
|
|
The packagegroup allarch rpm files for multilib can overwrite each other since
they are in theory indentical (in contrast to the other backends). We therefore
need to whitelist this to avoid build failures now this overwrite failure
is fatal.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Update the sstate file's timestamps after it is installed, it will be
very useful for removing the old sstate file, especially, it's not easy
to remove when use the shared SSTATE_DIR, we can easily remove them with
this change, for example:
$ find state-cache -type f -ctime +10 -exec rm -f {} \;
Will remove the sstate file which isn't used by recent 10 days.
We can use the -atime, but it is not always available, for example,
when mounted with "-o noatime".
The touch is a very light weight action, and the
scripts/sstate-cache-management.sh also requires this.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
When files overlap in the sysroot, something bad usually happened. We've had
two independent cases recently where a couple of months after one of these
warnings was shown, builds failed due to corruption.
This change moves the warning to become a fatal error. The complaint I've had
about this is that we need to tell the user what happened and more importantly
how to recover from it. If we could recover from it, great but the trouble is
we simply don't know what happened.
As a compromise, we can document several of the possible scenarios in the error
message. We don't normally go to this level of detail however in this case, I'm
lacking other viable alternatives.
I do believe it is important to stop as corruption occurs rather than letting the
build contunue into territory that is not deterministic amongst other things.
The complex message is followed by a simpler one in case the long message is too
much for the user.
(From OE-Core rev: 179ac7de03977b6e440409eddb2166819e07286a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
|
|
I've been giving things some thought, specifically why sstate doesn't
get used more and why we have people requesting external toolchains. I'm
guessing the issue is that people don't like how often sstate can change
and the lack of an easy way to lock it down.
Locking it down is actually quite easy so patch implements some basics
of how you can do this (for example to a specific toolchain). With an
addition like this to local.conf (or wherever):
SIGGEN_LOCKEDSIGS = "\
gcc-cross:do_populate_sysroot:a8d91b35b98e1494957a2ddaf4598956 \
eglibc:do_populate_sysroot:13e8c68553dc61f9d67564f13b9b2d67 \
eglibc:do_packagedata:bfca0db1782c719d373f8636282596ee \
gcc-cross:do_packagedata:4b601ff4f67601395ee49c46701122f6 \
"
the code at the end of the email will force the hashes to those values
for the recipes mentioned. The system would then find and use those
specific objects from the sstate cache instead of trying to build
anything.
Obviously this is a little simplistic, you might need to put an override
against this to only apply those revisions for a specific architecture
for example. You'd also probably want to put code in the sstate hash
validation code to ensure it really did install these from sstate since
if it didn't you'd want to abort the build.
This patch also implements support to add to bitbake -S which dumps the
locked sstate checksums for each task into a ready prepared include file
locked-sigs.inc (currently placed into cwd). There is a function,
bb.parse.siggen.dump_lockedsigs() which can be called to trigger the
same functionality from task space.
A warning is added to sstate.bbclass through a call back into the siggen
class to warn if objects are not used from the locked cache. The
SIGGEN_ENFORCE_LOCKEDSIGS variable controls whether this is just a warning
or a fatal error.
A script is provided to generate sstate directory from a locked-sigs file.
(From OE-Core rev: 7e14784f2493a19c6bfe3ec3f05a5cf9797a2f22)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The "grep -e (x|y)" doesn't work, for example:
$ echo xy | grep -e '(x|y)'
No output
We can use "grep -E" (extended regexp) or "grep -e x -e y" to fix it.
It only affected the cross recipes.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The use of [ and && here means $? is reset and the exit 1 error
interception wasn't working, leading to "file changed as we read it"
errors from sstate_create_package when heavily using hardlinks.
Fix this by placing $? into a variable.
(From OE-Core rev: 6e51f900b76b06c09a3d6927f8db7398e2c035ed)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In some cases we do either need to add extra sstate manipulation
functions, or change the existing modification functions. This patch
parametrizes it to SSTATEPOSTCREATEFUNCS after sstate_create_package
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In some cases we do either need to add extra sstate manipulation
functions, or change the existing modification functions. This patch
parametrises them to SSTATECREATEFUNCS and SSTATEPOSTUNPACKFUNCS and
abstracts the "hardcoded path" functions into separate functions using
these new variables.
We may use this new functionality to improve binary relocating using
patchelf for example, this at least lets us have the hooks to be able to
experiment.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
