| Age | Commit message (Collapse) | Author | Files |
|---|
|
It's rare but there are recipes that have individual files (as opposed
to archives) in SRC_URI using subdir= to put them under the source tree,
the examples in OE-Core being bzip2 and openssl. This broke devtool
update-recipe (and devtool finish) because the file wasn't unpacked into
the oe-local-files directory and thus when it came time to update the
recipe, the file was assumed to have been deleted by the user and thus
the file was erroneously removed. Add logic to handle these properly so
that this doesn't happen.
(We still have another potential problem in that these files become part
of the initial commit from upstream, which could be confusing because
they didn't come from there - but that's a separate issue and not one
that is trivially solved.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
When extracting patches from a git repository with PATCHTOOL = "git" we
cannot assume that all patches will be UTF-8 formatted, so as with other
places in this module, try latin-1 if utf-8 fails.
This fixes UnicodeDecodeError running devtool update-recipe or devtool
finish on the openssl recipe.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
|
|
The change to preserve extended attributes in copytree() and
copyhardlinktree() (e591d69103a40ec4f76d1132a6039d9cb1555103)
resulted in an incorrect cp invocation in copyhardlinktree() when
the source directory contained hidden files.
This was because the passed src was modified in place but some code
paths expected it to remain unmodified from the passed value.
Resolve the issue by constructing a new source string, rather than
modifying the passed in string.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Changed deployment directory from DEPLOY_DIR_IMAGE to
IMGDEPLOYDIR to make sstate machinery to do final deployment and
generate manifest.
Renamed variable deploy_dir to deploy_dir_image in selftest code
to avoid confusion with DEPLOYDIR variable.
Updated the code of rootfs.py:Rootfs class to use IMGDEPLOYDIR variable
as it's now used as a new deployment destination.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When using PATCHTOOL = "git", the user of the system is not really the
committer - it's the build system itself. Thus, specify "dummy" values
for username and email instead of using the user's configured values.
Various parts of the devtool code that need to make commits have also
been updated to use the same logic.
This allows PATCHTOOL = "git" and devtool to be used on systems where
git user.name / user.email has not been set (on versions of git where
it doesn't default a value under this circumstance).
If you want to return to the old behaviour where the externally
configured user name / email are used, set the following in your
local.conf:
PATCH_GIT_USER_NAME = ""
PATCH_GIT_USER_EMAIL = ""
Fixes [YOCTO #8703].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Pass appropriate options to tar invocations in copytree() and
copyhardlinktree() to ensure that any extended attributes on the files
are preserved during the copy.
We have to drop the use cpio in "Copy-pass" mode in copyhardlinktree()
because cpio doesn't support extended attributes on files. Instead we
revert back to using cp with different patterns depending on whether
or not the directory contains dot files.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Current functionality allows for the removal of certain packages
based on the read-only image feature. This patch extends this
functionality by adding the FORCE_RO_REMOVE variable, which will
remove these packages regardless of any image features.
[ YOCTO #9491 ]
Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
LICENSE should be a superset of all LICENSE_<pkg> values. That is,
LICENSE should contain all licenses and LICENSE_<pkg> can be used to
"filter" this on a per-package basis. LICENSE_<pkg> shouldn't contain
anything that isn't specified in LICENSE.
This patch implements simple checking of LICENSE_<pkg> values. It does
do not do advanced parsing/matching of license expressions, but,
checks that all licenses mentioned in LICENSE_<pkg> are also specified in
LICENSE. A warning is printed if problems are found.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
multi-configuration builds
Unfortunately to implenent multiconfig support in bitbake some APIs
had to change. This updates code in OE to match the changes in bitbake.
Its mostly periperhal changes around devtool/recipetool
[Will need a bitbake version requirement bump which I'll make when merging]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When we don't have uninative enabled there's more merging to be done in
the default configuration (SDK_EXT_TYPE = "full" which by default means
SDK_INCLUDE_TOOLCHAIN = "1") and there are likely files that already
exist in the sstate feed we're assembling, so we need to take care to
merge the directory contents rather than just moving the directories
over. Additionally we now only run this if uninative genuinely isn't
enabled (i.e. NATIVELSBSTRING is different to the fixed value of
"universal".)
In the process of fixing this I discovered an unusual behaviour in
os.rename() - when we're merging these feeds we're dealing with
hard-linked sstate artifacts, and whilst os.rename() is supposed to
silently overwrite an existing destination (permissions allowing), if
you have the source and destination as hardlinks to the same file then
the os.rename() call will just silently fail. As a result the code now
just checks if the destination exists and deletes the source if so
(since we know it will be the same file, we don't need to check in this
case.)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Give each rootfs its own RPM channel to use. This puts the RPM metadata
in a private subdirectory of $WORKDIR, rather than living in DEPLOY_DIR
where other tasks may race with it.
This allows us to reduce the time that the rpm.lock is held to only the
time needed to hardlink the RPMs, allowing the majority of the rootfs
operation to run in parallel.
Also, this fixes the smart tests by generating an index for all packages
at the time of the test, rather than using the one provided by the
rootfs process.
Original credit for the enhancement should go to Steven Walter
stevenrwalter@gmail.com.
Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Monitoring the process started by gnome-terminal was
spinning in a busy-loop. Insert some sleeping so that
we don't eat all the cpu.
Signed-off-by: Jacob Kroon <jacob.kroon@gmail.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
If you supplied an empty file to patch_recipe() (or an empty list to
patch_recipe_lines()) then the result was IndexError because the code
checking to see if it needed to add an extra line of padding didn't
check to see if there were in fact any lines before trying to access the
last line.
Fixes [YOCTO #9972].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use the new oe-check-sstate to filter the sstate artifacts shipped with
the extensible SDK by effectively running bitbake within the produced
eSDK and and getting it to tell us which tasks it will restore from
sstate. This has several benefits:
1) We drop the *-initial artifacts from the minimal + toolchain eSDK.
This still leaves us with a reasonably large SDK for this
configuration, however it does pave the way for future reductions
since we are actually filtering by what will be expected to be there
on install rather than hoping that whatever cuts we make will match.
2) We verify bitbake's basic operation within the eSDK, i.e. that
we haven't messed up the configuration
3) We verify that the sstate artifacts we expect to be present are
present (at least in the sstate cache for the build producing the
eSDK). Outside deletion of sstate artifacts has been a problem up to
now, and this should at least catch that earlier i.e. during the
build rather than when someone tries to install the eSDK.
This does add a couple of minutes to the do_populate_sdk_ext time, but
it seems like the most appropriate way to handle this.
Should mostly address [YOCTO #9083] and [YOCTO #9626].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If we're to completely replace the standard SDK with the extensible SDK,
we need to be able to provide the standard toolchain on install without
doing anything other than installing it, so that you can install the SDK
and then point your IDE at it. This is particularly applicable to the
minimal SDK which normally installs nothing by default.
NOTE: enabling this option currently adds ~280MB to the size of the
minimal eSDK installer. If we need to reduce this further we would have
to look at adjusting the dependencies and/or the sstate_depvalid()
function in sstate.bbclass which eliminates dependencies, or look at
reducing the size of the artifacts themselves.
Implements [YOCTO #9751].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
tar's --no-recursion flag only applies to files mentioned after the
flag, which made it a no-op in this invocation of tar, because it was at
the end of the command line.
This is simple to verify with GNU tar 1.29:
| $ mkdir foo
| $ mkdir foo/dir
| $ touch foo/dir/file
| $ tar -cf - foo --no-recursion | tar t
| foo/
| foo/dir/
| foo/dir/file
| $ tar -cf - --no-recursion foo | tar t
| foo/
Modify the code so that it actually does what the comment says by moving
the flag in front of the --files-from argument.
Signed-off-by: Clemens Lang <clemens.lang@bmw-carit.de>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
oe.utils.getstatusoutput() is a wrapper for subprocess.getstatusoutput() which
uses Universal Newlines, so the output is a str() not bytes().
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
It is possible in an attempt only install, that everything listed is not
available to be installed. This will have the effect of clearing the
package list. However, we only check for an empty package list at
the beginning of the function. We need to also check before running the
install, otherwise we can fail due to 'error: no package(s) given".
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Mingli Yu <mingli.yu@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When using PR service the buildhistory-diff output contains a lot of
PKGR changes: In practice the mass of PKGR updates hide other important
changes as they often account for 80% of all changes.
Skipped incremental and decremental changes of PKGR versions to reduce
amount of the script output. All changes are still included in the
output if script is run with -a/--report-all command line option.
[YOCTO #9755]
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This task runs all functions in IMAGE_QA_COMMANDS after the image
construction has completed in order to validate the resulting image.
Image sanity checks should either be Python functions which raise
bb.build.FuncFailed on failure or shell functions with return a
non-zero exit code.
Python functions may instead raise an oe.utils.ImageQAFailed
Exception which takes an extra argument, a description of the
failure.
python image_check_python_ok () {
if True:
raise bb.build.FuncFailed('This check always fails')
else:
bb.note("Nothing to see here")
}
image_check_shell_ok () {
if true
exit 1
else
exit 0
fi
}
[YOCTO #9448]
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Add a subcommand which will "finish" the work on a recipe. This is
effectively the same as update-recipe followed by reset, except that the
destination layer is required and it will do the right thing depending
on the situation - if the recipe file itself is in the workspace (e.g.
as a result of devtool add), the recipe file and any associated files
will be moved to the destination layer; or if the destination layer is
the one containing the original recipe, the recipe will be overwritten;
otherwise a bbappend will be created to apply the changes. In all cases
the layer path can be loosely specified - it could be a layer name, or
a partial path into a recipe. In the case of upgrades, devtool finish
will also take care of deleting the old recipe.
This avoids the user having to figure out the correct actions when
they're done - they just do "devtool finish recipename layername" and
it saves their work and then removes the recipe from the workspace.
Addresses [YOCTO #8594].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This provides us with the information we need to remove the original
version recipe and associated files when running "devtool finish" after
"devtool upgrade".
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
* Allow the function to be called with the base layer path (in which
case it will just return the same path)
* Ensure that the function doesn't recurse indefinitely if it's called
on a file that's not inside a layer
* Correct the doc comment for accuracy
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
During a parallel build it's possible for unrelated shlib files to be removed if
the recipe they came from is about to be rebuilt. They can't be involved in the
dependency chains as otherwise they wouldn't be removed, so just silently handle
files disappearing.
[ YOCTO #8555 ]
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
With Python 3, the encoding of a file is significant; several recipes in
OE-Core have patches which are not fully utf-8 decodable e.g. man,
lrzsz, and gstreamer1.0-libav, leading to errors when using devtool's
modify, upgrade or extract subcommands on these recipes. To work around
this, try reading the patch file as utf-8 first and if that fails try
latin-1 before giving up.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
With the new pid monitoring code we have for recent versions of
gnome-terminal we can just drop the --disable-factory code now since
the other solution handles this case as well.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Currently gnome-terminal just returns straight away, opening a terminal in a new
separate process we have no insight into. For patch resolution, this leads to
spawning many different terminal windows, for pydevshell, it just flashes a window
up and then closes.
We need to block until the command completes but gnome-terminal gives us no way
to do this. We therefore write the pid to a file using a "phonehome" wrapper
script, then monitor the pid until it exits.
[YOCTO #7254]
(also fixing do_devpyshell)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This was a copy-and-paste of subprocess.check_output() from when we supported
Python <2.7, so simply delete it and use subprocess.check_output() instead.
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
If we're building the extensible SDK we don't need to see the "Writing
locked sigs" message; it's only necessary when the user explicitly runs
bitbake -S none <target>.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Use the new task progress functionality to report progress during
do_rootfs. This is a little coarse and ideally we would have some
progress within the installation section, but it's better than
nothing.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
ordered
In RpmPM:insert_feeds_uris, the paths are kept in sets, which are unordered,
but they are later used to set the priority for the Smart channels, so
unexpected results could occur. Change the sets to lists and use the same
code as in create_configs() to add items to the list, rather than the set
operators.
[YOCTO #9717]
Signed-off-by: Bill Randle <william.c.randle@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Specify tmpdir for opkg via '-t' option so that opkg does not use
the default 'TMPDIR' which usually is '/tmp' on build host.
This would solve race problems like below.
sh: /tmp/opkg-rOG6Tl/opkg-intercept-iPoEp5/depmod: Permission denied
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Order is not preserved in dict() and this code depends on the order of
these lists of package architectures used when multilibs are enabled.
This caused 'random' breakage where sometimes the correct order was present
and sometimes it wasn't.
Use collections.OrderedDict() to avoid this problem.
Kudos to Bill Randle and Alejandro Hernandez who did most of the work debugging
this, I simply took the problem they identified and wrote a patch to fix it.
This unblocks the M1 build but this code needs auditing as there are clearly
other ordering issues (e.g. the set() usage).
[YOCTO #9717]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
create_socket: Use urllib because urllib2 is now urllib in python3
and proxies as argument are deprecated so export them in the environ
instead.
get_links_from_url: Change usage of sgmllib for parsing HTML because
is deprecated in python 3, use instead bs4 that is already imported
in the bitbake tree.
[YOCTO #9744]
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
On python 3, bytes variable types must be decoded if these are intended to be
used as strings, otherwise we get the following error exception:
TypeError: Type str doesn't support the buffer API
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Adds decoding needed by some commands output later used as
strings.
[YOCTO #9702]
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
string.rstrip() doesn't exist in Python 3, so use the .rstrip method on the
object itself instead.
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The read method of the data_stream File object now returns bytes,
not a str, so we must decode it.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The bytes type variables are threat different as strings variables in python3,
so convert bytes to strings. This was found when using the
script buildhistory-diff, where the interpreter was yielding this error
TypeError: Type str doesn't support the buffer API
Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This patch contains all the other misc pieces of the transition to
python3 which didn't make sense to be broken into individual patches.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Moved call of decode('utf-8') as close as possible to
call of subprocess API to avoid calling it in a lot of
other places.
Decoded binary data to utf-8 where appropriate to fix devtool
and recipetool tests in python 3 environment.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
|
|
Replaced iteritems -> items, itervalues -> values,
iterkeys -> keys or 'in'
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
|
|
xrange() no longer exists in python 3, use range()
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Iterators now return views, not lists in python3. Where we need
lists, handle this explicitly.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
python3 standardises its use of iteration operations. Update
the code to match the for python3 requires.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
In python3, strings are unicode by default. We need to encode/decode
from command pipelines and other places where we interface with the
real world using the correct locales. This patch updates various
call sites to use the correct encoding/decodings.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The syntax for octal values changed in python3, adapt to it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Add some more variables in appropriate places in recipe_progression such
that the patch_recipe_* functions are able to insert variables in the
right place within a recipe.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This function was assuming that what you wanted was that output lines
had trailing newline characters. If you're just outputting each line
verbatim to a text file then that's fine, but sometimes you start with
the assumption that the lines don't have trailing newlines; thus we
shouldn't allow for the possibility that the caller doesn't want them
and add a parameter to control it.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
