getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.

Search made with the following regex: getVar ?\(( ?[^,()]*), True\)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>

When there is a relative symlink in the layer, for example:
symA -> ../out/of/layer/file

symA will be invalid fater copied, it would be invalid from build time
if it points to a relative path, and would be invalid after extracted
the sdk if it points to a absolute py. Dereference symlink when copy
will fix the problem.

Use tar rather than shutil.copytree() to copy is because:
1) shutil.copytree(symlinks=Fasle) has bugs when dereference symlinks:
   https://bugs.python.org/issue21697
   And Ubunutu 1404 doesn't upgrade python3 to fix the problem.

2) shutil.copytree(symlinks=False) raises errors when there is a invalid
   symlink, and tar just prints a warning, tar is preferred here since
   the real world is unpredicatable

3) tar is faster than shutil.copytree() as said by oe.path.copytree()

So use tar to copy.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>

Indirect paths (e.g. ${TOPDIR}/../meta-something) do generally work if
used in BBLAYERS in bblayers.conf. However, if you built an extensible
SDK with this configuration then the creation of the workspace within
the SDK using devtool in do_populate_sdk_ext failed. This is because
the copy_buildsystem code was no longer correctly recognising that the
core layer ("meta") was part of a repository (e.g. openembedded-core /
poky) that should be shipped together - because of the indirection - and
thus it was splitting out the meta directory, and a number of places in
the code assume that the meta directory is next to the scripts
directory. Use os.path.abspath() to flatten out any indirections.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

When we don't have uninative enabled there's more merging to be done in
the default configuration (SDK_EXT_TYPE = "full" which by default means
SDK_INCLUDE_TOOLCHAIN = "1") and there are likely files that already
exist in the sstate feed we're assembling, so we need to take care to
merge the directory contents rather than just moving the directories
over. Additionally we now only run this if uninative genuinely isn't
enabled (i.e. NATIVELSBSTRING is different to the fixed value of
"universal".)

In the process of fixing this I discovered an unusual behaviour in
os.rename() - when we're merging these feeds we're dealing with
hard-linked sstate artifacts, and whilst os.rename() is supposed to
silently overwrite an existing destination (permissions allowing), if
you have the source and destination as hardlinks to the same file then
the os.rename() call will just silently fail. As a result the code now
just checks if the destination exists and deletes the source if so
(since we know it will be the same file, we don't need to check in this
case.)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>

Use the new oe-check-sstate to filter the sstate artifacts shipped with
the extensible SDK by effectively running bitbake within the produced
eSDK and and getting it to tell us which tasks it will restore from
sstate. This has several benefits:

1) We drop the *-initial artifacts from the minimal + toolchain eSDK.
   This still leaves us with a reasonably large SDK for this
   configuration, however it does pave the way for future reductions
   since we are actually filtering by what will be expected to be there
   on install rather than hoping that whatever cuts we make will match.

2) We verify bitbake's basic operation within the eSDK, i.e. that
   we haven't messed up the configuration

3) We verify that the sstate artifacts we expect to be present are
   present (at least in the sstate cache for the build producing the
   eSDK). Outside deletion of sstate artifacts has been a problem up to
   now, and this should at least catch that earlier i.e. during the
   build rather than when someone tries to install the eSDK.

This does add a couple of minutes to the do_populate_sdk_ext time, but
it seems like the most appropriate way to handle this.

Should mostly address [YOCTO #9083] and [YOCTO #9626].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

If we're to completely replace the standard SDK with the extensible SDK,
we need to be able to provide the standard toolchain on install without
doing anything other than installing it, so that you can install the SDK
and then point your IDE at it. This is particularly applicable to the
minimal SDK which normally installs nothing by default.

NOTE: enabling this option currently adds ~280MB to the size of the
minimal eSDK installer. If we need to reduce this further we would have
to look at adjusting the dependencies and/or the sstate_depvalid()
function in sstate.bbclass which eliminates dependencies, or look at
reducing the size of the artifacts themselves.

Implements [YOCTO #9751].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Iterators now return views, not lists in python3. Where we need
lists, handle this explicitly.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

python3 standardises its use of iteration operations. Update
the code to match the for python3 requires.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

When creating an eSDK ensure that any .git directories are not included.

[ YOCTO #9426 ]

Signed-off-by: Stephano Cetola <stephano.cetola@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Add a build-sdk command which is only available within the extensible
SDK that builds a derivative extensible SDK. The idea is recipes in the
workspace become a part of the new SDK - for example, this allows taking
a vendor provided SDK, adding a few libs and then producing a new SDK
with those included.

When normally building the extensible SDK, the workspace is excluded;
here we need to copy into the new SDK (renaming it in the process); the
recipes' task signatures become locked and thus the sources are no
longer needed, so they are removed along with the workspace bbappends
which would interfere with the locked signatures. Additionally we need
to just copy the configuration files (i.e. local.conf and auto.conf)
rather than filtering and appending to them since that work has already
been done when constructing the original SDK. The extra sstate artifacts
from workspace recipes are also determined and copied into the new SDK
in minimal mode (on the assumption that you won't set up a new sstate
mirror).

This reuses some code from build-image, so that needed to be
generalised to allow that.

Implements [YOCTO #8892].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>