If the user modifies files such as CMakeLists.txt in the case of cmake,
we want do_configure to re-run so that those changes can take effect. In
order to accomplish that, have a variable CONFIGURE_FILES which
specifies a list of files that will be put into do_configure's checksum
(either full paths, or just filenames which will be searched for in the
entire source tree). CONFIGURE_FILES then just needs to be set
appropriately depending on what do_configure is doing; for now I've set
this for autotools and cmake which are the most common cases.

Fixes [YOCTO #7617].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

To make sure changes to any source files are detected when externalsrc
is used, it sets BB_DONT_CACHE to force the recipe to be reparsed
every time. Previously, this was done conditionally based on whether
EXTERNALSRC was set. This worked fine for building the base recipe.
But if one tried to build, e.g., a native version of it (provided via
BBCLASSEXTEND), the recipe would not be reparsed as expected.

To solve the above problem, BB_DONT_CACHE is now set for the base
recipe if EXTERNALSRC is set for it or any of it derivatives.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

In python3, strings are unicode by default. We need to encode/decode
from command pipelines and other places where we interface with the
real world using the correct locales. This patch updates various
call sites to use the correct encoding/decodings.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Use a unique tempfile as the temporary git index file when determining
the git hash of the source tree. A fixed filename was obviously causing
races (with the git index.lock file) under oe-selftest where multiple
bitbake instances were run against the same source tree at the same
time.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Extend the srctree_hash_files() function in externalsrc.bbclass to
handle non-Git source trees.  If the source tree is not a git
repository, srctree_hash_files() now simply adds the whole source tree
as a dependency, causing bitbake to basically hash every file in it.
Hidden files and directories in the source tree root are ignored by the
glob currently used. This has the advantage of automatically ignoring
.git directory, for example.

During the first bitbake run preparing of the task runqueue may take
much longer if the source tree is not a git repository. The reason is
that all the files in the source tree are hashed.  Subsequent builds are
not significantly slower because (most) file hashes are found from the
cache.

[YOCTO #8853]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Be a bit more intelligent than mindlessly re-compiling every time.
Instead of always using 'nostamp' flag for do_compile run a python
function to get a list of files to add as 'file-checksums' flag. The
intention is to only re-run do_compile if something in the source tree
content changes.

This python function, srctree_hash_files(), works differently, depending
if the source tree is a git repository clone or not. If the source tree
is a git repository, the function runs 'git add .' and 'git write tree'
to get a hash of the working tree and writes this hash into a file under
the .git directory. This file containing the hash is then returned as
the file for the task to depend on. Hash is changed if any changes are
made in the source tree causing the task to be re-run. A trick is used
to parse the recipe every time so that the hash file gets updated.

If the source tree is not a git repository behaviour remains the same.
In this case srctree_hash_files() currently sets the 'nostamp' flag for
do_compile causing it to be re-run every time.

This method of tracking changes source tree changes to determine if
re-build is needed does not work perofectly, though. Many packages are
built under ${S} which effectively changes the source tree causing some
unwanted re-compilations.  However, if do_compile of the recipe does not
produce new/different artefacts on every run (as commonly is and should
be the case) the re-compilation loop stops. Thus, you should usually see
only one re-compilation (if any) after which the source tree is
"stabilized" and no more re-compilations happen.

[YOCTO #8853]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

We have a common build directory for all target architectures if
externalsrc is used and ${B}=${S}. In this case we also need to have a
common stamp directory. The reason is that e.g. changing MACHINE will
basically "invalidate" the artefacts in ${B} (wrt. to the new MACHINE)
but old stamp files with matching hashes from an earlier build might be
available in the machine-specific stamps directory and tasks would not
be correctly re-run.

Previous attempt in f44f12af346888bdeb3ae01a275cb5dd4396b505 to handle
this correctly was faulty as it threw some spurious warnings and caused
do_configure to be re-run in every build if that would not be necessary.

[YOCTO #8950]
[YOCTO #9237]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>

If the oe-workdir / oe-logs symlink exists and points to a different
path then the symlink needs to be removed before calling os.symlink() or
it'll fail.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Auto-create symlinks in the source directory to the work directory
(${WORKDIR}) and logs directory (${T}) so that they are easier for the
user to find. This is particularly useful within the extensible SDK
where the user is less likely to be familiar enough with the structure
of the build system to know where to find these things, but otherwise
they are a useful shortcut for anyone.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

If you're using externalsrc, it's very likely that you're going to want
to examine the intermediate build results even if the recipe builds
successfully; therefore you won't want rm_work to delete those.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>