openembedded-core.git/meta/classes/sstate.bbclass, branch thud

meta: remove True option to getVar calls (again)

2019-01-28T16:56:33+00:00

A couple have still been missed in the past despite multiple
attempts at doing so (or simply have re-appeared?).

Search & replace made using the following command:
    sed -e 's|\(d\.getVar \?\)( \?\([^,()]*\), \?True)|\1(\2)|g' \
        -i $(git grep -E 'getVar ?\( ?([^,()]*), ?True\)' \
             | cut -d':' -f1 \
             | sort -u)

(From OE-Core rev: 9f551d588693328e4d99d33be94f26684eafcaba)

Signed-off-by: André Draszik 
Signed-off-by: Richard Purdie 
Signed-off-by: Armin Kuster

sstate: add support for caching shared workdir tasks

2019-01-28T16:56:28+00:00

The sstate bbclass uses workdir as a hardcoded string in path
manipulations. This means that the sstate caching mechanism does
not work for the work-shared directory which the kernel uses to
share its build configuration and source files for out of tree
kernel modules.

This commit modifies the path manipulation mechanism to use the
work-shared directory if detected in the paths when handling the
sstate cache packages.

(From OE-Core rev: 27642449f95e38598f9c83948ce109c5891e5877)

Signed-off-by: Michael Ho 
Signed-off-by: Richard Purdie 
Signed-off-by: Armin Kuster

sstate.bbclass: Only remove sstate file when task is existed

2019-01-08T20:12:02+00:00

This can improve the performance a lot for "bitbake 
-ccleansstate" when there are a lot of sstate files.

For example:
* Before
  $ bitbake quilt-native -ccleansstate
  - Check log.do_cleansstate:
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_qa.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_packagedata.tgz*
  Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*

  There are no package tasks for quilt-native, so the first 4 lines doesn't
  make any sense, but the glob pattern "sstate-cache/*/*" is very time
  consuming when there are no disk caches. E.g., I have more than 600,000
  sstate files:
  - Without disk caches
  # echo 3 >/proc/sys/vm/drop_caches
  $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")'
    real    4m32.583s
    user    0m5.768s
    sys     0m12.892s

  - With disk caches (e.g., run it in the second time)
  $ time python3 -c 'import glob; glob.glob("/sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_package.tgz*")'
    real    0m5.128s
    user    0m2.772s
    sys     0m2.308s

  So the 4 removing *package* commands cost more than 20s or 272s in theory.

* After
  $ bitbake quilt-native -ccleansstate
  - Check log.do_cleansstate:
  Removing /sstate-cache/*/sstate:quilt-native::0.65:r0::3:*_populate_lic.tgz*
  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*

  We can see that it saved 20s or 272s in theory.

(From OE-Core rev: bb2d6349ea87f090c58001f0d4348b24c2982cde)

Signed-off-by: Robert Yang 
Signed-off-by: Richard Purdie 
Signed-off-by: Armin Kuster

sstate.bbclass: update SSTATE_DUPWHITELIST

2018-09-12T07:17:37+00:00

Update SSTATE_DUPWHITELIST in sstate.bbclass.

* remove ${DEPLOY_DIR_RPM}/noarch/ which is not overwritten any more
* add directories for package target-sdk-provides-dummy

Signed-off-by: Kai Kang 
Signed-off-by: Richard Purdie

sstate.bbclass: make SSTATE_PRUNE_OBSOLETEWORKDIR could be overwritten

2018-08-28T09:27:15+00:00

Define variable SSTATE_PRUNE_OBSOLETEWORKDIR with '?=' in sstate.bbclass,
then it could be overwritten by user configuration.

Signed-off-by: Kai Kang 
Signed-off-by: Richard Purdie

sstate: add intel-microcode to SSTATE_DUPWHITELIST

2018-08-23T06:45:32+00:00

intel-microcode multilib recipes can generate identical overlapping
files: microcode.cpio.

Signed-off-by: Yongxin Liu 
Signed-off-by: Richard Purdie

sstate: Optimise SSTATE_EXCLUDEDEPS_SYSROOT handling

2018-08-16T08:49:32+00:00

Using re.compile() is around six times faster than recompiling the regexp
each time so maintain a cache.

Signed-off-by: Richard Purdie

sstate: use SSTATE_EXCLUDEDEPS_SYSROOT for skipping base-passwd|shadow-sysroot recipes

2018-08-15T20:46:24+00:00

Use the newly introduced SSTATE_EXCLUDEDEPS_SYSROOT for specifying
the base-passwd|shadow-sysroot recipes to be excluded from a recipe sysroot.

Signed-off-by: André Draszik 
Signed-off-by: Richard Purdie

sstate: use SSTATE_EXCLUDEDEPS_SYSROOT for skipping *-initial recipes

2018-08-15T20:46:24+00:00

Use the newly introduced SSTATE_EXCLUDEDEPS_SYSROOT for specifying
the *-initial recipes to be excluded from a recipe sysroot.

Signed-off-by: André Draszik 
Signed-off-by: Richard Purdie

sstate: allow specifying indirect dependencies to exclude from sysroot

2018-08-15T20:46:24+00:00

Currently, a dependency on any -native recipe will pull in
all dependencies of that -native recipe in the recipe
sysroot. This behaviour might not always be wanted, e.g.
when that -native recipe depends on build-tools that are
not relevant for the current recipe.

This change adds a SSTATE_EXCLUDEDEPS_SYSROOT variable,
which will be evaluated for such recursive dependencies to
be excluded. The idea is similar to
   http://lists.openembedded.org/pipermail/openembedded-core/2018-January/146324.html
except that the list is not hard-coded anymore.

SSTATE_EXCLUDEDEPS_SYSROOT is evaluated as two regular
expressions of recipe and dependency to ignore, e.g. in
the above flex-native / bison-native use-case, one would
specify

    SSTATE_EXCLUDEDEPS_SYSROOT = ".*->(flex|bison)-native"

in layer.conf.

The existing special handling of "-initial" as well as
"base-passwd" and "shadow-sysroot" could also be
streamlined:

    SSTATE_EXCLUDEDEPS_SYSROOT += "\
        .*->.*-initial.* \
        .*(base-passwd|shadow-sysroot)->.* \
    "

Another anticipated user is meta-java, where certain newer
JDKs can only be bootstrapped (built) using older JDKs,
but it doesn't make much sense to copy all those older
JDKs and their own build tools (ant, etc.) into the
sysroot of recipes wanting to be built using the newer JDK
(only), e.g.:

    SSTATE_EXCLUDEDEPS_SYSROOT += "\
        openjdk-8-native->(ant-native|attr-native|coreutils-native|icedtea7-native|libxslt-native|make-native|openssl-native|zip-native|unzip-native) \
    "

Signed-off-by: André Draszik 
Signed-off-by: Richard Purdie