Devtool creates a git repository for extracted sources in a temporary
directory and then moves it to a final destination after patching is
done. Unfortunately devtool is not aware that some of its git operations
may have caused git garbage collector to start in background. If timing
is just right a repository move fails because GC is removing git objects
while they are being moved.

The issue was hit on Krogoth release, but the code that moves created
git repository is still the same.

Disable gc.autodetach to make GC run synchronously and block git until
it is done.

Traceback (most recent call last):
  File "<snip>/openembedded-core/scripts/devtool", line 342, in <module>
    ret = main()
  File "<snip>/openembedded-core/scripts/devtool", line 329, in main
    ret = args.func(args, config, basepath, workspace)
  File "<snip>/openembedded-core/scripts/lib/devtool/standard.py", line 352, in extract
    initial_rev = _extract_source(srctree, args.keep_temp, args.branch, False, rd)
  File "<snip>/openembedded-core/scripts/lib/devtool/standard.py", line 644, in _extract_source
    shutil.move(srcsubdir, srctree)
  File "/usr/lib/python2.7/shutil.py", line 299, in move
    copytree(src, real_dst, symlinks=True)
  File "/usr/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/5e', '<snip>/build/grpc/grpc_src/.git/objects/5e', "[Errno 2] No such file or directory: '/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/5e'"), ('/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/57', '<snip>/build/grpc/grpc_src/.git/objects/57', "[Errno 2] No such file or directory: '/tmp/devtool5RXkuX/workdir/grpc-1.2.5/.git/objects/57'"), many git objects ... ]

Signed-off-by: Taras Kondratiuk <takondra@cisco.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>

do_patch rule of SDK's workspace/appends/linux-*.bbhappend may fail if script are not written in Python

that was the case with Phytec's BSP, the fix was to replace the do_patch rule with :

do_patch[noexec]="1" when the file was generated in scripts/lib/devtool/standard.py

Signed-off-by: Yann CARDAILLAC <yann.cardaillac@smile.fr>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>

The check_commits logic assumes that both devtool-base and args.branch
exist in the git repo that it is operating on. In order to prevent
errors at that point it's best to first ensure that both of these refs
actually exist. If they don't both exist then the check_commits logic
should just be skipped, as it would be if the repo wasn't originally
checked out by devtool.

Previously if a user removed the args.branch branch from their devtool
cloned repo this code would crash on adding the repo with -n. The crash
would look like this:

Traceback (most recent call last):
  File "/home/ddedrick/src/poky/scripts/devtool", line 344, in <module>
    ret = main()
  File "/home/ddedrick/src/poky/scripts/devtool", line 331, in main
    ret = args.func(args, config, basepath, workspace)
  File "/home/ddedrick/src/poky/scripts/lib/devtool/standard.py", line 812, in modify
    (stdout, _) = bb.process.run('git log devtool-base..%s' % branch, cwd=srctree)
  File "/home/ddedrick/src/poky/bitbake/lib/bb/process.py", line 178, in run
    raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
bb.process.ExecutionError: Execution of 'git log devtool-base..devtool' failed with exit code 128:
fatal: ambiguous argument 'devtool-base..devtool': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

(From OE-Core rev: f13a3490fdb404bbd4c77e45b83540d6deec1358)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>

DEVTOOL_EXTRA_OVERRIDES only needs one entry for each instance of
overrides. Previous to these changes it would find every override to
SRC_URI and add it to the list. This would duplicate instances where
SRC_URI is modified multiple times with the same override like:
SRC_URI_append_foo += "file://0001-foo.patch"
SRC_URI_append_foo += "file://0002-bar.patch"

A bbappend might also overwrite a SRC_URI override, which would also
cause multiple instances to occur.

When there are multiple instances of the same override in
DEVTOOL_EXTRA_OVERRIDES it causes devtool modify to fail when creating
override branches. The failure occurs when attempting to create the same
override branch a second time and looks like this:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:devtool_post_patch(d)
     0003:
File: '/build/poky/meta/classes/devtool-source.bbclass', lineno: 202, function: devtool_post_patch
     0198:
     0199:        for override in extra_override_list:
     0200:            localdata = bb.data.createCopy(d)
     0201:            if override in default_overrides:
 *** 0202:                bb.process.run('git branch devtool-override-%s %s' % (override, devbranch), cwd=srcsubdir)
     0203:            else:
     0204:                # Reset back to the initial commit on a new branch
     0205:                bb.process.run('git checkout %s -b devtool-override-%s' % (initial_rev, override), cwd=srcsubdir)
     0206:                # Run do_patch function with the override applied
File: '/build/poky/bitbake/lib/bb/process.py', lineno: 178, function: run
     0174:        if not stderr is None:
     0175:            stderr = stderr.decode("utf-8")
     0176:
     0177:    if pipe.returncode != 0:
 *** 0178:        raise ExecutionError(cmd, pipe.returncode, stdout, stderr)
     0179:    return stdout, stderr
Exception: bb.process.ExecutionError: Execution of 'git branch devtool-override-foo devtool' failed with exit code 128:
fatal: A branch named 'devtool-override-foo' already exists.

(From OE-Core rev: 90f667db2219f04e6d61588cd61056d3d8da6d7d)

Signed-off-by: Dan Dedrick <ddedrick@lexmark.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>

This is a fixup for: e1ba46109ea4be3d3b310abaf7f2da3c84a83930

devtool deploy-target --strip foo root@192.168.7.2
ended up with:
Traceback (most recent call last):
  File "/home/user/bar_sdk/sysroots/x86_64-pokysdk-linux/usr/bin/
        devtool", line 344, in <module>
   ret = main()
  File "/home/user/bar_sdk/sysroots/x86_64-pokysdk-linux/usr/bin/
        devtool", line 331, in main
    ret = args.func(args, config, basepath, workspace)
  File "/home/user/bar_sdk/layers/poky/scripts/lib/devtool/
        deploy.py", line 187, in deploy
    rd.getVar('base_libdir'))
TypeError: strip_exbar() missing 1 required positional argument: 'd'

(From OE-Core rev: bc349b2d99d74f6d7fb590fd97750e5ff8d6e6c4)

Signed-off-by: Adrian Freihofer <adrian.freihofer@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>

Add -c (--clean) optiont to 'devtool build' command so that users
could easily clean things up when using devtool.

I encountered a problem about do_prepare_recipe_sysroot failure
when using `devtool build' command and I found myself in a situation
where I either have to use `bitbake' command to clean things up or
use `rm' to remove the directories under ${WORKDIR}.

So add a clean option as it would be helpful when users want to clean
things up to prepare an environment for a clean build.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

The warn method is deprecated. We should use the documented warning instead.

Quoting from the python's official doc:
"""
Note: There is an obsolete method warn which is functionally identical to warning.
      As warn is deprecated, please do not use it - use warning instead.
"""

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>

In most recipes in OE, beginline and endline are before md5 checksum.
We should obey this tradition in devtool's upgrade. Otherwise, we might
see meaningless change just because of the order change.

e.g.
-LIC_FILES_CHKSUM = "file://doc/LICENSE;md5=7765a3d787cb4fed3ccc3c9cee030af9 \
-                    file://plugins/sudoers/redblack.c;beginline=1;endline=41;md5=805782a8466975716f8376b2be9aedde \
-                    file://lib/util/reallocarray.c;beginline=3;endline=16;md5=85b0905b795d4d58bf2e00635649eec6 \
-                    file://lib/util/fnmatch.c;beginline=3;endline=27;md5=67f83ee9bd456557397082f8f1be0efd \
-                    file://lib/util/getcwd.c;beginline=5;endline=27;md5=449af4cc57fc7d46f42090608ba3e681 \
-                    file://lib/util/glob.c;beginline=6;endline=31;md5=5872733146b9eb0deb79e1f664815b85 \
-                    file://lib/util/snprintf.c;beginline=6;endline=34;md5=c82c1b3a5c32e08545c9ec5d71e41e50 \
-                    file://include/sudo_queue.h;beginline=5;endline=27;md5=449af4cc57fc7d46f42090608ba3e681 \
-                    file://lib/util/inet_pton.c;beginline=3;endline=17;md5=3970ab0518ab79cbd0bafb697f10b33a"
+LIC_FILES_CHKSUM = "file://doc/LICENSE;md5=cc4bf2366b059c9598e3947f885931ec \
+                    file://plugins/sudoers/redblack.c;md5=805782a8466975716f8376b2be9aedde;beginline=1;endline=41 \
+                    file://lib/util/reallocarray.c;md5=85b0905b795d4d58bf2e00635649eec6;beginline=3;endline=16 \
+                    file://lib/util/fnmatch.c;md5=67f83ee9bd456557397082f8f1be0efd;beginline=3;endline=27 \
+                    file://lib/util/getcwd.c;md5=449af4cc57fc7d46f42090608ba3e681;beginline=5;endline=27 \
+                    file://lib/util/glob.c;md5=5872733146b9eb0deb79e1f664815b85;beginline=6;endline=31 \
+                    file://lib/util/snprintf.c;md5=c82c1b3a5c32e08545c9ec5d71e41e50;beginline=6;endline=34 \
+                    file://include/sudo_queue.h;md5=449af4cc57fc7d46f42090608ba3e681;beginline=5;endline=27 \
+                    file://lib/util/inet_pton.c;md5=3970ab0518ab79cbd0bafb697f10b33a;beginline=3;endline=17 \
+                    "

After this change, it becomes:
-LIC_FILES_CHKSUM = "file://doc/LICENSE;md5=7765a3d787cb4fed3ccc3c9cee030af9 \
+LIC_FILES_CHKSUM = "file://doc/LICENSE;md5=cc4bf2366b059c9598e3947f885931ec \
                     file://plugins/sudoers/redblack.c;beginline=1;endline=41;md5=805782a8466975716f8376b2be9aedde \
                     file://lib/util/reallocarray.c;beginline=3;endline=16;md5=85b0905b795d4d58bf2e00635649eec6 \
                     file://lib/util/fnmatch.c;beginline=3;endline=27;md5=67f83ee9bd456557397082f8f1be0efd \
@@ -12,7 +12,8 @@ LIC_FILES_CHKSUM = "file://doc/LICENSE;md5=7765a3d787cb4fed3ccc3c9cee030af9 \
                     file://lib/util/glob.c;beginline=6;endline=31;md5=5872733146b9eb0deb79e1f664815b85 \
                     file://lib/util/snprintf.c;beginline=6;endline=34;md5=c82c1b3a5c32e08545c9ec5d71e41e50 \
                     file://include/sudo_queue.h;beginline=5;endline=27;md5=449af4cc57fc7d46f42090608ba3e681 \
-                    file://lib/util/inet_pton.c;beginline=3;endline=17;md5=3970ab0518ab79cbd0bafb697f10b33a"
+                    file://lib/util/inet_pton.c;beginline=3;endline=17;md5=3970ab0518ab79cbd0bafb697f10b33a \
+                    "

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>

It's possible that downloading file from updateserver fails. In
this case, we should error out instead of continue.

We have users reporting unexpected behavior of 'devtool sdk-update'.
When an invalid url is supplied, e.g., `devtool sdk-update http://invalid',
the program reports 'Note: Already up-to-date'.

This is obviously not expected. We should error out in such case.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

This is very useful for updating patch context so that any fuzz is eliminated.
Simply issue:

devtool modify <recipe>
devtool finish --force-patch-refresh <recipe> <layer_path>

Without this flag, devtool will not deem the commits in the workspace
different to patches in the layer, even if the commits have different,
up-to-date context line in them.

Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>