Rework installation of dev, dbg, doc, and locale packages

Use a similar mechanism that was previously used to install locales at
rootfs generation time to install other "complementary" packages (e.g.
*-dev packages) - i.e. install all of the explicitly requested packages
and their dependencies, then get a list of the packages that were
installed, and use that list to install the complementary packages. This
has been implemented by using a list of globs which should make it
easier to extend in future.

The previous locale package installation code assumed that the locale
packages did not have any dependencies that were not already installed;
now that we are installing non-locale packages this is no longer
correct. In practice only the rpm backend actually made use of this
assumption, so it needed to be changed to call into the existing package
backend code to do the complementary package installation rather than
calling rpm directly.

This fixes the doc-pkgs IMAGE_FEATURES feature to work correctly, and
also ensures that all dev/dbg packages get installed for
dev-pkgs/dbg-pkgs respectively even if the dependency chains between
those packages was not ensuring that already.

The code has also been adapted to work correctly with the new
SDK-from-image functionality. To that end, an SDKIMAGE_FEATURES variable
has been added to allow specifying what extra image features should go
into the SDK (extra, because by virtue of installing all of the packages
in the image into the target part of the SDK, we already include all of
IMAGE_FEATURES) with a default value of "dev-pkgs dbg-pkgs".

Fixes [YOCTO #2614].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>

2 files changed, 243 insertions, 0 deletions

diff --git a/scripts/oe-pkgdata-util b/scripts/oe-pkgdata-util
new file mode 100755
index 0000000000..2427f10d89
--- /dev/null
+++ b/scripts/oe-pkgdata-util
@@ -0,0 +1,167 @@
+#!/usr/bin/env python
+
+# OpenEmbedded pkgdata utility
+#
+# Written by: Paul Eggleton <paul.eggleton@linux.intel.com>
+#
+# Copyright 2012 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+#
+# Currently only has one function - mapping of packages to their dev/dbg/doc/locale etc. 
+# counterparts ("glob" command). Could be extended in future to perform other useful querying
+# functions on the pkgdata though.
+#
+
+import sys
+import os
+import os.path
+import fnmatch
+import re
+
+def usage():
+    print("syntax: pkgdata-util glob [-d] <pkgdatadir> <vendor-os> <pkglist> \"<globs>\"");
+
+
+
+def glob(args):
+    if len(args) < 4:
+        usage()
+        sys.exit(1)
+
+    pkgdata_dir = args[0]
+    target_suffix = args[1]
+    pkglist_file = args[2]
+    globs = args[3].split()
+
+    if target_suffix.startswith("-"):
+        target_suffix = target_suffix[1:]
+
+    skipregex = re.compile("-locale-|^locale-base-|-dev$|-doc$|-dbg$|-staticdev$|^kernel-module-")
+
+    mappedpkgs = set()
+    with open(pkglist_file, 'r') as f:
+        for line in f:
+            fields = line.rstrip().split()
+            if len(fields) < 2:
+                continue
+            pkg = fields[0]
+            arch = fields[1]
+            multimach_target_sys = "%s-%s" % (arch, target_suffix)
+
+            # Skip packages for which there is no point applying globs
+            if skipregex.search(pkg):
+                if debug:
+                    print("%s -> !!" % pkg)
+                continue
+
+            # Skip packages that already match the globs, so if e.g. a dev package
+            # is already installed and thus in the list, we don't process it any further
+            # Most of these will be caught by skipregex already, but just in case...
+            already = False
+            for g in globs:
+                if fnmatch.fnmatchcase(pkg, g):
+                    already = True
+                    break
+            if already:
+                if debug:
+                    print("%s -> !" % pkg)
+                continue
+
+            # Define some functions
+            def revpkgdata(pkgn):
+                return os.path.join(pkgdata_dir, multimach_target_sys, "runtime-reverse", pkgn)
+            def fwdpkgdata(pkgn):
+                return os.path.join(pkgdata_dir, multimach_target_sys, "runtime", pkgn)
+            def readpn(pkgdata_file):
+                pn = ""
+                with open(pkgdata_file, 'r') as f:
+                    for line in f:
+                        if line.startswith("PN:"):
+                            pn = line.split(': ')[1].rstrip()
+                return pn
+            def readrenamed(pkgdata_file):
+                renamed = ""
+                pn = os.path.basename(pkgdata_file)
+                with open(pkgdata_file, 'r') as f:
+                    for line in f:
+                        if line.startswith("PKG_%s:" % pn):
+                            renamed = line.split(': ')[1].rstrip()
+                return renamed
+
+            # Main processing loop
+            for g in globs:
+                mappedpkg = ""
+                # First just try substitution (i.e. packagename -> packagename-dev)
+                newpkg = g.replace("*", pkg)
+                revlink = revpkgdata(newpkg)
+                if os.path.exists(revlink):
+                    mappedpkg = os.path.basename(os.readlink(revlink))
+                    fwdfile = fwdpkgdata(mappedpkg)
+                    if os.path.exists(fwdfile):
+                        mappedpkg = readrenamed(fwdfile)
+                else:
+                    # That didn't work, so now get the PN, substitute that, then map in the other direction
+                    revlink = revpkgdata(pkg)
+                    if os.path.exists(revlink):
+                        pn = readpn(revlink)
+                        newpkg = g.replace("*", pn)
+                        fwdfile = fwdpkgdata(newpkg)
+                        if os.path.exists(fwdfile):
+                            mappedpkg = readrenamed(fwdfile)
+                    else:
+                        # Package doesn't even exist...
+                        if debug:
+                            print "%s is not a valid package!" % (pkg)
+                        break
+
+                if mappedpkg:
+                    if debug:
+                        print "%s (%s) -> %s" % (pkg, g, mappedpkg)
+                    mappedpkgs.add(mappedpkg)
+                else:
+                    if debug:
+                        print "%s (%s) -> ?" % (pkg, g)
+
+    if debug:
+        print "------"
+
+    print("\n".join(mappedpkgs))
+
+
+
+# Too lazy to use getopt
+debug = False
+noopt = False
+args = []
+for arg in sys.argv[1:]:
+    if arg == "--":
+        noopt = True
+    else:
+        if not noopt:
+            if arg == "-d":
+                debug = True
+                continue
+        args.append(arg)
+
+if len(args) < 1:
+    usage()
+    sys.exit(1)
+
+if args[0] == "glob":
+    glob(args[1:])
+else:
+    usage()
+    sys.exit(1)
diff --git a/scripts/opkg-query-helper.py b/scripts/opkg-query-helper.py
new file mode 100755
index 0000000000..b52284b325
--- /dev/null
+++ b/scripts/opkg-query-helper.py
@@ -0,0 +1,76 @@
+#!/usr/bin/env python
+
+# OpenEmbedded opkg query helper utility
+#
+# Written by: Paul Eggleton <paul.eggleton@linux.intel.com>
+#
+# Copyright 2012 Intel Corporation
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+#
+
+
+import sys
+import fileinput
+import re
+
+archmode = False
+filemode = False
+
+args = []
+for arg in sys.argv[1:]:
+    if arg == '-a':
+        archmode = True
+    elif arg == '-f':
+        filemode = True
+    else:
+        args.append(arg)
+
+# Regex for removing version specs after dependency items
+verregex = re.compile(' \([=<>]* [^ )]*\)')
+
+pkg = ""
+ver = ""
+for line in fileinput.input(args):
+    line = line.rstrip()
+    if ': ' in line:
+        if line.startswith("Package:"):
+            pkg = line.split(": ")[1]
+            ver = ""
+        else:
+            if archmode:
+                if line.startswith("Architecture:"):
+                    arch = line.split(": ")[1]
+                    print("%s %s" % (pkg,arch))
+            elif filemode:
+                if line.startswith("Version:"):
+                    ver = line.split(": ")[1]
+                elif line.startswith("Architecture:"):
+                    arch = line.split(": ")[1]
+                    print("%s %s_%s_%s.ipk" % (pkg,pkg,ver,arch))
+            else:
+                if line.startswith("Depends:"):
+                    depval = line.split(": ")[1]
+                    deps = depval.split(", ")
+                    for dep in deps:
+                        dep = verregex.sub('', dep)
+                        print("%s|%s" % (pkg,dep))
+                elif line.startswith("Recommends:"):
+                    recval = line.split(": ")[1]
+                    recs = recval.split(", ")
+                    for rec in recs:
+                        rec = verregex.sub('', rec)
+                        print("%s|%s [REC]" % (pkg, rec))
+