Implement BBVERSIONS

This implements a feature similar to BBCLASSEXTEND, but for generating
multiple versions of a given recipe.  For example: BBVERSIONS = "1.0 2.0 git".

In addition to the above, one can utilize [a-b] style patterns, and can have a
:<basever> postfix, which allows you to essentially name the range of
versions.  Both the current version and the basever end up in OVERRIDES, and
the basever gets placed into the BPV variable.  The default BPV, if none is
specified, is the original PV of the recipe, before bbversions processing.

In this way, you can do things like:
BBVERSIONS = "1.0.[0-6]:1.0.0+
              1.0.[7-9]:1.0.7+"

SRC_URI_append_1.0.7+ = "file://some_extra_patch.patch;patch=1"

Or you can create a recipe per range, and name the recipe file as such: nano_1.0.7+.bb.

(Bitbake rev: 4ee9a56e16f1eb3c1649eaa3127b09ab0e93d1ec)

Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>

4 files changed, 134 insertions, 30 deletions

diff --git a/bitbake/doc/manual/usermanual.xml b/bitbake/doc/manual/usermanual.xml
index 29cdf97a0a..6424a7ebd9 100644
--- a/bitbake/doc/manual/usermanual.xml
+++ b/bitbake/doc/manual/usermanual.xml
@@ -228,6 +228,18 @@ This event handler gets called every time an event is triggered. A global variab
 method one can get the name of the triggered event.</para><para>The above event handler prints the name
 of the event and the content of the <varname>FILE</varname> variable.</para>
             </section>
+            <section>
+                <title>Variants</title>
+                <para>Two Bitbake features exist to facilitate the creation of multiple buildable incarnations from a single recipe file.</para>
+                <para>The first is <varname>BBCLASSEXTEND</varname>.  This variable is a space separated list of classes to utilize to "extend" the recipe for each variant.  As an example, setting <screen>BBCLASSEXTEND = "native"</screen> results in a second incarnation of the current recipe being available.  This second incarantion will have the "native" class inherited.</para>
+                <para>The second feature is <varname>BBVERSIONS</varname>.  This variable allows a single recipe to be able to build multiple versions of a project from a single recipe file, and allows you to specify conditional metadata (using the <varname>OVERRIDES</varname> mechanism) for a single version, or an optionally named range of versions:</para>
+                <para><screen>BBVERSIONS = "1.0 2.0 git"
+SRC_URI_git = "git://someurl/somepath.git"</screen></para>
+                <para><screen>BBVERSIONS = "1.0.[0-6]:1.0.0+ \
+              1.0.[7-9]:1.0.7+"
+SRC_URI_append_1.0.7+ = "file://some_patch_which_the_new_versions_need.patch;patch=1"</screen></para>
+                <para>Note that the name of the range will default to the original version of the recipe, so given OE, a recipe file of foo_1.0.0+.bb will default the name of its versions to 1.0.0+.  This is useful, as the range name is not only placed into overrides, it's also made available for the metadata to use in the form of the <varname>BPV</varname> variable, for use in file:// search paths (<varname>FILESPATH</varname>).</para>
+            </section>
         </section>
         <section>
             <title>Dependency Handling</title>
@@ -270,7 +282,7 @@ of the event and the content of the <varname>FILE</varname> variable.</para>
             <section>
                 <title>Configuration Files</title>
                 <para>The first of the classifications of metadata in BitBake is configuration metadata.  This metadata is global, and therefore affects <emphasis>all</emphasis> packages and tasks which are executed.</para>
-                <papa>Bitbake will first search the current working directory for an optional "conf/bblayers.conf" configuration file. This file is expected to contain a BBLAYERS variable which is a space delimited list of 'layer' directories. For each directory in this list a "conf/layer.conf" file will be searched for and parsed with the LAYERDIR variable being set to the directory where the layer was found. The idea is these files will setup BBPATH and other variables correctly for a given build directory automatically for the user.</para>
+                <para>Bitbake will first search the current working directory for an optional "conf/bblayers.conf" configuration file. This file is expected to contain a BBLAYERS variable which is a space delimited list of 'layer' directories. For each directory in this list a "conf/layer.conf" file will be searched for and parsed with the LAYERDIR variable being set to the directory where the layer was found. The idea is these files will setup BBPATH and other variables correctly for a given build directory automatically for the user.</para>
                 <para>Bitbake will then expect to find 'conf/bitbake.conf' somewhere in the user specified <envar>BBPATH</envar>.  That configuration file generally has include directives to pull in any other metadata (generally files specific to architecture, machine, <emphasis>local</emphasis> and so on.</para>
                 <para>Only variable definitions and include directives are allowed in .conf files.</para>
             </section>
diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index 43091daa26..9a962ecc74 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -143,8 +143,8 @@ class Cache:
             if dep not in self.depends_cache[fn]["__depends"]:
                 self.depends_cache[fn]["__depends"].append(dep)
 
-        # Make sure BBCLASSEXTEND always makes the cache too
-        self.getVar('BBCLASSEXTEND', virtualfn, True)
+        # Make sure the variants always make it into the cache too
+        self.getVar('__VARIANTS', virtualfn, True)
 
         self.depends_cache[virtualfn]["CACHETIMESTAMP"] = bb.parse.cached_mtime(fn)
 
@@ -199,7 +199,7 @@ class Cache:
             self.cacheValidUpdate(fn)
 
         if self.cacheValid(fn):
-            multi = self.getVar('BBCLASSEXTEND', fn, True)
+            multi = self.getVar('__VARIANTS', fn, True)
             for cls in (multi or "").split() + [""]:
                 virtualfn = self.realfn2virtual(fn, cls)
                 if self.depends_cache[virtualfn]["__SKIPPED"]:
@@ -292,7 +292,7 @@ class Cache:
             self.clean[fn] = ""
 
         # Mark extended class data as clean too
-        multi = self.getVar('BBCLASSEXTEND', fn, True)
+        multi = self.getVar('__VARIANTS', fn, True)
         for cls in (multi or "").split():
             virtualfn = self.realfn2virtual(fn, cls)
             self.clean[virtualfn] = ""
@@ -443,7 +443,7 @@ class Cache:
 
         # Touch this to make sure its in the cache
         self.getVar('__BB_DONT_CACHE', file_name, True)
-        self.getVar('BBCLASSEXTEND', file_name, True)
+        self.getVar('__VARIANTS', file_name, True)
 
     def load_bbfile( self, bbfile , config):
         """
diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py
index 70a69b8d14..59aa44bee0 100644
--- a/bitbake/lib/bb/parse/ast.py
+++ b/bitbake/lib/bb/parse/ast.py
@@ -22,9 +22,11 @@
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
 import bb, re, string
+from itertools import chain
 
 __word__ = re.compile(r"\S+")
 __parsed_methods__ = bb.methodpool.get_parsed_dict()
+_bbversions_re = re.compile(r"\[(?P<from>[0-9]+)-(?P<to>[0-9]+)\]")
 
 class StatementGroup(list):
     def eval(self, data):
@@ -335,3 +337,115 @@ def finalise(fn, d):
 
     bb.event.fire(bb.event.RecipeParsed(fn), d)
 
+def _create_variants(datastores, names, function):
+    def create_variant(name, orig_d, arg = None):
+        new_d = bb.data.createCopy(orig_d)
+        function(arg or name, new_d)
+        datastores[name] = new_d
+
+    for variant, variant_d in datastores.items():
+        for name in names:
+            if not variant:
+                # Based on main recipe
+                create_variant(name, variant_d)
+            else:
+                create_variant("%s-%s" % (variant, name), variant_d, name)
+
+def _expand_versions(versions):
+    def expand_one(version, start, end):
+        for i in xrange(start, end + 1):
+            ver = _bbversions_re.sub(str(i), version, 1)
+            yield ver
+
+    versions = iter(versions)
+    while True:
+        try:
+            version = versions.next()
+        except StopIteration:
+            break
+
+        range_ver = _bbversions_re.search(version)
+        if not range_ver:
+            yield version
+        else:
+            newversions = expand_one(version, int(range_ver.group("from")),
+                                     int(range_ver.group("to")))
+            versions = chain(newversions, versions)
+
+def multi_finalize(fn, d):
+    safe_d = d
+
+    d = bb.data.createCopy(safe_d)
+    try:
+        finalise(fn, d)
+    except bb.parse.SkipPackage:
+        bb.data.setVar("__SKIPPED", True, d)
+    datastores = {"": safe_d}
+
+    versions = (d.getVar("BBVERSIONS", True) or "").split()
+    if versions:
+        pv = orig_pv = d.getVar("PV", True)
+        baseversions = {}
+
+        def verfunc(ver, d, pv_d = None):
+            if pv_d is None:
+                pv_d = d
+
+            overrides = d.getVar("OVERRIDES", True).split(":")
+            pv_d.setVar("PV", ver)
+            overrides.append(ver)
+            bpv = baseversions.get(ver) or orig_pv
+            pv_d.setVar("BPV", bpv)
+            overrides.append(bpv)
+            d.setVar("OVERRIDES", ":".join(overrides))
+
+        versions = list(_expand_versions(versions))
+        for pos, version in enumerate(list(versions)):
+            try:
+                pv, bpv = version.split(":", 2)
+            except ValueError:
+                pass
+            else:
+                versions[pos] = pv
+                baseversions[pv] = bpv
+
+        if pv in versions and not baseversions.get(pv):
+            versions.remove(pv)
+        else:
+            pv = versions.pop()
+
+            # This is necessary because our existing main datastore
+            # has already been finalized with the old PV, we need one
+            # that's been finalized with the new PV.
+            d = bb.data.createCopy(safe_d)
+            verfunc(pv, d, safe_d)
+            try:
+                finalise(fn, d)
+            except bb.parse.SkipPackage:
+                bb.data.setVar("__SKIPPED", True, d)
+
+        _create_variants(datastores, versions, verfunc)
+
+    extended = d.getVar("BBCLASSEXTEND", True) or ""
+    if extended:
+        pn = d.getVar("PN", True)
+        def extendfunc(name, d):
+            d.setVar("PN", "%s-%s" % (pn, name))
+            bb.parse.BBHandler.inherit([name], d)
+
+        safe_d.setVar("BBCLASSEXTEND", extended)
+        _create_variants(datastores, extended.split(), extendfunc)
+
+    for variant, variant_d in datastores.items():
+        if variant:
+            try:
+                finalise(fn, variant_d)
+            except bb.parse.SkipPackage:
+                bb.data.setVar("__SKIPPED", True, variant_d)
+
+    if len(datastores) > 1:
+        variants = filter(None, datastores.keys())
+        safe_d.setVar("__VARIANTS", " ".join(variants))
+
+    datastores[""] = d
+    return datastores
diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py b/bitbake/lib/bb/parse/parse_py/BBHandler.py
index 4641c13d9c..262c883c95 100644
--- a/bitbake/lib/bb/parse/parse_py/BBHandler.py
+++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py
@@ -152,30 +152,8 @@ def handle(fn, d, include):
         classes.remove(__classname__)
     else:
         if include == 0:
-            safe_d = d
-            d = bb.data.createCopy(safe_d)
-            try:
-                ast.finalise(fn, d)
-            except bb.parse.SkipPackage:
-                bb.data.setVar("__SKIPPED", True, d)
-            darray = {"": d}
-
-            extended = bb.data.getVar("BBCLASSEXTEND", d, True)
-            if extended:
-                bb.data.setVar("BBCLASSEXTEND", extended, safe_d)
-
-            for cls in (extended or "").split():
-                pn = data.getVar('PN', d, True)
-                variant_d = bb.data.createCopy(safe_d)
-                data.setVar('PN', pn + '-' + cls, variant_d)
-                inherit([cls], variant_d)
-                try:
-                    ast.finalise(fn, variant_d)
-                except bb.parse.SkipPackage:
-                    bb.data.setVar("__SKIPPED", True, variant_d)
-                darray[cls] = variant_d
-            return darray
-   
+            return ast.multi_finalize(fn, d)
+
     if oldfile:
         bb.data.setVar("FILE", oldfile, d)