summaryrefslogtreecommitdiff
path: root/scripts/lib/recipetool
AgeCommit message (Collapse)AuthorFiles
2016-07-08recipetool: create: drop unused convert_pkginfo() functionPaul Eggleton1
Code cleanup, no functional changes - this code was never used. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-07-08recipetool: create: avoid decoding errors with Python 3Paul Eggleton5
We're opening source files with the default encoding (utf-8) but we can't necessarily be sure that they are UTF-8 clean - for example, recipetool create ftp://mama.indstate.edu/linux/tree/tree-1.7.0.tgz prior to this patch resulted in a UnicodeDecodeError. Use the "surrogateescape" mode to avoid this. Fixes [YOCTO #9822]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-07-01recipetool: recognize less common makefile namesNathan Lynch1
GNU make looks for "makefile" and "GNUmakefile" in addition to "Makefile", so add these other names to the heuristic for detecting a make-based project. Signed-off-by: Nathan Lynch <nathan_lynch@mentor.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-06-02scripts: python3: convert iterables to listsEd Bartosh2
Converted return value of items() keys() and values() to lists when dictionary is modified in the loop and when the result is added to the list. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02scripts: Fix urlparse imports for python3Ed Bartosh1
Used urllib.parse instead of urlparse to make code working in python 3. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
2016-06-02scripts: Fix encoding errors for python3Ed Bartosh3
Moved call of decode('utf-8') as close as possible to call of subprocess API to avoid calling it in a lot of other places. Decoded binary data to utf-8 where appropriate to fix devtool and recipetool tests in python 3 environment. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
2016-06-02scripts: Replace basestring -> str for python3Ed Bartosh2
Python 3 doesn't have basestring type as all string are unicode strings. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
2016-06-02scripts: Fix deprecated dict methods for python3Ed Bartosh5
Replaced iteritems -> items, itervalues -> values, iterkeys -> keys or 'in' Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
2016-06-02classes/lib: Update to match python3 iter requirementsRichard Purdie1
python3 standardises its use of iteration operations. Update the code to match the for python3 requires. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-30recipetool / devtool: set a fixed SRCREV by default when fetching from gitPaul Eggleton1
If fetching source from a git repository, typically within OpenEmbedded we encourage setting SRCREV to a fixed revision, so change to do that by default and add a -a/--autorev option to use "${AUTOREV}" instead. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-30recipetool: create: use ${BP} for subdir for binary packagesPaul Eggleton1
If we use ${BP} for the subdirectory, the default value of S will work rather than having to have an ugly value derived from the package file name in both places. This does mean that we have to assume the default though (we can't just let the normal logic work because the value of BP is the default until later on, so the replacement doesn't work). Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-30recipetool: create: extract variable values from .deb/.ipk/.rpmPaul Eggleton1
Extract the metadata from package files and use it to set variable values in the recipe (including recipe name and version, LICENSE, SUMMARY, DESCRIPTION, SECTION and HOMEPAGE). For LICENSE we take care not to step on any value determined by our license file scan; if there is one we simply add a comment above the LICENSE setting so the user can resolve it. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-30recipetool: create: support extracting SUMMARY and HOMEPAGEPaul Eggleton3
Allow plugins to set any variable value through the extravalues dict, and use this to support extracting SUMMARY and HOMEPAGE values from spec files included with the source; additionally translate "License:" to a comment next to the LICENSE field (we have our own logic for setting LICENSE, but it will often be useful to see what the spec file says if one is present). Also use the same mechanism for setting the same variables for node.js modules; this was already supported but wasn't inserting the settings in the appropriate place in the file which this will now do. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06recipetool: create: fix falling back to declared license for npm packagesPaul Eggleton2
Fix two problems falling back to the "license" field from package.json when no license file is present: 1) The function that was supposed to return the license field value was always explicitly returning None, and this was never noticed (because the test cases never exercised the fallback as they provided license files for each module). 2) Fix the main package not falling back because it had a default of an empty list, which evaluates to '' instead of 'Unknown'. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-04-29recipetool.newappend: fix syntax error for 'not path_ok' errorChristopher Larson1
Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-03-20recipetool: create: don't create extra files directory unconditionallyPaul Eggleton1
The extra directory next to the recipe should only be created if there are files to put into it; currently only the npm plugin does this. I didn't notice the issue earlier because the test was actually able to succeed under these circumstances if the recipe file came first in the directory listing, which was a fault in my original oe-selftest test; apparently on some YP autobuilder machines the order came out reversed. With this change we can put the oe-selftest test that highlighted the issue back to the way it was, with an extra check to reinforce that only a single file should be created. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09recipetool: create: shrinkwrap and lockdown npm modulesPaul Eggleton1
"npm shrinkwrap" creates a file that ensures that the exact same versions get fetched the next time the recipe is built. lockdown is similar but also includes sha1sums of the modules thus validating they haven't changed between builds. These ensure that the build is reproducible. Fixes [YOCTO #9225]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09recipetool: create: support creation of additional files by pluginsPaul Eggleton1
Allow plugins to create additional files to go alongside the recipe. The plugins don't know what the output filename is going to be, so they need to put the files in a temporary location and add them to an "extrafiles" dict within extravalues where the destination filename is the key and the temporary path is the value. devtool add was also extended to ensure these files get moved in and preserved upon reset if they've been edited by the user. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09recipetool: create: check if npm available if npm:// URL specifiedPaul Eggleton1
If the user specifies an npm:// URL then the fetcher needs npm to be available to run, so check if it's available early rather than failing later. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09recipetool: create: split npm module dependencies into packagesPaul Eggleton2
Rather than rolling all of an npm module's dependencies into the same package, split them into one module per package, setting the SUMMARY and PKGV values from the package.json file for each package. Additionally, mark each package with the appropriate license using the license scanning we already do, falling back to the license stated in the package.json file for the module if unknown. All of this is mostly in aid of ensuring all modules and their licenses now show up in the manifests for the image. Additionally we set the main LICENSE value more concretely once we've calculated the per-package licenses, since we have more information at that point. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09recipetool: create: add license file crunchingPaul Eggleton1
Matching license texts directly to md5sums only goes so far. Some licenses make the copyright statement an intrinsic part of the license statement (e.g. MIT) which of course varies between projects. Also, people often seem to take standard license texts such as GPLv2 and reformat them cosmetically - re-wrapping lines at a different width or changing quoting styles are seemingly popular examples. In order to match license files to their actual licenses more effectively, "crunch" out these elements before comparing to an md5sum. (The existing plain md5sum matching has been left in since it's a shortcut, and our list of crunched md5sums isn't a complete replacement for it.) As always, this code isn't providing any guarantees (legal or otherwise) that it will always get the license correct - as indicated by the accompanying comments the LICENSE values it writes out to the recipe are indicative and you should verify them yourself by looking at the documentation supplied from upstream for the software being built if you have any concerns. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09recipetool: create: match *LICENSE* as a license filePaul Eggleton1
For example, this picks up a file named MIT-LICENSE.txt. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09recipetool: create: improve mapping for autotools program macrosPaul Eggleton2
Make the following improvements to mapping items specified in AC_CHECK_PROG, AC_PATH_PROG and AX_WITH_PROG to recipes/classes: * Produce a map of native recipe -> binary for all binaries currently in STAGING_BINDIR_NATIVE and use this when mapping items * Add some more entries to the class map * Ignore autotools binaries since they are covered by the inherit of autotools * Ignore coreutils-native since that would almost always be a bogus dependency Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-09recipetool: create: be more tolerant of spacing in configure.acPaul Eggleton1
Allow for whitespace in appropriate places, and ensure we match all whitespace chars not just the space character. (This fixes extracting dependencies from tmux's configure.ac, for example.) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-06recipetool: create: fix support for AX_CHECK_LIBRARYPaul Eggleton1
Clearly I didn't test this part of the code - lists don't have an "add" method. Needless to say I have tested it now. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-02recipetool: create: add basic support for new npm fetcher/classPaul Eggleton1
Add detection for npm modules and support for extracting the name and version from package.json as is usually part of an npm module contents. Note: this will likely only produce a buildable recipe if you use an npm:// URL; simply pointing to a node.js source repository isn't going to fetch the module's dependencies. It also doesn't set up the shrinkwrap/lockdown automatically, so there is some room for improvement later. Implements [YOCTO #8690]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-03-02recipetool: create: add basic support for generating linux kernel recipesPaul Eggleton1
Add support for detecting a Linux kernel source tree and generating a basic kernel recipe using meta-skeleton's linux-yocto-custom recipe as a base. Implements [YOCTO #8981]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-03-02recipetool: create: add support for out-of-tree kernel modulesPaul Eggleton1
Detect kernel modules by looking for #include <linux/module.h>, and handle the various styles of Makefile that appear to be used. I was able to use this code to successfully build a number of external kernel modules I found. Implements [YOCTO #8982]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-02-21recipetool: create: improve CMake package mappingPaul Eggleton2
* Package names are actually case sensitive near as I can tell, so we shouldn't be lowercasing them everywhere. * Look for CMake packages in pkgdata and map those back to recipes, so we aren't dependent on the hardcoded mappings (though those are still preserved). * Avoid duplicates in the unmapped package list Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21recipetool: create: add additional extension mechanismsPaul Eggleton2
Add a means of extending the dependency extraction for autotools and cmake. Note: in order to have this work, you need to have an __init__.py in the lib/recipetool directory within your layer along with the module implementing the handlers, and the __init__.py needs to contain: # Enable other layers to have modules in the same named directory from pkgutil import extend_path __path__ = extend_path(__path__, __name__) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-21devtool / recipetool: use common code for launching editorPaul Eggleton1
Looking at Chris Larson's code for starting the user's editor for "recipetool newappend" it was slightly better than what I wrote for "devtool edit-recipe" in that it checks VISUAL as well as EDITOR and defaults to vi if neither are set, so break this out to its own function and call it from both places. The broken out version passes shell=True however in case it's a more complicated command rather than just a name of an executable. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11recipetool: create: set S when we set SRC_URI from local git repoPaul Eggleton1
If you specify a local directory which happens to be a git repository with an origin remote (and it is in fact remote), we can use that for SRC_URI as implemented by OE-Core revision b143d414846854dc8b3e0a47358daf5646eded38, however we also need to set S if the recipe is going to be of any use fetching from that SRC_URI later. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11recipetool: create: convert http git URLs that don't end in .git but contain ↵Paul Eggleton1
/git/ When recipetool create is given a URL that starts with http(s):// and contains /git/, such as the URLs at git.yoctoproject.org, it's fairly safe to assume it's a git repository and not something that should be fetched with wget, so rewrite the URL. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11recipetool: create: ensure URL parameters don't make it into the namePaul Eggleton1
When auto-detecting the name for a recipe from the URL, strip off any parameters (";name=value...") before parsing the URL, otherwise this just ends up in the recipe name which is undesirable. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11recipetool: create: determine name/version from github/bitbucket URLsPaul Eggleton1
Both BitBucket and GitHub provide "release" downloads that unfortunately don't include the actual project name. For recipetool this means we have to have special handling for these URLs or else the current code assumes the name of the project is something like "v0.9.1". Borrow and adapt some code from autospec to do this, and put it in its own function for clarity. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11recipetool: create: support cmake find_library directivePaul Eggleton1
CMake supports a find_library() directive to find named libraries, so detect dependencies from this. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-11recipetool.newappend: add -e/--edit argumentChristopher Larson1
Signed-off-by: Christopher Larson <chris_larson@mentor.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-01-22recipetool: create: better fix for fetch error handlingPaul Eggleton1
I was a little bit hasty in OE-Core revision c2cc5abe34169eae92067d97ce1e747e7c1413f5 - it turns out BitBake's fetcher code is not consistent in whether it logs something useful or not; when fetching from an http URL it does but with a git repository it doesn't. In advance of any major reworking of fetch error handling in BitBake, let's just print the text of the exception and then we know we have shown something to the user. Additionally, we were only catching FetchException here but there are several other classes of exception that the fetcher can raise (e.g. MalformedUrl); catch the parent BBFetchException class instead so we avoid tracebacks for those other classes as well. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22recipetool: create: fix extraction of name from URLs ending in /Paul Eggleton1
If the URL ends in a / then we want to strip that off the path we split out of the URL before calling os.path.basename() on it. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22recipetool: create: extract SRC_URI from local git repositoriesPaul Eggleton1
If you specify a local directory which happens to be a git repository with an origin remote (and it is in fact remote), we can use that for SRC_URI rather than leaving it blank in the recipe. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22devtool / recipetool: support specifying a subdirectory within the fetched ↵Paul Eggleton1
source Sometimes you don't want to build an entire project, just a subdirectory of it; add a --src-subdir option to make that easier. (We still look for a single subdirectory in what gets unpacked, e.g. what you might find within a tarball, so whatever you specify with this option is added onto the end of that.) Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-22recipetool: create: strip quotes from values extracted from CMakeLists.txtPaul Eggleton1
Quoting is optional in CMakeLists.txt and is occasionally used, so strip out quotes if they are present. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19recipetool: create: add a couple more license checksumsPaul Eggleton1
I found these when I was looking at libftdi and they seem to be generic enough to show up in at least a couple of other packages so I figure I'll add them. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19recipetool: create: add basic support for extracting dependencies from cmakePaul Eggleton2
Add support for extracting dependencies from CMakeLists.txt. There's still a bunch of things missing that are outside the scope of OE-Core and we still lack a proper extension mechanism, but this is a good start. This also adds an oe-selftest test to exercise the new code a bit. Implements [YOCTO #7635]. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19recipetool: create: force GL libraries to virtual/*Paul Eggleton1
We want to specify dependencies on virtual/* rather than whatever library is selected in the current configuration. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19recipetool: create: move dependency mapping code to RecipeHandlerPaul Eggleton2
Some refactoring to allow access to the library/header/pkg-config mappings and the DEPENDS / unmapped dependency output code from other classes than AutotoolsRecipeHandler. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19recipetool: create: fix overzealous mapping of git URLsPaul Eggleton1
The regex for detecting git URLs was unanchored, leading to it matching where it shouldn't have. An example of where this went wrong was http://taglib.github.io/releases/taglib-1.9.1.tar.gz. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19recipetool: create: support additional autoconf macros from autoconf-archivePaul Eggleton1
Support a number of macros from autoconf-archive when reading configure.ac to extract dependencies. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19recipetool: create: detect flex/bison dependencyPaul Eggleton1
There are a few different macros that can be used to pick up these tools, add support for them all. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-01-19recipetool: create: pick up boost macros in configure.acPaul Eggleton1
The presence of BOOST_REQUIRE or AX_BOOST.* indicates that boost is a dependency. Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3 (git 2.25.1) at 2025-01-22 05:59:21 +0000