summaryrefslogtreecommitdiff
path: root/scripts/contrib/python/generate-manifest-3.5.py
AgeCommit message (Collapse)AuthorFiles
2018-01-20python3: Restructure python3 packaging and replace it with autopackagingAlejandro Hernandez1
See previous commit (python2 version) for more info, since mostly everything applies here as well. Old manifest file had several issues: - Its unorganized and hard to read and understand it for an average human being. - When a new package needs to be added, the user actually has to modify the script that creates the manifest, then call the script to create a new manifest, and then submit a patch for both the script and the manifest, so its a little convoluted. - Git complains every single time a patch is submitted to the manifest, since it violates some of its guidelines. - It changes or may change with every release of python, its impossible to know if the required files for a certain package have changed (it could have more or less dependencies), the only way of doing so would be to install and test them all one by one on separate individual images, and even then we wouldnt know if they require less dependencies, we would just know if an extra dependency is required since it would complain, lets face it, this isnt feasible. - The same thing happens for new packages, if someone wants to add a new package, its dependencies need to be checked manually one by one. Features/Fixes: - A new manifest format is used (JSON), easy to read and understand. This file is parsed by the python recipe and python packages read from here are passed directly to bitbake during parsing time. - It provides an automatic manifest creation task (explained on previous commit), which automagically checks for every package dependencies and adds them to the new manifest, hence we will have on each package exactly what that package needs to be run, providing finer granularity. - Dependencies are also checked automagically for new packages (explained on previous commit). This patch has the same features as the python2 version but it differs in the following ways: - Python3 handles precompiled bytecode files (*.pyc) differently. for this reason and since we are cross compiling, wildcards couldnt be avoided on python3 (See PEP #3147 [1]). Both the manifest and the manifest creation script handle this differently, the manifest for python3 has an extra field for cached files, which is how it lets the user install the cached files or not via : INCLUDE_PYCS = "1" on their local.conf. - Shared libraries nomenclature also changed on python3, so again, we use wildcards to deal with this issue ( See PEP #3149 [2]): - Fixes python3 manifest, python3-core should be base and everything should depend on it, hence several packages were deleted: python3-enum, re, gdbm, subprocess, signal, readline. - When building python3-native it adds as symlink to it called nativepython3, which is then isued by the create_manifest task. - Fixes [YOCTO #11513] while were at it. References: [1] https://www.python.org/dev/peps/pep-3147/ [2] https://www.python.org/dev/peps/pep-3149/ Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-12-13python-*-manifest/generators: add runpy; python3-plistlibTim Orling1
* runpy allows running modules/scripts with 'python -m foo' * python3-setuptools RDEPENDS on plistlib (present in python2) * pip3 RDEPENDS on _markupbase (add to python3-core) Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-12-13python-*-manifest/generators: fix long line lengthsTim Orling1
The generators create python-*-manifest.inc files with lines over 2500 characters long which breaks sending patches via git send-email (because of smtp limitation). This patchset formats all the long lines into multiple lines. Signed-off-by: Tim Orling <timothy.t.orling@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-08-03python: don't include -tests with modulesMark Asselstine1
Although 'test' is listed in the python module list (https://docs.python.org/3/py-modindex.html) it is meant only to be used 'internally' by folks developing python itself. Per the documentation: Note The test package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python’s standard library is discouraged as code mentioned here can change or be removed without notice between releases of Python. Per the above it is best not to include this module to discourage folks who might not head the above warnings. Additionally this module is one of the largest py modules going, by dropping this unneeded module from the 'modules' package we can reduce overall image size, something which is important for many embedded deployments. The generator scripts as well as the manifests have thus been modified accordingly, providing a generic mechanism to exclude modules from the 'all modules' package. Signed-off-by: Mark Asselstine <mark.asselstine@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-14python-3.5: Move bz2.py, lzma.py and _compression.py from python3-misc to ↵Martin Jansa1
python3-compression * the /usr/lib/python3.5/_compression.py file is possibly incorrectly included in python3-misc. This runtime dependency is needed in order to use e.g. gzip.py in runtime: >>> import tarfile, zlib, gzip Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3.5/gzip.py", line 12, in <module> import _compression ImportError: No module named '_compression' * at least python3-tests and lzma and bz2 still in python3-misc are using this as well: $ grep -R import.*_compression tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/ tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-compression/usr/lib/python3.5/gzip.py:import _compression tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-misc/usr/lib/python3.5/lzma.py:import _compression tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-misc/usr/lib/python3.5/bz2.py:import _compression tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-tests/usr/lib/python3.5/test/test_bz2.py:import _compression tmp-glibc/work/aarch64-oe-linux/python3/3.5.2-r1.0/packages-split/python3-tests/usr/lib/python3.5/test/test_lzma.py:import _compression and python3-tests are using it as well, so add new runtime dependency on python3-compression Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-06-06python3: add python3-modules-native to RPROVIDES for python3-nativeRoss Burton1
Also clean up the logic in the script to be more Pythonic. Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-05-29python3: Move config-*/Makefile from misc package to dev packageLi Zhou1
Move config-*/Makefile in libdir from misc package to dev package for python3, because it is only needed in development process. Signed-off-by: Li Zhou <li.zhou@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11python2/3: Move config/Makefile from core package to dev packageLi Zhou1
Move config/Makefile in libdir from core package to dev package for python, because it is only needed in development process. Signed-off-by: Li Zhou <li.zhou@windriver.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-08python3: fix run-time deps for core python3 librariesDmitry Rozhkov1
The http.server module from python3-netclient imports the html module which is in python3-html. Also xmlrpc.server imports pydoc which is a part of python3-pydoc. But those run-time dependencies are missing from python3-netclient and python3-xmlrpc respectively. Add the missing run-time dependencies. Signed-off-by: Dmitry Rozhkov <dmitry.rozhkov@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-01python: remove stale link to "Python for Embedded Systems Site"Bob Cochran1
Reference url is a stale, non existent site that returns a 404, so get rid of it Change impacts both the manifest files and the scripts that generate the manifests Run the following from within recipes-devtools/python ../../../scripts/contrib/python/generate-manifest-2.7.py > python-2.7-manifest.inc ../../../scripts/contrib/python/generate-manifest-2.7.py -n > python-native-2.7-manifest.inc ../../../scripts/contrib/python/generate-manifest-3.5.py > python-3.5-manifest.inc ../../../scripts/contrib/python/generate-manifest-3.5.py -n > python-native-3.5-manifest.inc Signed-off-by: Bob Cochran <openembedded@mindchasers.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-03-01generate-manifest-3.5.py: add logic to generate native manifestMing Liu1
python3-native supposes to RPROVIDE all native packages as added in generate-manifest-3.5.py, but it does not so far, this leads a problem that sometimes bitbake cant find a runtime provider for a python3-*-native when a new runtime dependency on it being required, this usualy happens after a new native python3-* recipe is created or the old native python3-* recipes are upgraded. To avoid manually extending RPROVIDE every time when a new runtime dependency is introduced, an argument '-n/--native' is added to the manifest generator, allowing it create a native python3 manifest, with a RPROVIDE line only, the RPROVIDE should contain all the sub-packages. The generated python-native-3.5-manifest.inc is also added which is included by python3-native recipe. Signed-off-by: Ming Liu <peter.x.liu@external.atlascopco.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-02-05python-3-manifest: split out typingAnders Darander1
This allows us to use typing.py without having to add the whole python3-misc package. Signed-off-by: Anders Darander <anders@chargestorm.se> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-02-05python-3-manifest: add ipaddress to python3-ioAnders Darander1
This allows us to use ipaddress without requiring the add the whole python3-misc. Signed-off-by: Anders Darander <anders@chargestorm.se> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-02-05python-3-manifest: add _compat_pickle to python3-pickleAnders Darander1
This allows us to depend on _compat_pickle.* wihtout having to add the whole python3-misc. Signed-off-by: Anders Darander <anders@chargestorm.se> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-02-05python-3-manifest: add socketserver to netserverAnders Darander1
socketserver.* should be part of python3-netserver. Signed-off-by: Anders Darander <anders@chargestorm.se> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-02-05python-3-manifest: add argparse to RDEPENDS for netclientAnders Darander1
http/server.py requires argparse. Signed-off-by: Anders Darander <anders@chargestorm.se> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-02-05python3-manifest: move htlm.py to python3-htmlAnders Darander1
This allows us to use html.py without importing misc. Signed-off-by: Anders Darander <anders@chargestorm.se> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-02-05python-3-manifest: fix adding imp to importlibAnders Darander1
Commit: 512334f102a33833d39af53467894315f0715d07 "python-3.5-manifest: Add imp to importlib" added imp to importlib in the generated manifest, but not in the generator script. Signed-off-by: Anders Darander <anders@chargestorm.se> Signed-off-by: Ross Burton <ross.burton@intel.com>
2017-01-16python-3.5-manifest: Add http module to the netclient packageDerek Straka1
Adding http module from Python's standard library. This allow use of the http module without installing all python-misc modules. Signed-off-by: Derek Straka <derek@asterius.io> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-11-30python3: Build and package precompiled modulesDominic Sacré1
Remove the patch that was applied in the python3 and python3-native recipes to skip compilation of python modules. Modify generate-manifest-3.5.py to match '__pycache__' directories in FILES_*. This is necessary because Python3 puts .pyc files in '__pycache__' subdirectories one level below the corresponding .py files, whereas in Python2 they used to be right next to the sources. This change significantly reduces the startup overhead of Python3 scripts. For example, on a Cortex-A9, "python3 -c pass" took 0.40s before, and 0.19s after. Signed-off-by: Dominic Sacré <dominic.sacre@gmx.de> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25python-3.5-manifest: Add argparse moduleFabio Berton1
Adding argparse module from Python's standard library. This allow use argparse without installing all python-misc modules. For compatibility, add python3-argparse as RDEPENDS to python3-misc. Signed-off-by: Fabio Berton <fabio.berton@ossystems.com.br> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-25python-3.5-manifest: Rename Queue module to queueFabio Berton1
The Queue module has been renamed to queue in Python 3. Signed-off-by: Fabio Berton <fabio.berton@ossystems.com.br> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-23python-3.5-manifest.inc: the signal module RDEPENDS on enumRobert Yang1
Fixed: $ python3 >>> import signal Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/path/to/sdk/sysroots/x86_64-pokysdk-linux/usr/lib/python3.5/signal.py", line 4, in <module> from enum import IntEnum as _IntEnum ImportError: No module named 'enum' Signed-off-by: Robert Yang <liezhi.yang@windriver.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-17python3.5-manifest: Fixes several dependencies on the newest python3Alejandro Hernandez1
This patch adds the following packages: python3-enum (needed by python3-git), python3-selectors (needed by python3-subprocess), python3-signal (needed by python3-subprocess), and it also fixes the following ones with missing dependencies: python3-subprocess, python3-compression, python3-datetime [YOCTO #10127] [YOCTO #10124] [YOCTO #10122] Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-08-17python-3.5-manifest: Add some missing RDEPENDSKyle Russell1
ctype's util.py needs subprocess lang's inspect.py needs importlib.machinery math's random.py needs crypt's hashlib subprocess imports threading Signed-off-by: Kyle Russell <bkylerussell@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-07-20python3: update manifest RDEPENDS for importlib and compression packagesDerek Straka1
zipfile.py has dependencies on importlib, threading, and shell importlib has a dependency on lang operator and contextlib added to the lang package instead of falling into misc Signed-off-by: Derek Straka <derek@asterius.io>
2016-07-12python3: Add compression to python3-shell dependenciesJussi Kukkonen1
python3-shell needs python3-compression for tarfile. Signed-off-by: Jussi Kukkonen <jussi.kukkonen@intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-06-02scripts: Rename ConfigParser -> configparser for python3Ed Bartosh1
The ConfigParser API was renamed to configparser in python 3. Renamed ConfigParser -> configparser in scripts/ to make the code working in python 3. Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
2016-06-02scripts: Fix deprecated dict methods for python3Ed Bartosh1
Replaced iteritems -> items, itervalues -> values, iterkeys -> keys or 'in' Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
2016-05-21meta/scripts: python3: rename file -> openRichard Purdie1
file() API doesn't exist in python 3, convert to open(). Also handle some cases where files aren't closed. Compatible with python 2.7. [Contributions from Ed and Richard] Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-03-12python: move ast module into python-coreRoss Burton1
The compile() method returns objects that inherit from ast.AST so it's best that python-core contains this class. [YOCTO #8684] Signed-off-by: Ross Burton <ross.burton@intel.com>
2016-02-11python: merge python-elementtree into python-xmlRoss Burton1
python-xml depends on python-elementtree as the latter just contains a C library used by the former. However there's no point to this split apart from increasing the number of packages, so merge -elementtree into python-xml. Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-02-04python-distutils: add missing dependency on python-emailDominique Hunziker1
Signed-off-by: Dominique Hunziker <dominique.hunziker@gmail.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
2015-12-01python3: Upgrade from 3.4.3 to 3.5Alejandro Hernandez1
python3-native_3.4.3.bb -> python3-native_3.5.0.bb: - changed version - changed cheksum - no license change, just dates python3_3.4.3.bb -> python3_3.5.0.bb: - changed version - changed cheksum - no license change, just dates New: - use_packed_importlib.patch: Fixes importlib on cross-compile environments Rebased: - Manifest - 000-cross-compile.patch - 020-dont-compile-python-files.patch - 04-default-is-optimized.patch - python-3.3-multilib.patch - distutils3-base.bbclass - distutils3-native-base.bbclass - python3native.bbclass Upstream: - makerace.patch Misc: - pip2 is handled as default on major distros, modified python3-pip to leave /usr/bin/pip available for pip2 - Fixed importing pip3 from python3 interpreter Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com> Signed-off-by: Ross Burton <ross.burton@intel.com>
generated by cgit v1.2.3 (git 2.25.1) at 2025-05-07 01:25:30 +0000