| Age | Commit message (Collapse) | Author | Files |
|---|
|
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
stdeout and stderr content returned by subprocess API has different
types in Python 3(bytes) and Python 2(string). Decoding it to
'utf-8' makes it unicode on both pythons.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Used tempfile.TemporaryFile() API instead of deprecated
os.tmpfile().
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Reduce is not a builtin function in python3.
It has to be imported from functools.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Changed old syle except statements 'except <exception>, var'
to new style 'except <exception> as var' as old style is not
supported in python3.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Converted return value of items() keys() and values() to
lists when dictionary is modified in the loop and when
the result is added to the list.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The ConfigParser API was renamed to configparser in python 3.
Renamed ConfigParser -> configparser in scripts/ to make the
code working in python 3.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
|
|
Replaced iteritems -> items, itervalues -> values,
iterkeys -> keys or 'in'
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
|
|
Used print function instead of print statement to make
the code work in python 3.
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
As suspected, invoking "git archive" with all intended files as
parameters can run into command line length limitations. Splitting up
the parameters into multiple invocations (xargs-style) works and was
tested after encountering the situation in practice.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When the "history" option is not set in the combo-layer.conf, the
intended default was to use the traditional method. Passing "True" as
default when querying the config was unintentional.
Also remove some left-over debugging code.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The core idea is that all commits get imported, including merge
commits, and joined into one big merge commit that imports the changes
from the individual components into the main branch of the combined
repository.
This is done by copying the files in each commit and removing deleted
ones, instead of trying to patch the combined repository.
The advantages of doing updates in this mode are:
- works for arbitrary upstream repos, not just those which
support conversion into a linear set of patches
- listing history shows that commits where developed
independently in the different components, instead of
artificially showing them as if they had been developed
one after the after (component "aaaa" before "bbbb", then "ccc", ...)
- bisecting becomes easier: when upstream repos only ensure consistency
when merging into their "master" branches, then those merge
commits are good candidates for test builds also in the combined
repo
- more data mining can be done, for example showing who merged a commit
and when
Selecting a subset of the files is supported, albeit with a slight
different semantic for wild card matching compared to other code paths
(/ is matched by * and ?). Empty commits get skipped because typically
they are a result of filtering (but that is not checked, so
intentionally empty commits also get skipped).
Other code paths are intentionally left unchanged, to avoid
regressions. However, the downside is that some opportunities for
refactoring (in particular regarding file filtering) were ignored.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Allow setting the environment. Due to a subprocess quirk, it must
always be set explicitly (reuses the one from the previous call if not
set, instead of falling back to os.environ).
Embedding nul characters will be useful for parsing git output more
reliably; support dumping such output a bit better.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
When setting "history = True" in combo-layer.conf consistently for the
components involved in an update or using "update" together with the
"--history" command line flag, a new mode for updating will be used
that does not rely on exporting/importing patches.
A config setting is used because it should be used consistently by
everyone using the same config, without having to remember to use an
additional command line parameter.
There are no real global settings, so the setting is checked
separately for each component although the setting has to be set
consistently. This restriction could be removed later.
In practice, putting "history" into the "[DEFAULT]" section is the
easiest approach for configuring it.
The actual code changes split up action_update and the
combo-layer.conf handling in preparation for this new mode, without
implementing the mode itself.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
There appears to have been a lot of copy and pasting of the code
which prints tracebacks upon failure and limits the stack trace to
5 entries. This obscures the real error and is very confusing to the user
it look me an age to work out why some tracebacks weren't useful.
This patch removes the limit, making tracebacks much more useful for
debugging.
[YOCTO #9230]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When importing the components during the "combo-layer init" with full
history and relocation into a destination directory, components with
empty commits were not handled because the "mv" command was invoked
with just one parameter.
Replacing that with a find/xargs pair avoids the problem and should
also fix the handling of filenames with special characters (in
particular spaces).
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
I ran into an issue where a patch just deleting a single file
within the repository (meta/recipes-devtools/m4/m4/make.patch)
would get skipped by combo-layer.
It turns out this has the patch header (commented to avoid breaking scripts):
: diff --git a/meta/recipes-devtools/m4/m4/make.patch b/meta/recipes-devtools/m4/m4/make.patch
: deleted file mode 100644
: index 79fb415..0000000
: --- a/meta/recipes-devtools/m4/m4/make.patch
: +++ /dev/null
: @@ -1,42 +0,0 @@
and this is classed as > 5 headers in filterdiff. When we piped the path
through filterdiff, the --- line disappears, then the second time we pass
through filterdiff, it shows no lines changed and the patch is assumed
to be empty and skipped.
Changing MAX_HEADERS in filterdiff is one way to fix this, another would
be to grep out "deleted file mode" lines. Instead, we can use new
git syntax to exclude files from the git format-patch instead and avoid
filterdiff entirely.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
There is no ability at the moment for situations where users would like
to keep section in combo-layer.conf but don't update it, unless explicitly
specified.
Now, by adding "update = no" to desired section would exclude
that repository from "combo-layer update" command.
It is still possible to explicitly update it by "combo-layer update $section".
By default, all repositories are assumed as "update = yes"
Signed-off-by: Alexander D. Kanevskiy <kad@kad.name>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
When reconfiguring the branch to something not already fetched,
action_pull fails with
error: pathspec '<new branch name>' did not match any file(s) known to git.
It is the "git checkout" which fails like that. To solve this,
try the faster "git checkout + git pull" first and only if that fails,
fall back to the slow "git fetch + git checkout".
In the conf.hard_reset case, do the checkout always after the git fetch.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The git operations can fail, for example when the branch is unknown
or misconfigured.
Better move the info message and extend it such that it is printed
first and provides the necessary context, because otherwise the
CalledProcessError exception gets dumped without mentioning for which
component it occurred.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Commit c908a423f85a84ddd8249abd00254f29d47df74b introduced a new
issue in combo-layer that leads to a traceback as
args.hard_reset is an unknown variable. This change defines an
appropriate destination for the command args parser and fixes the
reference.
Signed-off-by: Alexander Kanevskiy <kad@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
This option causes combo-layer to do git fetch and hard reset instead of
git pull in the component repositories. This makes sure that the local
component repositories are always in sync with the remote - tolerating
force pushes and overriding any locally made changes.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Only allow fast-forward merges in the component repositories when doing
git-pull. This makes it possible to spot problems (i.e. rewriting of
history) in the component upstream . Also, this change prevents the
creation of local-only merge commits in the component repositories.
These merges cause "last_revision" field of the combo-layer config to
point to a git commit that is only present in the users local component
repository but nowhere in upstream.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
This patch includes support for a global section in combo-layer.conf
called [combo-layer-settings]. Supported in this section is key
"commit_msg"; its value is the template for the git commit message
that updates the last_revision. The template can include substitution
for the updated component list: ${components}. The substituted value
will either be a comma-separated list of components or "all components",
if combo-layer was invoked without component list argument.
If the key is not present, the old default value is used for the commit
message.
Configuration file example:
[combo-layer-settings]
commit_msg = pulled in the latest changes for ${components}.
Signed-off-by: Ismo Puustinen <ismo.puustinen@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
If combo-layer is called from a non-interactive context we should exit
with a correct error code rather than try and drop to a shell.
This patch cleans up a few error case exit codes as well as
detecting and handling non-interactive usage.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The previous "clean up dest_dir checking" patch (f8cdbe7497) improved
handling of empty dest_dir but made handling of unset dest_dir worse:
instead showing the "Option dest_dir is not defined for component ..."
error, it fails with a Python exception.
Avoid that by providing a sane fallback for the unset case. With that
change, dest_dir is no longer strictly required, but the check for it
is kept to ensure that a combo-layer.conf also works with older
combo-layer versions.
[Yocto #7773]
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
When the head of a branch is a merge commit, combo-layer did not
record that commit as last_revision because it only considers applied
patches, and the merge commit never gets applied.
This causes problems when the merge commit leads to multiple patches
and the commit id that gets recorded only reaches some of these
patches. The next run then will try to re-apply the other patches.
This special case is now detected and dealt with by bumping
last_revision to the branch commit. The behavior where the head is a
normal commit is intentionally not changed, because some users might
prefer the traditional behavior.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The code detecting empty patches after removing files with
file_exclude failed for commits which were already empty before (like
the initial commit in some repos): such patches are completely empty
files, without a From line.
Detect that case and just let the normal empty patch detection deal
with it.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
"filterdiff -x ./some/file" does not remove changes for some/file.
We must be more careful about constructing the path name and
only add the prefix when it really means a directory.
While at it, also better normalize the path in copy_selected_files()
early on, to handle double slashes. Useful should the function ever
gets used for something other that dest_dir (which gets normalized in
sanity_check()).
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Empty dest_dir is basically undocumented behavior. The sample conf
only mentions using just a dot for the current directory. In practice,
the empty string does not work because of code like this:
def action_splitpatch(conf, args):
...
if dest_dir != ".":
filerange_root = '%s -x "%s/*"' % (filerange_root, dest_dir)
However, the empty string was not explicitly checked for, leading to
strange errors when trying to apply patches:
[12:50:23] Applying: foobar: xyz
fatal: unable to stat newly created file '/foobar': No such file or directory
This patch turns the empty string into an alias for the dot. This seems
more user-friendly than throwing an error. This alias is intentionally
not document in the sample conf, because the dot is clearer and works also
with older copies of combo-layer.
Instead of checking for both all the time and normalizing the path when
needed (as done in some places), rewrite the value in sanity_check()
and then only check for '.'.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Running an update operation with no repo/revision specified was failing.
This fixes that code path which worked until the change from:
http://git.yoctoproject.org/cgit.cgi/poky/commit/scripts/combo-layer?id=3592507a149b668c0a925e176535f7f2002fa543
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
The new "since_revision" property can be used to cut off the imported
history at some point. This is useful to keep the resulting repository
smaller while still preserving enough history that "git annotate"
reports the right author and commit for most lines.
The initial, squashed import commit shows up with "unknown" as author
in the "git annotate" output. It has the repository name as prefix
in the subject line; importing that commit works best with a
layer hook which does not add the repository name again when
it is already present. Adding it here is useful for hooks
which do not extend the subject line.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Instead of local graft entries rooting the imported branches in the
shared root commit, use replacement objects.
The advantage is that they get moved around by "git push" and "git
fetch", so everyone has the same, nicer view with everything starting
at the beginning of the combined repository.
If undesired, these objects can be removed with "git replace".
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The new --history parameter enables a new mode in "combo-layer init"
where it copies the entire history of the components into the new
combined repository. This also imports merge commits.
Moving into a destination directory and applying commit hooks
is done via "git filter-branch" of the upstream branch. File
filtering uses the same code as before and just applies it
to that filtered branch to create the final commit which
then gets merged into the master branch of the new repository.
When multiple components are involved, they all get merged
into a single commit with an octopus merge. This depends
on a common ancestor, which is grafted onto the filtered
branches via .git/info/grafts.
These grafts are currently left in place. However, they do not get
pushed, so the local view on the entire history (all branches rooted
in the initial, empty commit, temporarily diverging and then
converging) is not the same as what others will see (branches starting
independently and converging). Perhaps "git replace" should be used
instead.
The final commit needs to be done manually, as before. A commit
message with some tracking information is ready for use as-is. This
information should be sufficient to implement also "combo-layer
update" using this approach, if desired. The advantage would be that
merge commits with conflict resolution would not longer break
the update.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
The Configuration class mirrors all properties in local hashes.
When updating the configuration, also update these properties
to remain consistent.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Some combined repos intentionally do not include certain files.
For example, Poky does not include bitbake's setup files and
OE-core's sample files under meta/conf.
When these files get modified in the upstream repository, applying the
patches fails and requires manual intervention. That is merely a
nuisance for someone familiar with the problem, but a real show
stopper when having the import run automatically or by someone less
experienced.
Therefore this change introduces "file_exclude", a new per-repo list
of file patterns which removes all matching files when initializing or
updating a combined repository. Because fnmatch is used under the hood
to match full path strings, removing entire directories must be done
with a pattern ending in a '/*' (in contrast to file_filter).
For Poky, the additional configuration looks like this:
[bitbake]
...
file_exclude = classes/base.bbclass
conf/bitbake.conf
.gitignore
MANIFEST.in
setup.py
TODO
[openembedded-core]
...
file_exclude = meta/conf/bblayers.conf.sample
meta/conf/local.conf.sample
meta/conf/local.conf.sample.extended
meta/conf/site.conf.sample
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Allow the caller to specify a separate output stream. stderr is always
a temporary file opened by runcmd(), so read from that to capture
output for error reporting *and* the return value.
The reasoning for the latter is a) that this preserves the traditional
behavior when out=None and b) if the caller wants the content of
stdout, it can read from the stream itself, which is not possible for
the temporary stderr.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
It depends on the diligence of the person running the combo-layer tool
whether the Signed-off-by line added to each commit actually indicates
that the person was involved in validating the change.
When the import is purely automatic, it is better to not add the line,
because the history is more useful without it (searching for the person
really only lists changes he or she was involved with) and it would
be a false statement.
The 'signoff' property can be set per repository, like every
other property. But setting it in the special [DEFAULT] section
is more useful, so that is what the example shows.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
When updating a property (at the moment, only 'last_revision'),
first check whether the user already added it to the main config.
If so, update there even if there is a local config.
This way, 'last_revision' can be shared between developers
as part of the repository while still configuring per-developer
repo paths outside of the git repository in a local config.
An example of a repository which is set up like that is tizen-distro,
with instructions for such a collaborative maintenance found here:
https://review.tizen.org/git/?p=scm/bb/tizen-distro.git;a=blob;f=README;hb=refs/heads/tizen-ivi
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Support defining the top commit up to which to update. In other words,
this makes it possible to update up to certain point other than the
branch head. The update point (git commitish) is given on the command
line by appending the component name(s) with a colon and the commitish,
e.g.
$ combo-layer update my_component:sha1
Only the "update" action supports this.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
Change get_repos() to assume a list of repository names instead of full
list of command line arguments.
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
|
|
If file_filter is set, git format-patch takes account but git rev-list
does not. So revlist is going to get with wrong revisions. And last_revision
will be updated with wrong revision. The next time that user run
combo-layer it complain about applying patches.
So ensure that 'git rev-list' are using file_filter as 'git format-patch'.
Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
on more than one branch
If a revision is in more than one branch, the check_rev_branch() function can't
cope with it and the tool returns incorrect errror messages. This patch
ensures it copes with this situation.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If both branch and last_revision are specified for a component when
combo-layer init is run, ensure that the specified revision is actually
on the specified branch and error out if not. Also ensure that the error
message mentions the component.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
If last_revision is specified for a component when running combo-layer
init, then use that revision instead of the latest revision on the
branch. Also, remove unnecessary git checkout during init since we
specify the revision to all calls to git when dealing with the component
repositories.
Fixes [YOCTO #3040].
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
Handle the case during update where the configured branch is not
currently checked out in the component repository by just specifying
it in the places where it was not previously.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
|
|
If the local configuration is already split out, ensure the init action
works properly and does not error in the case that last_revision is not
set. Additionally, if the local configuration is within the repository,
prevent it from being committed and add it to .gitignore.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
Allow splitting the local parts of the configuration (mostly
local_repo_dir and last_revision, although there is no limitation) to
a side-by-side -local.conf file, with component sections optionally
tagged with the combo layer branch name. This effectively allows you to:
* avoid polluting the history by committing the updated last revision
to the combo repository for every update
* avoid putting local repo paths into the combo repository
* manage multiple branches of the combo repository whilst avoiding the
possibility of mixing the configuration for one branch with another.
An example split configuration (note, values may be artificial):
------------------- combo-layer.conf -------------------
[bitbake]
src_uri = git://git.openembedded.org/bitbake
dest_dir = bitbake
hook = scripts/combo-layer-hook-default.sh
[oe-core]
src_uri = git://git.openembedded.org/openembedded-core
dest_dir = .
hook = scripts/combo-layer-hook-default.sh
--------------------------------------------------------
---------------- combo-layer-local.conf ----------------
[bitbake]
local_repo_dir = ../repos/bitbake
[oe-core]
local_repo_dir = ../repos/oe-core
[bitbake|master]
branch = master
last_revision = db689a99beffea1a285cdfc74a58fe73f1666987
[oe-core|master]
branch = master
last_revision = 121a1499a81706366acc0081272a6bff634d4d62
[bitbake|denzil]
branch = 1.12
last_revision = 24b631acdaa143a4de39c6e1328849660c66f219
[oe-core|denzil]
branch = denzil
last_revision = 741146fa90f28f7ce8d82ee7f7e254872d519724
--------------------------------------------------------
It is assumed that the local config file will be added to .gitignore.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
|
If the user edits the configuration file by hand and sets last_revision,
we need to ensure that the revision is valid and on the specified
branch.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
|
