The code detecting empty patches after removing files with
file_exclude failed for commits which were already empty before (like
the initial commit in some repos): such patches are completely empty
files, without a From line.

Detect that case and just let the normal empty patch detection deal
with it.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

"filterdiff -x ./some/file" does not remove changes for some/file.
We must be more careful about constructing the path name and
only add the prefix when it really means a directory.

While at it, also better normalize the path in copy_selected_files()
early on, to handle double slashes. Useful should the function ever
gets used for something other that dest_dir (which gets normalized in
sanity_check()).

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Empty dest_dir is basically undocumented behavior. The sample conf
only mentions using just a dot for the current directory. In practice,
the empty string does not work because of code like this:

def action_splitpatch(conf, args):
    ...
        if dest_dir != ".":
            filerange_root = '%s -x "%s/*"' % (filerange_root, dest_dir)

However, the empty string was not explicitly checked for, leading to
strange errors when trying to apply patches:

[12:50:23] Applying: foobar: xyz
fatal: unable to stat newly created file '/foobar': No such file or directory

This patch turns the empty string into an alias for the dot. This seems
more user-friendly than throwing an error. This alias is intentionally
not document in the sample conf, because the dot is clearer and works also
with older copies of combo-layer.

Instead of checking for both all the time and normalizing the path when
needed (as done in some places), rewrite the value in sanity_check()
and then only check for '.'.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Running an update operation with no repo/revision specified was failing.
This fixes that code path which worked until the change from:

http://git.yoctoproject.org/cgit.cgi/poky/commit/scripts/combo-layer?id=3592507a149b668c0a925e176535f7f2002fa543

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

The new "since_revision" property can be used to cut off the imported
history at some point. This is useful to keep the resulting repository
smaller while still preserving enough history that "git annotate"
reports the right author and commit for most lines.

The initial, squashed import commit shows up with "unknown" as author
in the "git annotate" output. It has the repository name as prefix
in the subject line; importing that commit works best with a
layer hook which does not add the repository name again when
it is already present. Adding it here is useful for hooks
which do not extend the subject line.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>

Instead of local graft entries rooting the imported branches in the
shared root commit, use replacement objects.

The advantage is that they get moved around by "git push" and "git
fetch", so everyone has the same, nicer view with everything starting
at the beginning of the combined repository.

If undesired, these objects can be removed with "git replace".

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>

The new --history parameter enables a new mode in "combo-layer init"
where it copies the entire history of the components into the new
combined repository. This also imports merge commits.

Moving into a destination directory and applying commit hooks
is done via "git filter-branch" of the upstream branch. File
filtering uses the same code as before and just applies it
to that filtered branch to create the final commit which
then gets merged into the master branch of the new repository.

When multiple components are involved, they all get merged
into a single commit with an octopus merge. This depends
on a common ancestor, which is grafted onto the filtered
branches via .git/info/grafts.

These grafts are currently left in place. However, they do not get
pushed, so the local view on the entire history (all branches rooted
in the initial, empty commit, temporarily diverging and then
converging) is not the same as what others will see (branches starting
independently and converging). Perhaps "git replace" should be used
instead.

The final commit needs to be done manually, as before. A commit
message with some tracking information is ready for use as-is. This
information should be sufficient to implement also "combo-layer
update" using this approach, if desired. The advantage would be that
merge commits with conflict resolution would not longer break
the update.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>

The Configuration class mirrors all properties in local hashes.
When updating the configuration, also update these properties
to remain consistent.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>

Some combined repos intentionally do not include certain files.
For example, Poky does not include bitbake's setup files and
OE-core's sample files under meta/conf.

When these files get modified in the upstream repository, applying the
patches fails and requires manual intervention. That is merely a
nuisance for someone familiar with the problem, but a real show
stopper when having the import run automatically or by someone less
experienced.

Therefore this change introduces "file_exclude", a new per-repo list
of file patterns which removes all matching files when initializing or
updating a combined repository. Because fnmatch is used under the hood
to match full path strings, removing entire directories must be done
with a pattern ending in a '/*' (in contrast to file_filter).

For Poky, the additional configuration looks like this:

[bitbake]
...
file_exclude = classes/base.bbclass
	conf/bitbake.conf
	.gitignore
	MANIFEST.in
	setup.py
	TODO

[openembedded-core]
...
file_exclude = meta/conf/bblayers.conf.sample
	meta/conf/local.conf.sample
	meta/conf/local.conf.sample.extended
	meta/conf/site.conf.sample

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>

Allow the caller to specify a separate output stream. stderr is always
a temporary file opened by runcmd(), so read from that to capture
output for error reporting *and* the return value.

The reasoning for the latter is a) that this preserves the traditional
behavior when out=None and b) if the caller wants the content of
stdout, it can read from the stream itself, which is not possible for
the temporary stderr.

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>