From 2982e7c4059ffd0441c458a222f5437918c956fc Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Wed, 24 Nov 2010 08:43:34 -0800 Subject: documentation/kernel-manual/yocto-project-kernel-manual.xml and yocto-project-kernel-manual-customization.xsl: Renamed files with correct spelling for kernel. I had to rename these files because I had kernel mis-spelled when I committed the original manual files. Signed-off-by: Scott Rifenbark --- .../yocto-project-kernel-manual-customization.xsl | 8 + .../kernel-manual/yocto-project-kernel-manual.xml | 2175 ++++++++++++++++++++ 2 files changed, 2183 insertions(+) create mode 100644 documentation/kernel-manual/yocto-project-kernel-manual-customization.xsl create mode 100644 documentation/kernel-manual/yocto-project-kernel-manual.xml (limited to 'documentation') diff --git a/documentation/kernel-manual/yocto-project-kernel-manual-customization.xsl b/documentation/kernel-manual/yocto-project-kernel-manual-customization.xsl new file mode 100644 index 0000000000..8e6ea34dd4 --- /dev/null +++ b/documentation/kernel-manual/yocto-project-kernel-manual-customization.xsl @@ -0,0 +1,8 @@ + + + + + + + + diff --git a/documentation/kernel-manual/yocto-project-kernel-manual.xml b/documentation/kernel-manual/yocto-project-kernel-manual.xml new file mode 100644 index 0000000000..b1693500fc --- /dev/null +++ b/documentation/kernel-manual/yocto-project-kernel-manual.xml @@ -0,0 +1,2175 @@ + + +
+ + +
+ Yocto Project Kernel Architecture and Use Manual +
+ +
+ Introduction + + Yocto Project presents the kernel as a fully patched, history-clean git + repository. + The git tree represents the selected features, board support, + and configurations extensively tested by Yocto Project. + The Yocto Project kernel allows the end user to leverage community + best practices to seamlessly manage the development, build and debug cycles. + + + This manual describes the Yocto Project kernel by providing information + on its history, organization, benefits, and use. + The manual consists of two sections: + + Concepts - Describes concepts behind the kernel. + You will understand how the kernel is organized and why it is organized in + the way it is. You will understand the benefits of the kernel's organization + and the mechanisms used to work with the kernel and how to apply it in your + design process. + Using the Kernel - Describes best practices and "how-to" information + that lets you put the kernel to practical use. Some examples are "How to Build a + Project Specific Tree", "How to Examine Changes in a Branch", and "Saving Kernel + Modifications." + + + + For more information on the kernel, see the following links: + + + + + + + You can find more information on Yocto Project by visiting the website at + . + + +
+ +
+ Concepts + + This section provides conceptual information about the Yocto Project kernel: + + Kernel Goals + Yocto Project Kernel Development and Maintenance Overview + Kernel Architecture + Kernel Tools + + +
+ Kernel Goals + + The complexity of embedded kernel design has increased dramatically. + Whether it is managing multiple implementations of a particular feature or tuning and + optimizing board specific features, flexibility and maintainability are key concerns. + The Yocto Project Linux kernel is presented with the embedded + developer's needs in mind and has evolved to assist in these key concerns. + For example, prior methods such as applying hundreds of patches to an extracted + tarball have been replaced with proven techniques that allow easy inspection, + bisection and analysis of changes. + Application of these techniques also creates a platform for performing integration and + collaboration with the thousands of upstream development projects. + + + With all these considerations in mind, the Yocto Project kernel and development team + strives to attain these goals: + + Allow the end user to leverage community best practices to seamlessly + manage the development, build and debug cycles. + Create a platform for performing integration and collaboration with the + thousands of upstream development projects that exist. + Provide mechanisms that support many different work flows, front-ends and + management techniques. + Deliver the most up-to-date kernel possible while still ensuring that + the baseline kernel is the the most stable official release. + Include major technological features as part of Yocto Project's up-rev + strategy. + Present a git tree, that just like the upstream kernel.org tree, has a + clear and continuous history. + Deliver a key set of supported kernel types, where each type is tailored + to a specific use case (i.g. networking, consumer, devices, and so forth). + Employ a git branching strategy that from a customer's point of view + results in a linear path from the baseline kernel.org, through a select group of features and + ends with their BSP-specific commits. + + +
+ +
+ Yocto Project Kernel Development and Maintenance Overview + + Yocto Project kernel, like other kernels, is based off the Linux kernel release + from . + At the beginning of our major development cycle, we choose our Yocto Project kernel + based on factors like release timing, the anticipated release timing of "final" (i.e. non "rc") + upstream kernel.org versions, and Yocto Project feature requirements. + Typically this will be a kernel that is in the + final stages of development by the community (i.e. still in the release + candidate or "rc" phase) and not yet a final release. + But by being in the final stages of external development, we know that the + kernel.org final release will clearly land within the early stages of + the Yocto Project development window. + + + This balance allows us to deliver the most up-to-date kernel + as possible, while still ensuring that we have a stable official release as + our baseline kernel version. + + + The following figure represents the overall place the Yocto Project kernel fills. + + + + + + In the figure the ultimate source for the Yocto Project kernel is a released kernel + from kernel.org. + In addition to a foundational kernel from kernel.org the commercially released + Yocto Project kernel contains a mix of important new mainline + developments, non-mainline developments, Board Support Package (BSP) developments, + and custom features. + These additions result in a commercially released Yocto Project kernel that caters + to specific embedded designer needs for targeted hardware. + + + Once a Yocto Project kernel is officially released the Yocto Project team goes into + their next development cycle, or "uprev" cycle. + It is important to note that the most sustainable and stable way + to include feature development upstream is through a kernel uprev process. + Back-porting of hundreds of individual fixes and minor features from various + kernel versions is not sustainable and can easily compromise quality. + During the uprev cycle, the Yocto Project team uses an ongoing analysis of + kernel development, BSP support, and release timing to select the best + possible kernel.org version. + The team continually monitors community kernel + development to look for significant features of interest. + The illustration depicts this by showing the team looking back to kernel.org for new features, + BSP features, and significant bug fixes. + The team does consider back-porting large features if they have a significant advantage. + User or community demand can also trigger a back-port or creation of new + functionality in the Yocto Project baseline kernel during the uprev cycle. + + + Generally speaking, every new kernel both adds features and introduces new bugs. + These consequences are the basic properties of upstream kernel development and are + managed by the Yocto Project team's kernel strategy. + It is the Yocto Project team's policy to not back-port minor features to the released kernel. + They only consider back-porting significant technological jumps - and, that is done + after a complete gap analysis. + The reason for this policy is that simply back-porting any small to medium sized change + from an evolving kernel can easily create mismatches, incompatibilities and very + subtle errors. + + + These policies result in both a stable and a cutting + edge kernel that mixes forward ports of existing features and significant and critical + new functionality. + Forward porting functionality in the Yocto Project kernel can be thought of as a + "micro uprev." + The many “micro uprevs” produce a kernel version with a mix of + important new mainline, non-mainline, BSP developments and feature integrations. + This kernel gives insight into new features and allows focused + amounts of testing to be done on the kernel, which prevents + surprises when selecting the next major uprev. + The quality of these cutting edge kernels is evolving and the kernels are used in very special + cases for BSP and feature development. + +
+ +
+ Kernel Architecture + + This section describes the architecture of the Yocto Project kernel and provides information + on the mechanisms used to achieve that architecture. + + +
+ Overview + + As mentioned earlier, a key goal of Yocto Project is to present the developer with + a kernel that has a clear and continuous history that is visible to the user. + The architecture and mechanisms used achieve that goal in a manner similar to the + upstream kernel.org. + + + + You can think of the Yocto Project kernel as consisting of a baseline kernel with + added features logically structured on top of the baseline. + The features are tagged and organized by way of a branching strategy implemented by the + source code manager (SCM) git. + The result is that the user has the ability to see the added features and + the commits that make up those features. + In addition to being able to see added features, the user can also view the history of what + made up the baseline kernel as well. + + + The following illustration shows the conceptual Yocto Project kernel. + + + + + + In the illustration, the "kernel.org Branch Point" marks the specific spot (or release) from + which the Yocto Project kernel is created. From this point "up" in the tree features and + differences are organized and tagged. + + + The "Yocto Project Baseline Kernel" contains functionality that is common to every kernel + type and BSP that is organized further up the tree. Placing these common features in the + tree this way means features don't have to be duplicated along individual branches of the + structure. + + + From the Yocto Project Baseline Kernel branch points represent specific functionality + for individual BSPs as well as real-time kernels. + The illustration represents this through three BSP-specific branches and a real-time + kernel branch. + Each branch represents some unique functionality for the BSP or a real-time kernel. + + + The real-time kernel branch has common features for all real-time kernels and contains + more branches for individual BSP-specific real-time kernels. + The illustration shows three branches as an example. + Each branch points the way to specific, unique features for a respective real-time + kernel as they apply to a given BSP. + + + The resulting tree structure presents a clear path of markers (or branches) to the user + that for all practical purposes is the kernel needed for any given set of requirements. + +
+ +
+ Branching Strategy and Workflow + + The Yocto Project team creates kernel branches at points where functionality is + no longer shared and thus, needs to be isolated. + For example, board-specific incompatibilities would require different functionality + and would require a branch to separate the features. + Likewise, for specific kernel features the same branching strategy is used. + This branching strategy results in a tree that has features organized to be specific + for particular functionality, single kernel types, or a subset of kernel types. + This strategy results in not having to store the same feature twice internally in the + tree. + Rather we store the unique differences required to apply the feature onto the kernel type + in question. + + + BSP-specific code additions are handled in a similar manner to kernel-specific additions. + Some BSPs only make sense given certain kernel types. + So, for these types, we create branches off the end of that kernel type for all + of the BSPs that are supported on that kernel type. + From the perspective of the tools that create the BSP branch, the BSP is really no + different than a feature. + Consequently, the same branching strategy applies to BSPs as it does to features. + So again, rather than store the BSP twice, only the unique differences for the BSP across + the supported multiple kernels are uniquely stored. + + + While this strategy results in a tree with a significant number of branches, it is + important to realize that from the customer's point of view, there is a linear + path that travels from the baseline kernel.org, through a select group of features and + ends with their BSP-specific commits. + In other words, the divisions of the kernel are transparent and are not relevant + to the developer on a day-to-day basis. + From the customer's perspective, this is the "master" branch. + They do not need not be aware of the existence of any other branches at all. + Of course there is value in the existence of these branches + in the tree, should a person decide to explore them. + For example, a comparison between two BSPs at either the commit level or at the line-by-line + code diff level is now a trivial operation. + + + Working with the kernel as a structured tree follows recognized community best practices. + In particular, the kernel as shipped with the product should be + considered an 'upstream source' and viewed as a series of + historical and documented modifications (commits). + These modifications represent the development and stabilization done + by the Yocto Project kernel development team. + + + Because commits only change at significant release points in the product life cycle, + developers can work on a branch created + from the last relevant commit in the shipped Yocto Project kernel. + As mentioned previously, the structure is transparent to the user + because the kernel tree is left in this state after cloning and building the kernel. + +
+ +
+ Source Code Manager - git + + The Source Code Manager (SCM) is git and it is the obvious mechanism for meeting the + previously mentioned goals. + Not only is it the SCM for kernel.org but git continues to grow in popularity and + supports many different work flows, front-ends and management techniques. + + + It should be noted that you can use as much, or as little, of what git has to offer + as is appropriate to your project. + +
+
+ +
+ Kernel Tools + +Since most standard workflows involve moving forward with an existing tree by +continuing to add and alter the underlying baseline, the tools that manage +Yocto Project's kernel construction are largely hidden from the developer to +present a simplified view of the kernel for ease of use. + + +The fundamental properties of the tools that manage and construct the +kernel are: + + the ability to group patches into named, reusable features + to allow top down control of included features + the binding of kernel configuration to kernel patches/features + the presentation of a seamless git repository that blends Yocto Project value with the kernel.org history and development + + + +The tools that construct a kernel tree will be discussed later in this +document. The following tools form the foundation of the Yocto Project +kernel toolkit: + + git : distributed revision control system created by Linus Torvalds + guilt: quilt on top of git + *cfg : kernel configuration management and classification + kgit*: Yocto Project kernel tree creation and management tools + scc : series & configuration compiler + + +
+
+ + + + + + +
+ How to get things accomplished with the kernel + + This section describes how to accomplish tasks involving the kernel's tree structure. + The information covers the following: + + Tree construction + Build strategies + Series & Configuration Compiler + kgit + Workflow examples + Source Code Manager (SCM) + Board Support Package (BSP) template migration + BSP creation + Patching + Updating BSP patches and configuration + guilt + scc file example + "dirty" string + Transition kernel layer + + + +
+ Tree Construction + +The Yocto Project kernel repository, as shipped with the product, is created by +compiling and executing the set of feature descriptions for every BSP/feature +in the product. Those feature descriptions list all necessary patches, +configuration, branching, tagging and feature divisions found in the kernel. + + +The files used to describe all the valid features and BSPs in the Yocto Project +kernel can be found in any clone of the kernel git tree. The directory +wrs/cfg/kernel-cache/ is a snapshot of all the kernel configuration and +feature descriptions (.scc) that were used to build the kernel repository. +It should however be noted, that browsing the snapshot of feature +descriptions and patches is not an effective way to determine what is in a +particular kernel branch. Using git directly to get insight into the changes +in a branch is more efficient and a more flexible way to inspect changes to +the kernel. Examples of using git to inspect kernel commits are in the +following sections. + + +As a reminder, it is envisioned that a ground up reconstruction of the +complete kernel tree is an action only taken by Yocto Project staff during an +active development cycle. When an end user creates a project, it takes +advantage of this complete tree in order to efficiently place a git tree +within their project. + + +The general flow of the project specific kernel tree construction is as follows: + + a top level kernel feature is passed to the kernel build subsystem, + normally this is a BSP for a particular kernel type. + + the file that describes the top level feature is located by searching + system directories: + + + the kernel-cache under linux/wrs/cfg/kernel-cache + kernel-*-cache directories in layers + configured and default templates + + + In a typical build a feature description of the format: + <bsp name>-<kernel type>.scc is the target of the search. + + + once located, the feature description is compiled into a simple script + of actions, or an existing equivalent script which was part of the + shipped kernel is located. + + extra features are appended to the top level feature description. Extra + features can come from the command line, the configure script or + templates. + + each extra feature is located, compiled and appended to the script from + step #3 + + the script is executed, and a meta-series is produced. The meta-series + is a description of all the branches, tags, patches and configuration that + need to be applied to the base git repository to completely create the + "bsp_name-kernel_type". + + the base repository (normally kernel.org) is cloned, and the actions + listed in the meta-series are applied to the tree. + + the git repository is left with the desired branch checked out and any + required branching, patching and tagging has been performed. + + + + +The tree is now ready for configuration and compilation. Those two topics will +be covered below. + + +The end user generated meta-series adds to the kernel as shipped with + the Yocto Project release. Any add-ons and configuration data are applied + to the end of an existing branch. The full repository generation that + is found in the linux-2.6-windriver.git is the combination of all + supported boards and configurations. + + + +This technique is flexible and allows the seamless blending of an immutable +history with additional deployment specific patches. Any additions to the +kernel become an integrated part of the branches. + + +It is key that feature descriptions indicate if any branches are + required, since the build system cannot automatically decide where a + BSP should branch or if that branch point needs a name with + significance. There is a single restriction enforced by the compilation + phase: + + A BSP must create a branch of the format <bsp name>-<kernel type>. + + This means that all merged/support BSPs must indicate where to start + its branch from, with the right name, in its .scc files. The scc + section describes the available branching commands in more detail. + + + + +A summary of end user tree construction activities follow: + + compile and link a full top-down kernel description from feature descriptions + execute the complete description to generate a meta-series + interpret the meta-series to create a customized git repository for the + board + migrate configuration fragments and configure the kernel + checkout the BSP branch and build + + +
+ +
+ Build Strategy + +There are some prerequisites that must be met before starting the compilation +phase of the kernel build system: + + + There must be a kernel git repository indicated in the SRC_URI. + There must be a branch <bsp name>-<kernel type>. + + + +These are typically met by running tree construction/patching phase of the +build system, but can be achieved by other means. Examples of alternate work +flows such as bootstrapping a BSP are provided below. + + +Before building a kernel it is configured by processing all of the +configuration "fragments" specified by the scc feature descriptions. As the +features are compiled, associated kernel configuration fragments are noted +and recorded in the meta-series in their compilation order. The +fragments are migrated, pre-processed and passed to the Linux Kernel +Configuration subsystem (lkc) as raw input in the form of a .config file. +The lkc uses its own internal dependency constraints to do the final +processing of that information and generates the final .config that will +be used during compilation. + + +Kernel compilation is started, using the board's architecture and other +relevant values from the board template, and a kernel image is produced. + + +The other thing that you will first see once you configure a kernel is that +it will generate a build tree that is separate from your git source tree. +This build dir will be called "linux-<BSPname>-<kerntype>-build" where +kerntype is one of standard, cg`` +e, etc. This functionality is done by making +use of the existing support that is within the kernel.org tree by default. + + +What this means, is that all the generated files (that includes the final +".config" itself, all ".o" and ".a" etc) are now in this directory. Since +the git source tree can contain any number of BSPs, all on their own branch, +you now can easily switch between builds of BSPs as well, since each one also +has their own separate build directory. + +
+ +
+ Series & Configuration Compiler (SCC) + +In early versions of the product, kernel patches were simply listed in a flat +file called "patches.list", and then quilt was added as a tool to help +traverse this list, which in quilt terms was called a "series" file. + + +Before the 2.0 release, it was already apparent that a static series file was +too inflexible, and that the series file had to become more dynamic and rely +on certain state (like kernel type) in order to determine whether a patch was +to be used or not. The 2.0 release already made use of some stateful +construction of series files, but since the delivery mechanism was unchanged +(tar + patches + series files), most people were not aware of anything really +different. The 3.0 release continues with this stateful construction of +series files, but since the delivery mechanism is changed (git + branches) it +now is more apparent to people. + + +As was previously mentioned, scc is a "series and configuration +compiler". Its role is to combine feature descriptions into a format that can +be used to generate a meta-series. A meta series contains all the required +information to construct a complete set of branches that are required to +build a desired board and feature set. The meta series is interpreted by the +kgit tools to create a git repository that could be built. + + +To illustrate how scc works, a feature description must first be understood. +A feature description is simply a small bash shell script that is executed by +scc in a controlled environment. Each feature description describes a set of +operations that add patches, modify existing patches or configure the +kernel. It is key that feature descriptions can include other features, and +hence allow the division of patches and configuration into named, reusable +containers. + + +Each feature description can use any of the following valid scc commands: + + shell constructs: bash conditionals and other utilities can be used in a feature + description. During compilation, the working directory is the feature + description itself, so any command that is "raw shell" and not from the + list of supported commands, can not directly modify a git repository. + + patch <relative path>/<patch name>: outputs a patch to be included in a feature's patch set. Only the name of + the patch is supplied, the path is calculated from the currently set + patch directory, which is normally the feature directory itself. + + patch_trigger >condition< >action< <tgt>: indicate that a trigger should be set to perform an action on a + patch. + +The conditions can be: + + + arch:<comma separated arch list or "all"> + plat:<comma separated platform list or "all"> + +The action can be: + + exclude: This is used in exceptional situations where a patch + cannot be applied for certain reasons (arch or platform). + When the trigger is satisfied the patch will be removed from + the patch list. + include: This is used to include a patch only for a specific trigger. + Like exclude, this should only be used when necessary. + It takes 1 argument, the patch to include. + + + include <feature name> [after <feature>]: includes a feature for processing. The feature is "expanded" at the + position of the include directive. This means that any patches, + configuration or sub-includes of the feature will appear in the final + series before the commands that follow the include. + + include searches the include directories for a matching feature name, + include directories are passed to scc by the caller using -I <path> and + is transparent to the feature script. This means that <feature name> must + be relative to one of the search paths. For example, if + /opt/kernel-cache/feat/sched.scc is to be included and scc is invoked + with -I /opt/kernel-cache, then a feature would issue "include + feat/sched.scc" to include the feature. + + + The optional "after" directive allows a feature to modify the existing + order of includes and insert a feature after the named feature is + processed. Note: the "include foo after bar" must be issued before "bar" + is processed, so is normally only used by a new top level feature to + modify the order of features in something it is including. + + exclude <feature name>: Indicates that a particular feature should *not* be included even if an + 'include' directive is found. The exclude must be issued before the + include is processed, so is normally only used by a new top level feature + to modify the order of features in something it is including. + + git <command>: Issues any git command during tree construction. Note: this command is + not validated/sanitized so care must be taken to not damage the + tree. This can be used to script branching, tagging, pulls or other git + operations. + + dir <directory>: changes the working directory for "patch" directives. This can be used to + shorten a long sequence of patches by not requiring a common relative + directory to be issued each time. + + kconf <type> <fragment name>: associates a kernel config frag with the feature. + <type> can be + "hardware" or "non-hardware" and is used by the kernel configuration + subsystem to audit configuration. <fragment name> is the name of a file + in the current feature directory that contains a series of kernel + configuration options. There is no restriction on the chosen fragment + name, although a suffix of ".cfg" is recommended. Multiple fragment + specifications are supported. + + branch <branch name>: creates a branch in the tree. All subsequent patch commands will be + applied to the new branch and changes isolated from the rest of the + repository. + + scc_leaf <base feature> <branch name>: Performs a combination feature include and branch. This is mainly a + convenience directive, but has significance to some build system bindings + as a sentinel to indicate that this intends to create a branch that is + valid for kernel compilation. + + tag <tag name>: Tags the tree. The tag will be applied in processing order, so will + be after already applied patches and precede patches yet to be applied. + + define <var> <value>: Creates a variable with a particular value that can be used in subsequent + feature descriptions. + + + +
+ +
+ kgit Tools + +The kgit tools are responsible for constructing and maintaining the Wind +River kernel repository. These activities include importing, exporting, and +applying patches as well as sanity checking and branch management. From the +developers perspective, the kgit tools are hidden and rarely require +interactive use. But one tool in particular that warrants further description +is "kgit-meta". + + +kgit-meta is the actual application of feature description(s) to a kernel repo. +In other words, it is responsible for interpreting the meta series generated +from a scc compiled script. As a result, kgit-meta is coupled to the set of +commands permitted in a .scc feature description (listed in the scc section). +kgit-meta understands both the meta series format and how to use git and +guilt to modify a base git repository. It processes a meta-series line by +line, branching, tagging, patching and tracking changes that are made to the +base git repository. + + +Once kgit-meta has processed a meta-series, it leaves the repository with the +last branch checked out, and creates the necessary guilt infrastructure to +inspect the tree, or add to it via using guilt. As was previously mentioned, +guilt is not required, but is provided as a convenience. Other utilities such +as quilt, stgit, git or others can also be used to manipulate the git +repository. + +
+ +
+ Workflow Examples + + +As previously noted, the Yocto Project kernel has built in git/guilt +integration, but these utilities are not the only way to work with the kernel +repository. Yocto Project has not made changes to git, or other tools that +invalidate alternate workflows. Additionally, the way the kernel repository +is constructed uses only core git functionality allowing any number of tools +or front ends to use the resulting tree. + +This section contains several workflow examples. + + +
+ Change Inspection: Kernel Changes/Commits + +A common question when working with a BSP/kernel is: "What changes have been applied to this tree?" + + +In previous Yocto Project releases, there were a collection of directories that +contained patches to the kernel, those patches could be inspected, grep'd or +otherwise used to get a general feeling for changes. This sort of patch +inspection is not an efficient way to determine what has been done to the +kernel, since there are many optional patches that are selected based on the +kernel type and feature description, not to mention patches that are actually +in directories that are not being searched. + + +A more effective way to determine what has changed in the kernel is to use +git and inspect / search the kernel tree. This is a full view of not only the +source code modifications, but the reasoning behind the changes. + +
+ What Changed in a BSP? + +These examples could continue for some time, since the Yocto Project git +repository doesn't break existing git functionality and there are nearly +endless permutations of those commands. Also note that unless a commit range +is given (<kernel type>..<bsp>-<kernel type>), kernel.org history is blended +with Yocto Project changes + + + # full description of the changes + > git whatchanged <kernel type>..<bsp>-<kernel type> + > eg: git whatchanged standard..common_pc-standard + + # summary of the changes + > git log ‐‐pretty=oneline ‐‐abbrev-commit <kernel type>..<bsp>-<kernel type> + + # source code changes (one combined diff) + > git diff <kernel type>..<bsp>-<kernel type> + > git show <kernel type>..<bsp>-<kernel type> + + # dump individual patches per commit + > git format-patch -o <dir> <kernel type>..<bsp>-<kernel type> + + # determine the change history of a particular file + > git whatchanged <path to file> + + # determine the commits which touch each line in a file + > git blame <path to file> + +
+ +
+ Show a Particular Feature or Branch Change + +Significant features or branches are tagged in the Yocto Project tree to divide +changes. Remember to first determine (or add) the tag of interest. Note: +there will be many tags, since each BSP branch is tagged, kernel.org tags and +feature tags are all present. + + + # show the changes tagged by a feature + > git show <tag> + > eg: git show yaffs2 + + # determine which branches contain a feature + > git branch ‐‐contains <tag> + + # show the changes in a kernel type + > git whatchanged wrs_base..<kernel type> + > eg: git whatchanged wrs_base..standard + + +Many other comparisons can be done to isolate BSP changes, such as comparing +to kernel.org tags (v2.6.27.18, etc), per subsystem comparisons (git +whatchanged mm) or many other types of checks. + +
+
+ +
+ Development: Saving Kernel Modifications + +Another common operation is to build a Yocto Project supplied BSP, make some +changes, rebuild and test. Those local changes often need to be exported, +shared or otherwise maintained. + + +Since the Yocto Project kernel source tree is backed by git, this activity is +greatly simplified and is much easier than in previous releases. git tracks +file modifications, additions and deletions, which allows the developer to +modify the code and later realize that the changes should be saved, and +easily determine what was changed. It also provides many tools to commit, +undo and export those modifications. + + +There are many ways to perform this action, and the technique employed +depends on the destination for the patches, which could be any of: + + bulk storage + internal sharing either through patches or using git + external submission + export for integration into another SCM + + + +The destination of the patches also incluences the method of gathering them +due to issues such as: + + bisectability + commit headers + division of subsystems for separate submission / review + + + +
+ Bulk Export + +If patches are simply being stored outside of the kernel source repository, +either permanently or temporarily, then there are several methods that can be +used. + + +Note the "bulk" in this discussion, these techniques are not appropriate for +full integration of upstream submission, since they do not properly divide +changes or provide an avenue for per-change commit messages. This example +assumes that changes have not been committed incrementally during development +and simply must be gathered and exported. + + # bulk export of ALL modifications without separation or division + # of the changes + + > git add . + > git commit -s -a -m >commit message< + or + > git commit -s -a # and interact with $EDITOR + + + +These operations have captured all the local changes in the project source +tree in a single git commit, and that commit is also stored in the project's +source tree. + + +Once exported, those changes can then be restored manually, via a template or +through integration with the default_kernel. Those topics are covered in +future sections. + +
+ +
+ Incremental/Planned Sharing + +Note: unlike the previous "bulk" section, the following examples assume that +changes have been incrementally committed to the tree during development and +now are being exported. + + +During development the following commands will be of interest, but for full +git documentation refer to the git man pages or an online resource such as +http://github.com + + # edit a file + > vi >path</file + # stage the change + > git add >path</file + # commit the change + > git commit -s + # remove a file + > git rm >path</file + # commit the change + > git commit -s + + ... etc. + + + +Distributed development with git is possible by having a universally agreed +upon unique commit identifier (set by the creator of the commit) mapping to a +specific changeset with a specific parent. This ID is created for you when +you create a commit, and will be re-created when you amend/alter or re-apply +a commit. As an individual in isolation, this is of no interest, but if you +intend to share your tree with normal git push/pull operations for +distributed development, you should consider the ramifications of changing a +commit that you've already shared with others. + + +Assuming that the changes have *not* been pushed upstream, or pulled into +another repository, both the commit content and commit messages associated +with development can be update via: + + > git add >path</file + > git commit ‐‐amend + > git rebase or git rebase -i + + + +Again, assuming that the changes have *not* been pushed upstream, and that +there are no pending works in progress (use "git status" to check) then +commits can be reverted (undone) via: + + # remove the commit, update working tree and remove all + # traces of the change + > git reset ‐‐hard HEAD^ + # remove the commit, but leave the files changed and staged for re-commit + > git reset ‐‐soft HEAD^ + # remove the commit, leave file change, but not staged for commit + > git reset ‐‐mixed HEAD^ + + + +Branches can be created, changes cherry-picked or any number of git +operations performed until the commits are in good order for pushing upstream +or pull requests. After a push or pull, commits are normally considered +'permanent' and should not be modified, only incrementally changed in new +commits. This is standard "git" workflow and Yocto Project recommends the +kernel.org best practices. + +It is recommend to tag or branch before adding changes to a Yocto Project + BSP (or creating a new one), since the branch or tag provides a + reference point to facilitate locating and exporting local changes. + + +
+ Export Internally Via Patches + +Committed changes can be extracted from a working directory by exporting them +as patches. Those patches can be used for upstream submission, placed in a +Yocto Project template for automatic kernel patching or many other common uses. + + + # >first commit> can be a tag if one was created before development + # began. It can also be the parent branch if a branch was created + # before development began. + + > git format-patch -o <dir> <first commit>..<last commit> + + + + + In other words: + + # identify commits of interest. + + # if the tree was tagged before development + > git format-patch -o <save dir> <tag> + + # if no tags are available + > git format-patch -o <save dir> HEAD^ # last commit + > git format-patch -o <save dir> HEAD^^ # last 2 commits + > git whatchanged # identify last commit + > git format-patch -o <save dir> <commit id> + > git format-patch -o <save dir> <rev-list> + + + + +The result is a directory with sequentially numbered patches, that when +applied to a repository using "git am", will reproduce the original commit +and all related information (author, date, commit log, etc) will be +preserved. Note that new commit IDs will be generated upon reapplication, +reflecting that the commit is now applied to an underlying commit with a +different ID. + + +See the "template patching" example for how to use the patches to +automatically apply to a new kernel build. + +
+ +
+ Export Internally Via git + +Committed changes can also be exported from a working directory by pushing +(or by making a pull request) the changes into a master repository. Those +same change can then be pulled into a new kernel build at a later time using this command form: + + git push ssh://<master server>/<path to repo> <local branch>:<remote branch> + +For example: + + > push ssh://openlinux.windriver.com/pub/git/kernel-2.6.27 common_pc-standard:common_pc-standard + +A pull request entails using "git request-pull" to compose an email to the +maintainer requesting that a branch be pulled into the master repository, see +http://github.com/guides/pull-requests for an example. + + +Other commands such as 'git stash' or branching can also be used to save +changes, but are not covered in this document. + + +See the section "importing from another SCM" for how a git push to the +default_kernel, can be used to automatically update the builds of all users +of a central git repository. + +
+
+ +
+ Export for External (Upstream) Submission + +If patches are to be sent for external submission, they can be done via a +pull request if the patch series is large or the maintainer prefers to pull +changes. But commonly, patches are sent as email series for easy review and +integration. + + +Before sending patches for review ensure that you understand the +standard of the community in question and follow their best practices. For +example, kernel patches should follow standards such as: + + + + Documentation/SubmittingPatches (in any linux kernel source tree) + + + +The messages used to commit changes are a large part of these standards, so +ensure that the headers for each commit have the required information. If the +initial commits were not properly documented or don't meet those standards +rebasing via git rebase -i offer an opportunity to manipulate the commits and +get them into the required format. Other techniques such as branching and +cherry picking commits are also viable options. + + +Once complete, patches are sent via email to the maintainer(s) or lists that +review and integrate changes. "git send-email" is commonly used to ensure +that patches are properly formatted for easy application and avoid mailer +induced patch damage. + + +An example of dumping patches for external submission follows: + + # dump the last 4 commits + > git format-patch ‐‐thread -n -o ~/rr/ HEAD^^^^ + > git send-email ‐‐compose ‐‐subject '[RFC 0/N] <patch series summary>' \ + ‐‐to foo@yoctoproject.org ‐‐to bar@yoctoproject.org \ + ‐‐cc list@yoctoproject.org ~/rr + # the editor is invoked for the 0/N patch, and when complete the entire + # series is sent via email for review + + +
+ +
+ Export for Import into Other SCM + +Using any one of the previously discussed techniques, commits can be exported +as patches for import into another SCM. Note however, that if those patches +are manually applied to a secondary tree and then that secondary tree is +checked into the SCM, then it often results in lost information (like commit +logs) and so it is not recommended. + + +Many SCMs can directly import git commits, or can translate git patches to +not lose information. Those facilities are SCM dependent and should be used +whenever possible. + +
+
+ +
+ SCM: Working with the Yocto Project Kernel in Another SCM + +This is not the same as the exporting of patches to another SCM, but instead +is concerned with kernel development that is done completely in another +environment, but built with the Yocto Project build system. In this scenario two +things must happen: + + The delivered Yocto Project kernel must be exported into the second + SCM. + Development must be exported from that secondary SCM into a + format that can be used by the Yocto Project build system. + + +
+ Exporting Delivered Kernel to SCM + +Depending on the SCM it may be possible to export the entire Yocto Project +kernel git repository, branches and all, into a new environment. This is the +preferred method, since it has the most flexibility and potential to maintain +the meta data associated with each commit. + + +When a direct import mechanism is not available, it is still possible to +export a branch (or series of branches) and check them into a new +repository. + + +The following commands illustrate some of the steps that could be used to +import the common_pc-standard kernel into a secondary SCM + + > git checkout common_pc-standard + > cd .. ; echo linux/.git > .cvsignore + > cvs import -m "initial import" linux MY_COMPANY start + +The CVS repo could now be relocated and used in a centralized manner. + + +The following commands illustrate how two BSPs could be condensed and merged +into a second SCM: + + > git checkout common_pc-standard + > git merge cav_ebt5800-standard + # resolve any conflicts and commit them + > cd .. ; echo linux/.git > .cvsignore + > cvs import -m "initial import" linux MY_COMPANY start + + +
+ +
+ Importing Changes for Build + +Once development has reached a suitable point in the second development +environment, changes can either be exported as patches or imported into git +directly (if a conversion/import mechanism is available for the SCM). + +If changes are exported as patches, they can be placed in a template and +automatically applied to the kernel during patching. See the template patch +example for details. + + +If changes are imported directly into git, they must be propagated to the +wrll-linux-2.6.27/git/default_kernel bare clone of each individual build +to be present when the kernel is checked out. + +The following example illustrates one variant of this workflow: + + # on master git repository + > cd linux-2.6.27 + > git tag -d common_pc-standard-mark + > git pull ssh://<foo>@<bar>/pub/git/kernel-2.6.27 common_pc-standard:common_pc-standard + > git tag common_pc-standard-mark + + # on each build machine (or NFS share, etc) + > cd wrll-linux-2.6.27/git/default_kernel + > git fetch ssh://<foo>@<master server>/pub/git/kernel-2.6.27 + + # in the build, perform a from-scratch build of Linux and the new changes + # will be checked out and built. + > make linux + + +
+
+ +
+ BSP: Template Migration from 2.0 + +The move to a git-backed kernel build system in 3.0 introduced a small new +requirement for any BSP that is not integrated into the GA release of the +product: branching information. + + +As was previously mentioned in the background sections, branching information +is always required, since the kernel build system cannot make intelligent +branching decisions and must rely on the developer. This branching +information is provided via a .scc file. + + +A BSP template in 2.0 contained build system information (config.sh, etc) and +kernel patching information in the 'linux' subdirectory. The same holds true +in 3.0, with only minor changes in the kernel patching directory. +The ".smudge" files are now ".scc" files and now contain a full description + of the kernel branching, patching and configuration for the BSP. Where in + 2.0, they only contained kernel patching information. + + +The following illustrates the migration of a simple 2.0 BSP template to the +new 3.0 kernel build system. + + +Note: all operations are from the root of a customer layer. + + + templates/ + `‐‐ board + `‐‐ my_board + |‐‐ config.sh + |‐‐ include + `‐‐ linux + `‐‐ 2.6.x + |‐‐ knl-base.cfg + |‐‐ bsp.patch + `‐‐ my_bsp.smudge + + > mv templates/board/my_board/linux/2.6.x/* templates/board/my_board/linux + > rm -rf templates/board/my_board/linux/2.6.x/ + > mv templates/board/my_board/linux/my_bsp.smudge \ + templates/board/my_board/linux/my_bsp-standard.scc + > echo "kconf hardware knl-base.cfg" >> \ + templates/board/my_board/linux/my_bsp-standard.scc + > vi templates/board/my_board/linux/my_bsp-standard.scc + # add the following at the top of the file + scc_leaf ktypes/standard my_bsp-standard + + templates/ + `‐‐ board + `‐‐ my_board + |‐‐ config.sh + |‐‐ include + `‐‐ linux + |‐‐ knl-base.cfg + |‐‐ bsp.patch + `‐‐ my_bsp-standard.scc + + +That's it. Configure and build. + +There is a naming convention for the .scc file, which allows the build + system to locate suitable feature descriptions for a board: + + + <bsp name>-<kernel type>.scc + + + if this naming convention isn't followed your feature description will + not be located and a build error thrown. + +
+ +
+ BSP: Creating a New BSP + +Although it is obvious that the structure of a new BSP uses the migrated +directory structure from the previous example,the first question is whether +or not the BSP is started from scratch. + + +If Yocto Project has a similar BSP, it is often easier to clone and update, +rather than start from scratch. If the mainline kernel has support, it is +easier to branch f