path: root/documentation/poky-ref-manual/ref-bitbake.xml

<!DOCTYPE appendix PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">

<appendix id='ref-bitbake'>

    <title>Reference: BitBake</title>

    <para>
        BitBake is a program written in Python that interprets the metadata that makes up Poky. 
        At some point, people wonder what actually happens when you enter:
    <literallayout class='monospaced'>
     $ bitbake poky-image-sato
    </literallayout>
    </para>
   
    <para>
        This appendix provides an overview of what happens behind the scenes from BitBake's perspective.
    </para>

    <note><para>
        BitBake strives to be a generic "task" executor that is capable of handling complex dependency relationships. 
        As such, it has no real knowledge of what the tasks being executed actually do. 
        BitBake just considers a list of tasks with dependencies and handles metadata 
        that consists of variables in a certain format that get passed to the tasks.
    </para></note>

    <section id='ref-bitbake-parsing'>
        <title>Parsing</title>

        <para>
            BitBake parses configuration files, classes, and <filename>.bb</filename> files. 
        </para>

        <para>
            The first thing BitBake does is look for the <filename>bitbake.conf</filename> file.
            Poky keeps this file in <filename class="directory">meta/conf/</filename>.
            BitBake finds it by examining the <varname>BBPATH</varname> environment 
            variable and looking for the <filename class="directory">meta/conf/</filename> 
            directory.
        </para>

        <para>
            In Poky, <filename>bitbake.conf</filename> lists other configuration 
            files to include from a <filename class="directory">conf/</filename> 
            directory below the directories listed in <varname>BBPATH</varname>. 
            In general the most important configuration file from a user's perspective 
            is <filename>local.conf</filename>, which contains a user's customized 
            settings for Poky. 
            Other notable configuration files are the distribution 
            configuration file (set by the <glossterm><link linkend='var-DISTRO'>
            DISTRO</link></glossterm> variable) and the machine configuration file 
            (set by the <glossterm><link linkend='var-MACHINE'>MACHINE</link>
            </glossterm> variable).  
            The DISTRO and MACHINE environment variables are both usually set in 
            the <filename>local.conf</filename> file. 
            Valid distribution 
            configuration files are available in the <filename class="directory">
            meta/conf/distro/</filename> directory and valid machine configuration 
            files in the <filename class="directory">meta/conf/machine/</filename> 
            directory. 
            Within the <filename class="directory">meta/conf/machine/include/</filename> 
            directory are various <filename>tune-*.inc</filename> configuration files that provide common 
            "tuning" settings specific to and shared between particular architectures and machines.
        </para>

        <para>
            After the parsing of the configuration files some standard classes are included. 
            The <filename>base.bbclass</filename> file is always included.
            Other classes that are specified in the configuration using the 
            <glossterm><link linkend='var-INHERIT'>INHERIT</link></glossterm>
            variable are also inculded. 
            Class files are searched for in a classes subdirectory 
            under the paths in <varname>BBPATH</varname> in the same way as
            configuration files.
        </para>

        <para>
            After classes are included, the 
            variable <glossterm><link linkend='var-BBFILES'>BBFILES</link></glossterm> 
            is set, usually in
            <filename>local.conf</filename>, and defines the list of places to search for 
            <filename class="extension">.bb</filename> files. 
            By default, the BBFILES variable specifies the <filename class="directory">meta/packages/
            </filename> directory within Poky, but other directories such as
            <filename class="directory">meta-extras/</filename> can be included 
            too. 
            Adding extra content to BBFILES is best achieved through the use of BitBake
            <link linkend='usingpoky-changes-layers'>"layers"</link>.
        </para>

        <para>
            BitBake parses each <filename class="extension">.bb</filename> file in BBFILES and 
            stores the values of various variables.  
            In summary, for each <filename class="extension">.bb</filename> 
            file the configuration plus the base class of variables are set, followed 
            by the data in the <filename class="extension">.bb</filename> file 
            itself, followed by any inherit commands that
            <filename class="extension">.bb</filename> file might contain.
        </para>

        <para>
            Because parsing <filename class="extension">.bb</filename> files is a time 
            consuming process, a cache is kept to speed up subsequent parsing. 
            This cache is invalid if the timestamp of the <filename class="extension">.bb</filename> 
            file itself changes, or if the timestamps of any of the include, 
            configuration or class files the <filename class="extension">.bb</filename>
            file depends on changes.
        </para>
    </section>

    <section id='ref-bitbake-providers'>
        <title>Preferences and Providers</title>

        <para>
            Once all the <filename class="extension">.bb</filename> files have been 
            parsed, BitBake starts to build the target (poky-image-sato in the previous section's 
            example) and looks for providers of that target.
            Once a provider is selected, BitBake resolves all the dependencies for  
            the target. 
            In the case of "poky-image-sato", it would lead to <filename>task-base.bb</filename>,  
            which in turn leads to packages like <application>Contacts</application>, 
            <application>Dates</application> and <application>BusyBox</application>.
            These packages in turn depend on glibc and the toolchain.
        </para>

        <para>
            Sometimes a target might have multiple providers.
            An common example is "virtual/kernel", which is provided by each kernel package. 
            Each machine often elects the best kernel provider by using a line similar to the 
            following in the machine configuration file:
        </para>

        <programlisting>
PREFERRED_PROVIDER_virtual/kernel = "linux-rp"
        </programlisting>

        <para>
            The default <glossterm><link linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</link></glossterm> 
            is the provider with the same name as the target.
        </para>

        <para>
            Understanding how providers are chosen is made complicated by the fact
            that multiple versions might exist. 
            BitBake defaults to the highest version of a provider.
            Version comparisons are made using the same method as Debian. 
            You can use the <glossterm><link linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></glossterm>
            variable to specify a particular version (usually in the distro configuration).
            You can influence the order by using the 
            <glossterm><link linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></glossterm> 
            variable. 
            By default, files have a preference of "0". 
            Setting the DEFAULT_PREFERENCE to "-1" makes the package unlikely to be used unless it is
            explicitly referenced.
            Setting the DEFAULT_PREFERENCE to "1" makes it likely the package is used. 
            PREFERRED_VERSION overrides any DEFAULT_PREFERENCE setting.
            DEFAULT_PREFERENCE is often used to mark newer and more experimental package
            versions until they have undergone sufficient testing to be considered stable.
        </para>

        <para>
            In summary, BitBake has created a list of providers, which is prioritized, for each target.
        </para>
    </section>

    <section id='ref-bitbake-dependencies'>
        <title>Dependencies</title>

        <para>
            Each target BitBake builds consists of multiple tasks such as fetch, unpack, patch, configure, 
            and compile. 
            For best performance on multi-core systems, BitBake considers each task as an independent 
            entity with its own set of dependencies. 
        </para>
  
        <para>
            Dependencies are defined through several variables.
            You can find information about variables BitBake uses in the 
            <ulink url='http://bitbake.berlios.de/manual/'>BitBake manual</ulink>. 
            At a basic level it is sufficient to know that BitBake uses the 
            <glossterm><link linkend='var-DEPENDS'>DEPENDS</link></glossterm> and 
            <glossterm><link linkend='var-RDEPENDS'>RDEPENDS</link></glossterm> variables when 
            calculating dependencies. 
        </para>

    </section>

    <section id='ref-bitbake-tasklist'>
        <title>The Task List</title>

        <para>
            Based on the generated list of providers and the dependency information, 
            BitBake can now calculate exactly what tasks it needs to run and in what 
            order it needs to run them. 
            The build now starts with BitBake forking off threads up to the limit set in the 
            <glossterm><link linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS</link></glossterm> variable.
            BitBake continues to fork threads as long as there are tasks ready to run,
            those taksks have all their dependencies met, and the thread threshold has not been 
            exceeded.
        </para>

        <para>
            As each task completes, a timestamp is written to the directory specified by the 
            <glossterm><link linkend='var-STAMPS'>STAMPS</link></glossterm> variable (usually 
            <filename class="directory">build/tmp/stamps/*/</filename>). 
            On subsequent runs, BitBake looks at the STAMPS directory and does not rerun 
            tasks that are already completed unless a timestamp is found to be invalid. 
            Currently, invalid timestamps are only considered on a per 
            <filename class="extension">.bb</filename> file basis.
            So, for example, if the configure stamp has a timestamp greater than the 
            compile timestamp for a given target then the compile task would rerun.
            Running the compile task again, however, has no effect on other providers 
            that depend on that target. 
            This behavior could change or become configurable in future versions of BitBake. 
        </para>
 
        <note><para>
            Some tasks are marked as "nostamp" tasks.
            No timestamp file is created when these tasks are run.
            Consequently, "nostamp" tasks are always rerun.
        </para></note>
    </section>

    <section id='ref-bitbake-runtask'>
        <title>Running a Task</title>

        <para>
            Tasks can either be a shell task or a python task.
            For shell tasks, BitBake writes a shell script to 
            <filename>${WORKDIR}/temp/run.do_taskname.pid</filename> and then executes the script. 
            The generated shell script contains all the exported variables, and the shell functions 
            with all variables expanded. 
            Output from the shell script goes to the file <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
            Looking at the expanded shell functions in the run file and the output in the log files 
            is a useful debugging technique.
        </para>

        <para>
            For Python tasks, BitBake executes the task internally and logs information to the 
            controlling terminal. 
            Future versions of BitBake will write the functions to files similar to the way 
            shell tasks are handled.
            Logging will be handled in way similar to shell tasks as well.
        </para>

        <para>
            Once all the tasks have been completed BitBake exits.
        </para> 
    </section>


    <section id='ref-bitbake-commandline'>
        <title>BitBake Command Line</title>

        <para>
            Following is the bitbake manpage:
        </para>

        <screen>
$ bitbake --help
Usage: bitbake [options] [package ...]

Executes the specified task (default is 'build') for a given set of BitBake files.
It expects that BBFILES is defined, which is a space separated list of files to
be executed.  BBFILES does support wildcards.
Default BBFILES are the .bb files in the current directory.

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -b BUILDFILE, --buildfile=BUILDFILE
                        execute the task against this .bb file, rather than a
                        package from BBFILES.
  -k, --continue        continue as much as possible after an error. While the
                        target that failed, and those that depend on it,
                        cannot be remade, the other dependencies of these
                        targets can be processed all the same.
  -a, --tryaltconfigs   continue with builds by trying to use alternative
                        providers where possible.
  -f, --force           force run of specified cmd, regardless of stamp status
  -i, --interactive     drop into the interactive mode also called the BitBake
                        shell.
  -c CMD, --cmd=CMD     Specify task to execute. Note that this only executes
                        the specified task for the providee and the packages
                        it depends on, i.e. 'compile' does not implicitly call
                        stage for the dependencies (IOW: use only if you know
                        what you are doing). Depending on the base.bbclass a
                        listtasks tasks is defined and will show available
                        tasks
  -r FILE, --read=FILE  read the specified file before bitbake.conf
  -v, --verbose         output more chit-chat to the terminal
  -D, --debug           Increase the debug level. You can specify this more
                        than once.
  -n, --dry-run         don't execute, just go through the motions
  -p, --parse-only      quit after parsing the BB files (developers only)
  -d, --disable-psyco   disable using the psyco just-in-time compiler (not
                        recommended)
  -s, --show-versions   show current and preferred versions of all packages
  -e, --environment     show the global or per-package environment (this is
                        what used to be bbread)
  -g, --graphviz        emit the dependency trees of the specified packages in
                        the dot syntax
  -I IGNORED_DOT_DEPS, --ignore-deps=IGNORED_DOT_DEPS
                        Stop processing at the given list of dependencies when
                        generating dependency graphs. This can help to make
                        the graph more appealing
  -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
                        Show debug logging for the specified logging domains
  -P, --profile         profile the command and print a report
        </screen>
    </section>

    <section id='ref-bitbake-fetchers'>
        <title>Fetchers</title>

        <para>
            BitBake also contains a set of "fetcher" modules that allow 
            retrieval of source code from various types of sources. 
            For example, BitBake can get source code from a disk with the metadata, from websites, 
            from remote shell accounts or from Source Code Management (SCM) systems 
            like <filename>cvs/subversion/git</filename>. 
        </para>

        <para>
            Fetchers are usually triggered by entries in 
            <glossterm><link linkend='var-SRC_URI'>SRC_URI</link></glossterm>. 
            You can find information about the options and formats of entries for specific 
            fetchers in the <ulink url='http://bitbake.berlios.de/manual/'>BitBake manual</ulink>.
        </para>

        <para>
            One useful feature for certain SCM fetchers is the ability to 
            "auto-update" when the upstream SCM changes version. 
            Since this ability requires certain functionality from the SCM, not all
            systems support it.
            Currently Subversion, Bazaar and to a limited extent, Git support the ability to "auto-update". 
            This feature works using the <glossterm><link linkend='var-SRCREV'>SRCREV</link></glossterm> 
            variable. 
            See the 
            <link linkend='platdev-appdev-srcrev'>Developing within Poky with an External SCM-based Package</link> 
            section for more information.
        </para>

    </section>

</appendix>
<!-- 
vim: expandtab tw=80 ts=4 spell spelllang=en_gb
-->