path: root/documentation/poky-ref-manual/ref-bitbake.xml

<!DOCTYPE appendix PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">

<appendix id='ref-bitbake'>

    <title>Reference: BitBake</title>

    <para>
        BitBake is a program written in Python that interprets the metadata
        that makes up Poky. At some point, people wonder what actually happens 
        when you type <command>bitbake poky-image-sato</command>. This section 
        aims to give an overview of what happens behind the scenes from a
        BitBake perspective.
    </para>

    <para>
        It is worth noting that BitBake aims to be a generic "task" executor 
        capable of handling complex dependency relationships. As such it has no 
        real knowledge of what the tasks it is executing actually do. It just 
        considers a list of tasks with dependencies and handles metadata 
        consisting of variables in a certain format which get passed to the 
        tasks.
    </para>

    <section id='ref-bitbake-parsing'>
        <title>Parsing</title>

        <para>
            The first thing BitBake does is work out its configuration by 
            looking for a file called <filename>bitbake.conf</filename>.
            BitBake searches through the <varname>BBPATH</varname> environment 
            variable looking for a <filename class="directory">conf/</filename> 
            directory containing a <filename>bitbake.conf</filename> file and 
            adds the first <filename>bitbake.conf</filename> file found in
            <varname>BBPATH</varname> (similar to the PATH environment variable). 
            For Poky, <filename>bitbake.conf</filename> is found in <filename 
            class="directory">meta/conf/</filename>. 
        </para>

        <para>
            In Poky, <filename>bitbake.conf</filename> lists other configuration 
            files to include from a <filename class="directory">conf/</filename> 
            directory below the directories listed in <varname>BBPATH</varname>. 
            In general the most important configuration file from a user's perspective 
            is <filename>local.conf</filename>, which contains a users customized 
            settings for Poky. Other notable configuration files are the distribution 
            configuration file (set by the <glossterm><link linkend='var-DISTRO'>
            DISTRO</link></glossterm> variable) and the machine configuration file 
            (set by the <glossterm><link linkend='var-MACHINE'>MACHINE</link>
            </glossterm> variable).  The <glossterm><link linkend='var-DISTRO'>
            DISTRO</link></glossterm> and <glossterm><link linkend='var-MACHINE'>
            MACHINE</link></glossterm> environment variables are both usually set in 
            the <filename>local.conf</filename> file. Valid distribution 
            configuration files are available in the <filename class="directory">
            meta/conf/distro/</filename> directory and valid machine configuration 
            files in the <filename class="directory">meta/conf/machine/</filename> 
            directory. Within the <filename class="directory">
            meta/conf/machine/include/</filename> directory are various <filename>
            tune-*.inc</filename> configuration files which provide common 
            "tuning" settings specific to and shared between particular 
            architectures and machines.
        </para>

        <para>
            After the parsing of the configuration files some standard classes 
            are included. In particular, <filename>base.bbclass</filename> is 
            always included, as will any other classes 
            specified in the configuration using the <glossterm><link
                    linkend='var-INHERIT'>INHERIT</link></glossterm>
            variable. Class files are searched for in a classes subdirectory 
            under the paths in <varname>BBPATH</varname> in the same way as
            configuration files.
        </para>

        <para>
            After the parsing of the configuration files is complete, the 
            variable <glossterm><link linkend='var-BBFILES'>BBFILES</link></glossterm> 
            is set, usually in
            <filename>local.conf</filename>, and defines the list of places to search for 
            <filename class="extension">.bb</filename> files. By 
            default this specifies the <filename class="directory">meta/packages/
            </filename> directory within Poky, but other directories such as
            <filename class="directory">meta-extras/</filename> can be included 
            too. Adding extra content to
            <glossterm><link linkend='var-BBFILES'>BBFILES</link></glossterm> is best
            acheived through the use of BitBake
            <link linkend='usingpoky-changes-layers'>"layers"</link>.
        </para>

        <para>
            BitBake parses each <filename class="extension">.bb</filename> file in 
            <glossterm><link linkend='var-BBFILES'>BBFILES</link></glossterm> and 
            stores the values of various variables.  In summary, for each 
            <filename class="extension">.bb</filename> 
            file the configuration + base class of variables are set, followed 
            by the data in the <filename class="extension">.bb</filename> file 
            itself, followed by any inherit commands that
            <filename class="extension">.bb</filename> file might contain.
        </para>

        <para>
            Parsing <filename class="extension">.bb</filename> files is a time 
            consuming process, so a cache is kept to speed up subsequent parsing. 
            This cache is invalid if the timestamp of the <filename class="extension">.bb</filename> 
            file itself has changed, or if the timestamps of any of the include, 
            configuration or class files the <filename class="extension">.bb</filename>
            file depends on have changed.
        </para>
    </section>

    <section id='ref-bitbake-providers'>
        <title>Preferences and Providers</title>

        <para>
            Once all the <filename class="extension">.bb</filename> files have been 
            parsed, BitBake will proceed to build "poky-image-sato" (or whatever was
            specified on the commandline) and looks for providers of that target.
            Once a provider is selected, BitBake resolves all the dependencies for  
            the target. In the case of "poky-image-sato", it would lead to 
            <filename>task-base.bb</filename>  
            which in turn would lead to packages like <application>Contacts</application>, 
            <application>Dates</application>, <application>BusyBox</application>
            and these in turn depend on glibc and the toolchain.
        </para>

        <para>
            Sometimes a target might have multiple providers and a common example 
            is "virtual/kernel" that is provided by each kernel package. Each machine
            will often elect the best provider of its kernel with a line like the 
            following in the machine configuration file:
        </para>
        <programlisting><glossterm><link linkend='var-PREFERRED_PROVIDER'>PREFERRED_PROVIDER</link></glossterm>_virtual/kernel = "linux-rp"</programlisting>
        <para>
            The default <glossterm><link linkend='var-PREFERRED_PROVIDER'>
            PREFERRED_PROVIDER</link></glossterm> is the provider with the same name as
            the target.
        </para>

        <para>
            Understanding how providers are chosen is complicated by the fact
            multiple versions might be present. BitBake defaults to the highest 
            version of a provider by default. Version comparisons are made using 
            the same method as Debian. The <glossterm><link 
            linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></glossterm>
            variable can be used to specify a particular version 
            (usually in the distro configuration) but the order can 
            also be influenced by the <glossterm><link
                    linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></glossterm> 
            variable. By default files 
            have a preference of "0". Setting the
            <glossterm><link
                    linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></glossterm> to "-1" will 
            make a package unlikely to be used unless it was explicitly referenced and
            "1" makes it likely the package will be used. 
            <glossterm><link
                    linkend='var-PREFERRED_VERSION'>PREFERRED_VERSION</link></glossterm> overrides 
            any <glossterm><link
                    linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></glossterm>. <glossterm><link
                    linkend='var-DEFAULT_PREFERENCE'>DEFAULT_PREFERENCE</link></glossterm> 
            is often used to mark more 
            experimental new versions of packages until they've undergone sufficient 
            testing to be considered stable.
        </para>

        <para>
            The end result is that internally, BitBake has now built a list of 
            providers for each target it needs in order of priority.
        </para>
    </section>

    <section id='ref-bitbake-dependencies'>
        <title>Dependencies</title>

        <para>
            Each target BitBake builds consists of multiple tasks (e.g. fetch, 
            unpack, patch, configure, compile etc.). For best performance on 
            multi-core systems, BitBake considers each task as an independent 
            entity with a set of dependencies. There are many variables that 
            are used to signify these dependencies and more information can be found 
            about these in the <ulink url='http://bitbake.berlios.de/manual/'>
            BitBake manual</ulink>. At a basic level it is sufficient to know 
        that BitBake uses the <glossterm><link
                linkend='var-DEPENDS'>DEPENDS</link></glossterm> and 
        <glossterm><link linkend='var-RDEPENDS'>RDEPENDS</link></glossterm> variables when 
            calculating dependencies and descriptions of these variables are 
            available through the links. 
        </para>

    </section>

    <section id='ref-bitbake-tasklist'>
        <title>The Task List</title>

        <para>
            Based on the generated list of providers and the dependency information, 
            BitBake can now calculate exactly which tasks it needs to run and in what 
            order. The build now starts with BitBake forking off threads up to
            the limit set in the <glossterm><link
                    linkend='var-BB_NUMBER_THREADS'>BB_NUMBER_THREADS</link></glossterm> variable
            as long as there are tasks ready to run, i.e. tasks with all their
            dependencies met.
        </para>

        <para>
            As each task completes, a timestamp is written to the directory 
            specified by the <glossterm><link
                    linkend='var-STAMPS'>STAMPS</link></glossterm> variable (usually 
            <filename class="directory">build/tmp/stamps/*/</filename>). On 
            subsequent runs, BitBake looks at the <glossterm><link
                    linkend='var-STAMPS'>STAMPS</link></glossterm>
            directory and will not rerun 
            tasks its already completed unless a timestamp is found to be invalid. 
            Currently, invalid timestamps are only considered on a per <filename 
            class="extension">.bb</filename> file basis so if for example the configure stamp has a timestamp greater than the 
            compile timestamp for a given target the compile task would rerun but this 
            has no effect on other providers depending on that target. This could 
            change or become configurable in future versions of BitBake. Some tasks 
            are marked as "nostamp" tasks which means no timestamp file will be written 
            and the task will always rerun.
        </para>

        <para>Once all the tasks have been completed BitBake exits.</para> 

    </section>

    <section id='ref-bitbake-runtask'>
        <title>Running a Task</title>

        <para>
            It's worth noting what BitBake does to run a task. A task can either
            be a shell task or a python task. For shell tasks, BitBake writes a
            shell script to <filename>${WORKDIR}/temp/run.do_taskname.pid</filename>
            and then executes the script. The generated
            shell script contains all the exported variables, and the shell functions 
            with all variables expanded. Output from the shell script is 
            sent to the file <filename>${WORKDIR}/temp/log.do_taskname.pid</filename>.
            Looking at the
            expanded shell functions in the run file and the output in the log files 
            is a useful debugging technique.
        </para>

        <para>
            Python functions are executed internally to BitBake itself and 
            logging goes to the controlling terminal. Future versions of BitBake will 
            write the functions to files in a similar way to shell functions and 
            logging will also go to the log files in a similar way.
        </para>
    </section>


    <section id='ref-bitbake-commandline'>
        <title>Commandline</title>

        <para>
            To quote from "bitbake --help":
        </para>

        <screen>Usage: bitbake [options] [package ...]

Executes the specified task (default is 'build') for a given set of BitBake files.
It expects that BBFILES is defined, which is a space separated list of files to
be executed.  BBFILES does support wildcards.
Default BBFILES are the .bb files in the current directory.

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -b BUILDFILE, --buildfile=BUILDFILE
                        execute the task against this .bb file, rather than a
                        package from BBFILES.
  -k, --continue        continue as much as possible after an error. While the
                        target that failed, and those that depend on it,
                        cannot be remade, the other dependencies of these
                        targets can be processed all the same.
  -a, --tryaltconfigs   continue with builds by trying to use alternative
                        providers where possible.
  -f, --force           force run of specified cmd, regardless of stamp status
  -c CMD, --cmd=CMD     Specify task to execute. Note that this only executes
                        the specified task for the providee and the packages
                        it depends on, i.e. 'compile' does not implicitly call
                        stage for the dependencies (IOW: use only if you know
                        what you are doing). Depending on the base.bbclass a
                        listtasks tasks is defined and will show available
                        tasks
  -r FILE, --read=FILE  read the specified file before bitbake.conf
  -v, --verbose         output more chit-chat to the terminal
  -D, --debug           Increase the debug level. You can specify this more
                        than once.
  -n, --dry-run         don't execute, just go through the motions
  -S, --dump-signatures
                        don't execute, just dump out the signature
                        construction information
  -p, --parse-only      quit after parsing the BB files (developers only)
  -d, --disable-psyco   disable using the psyco just-in-time compiler (not
                        recommended)
  -s, --show-versions   show current and preferred versions of all packages
  -e, --environment     show the global or per-package environment (this is
                        what used to be bbread)
  -g, --graphviz        emit the dependency trees of the specified packages in
                        the dot syntax
  -I EXTRA_ASSUME_PROVIDED, --ignore-deps=EXTRA_ASSUME_PROVIDED
                        Assume these dependencies don't exist and are already
                        provided (equivalent to ASSUME_PROVIDED). Useful to
                        make dependency graphs more appealing
  -l DEBUG_DOMAINS, --log-domains=DEBUG_DOMAINS
                        Show debug logging for the specified logging domains
  -P, --profile         profile the command and print a report
  -u UI, --ui=UI        userinterface to use
  --revisions-changed   Set the exit code depending on whether upstream
                        floating revisions have changed or not</screen>

    </section>

    <section id='ref-bitbake-fetchers'>
        <title>Fetchers</title>

        <para>
            As well as the containing the parsing and task/dependency handling 
            code, BitBake also contains a set of "fetcher" modules which allow 
            fetching of source code from various types of sources. Example 
            sources might be from disk with the metadata, from websites, from 
            remote shell accounts or from SCM systems like cvs/subversion/git. 
        </para>

        <para>
            The fetchers are usually triggered by entries in 
            <glossterm><link linkend='var-SRC_URI'>SRC_URI</link></glossterm>. Information about the 
            options and formats of entries for specific fetchers can be found in the 
            <ulink url='http://bitbake.berlios.de/manual/'>BitBake manual</ulink>.
        </para>

        <para>
            One useful feature for certain SCM fetchers is the ability to 
            "auto-update" when the upstream SCM changes version. Since this 
            requires certain functionality from the SCM only certain systems 
            support it, currently Subversion, Bazaar and to a limited extent, Git. It 
            works using the <glossterm><link linkend='var-SRCREV'>SRCREV</link>
            </glossterm> variable. See the <link linkend='platdev-appdev-srcrev'>
            developing with an external SCM based project</link> section for more
            information.
        </para>

    </section>

</appendix>
<!-- 
vim: expandtab tw=80 ts=4 spell spelllang=en_gb
-->