path: root/docs/usermanual/chapters/usage.xml

<?xml version="1.0" encoding="UTF-8"?>
<chapter id="chapter_using_bitbake_and_oe">
  <title>Using bitbake and OpenEmbedded</title>

  <section id="usage_introduction" xreflabel="introduction">
    <title>Introduction</title>

    <para>If you're reading this manual you probably already have some idea of
    what OpenEmbedded is all about, which is taking a lot of software and
    creating something that you can run on another device. This involves
    downloading some source code, compiling it, creating packages (like .deb
    or .rpm) and/or creating boot images that can be written to flash on the
    device. The difficulties of cross-compiling and the variety of devices
    which can be supported lead to a lot more complexity in an OpenEmbedded
    based distribution than you'd find in a typical desktop distribution
    (where which cross-compiling isn't needed).</para>

    <para>A major part of OpenEmbedded deals with compiling source code for
    various projects. For each project this generally requires the same basic
    set of tasks:</para>

    <orderedlist>
      <listitem>
        <para>Download the source code, and any supporting files (such as
        initscripts);</para>
      </listitem>

      <listitem>
        <para>Extract the source code and apply any patches that might be
        wanted;</para>
      </listitem>

      <listitem>
        <para>Configure the software if needed (such as is done by running the
        configure script);</para>
      </listitem>

      <listitem>
        <para>Compile everything;</para>
      </listitem>

      <listitem>
        <para>Package up all the files into some package format, like .deb or
        .rpm or .ipk, ready for installation.</para>
      </listitem>
    </orderedlist>

    <para>There's nothing particular unusual about this process when building
    on the machine the package is to be installed on. What makes this
    difficult is:</para>

    <orderedlist>
      <listitem>
        <para>Cross-compiling: cross-compiling is difficult, and lots of
        software has no support for cross-compiling - all packages included in
        OE are cross-compiled;</para>
      </listitem>

      <listitem>
        <para>Target and host are different: This means you can't compile up a
        program and then run it - it's compiled to run on the target system,
        not on the system compiling it. Lots of software tries to build and
        run little helper and/or test applications and this won't work when
        cross-compiling.</para>
      </listitem>

      <listitem>
        <para>Tool chains (compiler, linker etc) are often difficult to
        compile. Cross tool chains are even more difficult. Typically you'd go
        out and download a tool chain made by someone else - but not when you're
        using OE. In OE the entire toolchain is built as part of the process.
        This may make things take longer initially and may make it more
        difficult to get started but makes it easier to apply patches and test
        out changes to the tool chain.</para>
      </listitem>
    </orderedlist>

    <para>Of course there's a lot more to OE then just compiling packages
    though. Some of the features that OE supports includes:</para>

    <itemizedlist>
      <listitem>
        <para>Support for both glibc and uclibc;</para>
      </listitem>

      <listitem>
        <para>Support for building for multiple target devices from the one
        code base;</para>
      </listitem>

      <listitem>
        <para>Automatically building anything that is required for the package
        to compile and/or run (build and run time dependencies);</para>
      </listitem>

      <listitem>
        <para>Creation of flash and disk images of any one of a number of
        types (jffs2, ext2.gz, squashfs etc) for booting directly on the
        target device;</para>
      </listitem>

      <listitem>
        <para>Support for various packaging formats;</para>
      </listitem>

      <listitem>
        <para>Automatic building all of the cross-compiling tools you'll
        need;</para>
      </listitem>

      <listitem>
        <para>Support for "native" packages that are built for the host
        computer and not for the target and used to help during the build
        process;</para>
      </listitem>
    </itemizedlist>

    <para>The rest of this chapter assumes you have mastered the Getting Start
    guides to OpenEmbedded (see the OpenEmbedded web site for details), and
    therefore have an appropriately configured setup and that you have managed
    to actually build the cross-compilers for your target. This section talks
    you through some of the background on what is happening with the aim of
    helping you understand how to debug and develop within
    OpenEmbedded.</para>

    <para>You'll also note a lot of references to variables that define specific
    directories or change the behaviour of some part of the build process. You
    should refer to <xref linkend="chapter_recipes" /> for full details on
    these variables.</para>
  </section>

  <section id="usage_configuration" xreflabel="configuration">
    <title>Configuration</title>

    <para>Configuration covers basic items such as where the various files can
    be found and where output should be placed to more specific items such as
    which hardware is being targeted and what features you want to have
    included in the final image. The main configuration areas in OE
    are:</para>

    <variablelist>
      <varlistentry>
        <term>conf/machine</term>

        <listitem>
          <para>This directory contains machine configuration information. For
          each physical device a configuration file is required in this
          directory that describes various aspects of the device, such as
          architecture of the device, hardware features of the device (does it
          have usb? a keyboard? etc), the type of flash or disk images needed
          for the device, the serial console settings (if any) etc. If you are
          adding support for a new device you would need to create a machine
          configuration in this directory for the device.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>conf/distro</term>

        <listitem>
          <para>This directory contains distribution related files. A
          distribution decides how various activities are handled in the final
          image, such as how networking is configured, if usb devices will be
          supported, what packaging system is used, which libc is used
          etc.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>conf/bitbake.conf</term>

        <listitem>
          <para>This is the main bitbake configuration file. This file is not
          to be edited but it is useful to look at it since it declares a
          larger number of the predefined variables used by OE and controls a
          lot of the base functionality provided by OE.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>conf/local.conf</term>

        <listitem>
          <para>This is the end-user specific configuration. This file needs
          to be copied and edited and is used to specify the various working
          directories, the machine to build for and the distribution to
          use.</para>
        </listitem>
      </varlistentry>
    </variablelist>
  </section>

  <section id="usage_workspace" xreflabel="workspace">
    <title>Work space</title>

    <para>Let's start out by taking a look at a typical working area. Note
    that this may not be exactly what you see - there are a lot of options that
    can effect exactly how things are done, but it gives us a pretty good idea
    of whats going on. What we are looking at here is the tmp directory (as
    specified by TMPDIR in your local.conf):<screen>$ <command>find</command> tmp -maxdepth 2 -type d 
tmp
tmp/stamps
tmp/cross
tmp/cross/bin
tmp/cross/libexec
tmp/cross/lib  
tmp/cross/share
tmp/cross/sh4-linux
tmp/cache
tmp/cache/titan
tmp/work
tmp/work/busybox-1.2.1-r13
tmp/work/libice-1_1.0.3-r0 
tmp/work/arpwatch-2.1a15-r2
...
tmp/rootfs
tmp/rootfs/bin
tmp/rootfs/usr
tmp/rootfs/media
tmp/rootfs/dev  
tmp/rootfs/var  
tmp/rootfs/lib  
tmp/rootfs/sbin 
tmp/rootfs/mnt  
tmp/rootfs/boot 
tmp/rootfs/sys  
tmp/rootfs/proc 
tmp/rootfs/etc  
tmp/rootfs/home 
tmp/rootfs/tmp  
tmp/staging     
tmp/staging/man 
tmp/staging/x86_64-linux
tmp/staging/pkgdata
tmp/staging/pkgmaps
tmp/staging/var
tmp/staging/sh4-linux
tmp/staging/local
tmp/staging/etc  
tmp/deploy
tmp/deploy/addons
tmp/deploy/ipk   
tmp/deploy/sources
tmp/deploy/images</screen></para>

    <para>The various top level directories under tmp include:</para>

    <variablelist>
      <varlistentry>
        <term>stamps</term>

        <listitem>
          <para>Nothing of interest to users in here. These time stamps are
          used by bitbake to keep track of what tasks it has completed and
          what tasks it still has outstanding. This is how it knows that
          certain actions have been completed and it doesn't need to do them
          again.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>cross</term>

        <listitem>
          <para>Contains the cross-compiler toolchain. That is the gcc and
          binutils that run on the host system but produce output for the
          target system.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>cache</term>

        <listitem>
          <para>Nothing of interest to users in here. This contains the
          bitbake parse cache and is used to avoid the need to parse all of
          the recipes each time bitbake is run. This makes bitbake a lot
          faster on the 2nd and subsequent runs.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>work</term>

        <listitem>
          <para>The work directory. This is the directory in which all
          packages are built - this is where the source code is extract,
          patches applied, software configure, compiled, installed and
          package. This is where you'll spend most of you time looking when
          working in OE.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>rootfs</term>

        <listitem>
          <para>The generated root filesystem image for your target device.
          This is the contents of the root filesystem (NOTE: fakeroot means it
          doesn't have the correct device special nodes and permissions to use
          directly).</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>sysroots</term>

        <listitem>
          <para>Contains the staging area, which is used to store natively
          compiled tools and and libraries and headers for the target that are
          required for building other software.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>deploy</term>

        <listitem>
          <para>Contains the final output from OE. This includes the
          installation packages (typically .ipkg packages) and flash and/or
          disk images. This is where you go to get the final product.</para>
        </listitem>
      </varlistentry>
    </variablelist>

    <para>When people refer to the <emphasis>"tmp directory"</emphasis> this
    is the directory they are talking about.</para>

    <para>To perform a complete rebuild from scratch you would usually rename
    or delete tmp and then restart your build. I recommend keeping one old
    version of tmp around to use for comparison if something goes wrong with
    your new build. For example:<screen>$ <command>rm</command> -fr tmp.OLD
$ <command>mv</command> tmp tmp.OLD
$ <command>bitbake</command> bootstrap-image</screen></para>

    <section id="usage_workdir" xreflabel="work directory">
      <title>work directory (tmp/work)</title>

      <para>The work directory is where all source code is unpacked into,
      where source is configured, compiled and packaged. In other words this
      is where all the action happens. Each bitbake recipe will produce a
      corresponding subdirectory in the work directory. The subdirectory
      name will contain the recipe name, version and the release number (as
      defined by the PR variable within the recipe).</para>