Difference between revisions of "Creating ABF build bots"

Revision as of 11:49, 13 April 2012

Build Server

Terms and Definitions

• Project — software source code, documentation and other artifacts accompanied by meta-information necessary to build and distribute the software product as a package.
• Group — users working on the same project. ABF allows to manage access privileges on both per-user and per-group levels.
• Platform — a group of repositories to be handled as a single entity. Such groupings are useful for creating different products from the same set of packages.
• Repository — a set of packages grouped together according to certain criteria: main packages, auxiliary packages, non-free software, etc.
• Product — ISO image of a distribution built on the basis of some platform.
• Reverse dependency — a package that requires the current one; "reverse build dependency" stands for a dependency necessary for building a package

Reverse dependencies

After building a package, one should normally rebuild all packages that depend on it (for example, to pick up library changes). In many cases packages built with older version can work incorrectly or fail to work at all with the new version. ABF is able to perform such rebuild automatically. In order to do this, it iteratively calculates reverse build dependency tree of the package being built. That is, it takes package "A", detects that it is required to build packages "B" and "C", then calculates reverse dependencies of "B" and "C" and so on, until a closure set is obtained.
The tree is built using width-first search algorithm

1. gcc
2. tar,kernel
3. mc,firefox,mesa

Build server builds packages level by level in multiple threads - that is, a package from some level will be built only when all packages from higher levels are already rebuilt successfully.
If loops are detected when calculating dependency tree (i.e., "A" requires "B" and "B" requires "A"), then the rebuild process will be performed twice: first, all packages on all tree levels will be build, and then they will be rebuilt using results from the first stage:

Main components

• Web UI
• Build core
• Build client (bot)
• ISO generation module

Build core

Responsible for:

• management of build clients (bots)
• creation/removal of platforms
• creation/removal of repositories
• creation/removal of build tasks
• publishing containers into repository
• creation/removal of projects
• calculation of reverse dependencies for the project
• calculation of direct dependencies for the project

The build core receives commands from Web UI using XML-RPC
Build server processes these requests and sends result back to WebUI

There are 2 kinds of platforms in ABF:

1. Main platforms
2. User platforms

Main platforms are supposed to build and maintain the whole distributions
User platforms are intended for building and maintaining user projects that can be built for different distributions

Project build workflow

1. WebUI sends a build request
2. Build core checks if the requested project can be built and puts appropriate task in the queue
3. The task is processed, reverse dependency tree is built, if necessary
4. Build task is formed for the build client; this task is put in the clients' queue
   • When connecting to the build server, every client reports for which distributions and hardware architectures it can build packages
5. A client receives the build task and processes it
6. When package is built, one can use WebUI to send a request on publishing the resulting container
7. Build server checks if it possible to publish the container puts appropriate task in the queue
8. During publishing all built packages are moved to repositories they belong to. The ABF database storing information about packages is updated.
9. WebUI is informed that container is published.

Build client (bot)

Client is used to build packages. Build clients can differ in the following ways:

• supported hardware architectures
• supported target distributions

Workflow:

First, build client receives a task from the core. Then packages from the task are built one by one.

Project build algorithm:

• Inform core that the project build has been started
• Take project sources from Git repository and increment release in the spec file (this increment is performed automatically for rpm-based distributions)
• Commit spec file changes back to the repo and create src.rpm
• Create list of repositories that should be enabled (this depends on the build type - is this a build for a personal repository or for the main one)
• Build package and publish container with updated metadata
• Send build status to the build core

If dependency loops are detected, the build process is performed once more using the results from the previous run.

When all projects are built, build client performs automated testing of resulting packages. During this testing, the client tries to install all packages into the chroot environment; this test is used to check that packages have all necessary requirements listed in their Requires tags.

If package tests are passed successfully, packages are sent to the build core.

1. To speed up the process, chroot is created in tmpfs
2. Repositories are connected using HTTP
3. Build results can be investigated using WebUI

ISO generation module

This module is used to build distribution ISO images. It is implemented as XML RPC server which receives tasks from the build core
WebUI allows to configure scripts and files necessary for ISO creation and add tarballs with additional files required for this process.

ISO creation workflow:

1. Web part sends a 'build ISO' request
2. The request is put into the module queue
3. WebUI receives notification when the generation starts
4. Tarballs with additional files specified in WebUI are downloaded, if any
5. ISO generation script is launched
6. When generation is finished, a corresponding notification is sent to WebUI

Adding support for more distributions

If you want to build your own distribution using ABF you should develop corresponding build client and ISO generation module. In order to do this, you can use existing templates. If you want to add support for your distribution in the main ABF instance, please send a request to info@rosalab.ru. You will be provided with necessary help, templates, documentation, etc. When your build client and ISO generator are ready you should send them to ABF developers for review. If everything is ok, ABF team will help to integrate the new stuff into the main ABF instance.