Difference between revisions of "Creating ABF build bots"

From Rosalab Wiki
Jump to: navigation, search
m (some more translations)
(→‎Main components: English picture)
 
(2 intermediate revisions by the same user not shown)
Line 8: Line 8:
 
*'''Product''' — ISO image of a distribution built on the basis of some platform.
 
*'''Product''' — ISO image of a distribution built on the basis of some platform.
 
*'''Reverse dependency''' — a package that requires the current one; "reverse build dependency" stands for a dependency necessary for building a package
 
*'''Reverse dependency''' — a package that requires the current one; "reverse build dependency" stands for a dependency necessary for building a package
−
 
−
== Reverse dependencies ==
 
−
After building a package, one should normally rebuild all packages that depend on it (for example, to pick up library changes). In many cases packages built with older version can work incorrectly or fail to work at all with the new version.
 
−
ABF is able to perform such rebuild automatically. In order to do this, it iteratively calculates ''reverse build dependency tree'' of the package being built. That is, it takes package "A", detects that it is required to build packages "B" and "C", then calculates reverse dependencies of "B" and "C" and so on, until a closure set is obtained.<br/>
 
−
'''The tree is built using width-first search algorithm'''
 
−
#gcc
 
−
#tar,kernel
 
−
#mc,firefox,mesa
 
−
 
−
Build server builds packages level by level in multiple threads - that is, a package from some level will be built only when all packages from higher levels are already rebuilt successfully.<br/>
 
−
If loops are detected when calculating dependency tree (i.e., "A" requires "B" and "B" requires "A"), then the rebuild process will be performed twice: first, all packages on all tree levels will be build, and then they will be rebuilt using results from the first stage:
 
−
[[File:Bbdt.jpg|center|400px]]
 
  
 
=== Main components ===
 
=== Main components ===
Line 27: Line 15:
 
* ISO generation module
 
* ISO generation module
  
−
[[File:SqUiMvkpq_lbw8D4twm3cRQ.png|center|400px]]
+
[[File:ABF_structure_en.png|center|400px]]
  
 
== Build core ==
 
== Build core ==
Line 65: Line 53:
 
Client is used to build packages.  
 
Client is used to build packages.  
 
Build clients can differ in the following ways:
 
Build clients can differ in the following ways:
−
*по поддерживаемым сборочным архитектурам. Под какие архитектуры сборочный клиент может собрать проект
+
* supported hardware architectures
−
*по типам дистрибутива, под которые данный сборочный клиент может собрать проект
+
* supported target distributions
 +
 
 +
'''Workflow:'''
 +
: First, build client receives a task from the core. Then packages from the task are built one by one.
 +
'''Project build algorithm:'''
 +
* Inform core that the project build has been started
 +
* Take project sources from Git repository and increment release in the spec file (this increment is performed automatically for rpm-based distributions)
 +
* Commit spec file changes back to the repo and create src.rpm
 +
* Create list of repositories that should be enabled (this depends on the build type - is this a build for a personal repository or for the main one)
 +
* Build package and publish container with updated metadata
 +
* Send build status to the build core
 +
 
 +
If dependency loops are detected, the build process is performed once more using the results from the previous run.
  
 +
When all projects are built, build client performs automated testing of resulting packages. During this testing, the client tries to install all packages into the chroot environment; this test is used to check that packages have all necessary requirements listed in their Requires tags.
  
−
'''Принцип работы:'''
+
If package tests are passed successfully, packages are sent to the build core.
−
: Сначала сборочный клиент забирает сборочное задание с ядра. Далее по очередности выполняется сборка каждого проекта в задании.
+
# To speed up the process, chroot is created in tmpfs
−
'''Порядок сборки проекта:'''
+
# Repositories are connected using HTTP
−
*Отправить статус что проект начал собираться на ядро
+
# Build results can be investigated using WebUI
−
*Забрать из гит репозитория исходники проекта и поднять релиз в spec файле (осуществляется универсальным скриптом для rpm дистрибутивов)
+
−
*передать изменения обратно в гит и создать src.rpm
+
−
*Создать список необходимых репозиториев в зависимости от типа сборки (сборка в персональный репозиторий сборка в основной репозиторий и тд)
+
−
*Собрать проект и выложить в контейнер обновив мета-дату
+
−
*Отправить статус сборки
+
  
−
Если присутствуют циклические зависимости, сборка происходит повторно с подключением предыдущих результатов<br/>
+
== ISO generation module ==
−
После сборки всех проектов происходит автоматическое тестирование результатов сборки. Оно представляет собой попытку установки всех пакетов в chroot. Проверка нужна для контроля зависимостей у собранных пакетов<br/>
+
This module is used to build distribution ISO images. It is implemented as XML RPC server which receives tasks from the build core<br/>  
−
После сборки всех проектов и автоматического тестирования результаты отправляются на сборочное ядро
+
WebUI allows to configure scripts and files necessary for ISO creation and add tarballs with additional files required for this process.
−
#Построение chroot происходит в tmpfs для ускорения работы
+
−
#Репозитории подключаются по http
+
−
#Результаты сборки можно увидеть через веб интерфейс
+
  
−
== Модуль сборки образов ==
+
'''ISO creation workflow:'''
−
Служит для сборки образов ОС. Представляет собой xml rpc сервер, который принимает с веб интерфейса задания на сборку дистрибутива.<br/>
+
# Web part sends a 'build ISO' request
−
В веб интерфейсе отображаются необходимые для сборки скрипты и файлы. Кроме того, есть возможность загружать  архивы с дополнительными файлами требуемыми для сборки образа.<br/>
+
# The request is put into the module queue
−
'''Процесс сборки:'''
+
# WebUI receives notification when the generation starts
−
#С веб интерфейса приходит задание на сборку
+
# Tarballs with additional files specified in WebUI are downloaded, if any
−
#Задание попадает в очередь
+
# ISO generation script is launched
−
#Перед началом выполнения задания на веб интерфейс отправляется статус о начале сборки
+
# When generation is finished, a corresponding notification is sent to WebUI
−
#Скачивается архив с дополнительными файлами с веб интерфейса
+
−
#Запускается скрипт сборки
+
−
#После сборки отправляется результат на веб интерфейс
+
  
 
== Adding support for more distributions ==
 
== Adding support for more distributions ==

Latest revision as of 09:57, 17 April 2012

Build Server

Terms and Definitions

  • Project — software source code, documentation and other artifacts accompanied by meta-information necessary to build and distribute the software product as a package.
  • Group — users working on the same project. ABF allows to manage access privileges on both per-user and per-group levels.
  • Platform — a group of repositories to be handled as a single entity. Such groupings are useful for creating different products from the same set of packages.
  • Repository — a set of packages grouped together according to certain criteria: main packages, auxiliary packages, non-free software, etc.
  • Product — ISO image of a distribution built on the basis of some platform.
  • Reverse dependency — a package that requires the current one; "reverse build dependency" stands for a dependency necessary for building a package

Main components

  • Web UI
  • Build core
  • Build client (bot)
  • ISO generation module
ABF structure en.png

Build core

Responsible for:

  • management of build clients (bots)
  • creation/removal of platforms
  • creation/removal of repositories
  • creation/removal of build tasks
  • publishing containers into repository
  • creation/removal of projects
  • calculation of reverse dependencies for the project
  • calculation of direct dependencies for the project

The build core receives commands from Web UI using XML-RPC
Build server processes these requests and sends result back to WebUI

There are 2 kinds of platforms in ABF:

  1. Main platforms
  2. User platforms

Main platforms are supposed to build and maintain the whole distributions
User platforms are intended for building and maintaining user projects that can be built for different distributions

Project build workflow

  1. WebUI sends a build request
  2. Build core checks if the requested project can be built and puts appropriate task in the queue
  3. The task is processed, reverse dependency tree is built, if necessary
  4. Build task is formed for the build client; this task is put in the clients' queue
    • When connecting to the build server, every client reports for which distributions and hardware architectures it can build packages
  5. A client receives the build task and processes it
  6. When package is built, one can use WebUI to send a request on publishing the resulting container
  7. Build server checks if it possible to publish the container puts appropriate task in the queue
  8. During publishing all built packages are moved to repositories they belong to. The ABF database storing information about packages is updated.
  9. WebUI is informed that container is published.

Build client (bot)

Client is used to build packages. Build clients can differ in the following ways:

  • supported hardware architectures
  • supported target distributions

Workflow:

First, build client receives a task from the core. Then packages from the task are built one by one.

Project build algorithm:

  • Inform core that the project build has been started
  • Take project sources from Git repository and increment release in the spec file (this increment is performed automatically for rpm-based distributions)
  • Commit spec file changes back to the repo and create src.rpm
  • Create list of repositories that should be enabled (this depends on the build type - is this a build for a personal repository or for the main one)
  • Build package and publish container with updated metadata
  • Send build status to the build core

If dependency loops are detected, the build process is performed once more using the results from the previous run.

When all projects are built, build client performs automated testing of resulting packages. During this testing, the client tries to install all packages into the chroot environment; this test is used to check that packages have all necessary requirements listed in their Requires tags.

If package tests are passed successfully, packages are sent to the build core.

  1. To speed up the process, chroot is created in tmpfs
  2. Repositories are connected using HTTP
  3. Build results can be investigated using WebUI

ISO generation module

This module is used to build distribution ISO images. It is implemented as XML RPC server which receives tasks from the build core
WebUI allows to configure scripts and files necessary for ISO creation and add tarballs with additional files required for this process.

ISO creation workflow:

  1. Web part sends a 'build ISO' request
  2. The request is put into the module queue
  3. WebUI receives notification when the generation starts
  4. Tarballs with additional files specified in WebUI are downloaded, if any
  5. ISO generation script is launched
  6. When generation is finished, a corresponding notification is sent to WebUI

Adding support for more distributions

If you want to build your own distribution using ABF you should develop corresponding build client and ISO generation module. In order to do this, you can use existing templates. If you want to add support for your distribution in the main ABF instance, please send a request to info@rosalab.ru. You will be provided with necessary help, templates, documentation, etc. When your build client and ISO generator are ready you should send them to ABF developers for review. If everything is ok, ABF team will help to integrate the new stuff into the main ABF instance.


ABF Documentation
Retrieved from "http://wiki.rosalab.ru/en/index.php?title=Creating_ABF_build_bots&oldid=226"