ABF: Working with project code and building projects

What links here (1) →

Working with Git repositories

Source code of all projects in ABF is stored in Git repositories; a separate repository is created for every project. Git can be accessed wither by HTTPS or by SSH protocols. URL to repository of a project named <project_name>, located in the group named <group_name> looks like <ABF_server_URL>/<group_name>/<project_name>.git. For example, in order to clone Git repository of the 0ad project from the import group using abf.io ABF server by means of HTTPS, you can run the following command on your local machine:

git clone https://abf.rosalinux.ru/import/0ad.git

In order to work with Git repositories on your local machine, you can use any command-line of GUI Git clients or ABF console client. You can also edit files in Git repositories directly in ABF web interface.

In order to explore project Git repository using ABF web interface, you should simply go to the project page. By default, the "Code" section will be displayed with Git repository content corresponding to default project branch. In the upper part of the window you will be able to see the following elements:

• a button with zip label; click on this button to download source code archive (corresponding to the chosen branch or tag) as zip or tar.gz format
• URLs to be used to clone project Git using HTTPS or SSH
• near the URL you can find a button with "question" label; click on it to get detailed help concerning project cloning
• "Current branch/tag" selector whic hcan be used to switch between Git branches and tags.

Project source code

Below that menu, you can see the following tabs:

• Files - project files in the chosen Git branch or tag. If a file is a text file, you can click on any file and edit it directly in the web interface. In this section you can also see project description, last commit in the current branch and buttons that can be used to start project build, fork it or create a pull request to another project.
• Commits - list of all changes in the current branch of project Git repository
• Branches - existing branches of project Git repository. Here you can compare, delete and create branches. Note that you cannot delete default branch (which is configured in the project settings)
• Tags - existing tags of project Git repository. Here you can view source code corresponding to particular tag or download this code as zip or tar.gz archive.

Storing binary files

In ABF, we do not encourage storing of binary files in Git repositories. This leads to large repository sizes and makes the clone processes slower. At the same time it doesn't make much sense to store binary files in version control system, since you cannot actually track modifications of their content and compare or merge different versions.

To store binary files, ABF uses File store - a separate file server which can be accessed by means of Web interface, REST API or ABF console client. The file store server allows to upload files and to extract them by their hash sums (computed using SHA1 algorithm).

ABF file store

Binary files should be uploaded to file store, and Git repository should only contain a special .abf.yml file (in YAML format) with list of binary files and their hash sums. Files should be listed in the sources section, for example:

sources:
  "myfile1.tar.gz": e80fd7a3d85e3a75e88780d36c80f672df8ddf64
  "myfile2.gif": a96c913906f6d36c30fa70f8c6ad259dbcfb0a98

When ABF starts a project build, it first analyzes .abf.yml and extracts all files listed there from the file store.

Since .abf.yml itself is stored in Git repository, one can track its modifications. And since the old versions of binary files are not removed from file store, you can always download versions of binary files corresponding to particular branch, tag or even commit.

Note that when ABF console client launches the "put command (i.e., you launch "abf put" inside project Git folder), it automatically uploads all binary files to file store, removes them from Git and adds necessary records to .abf.yml. When a project is created from SRPM package, biary files are automatically uploaded to file store, as well.

Building a project

If you project is a RPM package, then you can build it to repositories of distributions developed in ABF. The build is performed using spec file located at the root of Git repository. If Git doesn't contain any spec files or contain several spec files, then ABF will refuse to build the project.

A package can be built to personal repository of its owner (either group or user) and to repositories to which it was explicitly added. In order to add project to repository, you should go to repository's platform settings (see ABF: Platform Management for details).

To start project build from its page, go to the "Files" tab and press the "New build" button. On the appeared page you should choose target platform and repository (for which the project will be built), branch or tag of Git repository that will be used for the build and hardware architectures for which the package should be created. When you chooses target repository, then platform, branch and architectures will be chosen automatically on the basis of target platform settings. However, you can change the settings chosen automatically - in particular, you can build a package for for repository using a branch associated with another repository or build a package for a platform that by default uses packages from another group.

Then you should specify update type (it is used by package managers in users' systems - for example, to give users possibility to install only security updates and skip new packages). If needed, you can use additional repositories and results of other builds (not yet published to any repository) - this is useful when repositories used by default do not contain all packages necessary for the build. Note that you can use repositories of personal platforms only if you create a build for another personal platform. To add extra repository, specify its platform name, choose repository from the list appeared and press "Add".

Requirements for the extra build lists:

• one can add only a list with container (see below);
• for non-personal platforms, you can use only build lists for the same platform;
• ABF automatically analyzes hardware architecture of extra build lists and for every build uses only those lists which have the same architecture.

In the "Extra params" section one can specify additional arguments that will be passed to the tools that form build chroot and actually build packages (e.g., for 'urpmi' and 'mock-urpm' in ROSA Desktop).

In the "Preferences" section one can specify the following settings:

• should resulting packages be published to the target repository automatically in case of successful build;
• should container be created which will contain resulting packages;
• should testing repositories be used to resolve build dependencies (this option makes sense only if target platform has such repositories);
• if it is desired to use extra build nodes for the build (extra build nodes are provided by ABF users and are not controlled by ABF admins.

After the build setup is complete, press the "Start build" button.

Starting project build

Build monitoring

After the build is launched, you will be redirected to the task monitoring page where you will be able to view build tasks corresponding to your project. You can also track status of builds of any projects at the "Task monitoring" page in the ABF main menu. You can use filters in the upper part of monitoring window to track status of projects satisfying certain criteria.

For every build task, the monitoring page displays its id, short build parameters (target platform/repository, project version for already build packages, etc.) and its current status:

• Build pending - the build is not actually started yet, since all build nodes are busy;
• Build started - build process is ongoing; for such processes, ABF display time passed since the build was started and estimated time (calculated on the basis of previous builds of the same package for the same architecture)
• Build is rerun tests - user initiated tests relaunch for the successful build
• Build complete - the build was completed successfully but not published (since the automated publishing is disable by repository settings or user didn't choose this option when creating the build)
• Build is been published - the build was completed successfully and ABF is now publishing built packages to the target repository
• Build has been published - the build was completed successfully and built packages was published to the target repository
• Tests failed - the build was completed successfully, but packages failed to pass tests
• Build failed - build failed
• Publishing failed - the build was completed successfully, but there were errors when publishing built packages to repository.

To save screen space, similar tasks are grouped at the task monitoring page by default. More particular, ABF groups tasks launched for the same project using the same branch or tag but for different repositories or architectures (or just several builds with exactly the same parameters). For such a group of tasks, ABF displays status only one of the tasks (the choice is performed on the basis of statuses). To get details about all tasks, you can click on the arrow icon which is located on the left, near the task id. Additionally, a set of rectangle boxes is displayed near the build id. They number is equal to number of build tasks in the group, and color of every box corresponds to the task status (green for successful builds, red for build or publishing fails).

Build monitoring

Build results and containers

As a result of successful build, you will get an SRPM package and one or several RPM packages. You can find links to these packages on the build page, in the "Packages" section. Depending of build and platform configuration, these packages can be either published to the target repository automatically after they are built or ABF can create a separate container for them. Container is a standalone repository which contains only packages from the current build. Container has all necessary metadata files and you can use is in your system as an ordinary repository. The build page will provide you with a path to the container; if container was not created, you can push "Create container" button to create it.

Also note that on the build page you can create a new build with exactly the same parameters by clicking on "Recreate build list". The current build will not be destroyed.

A page with build results

If resulting packages were not published to repository automatically, you can initiate publishing process from the build page. If the build was already published, it can be published once again. This can be useful if target repository contains newer versions of the same packages, but that packages contain errors and it is necessary to revert to previous version. Note that "Publish" and "Publish again" buttons are available only if you have enough privileges to publish packages to the target repository.

In the "Logs" sections, you can find different build logs:

• abfworker::publish-worker-* - a log of publishing of built projects
• abfworker::rpm-worker-* - complete log of project build
• changelog.log - list of changes created on the basis of comments to Git commits; this changelog is automatically attached ton the Changelog section of built packages
• chroot-tree.log - complete list of files in the build environment; this log is available only for unsuccessful builds
• rpm-build.log - separate log of creation of SRPM and RPM packages
• rpm-qa.log - list of packages that were installed in the build environment
• rpm-root.log - log of creation of a chroot for building RPM packages
• rpm-state.log - a log of build state changes during RPM build
• rpmlint.log - a separate log of rpmlint results for the built packages
• src-rpm-build.log - a separate log of SRPM build
• src-rpm-root.log - log of creation of a chroot for building SRPM package
• src-rpm-state.log - a log of build state changes during SRPM build
• tests.log - log of tests launched for the built packages

Depending on build results and platform settings, some of these logs may be absent.

Pull Requests

If you have two different versions of the same project (e.g., two projects, one of which was forked from another in the past, or just two branches of the same project) and you want to merge changes from one version to another, you can use a Pull Request. In order to send a pull request, go to a page of the project from which the changes should be taken. On this page, press the «Pull request» button. Note that this button is available on for users that have write permissions for the project.

In the form appeared, choose a branch of the current project from which the changes should be taken and choose a target project and a target branch. After this is doen, press «Update commits».

Creating a pull request

ABF will analyze source code in the target branch and will show a form to input pull request name and description. You will also see «Diff» and «Commits» tabs where you can investigate in details which changes are going to be merged with the new pull request. ABF will also check if these changes can be merged automatically — that is, if git merge will succeed.

Creating a pull request

To complete pull request creation, press the «Create pull request» button.

After this you will be redirected to the page of the target project. In this page please check the «Pull Requests» tab — your new request will be displayed there.

Incoming pull requests

Everyone with write permissions for the project can click on pull request name and analyze its content. On this page one can also leave comments for the requests and discuss them.

Handling a pull request

If the changes can be merged automatically, you will see a button labeled «This pull request can be merged automatically». Press this button to automatically merge the requested changes.

If the changes conflict with the current state of the target project, then you will not be able to see such a button, and the request itself will be marked as «Blocked» (you will see corresponding lable in request title). Such conflicts should be resolved manually by cloning git repositories and merging their changes.

When you are finished with the pull request, press «Close pull request» to close it.