docs/en/Community-Developer-Guides/Source_Independent_Selection_Build.md

# openKylin source code independent selection build process

# I. Package version number selection

## 1. selection strategy

### 1.1 Software project selection strategy

![software_project_selection_strategies.png](./assets/source_independent_selection_build/software_project_selection_strategies.png)

- Continue to actively introduce full "A" software (version)

- Encourage the introduction of "A" and "B" software only (version) tory-view-113483.html

- Cautiously introduce software with "C" (version)

- Reject/delete software (version) with "D" or "C >= 3" in time

### 1.2 Software version selection strategy

- Long-term support version (for the time being, openKylin does not have a long-term support version, 1.0 is not a long-term support version at the moment)
  - Give priority to the version widely used by mainstream OS, taking into account both quality and stability.
  - The compatibility list is not affected by the upgrade.
  - No major version changes during the life cycle of the same version, only feature and patch rounds to solve quality problems.

- Community development version
  - Meet the basic compliance issues, integrate as much software as possible, giving priority to the latest version of the current stable branch of open source components.
  - Follow up on upstream community dynamics in a timely manner and merge new feature versions of open source software.

### 1.3 Software classification and compatibility principles

| **Level** | **Scope** | **Principles** |
| -------- | --------------------------------------- | ------------------------------------------------------------ |
| Level 1 | kernel, glibc, gcc, zlib, and other core system components | APIs and ABI remain stable for the lifetime of the main release, and as much as possible for the next main release |
| Level 2 | Base software such as dbus, oepnssl, bzip2, systemd, etc. | APIs and ABIs remain stable for the lifetime of a single major release, and dependent applications do not require major changes in principle for the lifetime of a single major release | Level 3 | Other applications, etc. | APIs and ABIs remain stable for the lifetime of a single major release.
| The API and ABI are not forced to remain stable during the life cycle of a single major version, and dependent applications are upgraded inline.

## The specific process

1. select the new projects to be introduced according to the "Software Project Selection Policy". 2.

2. select a version of the project according to the "software version selection policy"

3. output the "project selection report", including the compliance, compatibility, importance, activity, quality, security details of the new project, as well as the new features of the selected version, compatibility, impact domain, comparison of versions in other distributions, etc.

4. the technical committee adopted the following process to carry out specific work

### 2.1 Obtain the project address of the package

The project address is used to view the current project details, development status, compilation and installation, instructions, etc. of the package

There are several ways to obtain the project address of the package.

#### 2.1.1 The debian community's package tracking platform

Most packages can be found on the debian community's package tracker ([tracker.debian.org](https://tracker.debian.org/))

For example, to find the `libnftnl` package

![example_of_the_debian_community_package_tracking_platform.png](./assets/source_independent_selection_build/example_of_the_debian_community_package_tracking_platform.png)

The project address of the package can be accessed through the homepage information in the upper right corner, as shown above.

#### 2.1.2 dsc file

The dsc file of a package can be viewed on the debian community ([tracker.debian.org](https://tracker.debian.org/)), ubuntu (https://launchpad.net), etc. The dsc file may contain information describing the project address of the current package .

For example: `libnftnl` in the debian community to view the package [dsc file](https://deb.debian.org/debian/pool/main/libn/libnftnl) (hereinafter in [version 1.1.2-2](https://deb.debian.org/)) debian/pool/main/libn/libnftnl/libnftnl_1.1.2-2.dsc) as an example).

```Bash
# Remove gpg signature information for easy viewing
Format: 3.0 (quilt)
Source: libnftnl
Binary: libnftnl11, libnftnl-dev
Architecture: linux-any
Version: 1.1.2-2
Maintainer: Debian Netfilter Packaging Team <pkg-netfilter-team@lists.alioth.debian.org>
Uploaders: Arturo Borrero Gonzalez <arturo@debian.org>
Homepage: https://git.netfilter.org/libnftnl
Standards-Version: 4.2.1
Vcs-Browser: https://salsa.debian.org/pkg-netfilter-team/pkg-libnftnl
Vcs-Git: https://salsa.debian.org/pkg-netfilter-team/pkg-libnftnl
Build-Depends: debhelper (>= 11), libmnl-dev, libtool, pkg-config
package-list:
 libnftnl-dev deb libdevel optional arch=linux-any
 libnftnl11 deb libs optional arch=linux-any
Checksums-Sha1:
 c0f880588fabaa845a88fb5a2bdad0dea7481857 366014 libnftnl_1.1.2.orig.tar.bz2
 0a9063e79191955fbf59482d75c7bfb62aded816 912 libnftnl_1.1.2.orig.tar.bz2.asc
 505f8b44e3adb574dc3cf657580e9d73f0b0a853 8772 libnftnl_1.1.2-2.debian.tar.xz
Checksums-Sha256:
 a5c7b7a6c13c9c5898b13fcb1126fefce2015d5a96d7c354b19aaa40b6aece5d 366014 libnftnl_1.1.2.orig.tar.bz2
 4178552d7ad210e9f17df3c079ac6de032f62c72774d59bfac404a023d48950b 912 libnftnl_1.1.2.orig.tar.bz2.asc
 4ef80a0c344b23624d24b7dd4be1b73a7b96d86089813c545705f8093067ce13 8772 libnftnl_1.1.2-2.debian.tar.xz
Files:
 14093a238d5025d4a452e6d1cef88c58 366014 libnftnl_1.1.2.orig.tar.bz2
 e10dcb9138261d92e7569bd72b809a28 912 libnftnl_1.1.2.orig.tar.bz2.asc
 518ff0360f67c8031a31792a257a5c3b 8772 libnftnl_1.1.2-2.debian.tar.xz
```


As highlighted in.

- **Homepage** : project address

- **Vcs-Browser** : Version control system browsing platform (similar to github, gitee, gitlab, etc.)

- **Vcs-Git** : git repository address

#### 2.1.3 debian/control files

First download any source package from debian community, ubuntu community (launchpad.net) and unzip it.

```Bash
# Download the source package from the ubuntu community (launchpad.net)
# You can also use apt-source to download source code by adding deb-src source
# To use dget you need to install devscripts
dget -x https://launchpad.net/ubuntu/+archive/primary/+sourcefiles/libnftnl/1.2.3-1/libnftnl_1.2.3-1.dsc

# Go to the unpacked directory
cd libnftnl-1.2.3
```

Check the `debian/control` file for `Homepage`, `Vcs-Git`, `Vcs-Browser` and other related fields

For example.

``Bash
$ grep -E "^(Homepage|Vcs-*)" debian/control
Homepage: https://git.netfilter.org/libnftnl
Vcs-Git: https://salsa.debian.org/pkg-netfilter-team/pkg-libnftnl.git
Vcs-Browser: https://salsa.debian.org/pkg-netfilter-team/pkg-libnftnl
```

The descriptions of the fields highlighted above are the same as in subsection 2.1.2

#### 2.1.4 debian/watch files

As in subsection 2.1.3, the source code needs to be downloaded first. This time, the `debian/watch` file is viewed

The **debian/watch** file is mainly used to record the download address of the upstream source code (this file is not required, so it may not be present in some source packages)

``Bash
$ cat debian/watch  
version=3
opts=pgpsigurlmangle=s/$/.sig/ http://ftp.netfilter.org/pub/libnftnl/libnftnl-(\S+).tar.bz2
```

The upstream source code download address recorded in this file allows you to find its corresponding project address

For example.

```Bash
# The download address is usually an ftp or git release download address
# If the download address is hosted by another platform, then you may not know the project address
http://ftp.netfilter.org/pub/libnftnl

# The above address is a good guess for the project address
http://netfilter.org/pub/libnftnl
```

### 2.2 Getting the package version status

The latest stable version list of the current project is learned by looking at the project address.

This can be viewed in several ways.

- debian/watch file

If the package in subsection 2.1.4 has a debian/watch file, you can view the download address of the upstream package zip of the project.

```!
! The download address may contain a beta version with a name similar to xxxx-beta.tar.gz. Generally this beta version of the source code should not be selected.
```!

- git repositories

If it is a git repository, you can check if its counterpart has a release git release: 1.

1. If there is, then you can get a list of its releases on the git release page

2. If not, see if the corresponding `branch(branch)` or `tag(tag)` has the relevant version list information



# II. Building the source package

Select the version of the package to be built and the download address from the first section.

If there is already a project in the remote repository and you need to change the upstream version, then please check the contents of subsection 4.

## 1. Download the source code and modify the source code directory format

Download the source package locally and unzip it

```Bash
## Download the source code
wget https://www.netfilter.org/pub/libnftnl/libnftnl-1.2.3.tar.bz2

# Decompress
tar -xaf libnftnl-1.2.3.tar.bz2
```

Modify the source directory to conform to the **<source_name>-<source_version>** format. For example.

```Bash
$ ls -al
drwxr-x--@ - luzp 10 Aug 02:24 libnftnl-1.2.3
.rw-rw----@ 395k luzp 10 Aug 02:24 libnftnl-1.2.3.tar.bz2
```

## 2. build out the debian package directory

### 2.1 upstream source contains debian directory

```!
! Note: If the upstream source contains a debian directory, we need to manually rename the upstream debian directory to avoid other problems when we regenerate the debian directory
```

- Back up the original debian directory

```Bash
mv debian debian-orig
```

- Rebuild the debian directory

```Bash
debmake -u <upstream version number> -r <revision number> -t
```

This will create a new directory at the same level as the source code, in the format <name of the original source code directory> - <upstream version number>

Suppose.

The source tarball is `amdgcn-tools-13.tar.gz`

``Bash
# Extract the source code
# Unzip the source code to the amdgcn-tools-13 directory
tar -xaf amdgcn-tools-13.tar.gz

# Go to the source directory and backup the original debian directory
cd amdgcn-tools-13
mv debian debian-orig

# Rebuild the debian directory
debmake -u13 -rok1 -t

# After execution, a new directory will be created in the same level as amdgcn-tools-13
# <original source directory>-<upstream version number>, which is the following directory
ls amdgcn-tools-13-13
```
### 2.2 Building debian directories directly

If the upstream source code is pure source, then just build the debian directory directly.

``Bash
### Build the debian directory
debmake -z tar.bz2
```

-z Specify the tarball format for the source code
  - libnftnl-1.2.3.tar.bz2 would be specified as -z tar.bz2

  - The default source compression format is tar.gz

-b If it is a perl or python source package, you can specify this parameter to create more files for debmake in the corresponding programming language

```Bash
# If it is python
debmake -b ":python3"

# perl
debmake -b ":perl"
```

-t regenerate the upstream source package (i.e. xxx.orig.tar.gz)

More detailed debmake usage can be found in its man page

```Bash
man debmake
```

## 3. Modify build rules

The main thing is to modify the files in the debian directory.

### 3.1 control

The `control` file covers the contents of two sections.

#### 3.1.1 Source package section

A brief description of the current source package, and the compilation dependencies for building it

There are several fields that need to be noted.

- **Section** 

Describes the type of package, and is used to classify the package. For example, `mysql-8` should belong to `database`.

The options are.

admin, cli-mono, comm, database, debug, devel, doc, editors, education, electronics, embedded, fonts, games, gnome, gnu-r, gnustep, graphics, hamradio, haskell, httpd, interpreters, introspection, java, javascript, kde, kernel, libdevel, libs, lisp, localization, mail, math, metapackages, misc, net, news, ocaml, oldlibs, otherosfs, perl, php, python, ruby, rust, science, shells, sound, tasks, tex, text, utils, vcs, video, web, x11, xfce, zope

- **Priority**
  - Priority of the package

  - required System required packages
  - important Important packages, packages that are not required for the system, but preferably pre-installed on the system
  - standard Standard Relatively independent packages
  - optional This is the default priority option, which should be used unless the package is a system-required package

- **Maintainer** 
It needs to be written as the current openkylin sig group responsible for the package, e.g. `packaging sig` group is written as `Openkylin Developers <packaging@lists.openkylin.top>`

- **Vcs-Browser**:

Write it to the corresponding gitee repository address, e.g. https://gitee.com/openkylin/autoconf

- **Vcs-Git**.

Write to the corresponding gitee repository git address, e.g. https://gitee.com/openkylin/autoconf.git

- **Build-Depends and **

Build dependencies needed to compile architecture-related code

- **Build-Depends-Indep**

Compilation dependencies required to compile architecture-independent code

For **Build-Depends** and **Build-Depends-Indep** you can find the relevant compilation dependencies from the project address.

---
**Note:** If the control file references rules from another OS distribution, you need to remove or replace the version number rules in the other OS version, for example.

```
Build-Depends: 
  pkg1 (>= 1.0.0-0ubuntu1), 
  pkg1 (< 1.2.0-1debian2),
  pkg2 (> 1.0.0.build),
  pkg3 (> 2.0ubuntu19)
```

First of all, it should be clear that for quilt format version numbers generally have "-" in the format `<upstream_version>-<revision>`, while native format source packages generally do not contain "-"

The modification process can be done in two steps

**Step 1:**

For quilt-formatted versions, the version numbers 1.0.0-0ubuntu1 and 1.2.0-1debian2 in pkg1 contain the version number of the specific OS distribution, so they should be modified. It also becomes.

```
pkg1 (>= 1.0.0), 
pkg1 (< 1.2.0),
```

For version numbers in native format, you first need to determine its upstream version, for pkg2 and pkg3.

(1) Try to find the 1.0.0 version of pkg2, if you can find it, then 1.0.0 is the upstream version, and its compilation dependency should be changed to: pkg2 (> 1.0.0); same as pkg3

(2) If the upstream version of pkg2 cannot be found, and the current version number does not contain the name of other OS distributions, then it can be kept (because the source-package tool is used to repackage the source code, its upstream version number is also the current version number); but for pkg3, if pkg3 is OS self-developed (its source code name usually However, for pkg3, if pkg3 is OS developed (its source code name also contains OS name, e.g. ubuntu-keyring), then this package can be renamed and maintained by openkylin, with the corresponding source code name changed to openkylin-keyring and the version number redefined, e.g. 1:1.0.0; if it is not OS developed package, it is better for us to redefine the version number and maintain it according to If it is not an OS self-developed package, it is better that we also re-define the version number and maintain it according to our own version number, which can be discussed internally, e.g. 1:1.0.0 is also specified here.** Note that here, it is better to use the version number in : format, which indicates that the version number with : is always higher than the version number without. **

Then, for pkg2, pkg3, it is modified as follows

```
pkg2 (> 1.0.0.build),
pkg3 (> 1:1.0.0)
```
**Step 2 (native format is not handled):**

The second step is to improve on the first step. In general we need to clarify the content of the revisions in other OS distributions that follow the - horizontal line.

For <upstream_version>-<revision>, generally each revision is a fix for a bug, security vulnerability, etc. For ubuntu, each revision is made up of several patches.

So for pkg1 (>= 1.0.0-0ubuntu1), it is explicitly stated that it can only be used after revision 0ubuntu1, which means that using pkg1 version 1.0.0 is risky, and the most immediate phenomenon is that the compilation does not pass.

Therefore, it is necessary to combine our actual pkg1 version situation to.

(1) If pkg1 exists in the repository, its version is 1.0.0-ok1. In order to ensure that the above compilation passes, it is necessary to integrate the patch in ubuntu's 1.0.0-0ubuntu1. If the relevant patch is already integrated, then the compilation dependency becomes.

```
pkg1 (>= 1.0.0-ok1), 
pkg1 (< 1.2.0),
```

After the patch integration, the pkg1 version in the repository becomes 1.0.0-ok2. Then, where the dependencies are compiled, it becomes

```
pkg1 (>= 1.0.0-ok2), 
pkg1 (< 1.2.0),
```

(2) If pkg1 does not exist in the repository, then when we select the build independently for the first time, it is better to select the version between 1.0.0~1.2.0 to ensure that the compiled dependencies of each dependency package can be met.

The above is based on compilation dependencies. If the version number is also specified in the installation dependency field of the binary package, then it is handled in the same way as above.

#### 3.1.2 Binary package section

Describes which binary package programs the current source package can be divided into.

Common types of subpackages are

- lib

shared library, whose binary package name is usually `lib-xxx`.

- dev

Shared library headers and related development files, usually named `lib-xxx-dev`

- doc

Documentation of a package

If the documentation corresponds to the documentation of an executable, the binary package name is usually `xxx-doc`.

If the documentation is for a development library, the binary package name is usually `lib-xxx-doc`.

- bin

The binary package name is usually the name of the corresponding binary program of a package, such as the binary program of a compiled language (e.g., c/c++) and the executable program of an interpreted programming language (e.g., python, perl)

This sub-package can be used in combination with ``debmake`` when building the ``debian` package directory, e.g.

```Bash
# The package name is foo, which provides the binary and the documentation for the binary
debmake -b "foo:bin,foo-doc:doc"

# The package name is libfoo, which is the source code for a development library that provides documentation for the development library and the development library
debmake -b "libfoo:lib,libfoo-doc:doc"

# After execution, each binary package information template will be automatically written to debian/control file
```
### 3.2 rules

Package compilation and build rules

### 3.3 changelog

Modify the package version number in the format: <upstream_version>-<revisions>

In general, for the openkylin community, use `ok1` for the first revisions, e.g.

```Bash
autoconf (2.71-ok1) yangtze; urgency=low

  * Build for openKylin.

 -- Lu zhiping <luzhiping@kylinos.cn> Fri, 26 Aug 2022 15:25:09 +0800
```

### 3.4 source/format

The source package must be formatted as `quilt` format

```Bash
echo "3.0 (quilt)" > debian/source/format
```

```Bash
echo "3.0 (quilt)" > debian/source/format
```

### 3.5 Other files

xxx.install

xxx.doc

xxx.symbols

xxx.manpages

### 3.6 Applying patches from other OS distributions

If other OS distributions have patches for the package, then we need to analyze and consider whether to apply their patches.

You can start by considering whether to apply patches from other OS distributions in the following ways.

- What is fixed by the patch? Is it an irrelevant patch that has not been absorbed by the upstream community?

- Is it a patch that was only available in a previous, older release and has it been integrated in a new release?

- Is it a major vulnerability fix patch?

- Is it a patch that is only required for a specific OS distribution?

- If the patch is not applied is there a compilation problem and is there a problem with the installation?

```
Also, it's best to apply patches from the upstream community after building out the git repository, so that the information from the upstream community patches can be retained in the git commit
```

Here are the steps for applying patches after building out a git repository in Section 5.

If you need to apply patches from upstream communities, it is best to keep the content and author information of their patches. You can apply patches from upstream communities directly through the git tool.

- git am apply patch
The git am application patch will automatically keep the patch information

``Bash
git am xxxx.patch
```

- git apply applies patch

But some patches don't have the exact same information format as git am, so you can use git apply to import the patches and then commit to add the patch information

```Bash
# import patch information
git apply xxx.patch

# Add changes
git add .

# Fill in the edit box with information about the patch, such as
# the author of the patch, the content of the patch, etc.
git commit
git commit .

After importing a patch, if you want to compile and test the newly imported patch, you can use the gbp tool to generate the new source code (new dsc file)

```Bash
# Modify changelog to add a new version number
vim debian/changelog

# Compile the source code, which will generate a new dsc file
gbp buildpackage -S -sa -nc --git-prisine-tar --git-ignore-branch
```

## 4. Importing upstream source code

If the remote repository already has a project, you need to change the upstream version.

(1) Download the source tarball from the upstream community

(2) Re-compress the source tarball to match the ``<package-name>_<pacakge-version>.orig.tar.[gz|xz|bz2]` format

```Bash
# For example.
# The source tarball obtained from the upstream source community is requests-2.27.1.tar.gz
# First extract the source code
tar -xaf requests-2.27.1.tar.gz

# Change the source directory to <package-name>-<package-version>
# If the source tarball from the upstream community is extracted from the requests directory
mv requests requests-2.27.1

# Recompress to match <package-name>_<pacakge-version>.orig.tar.[gz|xz|bz2] 
tar -czf requests-2.27.1 requests-2.27.1.orig.tar.gz
```

(3) Clone the gitee remote repository source code to local, enter the source code directory

```Bash
# Clone the remote repository
git clone git@gitee.com:openkylin/requests.git

# Enter the source directory
cd requests
```
(4) Changing upstream source code

Refer to chapter 6 of the [openKylin source package git workflow](openKylin source package git workflow.md) section "Updating upstream versions".

- a. Import the upstream version (the zip generated in step 2)

- ``Bash
    gbp import-orig requests-2.27.1.orig.tar.gz --no-merge --pristine-tar --upstream-branch=upstream
    git checkout upstream
    git push
    git push --tags
    ```

- b. Filtering old patches (in conjunction with [openKylin source package git workflow] (openKylin source package git workflow.md) subsection 6.3.3)

- If the patch is an R&D patch, then you need to confirm with the relevant R&D that the patch is still required in the new upstream source code
  - If the patch is a patch from another OS distribution, then confirm with the OS distribution that the patch is still required in the latest version of the source code

- c. Introducing patches from other OS distributions

- You can refer to subsection 3.6, "Applying patches from other OS distributions".

  - Use git am or git apply or other methods to import patches from other OS distributions

- d. Modify other build files in the debian directory

- debian/control
  - debian/changelog needs to retain the original changelog information
  - debian/rules
  - deiban/other files

After the changes are done, you can follow section 5 and test the build locally first, then push it to the remote repository after the test is done.

## 5. Compile the source package

After completing the build rules for the debian directory, first compile a round of source packages based on this. The dsc file will be generated

``Bash
## Switch to the development branch
git checkout openkylin/yangtze
# Execute in the source directory
dpkg-source -b .
```



# III. Compile

After building the source package according to the previous section, compile it.

You can download a chroot environment of openkylin and compile and debug in the chroot environment.

The chroot for x86: http://files.build.openkylin.top/51159/chroot-openkylin-yangtze-amd64.tar.gz

riscv chroot: http://files.build.openkylin.top/51197/chroot-openkylin-yangtze-riscv64.tar.gz

At this stage, the package build rules in the debian package directory will be continuously modified and optimized until a complete installable binary package is packaged.



# IV. Installation Testing

Simply test whether the compiled binary is installable and whether the system works properly after installation, etc.

## V. Building a git repository

## 1. create gitee repository

fork openkylin's community repository

``Bash
# After fork to your personal repository, clone your personal repository to local
git clone git@gitee.com:<username>/community.git

# Then add the source package name to the corresponding sig group sig.yaml file
# For example, to add a package to the packaging sig, the source-name should be the actual package name
echo "- <source-name>" >> sig/packaging/sig.yaml

# Then commit the changes and push them to your personal repository
git add sig/packaging/sig.yaml
git commit -m "Add xxx package"
git push origin master
```

Create a PR on the gitee web side, and then have it reviewed by the community project leader. After the review, the CI platform will automatically create the corresponding git repository

## 2. Build the git repository

Build the git repository for the source package built in chapter 2.

- Download the tool

Follow the [source-packaging](https://gitee.com/openkylin/packaging-tools#源码反向构建工具-source-packing) tool instructions to download the tools and install the dependencies.

- Build the git repository

```Bash
. /source-packing import-to-git --dsc-file sqlite3_3.31.1-ok1.dsc --packaging-branch openkylin/yangtze
```

More detailed git workflow can be found in [openKylin source package git workflow](openKylin source package git workflow.md) 

## 3. Associating remote repositories and pushing to gitee

You need the package to be compiled in chapter 3 and installed and tested in chapter 4 before pushing it to the gitee repository

```Bash
# Go to the source code repository built in step 2
cd sqlite3
# Associate the remote repository and push to it
git remote add origin git@gitee.com:openkylin/sqlte3.git
git push --all && git push --tags
```



# VI. Other content

1. A brief description of gitee's ci process.

```Bash
1. package first push.
maintainer first pushes the package to the gitee repository by way of push, at which point the ci platform sends its package to the okbs compilation platform for compilation, and the current compilation address is (https://build.openkylin.top/~cibot/+archive/openkylin/ citest/+packages)

2. Subsequent modification contributions.
The developer needs to fork the repository to the personal repository, then develop changes on the personal repository, and then modify the changelog to add the version number after the source code is modified, and then submit the PR to the original repository.

maintainer review PR, PR review passed, CI platform automatically package the current PR sent to OKBS compilation platform for compilation, the compilation results will be added to the corresponding PR in the form of comments, maintainer according to the compilation results and CI platform gate check results to consider whether to accept the PR
```

2. The source address of each distribution, using gvfs as an example.

Debian 11: https://tracker.debian.org/pkg/gvfs

Ubuntu 22.04: https://launchpad.net/ubuntu/+source/gvfs

Fedora 36: https://src.fedoraproject.org/rpms/gvfs

Arch: https://archlinux.org/packages/extra/x86_64/gvfs/

Deepin: https://ftp.sjtu.edu.cn/deepin/pool/

openEuler: https://gitee.com/src-openeuler/gvfs