👹 👩🏼‍🍳 👩🏼 Creating reusable pipelines for GitLab CI in bash 🌙 🚏 ☕️

Over the past few years, I have grown very fond of GitLab CI . Mainly for its simplicity and functionality. It is enough just to create a file in the root of the repository .gitlab-ci.yml

, add a few lines of code there, and the next commit will start the pipeline with a set of jobs that will execute the specified commands.

And if you add the include and extends capabilities to this , you can do quite interesting things: create template jobs and pipelines, move them into separate repositories and reuse them in different projects without copying the code.

But unfortunately, not everything is as rosy as we would like. The instruction script

in GitLab CI is very low level. It simply executes the commands that are given to it in the form of strings. It is not very convenient to write large scripts inside YAML. As the logic becomes more complex, the number of scripts increases, they mix with YAML making configs unreadable and complicating their maintenance.

I really missed some mechanism that would simplify the development of large scripts. As a result, a microframework for the development of GitLab CI was born, which I want to talk about in this article (using the example of a simple pipeline for building docker images).

Example: Building docker images in GitLab

I want to consider the process of creating a pipeline using an example of a simple task that I often met with different teams: creating basic docker images in GitLab for reuse.

For example, a team writes microservices and wants to use their own base image for all of them with a set of preinstalled debugging utilities.

Or another example, a team writes tests and wants to use services directly in GitLab to create a temporary database (or queue, or something else) for them using their own image.

, docker- GitLab . .gitlab-ci.yml

:

services:
  - docker:dind

Build:
  image: docker
  script:
    - |
      docker login "$CI_REGISTRY" \
        --username "$CI_REGISTRY_USER" \
        --password "$CI_REGISTRY_PASSWORD"

      docker build \
        --file "Dockerfile" \
        --tag "$CI_REGISTRY/$CI_PROJECT_PATH:$CI_COMMIT_REF_SLUG" .

      docker push "$CI_REGISTRY/$CI_PROJECT_PATH:$CI_COMMIT_REF_SLUG"

Build

, GitLab Container Registry, . , ( , ).

, , :

, dockerfiles

.
:
- Build All
  
  - dockerfiles
  
  . ( ).
- Build Changed
  
  - dockerfiles
  
  , . ( ) .

. . , .

, .NET.

docker-, Container Registry :

, .

1: " "

. .gitlab-ci.yml

. :

services:
  - docker:dind

stages:
  - Build

Build All:
  stage: Build
  image: docker
  when: manual
  script:
    - |
      dockerfiles=$(find "dockerfiles" -name "*.Dockerfile" -type f)

      docker login "$CI_REGISTRY" \
        --username "$CI_REGISTRY_USER" \
        --password "$CI_REGISTRY_PASSWORD"

      for dockerfile in $dockerfiles; do
        path=$(echo "$dockerfile" | sed 's/^dockerfiles\///' | sed 's/\.Dockerfile$//')
        tag="$CI_REGISTRY/$CI_PROJECT_PATH/$path:$CI_COMMIT_REF_SLUG"

        echo "Building $dockerfile..."
        docker build --file "$dockerfile" --tag "$tag" .

        echo "Pushing $tag..."
        docker push "$tag"
      done

Build Changed:
  stage: Build
  image: docker
  only:
    changes:
      - 'dockerfiles/*.Dockerfile'
      - 'dockerfiles/**/*.Dockerfile'
  script:
    - |
      apk update
      apk add git #  ,     ,    ...

      dockerfiles=$(git diff --name-only HEAD HEAD~1 -- 'dockerfiles/***.Dockerfile')

      docker login "$CI_REGISTRY" \
        --username "$CI_REGISTRY_USER" \
        --password "$CI_REGISTRY_PASSWORD"

      for dockerfile in $dockerfiles; do
        path=$(echo "$dockerfile" | sed 's/^dockerfiles\///' | sed 's/\.Dockerfile$//')
        tag="$CI_REGISTRY/$CI_PROJECT_PATH/$path:$CI_COMMIT_REF_SLUG"

        echo "Building $dockerfile..."
        docker build --file "$dockerfile" --tag "$tag" .

        echo "Pushing $tag..."
        docker push "$tag"
      done

( step1)

Build All

, :
- , .. when: manual
  
  .
- :
  - find "dockerfiles" -name "*.Dockerfile" -type f
Build Changed

, :
- , .. only:changes
  
  .
- , :
  - git diff --name-only HEAD HEAD~1 -- 'dockerfiles/***.Dockerfile'
, , , dockerfiles/

.Dockerfile

, GitLab Container Registry.

GitLab CI .

GitLab CI Bootstrap

GitLab CI Bootstrap - GitLab CI. :

( .gitlab-ci.yml

) (bash shell), YAML .
, .
( ) , .

GitLab CI Bootstrap bootstrap.gitlab-ci.yml, .gitlab-ci.yml

include. . include:local:

include:
  - local: 'bootstrap.gitlab-ci.yml'

.bootstrap

, :

.bootstrap:
  before_script:
    - |
     	...

.bootstrap

, extends:

example:
  extends: '.bootstrap'
  script:
    - '...'

( ), GitLab.

. . , , .

bash shell. git, . .bootstrap

docker- .

, , docker-.

2:

, .bootstrap

.gitlab-ci.sh

. , .

, , .gitlab-ci.yml

:

.gitlab-ci.yml

:

include:
  - project: '$CI_PROJECT_NAMESPACE/bootstrap'
    ref: 'master'
    file: 'bootstrap.gitlab-ci.yml'

services:
  - docker:dind

stages:
  - Build

Build All:
  stage: Build
  image: docker
  extends: .bootstrap
  when: manual
  script:
    - search_all_dockerfiles_task
    - build_and_push_dockerfiles_task

Build Changed:
  stage: Build
  image: docker
  extends: .bootstrap
  only:
    changes:
      - 'dockerfiles/*.Dockerfile'
      - 'dockerfiles/**/*.Dockerfile'
  script:
    - install_git_task
    - search_changed_dockerfiles_task
    - build_and_push_dockerfiles_task

.gitlab-ci.sh

:

DOCKERFILES=""

function search_all_dockerfiles_task() {
    DOCKERFILES=$(find "dockerfiles" -name "*.Dockerfile" -type f)
}

function search_changed_dockerfiles_task() {
    DOCKERFILES=$(git diff --name-only HEAD HEAD~1 -- 'dockerfiles/***.Dockerfile')
}

function install_git_task() {
    #  ,     ,    ...
    apk update
    apk add git
}

function build_and_push_dockerfiles_task() {
    docker login "$CI_REGISTRY" \
        --username "$CI_REGISTRY_USER" \
        --password "$CI_REGISTRY_PASSWORD"

    for dockerfile in $DOCKERFILES; do
        path=$(echo "$dockerfile" | sed 's/^dockerfiles\///' | sed 's/\.Dockerfile$//')
        tag="$CI_REGISTRY/$CI_PROJECT_PATH/$path:$CI_COMMIT_REF_SLUG"

        echo "Building $dockerfile..."
        docker build --file "$dockerfile" --tag "$tag" .

        echo "Pushing $tag..."
        docker push "$tag"
    done
}

gitlab-ci.sh

, .gitlab-ci.yml

. search_all_dockerfiles_task

search_changed_dockerfiles_task

DOCKERFILES

, build_and_push_dockerfiles_task

.

3:

, , . , , . , - , , .

include.

. , , , - , . , .

, .bootstrap

CI_IMPORT

, : CI_{module}_PROJECT

, CI_{module}_REF

CI_{module}_FILE

, {module}

- CI_IMPORT

( ).

, .gitlab-ci.sh

, .

.gitlab-ci.yml

:

include:
  - project: '$CI_PROJECT_NAMESPACE/dockerfiles-example-ci'
    ref: 'step3'
    file: 'dockerfiles.gitlab-ci.yml'

dockerfiles.gitlab-ci.yml

( .gitlab-ci.yml

):

include:
  - project: '$CI_PROJECT_NAMESPACE/bootstrap'
    ref: 'master'
    file: 'bootstrap.gitlab-ci.yml'

services:
  - docker:dind

stages:
  - Build

variables:
  CI_DOCKERFILES_PROJECT: '$CI_PROJECT_NAMESPACE/dockerfiles-example-ci'
  CI_DOCKERFILES_REF: 'step3'
  CI_DOCKERFILES_FILE: 'dockerfiles.gitlab-ci.sh'

Build All:
  stage: Build
  image: docker
  extends: .bootstrap
  variables:
    CI_IMPORT: dockerfiles
  when: manual
  script:
    - search_all_dockerfiles_task
    - build_and_push_dockerfiles_task

Build Changed:
  stage: Build
  image: docker
  extends: .bootstrap
  variables:
    CI_IMPORT: dockerfiles
  only:
    changes:
      - 'dockerfiles/*.Dockerfile'
      - 'dockerfiles/**/*.Dockerfile'
  script:
    - search_changed_dockerfiles_task
    - build_and_push_dockerfiles_task

dockerfiles.gitlab-ci.sh

( .gitlab-ci.sh

):

DOCKERFILES=""

function search_all_dockerfiles_task() {
    DOCKERFILES=$(find "dockerfiles" -name "*.Dockerfile" -type f)
}

function search_changed_dockerfiles_task() {
    DOCKERFILES=$(git diff --name-only HEAD HEAD~1 -- 'dockerfiles/***.Dockerfile')
}

function build_and_push_dockerfiles_task() {
    docker login "$CI_REGISTRY" \
        --username "$CI_REGISTRY_USER" \
        --password "$CI_REGISTRY_PASSWORD"

    for dockerfile in $DOCKERFILES; do
        path=$(echo "$dockerfile" | sed 's/^dockerfiles\///' | sed 's/\.Dockerfile$//')
        tag="$CI_REGISTRY/$CI_PROJECT_PATH/$path:$CI_COMMIT_REF_SLUG"

        echo "Building $dockerfile..."
        docker build --file "$dockerfile" --tag "$tag" .

        echo "Pushing $tag..."
        docker push "$tag"
    done
}

Build All

Build Changed

CI_IMPORT

dockerfiles

. : CI_DOCKERFILES_PROJECT

, CI_DOCKERFILES_REF

CI_DOCKERFILES_FILE

, .

, dockerfiles.gitlab-ci.sh

, $CI_PROJECT_NAMESPACE/dockerfiles-example-ci

, step3

.

, install_git_task

. git , .. git git .

4:

. , .

dockerfiles.gitlab-ci.sh

. , .

, include

, , . , . include

, .

, , include

.

dockerfiles.gitlab-ci.sh

:

include "tasks/search.sh"
include "tasks/docker.sh"

tasks/search.sh

:

DOCKERFILES=""

function search_all_dockerfiles_task() {
    DOCKERFILES=$(find "dockerfiles" -name "*.Dockerfile" -type f)
}

function search_changed_dockerfiles_task() {
    DOCKERFILES=$(git diff --name-only HEAD HEAD~1 -- 'dockerfiles/***.Dockerfile')
}

tasks/docker.sh

:

function build_and_push_dockerfiles_task() {
    docker login "$CI_REGISTRY" \
        --username "$CI_REGISTRY_USER" \
        --password "$CI_REGISTRY_PASSWORD"

    for dockerfile in $DOCKERFILES; do
        path=$(echo "$dockerfile" | sed 's/^dockerfiles\///' | sed 's/\.Dockerfile$//')
        tag="$CI_REGISTRY/$CI_PROJECT_PATH/$path:$CI_COMMIT_REF_SLUG"

        echo "Building $dockerfile..."
        docker build --file "$dockerfile" --tag "$tag" .

        echo "Pushing $tag..."
        docker push "$tag"
    done
}

GitLab CI bash GitLab CI Bootstrap.

. :

. .. , . GitLab CI.
. , Python C#.
. , .

, .

, 1.0. , . , .

Sources: can be found here: https://gitlab.com/chakrygin/bootstrap

Repositories with examples here:

Creating reusable pipelines for GitLab CI in bash

Example: Building docker images in GitLab

1: " "

GitLab CI Bootstrap

2:

3:

4:

More articles: