Jenkins更新検証環境をDockerで作る

Jenkins・pluginの更新をする際に検証できる環境が欲しい!となったので、Docker上に構築したメモ

成果物

github.com

やりたかったこと

  • できるだけ運用環境と近い状態から以下の検証ができるようにする
    • Jenkins自体の更新確認
    • pluginの追加・更新確認

運用環境

Docker Hubからjenkins imageを探す

  ❯ docker pull jenkins/jenkins:2.138.3
  2.138.3: Pulling from jenkins/jenkins
  Status: Downloaded newer image for jenkins/jenkins:2.138.3
  docker.io/jenkins/jenkins:2.138.3

  # コンテナの中に入る
  ❯ docker run -it --name jenkinstest jenkins/jenkins:2.138.3 /bin/bash
  jenkins@88681d8f77c5:/$

  jenkins@88681d8f77c5:~$ cat /etc/issue
  Debian GNU/Linux 9 \n \l

Debian だったのでこれは🙅

  ❯ docker pull jenkins/jenkins:2.182-centos
  2.182-centos: Pulling from jenkins/jenkins
  Status: Downloaded newer image for jenkins/jenkins:2.182-centos
  docker.io/jenkins/jenkins:2.182-centos


  # 起動してみる
  ❯ docker run -p 8080:8080 -p 50000:50000 jenkins/jenkins:2.182-centos
  Running from: /usr/share/jenkins/jenkins.war
  webroot: EnvVars.masterEnvVars.get("JENKINS_HOME")

  # warファイルから起動してる??

  # コンテナの中に入る
  ❯ docker run -it --name jenkins-update-test jenkins/jenkins:2.182-centos /bin/bash
  bash-4.2$ cat /etc/redhat-release
  CentOS Linux release 7.6.1810 (Core)
  bash-4.2$ echo $HOME
  /var/jenkins_home

war ファイルからの起動のようなのでこれは🙅

Centos 7 のimageにJenkinsをyumインストールする方法を試す

rpmを探す

Jenkinsインストール

Javaランタイムインストール

[root@b92d081ebab0 /]# yum -y install java-1.8.0-openjdk

Jenkins公式yumリポジトリの追加

[root@b92d081ebab0 /]# curl -o /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo

[root@b92d081ebab0 /]# rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key

[root@b92d081ebab0 /]# yum info jenkins
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: ftp.jaist.ac.jp
 * extras: ftp.jaist.ac.jp
 * updates: ftp.jaist.ac.jp
jenkins                                                                                  | 2.9 kB  00:00:00
jenkins/primary_db                                                                       |  35 kB  00:00:00
Available Packages
Name        : jenkins
Arch        : noarch
Version     : 2.263.1
Release     : 1.1
Size        : 64 M
Repo        : jenkins
Summary     : Jenkins Automation Server
URL         : http://jenkins.io/
License     : MIT/X License, GPL/CDDL, ASL2
Description : Jenkins is an open source automation server which enables developers around the world to reliably
            : automate  their development lifecycle processes of all kinds, including build, document, test,
            : package, stage, deployment, static analysis and many?more.
            :
            : Jenkins is being widely used in areas of Continous Integration, Continuous Delivery, DevOps, and
            : other areas. And it is not only about software, the same automation techniques can be applied in
            : other areas like Hardware Engineering, Embedded Systems, BioTech, etc.
            :
            : For information see https://jenkins.io
            :
            :
            : Authors:
            : --------
            :     Kohsuke Kawaguchi <kk@kohsuke.org>

rpmインストール

[root@b92d081ebab0 tmp]# curl -OL http://mirrors.jenkins-ci.org/redhat-stable/jenkins-2.204.2-1.1.noarch.rpm

[root@b92d081ebab0 tmp]# ls
hsperfdata_root  jenkins-2.204.2-1.1.noarch.rpm  ks-script-DrRL8A  yum.log

[root@b92d081ebab0 tmp]# rpm -ivh jenkins-2.204.2-1.1.noarch.rpm

[root@b92d081ebab0 tmp]# systemctl start jenkins
Failed to get D-Bus connection: Operation not permitted

Failed to get D-Bus connection: Operation not permitted エラー

Dockerのエラーのようなので、以下を参考にして対応

qiita.com

一旦削除してdocker runやり直し 特権モードで起動させる

❯ docker run -d --privileged --name jenkins-test centos:centos7 /sbin/init

❯ docker ps
CONTAINER ID   IMAGE            COMMAND        CREATED          STATUS          PORTS     NAMES
cd5af186cc01   centos:centos7   "/sbin/init"   28 seconds ago   Up 26 seconds             jenkins-test

❯ docker exec -it jenkins-test /bin/bash
[root@cd5af186cc01 /]#

## ~~java install, repository 登録, rpm install~~

[root@cd5af186cc01 tmp]# systemctl start jenkins
Job for jenkins.service failed because the control process exited with error code. See "systemctl status jenkins.service" and "journalctl -xe" for details.
[root@cd5af186cc01 tmp]# systemctl status jenkins.service
● jenkins.service - LSB: Jenkins Automation Server
   Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2021-01-03 14:45:56 UTC; 15s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 355 ExecStart=/etc/rc.d/init.d/jenkins start (code=exited, status=1/FAILURE)

Jan 03 14:45:56 cd5af186cc01 systemd[1]: Starting LSB: Jenkins Automation Server...
Jan 03 14:45:56 cd5af186cc01 jenkins[355]: /etc/rc.d/init.d/jenkins: line 59: /etc/init.d/functions: No s...tory
Jan 03 14:45:56 cd5af186cc01 systemd[1]: jenkins.service: control process exited, code=exited status=1
Jan 03 14:45:56 cd5af186cc01 systemd[1]: Failed to start LSB: Jenkins Automation Server.
Jan 03 14:45:56 cd5af186cc01 systemd[1]: Unit jenkins.service entered failed state.
Jan 03 14:45:56 cd5af186cc01 systemd[1]: jenkins.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

/etc/rc.d/init.d/jenkins: line 59: /etc/init.d/functions: No such file or directory エラー

以下と同じエラーだったので、同様の対応をしてみる

qiita.com

[root@cd5af186cc01 tmp]# yum provides */init.d/functions*

[root@cd5af186cc01 tmp]# yum install -y initscripts

[root@cd5af186cc01 tmp]# ls /etc/init.d/functions
/etc/init.d/functions

[root@cd5af186cc01 tmp]# systemctl start jenkins
[root@cd5af186cc01 tmp]# systemctl status jenkins
● jenkins.service - LSB: Jenkins Automation Server
   Loaded: loaded (/etc/rc.d/init.d/jenkins; bad; vendor preset: disabled)
   Active: active (running) since Sun 2021-01-03 15:04:01 UTC; 7s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 464 ExecStart=/etc/rc.d/init.d/jenkins start (code=exited, status=0/SUCCESS)
   CGroup: /docker/cd5af186cc0158a972d74dc63069144faa3849d05296134356de89d37a8f7a04/system.slice/jenkins.service
           └─493 /etc/alternatives/java -Dcom.sun.akuma.Daemon=daemonized -Djava.awt.headless=true -DJENKINS_...

Jan 03 15:04:00 cd5af186cc01 systemd[1]: Starting LSB: Jenkins Automation Server...
Jan 03 15:04:00 cd5af186cc01 runuser[469]: pam_unix(runuser:session): session opened for user jenkins by (uid=0)
Jan 03 15:04:01 cd5af186cc01 runuser[469]: pam_unix(runuser:session): session closed for user jenkins
Jan 03 15:04:01 cd5af186cc01 jenkins[464]: Starting Jenkins [  OK  ]
Jan 03 15:04:01 cd5af186cc01 systemd[1]: Started LSB: Jenkins Automation Server.
Hint: Some lines were ellipsized, use -l to show in full.

起動🙆

Dockerfileを作る

以下を参考に作成

qiita.com

qiita.com

Jenkins updateを試す

  • サービスSTOP → yum install → サービスSTART
[root@localhost jenkins]# systemctl stop jenkins
[root@localhost jenkins]# yum install jenkins
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: ty1.mirror.newmediaexpress.com
 * extras: ty1.mirror.newmediaexpress.com
 * updates: ty1.mirror.newmediaexpress.com
Resolving Dependencies
--> Running transaction check
---> Package jenkins.noarch 0:2.204.2-1.1 will be updated
---> Package jenkins.noarch 0:2.273-1.1 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================
 Package                   Arch                     Version                     Repository                 Size
================================================================================================================
Updating:
 jenkins                   noarch                   2.273-1.1                   jenkins                    66 M

Transaction Summary
================================================================================================================
Upgrade  1 Package

Total download size: 66 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
jenkins-2.273-1.1.noarch.rpm                                                             |  66 MB  00:00:09
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : jenkins-2.273-1.1.noarch                                                                     1/2
  Cleanup    : jenkins-2.204.2-1.1.noarch                                                                   2/2
  Verifying  : jenkins-2.273-1.1.noarch                                                                     1/2
  Verifying  : jenkins-2.204.2-1.1.noarch                                                                   2/2

Updated:
  jenkins.noarch 0:2.273-1.1

Complete!
[root@localhost jenkins]#
[root@localhost jenkins]# systemctl start jenkins

workspaceが空だと一瞬で終わる模様

Jenkinsのjob,plugin等のバックアップは

sue445.hatenablog.com

のバックアップスクリプトを利用させてもらっている。 そこのコメントに気になる記述がある。

JenkinsCIををrpmでアップデートすると、$JENKINS_HOMEをchown -Rする糞コマンド入っているので、workspace/以下を綺麗にしておくかexport JENKINS_INSTALL_SKIP_CHOWN=trueしておかないと何時間もupdate終わらんとかになる

実際に更新する際は、 export JENKINS_INSTALL_SKIP_CHOWN=true をしておくのがよさそう

バックアップデータからの復元でハマったところ

Error migrating users

バックアップファイルを解凍してjenkinsディレクトリにコピーすればOK
Docker imageの作成→コンテナ起動→jenkins起動は問題なかった
http://localhost:8080 にアクセスしてみると以下のエラーが表示され、ログイン画面がでない

java.nio.file.DirectoryNotEmptyException: /var/lib/jenkins/users/hoge
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:498)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at hudson.model.UserIdMigrator.migrateUsers(UserIdMigrator.java:97)
    at hudson.model.UserIdMapper.load(UserIdMapper.java:184)
    at hudson.model.UserIdMapper.init(UserIdMapper.java:75)

users データの読み込みに失敗している?

/var/log/jenkins/jenkins.log も見ながら解消していく

2021-01-15 02:48:09.783+0000 [id=34]    INFO    hudson.model.UserIdMigrator#migrateUsers: Migrating user 'hoge' from 'users/hoge/' to 'users/hoge_6878077609320809469/'
2021-01-15 02:48:09.792+0000 [id=34]    SEVERE  hudson.model.UserIdMapper#load: Error migrating users.
java.nio.file.DirectoryNotEmptyException: /var/lib/jenkins/users/hoge
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:498)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at hudson.model.UserIdMigrator.migrateUsers(UserIdMigrator.java:97)
    at hudson.model.UserIdMapper.load(UserIdMapper.java:184)
    at hudson.model.UserIdMapper.init(UserIdMapper.java:75)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:104)
    at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:175)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1121)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
2021-01-15 02:48:09.807+0000 [id=34]    SEVERE  jenkins.InitReactorRunner$1#onTaskFailed: Failed UserIdMapper.init

エラー内容を調べる

jenkins-ci.361315.n4.nabble.com

www.jenkins.io

In this case, rename the directory $JENKINS_HOME/users/admin_6635238516816951048/ to $JENKINS_HOME/users/admin/ while Jenkins is not running. Repeat this for all other user directories listed in the log. Doing all of the above will undo the partial migration. Jenkins will re-run it on the next startup.

対応する

このコメントを参考

stackoverflow.com

service stop → users/users.xmlの削除 or rename → service start → 古いユーザーcredentialのまま使う

users/users.xmlはバックアップデータに存在しなかったので、以下の方法で対応

  • バックアップのusersディレクリ名をusers_tmpに変更
  • 空のusersディレクトリを作成
  • jenkins service startする
    • この時点でJenkinsの画面が表示されるか確認。job等は復元されている状態
    • ログインユーザー情報がないので、ログインはできない
  • jenkins service stopする
    • users_tmpディレクト名をusersに変更
    • jenkins service startする
    • ログインできるか確認

おわりに

Dockerfileからのimage作成、コンテナ起動などはシェルスクリプトを作成した。

Dockerで更新検証環境を作ったことで、pluginを気軽に試したり更新したりできるようになりそう😋