rootlessでDockerコマンドが使えるGitHub ARCをデプロイする
目次
Ubuntu 24.04上のMicrok8sでGitHub Actions Runner Controllerを動かして、dockerコマンドの使えるworkflowをdocker:dind-rootlessで動かそうとしたら結構苦労したのでメモ
概要
-
公式ドキュメント https://docs.github.com/en/enterprise-cloud@latest/actions/tutorials/use-actions-runner-controller/deploy-runner-scale-sets#example-running-dind-rootless が間違っている。
-
Ubuntu 24.04にて標準で有効化されているAppArmorのルールに引っかかり、dind(Dockerが動いているサイドカーコンテナ)の起動に失敗する。
環境
- OS: Ubuntu 24.04
- Microk8s: 1.34/stable (v1.34.1)
- gha-runner-scale-set-controller: 0.13.0
ARCのインストール
ここは特につらいところもないので、公式ドキュメントに従ってデプロイする。
NAMESPACE="arc-systems"
sudo microk8s helm install arc \
--namespace "${NAMESPACE}" \
--create-namespace \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller
AppArmorの制限
Ubuntuの公式ブログに記載がある通り、Ubuntu 23.10にてセキュリティを向上させるために、rootless環境で利用するunprivileged user namespacesの権限がAppArmorを使った許可制になった。
Restricted unprivileged user namespaces are coming to Ubuntu 23.10 https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces
頑張って起動しようとすると以下のインストラクションが出てきたため、これをそのまま実行してAppArmorのルールを入れる。
(出てきたメッセージ)
time="2025-11-02T12:24:56Z" level=warning msg="[rootlesskit:parent] This error might have happened because /proc/sys/kernel/apparmor_restrict_unprivileged_userns is set to 1" error="fork/exec /proc/self/exe: permission denied"
time="2025-11-02T12:24:56Z" level=warning msg="[rootlesskit:parent] Hint: try running the following commands:
########## BEGIN ##########
cat <<EOT | sudo tee "/etc/apparmor.d/usr.local.bin.rootlesskit"
# ref: https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces
abi <abi/4.0>,
include <tunables/global>
/usr/local/bin/rootlesskit flags=(unconfined) {
userns,
# Site-specific additions and overrides. See local/README for details.
include if exists <local/usr.local.bin.rootlesskit>
}
EOT
sudo systemctl restart apparmor.service
########## END ##########
"
ARC Runnersのデプロイ
ここで概要に書いた事象にあたるため、以下のようなvalues.yamlを書いてデプロイする。
この例では認証情報を pre-defined-secret という名前のSecretに入れている。
変更点としては
- dindコンテナの定義をinitContainersからcontainersへ移動
- runnerコンテナの定義から
securityContext.privileged=trueを削除- privileged=true でなくても動いているので不要だと思う。動かないものがあれば戻してください。
## githubConfigUrl is the GitHub url for where you want to configure runners
## ex: https://github.com/myorg/myrepo or https://github.com/myorg
githubConfigUrl: "https://github.com/suuei/action-sandbox"
## githubConfigSecret is the k8s secrets to use when auth with GitHub API.
## You can choose to use GitHub App or a PAT token
githubConfigSecret: pre-defined-secret
## maxRunners is the max number of runners the autoscaling runner set will scale up to.
maxRunners: 5
## minRunners is the min number of idle runners. The target number of runners created will be
## calculated as a sum of minRunners and the number of jobs assigned to the scale set.
minRunners: 0
## template is the PodSpec for each runner Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
template:
spec:
initContainers:
- name: init-dind-externals
image: ghcr.io/actions/actions-runner:latest
command: ["cp", "-r", "/home/runner/externals/.", "/home/runner/tmpDir/"]
volumeMounts:
- name: dind-externals
mountPath: /home/runner/tmpDir
- name: init-dind-rootless
image: docker:dind-rootless
command:
- sh
- -c
- |
set -x
cp -a /etc/. /dind-etc/
echo 'runner:x:1001:1001:runner:/home/runner:/bin/ash' >> /dind-etc/passwd
echo 'runner:x:1001:' >> /dind-etc/group
echo 'runner:100000:65536' >> /dind-etc/subgid
echo 'runner:100000:65536' >> /dind-etc/subuid
chmod 755 /dind-etc;
chmod u=rwx,g=rx+s,o=rx /dind-home
chown 1001:1001 /dind-home
securityContext:
runAsUser: 0
volumeMounts:
- mountPath: /dind-etc
name: dind-etc
- mountPath: /dind-home
name: dind-home
containers:
- name: dind
image: docker:dind-rootless
args:
- dockerd
- --host=unix:///run/user/1001/docker.sock
securityContext:
privileged: true
runAsUser: 1001
runAsGroup: 1001
restartPolicy: Always
startupProbe:
exec:
command:
- docker
- info
initialDelaySeconds: 0
failureThreshold: 24
periodSeconds: 5
volumeMounts:
- name: work
mountPath: /home/runner/_work
- name: dind-sock
mountPath: /run/user/1001
- name: dind-externals
mountPath: /home/runner/externals
- name: dind-etc
mountPath: /etc
- name: dind-home
mountPath: /home/runner
- name: runner
image: ghcr.io/actions/actions-runner:latest
command: ["/home/runner/run.sh"]
env:
- name: DOCKER_HOST
value: unix:///run/user/1001/docker.sock
securityContext:
runAsUser: 1001
runAsGroup: 1001
volumeMounts:
- name: work
mountPath: /home/runner/_work
- name: dind-sock
mountPath: /run/user/1001
volumes:
- name: work
emptyDir: {}
- name: dind-externals
emptyDir: {}
- name: dind-sock
emptyDir: {}
- name: dind-etc
emptyDir: {}
- name: dind-home
emptyDir: {}
悪戦苦闘のログ
そもそも
AppArmorに対して何もせずデプロイするとdindコンテナがエラーで立ち上がらずWorkflowの実行ができない。エラーメッセージはこんな感じ。
cat: can't open '/proc/net/ip_tables_names': Permission denied
cat: can't open '/proc/net/ip6_tables_names': Permission denied
cat: can't open '/proc/net/arp_tables_names': No such file or directory
Device "nf_tables" does not exist.
nf_tables 376832 12 nft_chain_nat,nft_compat
nfnetlink 20480 11 ip_set,nf_conntrack_netlink,nfnetlink_acct,nft_compat,nf_tables
libcrc32c 12288 5 nf_nat,nf_conntrack,nf_tables,btrfs,raid456
modprobe: can't change directory to '/lib/modules': No such file or directory
Device "ip_tables" does not exist.
ip_tables 32768 4 iptable_raw,iptable_mangle,iptable_nat,iptable_filter
x_tables 65536 22 xt_set,xt_multiport,ipt_rpfilter,xt_CT,iptable_raw,xt_addrtype,xt_nat,xt_tcpudp,xt_MASQUERADE,xt_mark,xt_nfacct,ip6table_filter,ip6table_mangle,xt_conntrack,iptable_mangle,ip6table_nat,ip6_tables,iptable_nat,nft_compat,iptable_filter,xt_comment,ip_tables
modprobe: can't change directory to '/lib/modules': No such file or directory
Device "ip6_tables" does not exist.
ip6_tables 36864 3 ip6table_filter,ip6table_mangle,ip6table_nat
x_tables 65536 22 xt_set,xt_multiport,ipt_rpfilter,xt_CT,iptable_raw,xt_addrtype,xt_nat,xt_tcpudp,xt_MASQUERADE,xt_mark,xt_nfacct,ip6table_filter,ip6table_mangle,xt_conntrack,iptable_mangle,ip6table_nat,ip6_tables,iptable_nat,nft_compat,iptable_filter,xt_comment,ip_tables
modprobe: can't change directory to '/lib/modules': No such file or directory
iptables v1.8.11 (nf_tables)
[rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: operation not permitted
dmesgを見るとこのようなログが出ている。
[ 87.823880] audit: type=1400 audit(1762078015.633:325): apparmor="AUDIT" operation="userns_create" class="namespace" info="Userns create - transitioning profile" profile="unconfined" pid=7789 comm="rootlesskit" requested="userns_create" target="unprivileged_userns"
[ 87.824105] audit: type=1400 audit(1762078015.633:326): apparmor="DENIED" operation="capable" class="cap" profile="unprivileged_userns" pid=7838 comm="rootlesskit" capability=21 capname="sys_admin"
(比較用)書いた手順で起動した時のLABEL
動いているpythonがコンテナ内(Workflow内でdocker runをして実行したコンテナ内)で実行しているプログラム。
LABEL USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
unconfined root 6772 0.1 0.2 1239152 16884 ? Sl 15:22 0:00 /snap/microk8s/8491/bin/containerd-shim-runc-v2 -namespace k8s.io -id bfcd2773f1a226815f9a3a7b3c9ec72622d0dc080e3c3a9ca058b036a79fb969 -address /var/snap/microk8s/common/run/containerd.sock
unconfined 65535 6795 0.0 0.0 1020 640 ? Ss 15:22 0:00 \_ /pause
/usr/local/bin/rootlesskit (unconfined) 1001 7245 0.0 0.1 1233988 9856 ? Ssl 15:22 0:00 \_ rootlesskit --net=vpnkit --mtu=1500 --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run docker-init -- dockerd --host=unix:///run/user/1001/docker.sock
/usr/local/bin/rootlesskit (unconfined) 1001 7297 1.4 0.1 1234884 14056 ? Sl 15:22 0:01 | \_ /proc/self/exe --net=vpnkit --mtu=1500 --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run docker-init -- dockerd --host=unix:///run/user/1001/docker.sock
/usr/local/bin/rootlesskit (unconfined) 1001 7331 0.0 0.0 1008 512 ? S 15:22 0:00 | | \_ docker-init -- dockerd --host=unix:///run/user/1001/docker.sock
/usr/local/bin/rootlesskit (unconfined) 1001 7335 2.2 1.1 2329708 84340 ? Sl 15:22 0:02 | | \_ dockerd --host=unix:///run/user/1001/docker.sock
/usr/local/bin/rootlesskit (unconfined) 1001 7645 0.0 0.4 1268104 36352 ? Ssl 15:22 0:00 | | \_ containerd --config /run/user/1001/docker/containerd/containerd.toml
/usr/local/bin/rootlesskit (unconfined) 1001 7314 2.1 0.7 121496 57744 ? Sl 15:22 0:02 | \_ vpnkit --ethernet /tmp/rootlesskit1802182924/vpnkit-ethernet.sock --mtu 1500 --host-ip 0.0.0.0
/usr/local/bin/rootlesskit (unconfined) 1001 13064 0.1 0.1 1239508 14848 ? Sl 15:24 0:00 | \_ /usr/local/bin/containerd-shim-runc-v2 -namespace moby -id 826113dad78968503e539c3f613a2645e8becf395161bfe5f049100e6929d6cc -address /run/user/1001/docker/containerd/containerd.sock
/usr/local/bin/rootlesskit (unconfined) 1001 13088 0.3 0.1 14016 9636 ? Ss 15:24 0:00 | \_ python -c import datetime, time; print(f'Start time: {datetime.datetime.now()}'); time.sleep(60); print(f'End time: {datetime.datetime.now()}')
unconfined 1001 7701 0.0 0.0 4368 3072 ? Ss 15:22 0:00 \_ /bin/bash /home/runner/run.sh
unconfined 1001 7763 0.0 0.0 4368 3200 ? S 15:22 0:00 \_ /bin/bash /home/runner/run-helper.sh
unconfined 1001 7771 2.2 1.5 274689324 115816 ? Sl 15:22 0:02 \_ /home/runner/bin/Runner.Listener run
unconfined 1001 11714 10.7 1.5 274251452 116516 ? Sl 15:24 0:02 \_ /home/runner/bin/Runner.Worker spawnclient 162 165
unconfined 1001 13046 0.0 0.0 4364 3200 ? S 15:24 0:00 \_ /usr/bin/bash -e /home/runner/_work/_temp/a2018b17-5057-435e-acd1-12d16c29da49.sh
unconfined 1001 13047 0.0 0.3 1255076 24576 ? Sl 15:24 0:00 \_ docker run --rm my-image-name:latest
values.yamlの修正だけでなんとかできないか試す
書かれているルールが flags=(unconfined) なので、Kubernetesの機能でunconfinedとしても同じだと考えて、values.yamlを修正して起動してみる。
...
containers:
- name: dind
image: docker:dind-rootless
args:
- dockerd
- --host=unix:///run/user/1001/docker.sock
securityContext:
privileged: true
runAsUser: 1001
runAsGroup: 1001
appArmorProfile:
type: Unconfined
結果は何も指定しなかった時と同じエラーで起動せず。そもそも元から profile="unconfined" と書いてあったからよく考えればそれはそうか。
[ 88.451629] audit: type=1400 audit(1762098204.253:327): apparmor="AUDIT" operation="userns_create" class="namespace" info="Userns create - transitioning profile" profile="unconfined" pid=7809 comm="rootlesskit" requested="userns_create" target="unprivileged_userns"
[ 88.452149] audit: type=1400 audit(1762098204.254:329): apparmor="DENIED" operation="capable" class="cap" profile="unprivileged_userns" pid=7920 comm="rootlesskit" capability=21 capname="sys_admin"
AppArmorのプロファイルを作ってコンテナ実行時に指定する
実行パスで制約を緩めると、悪意のあるコンテナが /usr/local/bin/rootlesskit に良くないファイルを置いて実行すればAppArmorを回避できてしまうのでは？と思い、新規のAppArmorプロファイルを作成し、Kubernetesの機能を使ってdindのコンテナに割り当ててみた。
cat << EOF | sudo tee "/etc/apparmor.d/k8s-dind-rootless"
# Special profile transitioned to by unconfined when creating an unprivileged
# user namespace.
#
abi <abi/4.0>,
include <tunables/global>
profile k8s-dind-rootless flags=(default_allow) {
userns,
# Site-specific additions and overrides. See local/README for details.
include if exists <local/k8s-dind-rootless>
}
EOF
sudo apparmor_parser -r /etc/apparmor.d/k8s-dind-rootless
values.yamlも少し修正してdindのコンテナに今作ったprofileを割り当てる。
...
containers:
- name: dind
image: docker:dind-rootless
args:
- dockerd
- --host=unix:///run/user/1001/docker.sock
securityContext:
privileged: true
runAsUser: 1001
runAsGroup: 1001
appArmorProfile:
type: Localhost
localhostProfile: k8s-dind-rootless
パス指定して起動した時とLABELの見た目は同じ。こっちの方が安全かな？
LABEL USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
unconfined root 11485 0.3 0.2 1239408 16832 ? Sl 15:10 0:00 /snap/microk8s/8491/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b09948da36086926aef8f6aa087e4da65ed117d49be353ffc3857f63787f3c6 -address /var/snap/microk8s/common/run/containerd.sock
unconfined 65535 11508 0.0 0.0 1020 640 ? Ss 15:10 0:00 \_ /pause
k8s-dind-rootless (unconfined) 1001 11851 0.0 0.1 1233988 9984 ? Ssl 15:10 0:00 \_ rootlesskit --net=vpnkit --mtu=1500 --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run docker-init -- dockerd --host=unix:///run/user/1001/docker.sock
k8s-dind-rootless (unconfined) 1001 11900 3.9 0.1 1234884 14216 ? Sl 15:10 0:01 | \_ /proc/self/exe --net=vpnkit --mtu=1500 --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run docker-init -- dockerd --host=unix:///run/user/1001/docker.sock
k8s-dind-rootless (unconfined) 1001 11931 0.0 0.0 1008 384 ? S 15:10 0:00 | | \_ docker-init -- dockerd --host=unix:///run/user/1001/docker.sock
k8s-dind-rootless (unconfined) 1001 11936 6.9 0.9 2403184 81508 ? Sl 15:10 0:03 | | \_ dockerd --host=unix:///run/user/1001/docker.sock
k8s-dind-rootless (unconfined) 1001 11945 0.1 0.4 1268360 36992 ? Ssl 15:10 0:00 | | \_ containerd --config /run/user/1001/docker/containerd/containerd.toml
k8s-dind-rootless (unconfined) 1001 11916 6.1 0.6 121240 51792 ? Sl 15:10 0:02 | \_ vpnkit --ethernet /tmp/rootlesskit3182120023/vpnkit-ethernet.sock --mtu 1500 --host-ip 0.0.0.0
k8s-dind-rootless (unconfined) 1001 15300 0.0 0.1 1239508 14720 ? Sl 15:10 0:00 | \_ /usr/local/bin/containerd-shim-runc-v2 -namespace moby -id e9d4922390298c0b940cbe727bf8dfe4e0b6273d855d555a802dfe63c6712310 -address /run/user/1001/docker/containerd/containerd.sock
k8s-dind-rootless (unconfined) 1001 15323 0.1 0.1 14016 9584 ? Ss 15:10 0:00 | \_ python -c import datetime, time; print(f'Start time: {datetime.datetime.now()}'); time.sleep(60); print(f'End time: {datetime.datetime.now()}')
unconfined 1001 12143 0.0 0.0 4368 3200 ? Ss 15:10 0:00 \_ /bin/bash /home/runner/run.sh
unconfined 1001 12162 0.0 0.0 4368 3200 ? S 15:10 0:00 \_ /bin/bash /home/runner/run-helper.sh
unconfined 1001 12167 5.2 1.2 274392164 101648 ? Sl 15:10 0:02 \_ /home/runner/bin/Runner.Listener run
unconfined 1001 12761 8.1 1.4 274177960 118564 ? Sl 15:10 0:03 \_ /home/runner/bin/Runner.Worker spawnclient 149 152
unconfined 1001 15277 0.0 0.0 4364 3072 ? S 15:10 0:00 \_ /usr/bin/bash -e /home/runner/_work/_temp/33b789e9-c4c4-480c-b8ce-69232597fe54.sh
unconfined 1001 15282 0.0 0.2 1255356 24576 ? Sl 15:10 0:00 \_ docker run --rm my-image-name:latest
AppArmorにもセキュリティにも詳しくないため、これが正しい方法なのかは分かりません。商用環境や重要な環境で試す場合は専門家に聞いてください。