Kubernetes搭建spinnaker服务

背景：

2017-2018年左右的吧，不记得看什么了看到了spinnaker，但是当时真的安装不起来。各种被墙裂。2020年底学习了泽阳大佬的spinnaker实践课程。通过Halyard方式搭建了spinnaker的集群，并与jenkins gitlab harbor k8s完成了集成。2021年初稍微玩了一下，就去整别的事情去了，没有能应用于线上环境。下半年了，jenkins k8s这些的流程现在基本都是清晰了。想把cd从jenkins中剥离出来教给spinnaker了，就重新温习一下spinnaker吧！

关于spinnaker

spinnaker是Netfix公司开源的一款持续部署工具，采用java语言编写，遵循微服务的设计思想，目标是为团队提供灵活的持续部署流水线并提供软件的部署效率

spinnaker的优势

• 支持多云部署
• 自动发布
• 内置部署最佳实践

spinnaker架构

关于spinnaker的架构说明

• deck-基于浏览器的 UI
• gate 微服务api网关，Spinnaker UI 和所有 api 调用者通过 Gate 与 Spinnaker 通信
• orca 流水线阶段编排引擎。它处理所有临时操作和管道。阅读有关 Orca 服务概述的更多信息
• clouddriver 负责对云提供商的所有变异调用以及索引/缓存所有部署的资源。
• front50 用于持久化应用程序、管道、项目和通知的元数据
• rosco 为各种云提供商生成不可变的 VM 映像（或映像模板）

  它用于生成机器映像（例如 GCE 映像 、 AWS AMI 、 Azure VM 映像 ）。它目前包装了 packer ，但将 被扩展以支持用于生成图像的其他机制。

• igor 用于通过 Jenkins 和 Travis CI 等系统中的持续集成作业触发管道，它允许在管道中使用 Jenkins/Travis 阶段
• echo 事件总线 它支持发送通知（例如 Slack、电子邮件、SMS），并对来自 Github 等服务的传入 webhook 采取行动。
• fiat 认证授权中心 它用于查询用户对帐户、应用程序和服务帐户的访问权限
• kayenta 自动金丝雀分析
• Keel 为管理交付提供动力

  注：这个还没有用过

• halyard 配置服务 管理上述每项服务的生命周期。它仅在 Spinnaker 启动、更新和回滚期间与这些服务交互。

服务依赖调用关系：

重要的事情： 这些东西去看官方文档很是详细，比其他的比较详细多了：https://spinnaker.io/docs/reference/architecture/microservices-overview/

Kubernetes搭建spinnaker服务

基本环境

腾讯云同一vpc内服务器，内网互通，ip为内网地址

注：个人尝试containerd运行halyard未能成功，最终使用docker方式运行halyard

基于docker runtime方式部署halyard的方式部署spinnaker

注： 关于halyard的操作都在k8s-01节点操作。另外声明一下k8s-01原主机名为k8s-02使用了hostnamectl set-hostname修改主机名。有些截图或者命令都依然为k8-02，实际为同一个台服务器。xshell早些时候打开10.0.2.17的窗口…

下载镜像，挂载本地配置文件目录，并启动容器

[root@k8s-01 ~]# docker pull registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 创建.hall文件夹后面持久化存储spinnaker生成文件 [root@k8s-01 ~]# mkdir -p /home/spinnaker/.hal 创建.kube文件夹并将集群中的config文件上传到此目录 [root@k8s-01 ~]# mkdir -p /home/spinnaker/.kube [root@k8s-01 ~]# ls /home/spinnaker/.kube config 启动halyard容器 [root@k8s-01 ~]# docker run -itd --name halyard -v /home/spinnaker/.hal:/home/spinnaker/.hal -v /home/spinnaker/.kube:/home/spinnaker/.kube registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 

特权身份进入容器关闭gcs

 以root身份进入容器，修改配置文件 [root@k8s-01 .kube]# docker exec -it -u root halyard bash bash-5.0# 

 修改spinnaker.config.input.gcs.enabled = false 。 vi /opt/halyard/config/halyard.yml spinnaker: artifacts: debian: https://dl.bintray.com/spinnaker-releases/debians docker: gcr.io/spinnaker-marketplace config: input: gcs: enabled: false writerEnabled: false bucket: halconfig 

重新启动halyard容器

 需要重启容器（如果此命令未重启，则需要退出容器然后 docker restart halyard） bash-5.0# hal shutdown Halyard Daemon Response: Shutting down, bye... 重启容器 [root@k8s-01 .kube]# docker start halyard halyard 

上传boms文件到服务器

通过rz命令上传制品库到运行halyard的服务器，并解压压缩包 [root@k8s-01 work]# ls 1.26.6-Install-Scripts.zip [root@k8s-01 work]# unzip 1.26.6-Install-Scripts.zip 

嗯看到了这个.boms的文件夹，将其copy到/home/spinnaker/.hal/目录下！

[root@k8s-01 1.26.6]# ls .boms/ bom clouddriver deck echo fiat front50 gate igor kayenta monitoring-daemon orca rosco [root@k8s-01 1.26.6]# cp -Ra .boms/ /home/spinnaker/.hal/ [root@k8s-01 1.26.6]# ls /home/spinnaker/.hal/.boms/ bom clouddriver deck echo fiat front50 gate igor kayenta monitoring-daemon orca rosco 

关于镜像的下载

镜像下载泽阳大佬的制品库下载中有下载镜像的脚本：

#!/bin/bash S_REGISTRY="gcr.io/spinnaker-marketplace" #T_REGISTRY="registry.cn-beijing.aliyuncs.com/spinnaker-cd" T_REGISTRY="docker.io/spinnakercd" NODES="node01.zy.com node02.zy.com" 下载镜像 function GetImages(){ echo -e "\033[43;34m =====GetImg===== \033[0m" IMAGES=$( cat tagfile.txt) for image in ${IMAGES} do for node in ${NODES} do echo -e "\033[32m ${node} ---> pull ---> ${image} \033[0m" ssh ${node} "docker pull ${T_REGISTRY}/${image}" echo -e "\033[32m ${node} ---> tag ---> ${image} \033[0m" ssh ${node} "docker tag ${T_REGISTRY}/${image} ${S_REGISTRY}/${image}" done done for node in ${NODES} do echo -e "\033[43;34m =====${node}===镜像信息===== \033[0m" ssh ${node} "docker images | grep 'spinnaker-marketplace' " done } GetImages 

But 我的集群的运行时是containerd。ctr crictl两个命令的区别有必要重新温习一下。crictl也没法修改标签啊？

#!/bin/bash S_REGISTRY="gcr.io/spinnaker-marketplace" #T_REGISTRY="registry.cn-beijing.aliyuncs.com/spinnaker-cd" T_REGISTRY="docker.io/spinnakercd" NODES="10.0.4.18 10.0.4.49 10.0.4.48 10.0.4.23 10.0.4.47 10.0.4.32" 下载镜像 function GetImages(){ echo -e "\033[43;34m =====GetImg===== \033[0m" IMAGES=$( cat tagfile.txt) for image in ${IMAGES} do for node in ${NODES} do echo -e "\033[32m ${node} ---> pull ---> ${image} \033[0m" ssh -p 36000 ${node} "crictl pull ${T_REGISTRY}/${image}" echo -e "\033[32m ${node} ---> tag ---> ${image} \033[0m" ssh -p 36000 ${node} "crictl images ${T_REGISTRY}/${image} ${S_REGISTRY}/${image}" done done for node in ${NODES} do echo -e "\033[43;34m =====${node}===镜像信息===== \033[0m" ssh -p 36000 ${node} "crictl images ls| grep 'spinnaker-marketplace' " done } GetImages 

[root@k8s-2 .hal]# mkdir -p /home/spinnaker/.hal/default/service-settings [root@k8s-2 .hal]# cd /home/spinnaker/.hal/default/service-settings [root@k8s-2 service-settings]# pwd /home/spinnaker/.hal/default/service-settings [root@k8s-2 service-settings]# ls clouddriver.yml deck.yml echo.yml fiat.yml front50.yml gate.yml igor.yml kayenta.yml orca.yml rosco.yml [root@k8s-2 service-settings]# cat * artifactId: docker.io/spinnakercd/clouddriver:8.0.4-028 artifactId: docker.io/spinnakercd/deck:3.7.2-020 artifactId: docker.io/spinnakercd/echo:2.17.1-836 artifactId: docker.io/spinnakercd/fiat:1.16.0-020 artifactId: docker.io/spinnakercd/front50:0.27.1-956 artifactId: docker.io/spinnakercd/gate:1.22.1-019 artifactId: docker.io/spinnakercd/igor:1.16.0-020 artifactId: docker.io/spinnakercd/kayenta:0.21.0-019 artifactId: docker.io/spinnakercd/orca:2.20.3-216 artifactId: docker.io/spinnakercd/rosco:0.25.0-020 

就不修改标签直接使用泽阳大佬docker的镜像仓库里面的镜像了免去下载镜像修改标签的步骤

Halyard配置管理

注： halyard的配置都在k8s-01节点执行默认在halyard容器内

设置Spinnaker版本，–version 指定版本

[root@k8s-01 .kube]# docker exec -it -u root halyard bash bash-5.0$ hal config version edit --version local:1.26.6 + Get current deployment Success - Edit Spinnaker version Failure Validation in Global: ! ERROR Failure writing your halconfig to path "/home/spinnaker/.hal/config": /home/spinnaker/.hal/config - Failed to update version. 

嗯强调一下 .hall目录要有读写权限啊

[root@k8s-01 1.26.6]# chmod 777 -R /home/spinnaker/.hal/ [root@k8s-01 1.26.6]# 

继续指定spinnaker版本并生成配置文件

bash-5.0$ hal config version edit --version local:1.26.6 + Get current deployment Success + Edit Spinnaker version Success + Spinnaker has been configured to update/install version "local:1.26.6". Deploy this version of Spinnaker with `hal deploy apply`. bash-5.0$ ls config default bash-5.0$ cat config currentDeployment: default deploymentConfigurations: - name: default version: local:1.26.6 providers: appengine: enabled: false accounts: [] aws: enabled: false accounts: [] bakeryDefaults: baseImages: [] defaultKeyPairTemplate: '{ 
  {name}}-keypair' defaultRegions: - name: us-west-2 defaults: iamRole: BaseIAMRole ecs: enabled: false accounts: [] azure: enabled: false accounts: [] bakeryDefaults: templateFile: azure-linux.json baseImages: [] dcos: enabled: false accounts: [] clusters: [] dockerRegistry: enabled: false accounts: [] google: enabled: false accounts: [] bakeryDefaults: templateFile: gce.json baseImages: [] zone: us-central1-f network: default useInternalIp: false huaweicloud: enabled: false accounts: [] bakeryDefaults: baseImages: [] kubernetes: enabled: false accounts: [] tencentcloud: enabled: false accounts: [] bakeryDefaults: baseImages: [] oracle: enabled: false accounts: [] bakeryDefaults: templateFile: oci.json baseImages: [] cloudfoundry: enabled: false accounts: [] deploymentEnvironment: size: SMALL type: LocalDebian imageVariant: SLIM updateVersions: true consul: enabled: false vault: enabled: false customSizing: {} sidecars: {} initContainers: {} hostAliases: {} affinity: {} tolerations: {} nodeSelectors: {} gitConfig: upstreamUser: spinnaker livenessProbeConfig: enabled: false haServices: clouddriver: enabled: false disableClouddriverRoDeck: false echo: enabled: false persistentStorage: azs: {} gcs: rootFolder: front50 redis: {} s3: rootFolder: front50 oracle: {} features: auth: false fiat: false chaos: false entityTags: false metricStores: datadog: enabled: false tags: [] prometheus: enabled: false add_source_metalabels: true stackdriver: enabled: false newrelic: enabled: false tags: [] period: 30 enabled: false notifications: slack: enabled: false twilio: enabled: false baseUrl: https://api.twilio.com/ github-status: enabled: false timezone: America/Los_Angeles ci: jenkins: enabled: false masters: [] travis: enabled: false masters: [] wercker: enabled: false masters: [] concourse: enabled: false masters: [] gcb: enabled: false accounts: [] codebuild: enabled: false accounts: [] repository: artifactory: enabled: false searches: [] security: apiSecurity: ssl: enabled: false uiSecurity: ssl: enabled: false authn: oauth2: enabled: false client: {} resource: {} userInfoMapping: {} saml: enabled: false userAttributeMapping: {} ldap: enabled: false x509: enabled: false iap: enabled: false enabled: false authz: groupMembership: service: EXTERNAL google: roleProviderType: GOOGLE github: roleProviderType: GITHUB file: roleProviderType: FILE ldap: roleProviderType: LDAP enabled: false artifacts: bitbucket: enabled: false accounts: [] gcs: enabled: false accounts: [] oracle: enabled: false accounts: [] github: enabled: false accounts: [] gitlab: enabled: false accounts: [] gitrepo: enabled: false accounts: [] http: enabled: false accounts: [] helm: enabled: false accounts: [] s3: enabled: false accounts: [] maven: enabled: false accounts: [] templates: [] pubsub: enabled: false google: enabled: false pubsubType: GOOGLE subscriptions: [] publishers: [] canary: enabled: false serviceIntegrations: - name: google enabled: false accounts: [] gcsEnabled: false stackdriverEnabled: false - name: prometheus enabled: false accounts: [] - name: datadog enabled: false accounts: [] - name: signalfx enabled: false accounts: [] - name: aws enabled: false accounts: [] s3Enabled: false - name: newrelic enabled: false accounts: [] reduxLoggerEnabled: true defaultJudge: NetflixACAJudge-v1.0 stagesEnabled: true templatesEnabled: true showAllConfigsEnabled: true spinnaker: extensibility: plugins: {} repositories: {} webhook: trust: enabled: false stats: enabled: true endpoint: https://stats.spinnaker.io instanceId: 01FKDR1B3P8PF35RRC93XTE9AS deploymentMethod: {} connectionTimeoutMillis: 3000 readTimeoutMillis: 5000 bash-5.0$ 

设置时区

# 设置时区 hal config edit --timezone Asia/Shanghai 

S3–no-validate

# 设置存储为s3(后面不用，但是必须配置bug) hal config storage edit --type s3 --no-validate 

访问方式,设置deck与gate的域名

# 访问方式：设置deck与gate的域名 hal config security ui edit --override-base-url http://spinnaker.xxxx.com hal config security api edit --override-base-url http://spin-gate.xxxx.com 

来对比一下执行以上命令后config文件的变化：


做这些对比是为了方便以后自己手动更改配置文件。大佬的可以忽略这些截图步骤。

添加镜像仓库（harbor）和k8s集群账户

开启镜像仓库配置并添加account

bash-5.0$ hal config provider docker-registry enable --no-validate + Get current deployment Success + Edit the dockerRegistry provider Success + Successfully enabled dockerRegistry bash-5.0$ hal config provider docker-registry account add my-harbor-registry \ > --address https://harbor.xxxx.com \ > --username xxxx \ > --password xxxx + Get current deployment Success + Add the my-harbor-registry account Success Validation in default.provider.dockerRegistry.my-harbor-registry: - WARNING Your docker registry has no repositories specified, and the registry's catalog is empty. Spinnaker will not be able to deploy any images until some are pushed to this registry. ? Manually specify some repositories for this docker registry to index. + Successfully added account my-harbor-registry for provider dockerRegistry. 

开启kubernetes配置并添加account

bash-5.0$ hal config provider kubernetes enable + Get current deployment Success + Edit the kubernetes provider Success Validation in default.provider.kubernetes: - WARNING Provider kubernetes is enabled, but no accounts have been configured. + Successfully enabled kubernetes bash-5.0$ hal config provider kubernetes account add default \ > --docker-registries my-harbor-registry \ > --context $(kubectl config current-context) \ > --service-account true \ > --omit-namespaces=kube-system,kube-public \ > --provider-version v2 \ > --no-validate + Get current deployment Success + Add the default account Success + Successfully added account default for provider kubernetes. 

再瞄一眼配置文件config:

指定部署使用account和命名空间，部署方式distributed（分布式）

​

bash-5.0$ hal config deploy edit \ > --account-name default \ > --type distributed \ > --location spinnaker 

看了一眼配置文件应该对应的是deploymentEnvironment下面的配置：

开启一些主要的功能（后期可以再追加）

bash-5.0$ hal config features edit --pipeline-templates true bash-5.0$ hal config features edit --artifacts true bash-5.0$ hal config features edit --managed-pipeline-templates-v2-ui true 

配置与jenkins CI集成

```shell # 配置Jenkins hal config ci jenkins enable JenkinsServer 需要用到账号和密码 hal config ci jenkins master add my-jenkins-master-01 \ --address https://jenkins.xxxx.com \ --username zhangpeng \ --password xxxx 启用csrf hal config ci jenkins master edit my-jenkins-master-01 --csrf true 

配置GitHub/GitLab集成

github的是泽阳大佬的。我这里就只集成了gitlab。github仅供参考在配置文件中也生成一下。方便对比配置文件。token的生成就不用做过多的赘述了！

```shell # GitHub 参考：https://spinnaker.io/setup/artifacts/github/ 创建token https://github.com/settings/tokens hal config artifact github enable hal config artifact github account add my-github-account \ --token xxxxxxxxxxxxxxxxxxxxxxx \ --username zeyangli # GitLab https://spinnaker.io/setup/artifacts/gitlab/ 创建一个个人的token（admin） hal config artifact gitlab enable hal config artifact gitlab account add my-gitlab-account \ --token xxxxxxxxxxxxxx 

artifacts下找到相关配置

使用外部redis集群

关于redis我是使用的腾讯云的云redis。正常该搞一个密码的。但是没有去仔细看下官方文档，就直接使用了免密的方式！

 ```shell service-settings bash-5.0$ pwd /home/spinnaker/.hal/default/service-settings vi .hal/default/service-settings/redis.yml overrideBaseUrl: redis://10.0.0.31:6379 skipLifeCycleManagement: true profiles /home/spinnaker/.hal/default/profiless bash-5.0$ pwd /home/spinnaker/.hal/default bash-5.0$ mkdir /home/spinnaker/.hal/default/profiles bash-5.0$ cd profiles/ bash-5.0$ vi gate-local.yml redis: configuration: secure: true 

使用SQL数据库

Clouddriver服务

创建数据库：

CREATE DATABASE `clouddriver` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, EXECUTE, SHOW VIEW ON `clouddriver`.* TO 'clouddriver_service'@'%' IDENTIFIED BY ''; GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, LOCK TABLES, EXECUTE, SHOW VIEW ON `clouddriver`.* TO 'clouddriver_migrate'@'%' IDENTIFIED BY ''; 

修改配置文件：

bash-5.0$ pwd /home/spinnaker/.hal/default/profiles bash-5.0$ vi clouddriver-local.yml sql: enabled: true # read-only boolean toggles `SELECT` or `DELETE` health checks for all pools. # Especially relevant for clouddriver-ro and clouddriver-ro-deck which can # target a SQL read replica in their default pools. read-only: false taskRepository: enabled: true cache: enabled: true # These parameters were determined to be optimal via benchmark comparisons # in the Netflix production environment with Aurora. Setting these too low # or high may negatively impact performance. These values may be sub-optimal # in some environments. readBatchSize: 500 writeBatchSize: 300 scheduler: enabled: true # Enable clouddriver-caching's clean up agent to periodically purge old # clusters and accounts. Set to true when using the Kubernetes provider. unknown-agent-cleanup-agent: enabled: false connectionPools: default: # additional connection pool parameters are available here, # for more detail and to view defaults, see: # https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt default: true jdbcUrl: jdbc:mysql://10.0.4.22:3306/clouddriver user: clouddriver_service password:  # The following tasks connection pool is optional. At Netflix, clouddriver # instances pointed to Aurora read replicas have a tasks pool pointed at the # master. Instances where the default pool is pointed to the master omit a # separate tasks pool. tasks: user: clouddriver_service jdbcUrl: jdbc:mysql://10.0.4.22:3306/clouddriver password:  migration: user: clouddriver_migrate jdbcUrl: jdbc:mysql://10.0.4.22:3306/clouddriver password:  redis: enabled: false cache: enabled: false scheduler: enabled: false taskRepository: enabled: false 

Front50服务

创建数据库

CREATE DATABASE `front50` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, EXECUTE, SHOW VIEW ON `front50`.* TO 'front50_service'@'%' IDENTIFIED BY ""; GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, LOCK TABLES, EXECUTE, SHOW VIEW ON `front50`.* TO 'front50_migrate'@'%' IDENTIFIED BY ""; 

修改配置文件

bash-5.0$ pwd /home/spinnaker/.hal/default/profiles bash-5.0$ vi front50-local.yml spinnaker: s3: enabled: false sql: enabled: true connectionPools: default: # additional connection pool parameters are available here, # for more detail and to view defaults, see: # https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt default: true jdbcUrl: jdbc:mysql://10.0.4.22:3306/front50 user: front50_service password:  migration: user: front50_migrate jdbcUrl: jdbc:mysql://10.0.4.22:3306/front50 password:  

Orca服务

创建数据库

set tx_isolation = 'REPEATABLE-READ'; CREATE SCHEMA `orca` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, EXECUTE, SHOW VIEW ON `orca`.* TO 'orca_service'@'%' IDENTIFIED BY "" ; GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, LOCK TABLES, EXECUTE, SHOW VIEW ON `orca`.* TO 'orca_migrate'@'%' IDENTIFIED BY "" ; 

修改配置文件

bash-5.0$ pwd /home/spinnaker/.hal/default/profiles bash-5.0$ vi front50-local.yml bash-5.0$ pwd /home/spinnaker/.hal/default/profiles bash-5.0$ vi orca-local.yml tasks: useManagedServiceAccounts: true sql: enabled: true connectionPool: jdbcUrl: jdbc:mysql://10.0.4.22:3306/orca user: orca_service password:  connectionTimeout: 5000 maxLifetime: 30000 # MariaDB-specific: maxPoolSize: 50 migration: jdbcUrl: jdbc:mysql://10.0.4.22:3306/orca user: orca_migrate password:  # Ensure we're only using SQL for accessing execution state executionRepository: sql: enabled: true redis: enabled: false # Reporting on active execution metrics will be handled by SQL monitor: activeExecutions: redis: false # Use SQL for Orca's work queue # Settings from Netflix and may require adjustment for your environment # Only validated with AWS Aurora MySQL 5.7 # Please PR if you have success with other databases keiko: queue: sql: enabled: true redis: enabled: false queue: zombieCheck: enabled: true pendingExecutionService: sql: enabled: true redis: enabled: false 

部署服务

bash-5.0$ hal deploy apply --no-validate 

创建Ingress访问web测试

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: spinnaker-service namespace: spinnaker annotations: kubernetes.io/ingress.class: traefik traefik.ingress.kubernetes.io/router.entrypoints: web spec: rules: - host: spinnaker.xxxx.com http: paths: - pathType: Prefix path: / backend: service: name: spin-deck port: number: 9000 - host: spin-gate.xxxx.com http: paths: - pathType: Prefix path: / backend: service: name: spin-gate port: number: 8084 

通过web浏览器访问https://spinnaker.xxxx.com/ 如下：

注：至于为什么访问https呢？因为我的代理是traefik slb上面做了跳转。当然了这里应该根据自己实际的环境出发！

集成ldap:

hal config security authn ldap edit \ --user-search-base 'ou=devops,dc=zy,dc=com' \ --url 'ldap://192.168.1.200:389' \ --user-search-filter 'cn={0}' \ --manager-dn 'cn=admin,dc=zy,dc=com' \ --manager-password '' hal config security authn ldap enable 

bash-5.0$ cd /home/spinnaker/.hal/ bash-5.0$ pwd /home/spinnaker/.hal bash-5.0$ cat config 

web访问如下：怀疑我traefik 强跳搞的

bash-5.0$ hal deploy apply --no-validate 

[root@k8s-master-01 ~]# kubectl get pods -n spinnaker 

等待pod起来


进入首页

关于授权

首先登陆ldap web管理页面两个用户组 groupOfUniqueNames yunwenzu devops两个组,根据ldap中组进行授权。

ldap创建用户组与用户

yunweizu-用户zhangpeng

将zhangpeng用户添加到组中：

devop用户组-用户huozhonghao

halyard中配置：

开启ldap security 配置。并增加相关配置：

hal config security authz ldap edit \ --url 'ldap://172.19.252.28:389/dc=xxxx,dc=com' \ --manager-dn 'cn=admin,dc=xxxx,dc=com' \ --manager-password 'xxxxxx' \ --user-dn-pattern 'cn={0}' \ --group-search-base 'ou=devops' \ --group-search-filter 'uniqueMember={0}' \ --group-role-attributes 'cn' \ --user-search-filter 'cn={0}' hal config security authz edit --type ldap hal config security authz enable 

设置那些用户可以访问集群账户、镜像仓库、应用程序

 配置yunweizu和group02角色的用户可以使用default这个集群账户 hal config provider kubernetes account edit default \ --add-read-permission yunweizu,group02 \ --add-write-permission yunweizu 配置yunweizu角色的用户可以使用my-harbor-registry账户 hal config provider docker-registry account edit my-harbor-registry \ --read-permissions yunweizu \ --write-permissions yunweizu 更新部署 hal deploy apply 

登陆spinnaker web尝试：

就先只看到这里的权限，警告提示告诉你read会所有用户锁定在此应用程序之外。
具体的权限是跟ldap绑定的那么应该是这样的：
1.在ldap管理页面中， 将用户zhangpeng加入devops组

2.spinnaker登陆zhangpeng用户新建一个应用，yunweizu 读写可执行，devops组仅仅可读。
​

3. 创建一个新的用户组platform将huozhonghao用户加入

4. spinnaker web登陆huozhonghao用户

嗯 这里也可以看到platform组了 修改一下权限试试，删除一下devops的试试：

增加platform组权限也是失败因为只有read权限，没有writer权限

开启管道权限

halyard容器中操作：

bash-5.0$ pwd /home/spinnaker/.hal/default/profiles bash-5.0$ cat /home/spinnaker/.hal/default/profiles/orca-local.yml tasks: useManagedServiceAccounts: true bash-5.0$ cat ~/.hal/default/profiles/settings-local.js window.spinnakerSettings.feature.managedServiceAccounts = true; bash-5.0$ hal deploy apply --no-validate 

注意：orca-local.yml中的开启。我其实在orca服务中早配置上了！

权限的一些测试

bash-5.0$ hal deploy apply --no-validate 

[root@k8s-master-01 develop]# kubectl get pods -n spinnaker 

[root@k8s-master-01 develop]#kubectl get svc -n spinnaker [root@k8s-master-01 develop]# curl -X POST http://172.19.254.33:7003/roles/sync [root@k8s-master-01 develop]#curl 172.19.254.33:7003/authorize/huozhonghao 

read权限依然无法看到accout!

kubernetes default account 添加devops组writer权限：

bash-5.0$ vi config bash-5.0$ hal deploy apply --no-validate 

一些失败的尝试（还是没有成功）

1. 下载Halyard 镜像并启动容器—ctr各种命令的复习

ctr pull

 [root@k8s-master-01 ~]# ctr image pull registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 [root@k8s-master-01 ~]# mkdir /root/.hal 

参考一下docker时代的启动方式：

docker run -itd --name halyard \ -v /root/.hal:/home/spinnaker/.hal \ -v /root/.kube:/home/spinnaker/.kube \ registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 

ctr run

依着葫芦画瓢一下？

ctr run -itd --name halyard \ -v /root/.hal:/home/spinnaker/.hal \ -v /root/.kube:/home/spinnaker/.kube \ registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 

ctr create

 [root@k8s-master-01 1.26.6]# ctr c create registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 halyard --mount type=bind,src=/root/.hal,dst=/home/spinnaker/.hal,options=rbind:row --mount type=bind,src=/root/.kube,dst=/home/spinnaker/.kube,options=rbind:ro --mount type=bind,src=/opt/halyard/config/,dst=/opt/halyard/config/,options=rbind:rw [root@k8s-master-01 1.26.6]# ctr c ls CONTAINER IMAGE RUNTIME halyard registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 io.containerd.runc.v2 

ctr t start

[root@k8s-master-01 1.26.6]# ctr t start -d halyard [root@k8s-master-01 1.26.6]# ctr t ls TASK PID STATUS halyard  RUNNING 

现在问题来了 如何进入容器呢？

ctr tasks exec -t –exec-id

[root@k8s-master-01 1.26.6]# ctr tasks list TASK PID STATUS halyard  RUNNING [root@k8s-master-01 1.26.6]# ctr tasks exec -t --exec-id  halyard sh / $ 

ctr c rm ctr c kill—-读写权限没有搞明白 只能采用挂载本地文件的方式重新搞一波了

[root@k8s-master-01 1.26.6]# ctr t ls TASK PID STATUS halyard  RUNNING [root@k8s-master-01 1.26.6]# ctr t kill halyard [root@k8s-master-01 1.26.6]# ctr t ls TASK PID STATUS halyard  STOPPED [root@k8s-master-01 1.26.6]# ctr t ls TASK PID STATUS halyard  STOPPED [root@k8s-master-01 1.26.6]# ctr c rm halyard [root@k8s-master-01 1.26.6]# ctr t ls TASK PID STATUS 

[root@k8s-master-01 1.26.6]# ctr c create registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 halyard --mount type=bind,src=/root/.hal,dst=/home/spinnaker/.hal,options=rbind:row --mount type=bind,src=/root/.kube,dst=/home/spinnaker/.kube,options=rbind:ro --mount type=bind,src=/opt/halyard/config/,dst=/opt/halyard/config/,options=rbind:rw [root@k8s-master-01 1.26.6]# ctr c ls CONTAINER IMAGE RUNTIME halyard registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 io.containerd.runc.v2 [root@k8s-master-01 1.26.6] # ctr t start -d halyard [root@k8s-master-01 1.26.6] # ctr t ls TASK PID STATUS halyard  RUNNING [root@k8s-master-01 1.26.6] # ctr tasks exec -t --exec-id  halyard sh 

下载镜像的尝试：

小伙伴们觉得下载镜像应该用下面哪个脚本？用ctr or crictl呢？最终使用镜像的是要kubernetes…应该是用crictl的。 ctr搞了kubernetes集群应用是发现不了镜像的！

#!/bin/bash S_REGISTRY="gcr.io/spinnaker-marketplace" #T_REGISTRY="registry.cn-beijing.aliyuncs.com/spinnaker-cd" T_REGISTRY="docker.io/spinnakercd" NODES="10.0.4.18 10.0.4.49 10.0.4.48 10.0.4.23 10.0.4.47 10.0.4.32" 下载镜像 function GetImages(){ echo -e "\033[43;34m =====GetImg===== \033[0m" IMAGES=$( cat tagfile.txt) for image in ${IMAGES} do for node in ${NODES} do echo -e "\033[32m ${node} ---> pull ---> ${image} \033[0m" ssh -p 36000 ${node} "crictl pull ${T_REGISTRY}/${image}" echo -e "\033[32m ${node} ---> tag ---> ${image} \033[0m" ssh -p 36000 ${node} "ctr image tag ${T_REGISTRY}/${image} ${S_REGISTRY}/${image}" done done for node in ${NODES} do echo -e "\033[43;34m =====${node}===镜像信息===== \033[0m" ssh -p 36000 ${node} "ctr image ls | grep 'spinnaker-marketplace' " done } GetImages 

#!/bin/bash S_REGISTRY="gcr.io/spinnaker-marketplace" #T_REGISTRY="registry.cn-beijing.aliyuncs.com/spinnaker-cd" T_REGISTRY="docker.io/spinnakercd" NODES="10.0.4.18 10.0.4.49 10.0.4.48 10.0.4.23 10.0.4.47 10.0.4.32" 下载镜像 function GetImages(){ echo -e "\033[43;34m =====GetImg===== \033[0m" IMAGES=$( cat tagfile.txt) for image in ${IMAGES} do for node in ${NODES} do echo -e "\033[32m ${node} ---> pull ---> ${image} \033[0m" ssh -p 36000 ${node} "crictl pull ${T_REGISTRY}/${image}" echo -e "\033[32m ${node} ---> tag ---> ${image} \033[0m" ssh -p 36000 ${node} "crictl images ${T_REGISTRY}/${image} ${S_REGISTRY}/${image}" done done for node in ${NODES} do echo -e "\033[43;34m =====${node}===镜像信息===== \033[0m" ssh -p 36000 ${node} "crictl images ls| grep 'spinnaker-marketplace' " done } GetImages 

当然了还有一个问题就是 crictl 可以更改镜像名字吗？貌似是不可以的…然后此方式就失败了。

各种失败的尝试-containerd下：

[root@k8s-master-01 .boms]# ctr c create registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 halyard --mount type=bind,src=/root/.hal,dst=/home/spinnaker/.hal,options=rbind:row --mount type=bind,src=/root/.kube,dst=/home/spinnaker/.kube,options=rbind:ro --mount type=bind,src=/opt/halyard/config/,dst=/opt/halyard/config/,options=rbind:rw [root@k8s-master-01 .boms]# ctr c ls CONTAINER IMAGE RUNTIME halyard registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 io.containerd.runc.v2 [root@k8s-master-01 .boms]# ctr t start -d halyard [root@k8s-master-01 .boms]# ctr t ls TASK PID STATUS halyard  RUNNING [root@k8s-master-01 .boms]# ctr tasks exec -t --exec-id  halyard sh / $ hal config version edit --version local:1.26.6 ~ $ cd /home/spinnaker/.hal/ vi config 

timezone: America/Los_Angeles timezone: Asia/Shanghai 

hal config storage edit --type s3 --no-validate 

hal config security ui edit --override-base-url http://spinnaker.xxxx.com hal config security api edit --override-base-url http://spin-gate.xxxx.com 

这都tmd怎么会事情…要疯了

[root@k8s-master-01 .boms]# ctr t kill --signal 9 halyard [root@k8s-master-01 .boms]# ctr c rm halyard 

[root@k8s-master-01 .boms]# ctr c create registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 halyard --mount type=bind,src=/root/.hal,dst=/home/spinnaker/.hal,options=rbind:row --mount type=bind,src=/root/.kube,dst=/home/spinnaker/.kube,options=rbind:ro --mount type=bind,src=/opt/halyard/config/,dst=/opt/halyard/config/,options=rbind:rw [root@k8s-master-01 .boms]# ctr c ls CONTAINER IMAGE RUNTIME halyard registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 io.containerd.runc.v2 [root@k8s-master-01 .boms]# ctr t start -d halyard [root@k8s-master-01 .boms]# ctr t ls TASK PID STATUS halyard  RUNNING [root@k8s-master-01 .boms]# ctr tasks exec -t --exec-id  halyard sh ~ $ cd /home/spinnaker/.hal/ ~/.hal $ cat config |grep time timezone: Asia/Shanghai ~/.hal $ cat config |grep s3 persistentStoreType: s3 s3: s3: s3Enabled: true ~/.hal $ cat config |grep com baseUrl: https://api.twilio.com/ overrideBaseUrl: http://spin-gate.xxxx.com overrideBaseUrl: http://spinnaker.xxxx.com 

~/.hal $ hal config provider kubernetes enable ~/.hal $ hal config provider kubernetes account add default \ --docker-registries my-harbor-registry \ --context $(kubectl config current-context) \ --service-account true \ --omit-namespaces=kube-system,kube-public \ --provider-version v2 \ --no-validate 

hal config deploy edit \ --account-name default \ --type distributed \ --location spinnaker 

hal config features edit --pipeline-templates true hal config features edit --artifacts true hal config features edit --managed-pipeline-templates-v2-ui true 

尼玛又疯了！。。。。。。。。。。。。。。。。。分隔符吧 我准备全部都修改好了这些文件了

我又开始怀疑了 一下人生：是不是我的服务器资源不够了？因为我这是kubernetes的master节点，然后呢资源只有4核心8g，我找一个资源多的server测试一下？
先copy一下 .kube下的config

[root@k8s-node-01 home]# mkdir -p /home/spinnaker/.hal [root@k8s-node-01 home]# mkdir -p /opt/halyard/config [root@k8s-node-01 home]# mkdir -p /home/spinnaker/.kube [root@k8s-node-01 home]# crictl pull registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 Image is up to date for sha256:8673f1670bcd8349b7d9843eb4fdd2e9f02d5fbe454c500d [root@k8s-node-01 home]# cd /home/spinnaker/.kube [root@k8s-node-01 .kube]# rz [root@k8s-node-01 .kube]# ls config [root@k8s-node-01 .kube]# ctr image pull registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 [root@k8s-node-01 .kube]# ctr c create registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 halyard --mount type=bind,src=/home/spinnaker/.hal,dst=/home/spinnaker/.hal,options=rbind:row --mount type=bind,src=/home/spinnaker/.kube,dst=/home/spinnaker/.kube,options=rbind:ro --mount type=bind,src=/opt/halyard/config/,dst=/opt/halyard/config/,options=rbind:rw 

[root@k8s-node-01 .boms]# pwd /home/spinnaker/.hal/.boms [root@k8s-node-01 .boms]# ls bom clouddriver deck echo fiat front50 gate igor kayenta monitoring-daemon orca rosco [root@k8s-node-01 .boms]# cd /opt/halyard/config/ [root@k8s-node-01 config]# cat halyard.yaml 

[root@k8s-node-01 ~]# ctr t ls TASK PID STATUS [root@k8s-node-01 ~]# ctr t start -d halyard [root@k8s-node-01 ~]# ctr t ls TASK PID STATUS halyard  RUNNING [root@k8s-node-01 ~]# ctr tasks exec -t --exec-id  halyard sh / $ hal config version edit --version local:1.26.6 + Get current deployment Success - Edit Spinnaker version Failure Validation in Global: ! ERROR Failure writing your halconfig to path "/home/spinnaker/.hal/config": /home/spinnaker/.hal/config - Failed to update version. / $ hal config version edit --version local:1.26.6 + Get current deployment Success + Edit Spinnaker version Success + Spinnaker has been configured to update/install version "local:1.26.6". Deploy this version of Spinnaker with `hal deploy apply`. / $ hal config edit --timezone Asia/Shanghai 又tmd sb了 不知道怎么回事不试了。直接改好配置文件直接启动了！ 

总结以上失败 执行啥也不行…最后决定直接把docker环境面config文件以及其他制品搞过来试试！

my config文件：

currentDeployment: default deploymentConfigurations: - name: default version: local:1.26.6 providers: appengine: enabled: false accounts: [] aws: enabled: false accounts: [] bakeryDefaults: baseImages: [] defaultKeyPairTemplate: '{ 
  {name}}-keypair' defaultRegions: - name: us-west-2 defaults: iamRole: BaseIAMRole ecs: enabled: false accounts: [] azure: enabled: false accounts: [] bakeryDefaults: templateFile: azure-linux.json baseImages: [] dcos: enabled: false accounts: [] clusters: [] dockerRegistry: enabled: true accounts: - name: my-harbor-registry requiredGroupMembership: [] providerVersion: V1 permissions: READ: - yunweizu WRITE: - yunweizu address: https://harbor.xxxx.com username: zhangpeng password: xxxx email:  cacheIntervalSeconds: 30 clientTimeoutMillis: 60000 cacheThreads: 1 paginateSize: 100 sortTagsByDate: false trackDigests: false insecureRegistry: false repositories: [] primaryAccount: my-harbor-registry google: enabled: false accounts: [] bakeryDefaults: templateFile: gce.json baseImages: [] zone: us-central1-f network: default useInternalIp: false huaweicloud: enabled: false accounts: [] bakeryDefaults: baseImages: [] kubernetes: enabled: true accounts: - name: default requiredGroupMembership: [] providerVersion: V2 permissions: READ: - yunweizu,group02 - devops WRITE: - yunweizu - devops dockerRegistries: - accountName: my-harbor-registry namespaces: [] context: kubernetes-admin@kubernetes configureImagePullSecrets: true serviceAccount: true cacheThreads: 1 namespaces: [] omitNamespaces: - kube-system - kube-public kinds: [] omitKinds: [] customResources: [] cachingPolicies: [] oAuthScopes: [] onlySpinnakerManaged: false primaryAccount: default tencentcloud: enabled: false accounts: [] bakeryDefaults: baseImages: [] oracle: enabled: false accounts: [] bakeryDefaults: templateFile: oci.json baseImages: [] cloudfoundry: enabled: false accounts: [] deploymentEnvironment: size: SMALL type: Distributed accountName: default imageVariant: SLIM updateVersions: true consul: enabled: false vault: enabled: false location: spinnaker customSizing: {} sidecars: {} initContainers: {} hostAliases: {} affinity: {} tolerations: {} nodeSelectors: {} gitConfig: upstreamUser: spinnaker livenessProbeConfig: enabled: false haServices: clouddriver: enabled: false disableClouddriverRoDeck: false echo: enabled: false persistentStorage: persistentStoreType: s3 azs: {} gcs: rootFolder: front50 redis: {} s3: rootFolder: front50 oracle: {} features: auth: false fiat: false chaos: false entityTags: false pipelineTemplates: true artifacts: true managedPipelineTemplatesV2UI: true metricStores: datadog: enabled: false tags: [] prometheus: enabled: false add_source_metalabels: true stackdriver: enabled: false newrelic: enabled: false tags: [] period: 30 enabled: false notifications: slack: enabled: false twilio: enabled: false baseUrl: https://api.twilio.com/ github-status: enabled: false timezone: Asia/Shanghai ci: jenkins: enabled: true masters: - name: my-jenkins-master-01 permissions: {} address: https://jenkins.xxxx.com username: zhangpeng password: xxxxx csrf: true travis: enabled: false masters: [] wercker: enabled: false masters: [] concourse: enabled: false masters: [] gcb: enabled: false accounts: [] codebuild: enabled: false accounts: [] repository: artifactory: enabled: false searches: [] security: apiSecurity: ssl: enabled: false overrideBaseUrl: https://spin-gate.xxxx.com uiSecurity: ssl: enabled: false overrideBaseUrl: https://spinnaker.xxxx.com authn: oauth2: enabled: false client: {} resource: {} userInfoMapping: {} saml: enabled: false userAttributeMapping: {} ldap: enabled: true url: ldap://172.19.252.28:389 userSearchBase: ou=devops,dc=xxxx,dc=com userSearchFilter: cn={0} managerDn: cn=admin,dc=xxxx,dc=com managerPassword: xxxx x509: enabled: false iap: enabled: false enabled: true authz: groupMembership: service: LDAP google: roleProviderType: GOOGLE github: roleProviderType: GITHUB file: roleProviderType: FILE path: /home/spinnaker/.hal/userrole.yml ldap: roleProviderType: LDAP url: ldap://172.19.252.28:389/dc=xxxx,dc=com managerDn: cn=admin,dc=xxxx,dc=com managerPassword: xxxx userDnPattern: cn={0} groupSearchBase: ou=devops userSearchFilter: cn={0} groupSearchFilter: uniqueMember={0} groupRoleAttributes: cn enabled: true artifacts: bitbucket: enabled: false accounts: [] gcs: enabled: false accounts: [] oracle: enabled: false accounts: [] github: enabled: true accounts: - name: my-github-account username: zeyangli token: xxxx gitlab: enabled: true accounts: - name: my-gitlab-account token: xxxx gitrepo: enabled: false accounts: [] http: enabled: false accounts: [] helm: enabled: false accounts: [] s3: enabled: false accounts: [] maven: enabled: false accounts: [] templates: [] pubsub: enabled: false google: enabled: false pubsubType: GOOGLE subscriptions: [] publishers: [] canary: enabled: false serviceIntegrations: - name: google enabled: false accounts: [] gcsEnabled: false stackdriverEnabled: false - name: prometheus enabled: false accounts: [] - name: datadog enabled: false accounts: [] - name: signalfx enabled: false accounts: [] - name: aws enabled: false accounts: [] s3Enabled: false - name: newrelic enabled: false accounts: [] reduxLoggerEnabled: true defaultJudge: NetflixACAJudge-v1.0 stagesEnabled: true templatesEnabled: true showAllConfigsEnabled: true spinnaker: extensibility: plugins: {} repositories: {} webhook: trust: enabled: false stats: enabled: true endpoint: https://stats.spinnaker.io instanceId: 01FKDR1B3P8PF35RRC93XTE9AS deploymentMethod: {} connectionTimeoutMillis: 3000 readTimeoutMillis: 5000 

继续

[root@k8s-master-01 .kube]# ctr c create registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 halyard --mount type=bind,src=/root/.hal,dst=/home/spinnaker/.hal,options=rbind:row --mount type=bind,src=/home/spinnaker/.kube,dst=/home/spinnaker/.kube,options=rbind:ro --mount type=bind,src=/opt/halyard/config/,dst=/opt/halyard/config/,options=rbind:rw [root@k8s-master-01 .kube]# ctr t start -d halyard [root@k8s-master-01 .kube]# ctr t ls TASK PID STATUS halyard  RUNNING [root@k8s-master-01 .kube]# ctr tasks exec -t --exec-id  halyard sh bash-5.0$ hal deploy apply --no-validate 

重新来一遍

[root@k8s-master-01 .kube]# ctr t kill --signal 9 halyard [root@k8s-master-01 .kube]# ctr c rm halyard 

[root@k8s-master-01 .hal]# ctr c create registry.cn-beijing.aliyuncs.com/spinnaker-cd/halyard:1.32.0 halyard --mount type=bind,src=/root/.hal,dst=/home/spinnaker/.hal,options=rbind:row --mount type=bind,src=/home/spinnaker/.kube,dst=/home/spinnaker/.kube,options=rbind:ro --mount type=bind,src=/opt/halyard/config/,dst=/opt/halyard/config/,options=rbind:rw [root@k8s-master-01 .hal]# ctr t start -d halyard [root@k8s-master-01 .hal]# ctr t ls TASK PID STATUS halyard  RUNNING [root@k8s-master-01 .hal]# ctr tasks exec -t --exec-id  halyard bash bash-5.0$ 

算了我放弃了…，containerd的安装方式

总结一下失败以及经验：

1. containerd or docker的运行时中都可以在文件夹 /home/spinnaker/.hal/default/service-settings本地写文件的件方式指定image tag，docker环境下还好，containerd方式下crictl 修改镜像标签自己掌握的不是很好！
2. containerd命令跟docker还是不一样。启动halyard的方式还是很不好弄，最好的方式还是在一台安装docker的机器上面运行halyard。
3. halyard执行脚本复制命令的空格格式问题
4. 部署过程中出现数据库地址写错问题…写成了TDSQL-C中的读地址…

​