使用Kubespray 部署kubernetes 高可用集群

  1. 云栖社区>
  2. 博客列表>
  3. 正文

使用Kubespray 部署kubernetes 高可用集群

科技探索者 2017-11-22 17:09:00 浏览743 评论0

摘要: Requirements Ansible v2.4 及以上版本,安装python-netaddr到运行Ansible commands的机器 Jinja 2.9 及以上版本,运行Ansible Playbooks 目标servers 必须可以访问外网,可以pull docker images 目标...

Requirements

  • Ansible v2.4 及以上版本,安装python-netaddr到运行Ansible commands的机器
  • Jinja 2.9 及以上版本,运行Ansible Playbooks
  • 目标servers 必须可以访问外网,可以pull docker images
  • 目标servers 配置允许 IPv4 forwarding
  • 将公钥复制到所有机器
  • 关闭防火墙
  • 请提前安装好docker,因为k8s不支持最新版docker,具体适配哪些docker版本,请看k8s上的changelog

0、环境

主机名 IP
master1 172.16.105.21
master2 172.16.105.22
master3 172.16.105.23
node1 172.16.105.24
node2 172.16.105.25
ansible-client 172.16.105.20

1、安装ansible和依赖

在172.16.105.20安装ansible

# 安装 python 及 epel
yum install -y epel-release python-pip python34 python34-pip
# 安装 ansible
yum install -y ansible
pip install netaddr
pip install --upgrade jinja2

2、建立公私钥,分发各服务器

在ansible-clinet机器生成免密密钥对

ssh-keygen -t rsa -P ''

将生成的公钥(id_rsa.pub)传到其他节点,这样ansible-client可以免密登陆其他机器

 cat id_rsa.pub >> ~/.ssh/authorized_keys

3、下载kuberspay源码

cd /usr/local/src/
wget https://github.com/kubernetes-incubator/kubespray/archive/v2.3.0.tar.gz

本版本所包含的组件版本

  • Kubernetes v1.8.1
  • Docker 1.13.1
  • etcd v3.2.4
  • Rkt v1.21.0 (optional)
  • Calico v2.5.0
  • Weave 2.0.4
  • Flannel v0.8.0

3.1 禁用docker yum仓和docker安装

vim roles/docker/tasks/main.yml

---
- name: gather os specific variables
  include_vars: "{{ item }}"
  with_first_found:
    - files:
        - "{{ ansible_distribution|lower }}-{{ ansible_distribution_version|lower|replace('/', '_') }}.yml"
        - "{{ ansible_distribution|lower }}-{{ ansible_distribution_release }}.yml"
        - "{{ ansible_distribution|lower }}-{{ ansible_distribution_major_version|lower|replace('/', '_') }}.yml"
        - "{{ ansible_distribution|lower }}.yml"
        - "{{ ansible_os_family|lower }}.yml"
        - defaults.yml
      paths:
        - ../vars
      skip: true
  tags:
    - facts

- include: set_facts_dns.yml
  when: dns_mode != 'none' and resolvconf_mode == 'docker_dns'
  tags:
    - facts

- name: check for minimum kernel version
  fail:
    msg: >
          docker requires a minimum kernel version of
          {{ docker_kernel_min_version }} on
          {{ ansible_distribution }}-{{ ansible_distribution_version }}
  when: (not ansible_os_family in ["CoreOS", "Container Linux by CoreOS"]) and (ansible_kernel|version_compare(docker_kernel_min_version, "<"))
  tags:
    - facts

#禁用docker仓库,已经使用清华源
#- name: ensure docker repository public key is installed
#  action: "{{ docker_repo_key_info.pkg_key }}"
#  args:
#    id: "{{item}}"
#    keyserver: "{{docker_repo_key_info.keyserver}}"
#    state: present
#  register: keyserver_task_result
#  until: keyserver_task_result|succeeded
#  retries: 4
#  delay: "{{ retry_stagger | random + 3 }}"
#  environment: "{{ proxy_env }}"
#  with_items: "{{ docker_repo_key_info.repo_keys }}"
#  when: not (ansible_os_family in ["CoreOS", "Container Linux by CoreOS"] or is_atomic)

#- name: ensure docker repository is enabled
#  action: "{{ docker_repo_info.pkg_repo }}"
#  args:
#    repo: "{{item}}"
#    state: present
#  with_items: "{{ docker_repo_info.repos }}"
#  when: not (ansible_os_family in ["CoreOS", "Container Linux by CoreOS"] or is_atomic) and(docker_repo_info.repos|length > 0)

#- name: Configure docker repository on RedHat/CentOS
#  template:
#    src: "rh_docker.repo.j2"
#    dest: "/etc/yum.repos.d/docker.repo"
#  when: ansible_distribution in ["CentOS","RedHat"] and not is_atomic

#- name: ensure docker packages are installed
#  action: "{{ docker_package_info.pkg_mgr }}"
#  args:
#    pkg: "{{item.name}}"
#    force: "{{item.force|default(omit)}}"
#    state: present
#  register: docker_task_result
#  until: docker_task_result|succeeded
#  retries: 4
#  delay: "{{ retry_stagger | random + 3 }}"
#  environment: "{{ proxy_env }}"
#  with_items: "{{ docker_package_info.pkgs }}"
#  notify: restart docker
#  when: not (ansible_os_family in ["CoreOS", "Container Linux by CoreOS"] or is_atomic) and (docker_package_info.pkgs|length > 0)

#对于docker的版本检测进行了保留

- name: check minimum docker version for docker_dns mode. You need at least docker version >= 1.12 for resolvconf_mode=docker_dns
  command: "docker version -f '{{ '{{' }}.Client.Version{{ '}}' }}'"
  register: docker_version
  failed_when: docker_version.stdout|version_compare('1.12', '<')
  changed_when: false
  when: dns_mode != 'none' and resolvconf_mode == 'docker_dns'

#对于docker的systemd配置,可以根据自己需求修改,但是注意会覆盖原来的
- name: Set docker systemd config
  include: systemd.yml

- name: ensure docker service is started and enabled
  service:
    name: "{{ item }}"
    enabled: yes
    state: started
  with_items:
    - docker

4、替换镜像

因为长城的原因,需要的镜像在安装的时候无法获取,所以需要改下源码,下载自己私有仓的镜像
脚本内容如下:

gcr_image_files=(
./kubespray/roles/download/defaults/main.yml
./kubespray/roles/dnsmasq/templates/dnsmasq-autoscaler.yml.j2
./kubespray/roles/kubernetes-apps/ansible/defaults/main.yml
)

for file in ${gcr_image_files[@]} ; do
    sed -i 's/gcr.io/docker.emarbox.com/g' $file
done

镜像列表,最好提前下载到私有仓,毕竟下载会很慢

gcr.io/google_containers/cluster-proportional-autoscaler-amd64:1.1.1
gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.5
gcr.io/google_containers/k8s-dns-kube-dns-amd64:1.14.5
gcr.io/google_containers/k8s-dns-dnsmasq-nanny-amd64:1.14.5
gcr.io/google_containers/pause-amd64:3.0
gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3

nginx:1.11.4-alpine
busybox:latest

quay.io/coreos/hyperkube:v1.8.1_coreos.0
quay.io/coreos/etcd:v3.2.4
quay.io/calico/ctl:v1.5.0
quay.io/calico/node:v2.5.0
quay.io/calico/routereflector:v0.4.0
quay.io/calico/cni:v1.10.0

5、配置文件内容

可以对auth的密码进行修改,网络插件默认calico,可替换成weave或flannel,还可以配置是否安装helm和efk,以及修改安装路径

more kubespray/kubespray-2.3.0/inventory/group_vars/k8s-cluster.yml

#6、生成自己的集群配置

因为kubespray自带的python脚本是Python3 ,所以要安装Python3

yum install -y python-pip python34 python34-pip
# 定义集群IP
IP=(
172.16.105.21
172.16.105.22
172.16.105.23
)

# 利用kubespray自带的python脚本生成配置
CONFIG_FILE=./kubespray/inventory/inventory.cfg python3 ./kubespray/contrib/inventory_builder/inventory.py ${IP[*]}

查看配置

cat ./kubespray/inventory/inventory.cfg

[all]
node1    ansible_host=172.16.105.21 ip=172.16.105.21
node2    ansible_host=172.16.105.22 ip=172.16.105.22
node3    ansible_host=172.16.105.23 ip=172.16.105.23

[kube-master]
node1    
node2    
node3

[kube-node]
node1    
node2    
node3    

[etcd]
node1    
node2    
node3    

[k8s-cluster:children]
kube-node        
kube-master      

[calico-rr]

[vault]
node1    
node2    
node3    

7、安装集群

ansible-playbook -i inventory/inventory.cfg cluster.yml -b -v

镜像地址
kubespray/roles/download/tasks/download_container.yml

8、问题

8.1

从1.8 开始,kubelet 会检测机器是否有swap,如果启用swap,kubelet会无法启动,需要手动添加参数。
去如下目录,修改kubelet参数

/usr/local/src/kubespray/kubespray-2.3.0/roles/kubernetes/node/defaults

### fail with swap on (default true)
kubelet_fail_swap_on: false

8.2

注意机器主机名,要符合k8s的规范

9、安装失败如何清理

rm -rf /etc/kubernetes/
rm -rf /var/lib/kubelet
rm -rf /var/lib/etcd
rm -rf /usr/local/bin/kubectl
rm -rf /etc/systemd/system/calico-node.service
rm -rf /etc/systemd/system/kubelet.service
systemctl stop etcd.service
systemctl disable etcd.service
systemctl stop calico-node.service
systemctl disable calico-node.service
docker stop $(docker ps -q)
docker rm $(docker ps -a -q)
systemctl restart docker

10、安装完成

[root@node2 .kube]# kubectl get nodes
NAME      STATUS    ROLES         AGE       VERSION
node1     Ready     master,node   9m        v1.8.1+coreos.0
node2     Ready     master,node   9m        v1.8.1+coreos.0
node3     Ready     master,node   9m        v1.8.1+coreos.0

11、扩展集群node

把需要添加的node 写入配置文件,然后执行ansible
以添加node4举例
修改 inventory.cfg

[all]
node1    ansible_host=172.16.105.21 ip=172.16.105.21
node2    ansible_host=172.16.105.22 ip=172.16.105.22
node3    ansible_host=172.16.105.23 ip=172.16.105.23
node4    ansible_host=172.16.105.37 ip=172.16.105.37

[kube-master]
node1    
node2    
node3

[kube-node]
node1    
node2    
node3
node4    

[etcd]
node1    
node2    
node3    

[k8s-cluster:children]
kube-node        
kube-master      

[calico-rr]

[vault]
node1    
node2    
node3 
ansible-playbook -i inventory/inventory.cfg scale.yml -b -v \
  --private-key=~/.ssh/private_key

后感

了解ansible,也就能自己随便搞kubespray,这个东西比较透明,不像kubeadm,封装很严,不知道具体流程,出问题也不知道怎么处理。

本文转自银狐博客51CTO博客,原文链接http://blog.51cto.com/foxhound/2044766如需转载请自行联系原作者


战狐

用云栖社区APP,舒服~

【云栖快讯】诚邀你用自己的技术能力来用心回答每一个问题,通过回答传承技术知识、经验、心得,问答专家期待你加入!  详情请点击

网友评论

科技探索者
文章5942篇 | 关注21
关注
提供了高性能可伸缩的容器应用管理服务,支持在一组云服务器上通过Docker容器来进行应用生命... 查看详情
以阿里云成熟的商业化云服务为基础,为游戏开发者、运营企业提供专属集群、尊享VIP服务、专项扶... 查看详情
API 网关(API Gateway),提供高性能、高可用的 API 托管服务,帮助用户对外... 查看详情
为您提供简单高效、处理能力可弹性伸缩的计算服务,帮助您快速构建更稳定、安全的应用,提升运维效... 查看详情
阿里云9.10会员日

阿里云9.10会员日