use ip modify route table use vip as src trans in multi-IP bonded env

  1. 云栖社区>
  2. 博客列表>
  3. 正文

use ip modify route table use vip as src trans in multi-IP bonded env

德哥 2016-03-31 17:17:10 浏览819 评论0

摘要: 在某些场景中, 当一个服务器的一块网卡上面配置了多个IP时, 例如虚拟IP, 可能想指定虚拟IP地址作为出口地址. 例如对这个虚拟IP有鉴权要求的场景. 如PostgreSQL的pg_hba.conf.

在某些场景中, 当一个服务器的一块网卡上面配置了多个IP时, 例如虚拟IP, 可能想指定虚拟IP地址作为出口地址.
例如对这个虚拟IP有鉴权要求的场景. 如PostgreSQL的pg_hba.conf.
如在集中的流复制standby场景, 当集中的主机DOWN掉的话, 希望把虚拟IP切走, 同时生产机也只允许这个虚拟IP来访问的情况.
那么需要改写集中流复制的主机的路由表, 让其出口为虚拟IP. 

默认情况下当有多个IP时, 路由是primary优先的. 
如下, 默认的出口是    inet 172.16.3.111/24 brd 172.16.3.255 scope global eth0 而不是eth0:1
[root@db-172-16-3-111 network-scripts]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether b8:ca:3a:6d:fe:b0 brd ff:ff:ff:ff:ff:ff
    inet 172.16.3.111/24 brd 172.16.3.255 scope global eth0
    inet 172.16.3.1/24 brd 172.16.1.255 scope global secondary eth0:1

为了达到指定出口IP的目的, 可以通过修改路由表来实现.
[root@db-172-16-3-111 ~]# which ip
/sbin/ip
在没有创建eth0:1的情况下的默认路由如下 : 
[root@db-172-16-3-111 ~]# ip route
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111 
169.254.0.0/16 dev eth0  scope link  metric 1002 
default via 172.16.3.1 dev eth0 
新增一个IP地址
[root@db-172-16-3-111 ~]# cd /etc/sysconfig/network-scripts/

[root@db-172-16-3-111 network-scripts]# cp ifcfg-eth0 ifcfg-eth0:1
[root@db-172-16-3-111 network-scripts]# vi ifcfg-eth0:1
DEVICE="eth0:1"
BOOTPROTO="static"
IPADDR="172.16.3.105"
NETMASK="255.255.255.0"
HWADDR="B8:CA:3A:6D:FE:B0"
IPV6INIT="no"
MTU="1500"
NM_CONTROLLED="no"
ONBOOT="no"
TYPE="Ethernet"
UUID="37fb27ad-4293-43ba-a3b7-8e94fa563384"

[root@db-172-16-3-111 network-scripts]# ifup eth0:1
Determining if ip address 172.16.3.105 is already in use for device eth0...
[root@db-172-16-3-111 network-scripts]# ifconfig
eth0      Link encap:Ethernet  HWaddr B8:CA:3A:6D:FE:B0  
          inet addr:172.16.3.111  Bcast:172.16.3.255  Mask:255.255.255.0
          inet6 addr: fe80::baca:3aff:fe6d:feb0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:536391 errors:0 dropped:0 overruns:0 frame:0
          TX packets:88849 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:718980563 (685.6 MiB)  TX bytes:4883966 (4.6 MiB)
          Memory:dcb00000-dcc00000 

eth0:1    Link encap:Ethernet  HWaddr B8:CA:3A:6D:FE:B0  
          inet addr:172.16.3.105  Bcast:172.16.3.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Memory:dcb00000-dcc00000 

然后添加路由, 目的是增加一条metric=1的路由, 使得去整个广播域和网关走新增的这个地址出去.
[root@db-172-16-3-111 network-scripts]# ip route add default via 172.16.3.1 dev eth0 src 172.16.3.105 metric 1
[root@db-172-16-3-111 network-scripts]# ip route add 172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.105 metric 1
然后删除原路由条目
[root@db-172-16-3-111 network-scripts]# ip route del default via 172.16.3.1 dev eth0
[root@db-172-16-3-111 network-scripts]# ip route del 172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111
注意, 最好增加2个条目, 如果虚拟IP切走的话, 还能正常走eth0出去的路由. 只是metric改大一点.
[root@db-172-16-3-111 network-scripts]# ip route add default via 172.16.3.1 dev eth0 src 172.16.3.111 metric 100
[root@db-172-16-3-111 network-scripts]# ip route add 172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111 metric 100
添加完后的路由表
[root@db-172-16-3-111 network-scripts]# ip route
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.105  metric 1 
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111  metric 100 
169.254.0.0/16 dev eth0  scope link  metric 1002 
default via 172.16.3.1 dev eth0  src 172.16.3.105  metric 1 
default via 172.16.3.1 dev eth0  metric 100 
看看路由有没有生效, 已经使用了我们配置的metric=1的路由条目.
[root@db-172-16-3-111 network-scripts]# ip route get 192.168.1.1
192.168.1.1 via 172.16.3.1 dev eth0  src 172.16.3.105 
    cache  mtu 1500 advmss 1460 hoplimit 64
[root@db-172-16-3-111 network-scripts]# ip route get 172.16.3.1
172.16.3.1 dev eth0  src 172.16.3.105 
    cache  mtu 1500 advmss 1460 hoplimit 64

关闭这个虚拟IP, 走metric=100的路由条目
[root@db-172-16-3-111 network-scripts]# ifdown eth0:1
[root@db-172-16-3-111 network-scripts]# ip route get 172.16.3.1
172.16.3.1 dev eth0  src 172.16.3.111 
    cache  mtu 1500 advmss 1460 hoplimit 64
[root@db-172-16-3-111 network-scripts]# ip route get 192.168.1.1
192.168.1.1 via 172.16.3.1 dev eth0  src 172.16.3.111 
    cache  mtu 1500 advmss 1460 hoplimit 64

关闭eth0:1后, 路由条目自动消失. 
[root@db-172-16-3-111 network-scripts]# ip route
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111  metric 100 
169.254.0.0/16 dev eth0  scope link  metric 1002 
default via 172.16.3.1 dev eth0  metric 100 

[root@db-172-16-3-111 network-scripts]# ifup eth0:1
Determining if ip address 172.16.3.105 is already in use for device eth0...

[root@db-172-16-3-111 network-scripts]# ip route
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111  metric 100 
169.254.0.0/16 dev eth0  scope link  metric 1002 
default via 172.16.3.1 dev eth0 
default via 172.16.3.1 dev eth0  metric 100 

如果虚拟IP切回来, 又要重新写路由, 比较土的方法是重启network服务, 然后执行前面一样的步骤添加路由.
[root@db-172-16-3-111 network-scripts]# service network restart
Shutting down interface eth0:  [  OK  ]
Shutting down loopback interface:  [  OK  ]
Bringing up loopback interface:  [  OK  ]
Bringing up interface eth0:  Determining if ip address 172.16.3.111 is already in use for device eth0...
[  OK  ]

[root@db-172-16-3-111 network-scripts]# ip route
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111 
169.254.0.0/16 dev eth0  scope link  metric 1002 
default via 172.16.3.1 dev eth0 

[root@db-172-16-3-111 network-scripts]# ip route add default via 172.16.3.1 dev eth0 src 172.16.3.105 metric 1
[root@db-172-16-3-111 network-scripts]# ip route add 172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.105 metric 1

[root@db-172-16-3-111 network-scripts]# ip route del default via 172.16.3.1 dev eth0
[root@db-172-16-3-111 network-scripts]# ip route del 172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111

[root@db-172-16-3-111 network-scripts]# ip route add default via 172.16.3.1 dev eth0 src 172.16.3.111 metric 100
[root@db-172-16-3-111 network-scripts]# ip route add 172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111 metric 100

比较方便的方法是写配置文件, 或者是使用iptables .
配置文件比较多, 参考
/usr/share/doc/initscripts-9.03.40/sysconfig.txt
这里用到ifup-post调用的ifup-routes
/etc/sysconfig/network-scripts/route-<interface-name>

  Contains lines that specify additional routes that should be added when the
  associated interface is brought up.

  The files are processed by the ifup-routes script and uses the /sbin/ipcalc
  utility for all network masks and numbers. Routes are specified using the
  syntax:

    ADDRESSn=<network>
    NETMASKn=<network/prefix mask>
    GATEWAYn=<next-hop router/gateway IP address>

  The "n" is expected to be consecutive positive integers starting from 0.
  For example:

    ADDRESS0=192.168.2.0
    NETMASK0=255.255.255.0
    GATEWAY0=192.168.1.1

  adds a network route to the 192.168.2.0 network via the gateway at
  192.168.1.1. Since you must already have a route to the network of the
  gateway, there is no need to specify a device.

  Note: The ifup-routes script also supports an older syntax designed to be
  used directly as an argument to "/sbin/ip route add".
  If no "ADDRESSn" lines are found the following will still
  work:
  
  192.168.2.0/24 dev ppp0
  
  adds a network route to the 192.168.2.0 network through ppp0.
所以需要给绑定了多个IP的网络设备写一个自定义路由配置文件
/etc/sysconfig/network-scripts/route-eth0
内容如下 : 
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111 metric 100
default via 172.16.3.1 dev eth0 src 172.16.3.111 metric 100
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.105 metric 1
default via 172.16.3.1 dev eth0 src 172.16.3.105 metric 1
这个配置文件是在ifup-post后调用ip route add来添加路由.
但是仅此还有问题, 因为默认的话还是会添加2条路由, 网关和本网段的路由.
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111 
default via 172.16.3.1 dev eth0 
[root@db-172-16-3-111 ~]# ip route
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111 
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.105  metric 1 
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111  metric 100 
169.254.0.0/16 dev eth0  scope link  metric 1002 
default via 172.16.3.1 dev eth0 
default via 172.16.3.1 dev eth0  src 172.16.3.105  metric 1 
default via 172.16.3.1 dev eth0  src 172.16.3.111  metric 100 
为了去除这两条路由, 需要调整一下配置文件.

[最终的配置 : ]
把网关去掉
vi /etc/sysconfig/network
and
vi /etc/sysconfig/network-scripts/ifcfg-eth0
vi /etc/sysconfig/network-scripts/ifcfg-eth0:1
GATEWAY=172.16.3.1删除
同时调整
/etc/sysconfig/network-scripts/route-eth0
最终的内容如下, 优先匹配小的子网掩码 : 
default via 172.16.3.1 dev eth0 src 172.16.3.111 metric 100
172.16.3.0/25 dev eth0  proto kernel  scope link  src 172.16.3.105 metric 1
172.16.3.128/25 dev eth0  proto kernel  scope link  src 172.16.3.105 metric 1
default via 172.16.3.1 dev eth0 src 172.16.3.105 metric 1
重启network服务后, 路由表如下 : 
[root@db-172-16-3-111 network-scripts]# ip route
172.16.3.0/25 dev eth0  proto kernel  scope link  src 172.16.3.105  metric 1 
172.16.3.128/25 dev eth0  proto kernel  scope link  src 172.16.3.105  metric 1 
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111 
169.254.0.0/16 dev eth0  scope link  metric 1002 
default via 172.16.3.1 dev eth0  src 172.16.3.105  metric 1 
default via 172.16.3.1 dev eth0  src 172.16.3.111  metric 100 
ifup , ifdown, service network等都会自动读取这个配置文件, 所以不需要在手工修改了.
另外注意, 不要直接使用ip命令对IP进行配置.
[root@db-172-16-3-111 network-scripts]# ip route get 192.168.1.1
192.168.1.1 via 172.16.3.1 dev eth0  src 172.16.3.105 
    cache  mtu 1500 advmss 1460 hoplimit 64
[root@db-172-16-3-111 network-scripts]# ip route get 172.16.3.1
172.16.3.1 dev eth0  src 172.16.3.105 
    cache  mtu 1500 advmss 1460 hoplimit 64
[root@db-172-16-3-111 network-scripts]# ip route get 172.16.3.0
172.16.3.0 dev eth0  src 172.16.3.105 
    cache  mtu 1500 advmss 1460 hoplimit 64
[root@db-172-16-3-111 network-scripts]# ip route get 172.16.3.127
172.16.3.127 dev eth0  src 172.16.3.105 
    cache  mtu 1500 advmss 1460 hoplimit 64
[root@db-172-16-3-111 network-scripts]# ip route get 172.16.3.128
172.16.3.128 dev eth0  src 172.16.3.105 
    cache  mtu 1500 advmss 1460 hoplimit 64

[root@db-172-16-3-111 network-scripts]# ifdown eth0:1
[root@db-172-16-3-111 network-scripts]# ip route
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111 
169.254.0.0/16 dev eth0  scope link  metric 1002 
default via 172.16.3.1 dev eth0  src 172.16.3.111  metric 100 
[root@db-172-16-3-111 network-scripts]# ifup eth0:1
Determining if ip address 172.16.3.105 is already in use for device eth0...
RTNETLINK answers: File exists
[root@db-172-16-3-111 network-scripts]# ip route
172.16.3.0/25 dev eth0  proto kernel  scope link  src 172.16.3.105  metric 1 
172.16.3.128/25 dev eth0  proto kernel  scope link  src 172.16.3.105  metric 1 
172.16.3.0/24 dev eth0  proto kernel  scope link  src 172.16.3.111 
169.254.0.0/16 dev eth0  scope link  metric 1002 
default via 172.16.3.1 dev eth0  src 172.16.3.105  metric 1 
default via 172.16.3.1 dev eth0  src 172.16.3.111  metric 100 

[参考]
1. /usr/share/doc/initscripts-9.03.40/sysconfig.txt
2. man ip
3. man ifup
【云栖快讯】阿里云栖开发者沙龙(Java技术专场)火热来袭!快来报名参与吧!  详情请点击

网友评论

德哥
文章1946篇 | 关注6440
关注
独立的公网IP资源,可以绑定到阿里云专有网络VPC类型的ECS、NAT网关、私网负载均衡SL... 查看详情
针对互联网服务器(包括非阿里云主机)在遭受大流量的DDoS攻击后导致服务不可用的情况下,推出... 查看详情
以阿里云成熟的商业化云服务为基础,为游戏开发者、运营企业提供专属集群、尊享VIP服务、专项扶... 查看详情
为您提供简单高效、处理能力可弹性伸缩的计算服务,帮助您快速构建更稳定、安全的应用,提升运维效... 查看详情
双12

双12