LinuCエヴァンジェリストの鯨井貴博@opensourcetechです。
今回は、HAProxyとKeepalivedを使って、Load Balancerを構築してみます。
役割としては、Keepalivedで仮想IP(VIP)を、HAProxyでFrontend(通信の受け側)・Backend(振り分け先のサーバ群)を設定します。
まず、BackendとなるWebサーバをDockerコンテナで起動しておきます。
具体的には、以下の4台を用意しました。
Web1-1 = 192.168.1.254:10001
Web1-2 = 192.168.1.254:10002
Web2-1 = 192.168.1.254:20001
Web2-2 = 192.168.1.254:20002
takahiro@ubuntu2004server:~$ sudo docker pull nginx Using default tag: latest latest: Pulling from library/nginx . . . sha256:0e188877aa60537d1a1c6484b8c3929cfe09988145327ee47e8e91ddf6f76f5c Status: Downloaded newer image for nginx:latest docker.io/library/nginx:latest takahiro@ubuntu2004server:~$ sudo docker imaeges REPOSITORY TAG IMAGE ID CREATED SIZE nginx latest 8cf1bfb43ff5 12 days ago 132MB 6ec8c9369e08: Pulling fs layer takahiro@ubuntu2004server:~$ sudo docker run --name web1-1 -d -p 10001:80 nginx:latest 2ab9cee364f184f6419c2d891c18b82ede2c32a0e90abe4d57ea8317cddc18f7 takahiro@ubuntu2004server:~$ sudo docker run --name web1-2 -d -p 10001:80 nginx:latest 79ecd73bcb93c028e6a38a76a830f0a518e2d3ba4d5c239313a334e85642a738 takahiro@ubuntu2004server:~$ sudo docker run --name web2-1 -d -p 20001:80 nginx:latest 99e93ac4fa2309770509180a16d693bfc68a571a1996c79a3403ef50469cb9f8 takahiro@ubuntu2004server:~$ sudo docker run --name web2-2 -d -p 20002:80 nginx:latest 14f201b26ccb2846935c3dd8aa6373337d2fd263c677af293e796c9f73994e3c takahiro@ubuntu2004server:~$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 14f201b26ccb nginx:latest "/docker-entrypoint.…" 10 minutes ago Up 10 minutes 0.0.0.0:20002->80/tcp web2-2 99e93ac4fa23 nginx:latest "/docker-entrypoint.…" 10 minutes ago Up 10 minutes 0.0.0.0:20001->80/tcp web2-1 79ecd73bcb93 nginx:latest "/docker-entrypoint.…" 10 minutes ago Up 10 minutes 0.0.0.0:10002->80/tcp web1-2 2ab9cee364f1 nginx:latest "/docker-entrypoint.…" 11 minutes ago Up 11 minutes 0.0.0.0:10001->80/tcp web1-1
続いて、CentOS7にkeepalivedとHAProxyをインストールします。
[root@localhost ~]# yum install haproxy keepalived 読み込んだプラグイン:fastestmirror Loading mirror speeds from cached hostfile * base: ftp-srv2.kddilabs.jp * extras: ftp-srv2.kddilabs.jp * updates: ty1.mirror.newmediaexpress.com base | 3.6 kB 00:00:00 c7-media | 3.6 kB 00:00:00 extras | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 (1/2): extras/7/x86_64/primary_db | 205 kB 00:00:00 (2/2): updates/7/x86_64/primary_db 70% [=================- ] 0.0 B/s | 2.7 MB . . 省略 . . インストール: haproxy.x86_64 0:1.5.18-9.el7 keepalived.x86_64 0:1.3.5-16.el7 依存性関連をインストールしました: lm_sensors-libs.x86_64 0:3.4.0-8.20160601gitf9185e5.el7 net-snmp-agent-libs.x86_64 1:5.7.2-48.el7_8.1 net-snmp-libs.x86_64 1:5.7.2-48.el7_8.1 perl.x86_64 4:5.16.3-295.el7 perl-Carp.noarch 0:1.26-244.el7 perl-Encode.x86_64 0:2.51-7.el7 perl-Exporter.noarch 0:5.68-3.el7 perl-File-Path.noarch 0:2.09-2.el7 perl-File-Temp.noarch 0:0.23.01-3.el7 perl-Filter.x86_64 0:1.49-3.el7 perl-Getopt-Long.noarch 0:2.40-3.el7 perl-HTTP-Tiny.noarch 0:0.033-3.el7 perl-PathTools.x86_64 0:3.40-5.el7 perl-Pod-Escapes.noarch 1:1.04-295.el7 perl-Pod-Perldoc.noarch 0:3.20-4.el7 perl-Pod-Simple.noarch 1:3.28-4.el7 perl-Pod-Usage.noarch 0:1.63-3.el7 perl-Scalar-List-Utils.x86_64 0:1.27-248.el7 perl-Socket.x86_64 0:2.010-5.el7 perl-Storable.x86_64 0:2.45-3.el7 perl-Text-ParseWords.noarch 0:3.29-4.el7 perl-Time-HiRes.x86_64 4:1.9725-3.el7 perl-Time-Local.noarch 0:1.2300-2.el7 perl-constant.noarch 0:1.27-2.el7 perl-libs.x86_64 4:5.16.3-295.el7 perl-macros.x86_64 4:5.16.3-295.el7 perl-parent.noarch 1:0.225-244.el7 perl-podlators.noarch 0:2.5.1-3.el7 perl-threads.x86_64 0:1.87-4.el7 perl-threads-shared.x86_64 0:1.43-6.el7 完了しました!
HAProxyの設定。/etc/haproxy/haproxy.confを編集します。
[root@localhost ~]# cat /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Example configuration for a possible web application. See the
# full configuration options online.
#
# http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
#frontend main *:5000
# acl url_static path_beg -i /static /images /javascript /stylesheets
# acl url_static path_end -i .jpg .gif .png .css .js
#
# use_backend static if url_static
# default_backend app
frontend main *:80 =======>仮想IPのTCP80で待ち受けしますという意味
default_backend app =======>バックエンド「app」を使うという意味
#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend static
balance roundrobin
server static 127.0.0.1:4331 check
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
#backend app
# balance roundrobin
# server app1 127.0.0.1:5001 check
# server app2 127.0.0.1:5002 check
# server app3 127.0.0.1:5003 check
# server app4 127.0.0.1:5004 check
backend app =======>バックエンド「app」の内容
# balance source
balance roundrobin =======>ラウンドロビンを使うという意味
server sv1 192.168.1.254:10001 check weight 1 =======>振り分けサーバの設定 ※重み設定してます。
server sv2 192.168.1.254:10002 check weight 2
backend app2
# balance source
balance roundrobin
server sv3 192.168.1.254:20001 check weight 1
server sv4 192.168.1.254:20002 check weight 2
続いて、keepalivedの設定。/etc/keepalived/keepalived.confを編集します。
[root@localhost ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
# notification_email {
# acassen@firewall.loc
# failover@firewall.loc
# sysadmin@firewall.loc
# }
# notification_email_from Alexandre.Cassen@firewall.loc
# smtp_server 192.168.200.1
# smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
vrrp_strict
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_instance VI_1 { =======>1つ目のVIP設定
state MASTER
interface team0
virtual_router_id 51
priority 50
advert_int 1
# authentication {
# auth_type PASS
# auth_pass 1111
# }
virtual_ipaddress {
192.168.1.219/24
# 192.168.200.17
# 192.168.200.18
}
}
vrrp_instance VI_2 { =======>2つ目のVIP設定
state MASTER
interface team0
virtual_router_id 52
priority 50
advert_int 1
virtual_ipaddress {
192.168.1.220/24
}
}
#virtual_server 192.168.200.100 443 {
# delay_loop 6
# lb_algo rr
# lb_kind NAT
# persistence_timeout 50
# protocol TCP
#
# real_server 192.168.201.100 443 {
# weight 1
# SSL_GET {
# url {
# path /
# digest ff20ad2481f97b1754ef3e12ecd3a9cc
# }
# url {
# path /mrtg/
# digest 9b3a0c85a887a256d6939da88aabd8cd
# }
# connect_timeout 3
# nb_get_retry 3
# delay_before_retry 3
# }
# }
#}
#
#virtual_server 10.10.10.2 1358 {
# delay_loop 6
# lb_algo rr
# lb_kind NAT
# persistence_timeout 50
# protocol TCP
#
# sorry_server 192.168.200.200 1358
#
# real_server 192.168.200.2 1358 {
# weight 1
# HTTP_GET {
# url {
# path /testurl/test.jsp
# digest 640205b7b0fc66c1ea91c463fac6334d
# }
# url {
# path /testurl2/test.jsp
# digest 640205b7b0fc66c1ea91c463fac6334d
# }
# url {
# path /testurl3/test.jsp
# digest 640205b7b0fc66c1ea91c463fac6334d
# }
# connect_timeout 3
# nb_get_retry 3
# delay_before_retry 3
# }
# }
#
# real_server 192.168.200.3 1358 {
# weight 1
# HTTP_GET {
# url {
# path /testurl/test.jsp
# digest 640205b7b0fc66c1ea91c463fac6334c
# }
# url {
# path /testurl2/test.jsp
# digest 640205b7b0fc66c1ea91c463fac6334c
# }
# connect_timeout 3
# nb_get_retry 3
# delay_before_retry 3
# }
# }
#}
#
#virtual_server 10.10.10.3 1358 {
# delay_loop 3
# lb_algo rr
# lb_kind NAT
# persistence_timeout 50
# protocol TCP
#
# real_server 192.168.200.4 1358 {
# weight 1
# HTTP_GET {
# url {
# path /testurl/test.jsp
# digest 640205b7b0fc66c1ea91c463fac6334d
# }
# url {
# path /testurl2/test.jsp
# digest 640205b7b0fc66c1ea91c463fac6334d
# }
# url {
# path /testurl3/test.jsp
# digest 640205b7b0fc66c1ea91c463fac6334d
# }
# connect_timeout 3
# nb_get_retry 3
# delay_before_retry 3
# }
# }
#
# real_server 192.168.200.5 1358 {
# weight 1
# HTTP_GET {
# url {
# path /testurl/test.jsp
# digest 640205b7b0fc66c1ea91c463fac6334d
# }
# url {
# path /testurl2/test.jsp
# digest 640205b7b0fc66c1ea91c463fac6334d
# }
# url {
# path /testurl3/test.jsp
# digest 640205b7b0fc66c1ea91c463fac6334d
# }
# connect_timeout 3
# nb_get_retry 3
# delay_before_retry 3
# }
# }
#}
ファイアウォールでTCP80を受け付けます。
[root@localhost ~]# firewall-cmd --list-all public (active) target: default icmp-block-inversion: no interfaces: enp0s3 enp0s8 team0 sources: services: dhcpv6-client http ssh ports: protocols: masquerade: no forward-ports: source-ports: icmp-blocks: rich ruls:
両プログラムを起動します。
[root@localhost ~]# systemctl start haproxy
[root@localhost ~]# systemctl start keepalived
[root@localhost ~]# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
Active: active (running) since 火 2020-08-04 19:10:07 JST; 4s ago
Main PID: 1626 (haproxy-systemd)
CGroup: /system.slice/haproxy.service
tq1626 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy....
tq1627 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
mq1628 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
8月 04 19:10:07 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
8月 04 19:10:07 localhost.localdomain haproxy-systemd-wrapper[1626]: haproxy-systemd-wrapper: ...
Hint: Some lines were ellipsized, use -l to show in full.
[root@localhost ~]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
Active: active (running) since 火 2020-08-04 19:10:17 JST; 5s ago
Process: 1636 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
CGroup: /system.slice/keepalived.service
mq1637 /usr/sbin/keepalived -D
8月 04 19:10:17 localhost.localdomain Keepalived_vrrp[1639]: Registering Kernel netlink reflector
8月 04 19:10:17 localhost.localdomain Keepalived_vrrp[1639]: Registering Kernel netlink comma...l
8月 04 19:10:17 localhost.localdomain Keepalived_vrrp[1639]: Registering gratuitous ARP share...l
8月 04 19:10:17 localhost.localdomain Keepalived_vrrp[1639]: Opening file '/etc/keepalived/ke....
8月 04 19:10:17 localhost.localdomain Keepalived_vrrp[1639]: Cant find interface eth0 for vrr...!
8月 04 19:10:17 localhost.localdomain Keepalived_vrrp[1639]: Default interface eth0 does not ....
8月 04 19:10:17 localhost.localdomain Keepalived_vrrp[1639]: (VI_1): No VIP specified; at lea...d
8月 04 19:10:17 localhost.localdomain Keepalived_healthcheckers[1638]: Opening file '/etc/keep...
8月 04 19:10:18 localhost.localdomain Keepalived[1637]: Keepalived_vrrp exited with permanen...ng
8月 04 19:10:18 localhost.localdomain Keepalived[1637]: Stopping
Hint: Some lines were ellipsized, use -l to show in full.
仮想IP(VIP)の確認。ip addr showコマンドを使います。
[root@localhost ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master team0 state UP group default qlen 1000
link/ether 08:00:27:7f:98:3f brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master team0 state UP group default qlen 1000
link/ether 08:00:27:28:fa:b0 brd ff:ff:ff:ff:ff:ff
4: team0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 08:00:27:7f:98:3f brd ff:ff:ff:ff:ff:ff
inet 192.168.1.218/24 brd 192.168.1.255 scope global noprefixroute team0
valid_lft forever preferred_lft forever
inet 192.168.1.219/24 scope global secondary team0 =======>VIP1がある
valid_lft forever preferred_lft forever
inet 192.168.1.220/24 scope global secondary team0 =======>VIP2がある
valid_lft forever preferred_lft forever
inet6 240f:32:57b8:1:6dfa:9033:f237:58aa/64 scope global noprefixroute dynamic
valid_lft 265sec preferred_lft 265sec
inet6 fe80::75a9:1afe:b90c:2ae8/64 scope link noprefixroute
valid_lft forever preferred_lft forever
そして、いよいよLoad Balancerの動作確認。
VIP(192.168.1.219/192.168.1.220)にアクセスしたら、Backend(192.168.1.254:10001/192.168.1.254:10002)へ1:2で分散すればOKです。
Firefoxのネットワークモニター(crtl+shift+eで表示される)を使って確認すると、指定した比率で分散していることが分かります。
※その他、パケットキャプチャしたものをWiresharkなどで確認する・サーバログを確認するなどでも同様のことはできます。
