Opensourcetechブログ

OpensourcetechによるNGINX/Zabbix/Neo4j/Linuxなどオープンソース技術に関するブログです。

KeepalivedとHAProxyでロードバランサーを構築する

DRBDのスプリットブレイン(StandAlone) Pacemaker/Corosync/DRBDのIPアドレス変更 排水管掃除


LinuCエヴァンジェリストの鯨井貴博@opensourcetechです。

今回は、HAProxyとKeepalivedを使って、Load Balancerを構築してみます。
役割としては、Keepalivedで仮想IP(VIP)を、HAProxyでFrontend(通信の受け側)・Backend(振り分け先のサーバ群)を設定します。


まず、BackendとなるWebサーバをDockerコンテナで起動しておきます。
具体的には、以下の4台を用意しました。


Web1-1 = 192.168.1.254:10001
Web1-2 = 192.168.1.254:10002
Web2-1 = 192.168.1.254:20001
Web2-2 = 192.168.1.254:20002

takahiro@ubuntu2004server:~$ sudo docker pull nginx
Using default tag: latest
latest: Pulling from library/nginx
.
.
.
sha256:0e188877aa60537d1a1c6484b8c3929cfe09988145327ee47e8e91ddf6f76f5c
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest

takahiro@ubuntu2004server:~$ sudo docker imaeges
REPOSITORY                TAG                 IMAGE ID            CREATED             SIZE
nginx                     latest              8cf1bfb43ff5        12 days ago         132MB

6ec8c9369e08: Pulling fs layer 
takahiro@ubuntu2004server:~$ sudo docker run --name web1-1 -d -p 10001:80 nginx:latest 
2ab9cee364f184f6419c2d891c18b82ede2c32a0e90abe4d57ea8317cddc18f7

takahiro@ubuntu2004server:~$ sudo docker run --name web1-2 -d -p 10001:80 nginx:latest
79ecd73bcb93c028e6a38a76a830f0a518e2d3ba4d5c239313a334e85642a738

takahiro@ubuntu2004server:~$ sudo docker run --name web2-1 -d -p 20001:80 nginx:latest
99e93ac4fa2309770509180a16d693bfc68a571a1996c79a3403ef50469cb9f8

takahiro@ubuntu2004server:~$ sudo docker run --name web2-2 -d -p 20002:80 nginx:latest
14f201b26ccb2846935c3dd8aa6373337d2fd263c677af293e796c9f73994e3c

takahiro@ubuntu2004server:~$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                   NAMES
14f201b26ccb        nginx:latest        "/docker-entrypoint.…"   10 minutes ago      Up 10 minutes       0.0.0.0:20002->80/tcp   web2-2
99e93ac4fa23        nginx:latest        "/docker-entrypoint.…"   10 minutes ago      Up 10 minutes       0.0.0.0:20001->80/tcp   web2-1
79ecd73bcb93        nginx:latest        "/docker-entrypoint.…"   10 minutes ago      Up 10 minutes       0.0.0.0:10002->80/tcp   web1-2
2ab9cee364f1        nginx:latest        "/docker-entrypoint.…"   11 minutes ago      Up 11 minutes       0.0.0.0:10001->80/tcp   web1-1



続いて、CentOS7にkeepalivedとHAProxyをインストールします。

[root@localhost ~]# yum install haproxy keepalived
読み込んだプラグイン:fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp-srv2.kddilabs.jp
 * extras: ftp-srv2.kddilabs.jp
 * updates: ty1.mirror.newmediaexpress.com
base                                                                        | 3.6 kB  00:00:00     
c7-media                                                                    | 3.6 kB  00:00:00     
extras                                                                      | 2.9 kB  00:00:00     
updates                                                                     | 2.9 kB  00:00:00     
(1/2): extras/7/x86_64/primary_db                                           | 205 kB  00:00:00     
(2/2): updates/7/x86_64/primary_db 70% [=================-       ]  0.0 B/s | 2.7 MB
.
.
省略
.
.
インストール:
  haproxy.x86_64 0:1.5.18-9.el7                  keepalived.x86_64 0:1.3.5-16.el7                 

依存性関連をインストールしました:
  lm_sensors-libs.x86_64 0:3.4.0-8.20160601gitf9185e5.el7                                          
  net-snmp-agent-libs.x86_64 1:5.7.2-48.el7_8.1                                                    
  net-snmp-libs.x86_64 1:5.7.2-48.el7_8.1                                                          
  perl.x86_64 4:5.16.3-295.el7                                                                     
  perl-Carp.noarch 0:1.26-244.el7                                                                  
  perl-Encode.x86_64 0:2.51-7.el7                                                                  
  perl-Exporter.noarch 0:5.68-3.el7                                                                
  perl-File-Path.noarch 0:2.09-2.el7                                                               
  perl-File-Temp.noarch 0:0.23.01-3.el7                                                            
  perl-Filter.x86_64 0:1.49-3.el7                                                                  
  perl-Getopt-Long.noarch 0:2.40-3.el7                                                             
  perl-HTTP-Tiny.noarch 0:0.033-3.el7                                                              
  perl-PathTools.x86_64 0:3.40-5.el7                                                               
  perl-Pod-Escapes.noarch 1:1.04-295.el7                                                           
  perl-Pod-Perldoc.noarch 0:3.20-4.el7                                                             
  perl-Pod-Simple.noarch 1:3.28-4.el7                                                              
  perl-Pod-Usage.noarch 0:1.63-3.el7                                                               
  perl-Scalar-List-Utils.x86_64 0:1.27-248.el7                                                     
  perl-Socket.x86_64 0:2.010-5.el7                                                                 
  perl-Storable.x86_64 0:2.45-3.el7                                                                
  perl-Text-ParseWords.noarch 0:3.29-4.el7                                                         
  perl-Time-HiRes.x86_64 4:1.9725-3.el7                                                            
  perl-Time-Local.noarch 0:1.2300-2.el7                                                            
  perl-constant.noarch 0:1.27-2.el7                                                                
  perl-libs.x86_64 4:5.16.3-295.el7                                                                
  perl-macros.x86_64 4:5.16.3-295.el7                                                              
  perl-parent.noarch 1:0.225-244.el7                                                               
  perl-podlators.noarch 0:2.5.1-3.el7                                                              
  perl-threads.x86_64 0:1.87-4.el7                                                                 
  perl-threads-shared.x86_64 0:1.43-6.el7                                                          

完了しました!



HAProxyの設定。/etc/haproxy/haproxy.confを編集します。

[root@localhost ~]# cat /etc/haproxy/haproxy.cfg 
#---------------------------------------------------------------------
# Example configuration for a possible web application.  See the
# full configuration options online.
#
#   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
#
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
#frontend  main *:5000
#    acl url_static       path_beg       -i /static /images /javascript /stylesheets
#    acl url_static       path_end       -i .jpg .gif .png .css .js
#
#    use_backend static          if url_static
#    default_backend             app

frontend  main *:80     =======>仮想IPのTCP80で待ち受けしますという意味
    default_backend             app          =======>バックエンド「app」を使うという意味

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend static
    balance     roundrobin
    server      static 127.0.0.1:4331 check

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
#backend app
#    balance     roundrobin
#    server  app1 127.0.0.1:5001 check
#    server  app2 127.0.0.1:5002 check
#    server  app3 127.0.0.1:5003 check
#    server  app4 127.0.0.1:5004 check

backend app   =======>バックエンド「app」の内容
#    balance     source
    balance     roundrobin       =======>ラウンドロビンを使うという意味
    server  sv1 192.168.1.254:10001 check weight 1        =======>振り分けサーバの設定 ※重み設定してます。
    server  sv2 192.168.1.254:10002 check weight 2

backend app2
#    balance     source
    balance     roundrobin
    server  sv3 192.168.1.254:20001 check weight 1 
    server  sv4 192.168.1.254:20002 check weight 2



続いて、keepalivedの設定。/etc/keepalived/keepalived.confを編集します。

[root@localhost ~]# cat /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {
#   notification_email {
#     acassen@firewall.loc
#     failover@firewall.loc
#     sysadmin@firewall.loc
#   }
#   notification_email_from Alexandre.Cassen@firewall.loc
#   smtp_server 192.168.200.1
#   smtp_connect_timeout 30
   router_id LVS_DEVEL
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {       =======>1つ目のVIP設定
    state MASTER
    interface team0
    virtual_router_id 51
    priority 50
    advert_int 1
#    authentication {
#        auth_type PASS
#        auth_pass 1111
#    }
    virtual_ipaddress {
        192.168.1.219/24
#        192.168.200.17
#        192.168.200.18
    }
}

vrrp_instance VI_2 {       =======>2つ目のVIP設定
    state MASTER
    interface team0
    virtual_router_id 52
    priority 50
    advert_int 1
    virtual_ipaddress {
        192.168.1.220/24
    }
}



#virtual_server 192.168.200.100 443 {
#    delay_loop 6
#    lb_algo rr
#    lb_kind NAT
#    persistence_timeout 50
#    protocol TCP
#
#    real_server 192.168.201.100 443 {
#        weight 1
#        SSL_GET {
#            url {
#              path /
#              digest ff20ad2481f97b1754ef3e12ecd3a9cc
#            }
#            url {
#              path /mrtg/
#              digest 9b3a0c85a887a256d6939da88aabd8cd
#            }
#            connect_timeout 3
#            nb_get_retry 3
#            delay_before_retry 3
#        }
#    }
#}
#
#virtual_server 10.10.10.2 1358 {
#    delay_loop 6
#    lb_algo rr 
#    lb_kind NAT
#    persistence_timeout 50
#    protocol TCP
#
#    sorry_server 192.168.200.200 1358
#
#    real_server 192.168.200.2 1358 {
#        weight 1
#        HTTP_GET {
#            url { 
#              path /testurl/test.jsp
#              digest 640205b7b0fc66c1ea91c463fac6334d
#            }
#            url { 
#              path /testurl2/test.jsp
#              digest 640205b7b0fc66c1ea91c463fac6334d
#            }
#            url { 
#              path /testurl3/test.jsp
#              digest 640205b7b0fc66c1ea91c463fac6334d
#            }
#            connect_timeout 3
#            nb_get_retry 3
#            delay_before_retry 3
#        }
#    }
#
#    real_server 192.168.200.3 1358 {
#        weight 1
#        HTTP_GET {
#            url { 
#              path /testurl/test.jsp
#              digest 640205b7b0fc66c1ea91c463fac6334c
#            }
#            url { 
#              path /testurl2/test.jsp
#              digest 640205b7b0fc66c1ea91c463fac6334c
#            }
#            connect_timeout 3
#            nb_get_retry 3
#            delay_before_retry 3
#        }
#    }
#}
#
#virtual_server 10.10.10.3 1358 {
#    delay_loop 3
#    lb_algo rr 
#    lb_kind NAT
#    persistence_timeout 50
#    protocol TCP
#
#    real_server 192.168.200.4 1358 {
#        weight 1
#        HTTP_GET {
#            url { 
#              path /testurl/test.jsp
#              digest 640205b7b0fc66c1ea91c463fac6334d
#            }
#            url { 
#              path /testurl2/test.jsp
#              digest 640205b7b0fc66c1ea91c463fac6334d
#            }
#            url { 
#              path /testurl3/test.jsp
#              digest 640205b7b0fc66c1ea91c463fac6334d
#            }
#            connect_timeout 3
#            nb_get_retry 3
#            delay_before_retry 3
#        }
#    }
#
#    real_server 192.168.200.5 1358 {
#        weight 1
#        HTTP_GET {
#            url { 
#              path /testurl/test.jsp
#              digest 640205b7b0fc66c1ea91c463fac6334d
#            }
#            url { 
#              path /testurl2/test.jsp
#              digest 640205b7b0fc66c1ea91c463fac6334d
#            }
#            url { 
#              path /testurl3/test.jsp
#              digest 640205b7b0fc66c1ea91c463fac6334d
#            }
#            connect_timeout 3
#            nb_get_retry 3
#            delay_before_retry 3
#        }
#    }
#}



ファイアウォールでTCP80を受け付けます。

[root@localhost ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s3 enp0s8 team0
  sources: 
  services: dhcpv6-client http ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich ruls: 



両プログラムを起動します。

[root@localhost ~]# systemctl start haproxy
[root@localhost ~]# systemctl start keepalived
[root@localhost ~]# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
   Active: active (running) since 火 2020-08-04 19:10:07 JST; 4s ago
 Main PID: 1626 (haproxy-systemd)
   CGroup: /system.slice/haproxy.service
           tq1626 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy....
           tq1627 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
           mq1628 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds

 8月 04 19:10:07 localhost.localdomain systemd[1]: Started HAProxy Load Balancer.
 8月 04 19:10:07 localhost.localdomain haproxy-systemd-wrapper[1626]: haproxy-systemd-wrapper: ...
Hint: Some lines were ellipsized, use -l to show in full.
[root@localhost ~]# systemctl status keepalived
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
   Active: active (running) since 火 2020-08-04 19:10:17 JST; 5s ago
  Process: 1636 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/keepalived.service
           mq1637 /usr/sbin/keepalived -D

 8月 04 19:10:17 localhost.localdomain Keepalived_vrrp[1639]: Registering Kernel netlink reflector
 8月 04 19:10:17 localhost.localdomain Keepalived_vrrp[1639]: Registering Kernel netlink comma...l
 8月 04 19:10:17 localhost.localdomain Keepalived_vrrp[1639]: Registering gratuitous ARP share...l
 8月 04 19:10:17 localhost.localdomain Keepalived_vrrp[1639]: Opening file '/etc/keepalived/ke....
 8月 04 19:10:17 localhost.localdomain Keepalived_vrrp[1639]: Cant find interface eth0 for vrr...!
 8月 04 19:10:17 localhost.localdomain Keepalived_vrrp[1639]: Default interface eth0 does not ....
 8月 04 19:10:17 localhost.localdomain Keepalived_vrrp[1639]: (VI_1): No VIP specified; at lea...d
 8月 04 19:10:17 localhost.localdomain Keepalived_healthcheckers[1638]: Opening file '/etc/keep...
 8月 04 19:10:18 localhost.localdomain Keepalived[1637]: Keepalived_vrrp exited with permanen...ng
 8月 04 19:10:18 localhost.localdomain Keepalived[1637]: Stopping
Hint: Some lines were ellipsized, use -l to show in full.



仮想IP(VIP)の確認。ip addr showコマンドを使います。

[root@localhost ~]# ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master team0 state UP group default qlen 1000
    link/ether 08:00:27:7f:98:3f brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master team0 state UP group default qlen 1000
    link/ether 08:00:27:28:fa:b0 brd ff:ff:ff:ff:ff:ff
4: team0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 08:00:27:7f:98:3f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.218/24 brd 192.168.1.255 scope global noprefixroute team0
       valid_lft forever preferred_lft forever
    inet 192.168.1.219/24 scope global secondary team0       =======>VIP1がある
       valid_lft forever preferred_lft forever
    inet 192.168.1.220/24 scope global secondary team0       =======>VIP2がある
       valid_lft forever preferred_lft forever
    inet6 240f:32:57b8:1:6dfa:9033:f237:58aa/64 scope global noprefixroute dynamic 
       valid_lft 265sec preferred_lft 265sec
    inet6 fe80::75a9:1afe:b90c:2ae8/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever



そして、いよいよLoad Balancerの動作確認。
VIP(192.168.1.219/192.168.1.220)にアクセスしたら、Backend(192.168.1.254:10001/192.168.1.254:10002)へ1:2で分散すればOKです。

Firefoxのネットワークモニター(crtl+shift+eで表示される)を使って確認すると、指定した比率で分散していることが分かります。
f:id:opensourcetech:20200805161014p:plain ※その他、パケットキャプチャしたものをWiresharkなどで確認する・サーバログを確認するなどでも同様のことはできます。

Opensourcetech by Takahiro Kujirai