LinuCエヴァンジェリストの鯨井貴博@opensourcetechです。
今回は、以下の2つの記事を組み合わせて、Oracle XEのクラスター環境を構築してみます。
1.Oracle Database Express Edition (XE) on CentOS7
2.DRBD9/Pacemaker1.1/Corocync2/PCSによるストレージ冗長構成構築 on CentOS7
なお、「2.」の記事の内容が構築済みである前提となります。
一応、設定のみ載せておきます。
[root@db01 ~]# cat /etc/corosync/corosync.conf totem { version: 2 crypto_cipher: none cluster_name: drbd_cluster crypto_hash: none interface { ringnumber: 0 bindnetaddr: 192.168.1.0 mcastport: 5405 ttl: 1 } transport: udpu } logging { fileline: off to_logfile: yes to_syslog: no logfile: /var/log/cluster/corosync.log debug: off timestamp: on logger_subsys { subsys: QUORUM debug: off } } nodelist { node { name: db01 ring0_addr: 192.168.1.222 nodeid: 1 } node { name: db02 ring0_addr: 192.168.1.223 nodeid: 2 } } quorum { provider: corosync_votequorum expected_votes: 2 two_node: 1 } [root@db01 ~]# cat /etc/drbd.conf # You can find an example in /usr/share/doc/drbd.../drbd.conf.example include "drbd.d/global_common.conf"; include "drbd.d/*.res"; [root@db01 ~]# cat /etc/drbd.d/ global_common.conf r0.res [root@db01 ~]# cat /etc/drbd.d/global_common.conf # DRBD is the result of over a decade of development by LINBIT. # In case you need professional services for DRBD or have # feature requests visit http://www.linbit.com global { # usage-count yes; usage-count no; # Decide what kind of udev symlinks you want for "implicit" volumes # (those without explicit volume <vnr> {} block, implied vnr=0): # /dev/drbd/by-resource/<resource>/<vnr> (explicit volumes) # /dev/drbd/by-resource/<resource> (default for implict) udev-always-use-vnr; # treat implicit the same as explicit volumes # minor-count dialog-refresh disable-ip-verification # cmd-timeout-short 5; cmd-timeout-medium 121; cmd-timeout-long 600; } common { handlers { # These are EXAMPLE handlers only. # They may have severe implications, # like hard resetting the node under certain circumstances. # Be careful when choosing your poison. # pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f"; # pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f"; # local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f"; # fence-peer "/usr/lib/drbd/crm-fence-peer.sh"; # split-brain "/usr/lib/drbd/notify-split-brain.sh root"; # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root"; # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k"; # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh; # quorum-lost "/usr/lib/drbd/notify-quorum-lost.sh root"; } startup { # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb } options { auto-promote yes; # cpu-mask on-no-data-accessible # RECOMMENDED for three or more storage nodes with DRBD 9: # quorum majority; # on-no-quorum suspend-io | io-error; } disk { # size on-io-error fencing disk-barrier disk-flushes # disk-drain md-flushes resync-rate resync-after al-extents # c-plan-ahead c-delay-target c-fill-target c-max-rate # c-min-rate disk-timeout } net { # protocol timeout max-epoch-size max-buffers # connect-int ping-int sndbuf-size rcvbuf-size ko-count # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri # after-sb-1pri after-sb-2pri always-asbp rr-conflict # ping-timeout data-integrity-alg tcp-cork on-congestion # congestion-fill congestion-extents csums-alg verify-alg # use-rle } } [root@db01 ~]# cat /etc/drbd.d/r0.res resource r0 { protocol C; disk /dev/sdb1; device /dev/drbd0; meta-disk internal; on db01 { address 192.168.1.222:7789; } on db02 { address 192.168.1.223:7789; } }
Oracle XEのインストール ※両系で実施
[root@db01 ~]# cd /var/tmp/ [root@db01 tmp]# ll 合計 317916 -rw-r--r--. 1 root root 722808 5月 22 18:03 drbd90-utils-9.10.0-1.el7.elrepo.x86_64.rpm -rw-r--r--. 1 root root 287020 5月 22 18:03 kmod-drbd90-9.0.20-1.el7_7.elrepo.x86_64.rpm -rw-r--r--. 1 root root 315891481 5月 29 18:43 oracle-xe-11.2.0-1.0.x86_64.rpm.zip -rw-r--r--. 1 root root 8614312 5月 22 18:27 pacemaker-repo-1.1.21-1.1.el7.x86_64.rpm [root@db01 Disk1]# rpm -ivh oracle-xe-11.2.0-1.0.x86_64.rpm 準備しています... (100%)# (100%)## (100%)### (100%)#### (100%)##### (100%)###### (100%)####### (100%)######## (100%)######### (100%)########## (100%)################################# [100%] /var/tmp/rpm-tmp.ghPJXM: 257 行: [: 18446744073692774399: 整数の式が予期されます /var/tmp/rpm-tmp.ghPJXM: 271 行: [: 18446744073692774399: 整数の式が予期されます 更新中 / インストール中... 1:oracle-xe-11.2.0-1.0 ( 99%)################################# [100%] Executing post-install steps... You must run '/etc/init.d/oracle-xe configure' as the root user to configure the database.
なお、Oracle XEの依存関係にあるbcコマンドが未インストールの場合、以下のように「/u01/app/oracle/product/11.2.0/xe/config/scripts/init.ora 」の32行目が空memory_target=
となり、エラーが起きるので注意。
※一度Oracle XEをアンインストールして、bc・Oracle XEの順でインストールし直せば解決します。
[root@db01 Disk1]# rpm -ivh oracle-xe-11.2.0-1.0.x86_64.rpm 準備しています... (100%)# (100%)## (100%)### (100%)#### (100%)##### (100%)###### (100%)####### (100%)######## (100%)######### (100%)########## (100%)################################# [100%] /var/tmp/rpm-tmp.y5eGlP: 行 186: bc: コマンドが見つかりません /var/tmp/rpm-tmp.y5eGlP: 257 行: [: 18446744073692774399: 整数の式が予期されます /var/tmp/rpm-tmp.y5eGlP: 271 行: [: 18446744073692774399: 整数の式が予期されます 更新中 / インストール中... 1:oracle-xe-11.2.0-1.0 ( 99%)################################# [100%] Executing post-install steps... /var/tmp/rpm-tmp.oN8D23: 行 76: bc: コマンドが見つかりません /var/tmp/rpm-tmp.oN8D23: 行 77: bc: コマンドが見つかりません /var/tmp/rpm-tmp.oN8D23: 78 行: [: -gt: 単項演算子が予期されます /var/tmp/rpm-tmp.oN8D23: 行 82: bc: コマンドが見つかりません You must run '/etc/init.d/oracle-xe configure' as the root user to configure the database. [root@db01 Disk1]# yum -y install bc 読み込んだプラグイン:fastestmirror Loading mirror speeds from cached hostfile * base: ftp-srv2.kddilabs.jp * extras: ftp-srv2.kddilabs.jp * updates: ftp-srv2.kddilabs.jp base | 3.6 kB 00:00:00 extras | 2.9 kB 00:00:00 linux-ha-ja-pacemaker | 2.9 kB 00:00:00 updates | 2.9 kB 00:00:00 依存性の解決をしています --> トランザクションの確認を実行しています。 ---> パッケージ bc.x86_64 0:1.06.95-13.el7 を インストール --> 依存性解決を終了しました。 依存性を解決しました =========================================================================================================== Package アーキテクチャー バージョン リポジトリー 容量 =========================================================================================================== インストール中: bc x86_64 1.06.95-13.el7 base 115 k トランザクションの要約 =========================================================================================================== インストール 1 パッケージ 総ダウンロード容量: 115 k インストール容量: 215 k Downloading packages: bc-1.06.95-13.el7.x86_64.rpm | 115 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction 警告: RPMDB は yum 以外で変更されました。 インストール中 : bc-1.06.95-13.el7.x86_64 [############################################## ] 1/1 検証中 : bc-1.06.95-13.el7.x86_64 1/1 インストール: bc.x86_64 0:1.06.95-13.el7 完了しました! [root@db01 Disk1]# /etc/init.d/oracle-xe configure Oracle Database 11g Express Edition Configuration ------------------------------------------------- This will configure on-boot properties of Oracle Database 11g Express Edition. The following questions will determine whether the database should be starting upon system boot, the ports it will use, and the passwords that will be used for database accounts. Press <Enter> to accept the defaults. Ctrl-C will abort. Specify the HTTP port that will be used for Oracle Application Express [8080]: Specify a port that will be used for the database listener [1521]: Specify a password to be used for database accounts. Note that the same password will be used for SYS and SYSTEM. Oracle recommends the use of different passwords for each database account. This can be done after initial configuration: Confirm the password: Do you want Oracle Database 11g Express Edition to be started on boot (y/n) [y]: Starting Oracle Net Listener...Done Configuring database... Database Configuration failed. Look into /u01/app/oracle/product/11.2.0/xe/config/log for details [root@db01 Disk1]# cat -n /u01/app/oracle/product/11.2.0/xe/config/scripts/init.ora 1 ############################################################################## 2 # Copyright (c) 1991, 2011, Oracle and/or its affiliates. All rights reserved. 3 ############################################################################## 4 5 ########################################### 6 # Cursors and Library Cache 7 ########################################### 8 open_cursors=300 9 10 ########################################### 11 # Database Identification 12 ########################################### 13 db_name=XE 14 15 ########################################### 16 # File Configuration 17 ########################################### 18 control_files=("/u01/app/oracle/oradata/XE/control.dbf") 19 DB_RECOVERY_FILE_DEST=/u01/app/oracle/fast_recovery_area 20 DB_RECOVERY_FILE_DEST_SIZE=10G 21 22 ########################################### 23 # Job Queues 24 ########################################### 25 job_queue_processes=4 26 27 ########################################### 28 # Miscellaneous 29 ########################################### 30 compatible=11.2.0.0.0 31 diagnostic_dest=/u01/app/oracle 32 memory_target= 33 34 ########################################### 35 # Sessions 36 ########################################### 37 sessions=20 38 39 ########################################### 40 # Security and Auditing 41 ########################################### 42 audit_file_dest=/u01/app/oracle/admin/XE/adump 43 remote_login_passwordfile=EXCLUSIVE 44 45 ########################################### 46 # Shared Server 47 ########################################### 48 dispatchers="(PROTOCOL=TCP) (SERVICE=XEXDB)" 49 shared_servers=4 50 51 ########################################### 52 # System Managed Undo and Rollback Segments 53 ########################################### 54 undo_management=AUTO 55 undo_tablespace=UNDOTBS1 [root@db01 Disk1]# cat -n /u01/app/oracle/product/11.2.0/xe/config/log/cloneDBCreation.log 1 begin 2 * 3 ERROR at line 1: 4 ORA-01034: ORACLE not available 5 Process ID: 0 6 Session ID: 0 Serial number: 0 7 8 9 create spfile='/u01/app/oracle/product/11.2.0/xe/dbs/spfileXE.ora' FROM pfile='/u01/app/oracle/product/11.2.0/xe/config/scripts/init.ora' 10 * 11 ERROR at line 1: 12 ORA-01078: failure in processing system parameters 13 LRM-00116: syntax error at 'sessions' following '=' 14 15 16 ORA-01034: ORACLE not available 17 ORA-27101: shared memory realm does not exist 18 Linux-x86_64 Error: 2: No such file or directory 19 ORA-01078: failure in processing system parameters 20 LRM-00109: could not open parameter file '/u01/app/oracle/product/11.2.0/xe/dbs/initXE.ora' 21 select 'utl_recomp_begin: ' || to_char(sysdate, 'HH:MI:SS') from dual 22 * 23 ERROR at line 1: 24 ORA-01034: ORACLE not available 25 Process ID: 0 26 Session ID: 0 Serial number: 0 27 28 29 BEGIN utl_recomp.recomp_serial(); END; 30 31 * 32 ERROR at line 1: 33 ORA-01034: ORACLE not available 34 Process ID: 0 35 Session ID: 0 Serial number: 0 36 37 38 select 'utl_recomp_end: ' || to_char(sysdate, 'HH:MI:SS') from dual 39 * 40 ERROR at line 1: 41 ORA-01034: ORACLE not available 42 Process ID: 0 43 Session ID: 0 Serial number: 0 44 45
ログファイルにも、以下のエラーが出ます。
1 begin 2 * 3 ERROR at line 1: 4 ORA-01034: ORACLE not available 5 Process ID: 0 6 Session ID: 0 Serial number: 0 7 8 9 create spfile='/u01/app/oracle/product/11.2.0/xe/dbs/spfileXE.ora' FROM pfile='/u01/app/oracle/product/11.2.0/xe/config/scripts/init.ora' 10 * 11 ERROR at line 1: 12 ORA-01078: failure in processing system parameters 13 LRM-00116: syntax error at 'sessions' following '='
Oracle XEの環境設定 ※両系で実施
続いて、Oracle XEの環境設定。
[root@db01 Disk1]# /etc/init.d/oracle-xe configure Oracle Database 11g Express Edition Configuration ------------------------------------------------- This will configure on-boot properties of Oracle Database 11g Express Edition. The following questions will determine whether the database should be starting upon system boot, the ports it will use, and the passwords that will be used for database accounts. Press <Enter> to accept the defaults. Ctrl-C will abort. Specify the HTTP port that will be used for Oracle Application Express [8080]: Specify a port that will be used for the database listener [1521]: Specify a password to be used for database accounts. Note that the same password will be used for SYS and SYSTEM. Oracle recommends the use of different passwords for each database account. This can be done after initial configuration: Confirm the password: Do you want Oracle Database 11g Express Edition to be started on boot (y/n) [y]: Starting Oracle Net Listener...Done Configuring database...Done Starting Oracle Database 11g Express Edition instance...Done Installation completed successfully. [root@db01 Disk1]# cat -n /u01/app/oracle/product/11.2.0/xe/config/scripts/init.ora 1 ############################################################################## 2 # Copyright (c) 1991, 2011, Oracle and/or its affiliates. All rights reserved. 3 ############################################################################## 4 5 ########################################### 6 # Cursors and Library Cache 7 ########################################### 8 open_cursors=300 9 10 ########################################### 11 # Database Identification 12 ########################################### 13 db_name=XE 14 15 ########################################### 16 # File Configuration 17 ########################################### 18 control_files=("/u01/app/oracle/oradata/XE/control.dbf") 19 DB_RECOVERY_FILE_DEST=/u01/app/oracle/fast_recovery_area 20 DB_RECOVERY_FILE_DEST_SIZE=10G 21 22 ########################################### 23 # Job Queues 24 ########################################### 25 job_queue_processes=4 26 27 ########################################### 28 # Miscellaneous 29 ########################################### 30 compatible=11.2.0.0.0 31 diagnostic_dest=/u01/app/oracle 32 memory_target=770703360 33 34 ########################################### 35 # Sessions 36 ########################################### 37 sessions=20 38 39 ########################################### 40 # Security and Auditing 41 ########################################### 42 audit_file_dest=/u01/app/oracle/admin/XE/adump 43 remote_login_passwordfile=EXCLUSIVE 44 45 ########################################### 46 # Shared Server 47 ########################################### 48 dispatchers="(PROTOCOL=TCP) (SERVICE=XEXDB)" 49 shared_servers=4 50 51 ########################################### 52 # System Managed Undo and Rollback Segments 53 ########################################### 54 undo_management=AUTO 55 undo_tablespace=UNDOTBS1 [root@db01 Disk1]# cd /u01/app/oracle/product/11.2.0/xe/bin [root@db01 bin]# ll 合計 191312 -rwxr-xr-x. 1 oracle dba 13091 8月 29 2011 adapters -rwxr-xr-x. 1 oracle dba 8614 8月 29 2011 createdb.sh -rwxr-xr-x. 1 oracle dba 523832 8月 29 2011 ctxkbtc -rwxr-xr-x. 1 oracle dba 453648 8月 29 2011 ctxlc -rwxr-xr-x. 1 oracle dba 193080 8月 29 2011 ctxload -rwxr-xr-x. 1 oracle dba 6840 8月 29 2011 cursize -rwxr-xr-x. 1 oracle dba 558768 8月 29 2011 dbfs_client -rwxr-xr-x. 1 oracle dba 6856 8月 29 2011 dbfsize -rwxr-xr-x. 1 oracle dba 2415 8月 29 2011 dbhome -rwxr-xr-x. 1 oracle dba 321904 8月 29 2011 dbv -rwxr-xr-x. 1 oracle dba 340072 8月 29 2011 dg4odbc -rwxr-xr-x. 1 oracle dba 349320 8月 29 2011 dgmgrl -rwxr-xr-x. 1 oracle dba 8296 8月 29 2011 dumpsga -rwxr-xr-x. 1 oracle dba 614568 8月 29 2011 exp -rwxr-xr-x. 1 oracle dba 126440 8月 29 2011 expdp -rwxr-xr-x. 1 oracle dba 608048 8月 29 2011 extjob -rwxr-xr-x. 1 oracle dba 608048 8月 29 2011 extjobo -rwxr-xr-x. 1 oracle dba 5944 8月 29 2011 extproc -rwxr-xr-x. 1 oracle dba 300968 8月 29 2011 imp -rwxr-xr-x. 1 oracle dba 136904 8月 29 2011 impdp -rwxr-xr-x. 1 oracle dba 368456 8月 29 2011 kgmgr -rwxr-xr-x. 1 oracle dba 12536 8月 29 2011 loadpsp -rwxr-xr-x. 1 oracle dba 91016 8月 29 2011 lsnrctl -rwxr-xr-x. 1 oracle dba 511824 8月 29 2011 lxchknlb -rwxr-xr-x. 1 oracle dba 7960 8月 29 2011 mapsga -rwxr-xr-x. 1 oracle dba 6872 8月 29 2011 maxmem -rwxr-xr-x. 1 oracle dba 58968 8月 29 2011 nid -rwxr-xr-x. 1 oracle dba 16759 8月 29 2011 nls_lang.sh -rwsr-s--x. 1 oracle dba 165700472 8月 29 2011 oracle -rwxr-xr-x. 1 oracle dba 156 8月 29 2011 oracle_env.csh -rwxr-xr-x. 1 oracle dba 156 8月 29 2011 oracle_env.sh -rwxr-xr-x. 1 oracle dba 52800 8月 29 2011 oradism -rwxr-xr-x. 1 oracle dba 6183 8月 29 2011 oraenv -rwxr-xr-x. 1 oracle dba 15232 8月 29 2011 orapwd -rwxr-xr-x. 1 oracle dba 19936 8月 29 2011 osdbagrp -rwxr-xr-x. 1 oracle dba 20040 8月 29 2011 osh -rwxr-xr-x. 1 oracle dba 946 8月 29 2011 ott -rwxr-xr-x. 1 oracle dba 14087984 8月 29 2011 rman -rwxr-xr-x. 1 oracle dba 1313712 8月 29 2011 sqlldr -rwxr-x--x. 1 oracle dba 4968 8月 29 2011 sqlplus -rwxr-xr-x. 1 oracle dba 13352 8月 29 2011 sysresv -rwxr-xr-x. 1 oracle dba 85096 8月 29 2011 tkprof -rwxr-xr-x. 1 oracle dba 738440 8月 29 2011 tnslsnr -rwxr-xr-x. 1 oracle dba 11040 8月 29 2011 tnsping -rwxr-xr-x. 1 oracle dba 16496 8月 29 2011 trcroute -rwxr-xr-x. 1 oracle dba 144784 8月 29 2011 unzip -rwxr-xr-x. 1 oracle dba 7097808 8月 29 2011 wrap -rwxr-xr-x. 1 oracle dba 208384 8月 29 2011 zip [root@db01 bin]# cat -n oracle_env.sh 1 export ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe 2 export ORACLE_SID=XE 3 export NLS_LANG=`$ORACLE_HOME/bin/nls_lang.sh` 4 export PATH=$ORACLE_HOME/bin:$PATH [root@db01 bin]# . ./oracle_env.sh [root@db01 bin]# echo $ORACLE_HOME /u01/app/oracle/product/11.2.0/xe [root@db01 ~]# cat /etc/passwd root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt mail:x:8:12:mail:/var/spool/mail:/sbin/nologin operator:x:11:0:operator:/root:/sbin/nologin games:x:12:100:games:/usr/games:/sbin/nologin ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin nobody:x:99:99:Nobody:/:/sbin/nologin systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin dbus:x:81:81:System message bus:/:/sbin/nologin polkitd:x:999:998:User for polkitd:/:/sbin/nologin sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin postfix:x:89:89::/var/spool/postfix:/sbin/nologin chrony:x:998:996::/var/lib/chrony:/sbin/nologin oracle:x:1000:1000:oracle:/home/oracle:/bin/bash rpc:x:32:32:Rpcbind Daemon:/var/lib/rpcbind:/sbin/nologin rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin tss:x:59:59:Account used by the trousers package to sandbox the tcsd daemon:/dev/null:/sbin/nologin hacluster:x:189:189:cluster user:/home/hacluster:/sbin/nologin
各ユーザ(root/oracle)のホームディレクトリや/etc/skelにある.bashrc・.bash_profileにも環境変数を追記しておきます。
続いて、Oracle XE起動時に自動起動するインスタンス(XE)の設定をします。
[root@db01 skel]# cat /etc/oratab XE:/u01/app/oracle/product/11.2.0/xe:N [root@db01 skel]# vi /etc/oratab [root@db01 skel]# vi /etc/oratab cat XE:/u01/app/oracle/product/11.2.0/xe:Y
Oracle XEに関するクラスター設定 ※Primaryとなる側でのみ実施
まず、現状のクラスター設定および動作確認。
[root@db01 skel]# crm configure show node 1: db01 node 2: db02 \ attributes primitive FS Filesystem \ params device="/dev/drbd0" directory="/mnt/drbd" fstype=xfs \ op monitor interval=10s on-fail=restart timeout=60s \ op start interval=0s on-fail=restart timeout=60s \ op stop interval=0s on-fail=block timeout=60s primitive VIP IPaddr2 \ params cidr_netmask=24 ip=192.168.1.221 \ op monitor interval=10s \ op start interval=0s timeout=20s \ op stop interval=0s timeout=20s primitive prmPing ocf:pacemaker:ping \ params attempts=5 debug=true host_list="192.168.1.222 192.168.1.223" multiplier=1 name=default_ping _set timeout=50 \ op monitor interval=10s on-fail=restart timeout=60s \ op start interval=0s on-fail=restart timeout=60s \ op stop interval=0s on-fail=ignore timeout=60s primitive vipcheck VIPcheck \ params count=3 target_ip=192.168.1.221 wait=5 \ op monitor interval=10s on-fail=restart start-delay=0 timeout=60s \ op start interval=0s on-fail=restart timeout=60s \ : op stop interval=0s on-fail=ignore timeout=60s group Group-FS FS vipcheck VIP clone prmPing-clone prmPing property cib-bootstrap-options: \ have-watchdog=false \ dc-version=1.1.21-1.el7-f14e36f \ cluster-infrastructure=corosync \ stonith-enabled=false \ no-quorum-policy=ignore \ cluster-name=drbd_cluster rsc_defaults rsc_defaults-options: \ migration-threshold=1 [root@db01 ~]# pcs status Cluster name: drbd_cluster WARNINGS: Corosync and pacemaker node names do not match (IPs used in setup?) Stack: corosync Current DC: db01 (version 1.1.21-1.el7-f14e36f) - partition with quorum Last updated: Fri May 29 15:50:40 2020 Last change: Thu May 28 20:46:10 2020 by root via cibadmin on db01 2 nodes configured 5 resources configured Online: [ db01 db02 ] Full list of resources: Resource Group: Group-FS FS (ocf::heartbeat:Filesystem): Started db01 vipcheck (ocf::heartbeat:VIPcheck): Started db01 VIP (ocf::heartbeat:IPaddr2): Started db01 Clone Set: prmPing-clone [prmPing] Started: [ db01 db02 ] Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
以下にある、oralsnr
やoracle
を使います。
[root@db01 ~]# find / -name oralsnr /usr/lib/ocf/resource.d/heartbeat/oralsnr [root@db01 ~]# ls /usr/lib/ocf/resource.d/pacemaker/ ClusterMon HealthCPU Stateful SystemHealth controld ifspeed ping remote Dummy HealthSMART SysInfo attribute diskd o2cb pingd [root@db01 ~]# ls /usr/lib/ocf/resource.d/pacemaker/heartbeat/ AoEtarget SendArp conntrackd kamailio podman AudibleAlarm ServeRAID db2 ldirectord portblock CTDB SphinxSearchDaemon dhcpd lvmlockd postfix ClusterMon Squid dnsupdate lxc pound Delay Stateful docker lxd-info proftpd Dummy SysInfo dovecot machine-info rabbitmq-cluster EvmsSCC VIPArip eDir88 mariadb redis Evmsd VIPcheck ethmonitor minio rkt Filesystem VirtualDomain exportfs mpathpersist rsyncd ICP WAS fio mysql rsyslog IPaddr WAS6 galera mysql-proxy scsi2reservation IPaddr2 WinPopup garbd nagios sfex IPsrcaddr Xen gcp-pd-move named sg_persist IPv6addr Xinetd gcp-vpc-move-ip nfsnotify slapd LVM ZFS gcp-vpc-move-route nfsserver sybaseASE LVM-activate aliyun-vpc-move-ip gcp-vpc-move-vip nginx symlink LinuxSCSI anything hulft openstack-cinder-volume syslog-ng MailTo apache iSCSILogicalUnit openstack-floating-ip tomcat ManageRAID asterisk iSCSITarget openstack-info varnish ManageVE aws-vpc-move-ip ids oraasm vdo-vol NodeUtilization aws-vpc-route53 iface-bridge oracle vmware Pure-FTPd awseip iface-vlan oralsnr vsftpd Raid1 awsvip ipsec ovsmonitor zabbixserver Route azure-events iscsi pgagent SAPDatabase azure-lb jboss pgsql SAPInstance clvm jira pingd
クラスター設定追加。
[root@db01 ~]# pcs resource create OraLsnr ocf:heartbeat:oralsnr sid=XE op start interval="0s" timeout="60ss" op monitor interval="30s" timeout="60s" depth=0 op stop interval="0s" timeout="60s" [root@db01 ~]# pcs constraint colocation add OraLsnr-with-FS FS OraLsnr [root@db01 ~]# pcs constraint order set FS OraLsnr [root@db01 ~]# pcs resource create OraSrv ocf:heartbeat:oracle sid=XE home="/u01/app/oracle/product/11.2.0 /xe" user=oracle op start interval="0s" timeout="120s" op monitor interval="120s" timeout="60s" op stop int erval="0s" timeout="120s" [root@db01 ~]# pcs constraint colocation add OraLsnr OraSrv [root@db01 ~]# pcs constraint order set OraLsnr OraSrv
[root@db01 ~]# pcs config Cluster Name: drbd_cluster Corosync Nodes: 192.168.1.222 192.168.1.223 Pacemaker Nodes: db01 db02 Resources: Group: Group-FS Resource: FS (class=ocf provider=heartbeat type=Filesystem) Attributes: device=/dev/drbd0 directory=/mnt/drbd fstype=xfs run_fsck=no Operations: monitor interval=15s timeout=40s (FS-monitor-15s) start interval=0s timeout=60s (FS-start-interval-0s) stop interval=0s on-fail=ignore timeout=60s (FS-stop-interval-0s) Resource: vipcheck (class=ocf provider=heartbeat type=VIPcheck) Attributes: count=3 target_ip=192.168.1.221 wait=5 Operations: monitor interval=10s on-fail=restart start-delay=0 timeout=60s (vipcheck-monitor-interval-10s) start interval=0s on-fail=restart timeout=60s (vipcheck-start-interval-0s) stop interval=0s on-fail=ignore timeout=60s (vipcheck-stop-interval-0s) Resource: VIP (class=ocf provider=heartbeat type=IPaddr2) Attributes: cidr_netmask=24 ip=192.168.1.221 nic=enp0s3 Operations: monitor interval=30s (VIP-monitor-30s) start interval=0s timeout=20s (VIP-start-interval-0s) stop interval=0s timeout=20s (VIP-stop-interval-0s) Resource: prmPing (class=ocf provider=pacemaker type=ping) Attributes: attempts=5 debug=true host_list=192.168.1.1 multiplier=1 name=default_ping_set timeout=50s Operations: monitor interval=10s on-fail=restart timeout=60s (prmPing-monitor-interval-10s) start interval=0s on-fail=restart timeout=60s (prmPing-start-interval-0s) stop interval=0s on-fail=fence timeout=60s (prmPing-stop-interval-0s) Resource: OraLsnr (class=ocf provider=heartbeat type=oralsnr) Attributes: sid=XE Operations: monitor interval=30s timeout=120s depth=0 (OraLsnr-monitor-interval-30s) start interval=0s timeout=120s (OraLsnr-start-interval-0s) stop interval=0s timeout=120s (OraLsnr-stop-interval-0s) Resource: OraSrv (class=ocf provider=heartbeat type=oracle) Attributes: home=/u01/app/oracle/product/11.2.0/xe sid=XE user=oracle Operations: monitor interval=120s timeout=60s (OraSrv-monitor-interval-120s) start interval=0s timeout=120s (OraSrv-start-interval-0s) stop interval=0s timeout=120s (OraSrv-stop-interval-0s) Stonith Devices: Fencing Levels: Location Constraints: Ordering Constraints: Resource Sets: set FS OraLsnr (id:pcs_rsc_set_FS_OraLsnr) (id:pcs_rsc_order_set_FS_OraLsnr) set OraLsnr OraSrv (id:pcs_rsc_set_OraLsnr_OraSrv) (id:pcs_rsc_order_set_OraLsnr_OraSrv) Colocation Constraints: FS with OraLsnr (score:INFINITY) (id:colocation-FS-OraLsnr-INFINITY) OraLsnr with OraSrv (score:INFINITY) (id:colocation-OraLsnr-OraSrv-INFINITY) Ticket Constraints: Alerts: No alerts defined Resources Defaults: migration-threshold=1 resource-stickiness=INFINITY Operations Defaults: No defaults set Cluster Properties: cluster-infrastructure: corosync cluster-name: drbd_cluster dc-version: 1.1.21-1.el7-f14e36f have-watchdog: false last-lrm-refresh: 1560163421 no-quorum-policy: ignore stonith-enabled: false Quorum: Options:
[root@db01 skel]# crm configure show node 1: db01 \ attributes standby=off node 2: db02 \ attributes primitive FS Filesystem \ params run_fsck=no device="/dev/drbd0" directory="/mnt/drbd" fstype=xfs \ op monitor interval=15s timeout=40s \ op start interval=0s timeout=60s \ op stop interval=0s on-fail=ignore timeout=60s primitive OraLsnr oralsnr \ params sid=XE \ op monitor interval=30s timeout=120s depth=0 \ op start interval=0s timeout=120s \ op stop interval=0s timeout=120s primitive OraSrv oracle \ params home="/u01/app/oracle/product/11.2.0/xe" sid=XE user=oracle \ op monitor interval=120s timeout=60s \ op start interval=0s timeout=120s \ op stop interval=0s timeout=120s primitive VIP IPaddr2 \ params cidr_netmask=24 ip=192.168.1.221 nic=enp0s3 \ op monitor interval=30s \ op start interval=0s timeout=20s \ op stop interval=0s timeout=20s primitive prmPing ocf:pacemaker:ping \ params attempts=5 debug=true host_list=192.168.1.1 multiplier=1 name=default_ping_set timeout=50s \ op monitor interval=10s on-fail=restart timeout=60s \ op start interval=0s on-fail=restart timeout=60s \ op stop interval=0s on-fail=fence timeout=60s primitive vipcheck VIPcheck \ params count=3 target_ip=192.168.1.221 wait=5 \ op monitor interval=10s on-fail=restart start-delay=0 timeout=60s \ op start interval=0s on-fail=restart timeout=60s \ op stop interval=0s on-fail=ignore timeout=60s group Group-FS FS vipcheck VIP prmPing colocation colocation-FS-OraLsnr-INFINITY inf: FS OraLsnr colocation colocation-OraLsnr-OraSrv-INFINITY inf: OraLsnr OraSrv order pcs_rsc_order_set_FS_OraLsnr _rsc_set_ FS OraLsnr order pcs_rsc_order_set_OraLsnr_OraSrv _rsc_set_ OraLsnr OraSrv property cib-bootstrap-options: \ have-watchdog=false \ dc-version=1.1.21-1.el7-f14e36f \ cluster-infrastructure=corosync \ stonith-enabled=false \ no-quorum-policy=ignore \ cluster-name=drbd_cluster \ last-lrm-refresh=1560163421 rsc_defaults rsc_defaults-options: \ resource-stickiness=INFINITY \ migration-threshold=1
[root@db01 ~]# crm configure save /var/tmp/oracle_pacemaker.crm [root@db01 ~]# cat /var/tmp/oracle_pacemaker.crm node 1: db01 \ attributes standby=off node 2: db02 \ attributes primitive FS Filesystem \ params run_fsck=no device="/dev/drbd0" directory="/mnt/drbd" fstype=xfs \ op monitor interval=15s timeout=40s \ op start interval=0s timeout=60s \ op stop interval=0s on-fail=ignore timeout=60s primitive OraLsnr oralsnr \ params sid=XE \ op monitor interval=30s timeout=120s depth=0 \ op start interval=0s timeout=120s \ op stop interval=0s timeout=120s primitive OraSrv oracle \ params home="/u01/app/oracle/product/11.2.0/xe" sid=XE user=oracle \ op monitor interval=120s timeout=60s \ op start interval=0s timeout=120s \ op stop interval=0s timeout=120s primitive VIP IPaddr2 \ params cidr_netmask=24 ip=192.168.1.221 nic=enp0s3 \ op monitor interval=30s \ op start interval=0s timeout=20s \ op stop interval=0s timeout=20s primitive prmPing ocf:pacemaker:ping \ params attempts=5 debug=true host_list=192.168.1.1 multiplier=1 name=default_ping_set timeout=50s \ op monitor interval=10s on-fail=restart timeout=60s \ op start interval=0s on-fail=restart timeout=60s \ op stop interval=0s on-fail=fence timeout=60s primitive vipcheck VIPcheck \ params count=3 target_ip=192.168.1.221 wait=5 \ op monitor interval=10s on-fail=restart start-delay=0 timeout=60s \ op start interval=0s on-fail=restart timeout=60s \ op stop interval=0s on-fail=ignore timeout=60s group Group-FS FS vipcheck VIP prmPing colocation colocation-FS-OraLsnr-INFINITY inf: FS OraLsnr colocation colocation-OraLsnr-OraSrv-INFINITY inf: OraLsnr OraSrv order pcs_rsc_order_set_FS_OraLsnr _rsc_set_ FS OraLsnr order pcs_rsc_order_set_OraLsnr_OraSrv _rsc_set_ OraLsnr OraSrv property cib-bootstrap-options: \ have-watchdog=false \ dc-version=1.1.21-1.el7-f14e36f \ cluster-infrastructure=corosync \ stonith-enabled=false \ no-quorum-policy=ignore \ cluster-name=drbd_cluster \ last-lrm-refresh=1560163421 rsc_defaults rsc_defaults-options: \ resource-stickiness=INFINITY \ migration-threshold=1 [root@db01 ~]# cat -n /var/tmp/oracle_pacemaker.crm 1 node 1: db01 \ 2 attributes standby=off 3 node 2: db02 \ 4 attributes 5 primitive FS Filesystem \ 6 params run_fsck=no device="/dev/drbd0" directory="/mnt/drbd" fstype=xfs \ 7 op monitor interval=15s timeout=40s \ 8 op start interval=0s timeout=60s \ 9 op stop interval=0s on-fail=ignore timeout=60s 10 primitive OraLsnr oralsnr \ 11 params sid=XE \ 12 op monitor interval=30s timeout=120s depth=0 \ 13 op start interval=0s timeout=120s \ 14 op stop interval=0s timeout=120s 15 primitive OraSrv oracle \ 16 params home="/u01/app/oracle/product/11.2.0/xe" sid=XE user=oracle \ 17 op monitor interval=120s timeout=60s \ 18 op start interval=0s timeout=120s \ 19 op stop interval=0s timeout=120s 20 primitive VIP IPaddr2 \ 21 params cidr_netmask=24 ip=192.168.1.221 nic=enp0s3 \ 22 op monitor interval=30s \ 23 op start interval=0s timeout=20s \ 24 op stop interval=0s timeout=20s 25 primitive prmPing ocf:pacemaker:ping \ 26 params attempts=5 debug=true host_list=192.168.1.1 multiplier=1 name=default_ping_set timeout=50s \ 27 op monitor interval=10s on-fail=restart timeout=60s \ 28 op start interval=0s on-fail=restart timeout=60s \ 29 op stop interval=0s on-fail=fence timeout=60s 30 primitive vipcheck VIPcheck \ 31 params count=3 target_ip=192.168.1.221 wait=5 \ 32 op monitor interval=10s on-fail=restart start-delay=0 timeout=60s \ 33 op start interval=0s on-fail=restart timeout=60s \ 34 op stop interval=0s on-fail=ignore timeout=60s 35 group Group-FS FS vipcheck VIP prmPing 36 colocation colocation-FS-OraLsnr-INFINITY inf: FS OraLsnr 37 colocation colocation-OraLsnr-OraSrv-INFINITY inf: OraLsnr OraSrv 38 order pcs_rsc_order_set_FS_OraLsnr _rsc_set_ FS OraLsnr 39 order pcs_rsc_order_set_OraLsnr_OraSrv _rsc_set_ OraLsnr OraSrv 40 property cib-bootstrap-options: \ 41 have-watchdog=false \ 42 dc-version=1.1.21-1.el7-f14e36f \ 43 cluster-infrastructure=corosync \ 44 stonith-enabled=false \ 45 no-quorum-policy=ignore \ 46 cluster-name=drbd_cluster \ 47 last-lrm-refresh=1560163421 48 rsc_defaults rsc_defaults-options: \ 49 resource-stickiness=INFINITY \ 50 migration-threshold=1
設定の反映等確認。
[root@db01 ~]# pcs status Cluster name: drbd_cluster WARNINGS: Corosync and pacemaker node names do not match (IPs used in setup?) Stack: corosync Current DC: db01 (version 1.1.21-1.el7-f14e36f) - partition with quorum Last updated: Fri May 29 16:11:51 2020 Last change: Fri May 29 16:11:37 2020 by root via cibadmin on db01 2 nodes configured 7 resources configured Online: [ db01 db02 ] Full list of resources: Resource Group: Group-FS FS (ocf::heartbeat:Filesystem): Started db01 vipcheck (ocf::heartbeat:VIPcheck): Started db01 VIP (ocf::heartbeat:IPaddr2): Started db01 Clone Set: prmPing-clone [prmPing] Started: [ db01 db02 ] OraLsnr (ocf::heartbeat:oralsnr): Started db01 OraSrv (ocf::heartbeat:oracle): Started db01 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@db01 ~]# ss -tan State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 128 *:111 *:* LISTEN 0 128 *:22 *:* ESTAB 0 0 192.168.1.222:22 192.168.1.131:50173 ESTAB 0 0 192.168.1.222:7789 192.168.1.223:58679 ESTAB 0 0 192.168.1.222:54307 192.168.1.223:7789 ESTAB 0 0 192.168.1.222:53214 192.168.1.222:1521 LISTEN 0 100 [::1]:25 [::]:* LISTEN 0 128 [::]:33221 [::]:* LISTEN 0 128 [::]:111 [::]:* LISTEN 0 128 [::]:8080 [::]:* LISTEN 0 128 [::]:2224 [::]:* LISTEN 0 128 [::]:1521 [::]:* LISTEN 0 128 [::]:22 [::]:* TIME-WAIT 0 0 [::ffff:192.168.1.222]:1521 [::ffff:192.168.1.222]:53204 TIME-WAIT 0 0 [::ffff:192.168.1.222]:1521 [::ffff:192.168.1.222]:53202 ESTAB 0 0 [::ffff:192.168.1.222]:1521 [::ffff:192.168.1.222]:53214
DB接続の確認。
[root@db01 ~]# pcs status Cluster name: drbd_cluster WARNINGS: Corosync and pacemaker node names do not match (IPs used in setup?) Stack: corosync Current DC: db01 (version 1.1.21-1.el7-f14e36f) - partition with quorum Last updated: Fri May 29 16:11:51 2020 Last change: Fri May 29 16:11:37 2020 by root via cibadmin on db01 2 nodes configured 7 resources configured Online: [ db01 db02 ] Full list of resources: Resource Group: Group-FS FS (ocf::heartbeat:Filesystem): Started db01 vipcheck (ocf::heartbeat:VIPcheck): Started db01 VIP (ocf::heartbeat:IPaddr2): Started db01 Clone Set: prmPing-clone [prmPing] Started: [ db01 db02 ] OraLsnr (ocf::heartbeat:oralsnr): Started db01 OraSrv (ocf::heartbeat:oracle): Started db01 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@db01 ~]# ss -tan State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 100 127.0.0.1:25 *:* LISTEN 0 128 *:111 *:* LISTEN 0 128 *:22 *:* ESTAB 0 0 192.168.1.222:22 192.168.1.131:50173 ESTAB 0 0 192.168.1.222:7789 192.168.1.223:58679 ESTAB 0 0 192.168.1.222:54307 192.168.1.223:7789 ESTAB 0 0 192.168.1.222:53214 192.168.1.222:1521 LISTEN 0 100 [::1]:25 [::]:* LISTEN 0 128 [::]:33221 [::]:* LISTEN 0 128 [::]:111 [::]:* LISTEN 0 128 [::]:8080 [::]:* LISTEN 0 128 [::]:2224 [::]:* LISTEN 0 128 [::]:1521 [::]:* LISTEN 0 128 [::]:22 [::]:* TIME-WAIT 0 0 [::ffff:192.168.1.222]:1521 [::ffff:192.168.1.222]:53204 TIME-WAIT 0 0 [::ffff:192.168.1.222]:1521 [::ffff:192.168.1.222]:53202 ESTAB 0 0 [::ffff:192.168.1.222]:1521 [::ffff:192.168.1.222]:53214
大丈夫そうですね♪
障害試験
では、続いて障害試験。
Primaryで動作している側をshutdownして切り替え、起動後の切り戻りをチェックします。
[root@db01 ~]# reboot
db01の再起動後、状態確認。
[root@db01 ~]# pcs status Cluster name: drbd_cluster WARNINGS: Corosync and pacemaker node names do not match (IPs used in setup?) Stack: corosync Current DC: db02 (version 1.1.21-1.el7-f14e36f) - partition with quorum Last updated: Fri May 29 18:17:17 2020 Last change: Fri May 29 16:11:37 2020 by root via cibadmin on db01 2 nodes configured 7 resources configured Online: [ db01 db02 ] Full list of resources: Resource Group: Group-FS FS (ocf::heartbeat:Filesystem): Started db02 vipcheck (ocf::heartbeat:VIPcheck): Started db02 VIP (ocf::heartbeat:IPaddr2): Started db02 Clone Set: prmPing-clone [prmPing] Started: [ db01 ] Stopped: [ db02 ] OraLsnr (ocf::heartbeat:oralsnr): Started db02 OraSrv (ocf::heartbeat:oracle): Started db02 Failed Resource Actions: * VIP_monitor_10000 on db01 'unknown error' (1): call=67, status=Timed Out, exitreason='', last-rc-change='Fri May 29 18:16:56 2020', queued=0ms, exec=0ms * prmPing_monitor_10000 on db02 'unknown error' (1): call=122, status=Timed Out, exitreason='', last-rc-change='Fri May 29 18:16:55 2020', queued=0ms, exec=0ms Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
2系に切り替わってますね。
※「Failed Resource Actions;」は、こちらにある方法で消しておいてください。
では、1系に戻します。
[root@db01 ~]# pcs cluster standby db02 [root@db01 ~]# pcs status Cluster name: drbd_cluster WARNINGS: Corosync and pacemaker node names do not match (IPs used in setup?) Stack: corosync Current DC: db02 (version 1.1.21-1.el7-f14e36f) - partition with quorum Last updated: Fri May 29 18:19:20 2020 Last change: Fri May 29 18:18:56 2020 by root via cibadmin on db01 2 nodes configured 7 resources configured Node db02: standby Online: [ db01 ] Full list of resources: Resource Group: Group-FS FS (ocf::heartbeat:Filesystem): Started db01 vipcheck (ocf::heartbeat:VIPcheck): Started db01 VIP (ocf::heartbeat:IPaddr2): Started db01 Clone Set: prmPing-clone [prmPing] Started: [ db01 ] Stopped: [ db02 ] OraLsnr (ocf::heartbeat:oralsnr): Started db01 OraSrv (ocf::heartbeat:oracle): Started db01 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled [root@db01 ~]# pcs cluster unstandby db02 [root@db01 ~]# pcs status Cluster name: drbd_cluster WARNINGS: Corosync and pacemaker node names do not match (IPs used in setup?) Stack: corosync Current DC: db02 (version 1.1.21-1.el7-f14e36f) - partition with quorum Last updated: Fri May 29 18:19:26 2020 Last change: Fri May 29 18:19:24 2020 by root via cibadmin on db01 2 nodes configured 7 resources configured Online: [ db01 db02 ] Full list of resources: Resource Group: Group-FS FS (ocf::heartbeat:Filesystem): Started db01 vipcheck (ocf::heartbeat:VIPcheck): Started db01 VIP (ocf::heartbeat:IPaddr2): Started db01 Clone Set: prmPing-clone [prmPing] Started: [ db01 ] Stopped: [ db02 ] OraLsnr (ocf::heartbeat:oralsnr): Started db01 OraSrv (ocf::heartbeat:oracle): Started db01 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
大丈夫ですね。
また、クラスター設定の再編集や保存・読み込みが必要な場合には、以下の記事にある方法で実施可能なので、
恐れる必要はありません。
-Pacemakerの設定を編集する(crm configure edit)
-Pacemakerの設定を保存する/読み込む(crm configure save/crm configure load)