UDN-企业互联网技术人气社区

板块导航

浏览  : 4361
回复  : 0

[讨论交流] MySQL HA方案之MySQL半复制+MHA+Keepalived+Atlas+LVS

[复制链接]
胭脂粉的头像 楼主
发表于 2016-3-24 10:23:34 | 显示全部楼层 |阅读模式
  目前Mysql高可用的方案有好多,比如MMM,heartbeat+drbd,Cluster等,还有percona的GaleraCluster等。这些高可用软件各有优劣。在进行高可用方案选择时,主要是看业务还有对数据一致性方面的要求。本文探讨的是MHA这种方案。

  至于DAL层,也有很多方案,现在主流的一是在应用层写数据库路由,当然这个效率也是最快的,但是最大的缺陷就是运维难度大,技术难度也大,需要有强大的技术团队支持。第二个方案是proxy中间件,现在开源的中间件很多,比如TDDL、Cobar、Atlas、MyCat、MysqlProxy、Oceanus等等,中间件的最大的好处是对数据层解耦,减轻了运维难度,当然在服务器层与数据库层加了一层proxy,使得效率明显不如直接访问数据库。按业务实际需求选择合适的方案,在本篇文章中,选的是360开源的DAL中间件Atlas,详细信息官方文档(https://github.com/Qihoo360/Atlas)以描述的很清楚,这里不做过多的描述。

  现在继续讨论MHA,MHA(MasterHighAvailability)目前在MySQL高可用方面是一个相对成熟的解决方案,它由日本DeNA公司youshimaton(现就职于Facebook公司)开发,是一套优秀的作为MySQL高可用性环境下故障切换和主从提升的高可用软件。在MySQL故障切换过程中,MHA能做到在0~30秒之内自动完成数据库的故障切换操作,并且在进行故障切换的过程中,MHA能在最大程度上保证数据的一致性,以达到真正意义上的高可用。

  该软件由两部分组成:MHAManager(管理节点)和MHANode(数据节点)。MHAManager可以单独部署在一台独立的机器上管理多个master-slave集群,也可以部署在一台slave节点上。MHANode运行在每台MySQL服务器上,MHAManager会定时探测集群中的master节点,当master出现故障时,它可以自动将最新数据的slave提升为新的master,然后将所有其他的slave重新指向新的master。整个故障转移过程对应用程序完全透明。

  在MHA自动故障切换过程中,MHA试图从宕机的主服务器上保存二进制日志,最大程度的保证数据的不丢失,但这并不总是可行的。例如,如果主服务器硬件故障或无法通过ssh访问,MHA没法保存二进制日志,只进行故障转移而丢失了最新的数据。使用MySQL5.5的半同步复制,可以大大降低数据丢失的风险。MHA可以与半同步复制结合起来。如果只有一个slave已经收到了最新的二进制日志,MHA可以将最新的二进制日志应用于其他所有的slave服务器上,因此可以保证所有节点的数据一致性。

  我们自己使用其实也可以使用1主1从,但是master主机宕机后无法切换,以及无法补全binlog。master的mysqld进程crash后,还是可以切换成功,以及补全binlog的。

  
2015-06-18_172304.jpg


    现在说下架构图的层级关系,最底层的是mysql主从,其中有从机作为备选的主机,备选主机与主机之间由keepalived维护着心跳。每个mysqlserver都是MHAnode,由MHAManager监控着Mysql的节点,当MHAManager监控(keepalived的心跳检测)到主机出现问题,会切换到备选主机,并将备选主机提升为新主机,其他的从机成为新主机的从机,并且VIP也漂移到新主机,这样就实现了Mysql的HA。db上面的一层是AtlasProxy,Atlas主要的功能是读写分离、从库的负载均衡、自动分表、自动摘除宕机的DB、DBA可平滑的上下线DB及IP过滤,其中读写分离及自动摘除宕机的DB是当初选择该中间件的最主要原因,自动摘除宕机的从机保障了整个架构系统持续性,而不影响业务的正常运行。后期根据数据量的增大可以切换到Mycat这个方案,支持分库分表、主从切换等等,并且社区活跃度也很高,感兴趣的可以查看下官方文档(https://github.com/MyCATApache)。回到正题,这里有两个Atlas做HA,上层的LVS做Atlas的负载均衡,两个LVS之间利用keepalived心跳实现热主备模式,防止出现单点故障。再上层的就是应用server,这里就不在讨论了。

  角色

  角色ip地址主机名

  MHAManger192.168.3.123B-dev23

  MysqlMaster192.168.3.110B-dev10

  Mysqlslave1192.168.3.115B-dev15

  Mysqlslave2192.168.3.117B-dev17

  Candicatemaster192.168.3.118B-dev18

  Atlas1192.168.3.124B-dev24

  Atlas2192.168.3.125B-dev25

  LVSMaster192.168.3.119B-dev19

  LVSBackup192.168.3.120B-dev20

  MHA

  1.MHA的基本原理

  (1)从宕机崩溃的master保存二进制日志事件(binlogevents);

  (2)识别含有最新更新的slave;

  (3)应用差异的中继日志(relaylog)到其他的slave;

  (4)应用从master保存的二进制日志事件(binlogevents);

  (5)提升一个slave为新的master;

  (6)使其他的slave连接新的master进行复制;

  MHA软件由两部分组成,Manager工具包和Node工具包,具体的说明如下。

  Manager工具包主要包括以下几个工具:

  masterha_check_ssh检查MHA的SSH配置状况

  masterha_check_repl检查MySQL复制状况

  masterha_manger启动MHA

  masterha_check_status检测当前MHA运行状态

  masterha_master_monitor检测master是否宕机

  masterha_master_switch控制故障转移(自动或者手动)

  masterha_conf_host添加或删除配置的server信息

  Node工具包(这些工具通常由MHAManager的脚本触发,无需人为操作)主要包括以下几个工具:

  save_binary_logs保存和复制master的二进制日志

  apply_diff_relay_logs识别差异的中继日志事件并将其差异的事件应用于其他的slave

  filter_mysqlbinlog去除不必要的ROLLBACK事件(MHA已不再使用这个工具)

  purge_relay_logs清除中继日志(不会阻塞SQL线程)

  注意:

  为了尽可能的减少主库硬件损坏宕机造成的数据丢失,因此在配置MHA的同时建议配置成MySQL5.5的半同步复制,关于半同步复制原理及安装流程下面会详细介绍。

  2.部署MHA

  接下来部署MHA,具体的搭建环境如下:

  其中master对外提供写服务,slave提供相关的读服务,一旦master宕机,将会把备选master提升为新的master,slave指向新的master。

  (1)在所有节点安装MHAnode所需的perl模块(DBD:mysql),安装脚本如下:

  [root@192.168.3.110~]#catinstall.sh

  #!/bin/bash

  wgethttp://xrl.us/cpanm--no-check-certificate

  mvcpanm/usr/bin

  chmod755/usr/bin/cpanm

  cat>/root/list<<EOF

  installDBD::mysql

  EOF

  forpackagein`cat/root/list`

  do

  cpanm$package

  done

  [root@192.168.3.110~]#

  如果有安装epel源,也可以使用yum安装

  yuminstallperl-DBD-MySQL-y

  (2)在所有的节点安装mhanode:

  wgethttp://mysql-master-ha.googlecod ... ql-node-0.53.tar.gz

  tarxfmha4mysql-node-0.53.tar.gz

  cdmha4mysql-node-0.53

  perlMakefile.PL

  make&&makeinstall

  安装完成后会在/usr/local/bin目录下生成以下脚本文件:

  [root@192.168.3.110~]#pwd

  /usr/local/bin

  [root@192.168.3.110bin]#ll

  total40

  -r-xr-xr-x1rootroot15498Jun2010:05apply_diff_relay_logs

  -r-xr-xr-x1rootroot4807Jun2010:05filter_mysqlbinlog

  -r-xr-xr-x1rootroot7401Jun2010:05purge_relay_logs

  -r-xr-xr-x1rootroot7263Jun2010:05save_binary_logs

  [root@192.168.3.110bin]#

  关于上面脚本的功能,上面已经介绍过了,这里不再重复了。

  3.安装MHAManager

  MHAManager中主要包括了几个管理员的命令行工具,例如master_manger,master_master_switch等。MHAManger也依赖于perl模块,具体如下:

  (1)安装MHANode软件包之前需要安装依赖。我这里使用yum完成,没有epel源的可以使用上面提到的脚本(epel源安装也简单)。注意:在MHAManager的主机也是需要安装MHANode。

  rpm-ivhhttp://dl.fedorJunoject.org/pub/ ... ease-6-8.noarch.rpm

  yuminstallperl-DBD-MySQL-y

  安装MHANode软件包,和上面的方法一样,如下:

  wgethttp://mysql-master-ha.googlecod ... ql-node-0.53.tar.gz

  tarxfmha4mysql-node-0.53.tar.gz

  cdmha4mysql-node-0.53perlMakefile.PL

  make&&makeinstall

  (2)安装MHAManager。首先安装MHAManger依赖的perl模块(我这里使用yum安装):

  yuminstallperl-DBD-MySQLperl-Config-Tinyperl-Log-Dispatchperl-Parallel-ForkManagerperl-Time-HiRes-y

  安装MHAManager软件包:

  wgethttp://mysql-master-ha.googlecod ... manager-0.53.tar.gz

  tarxfmha4mysql-manager-0.53.tar.gz

  cdmha4mysql-manager-0.53

  perlMakefile.PL

  make&&makeinstall

  安装完成后会在/usr/local/bin目录下面生成以下脚本文件,前面已经说过这些脚本的作用,这里不再重复。

  [root@192.168.3.123bin]#pwd

  /usr/local/bin

  [root@192.168.3.123bin]#ll

  total76

  -r-xr-xr-x1rootroot15498Jun2010:58apply_diff_relay_logs

  -r-xr-xr-x1rootroot4807Jun2010:58filter_mysqlbinlog

  -r-xr-xr-x1rootroot1995Jun2011:33masterha_check_repl

  -r-xr-xr-x1rootroot1779Jun2011:33masterha_check_ssh

  -r-xr-xr-x1rootroot1865Jun2011:33masterha_check_status

  -r-xr-xr-x1rootroot3201Jun2011:33masterha_conf_host

  -r-xr-xr-x1rootroot2517Jun2011:33masterha_manager

  -r-xr-xr-x1rootroot2165Jun2011:33masterha_master_monitor

  -r-xr-xr-x1rootroot2373Jun2011:33masterha_master_switch

  -r-xr-xr-x1rootroot3749Jun2011:33masterha_secondary_check

  -r-xr-xr-x1rootroot1739Jun2011:33masterha_stop

  -r-xr-xr-x1rootroot7401Jun2010:58purge_relay_logs

  -r-xr-xr-x1rootroot7263Jun2010:58save_binary_logs

  复制相关脚本到/usr/local/bin目录

  [root@192.168.3.123scripts]#pwd

  /root/mha4mysql-manager-0.53/samples/scripts

  [root@192.168.3.123scripts]#ll

  total32

  -rwxr-xr-x1rootroot3443Jun82015master_ip_failover#自动切换时vip管理的脚本,不是必须,如果我们使用keepalived的,我们可以自己编写脚本完成对vip的管理,比如监控mysql,如果mysql异常,我们停止keepalived就行,这样vip就会自动漂移

  -rwxr-xr-x1rootroot9186Jun82015master_ip_online_change#在线切换时vip的管理,不是必须,同样可以可以自行编写简单的shell完成

  -rwxr-xr-x1rootroot11867Jun82015power_manager#故障发生后关闭主机的脚本,不是必须

  -rwxr-xr-x1rootroot1360Jun82015send_report#因故障切换后发送报警的脚本,不是必须,可自行编写简单的shell完成。

  [root@192.168.3.123scripts]#cp*/usr/local/bin/

  4.配置SSH免密码登录(key模式)

  这里需要注意的是所有的mysqlserver节点两两之间必须配成SSH无密码免登录双向模式,否则后面会报错。SSH免密码登录配置的文档可以查看Linux(使用ssh-keygen)设置SSH免密码登录。

  5.搭建主从复制环境

  搭建MySQL主从复制的文档可以查看linux(Ubuntu)下mysql安装及主从复制,MySQL半复制的文档可以查看Mysql半复制浅谈,这里不再赘述。

  6.配置MHA

  (1)创建MHA的工作目录,并且创建相关配置文件(在软件包解压后的目录里面有样例配置文件)。

  [root@192.168.3.123~]#mkdir-p/etc/masterha

  [root@192.168.3.123~]#cpmha4mysql-manager-0.53/samples/conf/app1.cnf/etc/masterha/

  修改app1.cnf配置文件,修改后的文件内容如下(注意,配置文件中的注释需要去掉,我这里是为了解释清楚):

  [root@192.168.3.123~]#cat/etc/masterha/app1.cnf

  [serverdefault]

  manager_workdir=/var/log/masterha/app1.log//设置manager的工作目录

  manager_log=/var/log/masterha/app1/manager.log//设置manager的日志

  master_binlog_dir=/data/mysql//设置master保存binlog的位置,以便MHA可以找到master的日志,我这里的也就是mysql的数据目录

  #master_ip_failover_script=/usr/local/bin/master_ip_failover//设置自动failover时候的切换脚本,这里先注释掉后面集成keepalived后再开启

  master_ip_online_change_script=/usr/local/bin/master_ip_online_change//设置手动切换时候的切换脚本

  password=123456//设置mysql中root用户的密码,这个密码是前文中创建监控用户的那个密码

  user=root设置监控用户root

  ping_interval=1//设置监控主库,发送ping包的时间间隔,默认是3秒,尝试三次没有回应的时候自动进行railover

  remote_workdir=/tmp//设置远端mysql在发生切换时binlog的保存位置

  repl_password=123456//设置复制用户的密码

  repl_user=repl//设置复制环境中的复制用户名

  report_script=/usr/local/send_report//设置发生切换后发送的报警的脚本

  secondary_check_script=/usr/local/bin/masterha_secondary_check-sB-dev18-sB-dev10--user=root--master_host=B-dev10--master_ip=192.168.3.110--master_port=3306//一旦MHA到B-dev10的监控之间出现问题,MHAManager将会尝试从B-dev18登录到B-dev10

  shutdown_script=""//设置故障发生后关闭故障主机脚本(该脚本的主要作用是关闭主机放在发生脑裂,这里没有使用)

  ssh_user=root//设置ssh的登录用户名

  [server1]

  hostname=192.168.3.110

  port=3306

  [server2]

  hostname=192.168.3.118

  port=3306

  candidate_master=1//设置为候选master,如果设置该参数以后,发生主从切换以后将会将此从库提升为主库,即使这个主库不是集群中事件最新的slave

  check_repl_delay=0//默认情况下如果一个slave落后master100M的relaylogs的话,MHA将不会选择该slave作为一个新的master,因为对于这个slave的恢复需要花费很长时间,通过设置check_repl_delay=0,MHA触发切换在选择一个新的master的时候将会忽略复制延时,这个参数对于设置了candidate_master=1的主机非常有用,因为这个候选主在切换的过程中一定是新的master

  [server3]

  hostname=192.168.3.115

  port=3306

  [server4]

  hostname=192.168.3.117

  port=3306

  [root@192.168.3.123~]#

  (2)设置relaylog的清除方式(在每个slave节点上):

  [root@192.168.3.115~]#mysql-e'setglobalrelay_log_purge=0'

  [root@192.168.3.117~]#mysql-e'setglobalrelay_log_purge=0'

  [root@192.168.3.118~]#mysql-e'setglobalrelay_log_purge=0'

  注意:

  MHA在发生切换的过程中,从库的恢复过程中依赖于relaylog的相关信息,所以这里要将relaylog的自动清除设置为OFF,采用手动清除relaylog的方式。在默认情况下,从服务器上的中继日志会在SQL线程执行完毕后被自动删除。但是在MHA环境中,这些中继日志在恢复其他从服务器时可能会被用到,因此需要禁用中继日志的自动删除功能。定期清除中继日志需要考虑到复制延时的问题。在ext3的文件系统下,删除大的文件需要一定的时间,会导致严重的复制延时。为了避免复制延时,需要暂时为中继日志创建硬链接,因为在linux系统中通过硬链接删除大文件速度会很快。(在mysql数据库中,删除大表时,通常也采用建立硬链接的方式)

  MHA节点中包含了pure_relay_logs命令工具,它可以为中继日志创建硬链接,执行SETGLOBALrelay_log_purge=1,等待几秒钟以便SQL线程切换到新的中继日志,再执行SETGLOBALrelay_log_purge=0。

  pure_relay_logs脚本参数如下所示:

  --usermysql用户名

  --passwordmysql密码

  --port端口号

  --workdir指定创建relaylog的硬链接的位置,默认是/var/tmp,由于系统不同分区创建硬链接文件会失败,故需要执行硬链接具体位置,成功执行脚本后,硬链接的中继日志文件被删除

  --disable_relay_log_purge默认情况下,如果relay_log_purge=1,脚本会什么都不清理,自动退出,通过设定这个参数,当relay_log_purge=1的情况下会将relay_log_purge设置为0。清理relaylog之后,最后将参数设置为OFF。

  (3)设置定期清理relay脚本(三台slave服务器)

  [root@192.168.3.115~]#catpurge_relay_log.sh

  #!/bin/bash

  user=root

  passwd=123456

  port=3306

  log_dir='/data/masterha/log'

  work_dir='/data'

  purge='/usr/local/bin/purge_relay_logs'

  if[!-d$log_dir]

  then

  mkdir$log_dir-p

  fi

  $purge--user=$user--password=$passwd--disable_relay_log_purge--port=$port--workdir=$work_dir>>$log_dir/purge_relay_logs.log2>&1

  [root@192.168.3.115~]#

  添加到crontab定期执行

  [root@192.168.3.115~]#crontab-l

  04***/bin/bash/root/purge_relay_log.sh

  [root@192.168.3.115~]#

  purge_relay_logs脚本删除中继日志不会阻塞SQL线程。下面我们手动执行看看什么情况。

  [root@192.168.3.115~]#purge_relay_logs--user=root--password=123456--port=3306-disable_relay_log_purge--workdir=/data/

  2015-06-2015:47:24:purge_relay_logsscriptstarted.

  Foundrelay_log.info:/data/mysql/relay-log.info

  RemovinghardlinkedrelaylogfilesB-dev15-relay-bin*under/data/..done.

  Currentrelaylogfile:/data/mysql/B-dev15-relay-bin.000002

  Archivingunusedrelaylogfiles(upto/data/mysql/B-dev15-relay-bin.000001)...

  Creatinghardlinkfor/data/mysql/B-dev15-relay-bin.000001under/data//B-dev15-relay-bin.000001..ok.

  Creatinghardlinksforunusedrelaylogfilescompleted.

  ExecutingSETGLOBALrelay_log_purge=1;FLUSHLOGS;sleepingafewsecondssothatSQLthreadcandeleteolderrelaylogfiles(ifitkeepsup);SETGLOBALrelay_log_purge=0;..ok.

  RemovinghardlinkedrelaylogfilesB-dev15-relay-bin*under/data/..done.

  2015-06-2015:47:27:Allrelaylogpurgingoperationssucceeded.

  [root@192.168.3.115~]#

  7.检查SSH配置

  检查MHAManger到所有MHANode的SSH连接状态:

  [root@192.168.3.123~]#masterha_check_ssh--conf=/etc/masterha/app1.cnf

  WedJun2402:04:242015-[warning]Globalconfigurationfile/etc/masterha_default.cnfnotfound.Skipping.

  WedJun2402:04:242015-[info]Readingapplicationdefaultconfigurationsfrom/etc/masterha/app1.cnf..

  WedJun2402:04:242015-[info]Readingserverconfigurationsfrom/etc/masterha/app1.cnf..

  WedJun2402:04:242015-[info]StartingSSHconnectiontests..

  WedJun2402:04:262015-[debug]

  WedJun2402:04:242015-[debug]ConnectingviaSSHfromroot@192.168.3.110(192.168.3.110:22)toroot@192.168.3.118(192.168.3.118:22)..

  WedJun2402:04:242015-[debug]ok.

  WedJun2402:04:242015-[debug]ConnectingviaSSHfromroot@192.168.3.110(192.168.3.110:22)toroot@192.168.3.115(192.168.3.115:22)..

  WedJun2402:04:252015-[debug]ok.

  WedJun2402:04:252015-[debug]ConnectingviaSSHfromroot@192.168.3.110(192.168.3.110:22)toroot@192.168.3.117(192.168.3.117:22)..

  WedJun2402:04:262015-[debug]ok.

  WedJun2402:04:272015-[debug]

  WedJun2402:04:242015-[debug]ConnectingviaSSHfromroot@192.168.3.118(192.168.3.118:22)toroot@192.168.3.110(192.168.3.110:22)..

  WedJun2402:04:252015-[debug]ok.

  WedJun2402:04:252015-[debug]ConnectingviaSSHfromroot@192.168.3.118(192.168.3.118:22)toroot@192.168.3.115(192.168.3.115:22)..

  WedJun2402:04:262015-[debug]ok.

  WedJun2402:04:262015-[debug]ConnectingviaSSHfromroot@192.168.3.118(192.168.3.118:22)toroot@192.168.3.117(192.168.3.117:22)..

  WedJun2402:04:272015-[debug]ok.

  WedJun2402:04:282015-[debug]

  WedJun2402:04:252015-[debug]ConnectingviaSSHfromroot@192.168.3.115(192.168.3.115:22)toroot@192.168.3.110(192.168.3.110:22)..

  WedJun2402:04:252015-[debug]ok.

  WedJun2402:04:252015-[debug]ConnectingviaSSHfromroot@192.168.3.115(192.168.3.115:22)toroot@192.168.3.118(192.168.3.118:22)..

  WedJun2402:04:272015-[debug]ok.

  WedJun2402:04:272015-[debug]ConnectingviaSSHfromroot@192.168.3.115(192.168.3.115:22)toroot@192.168.3.117(192.168.3.117:22)..

  WedJun2402:04:282015-[debug]ok.

  WedJun2402:04:282015-[debug]

  WedJun2402:04:252015-[debug]ConnectingviaSSHfromroot@192.168.3.117(192.168.3.117:22)toroot@192.168.3.110(192.168.3.110:22)..

  WedJun2402:04:262015-[debug]ok.

  WedJun2402:04:262015-[debug]ConnectingviaSSHfromroot@192.168.3.117(192.168.3.117:22)toroot@192.168.3.118(192.168.3.118:22)..

  WedJun2402:04:272015-[debug]ok.

  WedJun2402:04:272015-[debug]ConnectingviaSSHfromroot@192.168.3.117(192.168.3.117:22)toroot@192.168.3.115(192.168.3.115:22)..

  WedJun2402:04:282015-[debug]ok.

  WedJun2402:04:282015-[info]AllSSHconnectiontestspassedsuccessfully.

  可以看见各个节点ssh验证都是ok的。

  8.检查整个复制环境状况

  通过masterha_check_repl脚本查看整个集群的状态

  [root@192.168.123~]#masterha_check_repl--conf=/etc/masterha/app1.cnf

  WedJun2401:51:172015-[warning]Globalconfigurationfile/etc/masterha_default.cnfnotfound.Skipping.

  WedJun2401:51:172015-[info]Readingapplicationdefaultconfigurationsfrom/etc/masterha/app1.cnf..

  WedJun2401:51:172015-[info]Readingserverconfigurationsfrom/etc/masterha/app1.cnf..

  WedJun2401:51:172015-[info]MHA::MasterMonitorversion0.53.

  WedJun2401:51:182015-[info]DeadServers:

  WedJun2401:51:182015-[info]AliveServers:

  WedJun2401:51:182015-[info]192.168.3.110(192.168.3.110:3306)

  WedJun2401:51:182015-[info]192.168.3.118(192.168.3.118:3306)

  WedJun2401:51:182015-[info]192.168.3.115(192.168.3.115:3306)

  WedJun2401:51:182015-[info]192.168.3.117(192.168.3.117:3306)

  WedJun2401:51:182015-[info]AliveSlaves:

  WedJun2401:51:182015-[info]192.168.3.118(192.168.3.118:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  WedJun2401:51:182015-[info]Replicatingfrom192.168.3.110(192.168.3.110:3306)

  WedJun2401:51:182015-[info]PrimarycandidateforthenewMaster(candidate_masterisset)

  WedJun2401:51:182015-[info]192.168.3.115(192.168.3.115:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  WedJun2401:51:182015-[info]Replicatingfrom192.168.3.110(192.168.3.110:3306)

  WedJun2401:51:182015-[info]192.168.3.117(192.168.3.117:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  WedJun2401:51:182015-[info]Replicatingfrom192.168.3.110(192.168.3.110:3306)

  WedJun2401:51:182015-[info]CurrentAliveMaster:192.168.3.110(192.168.3.110:3306)

  WedJun2401:51:182015-[info]Checkingslaveconfigurations..

  WedJun2401:51:182015-[info]read_only=1isnotsetonslave192.168.3.118(192.168.3.118:3306).

  WedJun2401:51:182015-[warning]relay_log_purge=0isnotsetonslave192.168.3.118(192.168.3.118:3306).

  WedJun2401:51:182015-[info]read_only=1isnotsetonslave192.168.3.115(192.168.3.115:3306).

  WedJun2401:51:182015-[warning]relay_log_purge=0isnotsetonslave192.168.3.115(192.168.3.115:3306).

  WedJun2401:51:182015-[info]read_only=1isnotsetonslave192.168.3.117(192.168.3.117:3306).

  WedJun2401:51:182015-[warning]relay_log_purge=0isnotsetonslave192.168.3.117(192.168.3.117:3306).

  WedJun2401:51:182015-[info]Checkingreplicationfilteringsettings..

  WedJun2401:51:182015-[info]binlog_do_db=,binlog_ignore_db=

  WedJun2401:51:182015-[info]Replicationfilteringcheckok.

  WedJun2401:51:182015-[info]StartingSSHconnectiontests..

  WedJun2401:51:232015-[info]AllSSHconnectiontestspassedsuccessfully.

  WedJun2401:51:232015-[info]CheckingMHANodeversion..

  WedJun2401:51:242015-[info]Versioncheckok.

  WedJun2401:51:242015-[info]CheckingSSHpublickeyauthenticationsettingsonthecurrentmaster..

  WedJun2401:51:252015-[info]HealthCheck:SSHto192.168.3.110isreachable.

  WedJun2401:51:262015-[info]MasterMHANodeversionis0.53.

  WedJun2401:51:262015-[info]Checkingrecoveryscriptconfigurationsonthecurrentmaster..

  WedJun2401:51:262015-[info]Executingcommand:save_binary_logs--command=test--start_pos=4--binlog_dir=/var/lib/mysql--output_file=/tmp/save_binary_logs_test--manager_version=0.53--start_file=mysql-bin.000002

  WedJun2401:51:262015-[info]Connectingtoroot@192.168.3.110(192.168.3.110)..

  Creating/tmpifnotexists..ok.

  Checkingoutputdirectoryisaccessibleornot..

  ok.

  Binlogfoundat/var/lib/mysql,uptomysql-bin.000002

  WedJun2401:51:262015-[info]Mastersettingcheckdone.

  WedJun2401:51:262015-[info]CheckingSSHpublickeyauthenticationandcheckingrecoveryscriptconfigurationsonallaliveslaveservers..

  WedJun2401:51:262015-[info]Executingcommand:apply_diff_relay_logs--command=test--slave_user=root--slave_host=192.168.3.118--slave_ip=192.168.3.118--slave_port=3306--workdir=/tmp--target_version=5.6.25-log--manager_version=0.53--relay_log_info=/var/lib/mysql/relay-log.info--relay_dir=/var/lib/mysql/--slave_pass=xxx

  WedJun2401:51:262015-[info]Connectingtoroot@192.168.3.118(192.168.3.118:22)..

  Checkingslaverecoveryenvironmentsettings..

  Opening/var/lib/mysql/relay-log.info...ok.

  Relaylogfoundat/var/lib/mysql,uptomysqld-relay-bin.000004

  Temporaryrelaylogfileis/var/lib/mysql/mysqld-relay-bin.000004

  Testingmysqlconnectionandprivileges..Warning:Usingapasswordonthecommandlineinterfacecanbeinsecure.

  done.

  Testingmysqlbinlogoutput..done.

  Cleaninguptestfile(s)..done.

  WedJun2401:51:272015-[info]Executingcommand:apply_diff_relay_logs--command=test--slave_user=root--slave_host=192.168.3.115--slave_ip=192.168.3.115--slave_port=3306--workdir=/tmp--target_version=5.6.25-log--manager_version=0.53--relay_log_info=/var/lib/mysql/relay-log.info--relay_dir=/var/lib/mysql/--slave_pass=xxx

  WedJun2401:51:272015-[info]Connectingtoroot@192.168.3.115(192.168.3.115:22)..

  Checkingslaverecoveryenvironmentsettings..

  Opening/var/lib/mysql/relay-log.info...ok.

  Relaylogfoundat/var/lib/mysql,uptomysqld-relay-bin.000004

  Temporaryrelaylogfileis/var/lib/mysql/mysqld-relay-bin.000004

  Testingmysqlconnectionandprivileges..Warning:Usingapasswordonthecommandlineinterfacecanbeinsecure.

  done.

  Testingmysqlbinlogoutput..done.

  Cleaninguptestfile(s)..done.

  WedJun2401:51:292015-[info]Executingcommand:apply_diff_relay_logs--command=test--slave_user=root--slave_host=192.168.3.117--slave_ip=192.168.3.117--slave_port=3306--workdir=/tmp--target_version=5.6.25-log--manager_version=0.53--relay_log_info=/var/lib/mysql/relay-log.info--relay_dir=/var/lib/mysql/--slave_pass=xxx

  WedJun2401:51:292015-[info]Connectingtoroot@192.168.3.117(192.168.3.117:22)..

  Checkingslaverecoveryenvironmentsettings..

  Opening/var/lib/mysql/relay-log.info...ok.

  Relaylogfoundat/var/lib/mysql,uptomysqld-relay-bin.000003

  Temporaryrelaylogfileis/var/lib/mysql/mysqld-relay-bin.000003

  Testingmysqlconnectionandprivileges..Warning:Usingapasswordonthecommandlineinterfacecanbeinsecure.

  done.

  Testingmysqlbinlogoutput..done.

  Cleaninguptestfile(s)..done.

  WedJun2401:51:312015-[info]Slavessettingscheckdone.

  WedJun2401:51:312015-[info]

  192.168.3.110(currentmaster)

  +--192.168.3.118

  +--192.168.3.115

  +--192.168.3.117

  WedJun2401:51:312015-[info]Checkingreplicationhealthon192.168.3.118..

  WedJun2401:51:312015-[info]ok.

  WedJun2401:51:312015-[info]Checkingreplicationhealthon192.168.3.115..

  WedJun2401:51:312015-[info]ok.

  WedJun2401:51:312015-[info]Checkingreplicationhealthon192.168.3.117..

  WedJun2401:51:312015-[info]ok.

  WedJun2401:51:312015-[info]Checkingmaster_ip_failover_scriptstatus:

  WedJun2401:51:312015-[info]/usr/local/bin/master_ip_failover--command=status--ssh_user=root--orig_master_host=192.168.3.110--orig_master_ip=192.168.3.110--orig_master_port=3306

  INSCRIPTTEST====/etc/init.d/keepalivedstop==/etc/init.d/keepalivedstart===

  CheckingtheStatusofthescript..OK

  WedJun2401:51:312015-[info]OK.

  WedJun2401:51:312015-[warning]master_ip_failoverisnotdefined.

  WedJun2401:51:312015-[warning]shutdown_scriptisnotdefined.

  WedJun2401:51:312015-[info]Gotexitcode0(Notmasterdead).

  MySQLReplicationHealthisOK.

  没有报错信息,只有两个警告而已,复制也显示正常了。

  9.检查MHAManager的状态

  通过master_check_status脚本查看Manager的状态:

  [root@192.168.3.123~]#masterha_check_status--conf=/etc/masterha/app1.cnf

  app1isstopped(2:NOT_RUNNING).

  [root@192.168.3.123~]#

  注意:如果正常,会显示”PING_OK”,否则会显示”NOT_RUNNING”,这代表MHA监控没有开启。

  10.开启MHAManager监控

  [root@192.168.3.123~]#nohupmasterha_manager--conf=/etc/masterha/app1.cnf--remove_dead_master_conf--ignore_last_failover</dev/null>/var/log/masterha/app1/manager.log2>&1&

  [1]30867

  [root@192.168.3.123~]#

  启动参数介绍:

  –remove_dead_master_conf该参数代表当发生主从切换后,老的主库的ip将会从配置文件中移除。

  –manger_log日志存放位置

  –ignore_last_failover在缺省情况下,如果MHA检测到连续发生宕机,且两次宕机间隔不足8小时的话,则不会进行Failover,之所以这样限制是为了避免ping-pong效应。该参数代表忽略上次MHA触发切换产生的文件,默认情况下,MHA发生切换后会在日志目录,也就是上面我设置的/data产生app1.failover.complete文件,下次再次切换的时候如果发现该目录下存在该文件将不允许触发切换,除非在第一次切换后收到删除该文件,为了方便,这里设置为–ignore_last_failover。

  查看MHAManager监控是否正常:

  [root@192.168.3.123~]#masterha_check_status--conf=/etc/masterha/app1.cnf

  app1(pid:20386)isrunning(0:PING_OK),master:192.168.3.110

  [root@192.168.3.123~]#

  可以看见已经在监控了,而且master的主机为192.168.3.110

  11.查看启动日志

  [root@192.168.3.123~]#tail-f/var/log/masterha/app1/manager.log

  SunJun2019:12:012015-[info]Connectingtoroot@192.168.3.117(192.168.3.117:22)..

  Checkingslaverecoveryenvironmentsettings..

  Opening/data/mysql/relay-log.info...ok.

  Relaylogfoundat/data/mysql,uptoB-dev17-relay-bin.000002

  Temporaryrelaylogfileis/data/mysql/B-dev17-relay-bin.000002

  Testingmysqlconnectionandprivileges..done.

  Testingmysqlbinlogoutput..done.

  Cleaninguptestfile(s)..done.

  SunJun2019:12:012015-[info]Connectingtoroot@192.168.3.115(192.168.3.117:22)..

  Checkingslaverecoveryenvironmentsettings..

  Opening/data/mysql/relay-log.info...ok.

  Relaylogfoundat/data/mysql,uptoB-dev15-relay-bin.000002

  Temporaryrelaylogfileis/data/mysql/B-dev15-relay-bin.000002

  Testingmysqlconnectionandprivileges..done.

  Testingmysqlbinlogoutput..done.

  Cleaninguptestfile(s)..done.

  SunJun2019:12:012015-[info]Slavessettingscheckdone.

  SunJun2019:12:012015-[info]

  192.168.3.110(currentmaster)

  +--192.168.3.115

  +--192.168.3.117

  +--192.168.3.123

  SunApr2019:12:012015-[warning]master_ip_failover_scriptisnotdefined.

  SunApr2019:12:012015-[warning]shutdown_scriptisnotdefined.

  SunApr2019:12:012015-[info]Setmasterpinginterval1seconds.

  SunApr2019:12:012015-[info]Setsecondarycheckscript:/usr/local/bin/masterha_secondary_check-sB-dev18-sB-dev10--user=root--master_host=B-dev10--master_ip=192.168.3.110--master_port=3306

  SunApr2019:12:012015-[info]Startingpinghealthcheckon192.168.3.123(192.168.3.123:3306)..

  SunApr2019:12:012015-[info]Ping(SELECT)succeeded,waitinguntilMySQLdoesn'trespond..

  [root@192.168.3.123~]#

  其中”Ping(SELECT)succeeded,waitinguntilMySQLdoesn’trespond..”说明整个系统已经开始监控了。

  12.关闭MHAManage监控

  关闭很简单,使用masterha_stop命令完成。

  [root@192.168.3.123~]#masterha_stop--conf=/etc/masterha/app1.cnf

  Stoppedapp1successfully.

  [1]+Exit1nohupmasterha_manager--conf=/etc/masterha/app1.cnf--remove_dead_master_conf--ignore_last_failover--manager_log=/data/mamanager.log

  [root@192.168.3.123~]#

  13.配置VIP

  vip配置可以采用两种方式,一种通过keepalived的方式管理虚拟ip的漂移;另外一种通过脚本方式启动虚拟ip的方式(即不需要keepalived或者heartbeat类似的软件)。这里采用的是keepalived的方式管理vip的漂移。

  keepalived方式管理虚拟ip,keepalived配置方法如下:

  (1)下载软件进行并进行安装(两台master,准确的说一台是master,另外一台是备选master,在没有切换以前是slave):

  [root@192.168.3.110~]#wgethttp://www.keepalived.org/software/keepalived-1.2.17.tar.gz

  tarxfkeepalived-1.2.17.tar.gz

  cdkeepalived-1.2.17

  ./configure--prefix=/usr/local/keepalived

  make&&makeinstall

  cp/usr/local/keepalived/etc/rc.d/init.d/keepalived/etc/init.d/

  cp/usr/local/keepalived/etc/sysconfig/keepalived/etc/sysconfig/

  mkdir/etc/keepalived

  cp/usr/local/keepalived/etc/keepalived/keepalived.conf/etc/keepalived/

  cp/usr/local/keepalived/sbin/keepalived/usr/sbin/

  (2)配置keepalived的配置文件,在master上配置(192.168.3.110)

  [root@192.168.3.110~]#cat/etc/keepalived/keepalived.conf

  !ConfigurationFileforkeepalived

  global_defs{

  notification_email{

  acassen@firewall.loc

  failover@firewall.loc

  sysadmin@firewall.loc

  }

  notification_email_fromAlexandre.Cassen@firewall.loc

  smtp_server127.0.0.1

  smtp_connect_timeout30

  router_idLVS_DEVEL

  }

  VRrp_instanceVI_1{

  stateBACKUP

  interfaceeth0

  virtual_router_id51

  priority80

  advert_int1

  nopreempt

  authentication{

  auth_typePASS

  auth_pass1111

  }

  virtual_ipaddress{

  192.168.3.250

  }

  }

  [root@192.168.3.110~]#

  其中router_idMySQLHA表示设定keepalived组的名称,将192.168.3.250这个虚拟ip绑定到该主机的eth0网卡上,并且设置了状态为backup模式,将keepalived的模式设置为非抢占模式(nopreempt),priority80表示设置的优先级为80。下面的配置略有不同,但是都是一个意思。

  (3)在候选master上配置(192.168.3.118)

  [root@192.168.3.118~]#cat/etc/keepalived/keepalived.conf

  !ConfigurationFileforkeepalived

  global_defs{

  notification_email{

  acassen@firewall.loc

  failover@firewall.loc

  sysadmin@firewall.loc

  }

  notification_email_fromAlexandre.Cassen@firewall.loc

  smtp_server127.0.0.1

  smtp_connect_timeout30

  router_idLVS_DEVEL

  }

  vrrp_instanceVI_1{

  stateBACKUP

  interfaceeth0

  virtual_router_id51

  priority60

  advert_int1

  nopreempt

  authentication{

  auth_typePASS

  auth_pass1111

  }

  virtual_ipaddress{

  192.168.3.250

  }

  }

  [root@192.168.3.118~]#

  (4)启动keepalived服务,在master上启动并查看日志

  [root@192.168.3.110~]#/etc/init.d/keepalivedstart

  Startingkeepalived:[OK]

  [root@192.168.3.110~]#tail-f/var/log/messages

  Jun2020:22:16192Keepalived_healthcheckers[15334]:Openingfile'/etc/keepalived/keepalived.conf'.

  Jun2020:22:16192Keepalived_healthcheckers[15334]:Configurationisusing:7231Bytes

  Jun2020:22:16192kernel:IPVS:Connectionhashtableconfigured(size=4096,memory=64Kbytes)

  Jun2020:22:16192kernel:IPVS:ipvsloaded.

  Jun2020:22:16192Keepalived_healthcheckers[15334]:UsingLinkWatchkernelnetlinkreflector...

  Jun2020:22:19192Keepalived_vrrp[15335]:VRRP_Instance(VI_1)TransitiontoMASTERSTATE

  Jun2020:22:20192Keepalived_vrrp[15335]:VRRP_Instance(VI_1)EnteringMASTERSTATE

  Jun2020:22:20192Keepalived_vrrp[15335]:VRRP_Instance(VI_1)settingprotocolVIPs.

  Jun2020:22:20192Keepalived_vrrp[15335]:VRRP_Instance(VI_1)SendinggratuitousARPsoneth0for192.168.3.250

  Jun2020:22:20192Keepalived_healthcheckers[15334]:NetlinkreflectorreportsIP192.168.3.250added

  Jun2020:22:25192Keepalived_vrrp[15335]:VRRP_Instance(VI_1)SendinggratuitousARPsoneth0for192.168.3.250

  发现已经将虚拟ip192.168.3.250绑定了网卡eth0上。

  (5)查看绑定情况

  [root@192.168.3.110~]#ipaddr|grepeth0

  2:eth0:<BROADCAST,MULTICAST,UP,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000

  inet192.168.3.110/24brd192.168.3.255scopeglobaleth0

  inet192.168.3.250/32scopeglobaleth0

  [root@192.168.3.110~]#

  在另外一台服务器,候选master上启动keepalived服务,并观察

  [root@192.168.3.118~]#/etc/init.d/keepalivedstart

  Startingkeepalived:[OK]

  [root@192.168.3.110~]#tail-f/var/log/messages

  Jun2020:26:18192Keepalived_vrrp[9472]:RegisteringgratuitousARPsharedchannel

  Jun2020:26:18192Keepalived_vrrp[9472]:Openingfile'/etc/keepalived/keepalived.conf'.

  Jun2020:26:18192Keepalived_vrrp[9472]:Configurationisusing:62976Bytes

  Jun2020:26:18192Keepalived_vrrp[9472]:UsingLinkWatchkernelnetlinkreflector...

  Jun2020:26:18192Keepalived_vrrp[9472]:VRRP_Instance(VI_1)EnteringBACKUPSTATE

  Jun2020:26:18192Keepalived_vrrp[9472]:VRRPsockpool:[ifindex(3),proto(112),unicast(0),fd(10,11)]

  Jun2020:26:18192Keepalived_healthcheckers[9471]:NetlinkreflectorreportsIP192.168.3.250added

  Jun2020:26:18192Keepalived_healthcheckers[9471]:NetlinkreflectorreportsIP192.168.3.118added

  Jun2020:26:18192Keepalived_healthcheckers[9471]:NetlinkreflectorreportsIPfe80::20c:29ff:fe9d:6a9eadded

  Jun2020:26:18192Keepalived_healthcheckers[9471]:NetlinkreflectorreportsIPfe80::20c:29ff:fe9d:6aa8added

  Jun2020:26:18192Keepalived_healthcheckers[9471]:RegisteringKernelnetlinkreflector

  Jun2020:26:18192Keepalived_healthcheckers[9471]:RegisteringKernelnetlinkcommandchannel

  Jun2020:26:18192Keepalived_healthcheckers[9471]:Openingfile'/etc/keepalived/keepalived.conf'.

  Jun2020:26:18192Keepalived_healthcheckers[9471]:Configurationisusing:7231Bytes

  Jun2020:26:18192kernel:IPVS:Registeredprotocols(TCP,UDP,AH,ESP)

  Jun2020:26:18192kernel:IPVS:Connectionhashtableconfigured(size=4096,memory=64Kbytes)

  Jun2020:26:18192kernel:IPVS:ipvsloaded.

  Jun2020:26:18192Keepalived_healthcheckers[9471]:UsingLinkWatchkernelnetlinkreflector...

  从上面的信息可以看到keepalived已经配置成功。

  注意:

  上面两台服务器的keepalived都设置为了BACKUP模式,在keepalived中2种模式,分别是master->backup模式和backup->backup模式。这两种模式有很大区别。在master->backup模式下,一旦主库宕机,虚拟ip会自动漂移到从库,当主库修复后,keepalived启动后,还会把虚拟ip抢占过来,即使设置了非抢占模式(nopreempt)抢占ip的动作也会发生。在backup->backup模式下,当主库宕机后虚拟ip会自动漂移到从库上,当原主库恢复和keepalived服务启动后,并不会抢占新主的虚拟ip,即使是优先级高于从库的优先级别,也不会发生抢占。为了减少ip漂移次数,通常是把修复好的主库当做新的备库。

  (6)MHA引入keepalived

  要想把keepalived服务引入MHA,我们只需要修改切换是触发的脚本文件master_ip_failover即可,在该脚本中添加在master发生宕机时对keepalived的处理。

  编辑脚本/usr/local/bin/master_ip_failover,修改后如下,这里完整贴出该脚本(192.168.3.123)。

  在MHAManager修改脚本修改后的内容如下:

  #!/usr/bin/envperl

  usestrict;

  usewarningsFATAL=>'all';

  useGetopt::Long;

  my(

  $command,$ssh_user,$orig_master_host,$orig_master_ip,

  $orig_master_port,$new_master_host,$new_master_ip,$new_master_port

  );

  my$vip='192.168.3.250';

  my$ssh_start_vip="/etc/init.d/keepalivedstart";

  my$ssh_stop_vip="/etc/init.d/keepalivedstop";

  GetOptions(

  'command=s'=>\$command,

  'ssh_user=s'=>\$ssh_user,

  'orig_master_host=s'=>\$orig_master_host,

  'orig_master_ip=s'=>\$orig_master_ip,

  'orig_master_port=i'=>\$orig_master_port,

  'new_master_host=s'=>\$new_master_host,

  'new_master_ip=s'=>\$new_master_ip,

  'new_master_port=i'=>\$new_master_port,

  );

  exit&main();

  submain{

  print"\n\nINSCRIPTTEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

  if($commandeq"stop"||$commandeq"stopssh"){

  my$exit_code=1;

  eval{

  print"DisablingtheVIPonoldmaster:$orig_master_host\n";

  &stop_vip();

  $exit_code=0;

  };

  if($@){

  warn"GotError:$@\n";

  exit$exit_code;

  }

  exit$exit_code;

  }

  elsif($commandeq"start"){

  my$exit_code=10;

  eval{

  print"EnablingtheVIP-$viponthenewmaster-$new_master_host\n";

  &start_vip();

  $exit_code=0;

  };

  if($@){

  warn$@;

  exit$exit_code;

  }

  exit$exit_code;

  }

  elsif($commandeq"status"){

  print"CheckingtheStatusofthescript..OK\n";

  #`ssh$ssh_user\@cluster1\"$ssh_start_vip\"`;

  exit0;

  }

  else{

  &usage();

  exit1;

  }

  }

  #AsimplesystemcallthatenabletheVIPonthenewmastersubstart_vip(){

  `ssh$ssh_user\@$new_master_host\"$ssh_start_vip\"`;

  }

  #AsimplesystemcallthatdisabletheVIPontheold_master

  substop_vip(){

  return0unless($ssh_user);

  `ssh$ssh_user\@$orig_master_host\"$ssh_stop_vip\"`;

  }

  subusage{

  print

  "Usage:master_ip_failover--command=start|stop|stopssh|status--orig_master_host=host--orig_master_ip=ip--orig_master_port=port--new_master_host=host--new_master_ip=ip--new_master_port=port\n";

  }

  现在已经修改这个脚本了,我们现在打开在上面提到过的参数,再检查集群状态,看是否会报错。

  [root@192.168.3.123~]#grep'master_ip_failover_script'/etc/masterha/app1.cnf

  master_ip_failover_script=/usr/local/bin/master_ip_failover

  [root@192.168.3.123~]#

  [root@192.168.3.123~]#masterha_check_repl--conf=/etc/masterha/app1.cnf

  WedJun2402:02:202015-[warning]Globalconfigurationfile/etc/masterha_default.cnfnotfound.Skipping.

  WedJun2402:02:202015-[info]Readingapplicationdefaultconfigurationsfrom/etc/masterha/app1.cnf..

  WedJun2402:02:202015-[info]Readingserverconfigurationsfrom/etc/masterha/app1.cnf..

  WedJun2402:02:202015-[info]MHA::MasterMonitorversion0.53.

  WedJun2402:02:202015-[info]DeadServers:

  WedJun2402:02:202015-[info]AliveServers:

  WedJun2402:02:202015-[info]192.168.3.110(192.168.3.110:3306)

  WedJun2402:02:202015-[info]192.168.3.118(192.168.3.118:3306)

  WedJun2402:02:202015-[info]192.168.3.115(192.168.3.115:3306)

  WedJun2402:02:202015-[info]192.168.3.117(192.168.3.117:3306)

  WedJun2402:02:202015-[info]AliveSlaves:

  WedJun2402:02:202015-[info]192.168.3.118(192.168.3.118:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  WedJun2402:02:202015-[info]Replicatingfrom192.168.3.110(192.168.3.110:3306)

  WedJun2402:02:202015-[info]PrimarycandidateforthenewMaster(candidate_masterisset)

  WedJun2402:02:202015-[info]192.168.3.115(192.168.3.115:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  WedJun2402:02:202015-[info]Replicatingfrom192.168.3.110(192.168.3.110:3306)

  WedJun2402:02:202015-[info]192.168.3.117(192.168.3.117:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  WedJun2402:02:202015-[info]Replicatingfrom192.168.3.110(192.168.3.110:3306)

  WedJun2402:02:202015-[info]CurrentAliveMaster:192.168.3.110(192.168.3.110:3306)

  WedJun2402:02:202015-[info]Checkingslaveconfigurations..

  WedJun2402:02:202015-[info]read_only=1isnotsetonslave192.168.3.118(192.168.3.118:3306).

  WedJun2402:02:202015-[warning]relay_log_purge=0isnotsetonslave192.168.3.118(192.168.3.118:3306).

  WedJun2402:02:202015-[info]read_only=1isnotsetonslave192.168.3.115(192.168.3.115:3306).

  WedJun2402:02:202015-[warning]relay_log_purge=0isnotsetonslave192.168.3.115(192.168.3.115:3306).

  WedJun2402:02:202015-[info]read_only=1isnotsetonslave192.168.3.117(192.168.3.117:3306).

  WedJun2402:02:202015-[warning]relay_log_purge=0isnotsetonslave192.168.3.117(192.168.3.117:3306).

  WedJun2402:02:202015-[info]Checkingreplicationfilteringsettings..

  WedJun2402:02:202015-[info]binlog_do_db=,binlog_ignore_db=

  WedJun2402:02:202015-[info]Replicationfilteringcheckok.

  WedJun2402:02:202015-[info]StartingSSHconnectiontests..

  WedJun2402:02:252015-[info]AllSSHconnectiontestspassedsuccessfully.

  WedJun2402:02:252015-[info]CheckingMHANodeversion..

  WedJun2402:02:262015-[info]Versioncheckok.

  WedJun2402:02:262015-[info]CheckingSSHpublickeyauthenticationsettingsonthecurrentmaster..

  WedJun2402:02:272015-[info]HealthCheck:SSHto192.168.3.110isreachable.

  WedJun2402:02:272015-[info]MasterMHANodeversionis0.53.

  WedJun2402:02:272015-[info]Checkingrecoveryscriptconfigurationsonthecurrentmaster..

  WedJun2402:02:272015-[info]Executingcommand:save_binary_logs--command=test--start_pos=4--binlog_dir=/var/lib/mysql--output_file=/tmp/save_binary_logs_test--manager_version=0.53--start_file=mysql-bin.000002

  WedJun2402:02:272015-[info]Connectingtoroot@192.168.3.110(192.168.3.110)..

  Creating/tmpifnotexists..ok.

  Checkingoutputdirectoryisaccessibleornot..

  ok.

  Binlogfoundat/var/lib/mysql,uptomysql-bin.000002

  WedJun2402:02:282015-[info]Mastersettingcheckdone.

  WedJun2402:02:282015-[info]CheckingSSHpublickeyauthenticationandcheckingrecoveryscriptconfigurationsonallaliveslaveservers..

  WedJun2402:02:282015-[info]Executingcommand:apply_diff_relay_logs--command=test--slave_user=root--slave_host=192.168.3.118--slave_ip=192.168.3.118--slave_port=3306--workdir=/tmp--target_version=5.6.25-log--manager_version=0.53--relay_log_info=/var/lib/mysql/relay-log.info--relay_dir=/var/lib/mysql/--slave_pass=xxx

  WedJun2402:02:282015-[info]Connectingtoroot@192.168.3.118(192.168.3.118:22)..

  Checkingslaverecoveryenvironmentsettings..

  Opening/var/lib/mysql/relay-log.info...ok.

  Relaylogfoundat/var/lib/mysql,uptomysqld-relay-bin.000004

  Temporaryrelaylogfileis/var/lib/mysql/mysqld-relay-bin.000004

  Testingmysqlconnectionandprivileges..Warning:Usingapasswordonthecommandlineinterfacecanbeinsecure.

  done.

  Testingmysqlbinlogoutput..done.

  Cleaninguptestfile(s)..done.

  WedJun2402:02:292015-[info]Executingcommand:apply_diff_relay_logs--command=test--slave_user=root--slave_host=192.168.3.115--slave_ip=192.168.3.115--slave_port=3306--workdir=/tmp--target_version=5.6.25-log--manager_version=0.53--relay_log_info=/var/lib/mysql/relay-log.info--relay_dir=/var/lib/mysql/--slave_pass=xxx

  WedJun2402:02:292015-[info]Connectingtoroot@192.168.3.115(192.168.3.115:22)..

  Checkingslaverecoveryenvironmentsettings..

  Opening/var/lib/mysql/relay-log.info...ok.

  Relaylogfoundat/var/lib/mysql,uptomysqld-relay-bin.000004

  Temporaryrelaylogfileis/var/lib/mysql/mysqld-relay-bin.000004

  Testingmysqlconnectionandprivileges..Warning:Usingapasswordonthecommandlineinterfacecanbeinsecure.

  done.

  Testingmysqlbinlogoutput..done.

  Cleaninguptestfile(s)..done.

  WedJun2402:02:302015-[info]Executingcommand:apply_diff_relay_logs--command=test--slave_user=root--slave_host=192.168.3.117--slave_ip=192.168.3.117--slave_port=3306--workdir=/tmp--target_version=5.6.25-log--manager_version=0.53--relay_log_info=/var/lib/mysql/relay-log.info--relay_dir=/var/lib/mysql/--slave_pass=xxx

  WedJun2402:02:302015-[info]Connectingtoroot@192.168.3.117(192.168.3.117:22)..

  Checkingslaverecoveryenvironmentsettings..

  Opening/var/lib/mysql/relay-log.info...ok.

  Relaylogfoundat/var/lib/mysql,uptomysqld-relay-bin.000003

  Temporaryrelaylogfileis/var/lib/mysql/mysqld-relay-bin.000003

  Testingmysqlconnectionandprivileges..Warning:Usingapasswordonthecommandlineinterfacecanbeinsecure.

  done.

  Testingmysqlbinlogoutput..done.

  Cleaninguptestfile(s)..done.

  WedJun2402:02:312015-[info]Slavessettingscheckdone.

  WedJun2402:02:312015-[info]

  192.168.3.110(currentmaster)

  +--192.168.3.118

  +--192.168.3.115

  +--192.168.3.117

  WedJun2402:02:312015-[info]Checkingreplicationhealthon192.168.3.118..

  WedJun2402:02:312015-[info]ok.

  WedJun2402:02:312015-[info]Checkingreplicationhealthon192.168.3.115..

  WedJun2402:02:312015-[info]ok.

  WedJun2402:02:312015-[info]Checkingreplicationhealthon192.168.3.117..

  WedJun2402:02:312015-[info]ok.

  WedJun2402:02:312015-[info]Checkingmaster_ip_failover_scriptstatus:

  WedJun2402:02:312015-[info]/usr/local/bin/master_ip_failover--command=status--ssh_user=root--orig_master_host=192.168.3.110--orig_master_ip=192.168.3.110--orig_master_port=3306

  INSCRIPTTEST====/etc/init.d/keepalivedstop==/etc/init.d/keepalivedstart===

  CheckingtheStatusofthescript..OK

  WedJun2402:02:312015-[info]OK.

  WedJun2402:02:312015-[warning]shutdown_scriptisnotdefined.

  WedJun2402:02:312015-[info]Gotexitcode0(Notmasterdead).

  MySQLReplicationHealthisOK.

  可以看见没有报错。/usr/local/bin/master_ip_failover添加或者修改的内容意思是当主库数据库发生故障时,会触发MHA切换,MHAManager会停掉主库上的keepalived服务,触发虚拟ip漂移到备选从库,从而完成切换。当然可以在keepalived里面引入脚本,这个脚本监控mysql是否正常运行,如果不正常,则调用该脚本杀掉keepalived进程。

  keepalived的vip漂移原理及配置项说明可以查看文档小谈keepalivedvip漂移原理与VRRP协议。

  14.MHA测试

  到此为止,基本MHA集群已经配置完毕。接下来就是实际的测试环节了。通过一些测试来看一下MHA到底是如何进行工作的。下面将从MHA自动failover、手动failover两种方式来介绍MHA的工作情况。

  14.1.自动Failover(必须先启动MHAManager,否则无法自动切换,当然手动切换不需要开启MHAManager监控)

  自动failover模拟测试的操作步骤如下:

  (1)使用sysbench生成测试数据(使用yum快速安装),sysbench的使用说明可以查看文档sysbench使用详解

  yuminstallsysbench-y

  在主库(192.168.3.110)上进行sysbench数据生成,在sbtest库下生成sbtest表,共100W记录。

  [root@192.168.3.110~]#sysbench--test=oltp--oltp-table-size=1000000--oltp-read-only=off--mysql-db=sps--oltp-table-name=sbtest1--init-rng=on--num-threads=16--max-requests=0--oltp-dist-type=uniform--max-time=1800--mysql-user=root--mysql-socket=/var/lib/mysql/mysql.sock--mysql-password=123456--db-driver=mysql--mysql-table-engine=innodb--oltp-test-mode=complexperpare

  (2)停掉slavesql线程,模拟主从延时(192.168.3.118)

  mysql>stopslaveio_thread;

  QueryOK,0rowsaffected(0.08sec)

  mysql>

  另外两台slave我们没有停止io线程,所以还在继续接收日志。

  (3)模拟sysbench压力测试

  在主库上(192.168.3.110)进行压力测试,持续时间为3分钟,产生大量的binlog。

  [root@192.168.3.110~]#sysbench--test=oltp--oltp-table-size=1000000--oltp-read-only=off--mysql-db=sps--oltp-table-name=sbtest1--init-rng=on--num-threads=16--max-requests=0--oltp-dist-type=uniform--max-time=1800--mysql-user=root--mysql-socket=/var/lib/mysql/mysql.sock--mysql-password=123456--db-driver=mysql--mysql-table-engine=innodb--oltp-test-mode=complexrun

  sysbench0.4.12:multi-threadedsystemevaluationbenchmark

  Runningthetestwithfollowingoptions:

  Numberofthreads:16

  Initializingrandomnumbergeneratorfromtimer.

  DoingOLTPtest.

  RunningmixedOLTPtest

  UsingUniformdistribution

  Using"BEGIN"forstartingtransactions

  Usingauto_incontheidcolumn

  Threadsstarted!

  Timelimitexceeded,exiting...

  (lastmessagerepeated15times)

  Done.

  OLTPteststatistics:

  queriesperformed:

  read:15092

  write:5390

  other:2156

  total:22638

  transactions:1078(5.92persec.)

  deadlocks:0(0.00persec.)

  read/writerequests:20482(112.56persec.)

  otheroperations:2156(11.85persec.)

  Testexecutionsummary:

  totaltime:181.9728s

  totalnumberofevents:1078

  totaltimetakenbyeventexecution:2910.4518

  per-requeststatistics:

  min:934.29ms

  avg:2699.86ms

  max:7679.95ms

  approx.95percentile:4441.47ms

  Threadsfairness:

  events(avg/stddev):67.3750/1.49

  executiontime(avg/stddev):181.9032/0.11

  (4)开启slave(192.168.3.118)上的IO线程,追赶落后于master的binlog。

  mysql>startslaveio_thread;

  QueryOK,0rowsaffected(0.00sec)

  mysql>

  (5)杀掉主库mysql进程,模拟主库发生故障,进行自动failover操作。

  [root@192.168.3.110~]#pkill-9mysqld

  (6)查看MHA切换日志,了解整个切换过程,在192.168.3.123上查看日志(注意:由于测试多次切换过程,下面摘录的是192.168.3.118为主机,192.168.3.110为候选主机的切换log):

  [root@192.168.3.123~]#tail-f/var/log/masterha/app1/manager.log

  INSCRIPTTEST====/etc/init.d/keepalivedstop==/etc/init.d/keepalivedstart===

  CheckingtheStatusofthescript..OK

  ThuJun1113:52:092015-[info]OK.

  ThuJun1113:52:092015-[warning]shutdown_scriptisnotdefined.

  ThuJun1113:52:092015-[info]Setmasterpinginterval1seconds.

  ThuJun1113:52:092015-[info]Setsecondarycheckscript:/usr/local/bin/masterha_secondary_check-sB-dev10-sB-dev18--user=root--master_host=B-dev18--master_ip=192.168.3.118--master_port=3306

  ThuJun1113:52:092015-[info]Startingpinghealthcheckon192.168.3.118(192.168.3.118:3306)..

  ThuJun1113:52:092015-[info]Ping(SELECT)succeeded,waitinguntilMySQLdoesn'trespond..

  ThuJun1113:54:012015-[warning]GoterroronMySQLselectping:2013(LostconnectiontoMySQLserverduringquery)

  ThuJun1113:54:022015-[info]ExecutingSSHcheckscript:save_binary_logs--command=test--start_pos=4--binlog_dir=/var/lib/mysql--output_file=/tmp/save_binary_logs_test--manager_version=0.53--binlog_prefix=mysql-bin

  ThuJun1113:54:022015-[info]Executingseconarynetworkcheckscript:/usr/local/bin/masterha_secondary_check-sB-dev10-sB-dev18--user=root--master_host=B-dev18--master_ip=192.168.3.118--master_port=3306--user=root--master_host=192.168.3.118--master_ip=192.168.3.118--master_port=3306

  ThuJun1113:54:022015-[info]HealthCheck:SSHto192.168.3.118isreachable.

  ThuJun1113:54:022015-[warning]GoterroronMySQLconnect:2013(LostconnectiontoMySQLserverat'readinginitialcommunicationpacket',systemerror:111)

  ThuJun1113:54:022015-[warning]Connectionfailed1time(s)..

  MonitoringserverB-dev10isreachable,MasterisnotreachablefromB-dev10.OK.

  ThuJun1113:54:032015-[warning]GoterroronMySQLconnect:2013(LostconnectiontoMySQLserverat'readinginitialcommunicationpacket',systemerror:111)

  ThuJun1113:54:032015-[warning]Connectionfailed2time(s)..

  MonitoringserverB-dev18isreachable,MasterisnotreachablefromB-dev18.OK.

  ThuJun1113:54:042015-[info]Masterisnotreachablefromallothermonitoringservers.Failovershouldstart.

  ThuJun1113:54:042015-[warning]GoterroronMySQLconnect:2013(LostconnectiontoMySQLserverat'readinginitialcommunicationpacket',systemerror:111)

  ThuJun1113:54:042015-[warning]Connectionfailed3time(s)..

  ThuJun1113:54:042015-[warning]Masterisnotreachablefromhealthchecker!

  ThuJun1113:54:042015-[warning]Master192.168.3.118(192.168.3.118:3306)isnotreachable!

  ThuJun1113:54:042015-[warning]SSHisreachable.

  ThuJun1113:54:042015-[info]Connectingtoamasterserverfailed.Readingconfigurationfile/etc/masterha_default.cnfand/etc/masterha/app1.cnfagain,andtryingtoconnecttoallserverstocheckserverstatus..

  ThuJun1113:54:042015-[warning]Globalconfigurationfile/etc/masterha_default.cnfnotfound.Skipping.

  ThuJun1113:54:042015-[info]Readingapplicationdefaultconfigurationsfrom/etc/masterha/app1.cnf..

  ThuJun1113:54:042015-[info]Readingserverconfigurationsfrom/etc/masterha/app1.cnf..

  ThuJun1113:54:132015-[info]DeadServers:

  ThuJun1113:54:132015-[info]192.168.3.118(192.168.3.118:3306)

  ThuJun1113:54:132015-[info]AliveServers:

  ThuJun1113:54:132015-[info]192.168.3.110(192.168.3.110:3306)

  ThuJun1113:54:132015-[info]192.168.3.115(192.168.3.115:3306)

  ThuJun1113:54:132015-[info]192.168.3.117(192.168.3.117:3306)

  ThuJun1113:54:132015-[info]AliveSlaves:

  ThuJun1113:54:132015-[info]192.168.3.110(192.168.3.110:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  ThuJun1113:54:132015-[info]Replicatingfrom192.168.3.118(192.168.3.118:3306)

  ThuJun1113:54:132015-[info]PrimarycandidateforthenewMaster(candidate_masterisset)

  ThuJun1113:54:132015-[info]192.168.3.115(192.168.3.115:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  ThuJun1113:54:132015-[info]Replicatingfrom192.168.3.118(192.168.3.118:3306)

  ThuJun1113:54:132015-[info]192.168.3.117(192.168.3.117:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  ThuJun1113:54:132015-[info]Replicatingfrom192.168.3.118(192.168.3.118:3306)

  ThuJun1113:54:132015-[info]Checkingslaveconfigurations..

  ThuJun1113:54:132015-[info]read_only=1isnotsetonslave192.168.3.110(192.168.3.110:3306).

  ThuJun1113:54:132015-[warning]relay_log_purge=0isnotsetonslave192.168.3.110(192.168.3.110:3306).

  ThuJun1113:54:132015-[info]read_only=1isnotsetonslave192.168.3.115(192.168.3.115:3306).

  ThuJun1113:54:132015-[info]read_only=1isnotsetonslave192.168.3.117(192.168.3.117:3306).

  ThuJun1113:54:132015-[info]Checkingreplicationfilteringsettings..

  ThuJun1113:54:132015-[info]Replicationfilteringcheckok.

  ThuJun1113:54:132015-[info]Masterisdown!

  ThuJun1113:54:132015-[info]Terminatingmonitoringscript.

  ThuJun1113:54:132015-[info]Gotexitcode20(Masterdead).

  ThuJun1113:54:132015-[info]MHA::MasterFailoverversion0.53.

  ThuJun1113:54:132015-[info]Startingmasterfailover.

  ThuJun1113:54:132015-[info]

  ThuJun1113:54:132015-[info]*Phase1:ConfigurationCheckPhase..

  ThuJun1113:54:132015-[info]

  ThuJun1113:54:192015-[info]DeadServers:

  ThuJun1113:54:192015-[info]192.168.3.118(192.168.3.118:3306)

  ThuJun1113:54:192015-[info]Checkingmasterreachabilityviamysql(doublecheck)..

  ThuJun1113:54:192015-[info]ok.

  ThuJun1113:54:192015-[info]AliveServers:

  ThuJun1113:54:192015-[info]192.168.3.110(192.168.3.110:3306)

  ThuJun1113:54:192015-[info]192.168.3.115(192.168.3.115:3306)

  ThuJun1113:54:192015-[info]192.168.3.117(192.168.3.117:3306)

  ThuJun1113:54:192015-[info]AliveSlaves:

  ThuJun1113:54:192015-[info]192.168.3.110(192.168.3.110:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  ThuJun1113:54:192015-[info]Replicatingfrom192.168.3.118(192.168.3.118:3306)

  ThuJun1113:54:192015-[info]PrimarycandidateforthenewMaster(candidate_masterisset)

  ThuJun1113:54:192015-[info]192.168.3.115(192.168.3.115:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  ThuJun1113:54:192015-[info]Replicatingfrom192.168.3.118(192.168.3.118:3306)

  ThuJun1113:54:192015-[info]192.168.3.117(192.168.3.117:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  ThuJun1113:54:192015-[info]Replicatingfrom192.168.3.118(192.168.3.118:3306)

  ThuJun1113:54:222015-[info]**Phase1:ConfigurationCheckPhasecompleted.

  ThuJun1113:54:222015-[info]

  ThuJun1113:54:222015-[info]*Phase2:DeadMasterShutdownPhase..

  ThuJun1113:54:222015-[info]

  ThuJun1113:54:222015-[info]Forcingshutdownsothatapplicationsneverconnecttothecurrentmaster..

  ThuJun1113:54:222015-[info]ExecutingmasterIPdeactivatationscript:

  ThuJun1113:54:222015-[info]/usr/local/bin/master_ip_failover--orig_master_host=192.168.3.118--orig_master_ip=192.168.3.118--orig_master_port=3306--command=stopssh--ssh_user=root

  INSCRIPTTEST====/etc/init.d/keepalivedstop==/etc/init.d/keepalivedstart===

  DisablingtheVIPonoldmaster:192.168.3.118

  ThuJun1113:54:232015-[info]done.

  ThuJun1113:54:232015-[warning]shutdown_scriptisnotset.Skippingexplicitshuttingdownofthedeadmaster.

  ThuJun1113:54:232015-[info]*Phase2:DeadMasterShutdownPhasecompleted.

  ThuJun1113:54:232015-[info]

  ThuJun1113:54:232015-[info]*Phase3:MasterRecoveryPhase..

  ThuJun1113:54:232015-[info]

  ThuJun1113:54:232015-[info]*Phase3.1:GettingLatestSlavesPhase..

  ThuJun1113:54:232015-[info]

  ThuJun1113:54:232015-[info]Thelatestbinarylogfile/positiononallslavesismysql-bin.000001:145958297

  ThuJun1113:54:232015-[info]Latestslaves(Slavesthatreceivedrelaylogfilestothelatest):

  ThuJun1113:54:232015-[info]192.168.3.110(192.168.3.110:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  ThuJun1113:54:232015-[info]Replicatingfrom192.168.3.118(192.168.3.118:3306)

  ThuJun1113:54:232015-[info]PrimarycandidateforthenewMaster(candidate_masterisset)

  ThuJun1113:54:232015-[info]Theoldestbinarylogfile/positiononallslavesismysql-bin.000001:142466719

  ThuJun1113:54:232015-[info]Oldestslaves:

  ThuJun1113:54:232015-[info]192.168.3.115(192.168.3.115:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  ThuJun1113:54:232015-[info]Replicatingfrom192.168.3.118(192.168.3.118:3306)

  ThuJun1113:54:232015-[info]

  ThuJun1113:54:232015-[info]*Phase3.2:SavingDeadMaster'sBinlogPhase..

  ThuJun1113:54:232015-[info]

  ThuJun1113:54:242015-[info]Fetchingdeadmaster'sbinarylogs..

  ThuJun1113:54:242015-[info]Executingcommandonthedeadmaster192.168.3.118(192.168.3.118:3306):save_binary_logs--command=save--start_file=mysql-bin.000001--start_pos=145958297--binlog_dir=/var/lib/mysql--output_file=/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlog--handle_raw_binlog=1--disable_log_bin=0--manager_version=0.53

  Creating/tmpifnotexists..ok.

  Concatbinary/relaylogsfrommysql-bin.000001pos145958297tomysql-bin.000001EOFinto/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlog..

  Dumpingbinlogformatdescriptionevent,fromposition0to120..ok.

  Dumpingeffectivebinlogdatafrom/var/lib/mysql/mysql-bin.000001position145958297totail(145969603)..ok.

  Concatsucceeded.

  ThuJun1113:54:262015-[info]scpfromroot@192.168.3.118:/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlogtolocal:/var/log/masterha/app1/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlogsucceeded.

  ThuJun1113:54:262015-[info]HealthCheck:SSHto192.168.3.110isreachable.

  ThuJun1113:54:272015-[info]HealthCheck:SSHto192.168.3.115isreachable.

  ThuJun1113:54:282015-[info]HealthCheck:SSHto192.168.3.117isreachable.

  ThuJun1113:54:292015-[info]

  ThuJun1113:54:292015-[info]*Phase3.3:DeterminingNewMasterPhase..

  ThuJun1113:54:292015-[info]

  ThuJun1113:54:292015-[info]Findingthelatestslavethathasallrelaylogsforrecoveringotherslaves..

  ThuJun1113:54:292015-[info]Checkingwhether192.168.3.110hasrelaylogsfromtheoldestposition..

  ThuJun1113:54:292015-[info]Executingcommand:apply_diff_relay_logs--command=find--latest_mlf=mysql-bin.000001--latest_rmlp=145958297--target_mlf=mysql-bin.000001--target_rmlp=142466719--server_id=110--workdir=/tmp--timestamp=20150611135413--manager_version=0.53--relay_log_info=/var/lib/mysql/relay-log.info--relay_dir=/var/lib/mysql/:

  Opening/var/lib/mysql/relay-log.info...ok.

  Relaylogfoundat/var/lib/mysql,uptomysqld-relay-bin.000003

  Fastrelaylogpositionsearchsucceeded.

  Targetrelaylogfile/positionfound.start_file:mysqld-relay-bin.000003,start_pos:79548.

  TargetrelaylogFOUND!

  ThuJun1113:54:302015-[info]OK.192.168.3.110hasallrelaylogs.

  ThuJun1113:54:302015-[info]Searchingnewmasterfromslaves..

  ThuJun1113:54:302015-[info]Candidatemastersfromtheconfigurationfile:

  ThuJun1113:54:302015-[info]192.168.3.110(192.168.3.110:3306)Version=5.6.25-log(oldestmajorversionbetweenslaves)log-bin:enabled

  ThuJun1113:54:302015-[info]Replicatingfrom192.168.3.118(192.168.3.118:3306)

  ThuJun1113:54:302015-[info]PrimarycandidateforthenewMaster(candidate_masterisset)

  ThuJun1113:54:302015-[info]Non-candidatemasters:

  ThuJun1113:54:302015-[info]Searchingfromcandidate_masterslaveswhichhavereceivedthelatestrelaylogevents..

  ThuJun1113:54:302015-[info]Newmasteris192.168.3.110(192.168.3.110:3306)

  ThuJun1113:54:302015-[info]Startingmasterfailover..

  ThuJun1113:54:302015-[info]

  From:

  192.168.3.118(currentmaster)

  +--192.168.3.110

  +--192.168.3.115

  +--192.168.3.117

  To:

  192.168.3.110(newmaster)

  +--192.168.3.115

  +--192.168.3.117

  ThuJun1113:54:302015-[info]

  ThuJun1113:54:302015-[info]*Phase3.3:NewMasterDiffLogGenerationPhase..

  ThuJun1113:54:302015-[info]

  ThuJun1113:54:302015-[info]Thisserverhasallrelaylogs.Noneedtogeneratedifffilesfromthelatestslave.

  ThuJun1113:54:302015-[info]Sendingbinlog..

  ThuJun1113:54:312015-[info]scpfromlocal:/var/log/masterha/app1/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlogtoroot@192.168.3.110:/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlogsucceeded.

  ThuJun1113:54:312015-[info]

  ThuJun1113:54:312015-[info]*Phase3.4:MasterLogApplyPhase..

  ThuJun1113:54:312015-[info]

  ThuJun1113:54:312015-[info]*NOTICE:Ifanyerrorhappensfromthisphase,manualrecoveryisneeded.

  ThuJun1113:54:312015-[info]Startingrecoveryon192.168.3.110(192.168.3.110:3306)..

  ThuJun1113:54:312015-[info]Generatingdiffssucceeded.

  ThuJun1113:54:312015-[info]Waitinguntilallrelaylogsareapplied.

  ThuJun1113:59:462015-[info]done.

  ThuJun1113:59:462015-[info]Gettingslavestatus..

  ThuJun1113:59:462015-[info]Thisslave(192.168.3.110)'sExec_Master_Log_PosequalstoRead_Master_Log_Pos(mysql-bin.000001:145958297).NoneedtorecoverfromExec_Master_Log_Pos.

  ThuJun1113:59:462015-[info]Connectingtothetargetslavehost192.168.3.110,runningrecoverscript..

  ThuJun1113:59:462015-[info]Executingcommand:apply_diff_relay_logs--command=apply--slave_user=root--slave_host=192.168.3.110--slave_ip=192.168.3.110--slave_port=3306--apply_files=/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlog--workdir=/tmp--target_version=5.6.25-log--timestamp=20150611135413--handle_raw_binlog=1--disable_log_bin=0--manager_version=0.53--slave_pass=xxx

  ThuJun1113:59:492015-[info]

  MySQLclientversionis5.6.25.Using--binary-mode.

  Applyingdifferentialbinary/relaylogfiles/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlogon192.168.3.110:3306.Thismaytakelongtime...

  Applyinglogfilessucceeded.

  ThuJun1113:59:492015-[info]Allrelaylogsweresuccessfullyapplied.

  ThuJun1113:59:492015-[info]Gettingnewmaster'sbinlognameandposition..

  ThuJun1113:59:492015-[info]mysql-bin.000001:11760

  ThuJun1113:59:492015-[info]Allotherslavesshouldstartreplicationfromhere.Statementshouldbe:CHANGEMASTERTOMASTER_HOST='192.168.3.110',MASTER_PORT=3306,MASTER_LOG_FILE='mysql-bin.000001',MASTER_LOG_POS=11760,MASTER_USER='repl',MASTER_PASSWORD='xxx';

  ThuJun1113:59:492015-[info]ExecutingmasterIPactivatescript:

  ThuJun1113:59:492015-[info]/usr/local/bin/master_ip_failover--command=start--ssh_user=root--orig_master_host=192.168.3.118--orig_master_ip=192.168.3.118--orig_master_port=3306--new_master_host=192.168.3.110--new_master_ip=192.168.3.110--new_master_port=3306

  INSCRIPTTEST====/etc/init.d/keepalivedstop==/etc/init.d/keepalivedstart===

  EnablingtheVIP-192.168.3.250onthenewmaster-192.168.3.110

  ThuJun1113:59:492015-[info]OK.

  ThuJun1113:59:492015-[info]**Finishedmasterrecoverysuccessfully.

  ThuJun1113:59:492015-[info]*Phase3:MasterRecoveryPhasecompleted.

  ThuJun1113:59:492015-[info]

  ThuJun1113:59:492015-[info]*Phase4:SlavesRecoveryPhase..

  ThuJun1113:59:492015-[info]

  ThuJun1113:59:492015-[info]*Phase4.1:StartingParallelSlaveDiffLogGenerationPhase..

  ThuJun1113:59:492015-[info]

  ThuJun1113:59:492015-[info]--Slavedifffilegenerationonhost192.168.3.115(192.168.3.115:3306)started,pid:17896.Checktmplog/var/log/masterha/app1/192.168.3.115_3306_20150611135413.logifittakestime..

  ThuJun1113:59:492015-[info]--Slavedifffilegenerationonhost192.168.3.117(192.168.3.117:3306)started,pid:17897.Checktmplog/var/log/masterha/app1/192.168.3.117_3306_20150611135413.logifittakestime..

  ThuJun1113:59:522015-[info]

  ThuJun1113:59:522015-[info]Logmessagesfrom192.168.3.117...

  ThuJun1113:59:522015-[info]

  ThuJun1113:59:492015-[info]Server192.168.3.117receivedrelaylogsupto:mysql-bin.000001:142482387

  ThuJun1113:59:492015-[info]Needtogetdiffsfromthelatestslave(192.168.3.110)upto:mysql-bin.000001:145958297(usingthelatestslave'srelaylogs)

  ThuJun1113:59:502015-[info]Connectingtothelatestslavehost192.168.3.110,generatingdiffrelaylogfiles..

  ThuJun1113:59:502015-[info]Executingcommand:apply_diff_relay_logs--command=generate_and_send--scp_user=root--scp_host=192.168.3.117--latest_mlf=mysql-bin.000001--latest_rmlp=145958297--target_mlf=mysql-bin.000001--target_rmlp=142482387--server_id=110--diff_file_readtolatest=/tmp/relay_from_read_to_latest_192.168.3.117_3306_20150611135413.binlog--workdir=/tmp--timestamp=20150611135413--handle_raw_binlog=1--disable_log_bin=0--manager_version=0.53--relay_log_info=/var/lib/mysql/relay-log.info--relay_dir=/var/lib/mysql/

  ThuJun1113:59:522015-[info]

  Opening/var/lib/mysql/relay-log.info...ok.

  Relaylogfoundat/var/lib/mysql,uptomysqld-relay-bin.000003

  Fastrelaylogpositionsearchsucceeded.

  Targetrelaylogfile/positionfound.start_file:mysqld-relay-bin.000003,start_pos:95216.

  Concatbinary/relaylogsfrommysqld-relay-bin.000003pos95216tomysqld-relay-bin.000003EOFinto/tmp/relay_from_read_to_latest_192.168.3.117_3306_20150611135413.binlog..

  Dumpingbinlogformatdescriptionevent,fromposition0to283..ok.

  Dumpingeffectivebinlogdatafrom/var/lib/mysql/mysqld-relay-bin.000003position95216totail(3571126)..ok.

  Concatsucceeded.

  Generatingdiffrelaylogsucceeded.Savedat/tmp/relay_from_read_to_latest_192.168.3.117_3306_20150611135413.binlog.

  scpB-dev10:/tmp/relay_from_read_to_latest_192.168.3.117_3306_20150611135413.binlogtoroot@192.168.3.117(22)succeeded.

  ThuJun1113:59:522015-[info]Generatingdifffilessucceeded.

  ThuJun1113:59:522015-[info]Endoflogmessagesfrom192.168.3.117.

  ThuJun1113:59:522015-[info]--Slavediffloggenerationonhost192.168.3.117(192.168.3.117:3306)succeeded.

  ThuJun1113:59:532015-[info]

  ThuJun1113:59:532015-[info]Logmessagesfrom192.168.3.115...

  ThuJun1113:59:532015-[info]

  ThuJun1113:59:492015-[info]Server192.168.3.115receivedrelaylogsupto:mysql-bin.000001:142466719

  ThuJun1113:59:492015-[info]Needtogetdiffsfromthelatestslave(192.168.3.110)upto:mysql-bin.000001:145958297(usingthelatestslave'srelaylogs)

  ThuJun1113:59:502015-[info]Connectingtothelatestslavehost192.168.3.110,generatingdiffrelaylogfiles..

  ThuJun1113:59:502015-[info]Executingcommand:apply_diff_relay_logs--command=generate_and_send--scp_user=root--scp_host=192.168.3.115--latest_mlf=mysql-bin.000001--latest_rmlp=145958297--target_mlf=mysql-bin.000001--target_rmlp=142466719--server_id=110--diff_file_readtolatest=/tmp/relay_from_read_to_latest_192.168.3.115_3306_20150611135413.binlog--workdir=/tmp--timestamp=20150611135413--handle_raw_binlog=1--disable_log_bin=0--manager_version=0.53--relay_log_info=/var/lib/mysql/relay-log.info--relay_dir=/var/lib/mysql/

  ThuJun1113:59:532015-[info]

  Opening/var/lib/mysql/relay-log.info...ok.

  Relaylogfoundat/var/lib/mysql,uptomysqld-relay-bin.000003

  Fastrelaylogpositionsearchsucceeded.

  Targetrelaylogfile/positionfound.start_file:mysqld-relay-bin.000003,start_pos:79548.

  Concatbinary/relaylogsfrommysqld-relay-bin.000003pos79548tomysqld-relay-bin.000003EOFinto/tmp/relay_from_read_to_latest_192.168.3.115_3306_20150611135413.binlog..

  Dumpingbinlogformatdescriptionevent,fromposition0to283..ok.

  Dumpingeffectivebinlogdatafrom/var/lib/mysql/mysqld-relay-bin.000003position79548totail(3571126)..ok.

  Concatsucceeded.

  Generatingdiffrelaylogsucceeded.Savedat/tmp/relay_from_read_to_latest_192.168.3.115_3306_20150611135413.binlog.

  scpB-dev10:/tmp/relay_from_read_to_latest_192.168.3.115_3306_20150611135413.binlogtoroot@192.168.3.115(22)succeeded.

  ThuJun1113:59:532015-[info]Generatingdifffilessucceeded.

  ThuJun1113:59:532015-[info]Endoflogmessagesfrom192.168.3.115.

  ThuJun1113:59:532015-[info]--Slavediffloggenerationonhost192.168.3.115(192.168.3.115:3306)succeeded.

  ThuJun1113:59:532015-[info]Generatingrelaydifffilesfromthelatestslavesucceeded.

  ThuJun1113:59:532015-[info]

  ThuJun1113:59:532015-[info]*Phase4.2:StartingParallelSlaveLogApplyPhase..

  ThuJun1113:59:532015-[info]

  ThuJun1113:59:532015-[info]--Slaverecoveryonhost192.168.3.115(192.168.3.115:3306)started,pid:17910.Checktmplog/var/log/masterha/app1/192.168.3.115_3306_20150611135413.logifittakestime..

  ThuJun1113:59:532015-[info]--Slaverecoveryonhost192.168.3.117(192.168.3.117:3306)started,pid:17911.Checktmplog/var/log/masterha/app1/192.168.3.117_3306_20150611135413.logifittakestime..

  ThuJun1114:14:142015-[info]

  ThuJun1114:14:142015-[info]Logmessagesfrom192.168.3.117...

  ThuJun1114:14:142015-[info]

  ThuJun1113:59:532015-[info]Sendingbinlog..

  ThuJun1113:59:532015-[info]scpfromlocal:/var/log/masterha/app1/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlogtoroot@192.168.3.117:/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlogsucceeded.

  ThuJun1113:59:532015-[info]Startingrecoveryon192.168.3.117(192.168.3.117:3306)..

  ThuJun1113:59:532015-[info]Generatingdiffssucceeded.

  ThuJun1113:59:532015-[info]Waitinguntilallrelaylogsareapplied.

  ThuJun1113:59:532015-[info]done.

  ThuJun1113:59:542015-[info]Gettingslavestatus..

  ThuJun1113:59:542015-[info]Thisslave(192.168.3.117)'sExec_Master_Log_Pos(mysql-bin.000001:142481989)doesnotequaltoRead_Master_Log_Pos(mysql-bin.000001:142482387).Itislikelythatrelaylogwascutduringtransaction.NeedtorecoverfromExec_Master_Log_Pos.

  ThuJun1113:59:542015-[info]Savinglocalrelaylogsfromexecpostoreadposon192.168.3.117:frommysqld-relay-bin.000002:142482152totheendoftherelaylog..

  ThuJun1113:59:542015-[info]Executingcommand:save_binary_logs--command=save--start_file=mysqld-relay-bin.000002--start_pos=142482152--output_file=/tmp/relay_from_exec_to_read_192.168.3.117_3306_20150611135413.binlog--handle_raw_binlog=1--disable_log_bin=0--manager_version=0.53--relay_log_info=/var/lib/mysql/relay-log.info--binlog_dir=/var/lib/mysql/

  ThuJun1113:59:542015-[info]

  Creating/tmpifnotexists..ok.

  Concatbinary/relaylogsfrommysqld-relay-bin.000002pos142482152tomysqld-relay-bin.000002EOFinto/tmp/relay_from_exec_to_read_192.168.3.117_3306_20150611135413.binlog..

  Dumpingbinlogformatdescriptionevent,fromposition0to283..ok.

  Dumpingeffectivebinlogdatafrom/var/lib/mysql/mysqld-relay-bin.000002position142482152totail(142482550)..ok.

  Concatsucceeded.

  ThuJun1113:59:542015-[info]Connectingtothetargetslavehost192.168.3.117,runningrecoverscript..

  ThuJun1113:59:542015-[info]Executingcommand:apply_diff_relay_logs--command=apply--slave_user=root--slave_host=192.168.3.117--slave_ip=192.168.3.117--slave_port=3306--apply_files=/tmp/relay_from_exec_to_read_192.168.3.117_3306_20150611135413.binlog,/tmp/relay_from_read_to_latest_192.168.3.117_3306_20150611135413.binlog,/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlog--workdir=/tmp--target_version=5.6.25-log--timestamp=20150611135413--handle_raw_binlog=1--disable_log_bin=0--manager_version=0.53--slave_pass=xxx

  ThuJun1114:14:122015-[info]

  Concatallapplyfilesto/tmp/total_binlog_for_192.168.3.117_3306.20150611135413.binlog..

  Copyingthefirstbinlogfile/tmp/relay_from_exec_to_read_192.168.3.117_3306_20150611135413.binlogto/tmp/total_binlog_for_192.168.3.117_3306.20150611135413.binlog..ok.

  Dumpingbinlogheadevents(rotateevents),skippingformatdescriptioneventsfrom/tmp/relay_from_read_to_latest_192.168.3.117_3306_20150611135413.binlog..dumpeduptopos283.ok.

  /tmp/relay_from_read_to_latest_192.168.3.117_3306_20150611135413.binloghaseffectivebinlogeventsfrompos283.

  Dumpingeffectivebinlogdatafrom/tmp/relay_from_read_to_latest_192.168.3.117_3306_20150611135413.binlogposition283totail(3476193)..ok.

  Dumpingbinlogheadevents(rotateevents),skippingformatdescriptioneventsfrom/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlog..dumpeduptopos120.ok.

  /tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binloghaseffectivebinlogeventsfrompos120.

  Dumpingeffectivebinlogdatafrom/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlogposition120totail(11426)..ok.

  Concatsucceeded.

  Allapplytargetbinarylogsareconcatinatedat/tmp/total_binlog_for_192.168.3.117_3306.20150611135413.binlog.

  MySQLclientversionis5.6.25.Using--binary-mode.

  Applyingdifferentialbinary/relaylogfiles/tmp/relay_from_exec_to_read_192.168.3.117_3306_20150611135413.binlog,/tmp/relay_from_read_to_latest_192.168.3.117_3306_20150611135413.binlog,/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlogon192.168.3.117:3306.Thismaytakelongtime...

  Applyinglogfilessucceeded.

  ThuJun1114:14:122015-[info]Allrelaylogsweresuccessfullyapplied.

  ThuJun1114:14:122015-[info]Resettingslave192.168.3.117(192.168.3.117:3306)andstartingreplicationfromthenewmaster192.168.3.110(192.168.3.110:3306)..

  ThuJun1114:14:142015-[info]ExecutedCHANGEMASTER.

  ThuJun1114:14:142015-[info]Slavestarted.

  ThuJun1114:14:142015-[info]Endoflogmessagesfrom192.168.3.117.

  ThuJun1114:14:142015-[info]--Slaverecoveryonhost192.168.3.117(192.168.3.117:3306)succeeded.

  ThuJun1114:14:372015-[info]

  ThuJun1114:14:372015-[info]Logmessagesfrom192.168.3.115...

  ThuJun1114:14:372015-[info]

  ThuJun1113:59:532015-[info]Sendingbinlog..

  ThuJun1113:59:532015-[info]scpfromlocal:/var/log/masterha/app1/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlogtoroot@192.168.3.115:/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlogsucceeded.

  ThuJun1113:59:532015-[info]Startingrecoveryon192.168.3.115(192.168.3.115:3306)..

  ThuJun1113:59:532015-[info]Generatingdiffssucceeded.

  ThuJun1113:59:532015-[info]Waitinguntilallrelaylogsareapplied.

  ThuJun1113:59:532015-[info]done.

  ThuJun1113:59:532015-[info]Gettingslavestatus..

  ThuJun1113:59:532015-[info]Thisslave(192.168.3.115)'sExec_Master_Log_Pos(mysql-bin.000001:142466642)doesnotequaltoRead_Master_Log_Pos(mysql-bin.000001:142466719).Itislikelythatrelaylogwascutduringtransaction.NeedtorecoverfromExec_Master_Log_Pos.

  ThuJun1113:59:532015-[info]Savinglocalrelaylogsfromexecpostoreadposon192.168.3.115:frommysqld-relay-bin.000002:142466805totheendoftherelaylog..

  ThuJun1113:59:532015-[info]Executingcommand:save_binary_logs--command=save--start_file=mysqld-relay-bin.000002--start_pos=142466805--output_file=/tmp/relay_from_exec_to_read_192.168.3.115_3306_20150611135413.binlog--handle_raw_binlog=1--disable_log_bin=0--manager_version=0.53--relay_log_info=/var/lib/mysql/relay-log.info--binlog_dir=/var/lib/mysql/

  ThuJun1113:59:542015-[info]

  Creating/tmpifnotexists..ok.

  Concatbinary/relaylogsfrommysqld-relay-bin.000002pos142466805tomysqld-relay-bin.000002EOFinto/tmp/relay_from_exec_to_read_192.168.3.115_3306_20150611135413.binlog..

  Dumpingbinlogformatdescriptionevent,fromposition0to283..ok.

  Dumpingeffectivebinlogdatafrom/var/lib/mysql/mysqld-relay-bin.000002position142466805totail(142466882)..ok.

  Concatsucceeded.

  ThuJun1113:59:542015-[info]Connectingtothetargetslavehost192.168.3.115,runningrecoverscript..

  ThuJun1113:59:542015-[info]Executingcommand:apply_diff_relay_logs--command=apply--slave_user=root--slave_host=192.168.3.115--slave_ip=192.168.3.115--slave_port=3306--apply_files=/tmp/relay_from_exec_to_read_192.168.3.115_3306_20150611135413.binlog,/tmp/relay_from_read_to_latest_192.168.3.115_3306_20150611135413.binlog,/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlog--workdir=/tmp--target_version=5.6.25-log--timestamp=20150611135413--handle_raw_binlog=1--disable_log_bin=0--manager_version=0.53--slave_pass=xxx

  ThuJun1114:14:352015-[info]

  Concatallapplyfilesto/tmp/total_binlog_for_192.168.3.115_3306.20150611135413.binlog..

  Copyingthefirstbinlogfile/tmp/relay_from_exec_to_read_192.168.3.115_3306_20150611135413.binlogto/tmp/total_binlog_for_192.168.3.115_3306.20150611135413.binlog..ok.

  Dumpingbinlogheadevents(rotateevents),skippingformatdescriptioneventsfrom/tmp/relay_from_read_to_latest_192.168.3.115_3306_20150611135413.binlog..dumpeduptopos283.ok.

  /tmp/relay_from_read_to_latest_192.168.3.115_3306_20150611135413.binloghaseffectivebinlogeventsfrompos283.

  Dumpingeffectivebinlogdatafrom/tmp/relay_from_read_to_latest_192.168.3.115_3306_20150611135413.binlogposition283totail(3491861)..ok.

  Dumpingbinlogheadevents(rotateevents),skippingformatdescriptioneventsfrom/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlog..dumpeduptopos120.ok.

  /tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binloghaseffectivebinlogeventsfrompos120.

  Dumpingeffectivebinlogdatafrom/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlogposition120totail(11426)..ok.

  Concatsucceeded.

  Allapplytargetbinarylogsareconcatinatedat/tmp/total_binlog_for_192.168.3.115_3306.20150611135413.binlog.

  MySQLclientversionis5.6.25.Using--binary-mode.

  Applyingdifferentialbinary/relaylogfiles/tmp/relay_from_exec_to_read_192.168.3.115_3306_20150611135413.binlog,/tmp/relay_from_read_to_latest_192.168.3.115_3306_20150611135413.binlog,/tmp/saved_master_binlog_from_192.168.3.118_3306_20150611135413.binlogon192.168.3.115:3306.Thismaytakelongtime...

  Applyinglogfilessucceeded.

  ThuJun1114:14:352015-[info]Allrelaylogsweresuccessfullyapplied.

  ThuJun1114:14:352015-[info]Resettingslave192.168.3.115(192.168.3.115:3306)andstartingreplicationfromthenewmaster192.168.3.110(192.168.3.110:3306)..

  ThuJun1114:14:372015-[info]ExecutedCHANGEMASTER.

  ThuJun1114:14:372015-[info]Slavestarted.

  ThuJun1114:14:372015-[info]Endoflogmessagesfrom192.168.3.115.

  ThuJun1114:14:372015-[info]--Slaverecoveryonhost192.168.3.115(192.168.3.115:3306)succeeded.

  ThuJun1114:14:372015-[info]Allnewslaveserversrecoveredsuccessfully.

  ThuJun1114:14:372015-[info]

  ThuJun1114:14:372015-[info]*Phase5:Newmastercleanupphease..

  ThuJun1114:14:372015-[info]

  ThuJun1114:14:372015-[info]Resettingslaveinfoonthenewmaster..

  ThuJun1114:14:372015-[info]192.168.3.110:Resettingslaveinfosucceeded.

  ThuJun1114:14:372015-[info]Masterfailoverto192.168.3.110(192.168.3.110:3306)completedsuccessfully.

  ThuJun1114:14:372015-[info]Deletedserver1entryfrom/etc/masterha/app1.cnf.

  ThuJun1114:14:372015-[info]

  -----FailoverReport-----

  app1:MySQLMasterfailover192.168.3.118to192.168.3.110succeeded

  Master192.168.3.118isdown!

  CheckMHAManagerlogsatB-dev23:/var/log/masterha/app1/manager.logfordetails.

  Startedautomated(non-interactive)failover.

  InvalidatedmasterIPaddresson192.168.3.118.

  Thelatestslave192.168.3.110(192.168.3.110:3306)hasallrelaylogsforrecovery.

  Selected192.168.3.110asanewmaster.

  192.168.3.110:OK:Applyingalllogssucceeded.

  192.168.3.110:OK:ActivatedmasterIPaddress.

  192.168.3.117:Generatingdifferentialrelaylogsupto192.168.3.110succeeded.

  192.168.3.115:Generatingdifferentialrelaylogsupto192.168.3.110succeeded.

  Generatingrelaydifffilesfromthelatestslavesucceeded.

  192.168.3.117:OK:Applyingalllogssucceeded.Slavestarted,replicatingfrom192.168.3.110.

  192.168.3.115:OK:Applyingalllogssucceeded.Slavestarted,replicatingfrom192.168.3.110.

  192.168.3.110:Resettingslaveinfosucceeded.

  Masterfailoverto192.168.3.110(192.168.3.110:3306)completedsuccessfully.

  看到最后的Masterfailoverto192.168.3.110(192.168.3.110:3306)completedsuccessfully.说明备选master现在已经上位了。

  从上面的输出可以看出整个MHA的切换过程,共包括以下的步骤:

  连接主机三次失败确认主机宕机,摘除宕机的vip

  检查从库中使用的最新binlogfile/pos

  检查从库中使用的最老binlogfile/pos

  获取宕掉主机的binlog:根据步骤1获取的binlog位置开始保存未同步到从机的binlog(导出binlog描述事件,导出影响的binlog数据);将导出的binlog发送到mha-manage服务器指定目录下

  healthcheck:监控ssh到其他的机器是否正常

  检查最新binlogfile/pos的从机是否有需要恢复其他从机所需的所有relaylog

  如果其他从机跟最新从机pos不一致,需要最新从机的relaylog进行一致性恢复

  检查是否设置候选主机

  开始候选主机切换:先检查候选主机是否有所有的relaylog,没有首先从最新从机恢复,然后将mha收到宕机主机缺省的binlog发送给候选主机,识别差异的relaylog并将差异的事件应用所有的relaylog,执行vip漂移到候选主机,候选主机恢复结束

  其他从机relaylog恢复:先检查从机是否有所有的relaylog,没有首先从最新从机恢复,然后将mha收到宕机主机缺省的binlog发送给从机,识别差异的relaylog并将差异的事件应用所有的relaylog

  将所有从机指向为候选主机

  启动MHAManger监控,查看集群里面现在谁是master。

  (7)检查切换后数据是主从数据是否一致

  在主机和候选主机上分别安装percona-toolkit,安装流程及使用文档请查看percona-toolkit简明教程。

  这里简单说下pt-table-checksum的工作原理:pt-table-checksum在主上执行检查语句在线检查mysql复制的一致性,生成replace语句,然后通过复制传递到从,再通过update更新master_src的值。通过检测从上this_src和master_src的值从而判断复制是否一致。注意:使用的时候选择业务地峰的时候运行,因为运行的时候会造成表的部分记录锁定。使用–max-load来指定最大的负载情况,如果达到那个负载这个暂停运行。如果发现有不一致的数据,可以使用pt-table-sync工具来修复。

  在切换后的新主机上使用命令:

  [root@192.168.3.110~]#pt-table-checksum--nocheck-replication-filters--chunk-size=1000000--no-check-binlog-format--replicate=sps.checksums--databases=spsh=127.0.0.1,u=root,p=123456,P=3306

  TSERRORSDIFFSROWSCHUNKSSKIPPEDTIMETABLE

  06-23T18:06:38000103.885sps.sps_account

  06-23T18:06:42003103.118sps.sps_activiti_goods

  06-23T18:06:46000104.613sps.sps_activiti_prop

  06-23T18:06:49002103.013sps.sps_activiti_re

  06-23T18:06:52003102.858sps.sps_activity

  06-23T18:06:55001103.359sps.sps_app_version

  06-23T18:06:58004102.894sps.sps_brand

  06-23T18:07:02003272103.367sps.sps_district

  06-23T18:07:050016102.935sps.sps_goods

  06-23T18:07:08006103.761sps.sps_goods_brand_cat

  06-23T18:07:1200332103.418sps.sps_goods_category

  06-23T18:07:16000104.175sps.sps_goods_category_template

  06-23T18:07:20005104.061sps.sps_goods_comment

  06-23T18:07:24009103.784sps.sps_goods_desc

  06-23T18:07:27004103.487sps.sps_goods_favorite

  06-23T18:07:330014105.847sps.sps_goods_media

  06-23T18:07:37002103.811sps.sps_goods_recommend

  06-23T18:07:41002103.623sps.sps_goods_recommend_group

  06-23T18:07:44004103.402sps.sps_goods_recommend_group_cate

  06-23T18:07:480012103.662sps.sps_goods_sku_attribute_value

  06-23T18:07:520019103.951sps.sps_goods_stock

  06-23T18:07:55006103.557sps.sps_goods_template

  06-23T18:07:58002103.226sps.sps_goods_tmpl_sku_attribute_line

  06-23T18:08:03005104.183sps.sps_goods_tmpl_sku_attribute_line_value

  06-23T18:08:0800208733105.422sps.sps_member

  06-23T18:08:11002103.304sps.sps_member_account_seq

  06-23T18:08:150027721103.575sps.sps_member_address

  06-23T18:08:18002103.116sps.sps_member_baby_birthday

  06-23T18:08:21001102.876sps.sps_member_grade_change_info

  06-23T18:08:24000103.563sps.sps_member_grade_dict

  06-23T18:08:280016103.848sps.sps_member_points_seq

  06-23T18:08:31000102.958sps.sps_menu

  06-23T18:08:35000103.687sps.sps_message

  06-23T18:08:39005103.589sps.sps_message_text

  06-23T18:08:4100746102.848sps.sps_order

  06-23T18:08:460094104.529sps.sps_order_goods_detail

  06-23T18:08:490023103.030sps.sps_order_status

  06-23T18:08:52000103.390sps.sps_payment

  06-23T18:08:570013104.559sps.sps_promotion

  06-23T18:09:000015102.986sps.sps_promotion_goods

  06-23T18:09:04000104.158sps.sps_role

  06-23T18:09:08000103.880sps.sps_role_menu

  06-23T18:09:110016102.997sps.sps_seller

  06-23T18:09:15004103.808sps.sps_shop_trolley

  06-23T18:09:18006103.409sps.sps_sign_in

  06-23T18:09:21002103.067sps.sps_sku_attribute

  06-23T18:09:25005103.871sps.sps_sku_attribute_value

  06-23T18:09:30001104.466sps.sps_store

  06-23T18:09:33000103.378sps.sps_store_account

  06-23T18:09:36006103.331sps.sps_sys_config

  06-23T18:09:40008103.391sps.sps_sys_message

  06-23T18:09:430014103.182sps.sps_ticket

  06-23T18:09:460012103.614sps.sps_ticket_goods

  06-23T18:09:5000140103.419sps.sps_ticket_his

  06-23T18:09:530012103.041sps.sps_ticket_prop

  06-23T18:09:56000103.091sps.sps_user_role

  06-23T18:10:00005103.592sps.sps_warehouse

  从结果中,我们可以看到DIFFS都为0,说明主从数据一致,故障切换流程没有数据丢失。

  14.2.手动Failover(MHAManager必须没有运行)

  手动failover,这种场景意味着在业务上没有启用MHA自动切换功能,当主服务器故障时,人工手动调用MHA来进行故障切换操作,具体命令如下:

  注意:如果MHAManager检测到没有dead的server,将报错,并结束failover:

  MonJun2121:23:332015-[info]DeadServers:

  MonJun2121:23:332015-[error][/usr/local/share/perl5/MHA/MasterFailover.pm,ln181]Noneofserverisdead.Stopfailover.

  MonJun2121:23:332015-[error][/usr/local/share/perl5/MHA/ManagerUtil.pm,ln178]GotERROR:at/usr/local/bin/masterha_master_switchline53

  进行手动切换命令如下:

  [root@192.168.3.123~]#masterha_master_switch--master_state=dead--conf=/etc/masterha/app1.cnf--dead_master_host=192.168.3.110--dead_master_port=3306--new_master_host=192.168.3.118--new_master_port=3306--ignore_last_failover

  14.3.修复宕机的Master

  通常情况下自动切换以后,原master可能已经废弃掉,待原master主机修复后,如果数据完整的情况下,可能想把原来master重新作为新主库的slave,这时我们可以借助当时自动切换时刻的MHA日志来完成对原master的修复。下面是提取相关日志的命令:

  [root@192.168.3.123app1]#grep-i"Allotherslavesshouldstart"manager.log

  MonJun2122:28:332015-[info]Allotherslavesshouldstartreplicationfromhere.Statementshouldbe:CHANGEMASTERTOMASTER_HOST='192.168.3.118',MASTER_PORT=3306,MASTER_LOG_FILE='mysql-bin.000022',MASTER_LOG_POS=506716,MASTER_USER='repl',MASTER_PASSWORD='xxx';

  [root@192.168.3.123app1]#

  获取上述信息以后,就可以直接在修复后的master上执行changemasterto相关操作,重新作为从库了。

  Atlas配置

  1.简介

  Atlas是由Qihoo360,Web平台部基础架构团队开发维护的一个基于MySQL协议的数据中间层项目。它是在mysql-proxy0.8.2版本的基础上,对其进行了优化,增加了一些新的功能特性。360内部使用Atlas运行的mysql业务,每天承载的读写请求数达几十亿条。

  主要功能:

  读写分离

  从库负载均衡

  IP过滤

  自动分表

  DBA可平滑上下线DB

  自动摘除宕机的DB

  2.安装(192.168.3.124,192.168.3.125)

  2.1RPM/DEB包安装

  从https://github.com/Qihoo360/Atlas/releases页面下载最新版RPM包,我这里下载的是Atlas-2.2.1.el6.x86_64.rpm,然后执行:sudorpm–iAtlas-2.2.1.el6.x86_64.rpm安装。

  如果是Debian/Ubuntu用户,下载DEB包后用dpkg-i安装即可。

  注意事项:

  (1).Atlas只能安装运行在64位的系统上。

  (2).Centos5.X安装Atlas-XX.el5.x86_64.rpm,Centos6.X安装Atlas-XX.el6.x86_64.rpm。

  (3).如果执行sudorpm–iAtlas-XX.el6.x86_64.rpm,提示类似:“file/usr/local/mysql-proxy/bin/encryptfrominstallofAtlas-2.0.1-1.x86_64conflictswithfilefrompackageAtlas-1.0.3-1.x86_64”错误,则表示该系统之前已经安装过Atlas-1.0.3-1.x86_64,需要执行:sudorpm–eAtlas-1.0.3-1.x86_64,将之前安装的Atlas删除掉,再执行sudorpm–iAtlas-XX.el6.x86_64.rpm安装新版本的Atlas。

  (4).后端mysql版本应大于5.1,建议使用Mysql5.6

  2.2源码包安装

  对Atlas系统不太熟悉的新用户,推荐使用rpm包的安装Atlas。对于想源码编译安装Atlas的用户,可参考http://blog.qixingzhong.com/2013/09/centos-install-atlas.html提供的安装方式进行安装。注意:Atlas为了提升内存管理效率,使用了jemalloc库,源码编译的时候需要额外安装jemalloc库。

  2.3配置文件修改

  Atlas运行需要依赖一个配置文件(test.cnf)。在运行Atlas之前,需要对该文件进行配置。Atlas的安装目录是/usr/local/mysql-proxy,进入安装目录下的conf目录,可以看到已经有一个名为test.cnf的默认配置文件,我们只需要修改里面的某些配置项,不需要从头写一个配置文件。

  配置文件如下:

  [mysql-proxy]

  #带#号的为非必需的配置项目

  #管理接口的用户名

  admin-username=atlas

  #管理接口的密码

  admin-password=hugnew

  #Atlas后端连接的MySQL主库的IP和端口,可设置多项,用逗号分隔(这里是Mysql主机提供的vip)

  proxy-backend-addresses=192.168.3.250:3306

  #Atlas后端连接的MySQL从库的IP和端口,@后面的数字代表权重,用来作负载均衡,若省略则默认为1,可设置多项,用逗号分隔

  proxy-read-only-backend-addresses=192.168.3.115:3306,192.168.3.117:3306

  #用户名与其对应的加密过的MySQL密码,密码使用PREFIX/bin目录下的加密程序encrypt加密,下行的user1和user2为示例,将其替换为你的MySQL的用户名和加密密码!

  pwds=root:or9RzYJBtvga8s/oWZlMvQ==

  #设置Atlas的运行方式,设为true时为守护进程方式,设为false时为前台方式,一般开发调试时设为false,线上运行时设为true,true后面不能有空格。

  daemon=false

  #设置Atlas的运行方式,设为true时Atlas会启动两个进程,一个为monitor,一个为worker,monitor在worker意外退出后会自动将其重启,设为false时只有worker,没有monitor,一般开发调试时设为false,线上运行时设为true,true后面不能有空格。

  keepalive=false

  #工作线程数,对Atlas的性能有很大影响,可根据情况适当设置

  event-threads=4

  #日志级别,分为message、warning、critical、error、debug五个级别

  log-level=message

  #日志存放的路径

  log-path=/usr/local/mysql-proxy/log

  #SQL日志的开关,可设置为OFF、ON、REALTIME,OFF代表不记录SQL日志,ON代表记录SQL日志,REALTIME代表记录SQL日志且实时写入磁盘,默认为OFF

  sql-log=ON

  #慢日志输出设置。当设置了该参数时,则日志只输出执行时间超过sql-log-slow(单位:ms)的日志记录。不设置该参数则输出全部日志。

  sql-log-slow=10

  #实例名称,用于同一台机器上多个Atlas实例间的区分(instance名与配置文件名test.cnf要一样)

  instance=test

  #Atlas监听的工作接口IP和端口

  proxy-address=0.0.0.0:1234

  #Atlas监听的管理接口IP和端口

  admin-address=0.0.0.0:2345

  #分表设置,此例中person为库名,mt为表名,id为分表字段,3为子表数量,可设置多项,以逗号分隔,若不分表则不需要设置该项

  #tables=person.mt.id.3

  #默认字符集,设置该项后客户端不再需要执行SETNAMES语句

  charset=utf8

  #允许连接Atlas的客户端的IP,可以是精确IP,也可以是IP段,以逗号分隔,若不设置该项则允许所有IP连接,否则只允许列表中的IP连接

  #client-ips=127.0.0.1,192.168.1

  #Atlas前面挂接的LVS的物理网卡的IP(注意不是虚IP),若有LVS且设置了client-ips则此项必须设置,否则可以不设置

  #lvs-ips=192.168.1.1

  2.4运行Atlas

  进入/usr/local/mysql-proxy/bin目录,执行下面的命令启动、重启或停止Atlas。

  (1).sudo./mysql-proxydteststart,启动Atlas。

  (2).sudo./mysql-proxydtestrestart,重启Atlas。

  (3).sudo./mysql-proxydteststop,停止Atlas。

  注意:

  (1).运行文件是:mysql-proxyd(不是mysql-proxy)。

  (2).test是conf目录下配置文件的名字,也是配置文件里instance项的名字,三者需要统一。

  (3).可以使用ps-ef|grepmysql-proxy查看Atlas是否已经启动或停止。

  3.Atlas测试

  1234端口是Atlas客户端的接口,在Mysql主服务器上执行命令:mysql-h127.0.0.1-P1234-uroot-p123456,这里的用户名密码是Mysql访问的用户名及密码,测试连接Mysql正常,client端连接Atlas执行几条SQL语句正常说明说明Atlas正常。

  2345端口是Atlas的管理接口,此接口提供了查看mysql状态,设置上线、下线后端mysql服务器。

  访问方法使用mysql命令

  mysql-h127.0.0.1-P2345-uatlas-phugnew

  这里的用户名密码是上面配置文件中的admin-username=admin、admin-password=password

  下面是管理接口的使用帮助

  SELECT*FROMhelp;#查看帮助信息

  SELECT*FROMbackends;#查看后端主机状态

  SETOFFLINE$backend_id#下线一台主机,$backend_id==SELECT*FROMbackends查出来的ID

  SETONLINE$backend_id#上线一台主机

  ADDMASTER$backend#添加一台master节点,如:ADDMASTER10.57.1.131;

  ADDSLAVE$backend#添加一台slave节点,如:ADDSLAVE10.57.1.132;

  REMOVEBACKEND$backend_id#删除一个节点

  更多资料查看官方地址:https://github.com/Qihoo360/Atlas

  LVS配置

  1.简介

  LVS集群采用IP负载均衡技术和基于内容请求分发技术。调度器具有很好的吞吐率,将请求均衡地转移到不同的服务器上执行,且调度器自动屏蔽掉服务器的故障,从而将一组服务器构成一个高性能的、高可用的虚拟服务器。整个服务器集群的结构对客户是透明的,而且无需修改客户端和服务器端的程序。

  LVS集群中实现的三种IP负载均衡技术是VS/NAT、VS/TUN和VS/DR技术。它们主要解决系统的可伸缩性和透明性问题,如何通过负载调度器将请求高效地分发到不同的服务器执行,使得由多台独立计算机组成的集群系统成为一台虚拟服务器;客户端应用程序与集群系统交互时,就像与一台高性能的服务器交互一样。

  在内核中的连接调度算法上,IPVS已实现了以下八种调度算法:

  轮叫调度(Round-RobinScheduling)

  加权轮叫调度(WeightedRound-RobinScheduling)

  最小连接调度(Least-ConnectionScheduling)

  加权最小连接调度(WeightedLeast-ConnectionScheduling)

  基于局部性的最少链接(Locality-BasedLeastConnectionsScheduling)

  带复制的基于局部性最少链接(Locality-BasedLeastConnectionswithReplicationScheduling)

  目标地址散列调度(DestinationHashingScheduling)

  源地址散列调度(SourceHashingScheduling)

  更多详细信息可以查看官方文档:http://www.linuxvirtualserver.org/zh/

  2.准备工作(每台LVS机器)

  首先禁掉防火墙和SeLinux,分为临时禁掉和永久禁掉,命令如下

  ####临时禁掉防火墙,重启服务器失效

  serviceiptablesstop

  ####临时禁掉Selinux,重启服务器失效

  setenforce0

  ####永久禁掉防火墙,需重启服务器

  chkconfigiptablesoff

  ####永久禁掉Selinux,需重启服务器

  修改/etc/selinux/config文件中设置SELINUX=disabled,然后重启服务器。

  3.安装LVS软件

  LVS软件包括二部分:

  IPVS模块,LVS已经是Linux标准内核的一部分,直接被编译在内核中!

  IPVS管理工具IPVSadm,如:IPVSadm-1.26

  (1)检查是否安装了IPVS模块

  LVS已经是Linux标准内核的一部分,直接被编译在内核中,可以直接使用modprobe命令即可查看。

  (2)安装IPVS管理软件及依赖包

  yum-yinstallipvsadm

  yum-yinstallkernel-devel

  yum-yinstalllibnl-devel

  yum-yinstallpopt-devel

  4.安装keepalived

  tarzxfkeepalived-1.2.14.tar.gz

  cdkeepalived-1.2.2

  ./configure--prefix=/usr/local/keepalived--sysconf=/etc--with-kernel-dir=/usr/src/kernels/2.6.32-71.el6.x86_64/

  直到出现四个:yes

  UseIPVSFramework:Yes

  IPVSsyncdaemonsupport:Yes

  IPVSuselibnl:Yes

  UseVRRPFramework:Yes

  然后进行编译

  make&&makeinstall

  ln-s/usr/local/sbin/keepalived/sbin///做个软链接

  执行命令keepalived–help命令检查安装keepalived是否成功。

  5.编辑配置文件

  5.1在主机上配置keepalived(192.168.3.119)

  keepalived的原理可以查看文档小谈keepalivedvip漂移原理与VRRP协议。

  vi/etc/keepalived/keepalived.conf

  !ConfigurationFileforkeepalived

  global_defs{

  notification_email{

  acassen@firewall.loc#设置报警邮件地址,可以设置多个,每行1个

  failover@firewall.loc#需开启邮件报警及本机的Sendmail服务。

  sysadmin@firewall.loc

  }

  notification_email_fromAlexandre.Cassen@firewall.loc

  smtp_server127.0.0.1#设置SMTPServer地址;

  smtp_connect_timeout30

  router_idLVS_DEVEL

  }

  ########VRRPInstance########

  vrrp_instanceVI_1{

  stateMASTER#指定Keepalived的角色,MASTER为主机服务器,BACKUP为备用服务器

  interfaceeth0#BACKUP为备用服务器

  virtual_router_id51

  priority100#定义优先级,数字越大,优先级越高,主机器必须大于备机。

  advert_int1

  authentication{

  auth_typePASS#设置验证类型,主要有PASS和AH两种

  auth_pass1111#设置验证密码

  }

  virtual_ipaddress{

  192.168.3.251#设置主DR的虚拟IP地址(virtualIP),可多设,但必须每行1个

  }

  }

  ########VirtualServer########

  virtual_server192.168.3.2511234{#注意IP地址与端口号之间用空格隔开

  delay_loop6#设置健康检查时间,单位是秒

  lb_algolc#设置负载调度算法,默认为rr,即轮询算法,最优秀是wlc算法,这里使用的是最小链接算法

  lb_kindDR#设置LVS实现LB机制,有NAT、TUNN和DR三个模式可选

  #nat_mask255.255.255.0

  #persistence_timeout50#会话保持时间,单位为秒

  protocolTCP#指定转发协议类型,有TCP和UDP两种

  real_server192.168.3.1241234{

  weight1#配置节点权值,数字越大权值越高

  TCP_CHECK{

  connect_timeout3#表示3秒无响应,则超时

  nb_get_retry3#表示重试次数

  delay_before_retry3#表示重试间隔

  }

  }

  real_server192.168.3.1251234{#配置服务器节点,即RealServer2的publicIP

  weight3#配置节点权值,数字越大权值越高

  TCP_CHECK{

  connect_timeout3#表示3秒无响应,则超时

  nb_get_retry3#表示重试次数

  delay_before_retry3#表示重试间隔

  }

  }

  }

  5.2在备机上配置keepalived(192.168.3.120)

  备机配置文件与主机有两处不同,如下

  把“stateMASTER”修改为“stateBACKUP”#即备用服务器;

  把“priority100”修改为“priority90”#优先级为90。

  6.配置realserver节点(Atlas所有服务器)

  (1)安装httpd

  yum–yinstallhttpd

  (2)启动httpd服务

  修改httpd.conf配置文件

  vi/etc/httpd/conf/httpd.conf

  #265ServerNamewww.example.com:80//去掉前面的#号

  /etc/init.d/httpdrestart

  (3)编写ip绑定脚本realserver.sh

  #addforchkconfig

  #chkconfig:23457030#234都是文本界面,5就是图形界面X,70启动顺序号,30系统关闭,脚本

  #止顺序号

  #description:RealServer'sscript#关于脚本的简短描述

  #processname:realserver.sh#第一个进程名,后边设置自动时会用到

  #!/bin/bash

  VIP=192.168.3.251

  source/etc/rc.d/init.d/functions

  case"$1"in

  start)

  ifconfiglo:0$VIPnetmask255.255.255.255broadcast$VIP

  ifconfiglo:0$VIPnetmask255.255.255.255broadcast$VIP

  /sbin/routeadd-host$VIPdevlo:0

  echo"1">/proc/sys/net/ipv4/conf/lo/arp_ignore

  echo"2">/proc/sys/net/ipv4/conf/lo/arp_announce

  echo"1">/proc/sys/net/ipv4/conf/all/arp_ignore

  echo"2">/proc/sys/net/ipv4/conf/all/arp_announce

  echo"RealServerStartOK"

  ;;

  stop)

  ifconfiglo:0down

  routedel$VIP>/dev/null2>&1

  echo"0">/proc/sys/net/ipv4/conf/lo/arp_ignore

  echo"0">/proc/sys/net/ipv4/conf/lo/arp_announce

  echo"0">/proc/sys/net/ipv4/conf/all/arp_ignore

  echo"0">/proc/sys/net/ipv4/conf/all/arp_announce

  echo"RealServerStoped"

  ;;

  *)

  echo"Usage:$0{start|stop}"

  exit1

  esac

  exit0

  (4)为realserver.sh添加权限并执行

  chmod755realserver.sh

  ./realserver.shstart

  (5)检查ip绑定是否成功

  执行ipaddr命令

  1:lo:<LOOPBACK,UP,LOWER_UP>mtu65536qdiscnoqueuestateUNKNOWN

  link/loopback00:00:00:00:00:00brd00:00:00:00:00:00

  inet127.0.0.1/8scopehostlo

  inet192.168.3.251/32brd192.168.3.251scopegloballo:0

  inet6::1/128scopehost

  valid_lftforeverpreferred_lftforever

  2:eth0:<BROADCAST,MULTICAST,UP,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000

  link/ether00:1a:4a:e0:4c:41brdff:ff:ff:ff:ff:ff

  inet192.168.3.124/24brd192.168.3.255scopeglobaleth0

  inet6fe80::21a:4aff:fee0:4c41/64scopelink

  valid_lftforeverpreferred_lftforever

  可以看到ip已经绑定成功

  7.lvs启动keepalived并检查是否生效

  (1)lvs主机启动keepalived并检查是否生效

  启动keepalived

  /etc/init.d/keepalivedstart

  检查是否生效,执行ipvsadm-Ln命令

  IPVirtualServerversion1.2.1(size=4096)

  ProtLocalAddress:PortSchedulerFlags

  ->RemoteAddress:PortForwardWeightActiveConnInActConn

  TCP192.168.3.251:1234lc

  ->192.168.3.124:1234Route500

  ->192.168.3.125:1234Route500

  执行ipaddr命令

  1:lo:<LOOPBACK,UP,LOWER_UP>mtu65536qdiscnoqueuestateUNKNOWN

  link/loopback00:00:00:00:00:00brd00:00:00:00:00:00

  inet127.0.0.1/8scopehostlo

  inet6::1/128scopehost

  valid_lftforeverpreferred_lftforever

  2:eth0:<BROADCAST,MULTICAST,UP,LOWER_UP>mtu1500qdiscpfifo_faststateUPqlen1000

  link/ether00:1a:4a:e0:4c:32brdff:ff:ff:ff:ff:ff

  inet192.168.3.119/23brd192.168.3.255scopeglobaleth0

  inet6fe80::21a:4aff:fee0:4c32/64scopelink

  valid_lftforeverpreferred_lftforever

  可以看到lvs主机配置已经生效。

  (2)lvs备机启动keepalived并检查是否生效

  启动keepalived

  /etc/init.d/keepalivedstart

  检查是否生效,执行ipvsadm-Ln命令

  IPVirtualServerversion1.2.1(size=4096)

  ProtLocalAddress:PortSchedulerFlags

  ->RemoteAddress:PortForwardWeightActiveConnInActConn

  TCP192.168.3.251:1234lc

  ->192.168.3.124:1234Route500

  ->192.168.3.125:1234Route500

  可以看到lvs备机配置也已经生效。

  8.LVS测试

  对lvs的vip地址192.168.3.251:1234发请求,使用Linux抓包工具tcpdump在atlas服务器上抓包,抓包命令为tcpdump-ieth0-nnport1234andsrchost192.168.3.251,测试lvs负载请求到atlas是否正常。

  结语

  ok,Mysql半复制+MHA+Keepalived+Atlas+LVS的集成就说到这里,这里面涉及到知识点可能比较多也比较散,如果有疑问的可以先查看官方文档,也可以找我交流哦。
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关于我们
联系我们
  • 电话:010-86393388
  • 邮件:udn@yonyou.com
  • 地址:北京市海淀区北清路68号
移动客户端下载
关注我们
  • 微信公众号:yonyouudn
  • 扫描右侧二维码关注我们
  • 专注企业互联网的技术社区
版权所有:用友网络科技股份有限公司82041 京ICP备05007539号-11 京公网网备安1101080209224 Powered by Discuz!
快速回复 返回列表 返回顶部