Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision Both sides next revision
litespeed_wiki:lslb:ha [2018/08/03 18:41]
Jackson Zhang [Replication our of sync? What required?]
litespeed_wiki:lslb:ha [2018/08/03 20:05]
Lisa Clarke Proofreading
Line 1: Line 1:
-====== How to enable HA for Web ADC====== +====== How to Enable High Availability ​for Web ADC ====== 
-LiteSpeed Web ADC HA setup provides a failover setup for two ADC Nodes. When one node is temporarily unavailable,​ the other one will automatically detect and take over the traffic. ​+LiteSpeed Web ADC's High Availability (HA) configuration ​provides a failover setup for two ADC Nodes. When one node is temporarily unavailable,​ the other one will automatically detect and take over the traffic. ​
  
-LiteSpeed Web ADC HA will use keepalived ​to detect the failover.+LiteSpeed Web ADC HA will use Keepalived ​to detect the failover.
  
-===== Two example ​Nodes ===== +===== Two Example ​Nodes ===== 
-We will setup two nodes as an example:+We will set up two nodes as an example:
  
 Node1: 10.10.30.96 Node1: 10.10.30.96
Line 13: Line 13:
 Virtual IP: 10.10.31.31 Virtual IP: 10.10.31.31
  
-===== Install and configure ​keepalived ​===== +===== Install and configure ​Keepalived ​===== 
- +Before you configure ADC HA, You should install ​Keepalived ​on both node 1 and node 2.
-Before you configure ADC HA, You should install ​keepalived ​on both node 1 and node 2.+
 On CentOS, you can do yum install: On CentOS, you can do yum install:
   yum install keepalived   yum install keepalived
-or on Ubuntu/​Debian,​ you can do apt-get:+On Ubuntu/​Debian,​ you can do apt-get:
   apt-get install keepalived   apt-get install keepalived
  
-Then Start keepalived:+Then start Keepalived:
   service keepalived start   service keepalived start
   ​   ​
-Also need to setup autorestart during the system reboot:+You also need to set up autorestart during the system reboot:
   systemctl enable keepalived   systemctl enable keepalived
 or or
   chkconfig keepalived on   chkconfig keepalived on
    
-The keepalive ​configuration file is located at /​etc/​keepalived/​keepalived.conf,​ but you should not edit this configuration file directly, instead, you should use ADC Web Admin GUI -> HA config to add/config VIP.  ​The manually ​added VIP to keepalived ​config won't be picked up by ADC HA config. The VIP configure ​under ADC HA tab is just a GUI to update the keepalived ​config file. So you should just use the WebAdmin GUI to manage VIP if they want to see it in the status. We will explain on later steps on how to add VIP in GUI.+The Keepalived ​configuration file is located at ''​/​etc/​keepalived/​keepalived.conf''​, but you should not edit this configuration file directly. Instead, you should use **ADC Web Admin GUI > HA config** to add or configure a Virtual IP.  ​If you manually ​add a VIP to Keepalived ​config, it won't be picked up by ADC HA. The VIP configuration ​under the ADC's **HA** tab is just a GUI to update the Keepalived ​config file. 
  
 +You should alwaus use the WebAdmin GUI to manage VIPs if you want to see them in the status. ​
  
 ===== Configure HA on LiteSpeed Web ADC =====  ===== Configure HA on LiteSpeed Web ADC ===== 
- 
 ==== Node 1 ==== ==== Node 1 ====
- ​login ​to node 1 ADC Web Admin Console: sample ​configuration+Login to node 1 ADC Web Admin Console. Sample ​configuration:
  
   Server Address 10.10.30.96:​11122   Server Address 10.10.30.96:​11122
Line 48: Line 47:
 {{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha1.png?​800 |}} {{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha1.png?​800 |}}
  
- then "Add" ​HA interface:+Click **Add** in **HA Interfaces** to add a Virtual IP:
  
 {{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha1-add-internface.png?​800 |}}  ​ {{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha1-add-internface.png?​800 |}}  ​
  
-After VIP has been added through GUI, the configuration will be added to keepalived ​configuration and you will see keepalive configuration ​like:+After VIP has been added through ​the GUI, the configuration will be added to the Keepalived ​configuration and it will look like this:
  
-vi /​etc/​keepalived/​keepalived.conf+  ​vi /​etc/​keepalived/​keepalived.conf
  
   ###### start of VI_5 ######   ###### start of VI_5 ######
Line 76: Line 75:
  
 ==== Node 2 ==== ==== Node 2 ====
- +Login to node 1 ADC Web Admin Console. Sample ​configuration:
- ​login ​to node 1 ADC Web Admin Console: sample ​configuration+
  
   Server Address 10.10.30.97:​11122   Server Address 10.10.30.97:​11122
Line 90: Line 88:
 {{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha1.png?​800 |}}  ​ {{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha1.png?​800 |}}  ​
  
-then add HA interface:+Click **Add** in **HA Interfaces** to add a Virtual IP: 
 {{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha2-add-internface.png?​800 |}} {{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha2-add-internface.png?​800 |}}
  
-After VIP has been added through GUI, the configuration will be added to keepalived ​configuration and you will see keepalive configuration ​like:+After VIP has been added through ​the GUI, the configuration will be added to the Keepalived ​configurationand it will look like this:
  
 ###### start of VI_5 ###### ###### start of VI_5 ######
Line 115: Line 114:
  
 Note:  Note: 
-  - node1 virtual_router_id should be the same as node2 +  - node1 ''​virtual_router_id'' ​should be the same as node2  
-  - **"state MASTER/​BACKUP"​** ​doesn'​t really matter, since Higher ​priority one will be MASTER.+  - ''​state MASTER''​/''​BACKUP'' ​doesn'​t really matter, since the higher ​priority one will always ​be MASTER.
  
-===== Test IP failver ​===== +===== Test IP Failover ​===== 
-For IP failover, it is completely managed by keepalived, ​ADC just add a configuration management interface. ​So you should test IP failover:+IP failover is completely managed by Keepalived. The ADC just adds a configuration management interface. IP failover ​only happens when one server is completely down. The other server will then take over the IP. Shuting down LS ADC won't trigger an IP failover. ​
  
-1. check the master node, which currently is node 1, 10.10.30.96+It's a good idea to test IP failover. 
 + 
 +==== 1. check the master node, which currently is node 1, 10.10.30.96 ​====
  
  ​root@ha1-ubuntu:​~#​ ip a  ​root@ha1-ubuntu:​~#​ ip a
Line 141: Line 142:
 You can see the VIP 10.10.31.31. You can see the VIP 10.10.31.31.
  
-2.  Then test backup node, node 2, 10.10.30.96:+==== 2. Test backup node, node 2, 10.10.30.96 ​====
  
   root@ha2-ubuntu:​~#​ ip a   root@ha2-ubuntu:​~#​ ip a
Line 159: Line 160:
 You don't see VIP on node 2 when VIP is active on node 1, which is correct. You don't see VIP on node 2 when VIP is active on node 1, which is correct.
  
-3. Shutdown ​the master node 1, the VIP 10.10.31.31 should be migrated to the backup server node 2, you can check: ​+==== 3. Shut down the master node 1. ==== 
 +The VIP 10.10.31.31 should be migrated to the backup server node 2. You can check: 
 + 
   root@ha2-ubuntu:​~#​ ip a                                                                                                                                                ​   root@ha2-ubuntu:​~#​ ip a                                                                                                                                                ​
   1: lo: <​LOOPBACK,​UP,​LOWER_UP>​ mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000                                                                            ​   1: lo: <​LOOPBACK,​UP,​LOWER_UP>​ mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000                                                                            ​
Line 178: Line 181:
 You can see VIP 10.10.31.31 is assigned to node 2 now. You can see VIP 10.10.31.31 is assigned to node 2 now.
  
- +===== Check HA Status for Both Nodes ===== 
-IP failover is completely handled by keepalived, it only happens when one server is completely down, the other server will take over the IP. Shutingdown LS ADC won't trigger an IP failover.  +The HA Status page will look like the following when running:
- +
-For more fancy failover method, you may want to try BGP, like what cloudflare does, but it is not controlled by ADC.  +
-https://​blog.cloudflare.com/​cloudflares-architecture-eliminating-single-p/​ +
- +
- +
-===== Check HA Status for both nodes ===== +
- +
- HA Status page will look like the following when running:+
  
 On Node 1: On Node 1:
Line 195: Line 190:
 {{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha2-status.png?​800 |}} {{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha2-status.png?​800 |}}
  
-===== Replication our of sync? What required? ​=====+===== Fixing Out of Sync Replication ​=====
  
-Sometime ​you may see replication out of sync.+Sometimes ​you may see replication ​is out of sync:
  
 {{ :​litespeed_wiki:​lslb:​adc-ha-status-out-sync.png?​800 |}} {{ :​litespeed_wiki:​lslb:​adc-ha-status-out-sync.png?​800 |}}
  
-  - You will need to make sure node 1 and node 2 are configurated ​the same way. If they are configurated different way, you can not expect HA/​Replication working. ​ +Check the following: 
-  - If one ADC instance is down, the replication will be out of sync, that's expected, the ADC will try to restore synchronization in short time. +  - If one ADC instance is down, the replication will be out of sync, That's expected. The ADC will try to restore synchronization in short time. 
-===== Verify ​your listener,virtualhost, ClusterHTTP ​setup correctly ​=====+  - Make sure node 1 and node 2 are configurated the same way. If they are configurated differently,​ you can not expect HA/​Replication to work.  
 + 
 +===== Verify ​ListenerVirtual Host, ClusterHTTP ​are Set up Correctly ​=====
  
-Assuming ​you have configured the listener, virtual ​Host and backend clusterHTTP on both Node 1 and node 2 seperately.+We assume ​you have configured the listener, virtual ​host and backend clusterHTTP on both Node 1 and node 2 seperately. ​They should look something like this:
  
 Listener: ​ Listener: ​
-With IP failover, we recommend ​listener configuration ​listening on *:<​port>,​ instead of individual <​IP>:<​port>​.+With IP failover, we recommend listening on ''​*:<​port>​''​, instead of individual ​''​<​IP>:<​port>​''​.
  
 {{ :​litespeed_wiki:​lslb:​adc-ha-configuration-listener.png?​800 |}} {{ :​litespeed_wiki:​lslb:​adc-ha-configuration-listener.png?​800 |}}
Line 229: Line 226:
 ===== Testing VIP ===== ===== Testing VIP =====
  
-Try access ​10.10.31.31 (VIP) from the browser, you will see the backend server page. Disable one node, you can still see the webpage. Check ADC HA statuslive node will become Master when the other one down.+Try accessing ​10.10.31.31 (VIP) from the browser. You will see the backend server page. Disable one node, and you can still see the webpage. Check ADC HA status. The live node will become Master when the other one down.
  
 ===== Troubleshooting ===== ===== Troubleshooting =====
-==== Whe make configuration ​changes, ​it may need a full stop/​start ​====+==== Problems After Configuration Changes ==== 
 +When making ​changes ​to the configuration,​ such as changing the listener from ''<​IP>:​443''​ to ''​*:​443''​, a full stop/​start ​is required.
  
-When making changes ​to the configuration,​ such as changing the listener from <IP>:443 to "*:443"that requires a full stop/start.+==== Inconsistent HA Configuraton Between Boxes  ==== 
 +When you see an error similar ​to the following: 
 +  2018-08-03 16:24:​20.099467 [WARN] [REPL] HA configures are inconsistent between boxes, full replicaton can't proceed 
 +  2018-08-03 16:24:20.099520 [ERROR] [REPL] peer HA replication config is inconsistent,​ it must be fixed!
  
-==== HA configures are inconsistent between boxes  ==== +It is because the configuration is out of sync between two LS ADC instances. ​Replication ​only works if the two ADCs are serving the exact same sites. ​ You need to keep ADC configuration in sync. If it is out of sync temporarily,​ it will break the synchronization. Once the configurations are synced, ADC will restore replication synchronization. ​
-When you see similar error to the following:​ +
-2018-08-03 16:​24:​20.099467 [WARN] [REPL] HA configures are inconsistent between boxes, full replicaton can't proceed +
-2018-08-03 16:​24:​20.099520 [ERROR] [REPL] peer HA replication config is inconsistent,​ it must be fixed! +
- +
-It is because the configuration is out of sync between two LS ADC instances. ​replication ​only works if two ADC are serving the exact same sites. ​ You need to keep ADC configuration in sync. If it is out of sync temporarily,​ it will break the synchronization. Once config ​synced, ADC will restore replication synchronization. ​+
  
 +===== Alternative =====
 +For a more fancy failover method, you may want to try BGP, like what Cloudflare does, but it is not controlled by ADC. 
 +https://​blog.cloudflare.com/​cloudflares-architecture-eliminating-single-p/​
  
  • Admin
  • Last modified: 2019/03/18 12:21
  • by Kevin Fwu