Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
litespeed_wiki:lslb:ha [2018/08/03 18:28]
Jackson Zhang [Test IP failver] 		litespeed_wiki:lslb:ha [2020/11/18 15:36] (current)
Lisa Clarke Redirect to new Documentation Site
Line 1: Line 1:
-====== How to enable HA for Web ADC? ====== +~~REDIRECT>​https://​docs.litespeedtech.com/products/lsadc/settings/~~
-LiteSpeed Web ADC HA setup provides a failover setup for two ADC Nodes. When one node is temporarily unavailable,​ the other one will automatically detect and take over the traffic.  +
- +
-LiteSpeed Web ADC HA will use keepalived to detect the failover. +
- +
-===== Two example Nodes ===== +
-We will setup two nodes as an example: +
- +
-Node1: 10.10.30.96 +
- +
-Node2: 10.10.30.97 +
- +
-Virtual IP: 10.10.31.31 +
- +
-===== Install and configure keepalived ===== +
- +
-Before you configure ADC HA, You should install keepalived on both node 1 and node 2. +
-On CentOS, you can do yum install: +
-  yum install keepalived +
-or on Ubuntu/​Debian,​ you can do apt-get: +
-  apt-get install keepalived +
- +
-Then Start keepalived:​ +
-  service keepalived start +
-   +
-Also need to setup autorestart during the system reboot: +
-  systemctl enable keepalived +
-or +
-  chkconfig keepalived on +
-  +
-The keepalive configuration file is located at /​etc/​keepalived/​keepalived.conf,​ but you should not edit this configuration file directly, instead, you should use ADC Web Admin GUI -> HA config to add/config VIP.  The manually added VIP to keepalived config won't be picked up by ADC HA config. The VIP configure under ADC HA tab is just a GUI to update the keepalived config file. So you should just use the WebAdmin GUI to manage VIP if they want to see it in the status. We will explain on later steps on how to add VIP in GUI. +
- +
- +
-===== Configure HA on LiteSpeed Web ADC =====  +
- +
-==== Node 1 ==== +
- login to node 1 ADC Web Admin Console: sample configuration +
- +
-  Server Address 10.10.30.96:​11122 +
-  Replication Cluster 10.10.30.96:​11122,​10.10.30.97:​11122 +
-  Heart Beat Interval (secs) 10 +
-  Heart Beat Timeout (secs) 30 +
-  Is Gzip Stream  ​      Yes +
-  Enable incremental sync      Yes +
-  Is File Cache Enabled  ​      Yes +
-  File Cache Server Address ​ 10.10.30.96:​1447 +
-   +
-{{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha1.png?​800 |}} +
- +
- then "​Add"​ HA interface:​ +
- +
-{{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha1-add-internface.png?​800 |}}   +
- +
-After VIP has been added through GUI, the configuration will be added to keepalived configuration and you will see keepalive configuration like: +
- +
-vi /​etc/​keepalived/​keepalived.conf +
- +
-  ###### start of VI_5 ###### +
-  vrrp_instance VI_5 { +
-    state BACKUP +
-    interface ens160 +
-    lvs_sync_daemon_inteface ens160 +
-    garp_master_delay 2 +
-    virtual_router_id 110 +
-    priority 170 +
-    advert_int 1 +
-    authentication { +
-        auth_type PASS +
-        auth_pass test123 +
-    } +
-    virtual_ipaddress { +
-        10.10.31.31 +
-    } +
-  } +
-  ###### end of VI_5 ###### +
- +
-==== Node 2 ==== +
- +
- login to node 1 ADC Web Admin Console: sample configuration +
- +
-  Server Address 10.10.30.97:​11122 +
-  Replication Cluster 10.10.30.96:​11122,​10.10.30.97:​11122 +
-  Heart Beat Interval (secs) 10 +
-  Heart Beat Timeout (secs) 30 +
-  Is Gzip Stream  ​      Yes +
-  Enable incremental sync      Yes +
-  Is File Cache Enabled  ​      Yes +
-  File Cache Server Address ​ 10.10.30.97:​1447 +
-   +
-{{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha1.png?​800 |}}   +
- +
-then add HA interface:​ +
-{{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha2-add-internface.png?​800 |}} +
- +
-After VIP has been added through GUI, the configuration will be added to keepalived configuration and you will see keepalive configuration like: +
- +
-###### start of VI_5 ###### +
-  vrrp_instance VI_5 { +
-    state BACKUP +
-    interface ens160 +
-    lvs_sync_daemon_inteface ens160 +
-    garp_master_delay 2 +
-    virtual_router_id 110 +
-    priority 150 +
-    advert_int 1 +
-    authentication { +
-        auth_type PASS +
-        auth_pass test123 +
-    } +
-    virtual_ipaddress { +
-        10.10.31.31 +
-    } +
-  } +
-  ###### end of VI_5 ###### +
- +
-Note:  +
-  - node1 virtual_router_id should be the same as node2;  +
-  - **"​state MASTER/​BACKUP"​** doesn'​t really matter, since Higher priority one will be MASTER. +
- +
-===== Test IP failver ===== +
-For IP failover, it is completely managed by keepalived, ADC just add a configuration management interface. So you should test IP failover: +
- +
-1. check the master node, which currently is node 1, 10.10.30.96 +
- +
- ​root@ha1-ubuntu:​~# ip a +
-  1: lo: <​LOOPBACK,​UP,​LOWER_UP>​ mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 +
-    link/​loopback 00:​00:​00:​00:​00:​00 brd 00:​00:​00:​00:​00:​00 +
-    inet 127.0.0.1/8 scope host lo +
-       ​valid_lft forever preferred_lft forever +
-    inet6 ::1/128 scope host  +
-       ​valid_lft forever preferred_lft forever +
-  2: ens160: <​BROADCAST,​MULTICAST,​UP,​LOWER_UP>​ mtu 1500 qdisc fq_codel state UP group default qlen 1000 +
-    link/ether 00:​0c:​29:​c4:​09:​80 brd ff:​ff:​ff:​ff:​ff:​ff +
-    inet 10.10.30.96/​16 brd 10.10.255.255 scope global ens160 +
-       ​valid_lft forever preferred_lft forever +
-    inet 10.10.31.31/​32 scope global ens160 +
-       ​valid_lft forever preferred_lft forever +
-    inet6 fe80::​20c:​29ff:​fec4:​980/​64 scope link  +
-       ​valid_lft forever preferred_lft forever +
- +
-You can see the VIP 10.10.31.31. +
- +
-2.  Then test backup node, node 2, 10.10.30.96:​ +
- +
-  root@ha2-ubuntu:​~# ip a +
-  1: lo: <​LOOPBACK,​UP,​LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 +
-    link/​loopback 00:​00:​00:​00:​00:​00 brd 00:​00:​00:​00:​00:​00 +
-    inet 127.0.0.1/8 scope host lo +
-       ​valid_lft forever preferred_lft forever +
-    inet6 ::1/128 scope host  +
-       ​valid_lft forever preferred_lft forever +
-  2: ens160: <​BROADCAST,​MULTICAST,​UP,​LOWER_UP>​ mtu 1500 qdisc fq_codel state UP group default qlen 1000 +
-    link/ether 00:​0c:​29:​95:​67:​6d brd ff:​ff:​ff:​ff:​ff:​ff +
-    inet 10.10.30.97/​16 brd 10.10.255.255 scope global ens160 +
-       ​valid_lft forever preferred_lft forever +
-    inet6 fe80::​20c:​29ff:​fe95:​676d/​64 scope link  +
-       ​valid_lft forever preferred_lft forever +
- +
-You don't see VIP on node 2 when VIP is active on node 1, which is correct. +
- +
-3. Shutdown the master node 1, the VIP 10.10.31.31 should be migrated to the backup server node 2, you can check:  +
-  root@ha2-ubuntu:​~#​ ip a                                                                                                                                                 +
-  1: lo: <​LOOPBACK,​UP,​LOWER_UP>​ mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000                                                                             +
-    link/​loopback 00:​00:​00:​00:​00:​00 brd 00:​00:​00:​00:​00:​00 ​                                                                                                              +
-    inet 127.0.0.1/8 scope host lo                                                                                                                                      +
-       ​valid_lft forever preferred_lft forever +
-    inet6 ::1/128 scope host  +
-       ​valid_lft forever preferred_lft forever +
-  2: ens160: <​BROADCAST,​MULTICAST,​UP,​LOWER_UP>​ mtu 1500 qdisc fq_codel state UP group default qlen 1000 +
-    link/ether 00:​0c:​29:​95:​67:​6d brd ff:​ff:​ff:​ff:​ff:​ff +
-    inet 10.10.30.97/​16 brd 10.10.255.255 scope global ens160 +
-       ​valid_lft forever preferred_lft forever +
-    inet 10.10.31.31/​32 scope global ens160 +
-       ​valid_lft forever preferred_lft forever +
-    inet6 fe80::​20c:​29ff:​fe95:​676d/​64 scope link  +
-       ​valid_lft forever preferred_lft forever +
- +
-You can see VIP 10.10.31.31 is assigned to node 2 now. +
- +
- +
-IP failover is completely handled by keepalived, it only happens when one server is completely down, the other server will take over the IP. Shutingdown LS ADC won't trigger an IP failover.  +
- +
-For more fancy failover method, you may want to try BGP, like what cloudflare does, but it is not controlled by ADC.  +
-https://blog.cloudflare.com/cloudflares-architecture-eliminating-single-p/ +
- +
- +
-===== Check HA Status for both nodes ===== +
- +
- HA Status page will look like the following when running: +
- +
-On Node 1: +
-{{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha1-status.png?​800 |}} +
- +
-On Node 2: +
-{{ :​litespeed_wiki:​lslb:​adc-ha-configuration-ha2-status.png?​800 |}} +
- +
-===== Replication our of sync? What required? ===== +
- +
-Sometime you may see replication out of sync. +
- +
-You will need to make sure node 1 and node 2 are configurated the same way. If they are configurated different way, you can not expect HA/Replication working.  +
- +
- +
-===== Verify your listener,​virtualhost,​ ClusterHTTP setup correctly ===== +
- +
-Assuming you have configured the listener, virtual Host and backend clusterHTTP on both Node 1 and node 2 seperately. +
- +
-Listener:  +
-With IP failover, we recommend listener configuration listening on *:<​port>,​ instead of individual <​IP>:<​port>​. +
- +
-{{ :​litespeed_wiki:​lslb:​adc-ha-configuration-listener.png?​800 |}} +
- +
-Virtual Host: +
-{{ :​litespeed_wiki:​lslb:​adc-ha-configuration-vhost.png?​800 |}} +
- +
-{{ :​litespeed_wiki:​lslb:​adc-ha-configuration-vhost2.png?​800 |}} +
- +
- +
-ClusterHTTP setup: +
-{{ :​litespeed_wiki:​lslb:​adc-ha-configuration-clusterhttp.png?​800 |}} +
- +
-{{ :​litespeed_wiki:​lslb:​adc-ha-configuration-clusterhttp-2.png?​800 |}} +
- +
-{{ :​litespeed_wiki:​lslb:​adc-ha-configuration-clusterhttp-3.png?​800 |}} +
- +
-{{ :​litespeed_wiki:​lslb:​adc-ha-configuration-clusterhttp-4.png?​800 |}} +
- +
-===== Testing VIP ===== +
- +
-Try access 10.10.31.31 (VIP) from the browser, you will see the backend server page. Disable one node, you can still see the webpage. Check ADC HA status, live node will become Master when the other one down. +
- +
-===== Troubleshooting ===== +
-==== Whe make configuration changes, it may need a full stop/start ==== +
- +
-When making changes to the configuration,​ such as changing the listener from <​IP>:​443 to "​*:​443",​ that requires a full stop/​start. +
- +
-==== HA configures are inconsistent between boxes  ==== +
-When you see similar error to the following:​ +
-2018-08-03 16:​24:​20.099467 [WARN] [REPL] HA configures are inconsistent between boxes, full replicaton can't proceed +
-2018-08-03 16:​24:​20.099520 [ERROR] [REPL] peer HA replication config is inconsistent,​ it must be fixed! +
- +
-It is because the configuration is out of sync between two LS ADC instances. replication only works if two ADC are serving the exact same sites. ​ You need to keep ADC configuration in sync. If it is out of sync temporarily,​ it will break the synchronization. Once config synced, ADC will restore replication synchronization.  +
- +
  • Admin
  • Last modified: 2018/08/03 18:28
  • by Jackson Zhang