Simple Automatic Failover for Asterisk and FreePBX

asterisk_logoDid you ever wanted an automatic failover solution for Asterisk? With just a few seconds downtime? Here’s how you do it.

This setup consists of the following steps:

  1. Download and instal AsteriskNow image (a bundle with Asterisk and FreePBX ready to deploy)
  2. Download and install keepalived daemon to get IP switchover
  3. Set up MySQL Master – Master replication
  4. Enjoy peace of mind 🙂

I assume your setup will be AsteriskNow based (which is Red Hat / CentOS based) but it should be painlessly applied to other Linux distributions. Also, reserve 3 or more IP addresses, one for each Asterisk instance and the third one for the cluster. Finally, the commands / edits will be done as root. This is NOT a best practice, but I want to show an easy setup and I won’t assume your user is configured as sudoer.

DOWNLOAD AND INSTALL ASTERISKNOW

We’ll go with AsteriskNow as it’s a “ready to go” Asterisk + FreePBX bundle which can be downloaded at the Asterisk’s official site: Download AsteriskNow

Burn and install / Boot from the iso and follow the installation procedure. In the installation, you’ll have to specify a fixed IP address (we’ll use it later for mysql replication), but remember not to use the one the one for the cluster. The third (cluster) IP must be unused.

Now, go on and install the two AsteriskNow instances.

SETTING UP FAILOVER WITH KEEPALIVED

For this step, we’ll use keepalived. It’s available in the repos, but in case you are on another setup or you want some more information, you can read more about keepalived here.

Now, login into your primary Asterisk instance and install and configure keepalived:

 

# yum install keepalived

# cd /etc/keepalived/

# vi keepalived.conf

Change the bold marked parameters (higher priority means it’ll be the primary server):

vrrp_instance VI_1 {

interface eth0

state MASTER

virtual_router_id 51

priority 100

authentication {

auth_type PASS

auth_pass Add-Your-Password-Here

}

virtual_ipaddress {

192.168.15.23

172.12.1.45

}

}

Hints:

  • Priority: Higher priority means the server will be used if alive. It must be different in the primary and the secondary server

  • auth_pass: It only takes into account 8 characters

  • virtual_ipaddress: Those will be the cluster addresses and must be the same for primary and secondary server. It can be one or more addresses which can be on different IP ranges or even on another network interface. There’s quite some info for more advanced setups (also, there’s always #man keepalived.conf). We are going for a simple working setup

A SIMPLE WORKING KEEPALIVED CONFIGURATION FOR PRIMARY SERVER

Remember to configure you SMTP server and change bold parameters to adapt to your particular setup.

! Configuration File for keepalived

global_defs {

notification_email {

voip@example.com

failover@example.com

sysadmin@example.com

}

notification_email_from server.asterisk@example.com

smtp_server xxx.yyy.zzz.aaa

smtp_connect_timeout 30

router_id LVS_DEVEL

}

vrrp_instance VI_1 {

state MASTER

interface eth0

virtual_router_id 51

priority 101

advert_int 1

authentication {

auth_type PASS

auth_pass hunter2

}

virtual_ipaddress {

192.168.143.200

}

}

auth_pass must be the same in both servers, but please, change it. You don’t want to be THAT guy with hunter2 as a password.

A SIMPLE WORKING KEEPALIVED CONFIGURATION FOR SECONDARY SERVER

Remember to configure you SMTP server and change bold parameters to adapt to your particular setup.

! Configuration File for keepalived

global_defs {

notification_email {

voip@example.com

failover@example.com

sysadmin@example.com

}

notification_email_from server.asterisk@example.com

smtp_server xxx.yyy.zzz.aaa

smtp_connect_timeout 30

router_id LVS_DEVEL

}

vrrp_instance VI_1 {

state MASTER

interface eth0

virtual_router_id 51

priority 100

advert_int 1

authentication {

auth_type PASS

auth_pass hunter2

}

virtual_ipaddress {

192.168.143.200

}

}

TESTING FAILOVER WITH KEEPALIVED

Starting with the secondary server, you’ll have to start the service manually and once it’s tested, configure it to start on boot.

Let’s start the service to check for startup errors

# service keepalived start

Optionally, to check virtual IP and switchover info:

# tail -f /var/log/messages

From a third PC, launch a ping to your virtual IP to see if it’s alive (it should be) and leave it running.

Now start the service on the primary server and optionally take a look at switchover information:

# service keepalived start

# tail -f /var/log/messages

The running ping might have lost a couple of packets during the switchover, but by now, the primary server should have the cluster IP assigned.

Finally, configure keepalive to automatically run on boot on both servers

# chkconfig keepalived on

CONFIGURING MYSQL MASTER MASTER REPLICATION

Now that we have an automatic failover mechanism in place, it’s time to set up MySQL redundancy.

MODIFY MYSQL CONFIGURATION (MY.CNF)

The first step will be to modify MySQL configuration following those steps:

  • Enable remote conections

  • Identify the server. As a default when it’s not otherwise said, it takes a value of 1 so it’s better to take another id

  • Location of binary log

  • Explicitly add which databases we want to replicate

  • Explicitly add which databases we don’t want to replicate. This step shouldn’t be necessary but I’ve had some passwords related problems because mysql system database got replicated.

The configuration file is llocated at /etc/my.cnf and you should have something similar to the following:

Initial configuration:

[mysqld]

bind-address = 127.0.0.1

general_log = 0

general_log_file = /var/log/mysql/mysql.log

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

user=mysql

# Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0

[mysqld_safe]

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

Final working setup for primary server:

[mysqld]

#bind-address = 127.0.0.1

server-id = 2

log_bin = /var/log/mysql/mysql-bin.log

binlog_do_db = asterisk

binlog_do_db = asteriskcdrdb

binlog_ignore_db = mysql

general_log = 0

general_log_file = /var/log/mysql/mysql.log

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

user=mysql

# Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0

[mysqld_safe]

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

Restart mysql service:

# service mysqld restart

And we’re done with the primary server. For the secondary server, repeat the steps remembering to change server-id to 3 or any other value.

Final working setup for secondary server:

[mysqld]

#bind-address = 127.0.0.1

server-id = 3

log_bin = /var/log/mysql/mysql-bin.log

binlog_do_db = asterisk

binlog_do_db = asteriskcdrdb

binlog_ignore_db = mysql

general_log = 0

general_log_file = /var/log/mysql/mysql.log

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

user=mysql

# Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0

[mysqld_safe]

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

Restart mysql service:

# service mysqld restart

ADDING AND CONFIGURING REPLICATION USER

Log into primary server and add the user which will do the replication:

# mysql -u root -p

mysql> create user 'replicator'@'%' identified by 'ChangeMePlease!!!!!';

Query OK, 0 rows affected (0.00 sec)

Add permission for replication to the user:

mysql> grant replication slave on *.* to 'replicator'@'%';

Query OK, 0 rows affected (0.00 sec)

As far as I can tell, it’s not possible to grant permissions to specific databases for this to work. Now, let’s get information from which point on the primary server we want to replicate on the secondary server:


mysql> show master status;

FilePositionBinlog_Do_DBBinlog_Ignore_DB
mysql-bin.000001254163asterisk,asteriskcdrdbmysql
1 row in set (0.00 sec)

The important parts are File and Position. Take note of them as we’ll need them later.

Log into secondary server and add the user which will do the replication:

# mysql -u root -p

mysql> create user 'replicator'@'%' identified by 'ChangeMePlease!!!!!';

Query OK, 0 rows affected (0.00 sec)

Add permission for replication to the user:


mysql> grant replication slave on *.* to 'replicator'@'%';

Query OK, 0 rows affected (0.00 sec)

SETTING UP MYSQL MASTER MASTER REPLICATION

Now, on the secondary server, we’ll configure replication from primary to secondary server (remember to use server’s fixed IP address and not the virtual one):

mysql> slave stop;

Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> CHANGE MASTER TO MASTER_HOST = '192.168.143.201', MASTER_USER = 'replicator', MASTER_PASSWORD = 'ChangeMePlease!!!!!', MASTER_LOG_FILE = 'mysql-bin.000001', MASTER_LOG_POS = 254163;

Query OK, 0 rows affected (0.03 sec)

mysql> slave start;

Query OK, 0 rows affected (0.00 sec)

Still on the secondary server, we’ll get information for setting up secondary to primary replication:

mysql> show master status;

FilePositionBinlog_Do_DBBinlog_Ignore_DB
mysql-bin.000001383123asterisk,asteriskcdrdbmysql
1 row in set (0.00 sec)

Finally, set up replication on the primary server from secondary to primary (remember to use server's fixed IP address and not the virtual one):

mysql> slave stop;

Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> CHANGE MASTER TO MASTER_HOST = '192.168.143.202', MASTER_USER = 'replicator', MASTER_PASSWORD = 'ChangeMePlease!!!!!', MASTER_LOG_FILE = 'mysql-bin.000001', MASTER_LOG_POS = 383123;

Query OK, 0 rows affected (0.02 sec)

mysql> slave start;

Query OK, 0 rows affected (0.00 sec)

That’s it. Easy, isn’t it? Now you can go on and configure your installation.

FINAL THOUGHTS, NEXT STEPS AND HELPFUL LINKS

  • Depending on the client used, I’ve got under 5 seconds downtime.
  • Passwords, too open configuration …. Please change the passwords stated here. Also, the configuration can (and should) be thightened.
  • You can configure the whole installation in one server and the configuration will be synchronized at FreePBX level. You must apply changes on the secondary server for Asterisk to notice them (there are workarounds but see next point)
  • If you can afford it (if you need HA, it’s probable you can afford this), please consider purchasing FreePBX HA module. This way, they get paid so they can go on developing and everyone benefits on the long run (Disclaimer: I have no relation with FreePBX. I just think they’re developing an amazing piece of software)
  • Recordings, if you use them, they are (still) not synchronized on the filesystem. They get stored on the server which is running in the moment the call is made. You’ll have to set up a synchronization mechanism

Did you find the post useful? Would you invite me to a nice coffe or offer some help running the site ad-free?
1EpMM5szeNSSYxgDKBvksyCCizKHoAZmBs

LINKS I USED FOR SETTING EVERYTHING UP

keepalived

https://raymii.org/s/tutorials/Keepalived-Simple-IP-failover-on-Ubuntu.html

http://www.cyberciti.biz/faq/rhel-centos-fedora-keepalived-lvs-cluster-configuration/

http://tecadmin.net/ip-failover-setup-using-keepalived-on-centos-redhat-6/#

http://vmhacks.com/ip-address-activepassive-failover-setup-in-linux-centos-6-rhel-6-using-keepalived/

MySQL

http://dba.stackexchange.com/questions/72058/master-master-replication-with-3-nodes

https://www.digitalocean.com/community/tutorials/how-to-set-up-mysql-master-master-replication

Asterisk

http://www.asteriskdocs.org/en/2nd_Edition/asterisk-book-html-chunk/asterisk-APP-A-SECT-2.html

Nov 1, 2016

6 comments

  1. How did you install 2 instances of asterisk? Did you use virtualbox or something else?

    1. Hi

      in my particular case, for production each instance of Asterisk is running on a different VMware ESX server.

      Before that, for testing purposes, I installed it on a couple of virtual machines with Virtualbox.

      The only difference was that ESX was faster, but it was due to hardware (normal PCs vs Server hardware)

      1. Thanks for your reply! And how did you configured sip clients? I mean for example, i have two softphones (Zoiper) that have assigned two extensions. Both of them successfully register to the primary asterisk when it’s up. But when the primary will shutdown, these two extensions will not register to the secondary asterisk even if keepalived is configure properly accordingly upper steps.

        1. Hi again

          I don’t know this particular SIP client, but as it works with the primary server, it should work just fine. I think the problem might be in the config of the secondary server, but, could you do a little test?

          Let’s say that for your particular installation:

          192.168.1.1 is the virtual IP (the one managed by keepalived)
          192.168.1.2 is the primary server IP
          192.168.1.3 is the secondary server IP

          Let’s forget for a moment about virtual IP and keepalived and assume both servers as standalone with the same config. Can you register both phones to 192.168.1.2? (I assume yes from the comment) Modify the config from both clients so now they will only register to 192.168.1.3 Does this work?

          If the second step doesn’t work, then the problem should be in the config of the secondary server.

          Login to the secondary server and look for the apply config button on the top of the screen. If it’s not present, make a dummy change (add a dummy extension and delete it, some insignificant change, …) so this button appears. Apply config and try to register the phones again to 192.168.1.3

          When both phones are able to register to both servers independently, it’s time to take a look at keepalived. Modify the config of both clients so they now register to 192.168.1.1

          Make it so 192.168.1.1 and 192.168.1.2 are in the same server. If the clients are not able to register, then it seems that Asterisk is not listening to 192.168.1.1 When the phones register to this interface, make this server fail so now 192.168.1.1 and 192.168.1.3 are in the same server.
          If the phones are not able to register, the problem seems to be in Asterisk not listening to 192.168.1.1

          Also, take a look in the log of the secondary server (I don’t recall the exact menu but there’s a link from a FreePBX menu) as it migh give you hints to where the problem is.

          1. yes, finally it seems to be fully functional,.
            I’ve registred two extensions, zoiper on my mobile phones automatically registred to backup asterisk, when primary is down, interesting is that the zoiper on my laptop cant, i do not know, where is problem. However, it’s seems functional. Thank you very much!

          2. I’m glad I was helpful. I think I can give you one more hint so it will also work on your PC.

            Before provoking a failovet, can your PC register on the secondary 192.168.1.3) server? If it can’t, I would recheck the sip configuration on the server.

            But of it’s able to register, then I would think of a network problem. When the failover happens, the new association of IP address and MAC address can take some time to propagate (it depends on network hardware, and OS and version).

            Can you check asterisk’s log whether the registration petition arrives. There should appear the extension number.

            If it doesn’t appear, can you check the contents of the ARP table, before and after the failover?

            The MAC associated to 192.168.1.1 (cluster IP) should change before and after the failover. If it remains the same, try to flush the cache. If this fixes the problem, then it would mean it’s almost for sure related to your network configuration.

Leave a Reply

Your email address will not be published. Required fields are marked *