Hi all,

I have an issue where I have configured heartbeat to run on a 2 node httpd cluster, heartbeat seems to be running when i check logs and I see that node1 comes up on web page, but when i shutdown heartbeat so that node2 would failover, it does not work. This is the log i see on node1... 

tailf /var/log/ha-log
Jan 28 09:48:04 node1 heartbeat: [2420]: info: Configuration validated. Starting heartbeat 3.0.4
Jan 28 09:48:04 node1 heartbeat: [2421]: info: heartbeat: version 3.0.4
Jan 28 09:48:04 node1 heartbeat: [2421]: info: Heartbeat generation: 1422435302
Jan 28 09:48:04 node1 heartbeat: [2421]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0
Jan 28 09:48:04 node1 heartbeat: [2421]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0 - Status: 1
Jan 28 09:48:04 node1 heartbeat: [2421]: info: G_main_add_TriggerHandler: Added signal manual handler
Jan 28 09:48:04 node1 heartbeat: [2421]: info: G_main_add_TriggerHandler: Added signal manual handler
Jan 28 09:48:04 node1 heartbeat: [2421]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Jan 28 09:48:04 node1 heartbeat: [2421]: info: Local status now set to: 'up'
Jan 28 09:48:04 node1 heartbeat: [2421]: info: Link node1:eth0 up.
Jan 28 09:50:05 node1 heartbeat: [2421]: WARN: node node2: is dead
Jan 28 09:50:05 node1 heartbeat: [2421]: info: Comm_now_up(): updating status to active
Jan 28 09:50:05 node1 heartbeat: [2421]: info: Local status now set to: 'active'
Jan 28 09:50:05 node1 heartbeat: [2421]: WARN: No STONITH device configured.
Jan 28 09:50:05 node1 heartbeat: [2421]: WARN: Shared disks are not protected.
Jan 28 09:50:05 node1 heartbeat: [2421]: info: Resources being acquired from node2.
harc(default)[2433]:    2015/01/28_09:50:05 info: Running /etc/ha.d//rc.d/status status
mach_down(default)[2469]:       2015/01/28_09:50:05 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down(default)[2469]:       2015/01/28_09:50:05 info: mach_down takeover complete for node node2.
Jan 28 09:50:05 node1 heartbeat: [2421]: info: mach_down takeover complete.
Jan 28 09:50:05 node1 heartbeat: [2421]: info: Initial resource acquisition complete (mach_down)
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_172.31.29.243)[2501]:  2015/01/28_09:50:05 INFO:  Resource is stopped
Jan 28 09:50:05 node1 heartbeat: [2434]: info: Local Resource acquisition completed.
harc(default)[2588]:    2015/01/28_09:50:06 info: Running /etc/ha.d//rc.d/ip-request-resp ip-request-resp
ip-request-resp(default)[2588]: 2015/01/28_09:50:06 received ip-request-resp 172.31.29.243 OK yes
ResourceManager(default)[2611]: 2015/01/28_09:50:06 info: Acquiring resource group: node1 172.31.29.243 httpd
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_172.31.29.243)[2639]:  2015/01/28_09:50:06 INFO:  Resource is stopped
ResourceManager(default)[2611]: 2015/01/28_09:50:06 info: Running /etc/ha.d/resource.d/IPaddr 172.31.29.243 start
IPaddr(IPaddr_172.31.29.243)[2737]:     2015/01/28_09:50:06 INFO: Adding inet address 172.31.29.243/20 with broadcast address 172.31.31.255 to device eth0
IPaddr(IPaddr_172.31.29.243)[2737]:     2015/01/28_09:50:06 INFO: Bringing device eth0 up
IPaddr(IPaddr_172.31.29.243)[2737]:     2015/01/28_09:50:06 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-172.31.29.243 eth0 172.31.29.243 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_172.31.29.243)[2723]:  2015/01/28_09:50:06 INFO:  Success
Jan 28 09:50:16 node1 heartbeat: [2421]: info: Local Resource acquisition completed. (none)
Jan 28 09:50:16 node1 heartbeat: [2421]: info: local resource transition completed.





node2 i see this

tailf /var/log/ha-log
Jan 28 09:27:22 node2 heartbeat: [1646]: info: Configuration validated. Starting heartbeat 3.0.4
Jan 28 09:27:22 node2 heartbeat: [1647]: info: heartbeat: version 3.0.4
Jan 28 09:27:22 node2 heartbeat: [1647]: info: Heartbeat generation: 1422435301
Jan 28 09:27:22 node2 heartbeat: [1647]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0
Jan 28 09:27:22 node2 heartbeat: [1647]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0 - Status: 1
Jan 28 09:27:22 node2 heartbeat: [1647]: info: G_main_add_TriggerHandler: Added signal manual handler
Jan 28 09:27:22 node2 heartbeat: [1647]: info: G_main_add_TriggerHandler: Added signal manual handler
Jan 28 09:27:22 node2 heartbeat: [1647]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Jan 28 09:27:22 node2 heartbeat: [1647]: info: Local status now set to: 'up'
Jan 28 09:27:22 node2 heartbeat: [1647]: info: Link node2:eth0 up.
Jan 28 09:29:23 node2 heartbeat: [1647]: WARN: node node1: is dead
Jan 28 09:29:23 node2 heartbeat: [1647]: info: Comm_now_up(): updating status to active
Jan 28 09:29:23 node2 heartbeat: [1647]: info: Local status now set to: 'active'
Jan 28 09:29:23 node2 heartbeat: [1647]: WARN: No STONITH device configured.
Jan 28 09:29:23 node2 heartbeat: [1647]: WARN: Shared disks are not protected.
Jan 28 09:29:23 node2 heartbeat: [1647]: info: Resources being acquired from node1.
Jan 28 09:29:23 node2 heartbeat: [1656]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys node2] to acquire.
harc(default)[1655]:    2015/01/28_09:29:23 info: Running /etc/ha.d//rc.d/status status
mach_down(default)[1685]:       2015/01/28_09:29:23 info: Taking over resource group 172.31.29.243
ResourceManager(default)[1712]: 2015/01/28_09:29:23 info: Acquiring resource group: node1 172.31.29.243 httpd
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_172.31.29.243)[1740]:  2015/01/28_09:29:23 INFO:  Resource is stopped
ResourceManager(default)[1712]: 2015/01/28_09:29:23 info: Running /etc/ha.d/resource.d/IPaddr 172.31.29.243 start
IPaddr(IPaddr_172.31.29.243)[1838]:     2015/01/28_09:29:23 INFO: Adding inet address 172.31.29.243/20 with broadcast address 172.31.31.255 to device eth0
IPaddr(IPaddr_172.31.29.243)[1838]:     2015/01/28_09:29:23 INFO: Bringing device eth0 up
IPaddr(IPaddr_172.31.29.243)[1838]:     2015/01/28_09:29:23 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-172.31.29.243 eth0 172.31.29.243 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_172.31.29.243)[1824]:  2015/01/28_09:29:23 INFO:  Success
mach_down(default)[1685]:       2015/01/28_09:29:23 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down(default)[1685]:       2015/01/28_09:29:23 info: mach_down takeover complete for node node1.
Jan 28 09:29:23 node2 heartbeat: [1647]: info: mach_down takeover complete.
Jan 28 09:29:23 node2 heartbeat: [1647]: info: Initial resource acquisition complete (mach_down)
Jan 28 09:29:33 node2 heartbeat: [1647]: info: Local Resource acquisition completed. (none)
Jan 28 09:29:33 node2 heartbeat: [1647]: info: local resource transition completed.
^Z
[1]+  Stopped                 tailf /var/log/ha-log
[root@ip-172-31-29-242 ~]# tailf /var/log/ha-log
IPaddr(IPaddr_172.31.29.243)[1838]:     2015/01/28_09:29:23 INFO: Adding inet address 172.31.29.243/20 with broadcast address 172.31.31.255 to device eth0
IPaddr(IPaddr_172.31.29.243)[1838]:     2015/01/28_09:29:23 INFO: Bringing device eth0 up
IPaddr(IPaddr_172.31.29.243)[1838]:     2015/01/28_09:29:23 INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-172.31.29.243 eth0 172.31.29.243 auto not_used not_used
/usr/lib/ocf/resource.d//heartbeat/IPaddr(IPaddr_172.31.29.243)[1824]:  2015/01/28_09:29:23 INFO:  Success
mach_down(default)[1685]:       2015/01/28_09:29:23 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down(default)[1685]:       2015/01/28_09:29:23 info: mach_down takeover complete for node node1.
Jan 28 09:29:23 node2 heartbeat: [1647]: info: mach_down takeover complete.
Jan 28 09:29:23 node2 heartbeat: [1647]: info: Initial resource acquisition complete (mach_down)
Jan 28 09:29:33 node2 heartbeat: [1647]: info: Local Resource acquisition completed. (none)
Jan 28 09:29:33 node2 heartbeat: [1647]: info: local resource transition completed.