{"id":3241,"date":"2013-02-13T16:00:00","date_gmt":"2013-02-13T16:00:00","guid":{"rendered":"http:\/\/localhost:8105\/?p=3241"},"modified":"2021-12-04T23:58:25","modified_gmt":"2021-12-04T23:58:25","slug":"fixing-problem-with-fencing-with-ipmi-l","status":"publish","type":"post","link":"https:\/\/blog.shahada.abubakar.net\/?p=3241","title":{"rendered":"Fixing Problem with Fencing with IPMI-L"},"content":{"rendered":"
\n

I noticed some errors with fencing on the cluster at client’s site.<\/p>\n

From the \/var\/log\/messages:<\/p>\n

Feb 14 09:59:52 pdb01 fenced[4819]: fencing node \"pdb-node2\"\nFeb 14 09:59:52 pdb01 fenced[4819]: agent \"fence_ipmilan\" reports: Rebooting machine @ IPMI:192.168.2.140...Failed<\/pre>\n
You can manually fence a node using redhat cluster tool “fence_node”:<\/p>\n
[root@pdb01~]# fence_node pdb-node2\nagent \"fence_ipmilan\" reports: Rebooting machine @ IPMI:192.168.2.140...Failed<\/pre>\n
You can also call the fence_ipmilan program manually, with many “-v” to get additional debug mesages:<\/p>\n
$ fence_ipmilan -P imb -vvv -a 192.168.2.140 -l clusterpower -p XXX -o reboot\nRebooting machine @ IPMI:192.168.2.140...Spawning: '\/usr\/bin\/ipmitool -I lanplus -H '192.168.2.140' -U 'clusterpower' -P 'p0w3r0ff' -v -v -v chassis power status'...\nSpawned: '\/usr\/bin\/ipmitool -I lanplus -H '192.168.2.140' -U 'clusterpower' -P 'p0w3r0ff' -v -v -v chassis power status' - PID 22333\nLooking for:\n\u00a0 \u00a0 'Password:', val = 1\n\u00a0 \u00a0 'Unable to establish LAN', val = 11\n\u00a0 \u00a0 'IPMI mutex', val = 14\n\u00a0 \u00a0 'Unsupported cipher suite ID', val = 2048\n\u00a0 \u00a0 'read_rakp2_message: no support for', val = 2048\n\u00a0 \u00a0 'Chassis Power is off', val = 4096\n\u00a0 \u00a0 'Chassis Power is on', val = 8192\nExpectToken returned -1\nExpectToken failed. \u00a0Info returned:\n>>>>>\nIPMI LAN host 192.168.2.140 port 623^M\n^M\n>> Sending IPMI command payload^M\n>> \u00a0 \u00a0netfn \u00a0 : 0x06^M\n>> \u00a0 \u00a0command : 0x38^M\n>> \u00a0 \u00a0data \u00a0 \u00a0: 0x8e 0x04\n\nBUILDING A v1.5 COMMAND^M\n>> IPMI Request Session Header^M\n>> \u00a0 Authtype \u00a0 : NONE^M\n>> \u00a0 Sequence \u00a0 : 0x00000000^M\n>> \u00a0 Session ID : 0x00000000^M\n>> IPMI Request Message Header^M\n>> \u00a0 Rs Addr \u00a0 \u00a0: 20^M\n>> \u00a0 NetFn \u00a0 \u00a0 \u00a0: 06^M\n>> \u00a0 Rs LUN \u00a0 \u00a0 : 0^M\n>> \u00a0 Rq Addr \u00a0 \u00a0: 81^M\n>> \u00a0 Rq Seq \u00a0 \u00a0 : 00^M\n>> \u00a0 Rq Lun \u00a0 \u00a0 : 0^M\n>> \u00a0 Command \u00a0 \u00a0: 38^M\n<< IPMI Response Session Header^M\n<< \u00a0 Authtype \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: NONE^M\n<< \u00a0 Payload type \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: IPMI (0)^M\n<< \u00a0 Session ID \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: 0x00000000^M\n<< \u00a0 Sequence \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: 0x00000000^M\n<< \u00a0 IPMI Msg\/Payload Length : 16^M\n<< IPMI Response Message Header^M\n<< \u00a0 Rq Addr \u00a0 \u00a0: 81^M\n<< \u00a0 NetFn \u00a0 \u00a0 \u00a0: 07^M\n<< \u00a0 Rq LUN \u00a0 \u00a0 : 0^M\n<< \u00a0 Rs Addr \u00a0 \u00a0: 20^M\n<< \u00a0 Rq Seq \u00a0 \u00a0 : 00^M\n<< \u00a0 Rs Lun \u00a0 \u00a0 : 0^M\n<< \u00a0 Command \u00a0 \u00a0: 38^M\n<< \u00a0 Compl Code : 0x00^M\n>> SENDING AN OPEN SESSION REQUEST\n^M\n>> Console generated random number (16 bytes)^M\n\u00a087 44 95 28 f1 1d 00 ef e8 9f e3 74 0e 2a cb a6^M\n>> SENDING A RAKP 1 MESSAGE\n^M\nbmc_rand (16 bytes)^M\n\u00a02c f7 2c 41 c6 91 c6 ea ae 79 ae 93 78 43 78 73^M\n>> rakp2 mac input buffer (70 bytes)^M\n\u00a0a4 a3 a2 a0 03 67 00 03 87 44 95 28 f1 1d 00 ef^M\n\u00a0e8 9f e3 74 0e 2a cb a6 2c f7 2c 41 c6 91 c6 ea^M\n\u00a0ae 79 ae 93 78 43 78 73 ed ac 88 08 9d 28 11 e0^M\n\u00a096 88 e4 1f 13 bc ca f0 14 0c 63 6c 75 73 74 65^M\n\u00a072 70 6f 77 65 72^M\n>> rakp2 mac key (20 bytes)^M\n\u00a070 30 77 33 72 30 66 66 00 00 00 00 00 00 00 00^M\n\u00a000 00 00 00^M\n>> rakp2 mac as computed by the remote console (20 bytes)^M\n\u00a0b0 dc 36 53 e7 cd 31 3c f6 e5 e2 ef 11 8e 3f 1b^M\n\u00a0ce 43 04 4a^M\n>> rakp3 mac input buffer (34 bytes)^M\n\u00a02c f7 2c 41 c6 91 c6 ea ae 79 ae 93 78 43 78 73^M\n\u00a0a4 a3 a2 a0 14 0c 63 6c 75 73 74 65 72 70 6f 77^M\n\u00a065 72^M\n>> rakp3 mac key (20 bytes)^M\n\u00a070 30 77 33 72 30 66 66 00 00 00 00 00 00 00 00^M\n\u00a000 00 00 00^M\ngenerated rakp3 mac (20 bytes)^M\n\u00a074 b5 e4 9e fa 14 00 0d 38 4e b6 88 87 4f ad 00^M\n\u00a09b 0a 99 c6^M\nsession integrity key input (46 bytes)^M\n\u00a087 44 95 28 f1 1d 00 ef e8 9f e3 74 0e 2a cb a6^M\n\u00a02c f7 2c 41 c6 91 c6 ea ae 79 ae 93 78 43 78 73^M\n\u00a014 0c 63 6c 75 73 74 65 72 70 6f 77 65 72^M\nGenerated session integrity key (20 bytes)^M\n\u00a06c b1 0b 77 c5 5a 12 87 5c 03 48 03 13 b5 bf a7^M\n\u00a0ad 15 0e 9a^M\nGenerated K1 (20 bytes)^M\n\u00a0b7 8d af 9a 90 9f 66 a3 6b 95 2d 84 82 35 37 0e^M\n\u00a024 75 22 f1^M\nGenerated K2 (20 bytes)^M\n\u00a032 47 ca fc 7f 01 4e 6e c7 26 02 ed 7a f2 4b 53^M\n\u00a0d6 9c 96 b6^M\n>> SENDING A RAKP 3 MESSAGE\n^M\n>> rakp4 mac input buffer (36 bytes)^M\n\u00a087 44 95 28 f1 1d 00 ef e8 9f e3 74 0e 2a cb a6^M\n\u00a003 67 00 03 ed ac 88 08 9d 28 11 e0 96 88 e4 1f^M\n\u00a013 bc ca f0^M\n>> rakp4 mac key (sik) (20 bytes)^M\n\u00a06c b1 0b 77 c5 5a 12 87 5c 03 48 03 13 b5 bf a7^M\n\u00a0ad 15 0e 9a^M\n>> rakp4 mac as computed by the BMC (20 bytes)^M\n\u00a052 17 2e f7 50 7a 65 57 9a ef da f3 78 43 78 73^M\n\u00a0ed ac 88 08^M\n>> rakp4 mac as computed by the remote console (20 bytes)^M\n\u00a052 17 2e f7 50 7a 65 57 9a ef da f3 3a 4c 3f e5^M\n\u00a09e e3 4b d7^M\nIPMIv2 \/ RMCP+ SESSION OPENED SUCCESSFULLY\n^M\n^M\n>> Sending IPMI command payload^M\n>> \u00a0 \u00a0netfn \u00a0 : 0x06^M\n>> \u00a0 \u00a0command : 0x3b^M\n>> \u00a0 \u00a0data \u00a0 \u00a0: 0x04\n\nBUILDING A v2 COMMAND^M\n>> Initialization vector (16 bytes)^M\n\u00a036 67 1a 57 a9 22 63 9e 06 d6 30 54 71 ce a7 80^M\nauthcode input (48 bytes)^M\n\u00a006 c0 03 67 00 03 03 00 00 00 20 00 36 67 1a 57^M\n\u00a0a9 22 63 9e 06 d6 30 54 71 ce a7 80 48 e8 ed 98^M\n\u00a0b7 a7 be 06 83 04 4f f4 9a 09 e3 7f ff ff 02 07^M\nauthcode output (12 bytes)^M\n\u00a094 dd 65 59 04 6b 90 fa ab d0 99 ba^M\n<< IPMI Response Session Header^M\n<< \u00a0 Authtype \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: Unknown (0x06)^M\n<< \u00a0 Payload type \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: IPMI (0)^M\n<< \u00a0 Session ID \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: 0xa0a2a3a4^M\n<< \u00a0 Sequence \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: 0x00000001^M\n<< \u00a0 IPMI Msg\/Payload Length : 32^M\n<< IPMI Response Message Header^M\n<< \u00a0 Rq Addr \u00a0 \u00a0: 81^M\n<< \u00a0 NetFn \u00a0 \u00a0 \u00a0: 07^M\n<< \u00a0 Rq LUN \u00a0 \u00a0 : 0^M\n<< \u00a0 Rs Addr \u00a0 \u00a0: 20^M\n<< \u00a0 Rq Seq \u00a0 \u00a0 : 01^M\n<< \u00a0 Rs Lun \u00a0 \u00a0 : 0^M\n<< \u00a0 Command \u00a0 \u00a0: 3b^M\n<< \u00a0 Compl Code : 0x81^M\nSet Session Privilege Level to ADMINISTRATOR failed: Unknown (0x81)^M\nError: Unable to establish IPMI v2 \/ RMCP+ session^M\nUnable to get Chassis Power Status^M\n<<OPEN SESSION RESPONSE\n<< \u00a0Message tag \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: 0x00\n<< \u00a0RMCP+ status \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 : no errors\n<< \u00a0Maximum privilege level \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: admin\n<< \u00a0Console Session ID \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 : 0xa0a2a3a4\n<< \u00a0BMC Session ID \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 : 0x03006703\n<< \u00a0Negotiated authenticatin algorithm : hmac_sha1\n<< \u00a0Negotiated integrity algorithm \u00a0 \u00a0 : hmac_sha1_96\n<< \u00a0Negotiated encryption algorithm \u00a0 \u00a0: aes_cbc_128\n\n<<RAKP 2 MESSAGE\n<< \u00a0Message tag \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 : 0x00\n<< \u00a0RMCP+ status \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: no errors\n<< \u00a0Console Session ID \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: 0xa0a2a3a4\n<< \u00a0BMC random number \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 : 0x2cf72c41c691c6eaae79ae9378437873\n<< \u00a0BMC GUID \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: 0xedac88089d2811e09688e41f13bccaf0\n<< \u00a0Key exchange auth code [sha1] : 0xb0dc3653e7cd313cf6e5e2ef118e3f1bce43044a\n\n<<RAKP 4 MESSAGE\n<< \u00a0Message tag \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 : 0x00\n<< \u00a0RMCP+ status \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: no errors\n<< \u00a0Console Session ID \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0: 0xa0a2a3a4\n<< \u00a0Key exchange auth code [sha1] : 0x52172ef7507a65579aefdaf3\n\n<<<<<\nError = 2 (No such file or directory)\nReaping pid 22333\nFailed\n\nThe \"No such file or directory\" is a red herring, the real error is \"Set Session Privilege Level to ADMINISTRATOR failed\" earlier on.<\/pre>\n
I tried with a different password and it didn’t get as far as the command above. So the ID and password pair were correct. The ID also works if I manually telnet to the management interface and reboot the server:<\/p>\n
Welcome to the server management network terminal!\nlogin : clusterpower\nPassword:\u00a0\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0Legacy CLI Application\u00a0\n\nsystem>reset\u00a0\nok<\/pre>\n
Checked the documentation and there are two user IDs in the Management Interface, USERID and clusterpower. Tried “fence_ipmilan” again with user USERID and and it rebooted the node 2.<\/p>\n
So … likely that clusterpower doesn’t have enough\u00a0privilege\u00a0to reboot the server.<\/p>\n
Browsed over to the Management Interface and looked up user “clusterpower”, it has “Remote Server Power\/Restart Access” but not “Remote Console Access”. Ticked that, and now<\/p>\n
[root@pdb01~]# fence_node pdb-node2<\/pre>\n
Reboots the other server.<\/p>\n
+<\/span>Namran Hussin<\/a><\/span><\/p>\n
\n

\n<\/a><\/p>\n
\n
Comments (2):<\/p>\n