We an NMS Application build in J2EE Technology using SNMP 4j Library.
We have one issue, from one of our clients that CERTAIN SNMP Packets are not send out of Solaris SERVER.
Problem Found:
The First SNMP Request of a Retransmitted SNMP Message from Solaris Server, is NOT Seen in any of the Solaris Application server Interface via snoop.
Meant, the First SNMP Packet of a Retransmitted request is not forwarded out of Solaris server.
The lowest layer, of Application that send SNMP Messages is SNMP 4J Library.
And according to the SNMP 4j Debug logs,
It says that the Initial SNMP Request had been sent. and no error thrown.
But I am unable to see this SNMP Request in SNOOP on the available interface in Solaris Server.
CAN YOU KINDLY LET ME KNOW HOW TO GO ABOUT THIS ISSUE?
THANKS>>>>
The following traces, refer the SNMP Request ID : 1811419093
SNMP First Retry (Initial Request)
---------------------------------
2011-12-22 04:52:48,722 DEBUG [JM-120]-[org.snmp4j.mp.MPv3] Adding cache entry: StateReference[msgID=305877635,pduHandle=PduHandle[1811419093],securityEngineID=80:0 0:02:7d:03:00:19:8f:2f:f7:b9,securityModel=org.snm p4j.security.USM@42c238e8,securityName=AMSUser,sec urityLevel=3,contextEngineID=80:00:02:7d:03:00:19: 8f:2f:f7:b9,contextName=nt]
2011-12-22 04:52:48,722 DEBUG [JM-120]-[org.snmp4j.Snmp] Running pending sync request with handle PduHandle[1811419093] and retry count left 2
2011-12-22 04:52:48,722 DEBUG [JM-120]-[org.snmp4j.transport.DefaultUdpTransportMapping] Sending message to 10.32.16.4/161 with length 345: 30:82:01:55:02:01:03:30:11:02:04:12:3b:52:83:02:03 :00:ff:ff:04:01:07:02:01:03:04:39:30:37:04:0b:80:0 0:02:7d:03:00:19:8f:2f:f7:b9:02:01:15:02:04:00:d4: fd:9e:04:07:41:4d:53:55:73:65:72:04:0c:63:71:7e:b5 :da:6f:2b:e7:3f:4b:a6:b1:04:08:00:00:00:15:bc:20:d 9:44:04:82:01:00:17:ee:44:2b:c1:26:b2:f6:b4:5e:cc: 0a:64:03:49:79:d6:f7:6d:0d:c8:ab:9c:d9:b8:3c:1e:1a :95:d7:9c:a8:8f:09:ef:1f:a4:2c:04:25:d0:0d:2d:0a:6 c:ea:c5:ba:ec:2e:5a:ba:76:91:91:e1:61:ef:fa:88:e2: aa:c0:de:99:d9:44:e9:0d:2d:78:3c:37:47:71:e0:c3:0d :a9:fa:c0:4f:ff:2c:33:b2:28:b5:fc:47:85:19:2f:f4:0 1:e9:81:4c:9a:88:2b:ee:b5:ce:91:67:4c:14:36:f3:32: 05:bc:bc:ad:24:58:63:34:12:a2:aa:e8:f4:b2:46:1f:13 :60:07:4b:9c:3d:36:3c:e5:3b:9c:9e:27:7d:1d:28:39:b 6:e8:cd:fe:7a:70:2a:8c:29:00:49:19:05:11:cb:19:c0: f0:c2:5e:43:aa:ce:b8:40:2f:32:a7:0b:30:ef:d3:55:53 :3b:03:8a:fd:27:51:cb:6d:5f:62:e6:22:28:f1:74:b4:7 a:6d:ed:90:45:ca:39:d2:e3:7b:4c:3b:d7:99:1a:0a:1d: 1c:69:20:3d:f0:56:b2:fb:c1:35:00:ce:16:96:64:00:f2 :7b:55:2d:d1:7a:45:23:e1:48:76:b8:eb:6c:7b:8a:7b:e 4:9c:1e:8c:6f:86:06:98:15:1e:75:0d
……
SNMP 4j says, that it had send the SNMP Message, and did not receive the response (For Initial SNMP Request)
….
So, After 30 Seconds ( Timeout before first Re-transmission =~ 30 Seconds )
-----------------------
SNMP 4j attempts for Second Retry
-------------------------------------------
2011-12-22 04:53:18,730 DEBUG [Timer-17]-[org.snmp4j.Snmp] Running pending sync request with handle PduHandle[1811419093] and retry count left 1
2011-12-22 04:53:18,730 DEBUG [Timer-17]-[org.snmp4j.transport.DefaultUdpTransportMapping] Sending message to 10.32.16.4/161 with length 345: 30:82:01:55:02:01:03:30:11:02:04:12:3b:52:9b:02:03 :00:ff:ff:04:01:07:02:01:03:04:39:30:37:04:0b:80:0 0:02:7d:03:00:19:8f:2f:f7:b9:02:01:15:02:04:00:d4: fd:bc:04:07:41:4d:53:55:73:65:72:04:0c:ea:2a:5f:de :12:0e:cf:bd:53:27:b4:b4:04:08:00:00:00:15:bc:20:d 9:5c:04:82:01:00:34:89:7a:5a:63:04:17:c0:2b:da:f5: 54:e1:3c:bc:bd:2f:87:f6:00:c1:b8:f8:6e:3d:8d:9c:e5 :4e:b8:a3:ec:0d:a3:67:c7:6c:3a:bf:a3:34:a6:40:18:2 b:c3:63:af:e9:fa:bf:8b:6a:04:7e:8e:cb:2f:94:d4:ce: cf:79:8d:4f:38:33:a9:c7:88:b1:42:11:e8:b7:34:36:a5 :3e:77:b7:a3:28:e0:f8:3b:92:64:28:9e:88:bd:08:4c:3 1:70:31:68:44:4f:34:8a:83:c3:41:de:d6:68:f1:15:a1: 0c:d5:69:98:b0:66:b5:7f:1b:5e:d4:dd:68:fb:8d:48:d1 :ba:f9:f7:e0:e2:b9:59:1b:69:35:69:fc:6c:77:29:aa:9 f:ed:5c:84:6a:dd:55:35:ed:90:5f:93:3e:50:8a:28:5d: 33:ae:d0:a5:4b:34:d8:34:e8:84:b7:81:1e:c9:b6:e2:26 :12:3b:71:d7:75:21:b2:42:02:51:96:92:07:f2:05:98:9 0:dd:a7:04:68:d8:c6:17:84:fa:d8:9f:ce:0f:ce:09:89: 90:4a:07:2e:64:41:1c:be:b1:63:9a:1e:04:12:93:12:e0 :e1:3b:62:e9:76:f9:d8:6f:87:22:0e:7d:46:e1:c3:b8:6 7:ff:5c:95:14:3f:2c:d8:2b:d7:33:c7
....
...
...
2011-12-22 04:53:18,754 DEBUG [DefaultUDPTransportMapping_10.32.127.7/0]-[org.snmp4j.transport.DefaultUdpTransportMapping] Received message from /10.32.16.4/161 with length 417: 30:82:01:9d:02:01:03:30:11:02:04:12:3b:52:9b:02:03 :00:ff:ff:04:01:07:02:01:03:04:39:30:37:04:0b:80:0 0:02:7d:03:00:19:8f:2f:f7:b9:02:01:15:02:04:00:d4: fd:bc:04:07:41:4d:53:55:73:65:72:04:0c:d7:59:cf:ba :f4:35:71:7b:0a:db:1f:88:04:08:1c:2b:1c:2b:1c:2b:1 c:2b:04:82:01:48:b6:a2:57:f6:53:96:b7:1a:0e:72:85: ad:d9:7a:17:29:5c:5e:29:04:b4:f9:54:52:4d:01:e9:1d :ff:db:94:45:01:11:3e:67:23:00:49:e5:80:0f:b7:13:f a:df:7c:1d:f1:c5:cf:e1:32:be:cf:0d:76:8d:b9:c9:8c: 9f:eb:3e:35:9e:98:2d:fe:df:0a:07:b6:c6:64:73:ee:cd :44:9d:77:ad:21:46:19:7f:20:20:cf:34:2f:7f:9a:d2:f 3:e2:39:c2:88:12:6d:45:ae:1c:94:2d:35:93:aa:ea:77: b6:37:fb:52:72:bc:e6:e2:4b:99:0c:80:bd:09:6f:c1:cb :42:df:3b:19:d5:35:35:00:2c:ec:99:7a:a0:f0:3e:24:0 4:28:3a:97:8f:27:ed:f1:e8:a4:a4:58:24:7c:e1:df:64: 8f:5c:68:8b:ed:c4:f7:a9:76:90:da:7c:9c:55:5f:ac:2e :5b:7c:22:05:12:ba:cb:84:bc:cd:9e:2a:c2:1c:f8:77:3 4:89:75:99:00:60:04:fc:94:a4:3a:39:d9:48:8e:86:0e: 84:62:63:a7:34:d2:00:89:ea:c1:b0:db:07:0a:e0:f6:88 :03:30:24:34:05:1b:98:c1:d7:bd:9f:19:95:06:95:60:a 8:6b:f6:7e:09:0a:8f:0f:a2:ac:c8:75:e4:86:8a:e0:58: 40:8b:63:39:7b:b3:91:ad:1e:46:70:cc:dc:ad:0f:80:a6 :fb:97:ca:e7:a6:fa:46:b0:e5:86:1c:a1:38:01:7c:84:4 5:77:5f:6e:85:b8:9a:6a:88:e5:b5:93:ef:d6:5e:49:df: 28:41:36:87:c4:9c:c9:5f:b8:06:74:47:ee:bb:6f:de:c1
2011-12-22 04:53:18,754 DEBUG [DefaultUDPTransportMapping_10.32.127.7/0]-[org.snmp4j.mp.MPv3] SNMPv3 header decoded: msgId=305877659, msgMaxSize=65535, msgFlags=07, secModel=3
--------------------------------
Server Configuration.
=================
T5220 SUN Server 32 GB RAM, and 64 CPUS
--
Solaris Server in Configured in Cluster has three interfaces, namely.
Solaris server AP05:
bash-3.00$ ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv 4,VIRTUAL> mtu 8232 index 1
******* inet 127.0.0.1 netmask ff000000
aggr109: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
******* inet 10.32.125.10 netmask ffffffc0 broadcast 10.32.125.63
******* ether 0:21:28:3d:bd:5a
aggr110: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
******* inet 10.32.127.6 netmask ffffffe0 broadcast 10.32.127.31
******* ether 0:21:28:3d:bd:5b
aggr111: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
******* inet 10.32.125.99 netmask ffffffe0 broadcast 10.32.125.127
******* ether 0:21:28:3d:bd:5c
Packet Capture commands.
nohup /usr/sbin/snoop -o /var/tmp/All_snoop_AP5_ aggr109_Interface.log -d aggr109 &
nohup /usr/sbin/snoop -o /var/tmp/All_snoop_AP5_ aggr110_Interface.log -d aggr110 &
nohup /usr/sbin/snoop -o /var/tmp/All_snoop_AP5_ aggr111_Interface.log -d aggr111 &
Solaris server AP06:
bash-3.00$ ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv 4,VIRTUAL> mtu 8232 index 1
******* inet 127.0.0.1 netmask ff000000
aggr112: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
******* inet 10.32.125.12 netmask ffffffc0 broadcast 10.32.125.63
******* ether 0:21:28:3e:e:9c
aggr113: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
******* inet 10.32.127.7 netmask ffffffe0 broadcast 10.32.127.31
******* ether 0:21:28:3e:e:9d
aggr114: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
******* inet 10.32.125.100 netmask ffffffe0 broadcast 10.32.125.127
******* ether 0:21:28:3e:e:9e
Packet Capture commands.
nohup /usr/sbin/snoop -o /var/tmp/All_snoop_AP6_ aggr112_Interface.log -d aggr112 &
nohup /usr/sbin/snoop -o /var/tmp/All_snoop_AP6_ aggr113_Interface.log -d aggr113 &
nohup /usr/sbin/snoop -o /var/tmp/All_snoop_AP6_ aggr114_Interface.log -d aggr114 &