OpenSolaris Solaris Forum  
     

Left Nav FAQ Members List Default Password Members List Linux Commands Search Today's Posts Mark Forums Read Right Nav

Left Container Right Container
 

Go Back   OpenSolaris Solaris Forum Solaris Networking

Reply
 
LinkBack Thread Tools Display Modes
  #1 (permalink)  
Old 01-05-2012, 11:30 AM
[email protected]
Guest
 
Posts: n/a
Red face SNMP Messages Lost @ Solaris Kernel. ( How to Debug )

We an NMS Application build in J2EE Technology using SNMP 4j Library.
We have one issue, from one of our clients that CERTAIN SNMP Packets are not send out of Solaris SERVER.

Problem Found:
The First SNMP Request of a Retransmitted SNMP Message from Solaris Server, is NOT Seen in any of the Solaris Application server Interface via snoop.
Meant, the First SNMP Packet of a Retransmitted request is not forwarded out of Solaris server.

The lowest layer, of Application that send SNMP Messages is SNMP 4J Library.

And according to the SNMP 4j Debug logs,
It says that the Initial SNMP Request had been sent. and no error thrown.
But I am unable to see this SNMP Request in SNOOP on the available interface in Solaris Server.

CAN YOU KINDLY LET ME KNOW HOW TO GO ABOUT THIS ISSUE?
THANKS>>>>

The following traces, refer the SNMP Request ID : 1811419093


SNMP First Retry (Initial Request)
---------------------------------

2011-12-22 04:52:48,722 DEBUG [JM-120]-[org.snmp4j.mp.MPv3] Adding cache entry: StateReference[msgID=305877635,pduHandle=PduHandle[1811419093],securityEngineID=80:0 0:02:7d:03:00:19:8f:2f:f7:b9,securityModel=org.snm [email protected],securityName=AMSUser,sec urityLevel=3,contextEngineID=80:00:02:7d:03:00:19: 8f:2f:f7:b9,contextName=nt]
2011-12-22 04:52:48,722 DEBUG [JM-120]-[org.snmp4j.Snmp] Running pending sync request with handle PduHandle[1811419093] and retry count left 2
2011-12-22 04:52:48,722 DEBUG [JM-120]-[org.snmp4j.transport.DefaultUdpTransportMapping] Sending message to 10.32.16.4/161 with length 345: 30:82:01:55:02:01:03:30:11:02:04:12:3b:52:83:02:03 :00:ff:ff:04:01:07:02:01:03:04:39:30:37:04:0b:80:0 0:02:7d:03:00:19:8f:2f:f7:b9:02:01:15:02:04:00:d4: fd:9e:04:07:41:4d:53:55:73:65:72:04:0c:63:71:7e:b5 :da:6f:2b:e7:3f:4b:a6:b1:04:08:00:00:00:15:bc:20:d 9:44:04:82:01:00:17:ee:44:2b:c1:26:b2:f6:b4:5e:cc: 0a:64:03:49:79:d6:f7:6d:0d:c8:ab:9c:d9:b8:3c:1e:1a :95:d7:9c:a8:8f:09:ef:1f:a4:2c:04:25:d0:0d:2d:0a:6 c:ea:c5:ba:ec:2e:5a:ba:76:91:91:e1:61:ef:fa:88:e2: aa:c0:de:99:d9:44:e9:0d:2d:78:3c:37:47:71:e0:c3:0d :a9:fa:c0:4f:ff:2c:33:b2:28:b5:fc:47:85:19:2f:f4:0 1:e9:81:4c:9a:88:2b:ee:b5:ce:91:67:4c:14:36:f3:32: 05:bc:bc:ad:24:58:63:34:12:a2:aa:e8:f4:b2:46:1f:13 :60:07:4b:9c:3d:36:3c:e5:3b:9c:9e:27:7d:1d:28:39:b 6:e8:cd:fe:7a:70:2a:8c:29:00:49:19:05:11:cb:19:c0: f0:c2:5e:43:aa:ce:b8:40:2f:32:a7:0b:30:ef:d3:55:53 :3b:03:8a:fd:27:51:cb:6d:5f:62:e6:22:28:f1:74:b4:7 a:6d:ed:90:45:ca:39:d2:e3:7b:4c:3b:d7:99:1a:0a:1d: 1c:69:20:3d:f0:56:b2:fb:c1:35:00:ce:16:96:64:00:f2 :7b:55:2d:d1:7a:45:23:e1:48:76:b8:eb:6c:7b:8a:7b:e 4:9c:1e:8c:6f:86:06:98:15:1e:75:0d

SNMP 4j says, that it had send the SNMP Message, and did not receive the response (For Initial SNMP Request)
.
So, After 30 Seconds ( Timeout before first Re-transmission =~ 30 Seconds )
-----------------------

SNMP 4j attempts for Second Retry
-------------------------------------------
2011-12-22 04:53:18,730 DEBUG [Timer-17]-[org.snmp4j.Snmp] Running pending sync request with handle PduHandle[1811419093] and retry count left 1
2011-12-22 04:53:18,730 DEBUG [Timer-17]-[org.snmp4j.transport.DefaultUdpTransportMapping] Sending message to 10.32.16.4/161 with length 345: 30:82:01:55:02:01:03:30:11:02:04:12:3b:52:9b:02:03 :00:ff:ff:04:01:07:02:01:03:04:39:30:37:04:0b:80:0 0:02:7d:03:00:19:8f:2f:f7:b9:02:01:15:02:04:00:d4: fd:bc:04:07:41:4d:53:55:73:65:72:04:0c:ea:2a:5f:de :12:0e:cf:bd:53:27:b4:b4:04:08:00:00:00:15:bc:20:d 9:5c:04:82:01:00:34:89:7a:5a:63:04:17:c0:2b:da:f5: 54:e1:3c:bc:bd:2f:87:f6:00:c1:b8:f8:6e:3d:8d:9c:e5 :4e:b8:a3:ec:0d:a3:67:c7:6c:3a:bf:a3:34:a6:40:18:2 b:c3:63:af:e9:fa:bf:8b:6a:04:7e:8e:cb:2f:94:d4:ce: cf:79:8d:4f:38:33:a9:c7:88:b1:42:11:e8:b7:34:36:a5 :3e:77:b7:a3:28:e0:f8:3b:92:64:28:9e:88:bd:08:4c:3 1:70:31:68:44:4f:34:8a:83:c3:41:de:d6:68:f1:15:a1: 0c:d5:69:98:b0:66:b5:7f:1b:5e:d4:dd:68:fb:8d:48:d1 :ba:f9:f7:e0:e2:b9:59:1b:69:35:69:fc:6c:77:29:aa:9 f:ed:5c:84:6a:dd:55:35:ed:90:5f:93:3e:50:8a:28:5d: 33:ae:d0:a5:4b:34:d8:34:e8:84:b7:81:1e:c9:b6:e2:26 :12:3b:71:d7:75:21:b2:42:02:51:96:92:07:f2:05:98:9 0:dd:a7:04:68:d8:c6:17:84:fa:d8:9f:ce:0f:ce:09:89: 90:4a:07:2e:64:41:1c:be:b1:63:9a:1e:04:12:93:12:e0 :e1:3b:62:e9:76:f9:d8:6f:87:22:0e:7d:46:e1:c3:b8:6 7:ff:5c:95:14:3f:2c:d8:2b:d7:33:c7
....
...
...
2011-12-22 04:53:18,754 DEBUG [DefaultUDPTransportMapping_10.32.127.7/0]-[org.snmp4j.transport.DefaultUdpTransportMapping] Received message from /10.32.16.4/161 with length 417: 30:82:01:9d:02:01:03:30:11:02:04:12:3b:52:9b:02:03 :00:ff:ff:04:01:07:02:01:03:04:39:30:37:04:0b:80:0 0:02:7d:03:00:19:8f:2f:f7:b9:02:01:15:02:04:00:d4: fd:bc:04:07:41:4d:53:55:73:65:72:04:0c:d7:59:cf:ba :f4:35:71:7b:0a:db:1f:88:04:08:1c:2b:1c:2b:1c:2b:1 c:2b:04:82:01:48:b6:a2:57:f6:53:96:b7:1a:0e:72:85: ad:d9:7a:17:29:5c:5e:29:04:b4:f9:54:52:4d:01:e9:1d :ff:db:94:45:01:11:3e:67:23:00:49:e5:80:0f:b7:13:f a:df:7c:1d:f1:c5:cf:e1:32:be:cf:0d:76:8d:b9:c9:8c: 9f:eb:3e:35:9e:98:2d:fe:df:0a:07:b6:c6:64:73:ee:cd :44:9d:77:ad:21:46:19:7f:20:20:cf:34:2f:7f:9a:d2:f 3:e2:39:c2:88:12:6d:45:ae:1c:94:2d:35:93:aa:ea:77: b6:37:fb:52:72:bc:e6:e2:4b:99:0c:80:bd:09:6f:c1:cb :42:df:3b:19:d5:35:35:00:2c:ec:99:7a:a0:f0:3e:24:0 4:28:3a:97:8f:27:ed:f1:e8:a4:a4:58:24:7c:e1:df:64: 8f:5c:68:8b:ed:c4:f7:a9:76:90:da:7c:9c:55:5f:ac:2e :5b:7c:22:05:12:ba:cb:84:bc:cd:9e:2a:c2:1c:f8:77:3 4:89:75:99:00:60:04:fc:94:a4:3a:39:d9:48:8e:86:0e: 84:62:63:a7:34:d2:00:89:ea:c1:b0:db:07:0a:e0:f6:88 :03:30:24:34:05:1b:98:c1:d7:bd:9f:19:95:06:95:60:a 8:6b:f6:7e:09:0a:8f:0f:a2:ac:c8:75:e4:86:8a:e0:58: 40:8b:63:39:7b:b3:91:ad:1e:46:70:cc:dc:ad:0f:80:a6 :fb:97:ca:e7:a6:fa:46:b0:e5:86:1c:a1:38:01:7c:84:4 5:77:5f:6e:85:b8:9a:6a:88:e5:b5:93:ef:d6:5e:49:df: 28:41:36:87:c4:9c:c9:5f:b8:06:74:47:ee:bb:6f:de:c1
2011-12-22 04:53:18,754 DEBUG [DefaultUDPTransportMapping_10.32.127.7/0]-[org.snmp4j.mp.MPv3] SNMPv3 header decoded: msgId=305877659, msgMaxSize=65535, msgFlags=07, secModel=3
--------------------------------

Server Configuration.
=================
T5220 SUN Server 32 GB RAM, and 64 CPUS
--
Solaris Server in Configured in Cluster has three interfaces, namely.

Solaris server AP05:

bash-3.00$ ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv 4,VIRTUAL> mtu 8232 index 1
******* inet 127.0.0.1 netmask ff000000
aggr109: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
******* inet 10.32.125.10 netmask ffffffc0 broadcast 10.32.125.63
******* ether 0:21:28:3d:bd:5a
aggr110: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
******* inet 10.32.127.6 netmask ffffffe0 broadcast 10.32.127.31
******* ether 0:21:28:3d:bd:5b
aggr111: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
******* inet 10.32.125.99 netmask ffffffe0 broadcast 10.32.125.127
******* ether 0:21:28:3d:bd:5c

Packet Capture commands.
nohup /usr/sbin/snoop -o /var/tmp/All_snoop_AP5_ aggr109_Interface.log -d aggr109 &
nohup /usr/sbin/snoop -o /var/tmp/All_snoop_AP5_ aggr110_Interface.log -d aggr110 &
nohup /usr/sbin/snoop -o /var/tmp/All_snoop_AP5_ aggr111_Interface.log -d aggr111 &



Solaris server AP06:

bash-3.00$ ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv 4,VIRTUAL> mtu 8232 index 1
******* inet 127.0.0.1 netmask ff000000
aggr112: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
******* inet 10.32.125.12 netmask ffffffc0 broadcast 10.32.125.63
******* ether 0:21:28:3e:e:9c
aggr113: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
******* inet 10.32.127.7 netmask ffffffe0 broadcast 10.32.127.31
******* ether 0:21:28:3e:e:9d
aggr114: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 4
******* inet 10.32.125.100 netmask ffffffe0 broadcast 10.32.125.127
******* ether 0:21:28:3e:e:9e

Packet Capture commands.
nohup /usr/sbin/snoop -o /var/tmp/All_snoop_AP6_ aggr112_Interface.log -d aggr112 &
nohup /usr/sbin/snoop -o /var/tmp/All_snoop_AP6_ aggr113_Interface.log -d aggr113 &
nohup /usr/sbin/snoop -o /var/tmp/All_snoop_AP6_ aggr114_Interface.log -d aggr114 &
Reply With Quote
Sponsored Links

Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may post new threads
You may post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On


Similar Threads
Thread Thread Starter Forum Replies Last Post
Solaris SNMP syntax Unregistered Solaris General 0 08-04-2011 01:41 PM
snmp on solaris 8 lamou23 Solaris General 0 01-13-2011 07:26 PM
proc_t in kernel memory nccc Solaris General 0 11-01-2009 08:36 AM


All times are GMT. The time now is 12:28 AM.


Powered by vBulletin. Copyright 2000 - 2008, Jelsoft Enterprises Ltd.
Search Engine Friendly URLs by vBSEO 3.1.0 ©2007, Crawlability, Inc.