DiagAgent Troubleshooting

SAP Solution Manager 7.
0 EhP1
End-to-End Diagnostics
Diagnostics agent troubleshooting
Version 1.5, June 2009
END-TO-END DIAGNOSTICS
DIAGNOSTICS AGENT TROUBLESHOOTING
Typographic Conventions Icons

Type Style Represents Icon Meaning
Example Words or characters quoted Caution
Text from the screen. These
include field names, screen Example
titles, pushbuttons labels,
menu names, menu paths,
and menu options. Note
Cross-references to other
documentation. Recommendation
Example Emphasized words or
text phrases in body text, graphic Syntax
titles, and table titles.
EXAMPLE Technical names of system
TEXT
objects. These include report
names, program names,
transaction codes, table
names, and key concepts of
a programming language
when they are surrounded
by body text, for example,
SELECT and INCLUDE.
Example Output on the screen. This
text includes file and directory
names and their paths,
messages, names of
variables and parameters,
source text, and names of
installation, upgrade and
database tools.
Example Exact user entry. These are
text words or characters that you
enter in the system exactly
as they appear in the
documentation.
<Example Variable user entry. Angle
text> brackets indicate that you
replace these words and
characters with appropriate
entries to make entries in the
system.
EXAMPLE Keys on the keyboard, for
TEXT example, F2 or ENTER.
END-TO-END DIAGNOSTICS
DIAGNOSTICS AGENT TROUBLESHOOTING
Contents
1 Introduction ......................................................................................................................................... 4
2 Pre-requisites ...................................................................................................................................... 5
2.1 OS support ................................................................................................................................... 5
2.2 JDK compatibility......................................................................................................................... 6
2.3 Managed System.......................................................................................................................... 8
2.4 SLD production system............................................................................................................... 9
3 Agent installation strategy................................................................................................................ 10
3.1 Standard environment............................................................................................................... 10
3.2 HA/virtualization environment .................................................................................................. 10
4 Agent installer.................................................................................................................................... 13
4.1 SAPinst Documentation ............................................................................................................ 13
4.2 Troubleshooting with SAPinst .................................................................................................. 13
4.3 OS configuration checks........................................................................................................... 14
4.4 JVM parameters ......................................................................................................................... 14
4.5 Startup scripts ........................................................................................................................... 15
5 Agent connectivity ............................................................................................................................ 18
5.1 Managing system....................................................................................................................... 18
5.2 Managed system ........................................................................................................................ 21
5.3 Full Qualified Name resolution issue ....................................................................................... 24
5.4 SAP router.................................................................................................................................. 24
5.5 Self-Monitoring .......................................................................................................................... 25
6 Agent Maintenance............................................................................................................................ 26
6.1 Upgrade SAP Kernel (saposcol update)................................................................................... 26
6.2 Configuration ............................................................................................................................. 26
6.2.1 Change Path of the JDK....................................................................................................... 27
6.2.2 Change JVM Parameters .................................................................................................... 27
6.2.3 Change the virtual host name............................................................................................... 27
6.2.4 Usage of smdsetup script..................................................................................................... 28
6.2.5 Change Agent log level ........................................................................................................ 31
6.3 Managing J2EE user.................................................................................................................. 32
7 Tracking agent issues ....................................................................................................................... 33
7.1 Deprecated SAP notes .............................................................................................................. 33
7.2 OSS message pre-requisites..................................................................................................... 33
7.3 Agent crash investigation ......................................................................................................... 34
7.4 Thread dump .............................................................................................................................. 35
7.5 Activate P4 logs ......................................................................................................................... 36
7.6 Telnet SMD server commands .................................................................................................. 37
8 Scalability........................................................................................................................................... 37
8.1 Managing java stack settings.................................................................................................... 37
End-to-End Diagnostics 4
Diagnostic agent troubleshooting guide
1 Introduction
The goal of this guide is to help Solution Manager Users troubleshoot failures encountered when installing,
configuring and using the Diagnostics Agents.
It contains the description of the resolution of the most common known issues related to the Diagnostics
Agents. It gives also information, tooltips and procedure to isolate and identify the root cause of an issue
related to the Diagnostics Agent.
This guide is separated in different areas that correspond to the different phases of the Solution Manager life
cycle. The chapters of this document follow a logical order corresponding to the Diagnostics Agent’s life cycle.
However, in order to successful troubleshoot a Diagnostics Agent’s issue, you should be familiar with all parts
of this document.
This guide covers the following area:

Pre-requisites: Helps users to check that all pre-requisites are fulfilled in order to complete a correct
installation of the Diagnostics Agents.
Agent Installation Strategy: Answers to the most frequent questions rose during the planning phase of
Diagnostics Agent installations.
Agent Installer: Lists all common issues encountered during the installation of the Diagnostics Agents.
Agent Connectivity: Help users to understand and troubleshoot the connection mechanisms flow
between:
o The Diagnostics and the Managing System.
o The Diagnostics agent and the Managed systems.
It includes also the main resolution procedures related to network configuration issues.
Agent Maintenance: Describes the procedure and common errors related to the operation of the
Diagnostics Agent. It concerns the administration of the Diagnostics Agents including a description of
the smdsetup script.
Tracking Agent Issues: Explains where to find information related to the Diagnostics Agent activities. It
explains how to interpret this information in order to establish a correct diagnostics when
troubleshooting an Agent issue.
Scalability: Contains scalability and performance notes related to the Diagnostics Agent.
©SAP AG 2009
2 Pre-requisites
2.1 OS support
The following table describes the list of different OS supported by the Diagnostics agent:
OS Diagnostics Agent 7.00 / 7.01 Diagnostics Agent 7.11

Server 2003
X X
on IA32 32bit
Server 2003
Windows X X
on IA64 64bit
Server 2003 on x64
X X
64bit
5.3 64bit on Power
X X
4, 5
AIX
6.x 64bit on Power
X (SAP Kernel > 133) X
4, 5
11.31 on IA64
X X
64bit
HP
11.11,
X X
PA-RISC 64bit
10 on SPARC
Solaris X X
64bit
IA32 32bit X X
IA64 64bit X X
Linux x86_64 64bit X X
Power 64bit X X
SLES9 only on
X X
zSeries 64bit
I5/OS V5R3, V5R4 64bit X X
1.6, 1.7, 1.8, 1.9
Z/OS - -
64bit
TRU64 - -
©SAP AG 2009
2.2 JDK compatibility

The following table describes the list of different JDK supported by the Diagnostics agent for each platform:
OS Diagnostics Agent 7.00 / 7.01 Diagnostics Agent 7.11

Server 2003
SUN 1.4.2.13 to 1.4.2.17b061 or 1.4.2_21+ SAPJVM5 last version
on IA32 32bit
Server 2003
Windows SUN 1.4.2.13 to 1.4.2.17b061 or 1.4.2_21+ SAPJVM5 last version
on IA64 64bit
Server 2003 on x64
64bit
5.3 64bit on Power
IBM 1.4.2 SR12 caix64142-20080923 SAPJVM5 last version
4, 5
AIX
6.x 64bit on Power
IBM 1.4.2 SR12 caix64142-20080923 SAPJVM5 last version
4, 5
11.31 on IA64
HP 1.4.2.15 to 1.4.2.172 or 1.4.2.21+ SAPJVM5 last version
64bit
HP
11.11,
HP 1.4.2.15 to 1.4.2.172 or 1.4.2.21+ SAPJVM5 last version
PA-RISC 64bit
10 on SPARC
Solaris 64bit
10 on x86_64 SUN 1.4.2.13 to 1.4.2.17b06 or 1.4.2_21+ SAPJVM5 last version
IA32 32bit SUN 1.4.2.13 to 1.4.2.17b061 or 1.4.2_21+ SAPJVM5 last version
IA64 64bit SUN 1.4.2.13 to 1.4.2.17b061 or 1.4.2_21+ SAPJVM5 last version
Linux x86_64 64bit IBM 1.4.2 SR10 j9xa64142-20080130 SAPJVM5 last version
Power 64bit IBM 1.4.2 SR12 cxp64142-20080923 SAPJVM5 last version
SLES9 only on
IBM 1.4.2 SR12 cxz64142-20080923 SAPJVM5 last version
zSeries 64bit
I5/OS V5R3, V5R4 64bit IBM IT4J JVM SAPJVM5 last version
1.6, 1.7, 1.8, 1.9
Z/OS - -
64bit
TRU64 - -
For Diagnostics agent 711, the SAPJVM 5 is included with the SAP Kernel DVD. Therefore, you don’t need to install
separately the SAPJVM5 before installing the 711 agent.
The following SAP notes can be used to determine the JDK version but do not apply the JVM parameters proposed
in those sap notes since they are only specific to the sapj2ee engine.
Note 716604 for the Sun JDK (Windows, Linux, Solaris)

Note 716926 for the HP JDK (HP-UX)
Note 716927 for the IBM JDK (AIX)
Note 717376 for the IBM JDK (iSeries, OS/400)
Note 746299 for the IBM JDK (zSeries)
Note 810008 for the IBM JDK (Linux on POWER)
Note 861215 for the IBM JDK (Linux on AMD64/EMT46)
1 Until further notice please don't use 1.4.2.17b12, 1.4.2_18 and 1.4.2_19.
2 Until further notice please don't use 1.4.2.20 (SAP note 1301397)
©SAP AG 2009
Most common JDK problems with agent
Timeout occurred
This error is logged in SMDSystem.log
Exception: Time out occurred when calling method 'configure' on object

[com.sap.smd.api.IAgentContext_Stub@49f949f9] after 10000 ms.
possible cause: com.sap.smd.server.util.concurrent.TimeoutException at
com.sap.smd.server.exec.asio.AsioInvocationHandler.invoke
(AsioInvocationHandler.java:130)
This timeout issue is caused either by a wrong JDK Version or wrong JVM parameters. The timeout
occurs often on old IBM SDK or when the J9 technology is not enabled in the java parameters of the
Diagnostics agent. Check also the java stack thread settings (see chapter 8.1 Managing java stack
settings)
Agent applications are not updated
This error is still logged in SMDSystem.log
Exception: [Thread[Thread-5,5,main]] Error

[com.sap.smd.om.impl.broker.ObjectProxyImpl@76933bcb (states:core.patch
[proxy])]: error during load of states:core.patch
[EXCEPTION] com.sap.smd.om.ObjectLoadException: failed to instantiate
implementation class at
com.sap.smd.core.plugin.service.ServiceEntityHandle.entityLoad(ServiceEntityHandl
e.java:159)
...
...
Caused by: java.lang.UnsatisfiedLinkError: inode0
at sun.nio.ch.FileChannelImpl.inode0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:868)
This problem occurs only with HP JDK; re-install a JDK version recommended for Diagnostics agent
(currently 1.4.2.17 or 1.4.2.21+).
©SAP AG 2009
2.3 Managed System

The following list describes the SAP systems for which the SAPinst installs automatically a Diagnostics agent at the
end of the SAP instance (central or dialog) installation procedure.
System SAPinst 7.0 SR3 SAPInst 7.01 Ehp1 SAPInst 7.11 Ehp1
NW 2007 Abap stack - - -
NW 2007 Java stack X X -
Netweaver Double stack X X -
Solution Manager X X -
NW 711 Abap stack - - (X) Not scheduled yet
NW 711 Java stack - - (X) Not scheduled yet
PI 7.11 - - X
CE 7.11 - - X
PI 7.10 - - -
NW 7.10 Java stack - - -
If the Diagnostics Agent has not been installed automatically, you have to install it as a standalone engine using the
standard SAPinst (Please refer the Agent Setup Guide for more information).
To avoid libraries conflicts it is highly recommended to install a Diagnostics agent which has the same kernel release
than the SAP systems already installed on the host.
Ex: On a NW04 or NW04s Managed System, you should install a Diagnostics agent 700 or 701,
On a 711 or 710 Managed System, you should install a Diagnostics agent 711,
…
Note for System NW2004 (640), the recommendation is to install a Diagnostics agent NW2004s (700)
only if no previous Diagnostics agent 640 has been installed on system.
Ex: on a host with a sap system 640, if no agent is present -> install a new diagnostics agent 700(SR3)
on a host with sap system 640, if one or more agent 640 are present -> install a new 640 agent.
©SAP AG 2009
2.4 SLD production system

To register a Diagnostics agents to a productive SLD, the CIM Model version of the productive SLD must be 1.5.26 or
higher.
How to determine which version of the CIM Model is installed?

Go in SLD with the url: http://sldhostname:port/sld
Logon with a J2EE user having the role SAP_SLD_ADMINISTRATOR
Click on link ‘About’. The CIM Model version is displayed in a window popup.
How to upgrade the CIM Model?

Refer to SAP Note 669669 Updating the SAP Component Repository in the SLD
How to check if the Diagnostics agent is registered in SLD?

Go in SLD with the url: http://sldhostname:port/sld
Logon with a J2EE user having the role SAP_SLD_ADMINISTRATOR
Go in the section ‘Administration’
Click on link ‘Content Maintenance’
Select the subset ‘All with Instances and class ‘Diagnostics Agent Instance’
If the class ‘Diagnostics Agent Instance’ does not appear in drop down list, No Diagnostics agent is registered
In column Name of table, check if the Diagnostics agent is present.
Or
Go in Agent Administration tool and click on link “agent candidates”
Enter the information to connect to productive SLD
The list of Diagnostics agents registered in this SLD will be displayed.
Which is the J2EE Group to assign to the SLD user used by the Diagnostics Agent?
The SAP_SLD_CONFIGURATOR group or the security role DataSupplierLD should be assigned to the user
used by the Diagnostics Agent.
What is the SLD registration frequency of the Diagnostics Agents?

Every 12 hours, the Diagnostics agent sends an SLD request to update the agent model information.
What is the frequency used by the Diagnostics Agent to check if an association is present in SLD?
If the Diagnostics agent is not connected to the Managing System and registered in SLD, it will check every 20
minutes if an association has been created in SLD.
If the Diagnostics agent is still connected to Managing System and registered in SLD, it is going to check every
hour if a new association has been created in SLD.
Why the Diagnostics agent is not visible in Agent Admin tool?

You need to create an SLD association between the Diagnostics agent and the Managing system. To create
this association go in Agent Admin tool, click on link “agent candidate”. Enter the SLD connection data where
the Diagnostics agent is registered then select the agent and click on the “attach” button.
©SAP AG 2009
3 Agent installation strategy
3.1 Standard environment

The following check list must be validated before installing a new Diagnostics agent on a managed host:
Note: Don’t install a Diagnostics Agent on a Database or an SCS host without any Central Instance. This
is not necessary in the current E2E RCA scenarios.
Check if a Diagnostic agent has already been installed on the managed system host.
Verify if the managed system has been installed with a SAPInst version greater than NW04S SR3
Check on file system if the folder /usr/sap/DA or the folder /usr/sap/SMD exist.
In both cases, an agent is already installed on the system.
If the Diagnostics agent is already installed on system, check if the agent is present in the SLD server where
the managed system is registered. In this case attach the agent to the Solution Manager System using the
agent admin tool (agent candidate view) or connect the agent with the script ‘smdsetup’ (see part 6.2.4).
If the Diagnostics agent is not installed, retrieve the virtual host or physical host name of the managed system.
The rule is to install one diagnostics agent per virtual host.
Go in the sap profile of the managed system (/usr/sap/SID/SYS/profile).
Retrieve the hostname in the profile file name. (Ex: for the value W46_JC93_NCED60185755A, the
hostname is nced60185755a).
The Diagnostics agent must be installed with this hostname.
When the installation is finished, on UNIX systems add the Diagnostics agent in OS startup/shutdown script
(see chapter 4.5 Startup scripts)
For UNIX system, check these Operating System limits of OS agent user (see part 4.2).
3.2 HA/virtualization environment
In High Availability environment, the Diagnostics agents must be installed on each virtual host used by the SAP
Instance (Except for the hosts running the database or the Central Services).
The Diagnostics Agents must be integrated in the cluster fail-over scripts used to control the managed systems. For
this reason, the recommendation is to install the Diagnostics agent on a shared file system.
During a failover scenario, the SAP instance of the managed system and the Diagnostic agent must be moved
together.
When the SAP instance and the agent are moved to another system node, you should mount the agent
instance partition before the SAP instance partition. This is due to the fact that the classpath of the SAP java
instance depends on some jar files contained in the agent folder for the Wily IS instrumentation.
©SAP AG 2009
Microsoft Cluster (MSCS)
With MSCS you enable critical system components, Single Points of Failure (SPOFs), to be automatically switched
from one machine to the other in the event of hardware or software problems, or system maintenance affecting one
machine.
Until now, SAP has only supported the installation of one clustered SAP system in one MSCS cluster with two MSCS
nodes.
It is necessary to install a Diagnostics agent for each central instance and all dialog instances. The Diagnostics agents
must be installed on a local disk.
On the failover node, you should activate the windows service used to start the Diagnostics Agent.
Figure 1: ABAP or Java System with SPOFs, where non-SPOFs are installed locally on the MSCS Cluster Nodes
©SAP AG 2009
Figure 2: ABAP+Java System where the non-SPOFs are installed on hosts outside of the MS Cluster
More information is available in SAP Note 112266 - SAP and MS Cluster Server: Frequent questions and tips.
Flexframe
A FlexFrame-based server landscape for SAP usually consists of the following components:
2 highly available control nodes (without SAP software)

One or more network appliance filer(s) (IO subsystem, NFS server)
Application-node with Linux or Solaris operating systems.
Implemented high availability of SAP components (myAMC - FlexFrame autonomous agents).
Virtualization of SAP services (at least 1 IP address for each service irrespective of the server IP address). This
can result in an instance being "changed" to another server.
In addition, a check would be carried out to see whether the instance in question is already running on another
application node.
On this platform, install a Diagnostics agent for each service hosted on the application nodes identified by virtual IP
addresses.
Other virtualization solutions
Please refer to SAP Note 674851 - Virtualization on Windows to get a list of supported virtualization solutions on Windows.
©SAP AG 2009
4 Agent installer
4.1 SAPinst Documentation
The following SAP Notes contain the most recent version of the ‘Agent Setup guide’ documentation and describes the
problems that could occur during the installation:
SAP Note 1234387: Inst. 7.1 EhP1 - Diagnostics Agent installation

SAP Note 1234384: Inst. NW/SOLMAN 7.0 EhP1 - Diagnostics Agent installation
SAP Note 1144737 Inst. NW 7.0 SR3/SOLMAN 7.0 - Diagnostics Agent installation
Don’t use the SAPInst SPS13 for SMD AGENT 7.00 or lower (SMDAGENT13_0-20001420.SAR), this
installer is deprecated.
If you need to install more than 20 agents, it is recommended to use the Mass Deployement option of the
Diagnostics agent.
SAP Note 1172764: Mass Deployment of SAP Diagnostics Agent 7.0 & 7.01
SAP Note 1294713: Mass Deployment of SAP Diagnostics Agent 7.11.
4.2 Troubleshooting with SAPinst

This section describes how to proceed when errors occur during the installation of the Diagnostics Agent with SAPinst.
If an error occurs, SAPinst will stop the installation and display a dialog informing you about the error.
In that case, the procedure is the following:
a. To view the log file, choose View Logs.

b. If an error occurs during the dialog or processing phase, do either of the following:
Try to solve the problem
Abort the installation with Exit
Continue the installation by choosing Retry.
c. Check the log files of SAPinst in the installation directory sapinst_instdir SAPinst normally creates
this installation directory directly below the temporary directory. SAPinst finds the temporary directory by
checking the value of the environment variables TEMP, TMP, or TMPDIR.
If no value is set for these variables, SAPinst uses /tmp/sapinst_instdir as default installation directory
on UNIX platforms and %ProgramFiles%\sapinst_instdir on windows platforms. If you want to use an
alternative installation directory, set the environment variable TEMP, TMP, or TMPDIR to the required directory.
©SAP AG 2009
4.3 OS configuration checks

After the installation of the Diagnostics agent under UNIX platform, check the following OS parameters corresponding
to the Diagnostics agent OS user (ex: smdadm)
Check the Diagnostics agent OS user limit :

o Logon with the agent OS user ‘su – smdadm’
o Open ‘sh ‘shell with command ‘sh’
o Execute the command ‘ulimit –a’
o Compete the values with the following recommended values
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 4194304
memory(Kbytes) unlimited
coredump(blocks) 4108901
nofiles(descriptors) 8092
If the user’s limit parameters for the OS agent user are not correct, some OutOfMemory exceptions can occur
in the extractor jobs of the Diagnostics agents.
Check File System access right:
If during an extractor job (datacollect), an access denied occurs in the agent log ‘SMDApplication.log’, check if
the folder /usr/sap/<SID_AGT>/JXX/SMDAgent/temp is writable for the sapsys group. If is not the case,
execute the command ‘chmod –R 775 /usr/sap/<SID_AGT>/JXX/SMDAgent/temp’.
Furthermore the file system of the Managed system must be readable by the Diagnostics agent.
Logon with an agent OS user (su – smdadm), go in the managed system folder (ex:
/usr/sap/SID/SYS/profile), and open one profile with the command ‘more SID_INSTANCE_HOST’. If the
“more” command is not working, the access rights are not correct for the Diagnostics agents. In that case,
check with your system Administrator to adapt the file system rights.
4.4 JVM parameters

In this section, you will find the recommended Diagnostics agent’s JVM parameters for each JDK vendor.
JDK JVM Parameters SAP Note

-DP4ClassLoad=P4Connection -Xj9 -Xgcpolicy:gencon -Xmn50m -Xmx256m
IBM SR10 or higher 1093831
-Xms256m
SUN JDK -DP4ClassLoad=P4Connection -Xmx128m -Xms128m n.a
HP JDK -DP4ClassLoad=P4Connection -Xmx128m -Xms128m n.a
SAP JVM5 -DP4ClassLoad=P4Connection -Xmx128m -Xms128m n.a
Don’t use the java parameter verbose:gc in the Diagnostics agents JVM parameters, because this
parameter will steadily increase the size of the file work/std_SMDAgent.out
©SAP AG 2009
4.5 Startup scripts

Information about Diagnostics agent 7.0x
On Windows platform the SAP MMC allows to start or stop the Diagnostics agent. The NT service of the
Diagnostics agent is configured to start automatically during Windows startup.
The green status in SAP MMC only indicates that the agent is running. It doesn’t report if the agent is
connected to the Managing system. If the status is yellow after the agent start, it is probably due to the fact
that the agent process was stopped abnormally (see 7.3 Agent crash investigation)
On UNIX platform, the commands smdstart.sh, smdstop.sh and smdadmin.sh allow to operate the Diagnostics
agent. Since the SR3 release of SAPInst, those commands are included in the PATH variable of the OS agent
user. In that case, the commands can be called from every folder (not only in the agent script folder).
Command Action
smdstart.sh <SID> <InstanceNumber> Starts the Diagnostics agent and the sapstartsrv process.
smdstop.sh <SID> <InstanceNumber> Stops the Diagnostics agent and the sapstartsrv process
Launches jcmon tool to administrate the Diagnostics agent
smdadmin.sh <SID> <instanceNumber>
instance
On Solaris platform, an error (/usr/sap/SMD/exe/smdstart.sh: usage: ulimit [ -HSacdfnstv ] [ limit ])

occurs when the smdstart.sh is executed. Check the solution in sap note 1109366.
On HP-UNIX platform, an error (!! ERRROR !! current user is not the owner of this instance )
occurs when the smdstart.sh is executed. Check the solution in sap note 1109366.
The following table gives the list of the return code of the smdstart.sh script.
Exit code of smdstart.sh Reason

Success to launch the agent process (This is not the confirmation that the agent is
0
connected to SMD).
2 Input Parameters are invalid.
5 Agent is already running.
4 User limit are not set correctly.
3 Current OS user is invalided to start the agent.
For UNIX system, Diagnostics agents do not start automatically when the system reboots because the agent startup
scripts are not included in the system startup scripts.
The following command line can be used in the system startup scripts:
su -c /usr/sap/SMD/J98/script/smdstart.sh > /dev/null 2>&1 -s /bin/sh - smdadm
You will find below the command line for the shutdown sequence:
su -c /usr/sap/SMD/J98/script/smdstop.sh > /dev/null 2>&1 -s /bin/sh - smdadm
©SAP AG 2009
The following procedure describes how to insert an agent in the boot sequence for a Linux SUSE 10 system:
Create a file diagnostics_agents in folder /etc/init.d

Determine the init run level in file /etc/inittab
Create a symbolic link named S80diagnostics_agents in the folder
/etc/init.d/rc<INIT_LEVEL>.d/ to the file diagnostics_agents (The agent has to be started after
the initialization of the network services).
Create a symbolic link named S80diagnostics_agents in the folder /etc/init.d/rc6.d to the file
diagnostics_agents (runlevel 6 corresponds to the System reboot).
When smdstart.sh and smdstop.sh are embedded in the system startup scripts, the exit code of the smd
scripts cannot be retrieved. Further it is important to redirect the standard output and standar error with a pipe
like 2>&1 to avoid that the smdstart.sh and smdstop.sh are blocking the parent script.
#!/bin/sh
### BEGIN INIT INFO
# Provides: diagnostics_agents
# Required-Start: $network
# Required-Stop: $network
# Default-Start: 2 3 5
# Description: start diagnostics agents
### END INIT INFO
case "$1" in
'start')
su -c /usr/sap/SMD/J98/script/smdstart.sh > /dev/null 2>&1 -s /bin/sh - smdadm
;;
'stop')
su -c /usr/sap/SMD/J98/script/smdstop.sh > /dev/null -s /bin/sh - smdadm
;;
'status')
su -c /usr/sap/SMD/J98/script/smdadmin.sh -s /bin/sh - smdadm
;;
*)
echo "Usage: $0 { start | stop | status }"
;;
esac
exit 0
©SAP AG 2009
Information about Diagnostics agent 7.11 or higher
With Diagnostics agent 711 the startup system is compliant to the SAP startup mechanism. It means that the
commands smdstart.sh and smdstop.sh have been removed.
Under Windows the SAP MMC allows to start or stop the Diagnostics agent. The NT service of Diagnostics
agent is configured to start automatically during the Windows startup.
The green status in SAP MMC only indicates that the agent is running. It doesn’t report if the agent is
connected to the Managing system. If the status is yellow after the agent start, it is probably due to the fact
that the agent process was stopped abnormally (see 7.3 Agent crash investigation)
Under UNIX with an agent OS User, the commands startsap and stopsap allow to operate the Diagnostics
agent.
Command Action
startsap SMDA<Instance_Number> [-vhost <virtual hostname>] Start the Diagnostics agent.
stopsap SMDA<Instance_Number> [-vhost <virtual hostname>] Stop the Diagnostics agent.
The sapstartsrv process of Diagnostics agent is automatically started during the boot sequence of system if the sapinit
program has been installed on system. In that case, you can start or stop the Diagnostics agent with a SAP MMC
(remote).
About sapstartsrv process
SAP Note Description

927637 - Web service authentication in The Web service interface of sapstartsrv requires a user authentication
sapstartsrv as of Release 7.00 from the client (MMC, SAP MC, sapcontrol and so on) for protected
operations such as starting and stopping the SAP instance.
Note 823941 - SAP Start Service on Unix After an SAP upgrade, the SAP Start Service (sapstartsrv) on UNIX
platforms platforms is not automatically started when the operating system is
started.
©SAP AG 2009
5 Agent connectivity
5.1 Managing system

In this section, you will find a description of the different connection modes used from the agent to the Managing
System. Two modes are available:
Connection through the SAPJ2EE message server (RECOMMENDED mode)
Direct connection to SAPJ2EE p4 dispatcher.
Connection through the SAPJ2EE Message server
It is RECOMMENDED to connect the Diagnostics agent to Solution Manager with the SPA J2EE message
server mode. To establish this connection, the information about the hostname of the J2EE Message server
(SCS instance) and the HTTP port of message server must be collected from the SCS instance profile.
The agent connection mechanism can be described plainly:
A. During the installation of the Diagnostic agent, the SAPJ2EE Message server connection mode must be
selected.
B. This information is stored in the agent configuration (runtime.properties) and the credentials are stored in
the agent secure store file (secstore.properties).
C. When the Diagnostics Agent starts, the P4 framework will retrieve all J2EE entries point available from the
SCS message server (on HTTP port ex: 81xx).
D. Once the list of J2EE entries point has been collected, one P4 entry point is chosen from the list.
E. The Diagnostics agent will then establish a P4 connection to the selected J2EE entry point with the host
name and the port mapped to the P4 entry point.
©SAP AG 2009
Troubleshooting Message server connection

Before starting the agent installation or before executing the smdsetup script, you can check the content of
the SCS message server with the following procedure:
1. Open the URL: http://msgserver:81xx/msgserver/text/logon (xx corresponding to instance number

of the SCS instance)
2. Check if a P4 entry point is available,
3. On the managed system, open a shell with the <SID>adm user of the Managed system or the SMD
Agent.
4. Execute the command “ping <host of P4 entry point>”: The host must be accessible,
5. Execute the command “telnet <host of P4 entry point> <P4 port>”. If no error message
occurs (blank screen) the connection is OK.
Most common agent connection problems
All connection problems are logged in a file named SMDSystem.X.log (see chapter 6.2.5). Check this log
to find the root exception of the issue: Once the exception is found, find the root cause of the exception by
locating the last “caused by” string.
No Entry point found
Exception: com.sap.engine.services.jndi.persistent.exceptions.NamingException:
Exception while trying to get InitialContext. [Root exception is
com.sap.engine.interfaces.cross.DestinationException: there are no access points for
service: P4 registered on the message server]
No P4 entry point has been reported by the Message Server. This is probably due to the fact that it is not
the correct message server: it must be the Java and not the ABAP one. So check the content of the
Message server with the URL http://msgserver:81xx/msgserver/text/logon and verify if a P4 entry point is
available (as shown below).
Unknown host
Exception: [sapms:// yupxxxxx.domain,com:8136] Checking server availability...

Connecting to SMD server sapms://yupxxxxx.domain,com:8136 failed - error counter: 0
- com.sap.engine.services.jndi.persistent.exceptions. NamingException:
Exception while trying to get InitialContext. [Root exception is
com.sap.engine.interfaces.cross.DestinationException:
cannot establish connection with any of the available instances
#Nested exceptions are:
#java.net.UnknownHostException: yupxxxxx: yupxxxxx
]
This is a network resolution problem due to the fact that the selected P4 hostname retrieved from the SCS
Message server list cannot be resolved from the Managed system host. Retrieve the list of the P4 entries
point in the Message server as:
version 1.0
J2EE933894500
J2EE NCED6018XXXXA.dhcp.ncel.sap.corp 59300 LB=1
J2EES NCED6018XXXXA.dhcp.ncel.sap.corp 59301 LB=1
P4 NCED6018XXXXA.dhcp.ncel.sap.corp 59304 LB=1
P4S NCED6018XXXXA.dhcp.ncel.sap.corp 59306 LB=1
P4HTTP NCED6018XXXXA.dhcp.ncel.sap.corp 59305 LB=1
©SAP AG 2009
Execute the command “ping <hostname_of_P4_entry_point_5xx04>” in a Managed system shell.
If the host cannot be reached, then change the host definition in your DNS or in the /etc/hosts. You may
alternatively fix you engine installation to report the right host information in the message server (see
chapter 5.3).
Direct Connection to P4 Dispatcher
Solution Manager in cluster environment
If the solution manager system is installed on a High Availability environment, this direct connection
cannot be used because the agent would always establish a connection to the same java instance
corresponding to the same p4 port.
To enable HA, you must select the SAP J2EE Message server connection mode. In that case, the
connection of the diagnostics agent will be load-balanced to the different available Java instances.
©SAP AG 2009
5.2 Managed system

All connections to the Managed systems from the Diagnostics agent are executed locally on the same host. It is the
reason why it is really imperative to install the Diagnostics agent on the same virtual host than the SAP Managed
system.
The agent connects to the following different SAP System layers:
Java stack on port P4 (5xx04)

Sapstartsrv http port of Managed System (5xx13)
J2EE Message server (39xx and 81xx)
RFC call to Managed Abap stack
Troubleshooting P4 connection problems with Managed system
Check on which host the P4 socket is bound

o Run the OS command “netstat –an” to find the entry line containing the P4 port of the managed
system:
TCP 0.0.0.0:59304 0.0.0.0:0 LISTENING

Or
tcp 0 0 *.50304 *.* LISTEN
In this example we see that the p4 port is bound for all hosts where a p4 request can be emitted (* or
0.0.0.0), if the port is bound like “10.45.192.204:50304” only the p4 requests emitted from the host
10.45.192.204 will be accepted.
It explains why the Diagnostic agent must be installed on virtual host 10.45.192.204 with SAPInst. The
situation can be fixed by changing the profile property SAPLOCALHOST (see 6.2.3 Change the virtual
host name).
o After verifying the port in the OS configuration, check in the J2ee dispatcher configuration that the
property “bindHost” is set to 0.0.0.0. This should be visible with the J2EE configuration tool of the
managed system->Instance_xxx->dispatcher->services->p4. If the value is not configured to 0.0.0.0
(which accepts all hosts), add the agent IP or install the agent on one of the listed IP.
©SAP AG 2009
Check if the agent can access the J2ee message server (SCS) of the Managed system with the host and the
http port stored in the agent configuration (see the next section). For example :
o Open a shell with the agent OS user (smdadm) on the managed system.
o Execute the command “msprot -l -mshost <host_j2ee_msg_srv> -msserv 39<SCS> or
36<SCS> -d J2EE –s”, the values of host and port must be retrieved from the Agent configuration
(see the next section).
o The command must return the following content :
SAP Message Server Protocol Program, Version 1.3 (built: Jan 24 2008 01:27:19)
INFO CLUSTER-ID HOST GROUP-ID STATE J2EE-SERVICE-MASK
----------------------------------------------------------------------------------------------
---> LIST SERVICE:00000 NAME:ConfigurationManager
---> LIST SERVICE:00001 NAME:_CacheManager
---> LIST SERVICE:00003 NAME:{service_manager_deploy_distributor}
---> LIST SERVICE:00004 NAME:{service_manager_internal_connection}
---> LIST SERVICE:00005 NAME:shell
---> LIST SERVICE:00006 NAME:jmx_notification
---> LIST SERVICE:00007 NAME:p4
---> LIST SERVICE:00008 NAME:webservices
---> LIST SERVICE:00010 NAME:iiop
---> LIST SERVICE:00011 NAME:r3startup
---> LIST SERVICE:00012 NAME:telnet
---> LIST SERVICE:00013 NAME:log_configurator
---> LIST SERVICE:00014 NAME:jmx
---> LIST SERVICE:00016 NAME:http
---> LIST SERVICE:00022 NAME:com.sap.security.core.ume.service
---> LIST SERVICE:00023 NAME:security
---> LIST SERVICE:00024 NAME:tc~sec~certrevoc~service
---> LIST SERVICE:00025 NAME:servlet_jsp
---> LIST SERVICE:00026 NAME:webdynpro
LIST 0004105250 ncevhost.mydomain 0000041052 RUNNING XXXXXXXXOXXOXXXXXOXXXXXXXXXXXX
LIST 0004105200 ncevhost.mydomain 0000041052 RUNNING XXXXXXOXXXXXXXXXXOOOOOOOOOOOOO
In this content, the P4 service is listed as well as the j2ee nodes with hostnames (e.g.
ncevhost.mydomain).
Check if those hostnames can be resolved on your network by executing the command “ping
ncevhost.mydomain”
o Open the J2EE Visual admin of the Managed system and connect with the j2ee user used by the Diagnostics
agent (agent configuration see next section) to check if the user can be authenticated and if it has the
Administrator role.
o Check also the Managed system defaulttrace file to find some exception linked to P4 connection.
©SAP AG 2009
Connection configuration for managed systems
The following table lists all properties generated and updated by the Managed Setup Wizard allowing the Diagnostics
agent to connect to the Managed system in order to complete the E2E RCA operations.
Those properties are stored in the Diagnostics agent configuration under the scope named “server name”.
o Managed systems connection information:
Connection Type to Port Configuration stored in agelet Property

Managed system
P4 port 5xx04 com.sap.smd.agent.application.global.configuration [SID]/[INB]/sapj2ee/P4/port
http port 5xx00 com.sap.smd.agent.application.global.configuration [SID]/[INB]/sapj2ee/http/port
J2EE Message server Intern 39xx com.sap.smd.agent.application.global.configuration [SID]/sapj2ee/msgserver/port
Port
J2EE Message server Http 81xx com.sap.smd.agent.application.global.configuration [SID]/[INB]/sapj2ee/http/port
port
J2EE Message server host <Hostname> com.sap.smd.agent.application.global.configuration [SID]/sapj2ee/msgserver/host
o Credentials :
Credential Configuration stored in agelet Property user Property password

J2EE credentials com.sap.smd.agent.application.global.configuration [SID]/sapj2ee/admin/user [SID]/sapj2ee/admin/pwd
Credential Configuration stored in agelet Property Property user Property

client password
ABAP credentials com.sap.smd.agent.application.global.configuration [SID]/its/client [SID]/abap/com/user [SID]/its/client
Those values might be only changed in the agent Admin tool in case of debugging. The general recommendation is to
update those values with the Managed system setup wizard.
©SAP AG 2009
5.3 Full Qualified Name resolution issue

In most of the cases, a connection issue from the Diagnostics Agent to the Managing system or the managed reveals a
network configuration issue. You will find below a list of SAP notes which can help to solve some network resolution
problems in different part of SAP products.
SAP Note Resume

Note 611361 The use of hostnames that are too long or contain illegal characters may
Hostnames of SAP servers result in undetermined errors
Note 962955 You want to use one or more virtual host names for SAP software within an
Use of virtual TCP/IP host names SAP server landscape to hide the physical network identities for the pieces of
software from each other.
Note 773830 It is not clear which fully qualified host name (FQHN) the Internet
FQHN determination in ICM Communication Manager (ICM) is using.
Use of virtual TCP/IP host names SAP server landscape to hide the physical network identities for the pieces of
software from each other.
Use of virtual TCP/IP host names in SAP server landscape to hide the physical network identities for the pieces of
Windows software from each other.
Note 129997 Information about IP addresses lookup.
Hostname and IP address lookup
Note 1052122 You wonder if a hostname that occurs in the System Landscape Directory
Host names in SLD (SLD) is correct. In the SLD, hostnames primarily occur in technical systems
and destinations
Note 979277 In computer systems registered by the ABAP data supplier in the SLD,
Incorrect registration of computer attributes may be missing in certain cases (for example, details of the
systems in the SLD operating system).
5.4 SAP router

In some network configuration, the Diagnostics agent has to connect to the Managing system though a SAP router.
This feature will be fully supported in Solution Manager 7 Ehp2 (7.02).
In order to enable this feature in Ehp1, please contact the Diagnostics Agent support team to get information on how to
setup the agent and on which patch level you can perform the installation.
©SAP AG 2009
5.5 Self-Monitoring
During a self-monitoring setup of Solution Manager, some special configurations in the different setup wizard are
needed due to some restrictions about the local agent connectivity. All information about the setup configuration of the
“self-monitoring” is available in the following SAP Notes:
SAP Note Resume

1273028 - Introscope 8 Release BCI Agent 8.0 cannot be used for SMD self-monitoring. Please use Java
Notes agent version 7.2 for self monitoring of the Solution Manager Java stack.
917080 - SMD Auto-Monitoring : DEPRECATED, DO NOT USE ANYMORE
single user for SMD Agent setup
1292811 - Wily setup wizard is failing An error occurs when performing the setup of a wily introscope agent on
in auto-monitoring system solution manager itself (self-monitoring) although the user and password are
correct.
Error in Managed system setup wizard in Self-monitoring
End of stream is reached unexpectedly during input from Socket
Exception:
com.sap.smdagent.plugins.connectors.p4.exceptions.P4AuthorizationException: Access is
denied: Please make sure the <user> and <password> parameters are correct.; nested
exception is:
com.sap.engine.services.jndi.persistent.exceptions.NamingException: Exception during
getInitialContext operation.No server is running. [Root exception is
com.sap.engine.services.security.exceptions.BaseLoginException: Exception in creating new
RemoteLoginContext instance.]
at
com.sap.smd.agent.plugin.connectors.p4.P4ConnectionService.getP4Connection(P4ConnectionServ
ice.java:69)
...
at java.lang.Thread.run(Thread.java:838)
Caused by: com.sap.engine.services.jndi.persistent.exceptions.NamingException: Exception
during getInitialContext operation.No server is running. Root exception is
com.sap.engine.services.security.exceptions.BaseLoginException: Exception in creating new
RemoteLoginContext instance.
at
com.sap.engine.services.security.remote.login.RemoteLoginContextExt.<init>(RemoteLoginConte
xtExt.java:34)
...
at java.lang.Thread.run(Thread.java:838)
Caused by: java.io.EOFException: End of stream is reached unexpectedly during input from
Socket [addr=/56.zzz.xx.xx,port=50204,localport=22326]
at com.sap.engine.services.rmi_p4.Connection.run(Connection.java:423)
The problem is linked to the value if the host name registered in the Solution Manager SCS message server.
Check the hostname stored in SCS message server with the following url
http://scs_host:81<SCS_instance_number>/msgserver/text/logon.
The agent hostname reported in agent admin and SCS hostname of entry points must be equal.
Since 7.01 Ehp1 SPS19, add the property <SID>/selfmonitoring/enabled=true in the agelet global configuration
of the Diagnostics agent XD XD which do the self monitoring to avoid some auto-monitoring connection issue.
©SAP AG 2009
6 Agent Maintenance
6.1 Upgrade SAP Kernel (saposcol update)
For specific agent issues, you might be asked to upgrade the SAP kernel binaries of the Diagnostics agent (Ex: This is
requested when upgrading saposcol). In that case, follow the procedure:
For 7.00 and 7.01
Determine the SID and instance number of the Diagnostics agent to upgrade.
Determine the SAP Kernel version by opening a shell with <sid>adm for Unix or command prompt Windows in
folder /usr/sap/<SID_AGT>/exe
Execute the command “sapcontrol -prot NI_HTTP -nr <Instance_Number_AGT> -function
GetVersionInfo”
GetVersionInfo
OK
Filename, VersionInfo, Time
C:\usr\sap\SMD\J98\..\exe\sapstartsrv.exe, 700, patch 144, changelist 957098, optU, NTintel,
C:\usr\sap\SMD\J98\..\exe\msg_server.exe, 700, patch 144, changelist 957098, optU, NTintel,
C:\usr\sap\SMD\J98\..\exe\enserver.exe, 700, patch 144, changelist 957098, optU, NTintel,
C:\usr\sap\SMD\J98\..\exe\jcontrol.exe, 700, patch 144, changelist 957098, optU, NTintel,
C:\usr\sap\SMD\J98\..\exe\jlaunch.exe, 700, patch 144, changelist 957098, optU, NTintel,
Stop all Diagnostics agents with same SID (because they share the binaries)
Download the latest version of the SAP Kernel on service market place. Be careful if the agent kernel version is
700 you must upgrade to the latest available version of the 700 kernel. You shouldn’t upgrade a 700 kernel to
the latest version of the 701 kernel.
Extract the SAR File (ex: SAPEXE_179-20000978.SAR) in folder /usr/sap/<SID_AGT>/exe
For 711
Determine the SID and instance number of the Diagnostics agent to upgrade.
Determine the SAP Kernel version, by opening a shell with <sid>adm for Unix or command prompt Windows in
folder /usr/sap/<SID_AGT>/SYS/exe/uc/<OS>.
Execute the command “sapcontrol -prot NI_HTTP -nr <Instance_Number_AGT> -function
GetVersionInfo”
/usr/sap/DAA/SMDA96/exe/sapstartsrv, 711, patch 9, changelist 1010123, optu; sun_64,

/usr/sap/DAA/SMDA96/exe/gwrd, 711, patch 9, changelist 1010123, optu , sun_64,
/usr/sap/DAA/SMDA96/exe/icman, 711, patch 9, changelist 1010123, optu , sun_64,
/usr/sap/DAA/SMDA96/exe/jcontrol, 711, patch 9, changelist 1010123, optu , sun_64,
/usr/sap/DAA/SMDA96/exe/jlaunch, 711, patch 9, changelist 1010123, optu , sun_64,
/usr/sap/DAA/SMDA96/exe/jstart, 711, patch 9, changelist 1010123, optu , sun_64,
Stop all Diagnostics agents with same SID.

Download the latest version of the SAP Kernel 711 on service market place. (do not use SAP Kernel 710)
Extract the SAR File (ex: SAPEXE_20-20003746.SAR) in the folder
/usr/sap/<SID_AGT>/SYS/exe/uc/<OS>
6.2 Configuration
This section describes all the procedures to update the Diagnostics agent configuration elements which cannot be
changed centrally from Solution Manager.
©SAP AG 2009
6.2.1 Change Path of the JDK

If the JDK path changes after the Diagnostics agent installation, you need to update the path of JDK in the agent profile
file.
For 7.00 and 7.01
Open the profile <SID_AGT>_J<INSTANCE_NUMBER>_<hostname> in folder

/usr/sap/<SID_AGT>/sys/profile.
Update the parameter jstartup/vm/home by putting the absolute path to the JDK home.
Remove the file /usr/sap/<SID_AGT>/JXX/profile/smd.properties.vmprop.
Restart the Diagnostics agent.
For 7.11
As the Diagnostics agent 711 runs only on SAPJVM5, it is not recommended to change the path of the JDK.
6.2.2 Change JVM Parameters

If the JVM parameters need to be changed, you have to update the agent instance properties file.
For 7.00 and 7.01
Open the file smd.properties in folder /usr/sap/<SID_AGT>/J<Instance_Number>/profile.

Update the parameter “smdagent.javaParameters” (see 4.4 JVM parameters).
For 7.11
Open the file smdagent.properties in the folder

/usr/sap/<SID_AGT>/SMDA<Instance_Number>/SMDAgent.
Update the parameter “smdagent.javaParameters” (see 4.4 JVM parameters).
6.2.3 Change the virtual host name

If the Diagnostics agent host name reported in the Agent Admin tool is not the expected name, you can change it in the
agent instance profile.
For 7.00, 7.01 and 711
Open the profile <SID_AGT>_J<INSTANCE_NUMBER>_<hostname> in the folder

/usr/sap/<SID_AGT>/sys/profile.
Update or add the parameter SAPLOCALHOST by setting the short hostname desired (not the FQN).
©SAP AG 2009
6.2.4 Usage of smdsetup script

The script ’smdsetup’ allows to update some configuration parameters for Diagnostics agent. This script is delivered in
the Diagnostics Agent since Solution Manager SPS16. However, this script is always automatically patched to the
latest version when the Diagnostics agent connects to the SMD server.
So if the LMSERVICE has a version greater than SPS15, the smdsetup script of the Diagnostics agents will be updated
to the last available version.
To find out what is the version of the smdsetup script, you have to execute the command “smdsetup.sh” or
“smdsetup.bat” without any parameters
o If the command output asks 5 parameters like “smdsetup.sh $JAVA_HOME smdserverhost 53004
j2eadmin abc123 myhost”, it is a smdsetup with a version less than SPS13.
Example of smdsetup SP13 output:
SMD Agent Setup expects the following parameters:

1) Path to the Java Virtual Machine - JDK 1.4 is required
2) Host Name (or IP Address) of the SMD Server (managing SYSTEM)
Note: Make sure to prefix the host name by sapms:// when using
the message server port.
3) Message Server HTTP Port (ex: 8131) or P4 Port (ex: 50004) of the SMD Server
4) User (J2EE User member of 'Administrator' Role)
5) Password of the J2ee User
6) The so called 'SMSY Server Name' which is the unique key used
to declared the host in Solution Manager.
o If the output proposes some action as “managingconf”, the version of smdsetup version is greater than
SPS14.
Example of smdsetup SP14+ output:
>> Help Information

----------------------
Script which allows to update some diagnostics agent configuration.

smdsetup <action> <parameters>
List of actions:
* smdsetup secstorej2ee user:"<value>" pwd:"<value>"
* smdsetup secstoresld user:"<value>" pwd:"<value>"
* smdsetup sldconf hostname:"<fqn>" port:"<http port>"
[optional user:"<value>" pwd:"<value>"]
* smdsetup managingconf hostname:"sapms://<fqn>" port:"<MS httpPort>"
[optional user:"<...>" pwd:"<...>"
servername:"<...>"]
* smdsetup managingconf hostname:"<fqn>" port:"<p4 port>"
[optional user:"<value>" pwd:"<value>"
servername:"<value>"]
….
SMDSetup version <= SPS13
The smdsetup script in this version is located in /usr/sap/<SID_AGT>/J<Instance_Number>/SMDAgent.
©SAP AG 2009
In this version the script permits only to change the connection information and the credentials at the same time. It
means that all the input parameters are mandatory.
Classpath error
In case of a classpath error, you need to adapt the classpath of the smdsetup script SP12 or lower, because
in SPS13 the logging.jar has moved from the SMDAgent/lib/launcher directory to the SMDAgent/lib directory.
SMDSetup version >= SPS14
In 7.0 SR3 and 7.0 the smdsetup script in this version is located in the following folder:
o /usr/sap/<SID_AGT>/J<Instance_Number>/script
In 7.11 the smdsetup script is located in the following folder:

o /usr/sap/<SID_AGT>/SMDA<Instance_Number>/script
Action Description Availability

Change the managing system of the agent (host and port), restart
managingconf SPS14
agent after the changes.
Change the SLD server where the agent is registered (host and
sldconf SPS14
port), restart agent after the changes.
Change only the J2EE user used to connect to the Managing
secstorej2ee SPS14
system. (restart after the changes)
Change only the J2EE user used to register the agent to the SLD
secstoresld SPS14
system. (restart after the changes)
changeservername Change the server name of agent. (restart after the changes) SPS14
addsaprouter Full Implementation in Ehp2, so don’t use this command SPS18
saprouterpass Full Implementation in Ehp2, so don’t use this command SPS18
removesaprouter Full Implementation in Ehp2, so don’t use this command SPS18
Execute some agent health-checks and generate a zip containing
supportlogs SPS17 patch 3
all agents logs (including the agent check report)
MANAGINGCONF SYNTAX
To execute this command you need to stop the agent. The command updates the connection settings to the Managing
system:
Connect the agent by P4 direct connection:

smdsetup managingconf hostname:”myhost” port:”53004”
Connect the agent by J2EE Message server http port (not internal message server port 36xx):
smdsetup managingconf hostname:”sapms://myhost.domain.corp” port:”8101”
Optionally, you can add the j2ee user information in the command line to update also the agent secure store:
smdsetup managingconf hostname:”myhost.domain,corp” port:”53004” user:”SMD_ADMIN”
pwd:”init1234”
©SAP AG 2009
SLDCONF SYNTAX
To execute this command you need to stop the agent. The command updates the connection settings to the SLD
system of the agent registration:
Connect the agent to the SLD system by the http port of java stack.
smdsetup sldconf hostname:”mySLDhost.domain.corp” port:”50000”
Optionally, you can add the sld user information in the command line to update the agent secure store:
smdsetup sldconf hostname:”mySLDhost.domain.corp” port:”50000” user:”slddsuser” pwd:”xxxxx”
SECSTOREJEE SYNTAX
To execute this command you need to stop the agent. The command updates the credentials of the connection to
Managing system:
Update the agent secstore file:

smdsetup secstorej2ee user:”SMD_ADMIN” pwd:”init1234”
SECSTORESLD SYNTAX
To execute this command you need to stop the agent. The command updates the credentials of the connection to the
SLD system:
Update the agent secstore file

smdsetup secstoresld user:”slddsuser” pwd:”xxxxxx”
CHANGESERVERNAME SYNTAX
To execute this command you need to stop the agent. The command updates the “servername” attribute of the agent:
Update the agent configuration file

smdsetup changeservername servername:"myservername"
SUPPORTLOGS SYNTAX
The Diagnostics agent must be started before executing this action. This command retrieves all agent log files and
executes some basics tests to check the agent pre-requisites as:
jdk version check
jvm parameters check
hostname resolution check
tcp port check
p4 protocol check
message server connectivity check
To execute the command

smdsetup supportlogs
At the end of the execution of the command, a Zip file is generated in

/usr/sap/<AGENT_SID>/<AGENT_INSTANCE_NUMBER>/SMDAgent/supportLogs_<TIMESTAMP>.zip.
The zip content is the following:

SMD Agent log files
SMD Agent profile
SAPInst logs
Work directory
All properties files contained in the SMD agent folder.
©SAP AG 2009
6.2.5 Change Agent log level

The Diagnostics agent logs are stored in /usr/sap/<SID_AGT>/<InstNbr>/SMDAgent/log folder and organized as
follow:
Filename Default settings Default Description

level
SMDSystem.log 10 files of 1MBytes INFO This file reports all information about agent
framework like connection to managing system,
SLD report, agelets loading and patch sequence.
SMDAgentApplication.log 10 files of 1Mbytes WARNING This file reports all information about the agelets
used by RCA applications, Setup Wizards and
Wily IS setup.
P4.log 10 files of 1MBytes NONE This file reports all information about agent
connection layer with managing system.
When changing the log level of the agent logging, you need to consider two cases:
o The Diagnostic agent is connected to the Solution Manager System:
Go in Agent Administration tool, select the tab “Agent Log viewer”

Select the agent click on display button
Select the location
In section “Log Level Control”, select the level and click on the “apply” button.
o The Diagnostics agent is not connected to the Solution Manager System:
Stop the agent

Go in folder /usr/sap/<SID_AGT>/<InstNbr>/SMDAgent/configuration
Open the file log-configuration.xml
Adapt the level (attribute effective-severity) in following line (level values:
ALL,DEBUG,INFO,WARNING,ERROR,NONE):
For Filename
SMDSystem.log <log-controller effective-severity="INFO" maximum-severity="ALL" minimum-
severity="DEBUG" name="/SMDlogger/System">
SMDAgentApplication.log <log-controller effective-severity="WARNING" maximum-severity="ALL" minimum-
severity="DEBUG" name="/SMDlogger/AgentApp">
P4.log <log-controller effective-severity="NONE" maximum-severity="ALL" minimum-
severity="ALL" name="com.sap.engine.services.rmi_p4">
Save the file and restart the agent.
©SAP AG 2009
6.3 Managing J2EE user

Information about the agent j2ee user who connects to the Solution Manager System, is described in the “User
Administration guide”, section 2.3.3 [SOLMAN.AGT.COM]: Diagnostics agent Communication User.
If the J2EE user used by the Diagnostic agent cannot be authenticated (wrong password, missing role assignment,
user locked), the agent will shutdown automatically to avoid locking the corresponding account.
Most common agent connection problems
Authentication failed
all authentication problems are logged in file SMDSystem.log.
Exception: at com.sap.engine.core.thread.impl3.SingleThread.run
(SingleThread.java:170)
Caused by:
com.sap.engine.services.security.exceptions.BaseLoginException:
Authentication did not succeed.
at
com.sap.engine.services.security.login.ModulesProcessAction.run
(ModulesProcessAction.java:175)
... 13 more
Before restarting the agent It is important to execute some checks related to the J2EE user as follow:
1. Start the ABAP transaction SU01 on Solution Manager.

Check if the J2EE user (SMD_ADMIN) is not locked
Check that the Role assigned is compliant to SAP recommendations done in the “User Administration guide”.
2. Logon to the Agent administration tool with the J2EE user (SMD_ADMIN) to check that the logon module of
the Java stack works correctly.
3. Call the smdsetup to update only the credentials to be sure the password is up-to-date.
4. Restart the Diagnostic agent.
In dev_smdagent.log, if you find the exit code -11112, it is an authentication problem as described above.
[Thr 880] Wed Feb 04 09:28:47 2009

[Thr 880] JLaunchIStartFunc: end of main method (rc = 0)
[Thr 6000] JLaunchIExitJava: exit hook is called (rc = -11112)
[Thr 6000] *************************************************************
*** ERROR => The Java VM terminated with a non-zero exit code.
*** Please see SAP Note 943602 , section 'J2EE Engine exit codes'
*** for additional information and trouble shooting.
**********************************************************************
[Thr 6000] JLaunchCloseProgram: good bye (exitcode = -11112)
©SAP AG 2009
7 Tracking agent issues
7.1 Deprecated SAP notes
The following table contains a list of SAP notes which are now deprecated:
SAP Note Short description State

985140 SMD Agent/SAPinst: Virtual Host is not taken into account Deprecated
906248 Problem with missign CA directory with NW04s SP5 SMD install Deprecated
987092 Install an SMD Agent <= SP11 on IBM System i (i5/OS, OS400) Deprecated
954861 SMD agent installation with SAP Inst failed (SSL problem) Deprecated
985138 SMD Agent/SAPinst/IBM JDK: Java version wrongly detected Deprecated
1037767 SMD Agent/SAPinst: DNS Domain name error on AIX Deprecated
1055798 smdagent.sh start: java.lang.NoClassDefFoundError Deprecated
1069588 SMD Agent/SAPinst: How to use an alternate System Name Deprecated
7.2 OSS message pre-requisites

Before forwarding any OSS messages to the SAP support organization, it is important to check the following points:
The Diagnostics agent has been installed with SAPInst SR3, Ehp1 or 7.11.
The OS limit for Unix platform are compliant with the agent OS user (<AgentSID>adm),
JDK vendor and version is supported by Diagnostics agent,
JVM parameters of Diagnostics agent are compliant,
The Diagnostics agent connection to the Managing system is of type J2EE Message Server connection,
Recommendations for the auto-monitoring scenario Managed setup wizard execution have been applied.
The Java dispatcher of Solution Manager is correctly sized to avoid some agent disconnection (see part 8.1
Managing java stack settings.)
If those check points cannot solve the problem, follow the SAP Note 1076573 to create the OSS message. If the
LMService version is greater or equal to NW04S SPS17 Patch 3, attach the support logs Zip file generated by
the smdsetup script (see part 6.2.4).
©SAP AG 2009
7.3 Agent crash investigation

When the system process of the Diagnostics agent shutdowns abnormally, the procedure to determine the cause of
this crash is the following:
Go in folder /usr/sap/<SID_AGT>/J<INSTANCE_NUMBER>/work
Open the log file dev_jcontrol (This is the log file of the jcontrol process which is the controller of the
java agent process) to check if the jcontrol has started the jlaunch process as described below:
[Thr 3288] JControlICheckProcessList: process SMDAgent started (PID:3616)

If the process jlaunch cannot be start, the error is probably located in SAP profile parameters
The message JStartupICheckFrameworkPackage: can't find framework package

C:\usr\sap\SMD\exe\jvmx.jar is NOT a critical error. The Diagnostics agent does not use specific SAP
kernel functions and this message can be ignored.
If the jlaunch process has been started, open the file dev_SMDAgent which corresponds to the jlaunch
process output :
Thr 3860] JLaunchISetState: change state from [Initial (0)] to [Initial (0)]
[Thr 3860] JLaunchISetState: change state from [Initial (0)] to [Waiting for start (1)]
[Thr 3860] Thu Dec 25 12:11:35 2008

[Thr 3860] JLaunchISetState: change state from [Waiting for start (1)] to [Starting (2)]
[Thr 3860] JLaunchISetState: change state from [Starting (2)] to [Running (3)]
In this file, all information about the java of the Diagnostics agent is stored. You will information related to
JVM loading, process state, crash of VM or bad JVM parameters. In case of process crash, you will find in
this file, the corresponding Exit Code or Error message.
JVM Loading problem
all JVM loading problems are logged in file dev_SMDAgent.
Error: [Thr 1084238144] *** ERROR => DlLoadLib: dlopen()= libjvm.so: cannot open shared
object file: No such file or directory -> DLENOACCESS (0, Success) [dlux_mt.c 314]
[Thr 1084238144] *** ERROR => Can't load VM shared library (libjvm.so) (rc=-2)
[jhvmxx_mt.c 1576]
[Thr 1084238144] *** ERROR => Cannot load DLL for Java VM [jlnchxxi_mt.717]
You will find potential JDK installation issues like :

Wrong JDK version for Agent
JDK is not compatible with SAP Kernel (32bits/64bits)
Agent OS user have not rights to access to JDK Libraries
The SAP Note 940893 - Analysis of errors within the startup of J2EE Engine 6.40 can help to troubleshoot
JVM loading problems.
If the JVM is loaded correctly, now check the file std_SMDAgent.out and jvm_SMDAgent.out to find some
errors like OutOfMemory, StackOverflow or Nullpointer which could have stopped the agent process.
©SAP AG 2009
7.4 Thread dump

Facing a deadlock situation or a Diagnostic agent freeze, the only correct way to understand the cause of the problem
is to perform a thread dump on the agent process.
Make a Thread dump under Windows
The Diagnostics agent is like any other SAP system, available in the SAP MMC.
In SAP MMC, open the node “AS Java Process Table” of the Diagnostics agent identified by the SID and the instance
number. Only one process is displayed in the right part: It is the agent process.
Select this process and right click on it to display the context menu, press the item “Dump stack Trace”.
Make a Thread dump under Unix
Under UNIX platform, you need to use the command smdadmin.sh.
©SAP AG 2009
Start in an UNIX shell with the OS user of the Diagnostics agent. Execute the command “smdadmin.sh XX” (XX is the
instance number of Diagnostics agent).
For old agent installation, it is necessary to execute the command “../smdadmin.sh” from the folder
/usr/sap/<SID_AGT>/J<InstanceNumber>/script.
Once the command is executed, the JCMON menu is displaying, select the entry “20: Local administration menu” then
execute the action “10: Dump stacktrace” on process 0 (This is the process of the SMDAgent).
Where is the thread dump stored?
Once the thread dump is done, the output is written in file /usr/sap/<SID_AGT>/J<InstanceNumber>/work/
std_SMDAgent.out for Sun and HP-UNIX JDK.
For IBM JDK, the thread dump output is written in java.core.<timestamp>.txt in folder
usr/sap/<SID_AGT>/J<InstanceNumber>/SMDAgent.To find the absolute path of an IBM thread dump, open the file
std_SMDAgent.out and find the line beginning with “Java Dump written to”.
7.5 Activate P4 logs

In case of a persistent p4 connection problem with the managing system or the managed system (like timeout,
connection reset, ..) , you should enable the P4 logs before sending the OSS message to the Development Support
organization.
Connection problem to Managing System or Managed system
The procedure to enable the P4 logs on the Diagnostics agent is the following:
Stop the agent

Clean the log folder under SMDAgent directory
Set the log level (effective-severity) of SMDSystem location to ALL (see 6.2.5 Change Agent log level)
Set the log level (effective-severity) of P4 location to ALL (see 6.2.5 Change Agent log level)
Add the Java parameter –Ddebug=true in agent configuration (6.2.2 Change JVM Parameters)
Once the p4 log enabled on agent, go to Managing system or Managed system to execute the following instructions:
Open the Visual Administrator of Java stack

Select the server node Under SID node
Select the service “Log Configurator”
Go in tab Locations
Select the location com.sap.engine.services.rmi_p4
Increase the log level to DEBUG and save the modification.
Once all P4 locations are configured, restart the agent to reproduce the problem and attach the agent logs (use the
command ‘smdsetup supportlogs’) and the current defaulttrace.log of the Managing or the Managed system to the OSS
message.
Performance Impact
It is important to restore All log locations changed to their default level because the procedure above has a
severe impact on the performance of the Managed system or the Managing system.
©SAP AG 2009
7.6 Telnet SMD server commands

Some telnet commands are available in the Solution Manager Java stack to check the installation and configuration of
the SMD server.
To use those commands, start a telnet client to the SAP J2EE server of Solution Manager (default port 5xx18) and go
on server node with the command ‘jump <nodeid>’. Add the group smdserver to enable the SMD commands by typing
use the command ‘add smdserver’.
Once the group smdserver is added, execute the command ’man –g smdserver’ to see all available SMD server
commands.
SMD server telnet command Description

Smdstatus Display the version and running mode of SMD server.
Smdstatus –a Check connectivity of all agents
list_agents Display all diagnostics agents connected to SMD server with their status.
find_agent <servername> or <fqn> Retrieve an agent from name, fully qualified domain name or IP.
mm ON Enable the maintenance mode of SMD server
mm OFF Disable the maintenance mode of SMD server
kill_agent <servername> or <fqn> Reboot the Diagnostics agent process
start_agent <servername> or <fqn> Start the agent applications of selected Diagnostics agent
stop_agent <servername> or <fqn> Stop the agent applications of selected Diagnostics agent
host_mapping Display the mapping between server name and network names for all
agents.
known_agents -c Displays the list of known agents. (Agents that have been registered at least
once to the SMD Server)
known_agents -n Displays the list of known agents by FQN.
as Provides ping statistics for all agents.
es Provide statistics usage for the SMD server.
8 Scalability
8.1 Managing java stack settings

We recommend adapting the J2EE configuration of Solution Manager Java stack if an important number of Diagnostics
agents will connect to Solution Manager.
The Java dispatcher has an important role in the agent connection’s stability. Therefore, you need to adapt the
maximum number of threads in the pool and in the heap size configuration of the dispatcher to avoid some timeout
connection issues.
Dispatcher Parameter Recommended Value

Java Max Heap Size (-Xmx) Number of agents installed * 1.28 (minimum 128MB)
ThreadManager->MaxThreadCount ~ Number of agents installed * 1.75
Default value of MaxThreadCount
After an installation of Solution Manager, the default value of parameter MaxThreadCount in Java Dispatcher
configuration is set to 70. These settings allow by default to connect a maximum of 40 agents.
©SAP AG 2009

DiagAgent Troubleshooting

Cargado por

Información del documento

Descripción original:

Derechos de autor

Formatos disponibles

Compartir este documento

Compartir o incrustar documentos

Opciones para compartir

¿Le pareció útil este documento?

¿Este contenido es inapropiado?

Copyright:

Formatos disponibles

DiagAgent Troubleshooting

Cargado por

Copyright:

Formatos disponibles

SAP Solution Manager 7.

Typographic Conventions Icons

This guide covers the following area:

OS Diagnostics Agent 7.00 / 7.01 Diagnostics Agent 7.11

2.2 JDK compatibility

OS Diagnostics Agent 7.00 / 7.01 Diagnostics Agent 7.11

Note 716604 for the Sun JDK (Windows, Linux, Solaris)

Most common JDK problems with agent

This error is logged in SMDSystem.log

Exception: Time out occurred when calling method 'configure' on object

Agent applications are not updated

This error is still logged in SMDSystem.log

Exception: [Thread[Thread-5,5,main]] Error

2.3 Managed System

2.4 SLD production system

How to determine which version of the CIM Model is installed?

How to upgrade the CIM Model?

How to check if the Diagnostics agent is registered in SLD?

What is the SLD registration frequency of the Diagnostics Agents?

Why the Diagnostics agent is not visible in Agent Admin tool?

3 Agent installation strategy

3.1 Standard environment

3.2 HA/virtualization environment

Microsoft Cluster (MSCS)

2 highly available control nodes (without SAP software)

Other virtualization solutions

4.1 SAPinst Documentation

SAP Note 1234387: Inst. 7.1 EhP1 - Diagnostics Agent installation

SAP Note 1294713: Mass Deployment of SAP Diagnostics Agent 7.11.

4.2 Troubleshooting with SAPinst

In that case, the procedure is the following:

a. To view the log file, choose View Logs.

4.3 OS configuration checks

Check the Diagnostics agent OS user limit :

Check File System access right:

4.4 JVM parameters

JDK JVM Parameters SAP Note

4.5 Startup scripts

On Solaris platform, an error (/usr/sap/SMD/exe/smdstart.sh: usage: ulimit [ -HSacdfnstv ] [ limit ])

Exit code of smdstart.sh Reason

su -c /usr/sap/SMD/J98/script/smdstart.sh > /dev/null 2>&1 -s /bin/sh - smdadm

su -c /usr/sap/SMD/J98/script/smdstop.sh > /dev/null 2>&1 -s /bin/sh - smdadm

Create a file diagnostics_agents in folder /etc/init.d

Information about Diagnostics agent 7.11 or higher

About sapstartsrv process

SAP Note Description

5.1 Managing system

Connection through the SAPJ2EE Message server

The agent connection mechanism can be described plainly:

Troubleshooting Message server connection

1. Open the URL: http://msgserver:81xx/msgserver/text/logon (xx corresponding to instance number

Most common agent connection problems

No Entry point found

Exception: [sapms:// yupxxxxx.domain,com:8136] Checking server availability...

Execute the command “ping <hostname_of_P4_entry_point_5xx04>” in a Managed system shell.

Direct Connection to P4 Dispatcher

Solution Manager in cluster environment

5.2 Managed system

The agent connects to the following different SAP System layers:

Java stack on port P4 (5xx04)

Troubleshooting P4 connection problems with Managed system

Check on which host the P4 socket is bound

TCP 0.0.0.0:59304 0.0.0.0:0 LISTENING