Performance Analyzer visual snapshots when monitoring ITCAM Agents..

In this blog - I will report some of the visuals of the Performance Analyzer when monitoring the ITCAM agents.



Some of the TEP GUI rendering when PA is running and their explanations are captured ?














This snapshot below states  the historical tables are not created.

RRT_Transaction_Status H M and W Y etc etc. are not historically created by the SY and the WPA processes.





Here - the history is configured. but the tasks are failing - with the above message,
This is due to the fact that the tables are all empty.
RRT_LT_Fcast and also RRT_transaction-status etc etc. the history tables are not created.



are the only ones.

The error message in the log file is

2013-10-24 09:13:55: Java exception : com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704, SQLERRMC=ITMUSER.RRT_Transaction_Status_HV, DRIVER=3.62.56
2013-10-24 09:13:55: Stack trace : com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704, SQLERRMC=ITMUSER.RRT_Transaction_Status_HV, DRIVER=3.62.56
        at com.ibm.db2.jcc.am.fd.a(fd.java:676)
        at com.ibm.db2.jcc.am.fd.a(fd.java:60)
        at com.ibm.db2.jcc.am.fd.a(fd.java:127)
        at com.ibm.db2.jcc.am.jn.c(jn.java:2614)
        at com.ibm.db2.jcc.am.jn.d(jn.java:2602)
        at com.ibm.db2.jcc.am.jn.a(jn.java:2094)
        at com.ibm.db2.jcc.t4.cb.g(cb.java:141)
        at com.ibm.db2.jcc.t4.cb.a(cb.java:41)
        at com.ibm.db2.jcc.t4.q.a(q.java:32)
        at com.ibm.db2.jcc.t4.rb.i(rb.java:135)
        at com.ibm.db2.jcc.am.jn.gb(jn.java:2064)
        at com.ibm.db2.jcc.am.jn.a(jn.java:3089)
        at com.ibm.db2.jcc.am.jn.e(jn.java:1044)
        at com.ibm.db2.jcc.am.jn.execute(jn.java:1028)
        at com.tivoli.kpa.kpaxjdbc.execDirect(Unknown Source)
2013-10-24 09:13:55: Query failed:
2013-10-24 09:13:55: SELECT  "RRT_Transaction_Status_HV"."AVG_Average_Response_Time" , "RRT_Transaction_Status_HV"."Application" , "RRT_Transaction_Status_HV"."Client" , "RRT_Transaction_Status_HV"."Origin_Node" , "RRT_Transaction_Status_HV"."Server" , "RRT_Transaction_Status_HV"."Transaction" , "RRT_Transaction_Status_HV"."Server" , "RRT_Transaction_Status_HV"."Transaction" , "RRT_Transaction_Status_HV"."Client" , "RRT_Transaction_Status_HV"."Application" , "RRT_Transaction_Status_HV"."Origin_Node" , "RRT_Transaction_Status_HV".LAT_WRITETIME , "RRT_Transaction_Status_HV"."TMZDIFF"  FROM itmuser."RRT_Transaction_Status_HV" "RRT_Transaction_Status_HV" WHERE "RRT_Transaction_Status_HV"."Origin_Node" IN ( 'nc184132:T6') ORDER BY  "RRT_Transaction_Status_HV"."Client"  ,  "RRT_Transaction_Status_HV"."Origin_Node"  ,  "RRT_Transaction_Status_HV"."Server"  ,  "RRT_Transaction_Status_HV"."Transaction"  ,  "RRT_Transaction_Status_HV"."Application"  , "RRT_Transaction_Status_HV".LAT_WRITETIME
2013-10-24 09:13:55: Database SQL query failed with error: com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704, SQLERRMC=ITMUSER.RRT_Transaction_Status_HV, DRIVER=3.62.56

- Is the SPA and WPA processes are running ? If not start it. They are not running possibly why the messages in the kpacma.log is saying that .

- Is the t6 ( RRT ) agent running ?
- Is the task interval within reasonable limits - Can I restart the PA and see that I force a re-run of the task.


--

Does this mean that the PA  files like,   RRT_LT_Fcast and RRT_LT_Status is not created.

The RRT_Transaction_Status is there, but the PA related tables are not created.




how to resolve it,
close the browser or restarting the agent will not fix it.




----
Next.

I added the  managed systems for " RRT % Available."

no change.

Open the Historical configuration  and add the server underneath ?





This below means that the historical collections has the servers under the Available systems above - moved to the left, but there are insufficient data.








This just means that the historical collections has been started, but not enough data points ?
But here I have 57 measurements - so this "Not sufficient data-points " should not have showed up.
Diagnosed this to a sleeping PA process which had no swapspace and seemed like a run-away due to resource crunch on the RHEL server.

At this point , the itmcmd agent stop pa also did not work.
there was no free space either.
so I had to kill the process , restart the process and then I was able to rectify it.




After restarting the PA process. The Not -sufficient data points message went away.

The not sufficient data points should be only if less than 2 measurements.






If you see this message when bringing up the TEPS GUI - then it may mean the database is not started.





Sometimes, it might require restarting the TEMS, TEPS and reconfigure if this error continues.



----

How to resolve this  error below .
The test connection works - but the TEP client is showing an error.




If you look at the kpacma.log there are instances where the log indicates that the PA agent is unable to connect to the TEPS database.

To resolve
Check

su - db2 inst1


db2 => connect to warehous

   Database Connection Information

 Database server        = DB2/LINUX 9.7.4
 SQL authorization ID   = DB2INST1
 Local database alias   = WAREHOUS

db2 => list active databases

                           Active Databases

Database name                              = TEPS
Applications connected currently           = 3
Database path                              = /home/db2inst1/db2inst1/NODE0000/SQL00002/

Database name                              = WAREHOUS
Applications connected currently           = 1
Database path                              = /home/db2inst1/db2inst1/NODE0000/SQL00001/

db2 =>


In the logs ,  you will see that the connection failed - so fix /create the TEPS database first

2013-10-23 09:55:46: Evaluation total time 0 s
2013-10-23 09:55:46: Number of output resources is 0
2013-10-23 09:55:46: Waiting for a period of 60000 ms
2013-10-23 09:56:46: Entering task controller execution loop
2013-10-23 09:56:46: Failed to connect to TEPS database
2013-10-23 09:56:46: Evaluation total time 0 s
2013-10-23 09:56:46: Number of output resources is 0
2013-10-23 09:56:46: Waiting for a period of 60000 ms
2013-10-23 09:57:46: Entering task controller execution loop
2013-10-23 09:57:46: Failed to connect to TEPS database
2013-10-23 09:57:46: Evaluation total time 0 s
2013-10-23 09:57:46: Number of output resources is 0
2013-10-23 09:57:46: Waiting for a period of 60000 ms
2013-10-23 09:58:46: Entering task controller execution loop
2013-10-23 09:58:46: Failed to connect to TEPS database



Next, check that the cq agent created the TEPS database during the configuration.

Next, check that you are using the db2jcc.jar and the db2jcc_license.jar during the PA configuration - to resolve it.
Adding more jar files may also cause it.
See the kpacma.log where it exactly tells the jar files used for configuration.

2013-10-24 04:11:54: Loading tasksimp
2013-10-24 04:11:54: tasksimp is build 120551 (Feb 24 2012)
2013-10-24 04:11:54: TaskImporter starting
2013-10-24 04:11:54: TaskImporter finished
2013-10-24 04:11:54: Loaded tasksimp successfully
2013-10-24 04:11:54: Agent running - 6 subagents active
2013-10-24 04:11:54: TImpMainTask: KPA_JAVA_HOME is /opt/IBM/ITM/JRE/li6263
2013-10-24 04:11:54: TImpMainTask: CANDLEHOME is /opt/IBM/ITM
2013-10-24 04:11:54: TImpMainTask: config dir /opt/IBM/ITM/li6263/pa/config
2013-10-24 04:11:54: TImpMainTask: root dir /opt/IBM/ITM/li6263/pa/bin/
2013-10-24 04:11:54: TImpMainTask: todeploy dir /opt/IBM/ITM/li6263/pa/config/todeploy
2013-10-24 04:11:54: TImpMainTask: deployed dir /opt/IBM/ITM/li6263/pa/config/deployed
2013-10-24 04:11:54: Using jvm classpath /opt/IBM/ITM/li6263/pa/bin/kpaxjdbc.jar:/opt/ibm/db2/V9.7/java/db2jcc.jar
2013-10-24 04:11:54: TImpMainTask: Trying file ..
2013-10-24 04:11:54: TImpMainTask: Trying file .
2013-10-24 04:11:54: GenIraAgent::TakeSample(WRT_LT_Status) returns 0 rows in 0 s


----