DX Application Performance Management

 View Only
  • 1.  Enterprise Manager cannot reach TIM

    Posted Nov 11, 2013 04:55 PM

    Hi everyone:

    Sometimes I get the following messages in the CEM: ''Error retrieving URL http://172.29.104.39/wily/cem/tim/mod_python/timfiles/listEvents: connect timed out'' and ''Error retrieving response http://10.80.200.116/wily/cem/tim/mod_python/timfiles/listBTstats: Read timed out''. The EM's is fine, what is the cause?

     



  • 2.  RE: Enterprise Manager cannot reach TIM
    Best Answer

    Broadcom Employee
    Posted Nov 11, 2013 05:00 PM
    adan_85:

    Hi everyone:

    Sometimes I get the following messages in the CEM: ''Error retrieving URL http://172.29.104.39/wily/cem/tim/mod_python/timfiles/listEvents: connect timed out'' and ''Error retrieving response http://10.80.200.116/wily/cem/tim/mod_python/timfiles/listBTstats: Read timed out''. The EM's is fine, what is the cause?

     


    Hi Adan,

    If this is only an intermittent problem it is likely to be a transient Network related problem rather than configuration related.

    I have seen such intermittent errors reported (a few per day) when the EM and/or TIM are running under a virtual environment & there was no resolution aisde from just ignoring the messages.

    If you think this may be more "sinister" and need a more extensive review please raise an issue with us.

    Thanks,

    Lynn

    CA APM Support
     

     

     



  • 3.  RE: Enterprise Manager cannot reach TIM

    Posted Nov 11, 2013 06:14 PM

    Thank you a lot Lynn.

    Adan.



  • 4.  RE: Enterprise Manager cannot reach TIM

    Posted 22 days ago
    Hi Lynn,

    I am also getting the same error and our entire 7 servers which include one MOM one agent collector, one TIM collector and four TIM servers are on physical environment.
    why this error is coming in logs in CEM?
    https://knowledge.broadcom.com/external/article/30192/em-unable-to-process-defects-seeing-http.html -
    gone through this article too but unable to understand in point three where do we have to act, I mean out of seven servers where we need to troubleshoot

    Shiv


    ------------------------------
    Shiv Choudhary
    Tata Consultancy Services
    India
    ------------------------------



  • 5.  RE: Enterprise Manager cannot reach TIM

    Broadcom Employee
    Posted 22 days ago
    Edited by Jörg Mertin 22 days ago

    Hi Shiv,

    you have to go onto the TIM - and check (depending if you are running a plain TIM or a MTP) the mentioned directories for remaining files.

    Maybe you want to du a "du - /etc/wily/cem/tim/data/out/defects" (or replace with correct directory) just to see how much data is inside first.
    If you want, we can do WebEx to check it out together (Note that I am in Europe/Paris timezone). PM me to schedule it.




  • 6.  RE: Enterprise Manager cannot reach TIM

    Posted 14 days ago
    Edited by Shiv Choudhary 14 days ago
    Hi Jorg ,
    Thanks for your reply .
    I have a doubt in my mind that what exactly is under the defect folder in worker XX folder.. because when checked I found only large number of defects. timestamp. xml files and no other files.
    So I want to know can we delete worker XX folder contents? Will this cause any problems later on data collected or on any metrics data ?


    ------------------------------
    Shiv Choudhary
    India
    ------------------------------



  • 7.  RE: Enterprise Manager cannot reach TIM

    Broadcom Employee
    Posted 14 days ago
    Hi Shiv,

    the way the TIM works, is as follows.
    1. The TIM processes data based on a "required" base only, and will store detailed data only on identified defects.
      The Way TIM discards data is pretty efficient as Monitoring transactions is very resource intensive. Hence, discard as fast as possible it the general rule. When the data to be analyzed reaches the TIM, the following levels are passed (Assuming that if a decryption needed to take place, if was successfully handled):

      1. WebServer filters: If a filter is set, only packets matching the IP/IP-Range are taken into account. All others are discarded.
        Note that if there are no webServer filters set, the TIM will look at all traffic! This very bad practice. Avoid it!
      2. Transaction definitions: The TIM will now look for a match on the existing transaction definitions. If no match is found, the transaction/data is discarded.
      3. Private parameters: The TIM will by default anonymize all variables that are configured as Private parameters, replacing their content by ****.
      4. Transaction analysis: The TIM will now perform the analysis of the data and compute various parameters and identify if all computed parameters are well within range.
        2 possible outcomes exist:

        • All computed parameters within range: Data is Ok, transaction is OK, statistical data is stored, all the transaction data used to compute this information is discarded
        • At least one of the computed parameters identifies this transaction to be defective. TIM will store (depending on the configuration) the defective transaction details and mark the transaction to be defective, also compute statistics to on that transaction.

      5. TIM will store the statistics into the data/out/stats/Statistics file.
      6. TIM will store the defect information on a per transaction base to data/out/defects
      7. The Collector the TIM is connected to will collect the defects on a regular poll interval of 7 to 15 seconds. The Statistics will be downloaded once an hour only
      8. Once the collector has successfully downloaded a file from the TIM data/out directory, this file will be deleted from the TIM.

      On the CEM/TESS system, the regular configuration for data retention apply.

    If you now check the files under ./data/out - the timestamps will tell you from when (date and time) these files are. You can check if these files are old, and if they are older than 5minutes (defects) to the collector time, you can delete them. The collector will never collect them. For the statistics files,  after a day of age they will also not be collected.
    These files will remain on the file system, and will never be removed (unless manually by you).

    If by inadvertence you remove these files for a current active run, the data will just be missing on the collector/TESS UI for that time period of the removed file. So it is important to remove the old files only. So the only impact will be the missing metrics or defects, depending on which data-types you remove.




  • 8.  RE: Enterprise Manager cannot reach TIM

    Posted 5 days ago
    Edited by Shiv Choudhary 4 days ago

    Hi Jorg,

    We were facing error in one of our TIM server as the Worker XX folder under defects directory consuming lots of space . we have removed the files from the worker & noticed that our MOM is heavily loaded with defects data & the Database size is increasing at an alarming rate .
    huge number of defects data started creating in DB and we are running out of space due to this. I have observed database size increase by 10GB in a day.

    Please help me as i can't find any exact solution from Broadcom too . For this case 32989022 is also raised. PFB Screenshot for more clarity .




    ------------------------------
    Shiv Choudhary
    India
    ------------------------------



  • 9.  RE: Enterprise Manager cannot reach TIM

    Broadcom Employee
    Posted 3 days ago

    Hi Shiv,
    this is probably happening because of what we call a Defect Storm.

    Now that you have eliminated the condition where the collector wasn't able to collect the defects, it is now processing all that he got (also the old one) and tries to aggregate them all at the same time.

    You have 2 possibilities:

    1. Wait until the collector calms down and is able to also age out old defects. The data retention settings will control this over time. But you will need to have space for the database in the meantime.
    2. stop the collector and the TIM. Delete all defects on the TIM (Do not touch the statistics files themselves) that are in the out-queue. Clean up the collector directories (where it stores the defects to process) then restart the TIM and the collector.

    Regards

    Jörg




  • 10.  Moving thread.

    Posted Nov 19, 2013 05:30 PM
    Moving this thread to the General Discussion where questions are asked and answered. Mary