DX Unified Infrastructure Management

  • Private Community

  • 1.  maintenance routines that run at 00:40 every day inquiry

    Posted Jul 23, 2009 01:05 AM
    I'm curious to what occurs during the daily data management routines that are run by the data_engine every day at 00:40:00 by default.  We have version 7.02 of the data_engine probe.  We are starting to have an issue with the data_engine queue getting fairly large every morning because of the data maintenance routines.  The main complaint is that some business customers are trying to access reports when they come in at 8AM and the reports are not accurate because there is still 800MB's of data in the data_engine queue.  Until the data in the queue has been written to the database the reports don't show up accurately.  It's definitely an issue now as everyday the data_engine queue is backed up till around 7AM to 9AM.  The maintenane routines apparently are taking up to 7 hours to complete?


  • 2.  maintenance routines that run at 00:40 every day inquiry

    Posted Jul 23, 2009 08:18 AM
    We have struggled with the same issue and took to restarting the data_engine six days a week to prevent the maintenance from running more than once a week.

    The maintenance routines normally do the following:
    • Compress data from the raw (RN) data tables to the historic (HN) data tables
    • Purge old data from the raw data tables
    • Purge old data from the historic data tables
    The thresholds that determine which data is compressed and purged are set in the data_engine GUI.

    What I find especially interesting about the historic data tables is that they are never used.  The report_engine uses only the raw data, so there is currently no reason to allow compression to happen (as far as I know).  There was little that could be done about this in the past, but with data_engine 7.02 you should be able to disable compression by setting compress_raw_samples to 0.

    If you disable compression, the purging of historic data should go much faster as well.  If you have data in the historic tables now, you can probably delete it unless someone else knows of a reason to keep it around.

    Nimsoft has put quite a bit of work into the data_engine recently to address these issues.  In addition to the option of disabling compression and purges in 7.02, data_engine 7.31 is supposed to have the ability to split up the maintenance into multiple jobs that run on different nights.  I am not completely sure how this works yet; I have not had an opportunity to test it.  I think those changes should help a lot, but they are also very big and will need to be tested thoroughly.

    These database issues can be challenging, but make sure you carefully choose settings that are right for your environment.  I hope some of this information is helpful; let me know if you have questions.  Keep us posted on what you do to address this issue and how it works for you.

    Good luck,
    Keith


  • 3.  maintenance routines that run at 00:40 every day inquiry

    Posted Jul 23, 2009 10:16 PM
    I had one of our DBA's look at the database and he reindexed all the tables and this morning when I came in I could definitely tell that things were running smoother.  I notice looking at the probe documentation there is a screenshot of the 7.20 data_enginer which isn't GA yet.  This has a check box for reindexing.  I'm hoping that may help.


  • 4.  maintenance routines that run at 00:40 every day inquiry

    Posted Jul 24, 2009 12:34 AM
    Good point!  We have found that we need to reindex the tables every so often to keep our database performance at an acceptable level.  Of course, this operation can affect performance as well, so it is a balancing act.

    You might still want to check on the historic data and compression routine, since those things could still be impacting your performance and giving you no benefit.

    -Keith