Dear Team:
This is the updated KB on this topic
DST issue with APM 9.5.0, 9.5.1,
9.5.2 and Oracle DB – Partition
Creation Failed Message received after Daylight Savings Time change ORA-14074
Description:
While running APM 9.5.0, 9.5.1 or 9.5.2 with an Oracle APM Database, the following (or
similar) errors in the APM CE GUI And in TIM Collection Service EM Logs may occur when transitioning to/from Daylight Savings Time.
- Partition creation failed
- Administration Enterprise Manager 0.0.0.0 Partition creation failed.
- Disk or oracle table space probably exhausted.
Manager.com.timestock.tess.util.DbUtils] Execute failed for SQL DDL 'ALTER TABLE ts_us_sessions_map ADD PARTITION ts_us_sessions_map_20140309_03 VALUES
LESS THAN (TIMESTAMP '2014-03-09 4:00:00 -4:00')'
java.sql.SQLException: ORA-14074: partition bound must collate higher than that of the last partition"
This indicates that last table partitions for table ‘ts_us_sessions_map’ overlaps with the new partition and prevents its creation.
A Technical Advisory with more details is available at https://support.ca.com/phpdocs/0/5974/5974_critical_alert_DST_Issue_032014.pdf.
Impact:
Most of CEM processes stop and CEM is not reporting data (no new data in CEM UI).
Solution #1:
- Make a backup of the APMDB (Important!)
Drop the latest/newest TS_US_SESSIONS_MAP partition which is causing the overlap. (execute this while the EM with Tim
Collection Service [TCS] is running):
ALTER TABLE TS_US_SESSIONS_MAP DROP PARTITION TS_US_SESSIONS_MAP_201xxxxx_xx UPDATE GLOBAL INDEXES;
Replace “TS_US_SESSIONS_MAP_201xxxxx_xx” with the name of the latest table partition name.
2. If "duplicate partition name" error occurs after completing step #2 (this usually happens when DST ends and time moves one hour backwards) then repeat step #2
again and drop latest TS_US_SESSIONS_MAP partition again. Execute this again while the EM with Tim Collection Service [TCS] is running.
3. Restart EM running Tim Collection Service. (This step is
usually not necessary, however there are some cases when restart is needed - for
example when this issue lasts for more than 2 days).
4. Upload the TCS and Aggregation Service EM logs to CA Support (support.ca.com) a couple of hours after completing steps above for
analysis
Solution #2:
- Make a backup of the APMDB (Important!)
- Stop EM running Tim Collection Service
- Drop all TS_US_SESSIONS_MAP partition except TS_US_SESSIONS_MAP_TEMP
ALTER TABLE TS_US_SESSIONS_MAP DROP PARTITION TS_US_SESSIONS_MAP_201xxxxx_xx UPDATE GLOBAL INDEXES;
Replace “TS_US_SESSIONS_MAP_201xxxxx_xx” with name of the table partition name. Repeat it for ALL existing partitions except TS_US_SESSIONS_MAP_TEMP.
- Start EM running Tim Collection Service
- Upload the TCS and Aggregation EM logs to CA Support (support.ca.com) a couple of hours after completing the steps above for analysis.
Note:
If solutions above don’t resolve the issue or if this issue occurs outside of DST time shift window, you may be
experiencing a different issue and should check the EM logs for other messages and
errors.