After Upgrade to 14.3 - O365 account Explore and correlate failing

[Deactivated User]

After Upgrade from 14.1-> 14.3 - O365 account Explore and correlate failing

After vApp and all IM components 14.3 upgrade,  we are trying to explore and correlate o365 account of a user but its is failing with below logs in provisioning server logs after enabling full debugging.

===============================================================
20201204:174429:TID=3fcb70:Explore :E102:----:S: External Explore (eTDYNAccountName=usavarremn) Requested by User etaadmin - TenantN
20201204:174429:TID=3fcb70:Explore :E102:----:S:+otSet
20201204:174429:TID=3fcb70:Explore :E102:----:P: Class Name: User
20201204:174429:TID=3fcb70:Explore :E102:----:P: base-dn: eTDYNAccountName=usavarremn,eTDYNAccountContainerName=Accounts,eTDYNDi
20201204:174429:TID=3fcb70:Explore :E102:----:P:+ rectoryName=Office365_EPT,eTNamespaceName=Office 365,dc=im
20201204:174429:TID=3fcb70:Explore :E102:----:P: scope : SUBTREE
20201204:174429:TID=3fcb70:Explore :E102:----:P: filter : (objectClass=*)
20201204:174429:TID=3fcb70:Explore :E102:----:P: attrs : eTExploreUpdateEtrust, objectClass, eTSubclass
20201204:174429:TID=3fcb70:Search :D103:E102:S: DB Search (eTDYNAccountContainerName=Accounts) Requested by User etaadmin - Tenan
20201204:174429:TID=3fcb70:Search :D103:E102:S:+tNotSet
20201204:174429:TID=3fcb70:Search :D103:E102:P: URL: ldaps://ca-prov-srv:20391
20201204:174429:TID=3fcb70:Search :D103:E102:P: Class Name: Accounts
20201204:174429:TID=3fcb70:Search :D103:E102:P: base-dn: eTDYNAccountContainerName=Accounts,eTDYNDirectoryName=Office365_EPT,
20201204:174429:TID=3fcb70:Search :D103:E102:P:+ eTNamespaceName=Office 365,dc=im
20201204:174429:TID=3fcb70:Search :D103:E102:P: scope : ONE-LEVEL
20201204:174429:TID=3fcb70:Search :D103:E102:P: filter : (eTDYNAccountName=usavarremn)
20201204:174429:TID=3fcb70:Search :D103:E102:P: attrs : objectClass
20201204:174429:TID=3fcb70:Search :D103:E102:F: SUCCESS: DB Search (eTDYNAccountContainerName=Accounts), entry-count: 0
20201204:174429:TID=3fcb70:Search :D104:E102:S: DB Search (eTDYNDirectoryName=Office365_EPT) Requested by User etaadmin - TenantN
20201204:174429:TID=3fcb70:Search :D104:E102:S:+otSet
20201204:174429:TID=3fcb70:Search :D104:E102:P: URL: ldaps://ca-prov-srv:20391
20201204:174429:TID=3fcb70:Search :D104:E102:P: Class Name: Endpoint
20201204:174429:TID=3fcb70:Search :D104:E102:P: base-dn: eTDYNDirectoryName=Office365_EPT,eTNamespaceName=Office 365,dc=im
20201204:174429:TID=3fcb70:Search :D104:E102:P: scope : ONE-LEVEL
20201204:174429:TID=3fcb70:Search :D104:E102:P: filter : (eTDYNAccountContainerName=Accounts)
20201204:174429:TID=3fcb70:Search :D104:E102:P: attrs : objectClass
20201204:174429:TID=3fcb70:Search :D104:E102:F: SUCCESS: DB Search (eTDYNDirectoryName=Office365_EPT), entry-count: 1, attributes
20201204:174429:TID=3fcb70:Search :D104:E102:F:+: objectClass
20201204:174429:TID=3fcb70:Explore :E102:----:I: EXPLORE: Using Low-Memory algorithm
20201204:174429:TID=3fcb70:Explore :E102:----:I: FOUND EXISTING CACHED ITEM
20201204:174429:TID=3fcb70:Explore :E102:----:I: eTGlobalUserName=etaadmin,eTGlobalUserContainerName=Global Users,eTNamespaceName=
20201204:174429:TID=3fcb70:Explore :E102:----:I:+CommonObjects,dc=im
20201204:174429:TID=3fcb70:Explore :E102:----:I: USER-PRIV-CACHE STATS { SIZE: 3 of 20, USED 1 for 1, CAN 0 UNINIT 0 }
20201204:174429:TID=3fcb70:Explore :E102:----:I: AUTH CHECK: [MODIFY access to eTDYNDirectory objects in scope eTDYNDirectoryName=
20201204:174429:TID=3fcb70:Explore :E102:----:I:+Office365_EPT,eTNamespaceName=Office 365,dc=im] => GRANTED
20201204:174429:TID=3fcb70:Explore :E102:----:I: Subtree optimization skipped: Explore/Sync search operation
20201204:174429:TID=3fcb70:Explore :E102:----:I: EXPLORE: Exploring 'usavarremn' [Single Object]
20201204:174429:TID=3fcb70:Search :D105:E102:S: DB Search (eTConfigParamFolderName=Explore and Correlate) Requested by User etaad
20201204:174429:TID=3fcb70:Search :D105:E102:S:+min - TenantNotSet
20201204:174429:TID=3fcb70:Search :D105:E102:P: URL: ldaps://ca-prov-srv:20391
20201204:174429:TID=3fcb70:Search :D105:E102:P: base-dn: eTConfigParamFolderName=Explore and Correlate,eTConfigParamContainer
20201204:174429:TID=3fcb70:Search :D105:E102:P:+ Name=Parameters,eTConfigContainerName=Configuration,eTNamespaceName=CommonObj
20201204:174429:TID=3fcb70:Search :D105:E102:P:+ ects,dc=im
20201204:174429:TID=3fcb70:Search :D105:E102:P: scope : ONE-LEVEL
20201204:174429:TID=3fcb70:Search :D105:E102:P: filter : (eTConfigParamName=Correlation Attribute)
20201204:174429:TID=3fcb70:Search :D105:E102:P: attrs : <ALL>
20201204:174429:TID=3fcb70:Search :D105:E102:F: SUCCESS: DB Search (eTConfigParamFolderName=Explore and Correlate), entry-count:
20201204:174429:TID=3fcb70:Search :D105:E102:F:+1, attributes: objectClass,eTConfigParamName,eTConfigParamDescription,eTConfigPar
20201204:174429:TID=3fcb70:Search :D105:E102:F:+amProperties,eTConfigParamOrigin,eTConfigParamValue,eTUpdateUserid,eTUpdateUserNa
20201204:174429:TID=3fcb70:Search :D105:E102:F:+me,eTUpdateDate,eTUpdateTime,eTUpdateNode
20201204:174429:TID=3fcb70:Search :D106:E102:S: DB Search (eTNamespaceName=Office 365) Requested by User etaadmin - TenantNotSet
20201204:174429:TID=3fcb70:Search :D106:E102:P: URL: ldaps://ca-prov-srv:20391
20201204:174429:TID=3fcb70:Search :D106:E102:P: base-dn: eTNamespaceName=Office 365,dc=im
20201204:174429:TID=3fcb70:Search :D106:E102:P: scope : ONE-LEVEL
20201204:174429:TID=3fcb70:Search :D106:E102:P: filter : (eTDYNDirectoryName=Office365_EPT)
20201204:174429:TID=3fcb70:Search :D106:E102:P: attrs : <ALL>
20201204:174429:TID=3fcb70:Search :D106:E102:F: SUCCESS: DB Search (eTNamespaceName=Office 365), entry-count: 1, attributes: eTLo
20201204:174429:TID=3fcb70:Search :D106:E102:F:+gDestination,eTSubtreeExplored,eTAccountDeletable,eTDYNDirectoryName,eTAccountCon
20201204:174429:TID=3fcb70:Search :D106:E102:F:+tainer,eTPwdPropDisabled,eTLogETSeverity,eTLogStdoutSeverity,eTLogUnicenterSeveri
20201204:174429:TID=3fcb70:Search :D106:E102:F:+ty,eTLogWindowsEventSeverity,eTLogAuditSeverity,eTCreateNode,objectClass,eTDefaul
20201204:174429:TID=3fcb70:Search :D106:E102:F:+tPolicyDN,eTDirectoryName,eTDYN-str-multi-ca-01,eTDYN-str-ca-01,eTDYNUser,eTCreat
20201204:174429:TID=3fcb70:Search :D106:E102:F:+eUserid,eTCreateUserName,eTCreateDate,eTCreateTime,eTID,eTUpdateUserName,eTUpdate
20201204:174429:TID=3fcb70:Search :D106:E102:F:+Userid,eTDYNPassword,eTLog,eTLogFileSeverity,eTUpdateNode,eTUpdateDate,eTUpdateTi
20201204:174429:TID=3fcb70:Search :D106:E102:F:+me
20201204:174429:TID=3fcb70:Search :D107:E102:S: DB Search (dc=im) Requested by User etaadmin - TenantNotSet
20201204:174429:TID=3fcb70:Search :D107:E102:P: URL: ldaps://ca-prov-srv:20391
20201204:174429:TID=3fcb70:Search :D107:E102:P: base-dn: dc=im
20201204:174429:TID=3fcb70:Search :D107:E102:P: scope : ONE-LEVEL
20201204:174429:TID=3fcb70:Search :D107:E102:P: filter : (eTNamespaceName=Office 365)
20201204:174429:TID=3fcb70:Search :D107:E102:P: attrs : <ALL>
20201204:174429:TID=3fcb70:Search :D107:E102:F: SUCCESS: DB Search (dc=im), entry-count: 1, attributes: eTMetaData,eTNamespaceNam
20201204:174429:TID=3fcb70:Search :D107:E102:F:+e,objectClass,eTNamespaceType,eTNamespaceFamily,eTAgentNamespaceClass,eTCreateUse
20201204:174429:TID=3fcb70:Search :D107:E102:F:+rid,eTCreateUserName,eTCreateDate,eTCreateTime,eTID,eTUpdateDate,eTUpdateTime,eTU
20201204:174429:TID=3fcb70:Search :D107:E102:F:+pdateUserName,eTUpdateUserid
20201204:174429:TID=3fcb70:Explore :E102:----:I: DN: eTDYNAccountName=usavarremn,eTDYNAccountContainerName=Accounts,eTDYNDirectoryNa
20201204:174429:TID=3fcb70:Explore :E102:----:I:+me=Office365_EPT,eTNamespaceName=Office 365,dc=im.
20201204:174429:TID=3fcb70:Explore :E102:----:I: Associated CS(s): ldaps://101.120.366.185:20411, ldaps://101.120.366.186:20411
20201204:174429:TID=3fcb70:LDAP :----:----:*: Load Balancing: ldaps://101.120.366.185:20411(0.01), ldaps://101.120.366.186:20411(0.
20201204:174429:TID=3fcb70:LDAP :----:----:*:+02)
20201204:174429:TID=3fcb70:Search :S108:E102:S: Connector Server Search (eTDYNAccountName=usavarremn) Requested by User etaadmin -
20201204:174429:TID=3fcb70:Search :S108:E102:S:+TenantNotSet
20201204:174429:TID=3fcb70:Search :S108:E102:P: URL: ldaps://101.120.366.185:20411
20201204:174429:TID=3fcb70:Search :S108:E102:P: Class Name: User
20201204:174429:TID=3fcb70:Search :S108:E102:P: base-dn: eTDYNAccountName=usavarremn,eTDYNAccountContainerName=Accounts,eTDYNDi
20201204:174429:TID=3fcb70:Search :S108:E102:P:+ rectoryName=Office365_EPT,eTNamespaceName=Office 365,dc=im
20201204:174429:TID=3fcb70:Search :S108:E102:P: scope : BASE
20201204:174429:TID=3fcb70:Search :S108:E102:P: filter : (objectClass=*)
20201204:174429:TID=3fcb70:Search :S108:E102:P: attrs : objectClass, eTDYNAccountName, eTID, eTPolicyDN
20201204:174431:TID=3fcb70:Search :S108:E102:F: SUCCESS: Connector Server Search (eTDYNAccountName=usavarremn), entry-count: 1, att
20201204:174431:TID=3fcb70:Search :S108:E102:F:+ributes: eTDYNAccountName,eTDYN-str-multi-29,objectClass
20201204:174431:TID=3fcb70:Search :D109:E102:S: DB Search (eTDYNAccountName=usavarremn) Requested by User etaadmin - TenantNotSet
20201204:174431:TID=3fcb70:Search :D109:E102:P: URL: ldaps://ca-prov-srv:20391
20201204:174431:TID=3fcb70:Search :D109:E102:P: Class Name: User
20201204:174431:TID=3fcb70:Search :D109:E102:P: base-dn: eTDYNAccountName=usavarremn,eTDYNAccountContainerName=Accounts,eTDYNDi
20201204:174431:TID=3fcb70:Search :D109:E102:P:+ rectoryName=Office365_EPT,eTNamespaceName=Office 365,dc=im
20201204:174431:TID=3fcb70:Search :D109:E102:P: scope : BASE
20201204:174431:TID=3fcb70:Search :D109:E102:P: filter : <NONE>
20201204:174431:TID=3fcb70:Search :D109:E102:P: attrs : <ALL>
20201204:174431:TID=3fcb70:Search :D109:E102:F: FAILURE: DB Search (eTDYNAccountName=usavarremn)
20201204:174431:TID=3fcb70:Search :D109:E102:F: rc: 0x0020 (No such object)
20201204:174431:TID=3fcb70:Search :D109:E102:F: msg: DB Read failed: No such object (ldaps://ca-prov-srv:20391)
20201204:174431:TID=3fcb70:Add :C110:E102:S: Child Add (eTDYNAccountName=usavarremn) Requested by User etaadmin - TenantNotSet
20201204:174431:TID=3fcb70:Add :C110:E102:P: Class Name: User
20201204:174431:TID=3fcb70:Add :C110:E102:P: dn: eTDYNAccountName=usavarremn,eTDYNAccountContainerName=Accounts,eTDYNDirect
20201204:174431:TID=3fcb70:Add :C110:E102:P:+ oryName=Office365_EPT,eTNamespaceName=Office 365,dc=im
20201204:174431:TID=3fcb70:Add :C110:E102:P: eTDYNAccountName(User Name): [e-mail]
20201204:174431:TID=3fcb70:Add :C110:E102:P: eTDYN-str-multi-29(User Principal Name): [e-mail]
20201204:174431:TID=3fcb70:Add :C110:E102:P: objectClass: eTDYNAccount
20201204:174431:TID=3fcb70:Add :C110:E102:P: objectClass: top
20201204:174431:TID=3fcb70:Add :D111:C110:S: DB Add (eTDYNAccountName=usavarremn) Requested by User etaadmin - TenantNotSet
20201204:174431:TID=3fcb70:Add :D111:C110:P: URL: ldaps://ca-prov-srv:20391
20201204:174431:TID=3fcb70:Add :D111:C110:P: Class Name: User
20201204:174431:TID=3fcb70:Add :D111:C110:P: dn: eTDYNAccountName=usavarremn,eTDYNAccountContainerName=Accounts,eTDYNDirect
20201204:174431:TID=3fcb70:Add :D111:C110:P:+ oryName=Office365_EPT,eTNamespaceName=Office 365,dc=im
20201204:174431:TID=3fcb70:Add :D111:C110:P: eTDYNAccountName(User Name): [e-mail]
20201204:174431:TID=3fcb70:Add :D111:C110:P: objectClass: eTDYNAccount
20201204:174431:TID=3fcb70:Add :D111:C110:P: objectClass: top
20201204:174431:TID=3fcb70:Add :D111:C110:P: eTAccountName: usavarremn
20201204:174431:TID=3fcb70:Add :D111:C110:P: eTCreateUserid: etaadmin
20201204:174431:TID=3fcb70:Add :D111:C110:P: eTCreateUserName: etaadmin
20201204:174431:TID=3fcb70:Add :D111:C110:P: eTCreateDate: 0000120339
20201204:174431:TID=3fcb70:Add :D111:C110:P: eTCreateTime: 0006387100
20201204:174431:TID=3fcb70:Add :D111:C110:P: eTID: abd995dc-ca58-103a-8b70-ac4703b3947b
20201204:174431:TID=3fcb70:Add :D111:C110:F: FAILURE: DB Add (eTDYNAccountName=usavarremn)
20201204:174431:TID=3fcb70:Add :D111:C110:F: rc: 0x0013 (Constraint violation)
20201204:174431:TID=3fcb70:Add :D111:C110:F: msg: DB Add failed: Constraint violation (ldaps://ca-prov-srv:20391)
20201204:174431:TID=3fcb70:Add :C110:E102:F: FAILURE: Child Add (eTDYNAccountName=usavarremn)
20201204:174431:TID=3fcb70:Add :C110:E102:F: rc: 0x0013 (Constraint violation)
20201204:174431:TID=3fcb70:Add :C110:E102:F: msg: :ETA_E_0004<AAC>, User 'usavarremn' on 'Office365_EPT' creation failed: DB
20201204:174431:TID=3fcb70:Add :C110:E102:F:+ Add failed: Constraint violation (ldaps://ca-prov-srv:20391)
20201204:174431:TID=3fcb70:Explore :E102:----:F: FAILURE: External Explore (eTDYNAccountName=usavarremn)
20201204:174431:TID=3fcb70:Explore :E102:----:F: rc: 0x0001 (Operations error)
20201204:174431:TID=3fcb70:Explore :E102:----:F: msg: :ETA_E_0024<EAC>, User 'usavarremn' exploration failed: (objects added: 0,
20201204:174431:TID=3fcb70:Explore :E102:----:F:+ deleted: 0, updated: 0, unchanged: 0, failures: 1)

===================================

In windows JCS logs of Office365_EPT folder, I can see below

2020-12-04 17:46:02,170 401585973 [ApacheDS Worker-thread-104] Office_365_Office365_EPT (ObjectClassMappingVisitor.java:467) DEBUG - No childTypes to apply, fall-back to match-all (*) filter
2020-12-04 17:46:02,186 401585989 [ApacheDS Worker-thread-104] Office_365_Office365_EPT (MetaConnector.java:4830) DEBUG - getFilterInfo(): FilterInfo: matches names=[ eTDYNAccount] and classMaps=[ (eTDYNAccount)] with filter='(objectClass=*)'
2020-12-04 17:46:02,186 401585989 [ApacheDS Worker-thread-104] Office_365_Office365_EPT (AttributeMapAndConvertVisitor.java:88) DEBUG - pruning object class assertion: '(objectClass=*)'
2020-12-04 17:46:02,186 401585989 [ApacheDS Worker-thread-104] Office_365_Office365_EPT (AttributeMapAndConvertVisitor.java:190) DEBUG - after completing rewriting filter=: 'FilterInfo: matches names=[ eTDYNAccount] and classMaps=[ (eTDYNAccount)] with filter='null''
2020-12-04 17:46:02,186 401585989 [ApacheDS Worker-thread-104] Office_365_Office365_EPT (MetaConnector.java:4837) DEBUG - getFilterInfo(): after converting attrs: FilterInfo: matches names=[ eTDYNAccount] and classMaps=[ (eTDYNAccount)] with filter='null'
2020-12-04 17:46:02,186 401585989 [ApacheDS Worker-thread-104] Office_365_Office365_EPT (MetaConnector.java:5575) INFO - Ignoring target search filter expressed using: [PRESENCE]
2020-12-04 17:46:02,186 401585989 [ApacheDS Worker-thread-104] Office_365_Office365_EPT (MetaConnector.java:5576) INFO - Search result post-filtering needs activation: O365MetaConnector does not support filters using : [PRESENCE]
2020-12-04 17:46:02,186 401585989 [ApacheDS Worker-thread-104] Office_365_Office365_EPT (MetaConnector.java:6215) INFO - unknown return attribute 'ref' not allowed to be accepted and mapped by isMapAcceptedUnknownAttrIds()==false
2020-12-04 17:46:02,186 401585989 [ApacheDS Worker-thread-104] Office_365_Office365_EPT (MetaConnector.java:6218) INFO - unknown return attribute 'eTID'
2020-12-04 17:46:02,186 401585989 [ApacheDS Worker-thread-104] Office_365_Office365_EPT (MetaConnector.java:6132) INFO - unknown return attribute 'ref' not accepted due to isMapAcceptedUnknownAttrIds == false
2020-12-04 17:46:02,186 401585989 [ApacheDS Worker-thread-104] Office_365_Office365_EPT (MetaConnector.java:6137) INFO - unknown return attribute 'eTID'
2020-12-04 17:46:02,186 401585989 [ApacheDS Worker-thread-104] Office_365_Office365_EPT (MetaConnector.java:6137) INFO - unknown return attribute 'eTPolicyDN'
2020-12-04 17:46:02,186 401585989 [ApacheDS Worker-thread-104] Office_365_Office365_EPT (O365UserHelper.java:507) DEBUG - Looking up an Office 365 Account....
2020-12-04 17:46:10,842 401594645 [ApacheDS Worker-thread-104] Office_365_Office365_EPT (MetaConnector.java:3329) DEBUG - skipping validating outbound

=============

what could have caused this after upgrade? Any idea? I'm not getting much information from Logs. Any thing that we can try to resolve here. Please help.

Thanks,
Tanu Goyal

[Kenneth Verrastro]

The failure is happening when trying to add the object to the Provisioning Repository so it is the Directory layer and logs that would need to be reviewed further. 

20201204:174431:TID=3fcb70:Add :D111:C110:F: FAILURE: DB Add (eTDYNAccountName=usavarremn)
20201204:174431:TID=3fcb70:Add :D111:C110:F: rc: 0x0013 (Constraint violation)
20201204:174431:TID=3fcb70:Add :D111:C110:F: msg: DB Add failed: Constraint violation (ldaps://ca-prov-srv:20391)

[Deactivated User]

I have uploaded 2 log files from directory server and provisioning server.

Attachment(s)

etatrans20201207-2121.log   6.57 MB 1 version

ca-prov-srv-01-impd-main_trace.log   55 KB 1 version

[Kenneth Verrastro]

I think you would be best served by opening a support case to work through this further.

The logs showed the following:

20201207:213143:TID=7a2b70:Add :D040:C039:F: FAILURE: DB Add (eTDYNAccountName=mnavarre)
20201207:213143:TID=7a2b70:Add :D040:C039:F: rc: 0x0013 (Constraint violation)
20201207:213143:TID=7a2b70:Add :D040:C039:F: msg: DB Add failed: Constraint violation (ldaps://ca-prov-srv:20391)
20201207:213143:TID=7a2b70:Add :D040:C039:P: dn: eTDYNAccountName=mnavarre,eTDYNAccountContainerName=Accounts,eTDYNDirect
20201207:213143:TID=7a2b70:Add :D040:C039:P:+ oryName=Office365_EPT,eTNamespaceName=Office 365,dc=im
20201207:213143:TID=7a2b70:Add :D040:C039:P: eTDYNAccountName(User Name): [e-mail]
20201207:213143:TID=7a2b70:Add :D040:C039:P: objectClass: eTDYNAccount
20201207:213143:TID=7a2b70:Add :D040:C039:P: objectClass: top
20201207:213143:TID=7a2b70:Add :D040:C039:P: eTAccountName: mnavarre
20201207:213143:TID=7a2b70:Add :D040:C039:P: eTCreateUserid: etaadmin
20201207:213143:TID=7a2b70:Add :D040:C039:P: eTCreateUserName: etaadmin
20201207:213143:TID=7a2b70:Add :D040:C039:P: eTCreateDate: 0000120342
20201207:213143:TID=7a2b70:Add :D040:C039:P: eTCreateTime: 0007750300
20201207:213143:TID=7a2b70:Add :D040:C039:P: eTID: e846f0e6-ccd3-103a-9902-da385e076d8a


> [7] -> #1 [ca-prov-srv-01-imps-router] DSP ADD-ENTRY-REFUSE
> [7] invoke-id = 720 credit = 1
> [7] Attribute Error:
> [7] Entry:
> [7] <cosineDomainComponent "etadb">
> [7] <cosineDomainComponent "im">
> [7] <eTNamespaceName "Office 365">
> [7] <eTDYNDirectoryName utf8 "Office365_EPT">
> [7] <eTDYNAccountContainerName "Accounts">
> [7] <eTDYNAccountName "mnavarre">
> [7] Attribute: eTDYNAccountName
> [7] Problem: Constraint violation