Rally Software

 View Only

 AC2DB

Ben Gilles's profile image
Ben Gilles posted Jan 07, 2021 04:53 PM

I am trying to get data to transfer out of Rally using the python script AC2DB. I have been using it for a while and now I am getting an error. Below is from the log and I am hoping someone can tell me what is causing the issue with the "two chunks of data" and how I can over come this. 


[2021-01-07 19:54:33 Z] DEBUG: EnvironmentalKey.identVector - Obtained conf_owner_id as: **************
[2021-01-07 19:54:33 Z] DEBUG: EnvironmentalKey.identVector - Obtained config_name as: TFIC_Defect.yml
[2021-01-07 19:54:33 Z] INFO: DBConnector - Agile Central DBConnector version 1.1.0
[2021-01-07 19:54:33 Z] INFO: DBConnector - Python platform: Windows 10
[2021-01-07 19:54:33 Z] INFO: DBConnector - Python version: 3.6.0
[2021-01-07 19:54:34 Z] INFO: AgileCentralConnection - Initializing dbeif.AgileCentralConnection connection version 1.1.0
[2021-01-07 19:54:34 Z] INFO: AgileCentralConnection - Rally WSAPI Version v2.0
[2021-01-07 19:54:34 Z] INFO: MSSQLDBConnection - Initializing dbeif.DBConnection connection version 1.1.0
[2021-01-07 19:54:34 Z] INFO: DBConnector.establishConnections - Connected to ('Microsoft SQL Server 2019 (RTM-GDR) (KB4517790) - 15.0.2070.41 (X64) \n\tOct 28 2019 19:56:59 \n\tCopyright (C) 2019 Microsoft Corporation\n\tDeveloper Edition (64-bit) on Windows 10 Pro 10.0 <X64> (Build 18363: )\n', ) on I********************:80 database name: RallySandbox
[2021-01-07 19:54:34 Z] INFO: AgileCentralConnection.connect - Connecting to AgileCentral
[2021-01-07 19:54:38 Z] INFO: AgileCentralConnection.connect - Connected to Agile Central server: rally1.rallydev.com
[2021-01-07 19:54:38 Z] INFO: AgileCentralConnection.connect - Workspace: TFIC
[2021-01-07 19:54:38 Z] INFO: DBConnector.validate - Connector validation starting
[2021-01-07 19:54:38 Z] INFO: DBConnector.validate - AgileCentralConnection validation succeeded
[2021-01-07 19:54:38 Z] WARN: MSSQLDBConnection.validate - You have tables existing in the DB that will be dropped: Defect
[2021-01-07 19:54:38 Z] INFO: DBConnector.validate - DBConnection validation succeeded for RallySandbox
[2021-01-07 19:54:38 Z] INFO: DBConnector.validate - Connector validation completed
[2021-01-07 19:54:38 Z] INFO: DBConnector - Initialization complete: Delegate connections operational, ready for scan/reflect ops
[2021-01-07 19:54:38 Z] DEBUG: DBConnectorRunner._operateService - Got a DBConnector instance, calling the DBConnector.run ...
[2021-01-07 19:54:38 Z] INFO: DBConnector._processEntity - Processing type Defect
[2021-01-07 19:54:38 Z] DEBUG: DBConnector._processEntity - Preparing to write AC data to CSV file: C:\Users\**************\Documents\RallySQLconnector\dbeif-1.1.0\csv\RallySandbox/Defect-2021-01-07T13-54-38.csv
[2021-01-07 19:54:38 Z] DEBUG: AgileCentralConnection.getSchema - ACConn.getSchema(Defect) attrs whose ElementName is 'State':
ElementName: State Type: RATING Custom: No Constrained: Yes
AllowedValues: ['Submitted', 'Open', 'Fixed', 'Closed', 'Rejected']
[2021-01-07 19:54:38 Z] DEBUG: AgileCentralConnection.getSchema - ACConn.getSchema(Defect) attrs whose AttributeType is OBJECT:
ElementName: Blocker Type: OBJECT Custom: No Constrained: No
AllowedValues: unresolved_ref: typedefinition/-51067
ElementName: Iteration Type: OBJECT Custom: No Constrained: Yes
AllowedValues: unresolved_ref: typedefinition/-51015
ElementName: Owner Type: OBJECT Custom: No Constrained: Yes
AllowedValues: unresolved_ref: typedefinition/-51025
ElementName: Project Type: OBJECT Custom: No Constrained: Yes
AllowedValues: unresolved_ref: typedefinition/-51022
ElementName: Release Type: OBJECT Custom: No Constrained: Yes
AllowedValues: unresolved_ref: typedefinition/-51016
ElementName: Requirement Type: OBJECT Custom: No Constrained: No
AllowedValues: unresolved_ref: typedefinition/-51038
ElementName: RevisionHistory Type: OBJECT Custom: No Constrained: No
AllowedValues: unresolved_ref: typedefinition/-51023
ElementName: SubmittedBy Type: OBJECT Custom: No Constrained: Yes
AllowedValues: unresolved_ref: typedefinition/-51025
ElementName: Subscription Type: OBJECT Custom: No Constrained: No
AllowedValues: unresolved_ref: typedefinition/-51021
ElementName: TestCase Type: OBJECT Custom: No Constrained: No
AllowedValues: unresolved_ref: typedefinition/-51012
ElementName: TestCaseResult Type: OBJECT Custom: No Constrained: No
AllowedValues: unresolved_ref: typedefinition/-51017
ElementName: Workspace Type: OBJECT Custom: No Constrained: No
AllowedValues: unresolved_ref: typedefinition/-51020
[2021-01-07 19:54:38 Z] DEBUG: AgileCentralConnection.getSchema - ACConn.getSchema(Defect) pi_state_attrs:
[2021-01-07 19:54:38 Z] DEBUG: AgileCentralConnection.getSchema - ACConn.getSchema(Defect) returning partial_schema of: [SchemaItemAttribute for ObjectID, SchemaItemAttribute for FormattedID, SchemaItemAttribute for AcceptedDate, SchemaItemAttribute for Blocked, SchemaItemAttribute for BlockedReason, SchemaItemAttribute for ClosedDate, SchemaItemAttribute for CreationDate, SchemaItemAttribute for Description, SchemaItemAttribute for DisplayColor, SchemaItemAttribute for Environment, SchemaItemAttribute for FixedInBuild, SchemaItemAttribute for FoundInBuild, SchemaItemAttribute for InProgressDate, SchemaItemAttribute for LastUpdateDate, SchemaItemAttribute for Notes, SchemaItemAttribute for OpenedDate, SchemaItemAttribute for Package, SchemaItemAttribute for PlanEstimate, SchemaItemAttribute for Priority, SchemaItemAttribute for Ready, SchemaItemAttribute for Recycled, SchemaItemAttribute for ReleaseNote, SchemaItemAttribute for Resolution, SchemaItemAttribute for SalesforceCaseID, SchemaItemAttribute for SalesforceCaseNumber, SchemaItemAttribute for ScheduleState, SchemaItemAttribute for Severity, SchemaItemAttribute for State, SchemaItemAttribute for Tags, SchemaItemAttribute for TargetBuild, SchemaItemAttribute for TargetDate, SchemaItemAttribute for TaskActualTotal, SchemaItemAttribute for TaskEstimateTotal, SchemaItemAttribute for TaskRemainingTotal, SchemaItemAttribute for TaskStatus, SchemaItemAttribute for TestCaseStatus, SchemaItemAttribute for VerifiedInBuild, SchemaItemAttribute for Blocker, SchemaItemAttribute for Iteration, SchemaItemAttribute for Owner, SchemaItemAttribute for Project, SchemaItemAttribute for Release, SchemaItemAttribute for Requirement, SchemaItemAttribute for RevisionHistory, SchemaItemAttribute for SubmittedBy, SchemaItemAttribute for Subscription, SchemaItemAttribute for TestCase, SchemaItemAttribute for TestCaseResult, SchemaItemAttribute for Workspace, SchemaItemAttribute for Name]
[2021-01-07 19:54:38 Z] DEBUG: AgileCentralConnection.getInitialQualifier - Retrieving Agile Central items for Defect
[2021-01-07 19:54:38 Z] DEBUG: AgileCentralConnection.entityAttributes - Getting schema for AC entity: Defect
[2021-01-07 19:54:38 Z] DEBUG: AgileCentralConnection.entityAttributes - Obtained schema for AC entity: Defect, narrowed to: ObjectID, FormattedID, AcceptedDate, Blocked, BlockedReason, ClosedDate, CreationDate, Description, DisplayColor, Environment, FixedInBuild, FoundInBuild, InProgressDate, LastUpdateDate, Notes, OpenedDate, Package, PlanEstimate, Priority, Ready, Recycled, ReleaseNote, Resolution, SalesforceCaseID, SalesforceCaseNumber, ScheduleState, Severity, State, Tags, TargetBuild, TargetDate, TaskActualTotal, TaskEstimateTotal, TaskRemainingTotal, TaskStatus, TestCaseStatus, VerifiedInBuild, Blocker, Iteration, Owner, Project, Release, Requirement, RevisionHistory, SubmittedBy, Subscription, TestCase, TestCaseResult, Workspace, Name
[2021-01-07 19:54:38 Z] DEBUG: AgileCentralConnection.createFieldCategories - ACConn.createFieldCategories has pi_state_fields: [] and field_categories['pi_state'] of: []
[2021-01-07 20:01:11 Z] ERROR: DBConnector.run - RallyResponseError in DBConnector.run processing entity: Defect -- Unable to retrieve 2 chunks of data
File "C:\Users\*******\Documents\RallySQLconnector\dbeif-1.1.0\dbeif\connector.py", line 184, in run
stats[entity] = self._processEntity(entity, csv_dir_name, populate_db, trash_csv)
File "C:\Users\*********\Documents\RallySQLconnector\dbeif-1.1.0\dbeif\connector.py", line 259, in _processEntity
entity_schema, pk_field = self.retrieveAndSaveAgileCentralData(entity)
File "C:\Users\*********\Documents\RallySQLconnector\dbeif-1.1.0\dbeif\utils\metrixeur.py", line 13, in wrapper
result = f(*args, **kwargs)
File "C:\Users\**********\Documents\RallySQLconnector\dbeif-1.1.0\dbeif\connector.py", line 226, in retrieveAndSaveAgileCentralData
= self.agicen_conn.getAgileCentralData(workspace_name, project, entity, project_scope_up, project_scope_down, self.remapping)
File "C:\Users\**********\Documents\RallySQLconnector\dbeif-1.1.0\dbeif\agile_central_connection.py", line 615, in getAgileCentralData
for item in response:
File "C:\Users\**********\AppData\Local\Programs\Python\Python36\lib\site-packages\pyral\rallyresp.py", line 313, in __next__
self._page[:] = self.__retrieveNextPage()
File "C:\Users\**********\AppData\Local\Programs\Python\Python36\lib\site-packages\pyral\rallyresp.py", line 410, in __retrieveNextPage
chapter = self.__retrievePages()
File "C:\Users\*********\AppData\Local\Programs\Python\Python36\lib\site-packages\pyral\rallyresp.py", line 478, in __retrievePages
raise RallyResponseError("Unable to retrieve %d chunks of data" % num_threads)

<class 'pyral.rallyresp.RallyResponseError'> Unable to retrieve 2 chunks of data
[2021-01-07 20:01:11 Z] ERROR: DBConnectorRunner.run - Unable to retrieve 2 chunks of data
[2021-01-07 20:01:11 Z] INFO: DBConnectorRunner.run - run completed
Mike_Caron's profile image
Mike_Caron
I have a couple of thoughts, though am not familiar with AC2DB. 

1. I am wondering if you lost the connection or received an improperly filled packet since the state information of the defect is an empty list.

[2021-01-07 19:54:38 Z] DEBUG: AgileCentralConnection.createFieldCategories - ACConn.createFieldCategories has pi_state_fields: [] and field_categories['pi_state'] of: []

2. Have you retrieved defects before? It looks like the schema handling knows about state, but not some of the other variables based on "unresolved_ref" errors below.

[2021-01-07 19:54:38 Z] DEBUG: AgileCentralConnection.getSchema - ACConn.getSchema(Defect) attrs whose ElementName is 'State':
ElementName: State Type: RATING Custom: No Constrained: Yes
AllowedValues: ['Submitted', 'Open', 'Fixed', 'Closed', 'Rejected']
[2021-01-07 19:54:38 Z] DEBUG: AgileCentralConnection.getSchema - ACConn.getSchema(Defect) attrs whose AttributeType is OBJECT:
ElementName: Blocker Type: OBJECT Custom: No Constrained: No
AllowedValues: unresolved_ref: typedefinition/-51067
ElementName: Iteration Type: OBJECT Custom: No Constrained: Yes
AllowedValues: unresolved_ref: typedefinition/-51015
ElementName: Owner Type: OBJECT Custom: No Constrained: Yes
AllowedValues: unresolved_ref: typedefinition/-51025
ElementName: Project Type: OBJECT Custom: No Constrained: Yes
AllowedValues: unresolved_ref: typedefinition/-51022
ElementName: Release Type: OBJECT Custom: No Constrained: Yes
AllowedValues: unresolved_ref: typedefinition/-51016
ElementName: Requirement Type: OBJECT Custom: No Constrained: No
AllowedValues: unresolved_ref: typedefinition/-51038
ElementName: RevisionHistory Type: OBJECT Custom: No Constrained: No
AllowedValues: unresolved_ref: typedefinition/-51023
ElementName: SubmittedBy Type: OBJECT Custom: No Constrained: Yes
AllowedValues: unresolved_ref: typedefinition/-51025
ElementName: Subscription Type: OBJECT Custom: No Constrained: No
AllowedValues: unresolved_ref: typedefinition/-51021
ElementName: TestCase Type: OBJECT Custom: No Constrained: No
AllowedValues: unresolved_ref: typedefinition/-51012
ElementName: TestCaseResult Type: OBJECT Custom: No Constrained: No
AllowedValues: unresolved_ref: typedefinition/-51017
ElementName: Workspace Type: OBJECT Custom: No Constrained: No
AllowedValues: unresolved_ref: typedefinition/-51020
Ben Gilles's profile image
Ben Gilles
Mark,
Thanks for the input. When I run this I do get the message "Retrying the page_urls for the page group startIndexes: 1001 -> 2001". I think this relates to your first thought. this message I get multiple times. 
In reference to your second thought, I believe that the errors refer to custom fields that we created. That is another issue I am working on. 

AC2DB is a python script that takes data from Rally and performs an ETL process to convert it to a SQL database. 

Thank you for the input and i think i will try taking a smaller data sample and see if it is just the size that is causing the retry referred to above