After 1-2 Days: Object reference not set to an instance of an object

[Patrick Schmid]

Hi

 

I got a Problem with my Workflow, after 1-2 Days my Workflow gets following error:

 

Error,Mittwoch, 17. Oktober 2012 08:15:54,Error starting/continuing dialog model execution System.NullReferenceException: Object reference not set to an instance of an object.
   at LogicBase.Components.Default.ExchangeAdapters.LogicBaseExchangeWorkQueue.ClearWorkflow(String serviceID, String workflowTrackingID)
   at LogicBase.Core.Models.Workflow.WorkflowExecutionEngine.HandleExcecutionEnd(TLExecutionContext context, IData data, IData finalData)
   at LogicBase.Core.ExecutionEngine.AbstractExecutionEngine.RunEngine(List`1 executionQueue)
   at LogicBase.Core.ExecutionEngine.AbstractExecutionEngine.ExecuteComponent(IData data, IOrchestrationComponent component)
   at LogicBase.Core.Models.Workflow.WorkflowExecutionEngine.StartInternalWithProcessIDReturn(IData data, String modelName, IData globalData, String testRunId, TestingScenario testingScenario)
   at LogicBase.Core.Models.Workflow.EmbeddedDialogModel.WorkflowDialogHTTPHandler.StartWorkflow(HttpContext context, DialogSession session, WorkflowDeploymentInfo deploymentInfo)
   at LogicBase.Core.Models.Workflow.EmbeddedDialogModel.WorkflowDialogHTTPHandler.StartDialog(HttpContext context, DialogSession session)
   at LogicBase.Core.Models.Dialog.DialogHttpHandlerBase.System.Web.IHttpHandler.ProcessRequest(HttpContext context)

After I do a IIS Restart everything is working fine again, following I see in the logs from IIS that seems to be strange (this goes the whole day and night):

2012-10-17 07:00:42 ::1 POST /HGRSPurchaseOrderTest/WorkflowManagementService.asmx - 80 - ::1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5456) 200 0 0 7
2012-10-17 07:00:42 ::1 POST /hgrspurchaserequest/AutoInvokeDelegateService.asmx - 80 - ::1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5456) 200 0 0 5
2012-10-17 07:00:42 ::1 POST /HGRSPurchaseRequest/WorkflowManagementService.asmx - 80 - ::1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5456) 200 0 0 10
2012-10-17 07:01:42 ::1 POST /hgrspurchaseorder/AutoInvokeDelegateService.asmx - 80 - ::1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5456) 200 0 0 2
2012-10-17 07:01:42 ::1 POST /HGRSPurchaseOrder/WorkflowManagementService.asmx - 80 - ::1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5456) 200 0 0 3
2012-10-17 07:01:42 ::1 POST /hgrspurchaseordertest/AutoInvokeDelegateService.asmx - 80 - ::1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5456) 200 0 0 2
2012-10-17 07:01:42 ::1 POST /HGRSPurchaseOrderTest/WorkflowManagementService.asmx - 80 - ::1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5456) 200 0 0 3
2012-10-17 07:01:42 ::1 POST /hgrspurchaserequest/AutoInvokeDelegateService.asmx - 80 - ::1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5456) 200 0 0 2
2012-10-17 07:01:42 ::1 POST /HGRSPurchaseRequest/WorkflowManagementService.asmx - 80 - ::1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5456) 200 0 0 12
2012-10-17 07:02:42 ::1 POST /hgrspurchaseorder/AutoInvokeDelegateService.asmx - 80 - ::1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5456) 200 0 0 2
2012-10-17 07:02:42 ::1 POST /HGRSPurchaseOrder/WorkflowManagementService.asmx - 80 - ::1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5456) 200 0 0 3
2012-10-17 07:02:42 ::1 POST /hgrspurchaseordertest/AutoInvokeDelegateService.asmx - 80 - ::1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5456) 200 0 0 1
2012-10-17 07:02:42 ::1 POST /HGRSPurchaseOrderTest/WorkflowManagementService.asmx - 80 - ::1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5456) 200 0 0 3
2012-10-17 07:02:42 ::1 POST /hgrspurchaserequest/AutoInvokeDelegateService.asmx - 80 - ::1 Mozilla/4.0+(compatible;+MSIE+6.0;+MS+Web+Services+Client+Protocol+2.0.50727.5456) 200 0 0 3
 

 

And also this in the serverextensions.log (I don't know why, and after an IIS Reset it seems that there is a possibility that it changes in my example from hgrspurchaserequest to hgrspurchaseorder):

Error,Dienstag, 16. Oktober 2012 18:03:55,(WebServiceAutoInvokeDelegate) Error triggering service at http://hgrs-hbk-s1151//hgrspurchaserequest/AutoInvokeDelegateService.asmx System.Exception: exception invoking method ---> System.Web.Services.Protocols.SoapException: System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> System.ComponentModel.Win32Exception: Access is denied
   at System.Diagnostics.ProcessManager.OpenProcess(Int32 processId, Int32 access, Boolean throwIfExited)
   at System.Diagnostics.Process.GetProcessHandle(Int32 access, Boolean throwIfExited)
   at System.Diagnostics.Process.OpenProcessHandle()
   at System.Diagnostics.Process.get_Handle()
   at LogicBase.Framework.Utilities.ProcessorArchitectureDetection.Is64Bit()
   at LogicBase.Framework.Utilities.RegistryHelper.AddSoftwareKey(String key)
   at LogicBase.Core.ServerRegistration.get_EnsembleRelativePath()
   at LogicBase.Core.ServerRegistration.GetEnsembleURL()
   at LogicBase.Core.ApplicationProperty.Evaluate()
   at LogicBase.Core.ExecutionEngine.AbstractExecutionEngine.GetPropertyData(String cultureName)
   at LogicBase.Core.Data.OrchestrationData.GetValue(String name)
   at LogicBase.Components.Default.ExchangeAdapters.LogicBaseExchangeWorkQueue.SetProperties(IData properties)
   at LogicBase.Core.Models.Workflow.WorkQueueFactory.Setup(String typeName)
   at LogicBase.Core.Models.Workflow.WorkflowRuntimeSetup.Execute(AbstractOrchestrationProject project)
   at LogicBase.Core.ProjectRuntime.LoadProject()
   at LogicBase.Core.ProjectRuntime.get_Project()
   at LogicBase.Core.Models.Workflow.WorkflowServices.CheckTimeoutsAndEscalations()
   at LogicBase.Core.Models.Workflow.WorkflowManagementService.CheckTimeoutsAndEscalations()
   --- End of inner exception stack trace ---
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at WorkflowManagementService.CheckTimeoutsAndEscalations()
   --- End of inner exception stack trace ---
   at LogicBase.Framework.DynamicProxy.InvokeMethod(String MethodName, Object[] parameters)
   at LogicBase.ServerExtensions.Core.WSInvokerDelegate.DoInvoke()
Error,Mittwoch, 17. Oktober 2012 08:16:26,(WebServiceAutoInvokeDelegate) Error triggering service at http://hgrs-hbk-s1151//hgrspurchaseorder/AutoInvokeDelegateService.asmx System.Exception: exception invoking method ---> System.Net.WebException: The request failed with HTTP status 503: Service Unavailable.
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at AutoInvokeDelegateService.GetAutoInvokeTarget()
   --- End of inner exception stack trace ---
   at LogicBase.Framework.DynamicProxy.InvokeMethod(String MethodName, Object[] parameters)
   at LogicBase.ServerExtensions.Core.WSInvokerDelegate.DoInvoke()

 

Any help would be appreciated

Patrick

 

 

 

 

 

[Migration User]

 

These entries from IIS logs (second paragraph) are quite normal. This is an effect of background processing done by workflow server.

First the server looks for IIS apps, then calls theirs default web service "AutoInvokeDelegateService.asmx" which returns the URL and service name. Second the server calls retrived web service forcing process to check its timeouts and escalations.

 

When you restart an IIS the workflow server is not able to reach any process. So this is potentialy risky action to restart an IIS without stoping workflow server before. You can loose currently executed process instances.

 

The first error entry from "serverextensions.log" (third paragraph) says that your process has not priviliges to access its own system process handle. All these happens when the workflow server tries to call process for timeout checking. Maybe error is a result of restarting IIS without stopping workflow server system service?

 

The first error is a bit strange (first paragraph). It seems that the new instance of workflow process is started by form and its primary model is executed, but before reaching any DialogWorkflow component the model ends (flow reaches End component - it should never happen) and something goes wrong during clean up procedure.

[Patrick Schmid]

Ok good to hear that IIS Log normal is ( I'm new to the whole Workflow environment).

It seems it depends on how many people opened the Workflow, the last IIS restart was 12:30 and now on 14:30 I had the error again.

This time I did Shutdown the Service "Symantec Workflow Server" and did the IIS restart and afterwards started the Workflow Service again.

What I didn't write last time, after I do the IIS restart the first time I open the Workflow it triggers an Authentication Box. I can cancel and reload the site and it works or put in my credentials (maybe the Invoke logentries from serverextensions.log).

Authentication Settings in IIS for this Workflow Site:

ASP.NET Impersonation Enabled

Windows Authentication Enabled

 

And for WorkflowManagementService.asmx & AutoInvokeDelegateService.asmx following:

Anonymous Authentication Enabled

ASP.NET Impersonation Enabled

 

More info on the Workflow if that helps:

Primary Model is an "Dynamic Linked Model" to hand different version and it points to the newest Model

In the Newest Model I have A "Start" -> "Dialog Workflow" -> "End" and an "Exception Trigger"-> "End"

It seems it fills something up and can't clear it and a IIS Reset clears that. Because first the duration was around 2 days and now as more people use it happens faster.

 

 

[Migration User]

I don't see any suspicions in your description explaining such error.

If I migth guess - because the exception is thrown from ClearWorkflow method in LogicBaseExchangeWorkQueue class - I would bet on exchange configuration problem or memory management problem.

Did I understand you well?
After a short period of time this process you mention - when you try to run it - it every time shows error page and logs the same error (from the frist paragraph?).

[Patrick Schmid]

Ok I will check the exchange configuration, how can I check the memory management? I load different Collections in the Dialog Workflow with some hundred entries (all together I think around 500-800), I thought to try an Initialize Component and set all Collections to NULL.

 

Yes the people can access the Workflowsite several times and out of nowhere the error pops-up, and after IIS Reset it works again for everybody.

[Patrick Schmid]

some more infos

After a manual services restart from Workflow and IIS everything worked till the afternoon, but then I got 4 tiems following Error:

And after this error when I tried to access the Workflow I got the object not set to an instance error.

 

What seems suspicous is the Date Time, the first 2 are 1 minute and 1 Seconds apart, as are the last 2 invoke errors.

 

Edit:

I did a Server Restart and tried to open the Workflow. I prompted me for Authentication and afterwards went directly to the object not set to an instance error. And then again I stoped the Workflow Server Service, IIS Restart, started the Workflow Server Service. Auth Prompt and then it worked

[Migration User]

Hi Patrick,

Can you scrub any login or password data from your project and post it here? I think there is a loop somewhere that is causing your project to crash. It may be in one of the escalation dialouges or something similar.

 

One thing to check when this happens is to see how much memory this project is taking up. When you get the error, open the Task Manager, go to the processes window and take a look at the Memory for the project's process ID.

 

I had a project that was collecting a large amount of data, ran for a few hours and then was supposed to close out and flush out it's memory. Instead, the memory usage just kept getting larger and larger, and would eventually crash the project.

 

Rob

 

[Patrick Schmid]

Hi Rob

 

I checked the Task Manager keeps growing if I add items and everything, after the Workflow gets to EndDialog the consumed memory stays the same and doesn't free up.

Is there a special way to free the memory, I thougt it would do it automatically after it gets to the EndDialog.

(I'll post the Workflow as soon as all passwords etc are out)

 

Patrick

[Migration User]

DateTime is not suspicious.
Errors logged by LogicBase.ServerExtensions usually are effect of the background processing which works in cycles. If your process stopped responding by form start so possibly stopped responding for timeout processing done by server.

[Migration User]

It's not so simple.

First, because it is a .NET and memory management is done by Garbage Collector which recycles memory when it is needed.

Second, during process execution much data are stored in different queues (native exchange functionality of workflow server) like dialogs data kept in local.session, task messages kept in local.workflow.task.cache. These cache queues usually store theirs items in memory and they have different life time set (from 1,5 min to 1 hour). So decreasing allocated memory after process completion can take some time.

[Patrick Schmid]

Additional Info:

I just discovered the Workflowmanagement.aspx site and there I see under Current Processe 97 Processes since 1. October. All have the name from the "Dialogue Workflow Model" - New Dialog Workflow Task. Is there some way to Close this "Workflow Task" if they hit the End Component?

 

 

[Migration User]

It seems that all things your process does inside Dialog Workflow. Sometimes it's not a good solution. You have a lot of data and some integration operations done inside. The cruicial thing is to be sure that your process always completes - ends with a end page "Proces complete...". You can also use its own end page placing at the end of DW a Terminating Form Builder. This is so important because all what is happening during you process is inside web dialog. Each time you go from one web page to another IIS gives your process a precisely limited time to complete the request. By default it is a 90 sec. It's not so hard to exceed this time especially executing integration operations and at the end of DW when a larga pack of data is copied from DW context to main model context. Look for the "ThreadAbortException" in logs.
Conecting all those above with you remark about not finished tasks i can guess that maybe from time to time your process does not complete correctly (nobody cares about end page) and task are not completed and removed. Broken processes can degradate an application pool in which your process is running.
Try first very simple solution - remove all unnecessary variables at the end of dialog (it help quicker close the task).

You can also try to use an option "Is Dialog Start" on Assignemt page of Dialog Worfklow. It changes the way a task is created in exchagne queues - task is not stored in persistent queues and it not stay waiting for timeout if it is not completed.

P.S.
I noticed that you iterate collections to count thier items. You do not need to to this. All collections have a number property named "Count".

[Migration User]

If DW task is processed till end component which is not checked on Do Not Exit On Some Outputs list in DW settings it always and immediatly should be automatically closed (removed from exchange queues). If it is not - something wrong must have happend.

[Patrick Schmid]

Thanks AnaMan

I set it to "Is Dialog Start" and did "Definitions to Remove" with every variable, and also set the integration components directly after the workflow start and just enter the DW to the Web Form and tested it. The Processes and Tasks are closed and not anymore in "WorkflowManagement.aspx".

The Memory Consumption after the workflow was on 146 MB, I completed the workflow 2 more times and now its on 225. After several minutes still the same and Stays there.

Now I set after the Workflow for each Component from before DW in a seperate "Remove Data " Component (or is the initialize component better and set everything to NULL?).

I opend and completed several workflow and the consumption is 165 MB, after several minutes almost the same. I will check tomorrow the consumption again

.

[Migration User]

A few things to note...

All your actions are in the Dialog WF component, even though after the initial input there is no more user interaction required. all those actions should be performed ouside of the Dialog Workflow page. The reason here is that if somone opens the page, and never completes it, they just close the webpage out, the process stays in progress until the time out is reached - which is currently set at 28 days.

Also, you are using Windows Auth, so unless you've set anonymous access to the WorklfowManagementService.asmx and the AutoInvokeDelegateSerice.asmx pages, none of the escalations and time outs will run.

In general, you only want to have just the components necessary to interact with the users in a DW box, and any process or calculations can be done after you exit. This allows the DW to close, and release all the information that is no longer needed in memory. It also allows you to set a 1 or 2 hour timeout on the page, so if a person opens the WF and then abandons the request, you don't have that empty process hanging out there.

 

Rob

 

 

 

[Patrick Schmid]

OK now I have took everything that I can out of the DW, and let it run in the debug and I don't know if thats normal, at the end of the DW it goes to the DW end component but not further, although it does the rest (I receive the Mail etc).

It keeps hanging here:

 

 

Maybe this is normal not to show the steps after the DW?

 

 

 

Edit: Ok found it out, via debug time I see again the other parts. Sorry!

[Patrick Schmid]

Ok the error was here again, but there was a Serverrestart over night (strange that the workflow after a Server Restart doesn't work and goes directly to the error, but I tested it and restarted again now it worked). But is this normal that everytime after an IIS Restart the Authbox comes?

Yes I have Anonymous access to these 2 pages.

Well even if I clear all collections the memory consumption doesn't sink, is there a way to see where the data is stored?

 Also the "is dialog start" is unchecked after closing and opening the project. (I can check it again but will be unchecked after close/open project)

But after the first error today it hase mor infos:

 19.10.2012 08:27:18 Error LogicBase.Core.Models.Dialog.DialogHttpHandlerBase

 Error starting/continuing dialog model execution System.Exception: Execution Engine Exception  ---> System.Exception: Execution Exception ---> System.NullReferenceException: Object reference not set to an instance of an object.    at LogicBase.Components.Default.ExchangeAdapters.LogicBaseExchangeWorkQueue.Store(String serviceID, WorkQueueItem item)    at LogicBase.Components.Default.ExchangeAdapters.LogicBaseExchangeWorkQueue.AddWorkQueueItem(String serviceID, WorkQueueItem item)    at LogicBase.Core.Models.Workflow.WorkflowDelegate.Execute(IData data, IOrchestrationComponent comp, String& outputPath, IExecutionEngine engine, TLExecutionContext context)    at LogicBase.Core.ExecutionEngine.AbstractExecutionEngine.RunComponent(TLExecutionContext context, IData data, IOrchestrationComponent comp)    --- End of inner exception stack trace ---    at LogicBase.Core.ExecutionEngine.AbstractExecutionEngine.RunEngine(List`1 executionQueue)    --- End of inner exception stack trace ---    at LogicBase.Core.ExecutionEngine.AbstractExecutionEngine.RunEngine(List`1 executionQueue)    at LogicBase.Core.ExecutionEngine.AbstractExecutionEngine.ExecuteComponent(IData data, IOrchestrationComponent component)    at LogicBase.Core.Models.Workflow.WorkflowExecutionEngine.StartInternalWithProcessIDReturn(IData data, String modelName, IData globalData, String testRunId, TestingScenario testingScenario)    at LogicBase.Core.Models.Workflow.EmbeddedDialogModel.WorkflowDialogHTTPHandler.StartWorkflow(HttpContext context, DialogSession session, WorkflowDeploymentInfo deploymentInfo)    at LogicBase.Core.Models.Workflow.EmbeddedDialogModel.WorkflowDialogHTTPHandler.StartDialog(HttpContext context, DialogSession session)    at LogicBase.Core.Models.Dialog.DialogHttpHandlerBase.System.Web.IHttpHandler.ProcessRequest(HttpContext context)

 

 

19.10.2012 08:27:18 Error LogicBase.ExecutionEngine

System.Exception: Execution Exception ---> System.NullReferenceException: Object reference not set to an instance of an object.    at LogicBase.Components.Default.ExchangeAdapters.LogicBaseExchangeWorkQueue.Store(String serviceID, WorkQueueItem item)    at LogicBase.Components.Default.ExchangeAdapters.LogicBaseExchangeWorkQueue.AddWorkQueueItem(String serviceID, WorkQueueItem item)    at LogicBase.Core.Models.Workflow.WorkflowDelegate.Execute(IData data, IOrchestrationComponent comp, String& outputPath, IExecutionEngine engine, TLExecutionContext context)    at LogicBase.Core.ExecutionEngine.AbstractExecutionEngine.RunComponent(TLExecutionContext context, IData data, IOrchestrationComponent comp)    --- End of inner exception stack trace ---    at LogicBase.Core.ExecutionEngine.AbstractExecutionEngine.RunEngine(List`1 executionQueue)

 

19.10.2012 08:27:18 Error LogicBase.ExecutionEngine

resulting exception from execution  System.NullReferenceException: Object reference not set to an instance of an object.    at LogicBase.Components.Default.ExchangeAdapters.LogicBaseExchangeWorkQueue.Store(String serviceID, WorkQueueItem item)    at LogicBase.Components.Default.ExchangeAdapters.LogicBaseExchangeWorkQueue.AddWorkQueueItem(String serviceID, WorkQueueItem item)    at LogicBase.Core.Models.Workflow.WorkflowDelegate.Execute(IData data, IOrchestrationComponent comp, String& outputPath, IExecutionEngine engine, TLExecutionContext context)    at LogicBase.Core.ExecutionEngine.AbstractExecutionEngine.RunComponent(TLExecutionContext context, IData data, IOrchestrationComponent comp)

 

19.10.2012 08:27:18 Error LogicBase.ExecutionEngine

last component Add All Fixed to Collection(AddItemsToCollection)

 

19.10.2012 08:27:18 Error LogicBase.ExecutionEngine

current component Dialog Workflow(DialogWorkflowComponent)

 19.10.2012 08:27:18 Error LogicBase.ExecutionEngine

System.NullReferenceException: Object reference not set to an instance of an object.    at LogicBase.Components.Default.ExchangeAdapters.LogicBaseExchangeWorkQueue.Store(String serviceID, WorkQueueItem item)    at LogicBase.Components.Default.ExchangeAdapters.LogicBaseExchangeWorkQueue.AddWorkQueueItem(String serviceID, WorkQueueItem item)    at LogicBase.Core.Models.Workflow.WorkflowDelegate.Execute(IData data, IOrchestrationComponent comp, String& outputPath, IExecutionEngine engine, TLExecutionContext context)    at LogicBase.Core.ExecutionEngine.AbstractExecutionEngine.RunComponent(TLExecutionContext context, IData data, IOrchestrationComponent comp)

 

[Migration User]

Damn, you're right.

I confirmed the problem with unchecking "Is Dialog Start" by .NET Reflector. Programmers forgot to store the state of this property in model file!

Do as Rob said: set timeout to short time: 1-2 hours.

Are you still getting those error System.NullReferenceException: Object reference not set to an instance of an object. ??

Error thrown in LogicBaseExchangeWorkQueue class are dangerous. I still suspect problem with exchange or server configuration.

[Patrick Schmid]

Currently it seems to be good. It's around 197MB consumption even if I open it several times and close it (just differs a little bit goes up to 202 then back to 197 etc). I have it in a seperate AppPool, the DefaultAppPool still grows a bit but I guess this is normal

Additionally I set like said the timeout to 1 hour.

I think take everything out of the DW that is not needed there & additional remove data components did the trick.

 

Thanks so much AnaMan & Rob for the help, learned a lot! I'll wait till monday afternoon and if everything still is fine I'll mark one post of you as solution.

 

About the remove data, is there an easier way to Clear up the data? Currently I have created for each component a "Remove Data" and had to create 16 of them just for data clean up.

And about the exchange/server configuration I don't find any good documents (or my search skills are that bad), I think I configured it on SQL Persistance but do you have any link maybe? I'm really interested to understand this exchange and data store handling.

Thanks again!

[Migration User]

Good! Sounds like you are on the right track here! Please let us all know if this resolves the problem.

 

Rob

[Patrick Schmid]

And here the Workflow

 

 

[Migration User]

 

I think the Remove Data component is the most suitable because it explicitly assigns NULL to a variable signaling to garbage collector that allocated memory in no longer used.

 

But... I have an idea how to abuse a bit the way Embedded Model component works :-)

The Embedded Model component has a property Output Data. It is possible to define on this list all the root level variables with the same names, types and array option but with checked option Allow Null. Then inside Embedded Model for each variable from Output Data a mapping must be set - all these mappings should be set to directly return Null.

Finishing Embedded Model execution the engine merges data from component to main path. All variables with the same name should be set indirectly to Null. 

 

There is no sufficent documentation about exchange functionality of workflow server. There is a document "Managing SymQ" but it has only 8 pages!

I've spent weeks on .NET Reflctor to get known how it really works.

 

[Patrick Schmid]

Nice I'll try that with the Embedded Model. It still seems to work fine, several users have already done some Purchase Requests and it's still around 200mb, let's hope it still works on monday :-).

 

Yes I saw this "Managing SymQ" too, but the tool Logging and Configuration Tool is not in that folder. I think they renamed and did some UI changes in 7.1, so the "Workflow Explorer" does this now?

Well then there is much to learn with try and error on SymQ :-)

Thanks again!

[Migration User]

Yes, its new name is "Workflow Explorer".
The same app but opening on different tab is also "Log Wiever".

[Patrick Schmid]

OK I shortly connected from home and the error was again:

But I saw something, as I tried to open the Page it took some time, and afterwards it was immediatly to the error. RAM level is normal but the first time it went up about 50mb and then back to 140.

 

Error:

 

LogicBase.Components.Default.Workflow.DialogWorkflowComponent

Dialog Workflow

52e819c5-19bc-11e2-afc5-005056aa58bf

System.NullReferenceException

Object reference not set to an instance of an object.

System.NullReferenceException: Object reference not set to an instance of an object. at LogicBase.Components.Default.ExchangeAdapters.LogicBaseExchangeWorkQueue.Store(String serviceID, WorkQueueItem item) at LogicBase.Components.Default.ExchangeAdapters.LogicBaseExchangeWorkQueue.AddWorkQueueItem(String serviceID, WorkQueueItem item) at LogicBase.Core.Models.Workflow.WorkflowDelegate.Execute(IData data, IOrchestrationComponent comp, String& outputPath, IExecutionEngine engine, TLExecutionContext context) at LogicBase.Core.ExecutionEngine.AbstractExecutionEngine.RunComponent(TLExecutionContext context, IData data, IOrchestrationComponent comp)

A wild guess, could it be that the Collections from the "Integration" components are not cleared? Is there anything special with Integrations? I could imagine that the first time it loads it (takes longer to load the page), and every user afterwards it opens very fast. And today It took longer again and maybe it went "double" because it was somehow there and not cleared?

 

 

EDIT:

I checked the logviewer and it had again this error:

Application Name : LogicBase.ServerExtensions.exe

Process ID : 2296

Date :20.10.2012 19:09:16

Log Level :Error

Log Category :AbstractWebServiceURLInvoker

Machine Name : HGRS-HBK-S1151

Message : 

(WebServiceAutoInvokeDelegate) Error triggering service at http://hgrs-hbk-s1151//hgrspurchaserequest/AutoInvokeDelegateService.asmx System.Exception: exception invoking method ---> System.Web.Services.Protocols.SoapException: System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> System.ComponentModel.Win32Exception: Access is denied

   at System.Diagnostics.ProcessManager.OpenProcess(Int32 processId, Int32 access, Boolean throwIfExited)

   at System.Diagnostics.Process.GetProcessHandle(Int32 access, Boolean throwIfExited)

   at System.Diagnostics.Process.OpenProcessHandle()

   at System.Diagnostics.Process.get_Handle()

   at LogicBase.Framework.Utilities.ProcessorArchitectureDetection.Is64Bit()

   at LogicBase.Framework.Utilities.RegistryHelper.AddSoftwareKey(String key)

   at LogicBase.Core.ServerRegistration.get_EnsembleRelativePath()

   at LogicBase.Core.ServerRegistration.GetEnsembleURL()

   at LogicBase.Core.ApplicationProperty.Evaluate()

   at LogicBase.Core.ExecutionEngine.AbstractExecutionEngine.GetPropertyData(String cultureName)

   at LogicBase.Core.Data.OrchestrationData.GetValue(String name)

   at LogicBase.Components.Default.ExchangeAdapters.LogicBaseExchangeWorkQueue.SetProperties(IData properties)

   at LogicBase.Core.Models.Workflow.WorkQueueFactory.Setup(String typeName)

   at LogicBase.Core.Models.Workflow.WorkflowRuntimeSetup.Execute(AbstractOrchestrationProject project)

   at LogicBase.Core.ProjectRuntime.LoadProject()

   at LogicBase.Core.ProjectRuntime.get_Project()

   at LogicBase.Core.Models.Workflow.WorkflowServices.CheckTimeoutsAndEscalations()

   at LogicBase.Core.Models.Workflow.WorkflowManagementService.CheckTimeoutsAndEscalations()

   --- End of inner exception stack trace ---

   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)

   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)

   at WorkflowManagementService.CheckTimeoutsAndEscalations()

   --- End of inner exception stack trace ---

   at LogicBase.Framework.DynamicProxy.InvokeMethod(String MethodName, Object[] parameters)

   at LogicBase.ServerExtensions.Core.WSInvokerDelegate.DoInvoke()

 

Another wild guess :-), everytime I restart IIS I get 1 time prompted for credentials and afterwards it works fine. Could there be a missconfiguration in the IIS? Maybe the "credentials" expired at the time viewd above (20.10.2012 19:xx) and afterwards the workflow wouldn't work?

 

And about the exchange, is there a simple way to "repair" or "reset" the configuration? 

 

 

Here the Invoke Settings, is there maybe something at fault?

AutoInvoke:

 

Manual:

 

 

Again Edit :-): I did a repair via Workflow Setup, afterwards I had to get again "all resources" from SMP for my custom. Maybe that helps

[Patrick Schmid]

OK I verified one thing, everytime it comes to this Object reference not set to an instance of an object, I have this Invocation error first so I think there lies the problem, but no clue how to solve because I don't know what to configure.

 

EDIT: Ok I had this Invocation Error again 10:20 and now its 10:47 and it's still working

[Migration User]

 

You process is quite simple and normal. Event with large amount of data is should work fine on well configured server. I worked with processes storing task messages larger then 5 MB and it made some problems but never like yours.

 

As I said those errors "Object not referenced..." are very strange becasue the only case thay can be thrown is when already initialized in process massege queue handles (this is done during process initialization at first start) have been deleted/reset to null. I don't know what could made such thing. Maybe a workflow server service restart but what more?

 

Very interesting is what you said about connection between both problems.

I started suspecting that you heve some kind of permission problem.

My hypothetical scenario is: the process is started, you have to enter some kind of access data and then all seems to work fine. Till the time when permissions granted to this process becomes invalid (out of term etc.). Then the process lost ability to access some system functionality like opening own system process image or accesing message queues.

 

Could you attach configuration of application pool of your process?

[Patrick Schmid]

Since the repair it works, hopefully it stays like this.

About the message queues is it possible that they are not visible via sql explorer if it is on sql persistence? I only see some messages in the dbo.Messages table if I open a Workflow.

Well the connection thing is really strange, on sunday I thought if the error comes again I will set the "AutoInvoke" Credentials manually, because I don't know which user is default?

Here the configuration (I didn't change anything when I created the apppool):

[Migration User]

Most of settings are typical.

I personally use two differents settings. First I use a classic app pools instead of integrated. Second I use an named account instead of ApplicationPoolIdentity. I had some permission problems using ApplicationPoolIdentity. But in my case workflow tightly integrates with other system and I need use those settings.

[Patrick Schmid]

Hm that's a got hint about the ApplicationPoolIdentity, if it fails now I will change the AutoInvoke and the AppPoolIdentity (to be sure about the db_owner rights on the SQL with the Account).

[Patrick Schmid]

Unfortunately I had to restart IIS and Workflow, but it worked 1 1/2 days without an error. If it works now till thursday I think it's resolved

[Patrick Schmid]

Well it still works :-)

 

But what was it now? Good question I think it was a mix of everything. Thanks again!