Scheduling Recurring Dynamics CRM 2011/2013 workflows with FetchXML for more than 5000 records

For most of the CRM implementations, running batch job (to update status, trigger notification, etc.) is unavoidable. But when it comes to deploying those batch jobs console applications on the server, it is kind of a hassle to go to the server and replace the .exe of update the app.config file.

If you can replace those scheduler applications with custom workflow and trigger by timeout recursively, configuration and deployment of the logic is more seamless since everything can be packed up together in a single solution and deployed  remotely via CRM Solution Import. (By the way, here are some excellent tips from Dynamics CRM Tip of the Day about wait and timeout workflows)
Tip #121: Don’t use wait conditions
Tip #122: If you must use waiting workflows
Tip #123: Waiting for Godo in CRM workflows
Tip #124: But wait, there is more
Tip #125: The wait is over

But even with those recursive timeout workflows, there are some disadvantages. The waiting ones are still hogging some of the resource when it comes to thousands of workflow jobs. When the business rules is changed, those which are already triggered and in waiting state will not be affected.

Finally, I've found the best solution to replace the scheduled batch jobs which is the Scheduling recurring Dynamics CRM workflows with FetchXML by Lucas Alexander. Since I found out about this solution, I've replaced all the existing scheduled batch jobs with custom workflows and triggered with the LucasWorkflowTools.StartScheduledWorkflows. The only the problem I've found so far with the solution is that this solution cannot handle so many records. Due to one of the requirements in my current project, I need to trigger on 5000+ records every day. So, I've modified the code with Paging Retrieve Multiple, complied it and registered it to Isolation Mode None. I've also shared the modified code at the bottom of this post just in case any of you guys have such a requirement on your on-premise environment. This solution may not work on CRM online since there is a restriction on plugins not to run more than 2 mins in Sandbox isolation mode.

 FetchXmlToQueryExpressionRequest req = new FetchXmlToQueryExpressionRequest();  
         req.FetchXml = FetchXMLQuery.Get(executionContext);  
         FetchXmlToQueryExpressionResponse resp = (FetchXmlToQueryExpressionResponse)service.Execute(req);  
         QueryExpression query = resp.Query;  
         EntityCollection recordsToProcess = new EntityCollection();  
         RetrieveMultipleRequest multiRequest;  
         RetrieveMultipleResponse multiResponse = new RetrieveMultipleResponse();  
         int pageNumber = 1;  
         do  
         {  
           query.PageInfo.Count = 5000;  
           query.PageInfo.PagingCookie = (pageNumber == 1) ? null : multiResponse.EntityCollection.PagingCookie;  
           query.PageInfo.PageNumber = pageNumber++;  
           multiRequest = new RetrieveMultipleRequest();  
           multiRequest.Query = query;  
           multiResponse = (RetrieveMultipleResponse)service.Execute(multiRequest);  
           recordsToProcess.Entities.AddRange(multiResponse.EntityCollection.Entities);  
         }  
         while (multiResponse.EntityCollection.MoreRecords);  
         recordsToProcess.Entities.ToList().ForEach(a =>  
         {  
           ExecuteWorkflowRequest request = new ExecuteWorkflowRequest  
           {  
             EntityId = a.Id,  
             WorkflowId = (Workflow.Get(executionContext)).Id  
           };  
           service.Execute(request); //run the workflow  
         });