Access Flow Run History within a Record in Dynamics 365/Microsoft Dataverse

As recommended by Microsoft, I have been creating automated flows instead of background workflows for a while now (obedient me!). But not being able to see the list of Background Processes related to the record is one of the things that I miss from the classic background workflows (especially when I am having an issue 🐛 with a particular record).

Although we no longer have the Background Processes associated view in Unified Interface anyway, I still wish I could still see the list of flow run history related to a particular record in a similar fashion as background process history. In that way, if there is an issue with a particular record, I can quickly go and check out the related flow runs instead of listing all runs in Run History, downloading the .csv file, 🤿 diving through all rows and looking for the record GUID to find that URL of a particular run (phew! 😅).

Note: If you want to see approval flows associated with a CDS entity record, you can use this awesome Approval Status PCF control by Abdul Mounem to show the status of the approval flows associated with an entity record.

Because I missed the Background Processes so much, I delve into the magical realm of the World Wide Web to find a solution to my current problem. And by doing so, I found out that there is a way are several ways of getting the list of flow runs with such as Workflow Runs - List API (Update: I was advised that Logic Apps API cannot be used for flow runs history due to the lack of subscriptionId) Get-FlowRun cmdlet from Microsoft.PowerApps.PowerShell but all of those options require pro dev skills 🤓 - plus they are not easy to get the flow runs related to a particular CDS record only. I also looked into Power Automate for Admins and Power Automate Management connectors but I cannot seem to find any action available to list the Flow run history. And then finally, I attempted to see if I can log the history of the flow into one of the CDS records to replicate the functionality of the Background Processes list – and luckily that worked! 😉

If you do not want to log the history of all flow runs and just want to find the flow run with the record ID for troubleshooting purposes, I created an on-demand flow which accepts the name of the flow and the record GUID as parameters to get the flow run history. Check out more at my other blog post.

🛈 Disclaimer

This solution is inspired by a blog post by Rik de Koning on "Keeping up with the Microsoft Flow run history of your SharePoint item"

There are two ways of how we can log run history of the flow. The first approach is to log at the end of the flow (Approach 1) and the second one is to log with child flow at the beginning of the flow (Approach 2).

Let’s have a look in detail on how I managed to make those two approaches work and the step by step process in completing the flow.

First of all, I created a custom activity entity to store the flow run history data and named it "Flow Run". I used a custom activity entity so that I can make use of the Regarding lookup field to link to various entities later on. I renamed a few fields to make sense to what we are trying to achieve here and added a few more fields to capture the Status of the flow ("Successful", "Failed", "Timed Out"), the Trigger (Create, Update, Delete, Select) and the Flow Run URL.

Approach 1 (logging at the end of the flow)

I have a flow which triggers on Create/Update/Delete of an Account record. To test my first approach, I added the flow run logging mechanism at the end of this flow. Below are the steps you can follow and add in your automated flows to log the run history against the CDS records.

1. Initialize Flow Status variable
2. Scope - Main Flow
3. Set Flow Status variable
4. Initialize Trigger Event variable
5. Set Trigger Event variable
6. Create a new Flow Run
7. Update Flow Run to Set Triggering User
8. Update Flow Run to Set Regarding record
9. Terminate with same Status

1. Initialize Flow Status variable

This step to initialise the Flow Status comes right after the trigger because the value of the Flow Status variable needs to be set right after the next step.

2. Scope - Main Flow

This step is where you have to put the logic of your flow. Any action, condition, looping step of the flow must be enclosed in the Scope (except Initialise Variable steps which needs to be moved to the step above the Scope).

3. Set Flow Status variable

In the next step, add 3 parallel branches to capture the status of the Scope - Main Flow whether any of the steps in the Scope - Main Flow has failed or timed out or the whole Scope - Main Flow is successful.

Configure the first step on the left to run after the Scope - Main Flow is successful. This step will set the Flow Status variable with the option set value to Successful integer option set value.

In a similar way, configure the middle step to run after Scope - Main Flow has failed and set the variable to Failed option set value.

The step on the right will run after the Scope - Main Flow has timed out to set the variable to Timed Out option set value.

4. Initialize Trigger Event variable

The next step is just initialising another variable to store the trigger event. But this step needs to be configured to run after all three steps above are Successful or Skipped. For the default setting, the step will run only when all three steps above are Successful. Based on the configuration of the x3 Set Flow Status variable steps, only one of them will be Successful and the rest will be Skipped. There is no occasion that all three of them will be Successful and if you do not change the default setting, the flow will stop at this point and will never run this step.

5. Set Trigger Event variable

To find out if the flow is triggered when the record is Created, Updated, Deleted or Selected (on-demand flow), SdkMessage parameter from the triggerBody can be used to determine this.

triggerBody()?['body']?['SdkMessage']

SdkMessage will contain the string value 'Create', 'Update', 'Delete' for Common Data Service (current environment) connector trigger and based on the value, set the Trigger Event variable option set value. If there is no value for SdkMessage, we can assume that the flow is triggered by the "When a record is selected" trigger from Common Data Service connector.

6. Create a new Flow Run

In this step, we will create the Flow Run custom activity entity that we created at the beginning of this blog post. For the Flow Name field, set the flowDisplayName tag from the workflow() function. You can also read more details about workflow() function in the Dynamics 365 in the Field blog post.

workflow()['tags']['flowDisplayName']




For the Description field, you can store all error-related information such as Error Code, Message, Inner Error, etc. In this example, the error information for "List records (Dummy Record)" step, one of the flow steps in the Scope - Main Flow, is captured. You can capture more error info by using concat() function but make sure the error details do not exceed the 1048576 characters (max char limit for the Description field).

This way of error capturing seems like a bit of repetitive and tedious task but I cannot seem to find a way to get the error information for the whole Scope step or the first error of the flow. If you figure out a better way to capture the error information, please leave your suggestion in the comment section below.

if(
    equals(
        outputs('List_records_(Dummy_Record)')?['body']?['error'],
        null
    ),
    '', 
    concat(
        actions('List_records_(Dummy_Record)')?['name'],
        '
Error Code:',
        outputs('List_records_(Dummy_Record)')?['body']?['error']?['code'],
        '
Error Message: ',
        outputs('List_records_(Dummy_Record)')?['body']?['error']?['message'],
        '
Inner Error: ',
        outputs('List_records_(Dummy_Record)')?['body']?['error']?['innererror']
    )
)




To get the Start Time of the flow run, use the trigger function to get the start time.

trigger()?['startTime']

For more details about the trigger function, you can check out Pieter Veenstra's blog, "The INs and OUTs of triggers and actions".

The End Time is just the current date/time using utcNow() function and the Duration can be calculated by converting both date/time values into ticks and dividing by 600000000 (number of ticks in a minute). For most of the flows, it would not take more than 1 minute to run and the Duration field will be just 0 minute. (if it is not, you should be concerned about it ⚠️)

div(
    sub(
        ticks(trigger()?['startTime']),
        ticks(utcNow())
    ),
    600000000
)

Here comes the main act of the whole show, which is to build the flow run URL from the parameters from workflow() function. I got the formula from Rik's blog post and it works like a charm even if I remove the region subdomain from the URL.

concat(
    'https://flow.microsoft.com/manage/environments/',
    workflow()['tags']['environmentName'],
    '/flows/',
    workflow()['name'],
    '/runs/',
    workflow()['run']['name']
)

Keep in mind that Microsoft only keeps run logs for no longer than 28 days. So, the flow run URL will not work after 28 days and if you need to maintain a longer history, you will have to capture run histories before they are deleted.

Finally, set the Flow Status and Trigger Event variables from step 3 and 5 into corresponding option set fields.

7. Update Flow Run to Set Triggering User

The GUID of the user who triggers the flow can be found in the RunAsSystemUserId parameter of the trigger output body.

triggerOutputs()?['body']?['RunAsSystemUserId']

But there is no such information can be found in "When a record is selected" trigger from Common Data Service connector. That is the reason why this part is extracted in this sample flow as a separate condition to check if the value exists and populate it in a field only if the trigger output contains such information. In a normal scenario, if the trigger is not "When a record is selected" one, the information can be populated in the Flow Run creation at step 6.

In this sample flow, Owner field is re-used to store the information of the user who triggers the flow (as in the job owner). That will require all the users to have Read privilege to the Activity entity or else, the system will throw an error when the user is assigned to the Flow Run entity as an owner with no Read privilege. If there is any such scenario exists (users without Read privilege to Activity entity), create a new lookup field to the User entity and store the trigger user info in that field.

8. Update Flow Run to Set Regarding record

Again, another conditionally Update step to set the Regarding field of the Flow Run to avoid an error for Delete trigger (where the record no longer exists). Once we link the record in the Regarding field, we can see all the related (Create, Update, On-demand) flow runs of the record in the associated view.

If only the Common Data Service (current environment) connector allowed setting NULL value to the lookup field, we could have set the Regarding field conditionally using flow expression instead of creating such additional Update step. If you also think we should be able to set NULL value to the lookup field (who does not! 😁), vote for this idea on the Power Automate Ideas site.

9. Terminate with same Status

You might be wondering why there is a Terminate step at the end of the flow. In a normal scenario, we would not need a Terminate step and the flow will just stop after processing the last step. But for this case, we have those exception handling steps which triggers after the Scope - Main Flow has failed or timed out. That means the flow will always complete successfully no matter if there is an error in Scope - Main Flow or not. That will make it seem like all of the flow runs were successful in the run history. To show the correct flow status, you need to terminate it conditionally based on the Flow Status variable that is captured in step 3. Try/Catch/Log/Rethrow is probably not a good idea in the actual programming language but it seems like a good one in this scenario.

if(equals(variables('Flow Status'), 273510000), 'Succeeded', 'Failed')

Once you have set up everything, you should be able to see the flow history records related to the entity for each flow run and access the flow run URL easily.

Is that a lot to configure for each flow just to log the flow runs against the record? Do you already have dozen of flows that you want to log the flow runs without configuring hundreds of steps? Then have a look at the second approach below.

Approach 2 (logging with child flow at the beginning of the flow)

Here is an alternative solution for logging the flow run history using the child flow. I have extracted most of the important parts to build as a re-usable child flow. All you have to do is configure x2 steps for each flow. The first one to call the child flow and the second one to set Regarding field of the Flow Run activity created by the child flow based on the triggering entity.

The downside to this approach though is, unlike the first approach (logging at the end of the flow), we cannot log some of the information like Flow Status, Error Info, End Time and Duration, and that is because of the positioning of the flow. The flow history logging child flow is triggered up front as the first action right after the trigger so that the flow is captured in the log even if the flow fails down the line. The x4 pieces of information that we captured in the first approach are only available at the end of the flow, thus we cannot capture this information as the child flow runs at the beginning and not at the end.

"Log Flow Run History" child flow requires two string parameters. The whole JSON object of the workflow() function.

string(workflow())

And the whole JSON object of the trigger() function.

string(trigger())

Both of those JSON objects need to be wrapped around by string() function to convert as a string. Otherwise, you will get the following error and there is no Object input type available for the child flow either.

The input body for trigger 'manual' of type 'Request' did not match its schema definition. Error details: 'Invalid type. Expected String but got Object.,Invalid type. Expected String but got Object.'.

The next step is just to set the Regarding field value of the Flow Run record based on the current triggering entity. I tried to capture that information within the child flow but it seems to be impossible because of the following reasons:

1. Common Data Service (current environment) connector does not allow setting NULL value to the Regarding lookup field when I try to set conditionally with flow expression based on the entity type.
2. The standard Common Data Service connector cannot be used in the child flow and it fails with the following error if the child flow contains any step with the old CDS connector.

{
  "code": "InvokerConnectionOverrideFailed",
  "message": "Failed to parse invoker connections from trigger 'manual' outputs. Exception: Could not find property 'headers.X-MS-APIM-Tokens' in the trigger outputs. Workflow has connection references '[\"shared_commondataservice\"]' with invoker runtime source."
}

🛈 Note

It is important to Configure run after for the immediate step after the Set Regarding step and check all conditions. This configuration is important when the logging is done at the beginning of the flow and we would not want to let the flow fail just because of any issue in the logging mechanism. If you set that configuration properly, the flow continues to run the steps below even if the child flow failed.

This is the "Child Flow - Log Flow Run History" with the same steps 4-7 from the Approach 1 flow.

The only additional steps are x2 Parse JSON steps from the trigger input and Respond Flow Run record GUID to the parent flow for setting Regarding lookup.

One more difference in the child flow is setting "Configure run after" for all of the steps to proceed regardless of the condition of the preceding step. This precaution is taken so that the child flow will always be processed until the last step and respond to the parent flow. Otherwise, the parent flow will keep hanging until it will time out with the following error message before proceeding to the next step.

The server did not received a response from an upstream server.

Summary

Logging the flow run history against the Dynamics 365/CDS Record can be done by adding additional steps to the flows. Depending on the data that you want to capture and the effort that you can spend for each flow to add logging, you can decide which approach will work for you.

You can download the unmanaged solution from my GitHub repository via this link. The solution contains the following components:

• Flow Run custom activity entity
• Flow Trigger global option set
• Flow Run History Demo (sample flow for approach 1)
• Child Flow - Log Flow Run History (child flow from approach 2)
• Run Flow Run Logging Child Flow Demo (sample flow which calls the child flow above)

Note: The entity and the global option set in the solution have "lzw_" prefix. If you wish to use your own environment prefix, open the customizations.xml file inside the solution .zip file and Find & Replace all instances of "lzw_" prefix with your own.