- 12 minutes to read

Logging the Request/Response from the Azure API Management platform

In this guide, you will learn how to apply a Nodinite specific policy to enable logging from the Azure API Management platform

In this guide, there are two ways to create the Nodinite Log Events to enable logging to Nodinite from your Azure API Management platform.

  1. Azure Event Hub (recommended)
  2. API Call (supported but not recommended)
graph LR A[Azure API Management
with Policy] -->|1. Log Event| B(Azure Event Hub) B --> C{Pickup Service} C --> F[Nodinite's Log API] A -.-> |2. Log Event|F

The policy creates Nodinite Log Events and then puts the result to an Azure Event Hub using an Event Hub Logger. Next, the Nodinite Pickup LogEvents Service Logging Agent transfers these to the instance of Nodinite for use in self-service enabled Log Views.

Body

The payload of the message can optionally be logged to Nodinite, the code for this is provided in the policy.

Context

A Nodinite Log Event has a Context. This is a collection of key-value pairs. The HTTP Headers and programmatically added custom properties from the logging policy can be found in this context. These can be used to create Search Fields for use in self-service enabled Log Views.

Request Headers Response Headers

Nodinite Logging using an Azure Event Hub

Using an Azure Event Hub is the recommended way to get logging from your APIs to Nodinite

How to Enable Nodinite logging from your APIs in Azure API Mgmt Services:

When the policy is configured and active, the Azure API Management platform creates a Nodinite specific JSON Log Event that is posted to the named EventHub. Then, the properly configured Nodinite Pickup LogEvents Service Logging Agent asynchronously moves the JSON Log Event to Nodinite.

You can find additional information about Non-Events Monitoring here.

Event Hub Logger

You must first programmatically add one or more Event Hub loggers to enable logging to Nodinite using an Azure Event Hub

The Event Hub Logger is created by performing a PUT method with a Body, template provided below:

Method URL
PUT https://management.azure.com/subscriptions/{{subscriptionId}}/resourceGroups/{{resourceGroup}}/providers/Microsoft.ApiManagement/service/{{APIMGMTServiceName}}/loggers/{{loggerName}}?api-version=2019-12-01
DELETE https://management.azure.com/subscriptions/{{subscriptionId}}/resourceGroups/{{resourceGroup}}/providers/Microsoft.ApiManagement/service/{{APIMGMTServiceName}}/loggers/{{loggerName}}?api-version=2019-12-01

The {loggerName} is the unique identifier for this logger in your API Management Service ({{APIMGMTServiceName}}). This name is later used in the API Policy.
Replace {loggerName} with the name of the logger, this name should be unique per Event Hub used as the target for logging. Use at least one Event Hub per Environment (Prod, QA, Test, ...)

{
  "properties": {
    "loggerType": "azureEventHub",
    "description": "Sample Event Hub Logger enabling logging from a Policy in Azure API Services to Nodinite",
    "credentials": {
      "name": "{loggerName}",
      "connectionString": "Endpoint=sb://{namespace}.servicebus.windows.net/;SharedAccessKeyName={sharedAccessKeyName};SharedAccessKey={sharedAccessKey};EntityPath={entityPath}"
    }
  }
}

Use this Body as a template and modify it according to your preferences

Property Description
{loggerName} The name of the Event Logger. Created using this Azure REST API, and then it is to be used in policy for the API mgmt Service APIs
{namespace} The namespace with your Event Hub to log to
{sharedAccessKeyName} The name of the shared access key
{sharedAccessKey} The shared access key for provided key name
{entityPath} The name of the event hub to use as the intermediate destination for Logged Events from API Managment Service Policy
Make sure to use separate event hubs for differents purpose and needs, i.e. multiple Logic App Logging, multiple API management Services, multiple APIs, all according to your architecture

Event Hub Policy Configuration

Depending on what you want to log (Request and/or response), you must add a logging configuration into your policy either in the inbound, outbound, or in both configurations.

Direction Type
Inbound Request
Outbound Response

The template policy configuration below logs the following items:

  • Mandatory properties for a Nodinite Log Event
    • Feel free to modify content as required by your business requirements
  • HTTP Headers are retained as context properties (Note Some properties are excluded in the sample code)
  • Body

Replace the '{loggerName}' in both API management logger and the code snippets below for the Policy configuration

The example below creates a basic JSON Log Event with some common properties, required and optional.

  • Context is optional
  • Body is optional
<log-to-eventhub logger-id='{loggerName}'>
    @{
    var body =  context.Request.Body.As<string>(preserveContent: true);

    var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

    var headers = context.Request.Headers
                .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");

    Dictionary<string, string> contextProperties = new Dictionary<string, string>();
    foreach (var h in headers) {
        contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
    }
    
    var requestLogMessage = new {
        LogAgentValueId = 15,
        EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
        EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
        EndPointDirection = 10,
        EndPointTypeId = 71,
        OriginalMessageTypeName = "Nodinite.Schemas/Nodinite.Demo/5.0#Orders",
        LogDateTime = DateTime.UtcNow,
        Context = contextProperties,
        Body = bodyToLog
    };

    return JsonConvert.SerializeObject(requestLogMessage);
    }
</log-to-eventhub>

Example Azure Event Hub Policy

This complete policy includes a configuration to log both the inbound (Request) and the outbound (Response) information.

In this example, we will add a clientTrackingId (guid) if it was not provided in the x-ms-client-tracking-id HTTP header and some other optional properties for a Nodinite Log Event.

x-ms-client-tracking-id
<policies>
    <inbound>
        <!-- creating a tracking/correlation id -->
        <set-variable name="clientTrackingId" value="@{ 
            return Guid.NewGuid().ToString();
        }" />
        <!-- if x-ms-client-tracking-id exists use it, if not, use clientTrackingId -->
        <set-header name="x-ms-client-tracking-id" exists-action="skip">
            <value>@(context.Variables.GetValueOrDefault<string>("clientTrackingId"))</value>
        </set-header>
        <!-- Put on logger -->
        <log-to-eventhub logger-id='{loggerName}'>
            @{
                var body =  context.Request.Body.As<string>(preserveContent: true); // in case we need it later...

                var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

                var headers = context.Request.Headers
                            .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");

                Dictionary<string, string> contextProperties = new Dictionary<string, string>();
                contextProperties.Add("CorrelationId", context.Variables.GetValueOrDefault<string>("clientTrackingId"));
                foreach (var h in headers) {
                    contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
                }
                
                var requestLogMessage = new {
                    LogAgentValueId = 15,
                    EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                    EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                    EndPointDirection = 10,
                    EndPointTypeId = 71,
                    EventDirection=21,
                    OriginalMessageTypeName = "Nodinite.Schemas/Nodinite.Demo/5.0#Orders",
                    LogDateTime = DateTime.UtcNow,
                    LogStatus = context.Response.StatusCode,
                    ApplicationInterchangeId = context.Variables.GetValueOrDefault<string>("x-ms-client-tracking-id"),
                    Context = contextProperties,
                    Body = bodyToLog
                };

                return JsonConvert.SerializeObject(requestLogMessage);
            }
        </log-to-eventhub>
    </inbound>
    <backend>
        <forward-request />
    </backend>
    <outbound>
        <!-- if x-ms-client-tracking-id exists use it, if not, use clientTrackingId -->
        <set-header name="x-ms-client-tracking-id" exists-action="skip">
            <value>@(context.Variables.GetValueOrDefault<string>("Correlation Id"))</value>
        </set-header>
        <log-to-eventhub logger-id='{loggerName}'>
            @{
            var body =  context.Response.Body.As<string>(preserveContent: true); // in case we need it later...

            var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

            var headers = context.Response.Headers
                        .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");

            Dictionary<string, string> contextProperties = new Dictionary<string, string>();
            contextProperties.Add("CorrelationId", context.Variables.GetValueOrDefault<string>("x-ms-client-tracking-id"));
            foreach (var h in headers) {
                contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
            }
            string logText = "";

            if (context.Response.StatusCode > 300)
            {
                logText = "Guru meditation";
            }
            
            var requestLogMessage = new {
                LogAgentValueId = 15,
                EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointDirection = 10,
                EndPointTypeId = 71,
                EventDirection=25,
                OriginalMessageTypeName = "Nodinite.Schemas/Nodinite.Demo/5.0#OrderResponses",
                LogDateTime = DateTime.UtcNow,
                ApplicationInterchangeId = context.Variables.GetValueOrDefault<string>("x-ms-client-tracking-id"),
                Context = contextProperties,
                Body = bodyToLog,
                LogStatus = context.Response.StatusCode,
                LogText = logText
            };
            return JsonConvert.SerializeObject(requestLogMessage);
            }
        </log-to-eventhub>
    </outbound>
    <on-error />
</policies>

Logging using Rest API Call

This approach is typically used when you are using a consumption-based tier

Before logging to Nodinite's Log API, make sure that Nodinite's Log API is available to your Azure API Management. It could be part of the VNET or the Log API is available on a public endpoint.

graph LR A[Azure API Management] -->|Log Event| B(Nodinite's Log API)
<send-request mode="new" response-variable-name="nodiniteRequestResponse" timeout="60" ignore-error="true">
    <set-url>@("{Your public nodinite log api url")</set-url>
    <set-method>POST</set-method>
    <set-header name="content-type" exists-action="override">
        <value>application/json</value>
    </set-header>
    <set-body>@{

    var body =  context.Request.Body.As<string>(preserveContent: true); // in case we need it later...

    var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

    var headers = context.Request.Headers
                .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");

    Dictionary<string, string> contextProperties = new Dictionary<string, string>();
    contextProperties.Add("CorrelationId", context.Variables.GetValueOrDefault<string>("clientTrackingId"));
    foreach (var h in headers) {
        contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
    }
    
    var requestLogMessage = new {
        LogAgentValueId = 15,
        EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
        EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
        EndPointDirection = 0,
        EndPointTypeId = 71,
        OriginalMessageTypeName = "Azure.ApiManagement.Request",
        LogDateTime = DateTime.UtcNow,
        ApplicationInterchangeId = context.Variables.GetValueOrDefault<string>("clientTrackingId"),
        //Context = new {
            //CorrelationId = context.Variables.GetValueOrDefault<string>("clientTrackingId")
        //},
        Context = contextProperties,
        Body = bodyToLog
    };

    
    return JsonConvert.SerializeObject(requestLogMessage);
    }</set-body>
</send-request>

Example API call

<policies>
    <inbound>
        <!-- creating a tracking/correlation id -->
          <set-variable name="clientTrackingId" value="@{ 
            return Guid.NewGuid().ToString();
        }" />
        <!-- if x-ms-client-tracking-id exists use it, if not, use clientTrackingId -->
        <set-header name="x-ms-client-tracking-id" exists-action="skip">
            <value>@(context.Variables.GetValueOrDefault<string>("clientTrackingId"))</value>
        </set-header>
        <!-- Send request -->
        <send-request mode="new" response-variable-name="nodiniteRequestResponse" timeout="60" ignore-error="true">
            <set-url>@("{Your public nodinite log api url")</set-url>
            <set-method>POST</set-method>
            <set-header name="content-type" exists-action="override">
                <value>application/json</value>
            </set-header>
            <set-body>@{

            var body =  context.Request.Body.As<string>(preserveContent: true); // in case we need it later...

            var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

            var headers = context.Request.Headers
                        .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");

            Dictionary<string, string> contextProperties = new Dictionary<string, string>();
            contextProperties.Add("CorrelationId", context.Variables.GetValueOrDefault<string>("clientTrackingId"));
            foreach (var h in headers) {
                contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
            }
            
            var requestLogMessage = new {
                LogAgentValueId = 15,
                EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointDirection = 0,
                EndPointTypeId = 71,
                EventDirection=21,
                OriginalMessageTypeName = "SupplyChain.Schemas/1.0#Orders",
                LogDateTime = DateTime.UtcNow,
                ApplicationInterchangeId = correlationId,
                Context = contextProperties,
                LogText = "Forward the API Request to the Logic App", 
                Body = bodyToLog
            };

            
            return JsonConvert.SerializeObject(requestLogMessage);
            }</set-body>
        </send-request>
    </inbound>
    <backend>
        <forward-request />
    </backend>
    <outbound>
        <set-header name="x-ms-client-tracking-id" exists-action="override">
            <value>@(context.Variables.GetValueOrDefault<string>("clientTrackingId"))</value>
        </set-header>
        <send-request mode="new" response-variable-name="nodiniteRequestResponse" timeout="60" ignore-error="true">
            <set-url>@("{Your public nodinite log api url")</set-url>
            <set-method>POST</set-method>
            <set-header name="content-type" exists-action="override">
                <value>application/json</value>
            </set-header>
            <set-body>@{
            var body =  context.Response.Body.As<string>(preserveContent: true); // in case we need it later...

            var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

            var headers = context.Response.Headers
                        .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");

            Dictionary<string, string> contextProperties = new Dictionary<string, string>();
            contextProperties.Add("CorrelationId", context.Variables.GetValueOrDefault<string>("clientTrackingId"));
            contextProperties.Add("StatusCode", context.Response.StatusCode.ToString());
            foreach (var h in headers) {
                contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
            }
            
            string logText = "";

            if (context.Response.StatusCode > 300)
            {
                logText = "Guru meditation";
            }
            var requestLogMessage = new {
                LogAgentValueId = 15,
                EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointDirection = 1,
                EndPointTypeId = 71,
                EventDirection=25,
                OriginalMessageTypeName = "Azure.ApiManagement.Response",
                LogDateTime = DateTime.UtcNow,                                                                                        
                ApplicationInterchangeId = correlationId,
                Context = contextProperties,
                LogStatus = context.Response.StatusCode,
                Body = bodyToLog,
                LogText = logText
            };

            return JsonConvert.SerializeObject(requestLogMessage);
            }</set-body>
        </send-request>
    </outbound>
    <on-error />
</policies>

Next Step

JSON Log Event

Pickup LogEvents Service Logging Agent
Non Events Agent