- 14 minutes to read

Logging the Request/Response from the Azure API Management platform

This guide teaches how to apply a Nodinite specific policy to enable logging from the Azure API Management platform

In this guide, there are two ways to create the Nodinite Log Events to enable logging to Nodinite from your Azure API Management Service platform.

  1. Azure Event Hub (recommended)
  2. API Call (supported but not recommended)
graph LR A[Azure API Management
with Policy] -->|1. Log Event| B(Azure Event Hub) B --> C{Pickup Service} C --> F[Nodinite's Log API] A -.-> |2. Log Event|F

The policy creates Nodinite Log Events and then puts the result to an Azure Event Hub using an Event Hub Logger. Next, the Nodinite Pickup LogEvents Service Logging Agent transfers these to the instance of Nodinite for use in self-service enabled Log Views.

Body

The payload is optionally logged to Nodinite. The sample code is provided in a template policy in this user guide.

Context

A Nodinite Log Event has a Context, a collection of key-value pairs. The HTTP Headers and programmatically added custom properties from the logging policy can be found in this context. Use these to create the set of Search Fields of interest for use in self-service enabled Log Views.

Request Headers Response Headers

Nodinite Logging feature

Use a Policy and an Azure Event Hub to get logging from your APIs to Nodinite

How to Enable Nodinite logging from your APIs in Azure API Mgmt Services:

When the policy is configured and active, the Azure API Management platform creates a Nodinite specific JSON Log Event that is posted to the named Event Hub. Then, the properly configured Nodinite Pickup LogEvents Service Logging Agent asynchronously moves the JSON Log Event to Nodinite.

You can find additional information about Non-Events Monitoring here.

Event Hub

The target for logging from the API policy is an Event Hub entity. This named Event Hub entity CAN be re-used if the content logged, is a Nodinite JSON Log Event.

graph LR subgraph "Integration solution" C[Azure Function
with Serilog logging] ---|MAX 1 MB payload|C1(Serilog Event Hub Sink) A[Azure API Management Service API
with Policy] --- |MAX 1 MB payload| A1(Policy) B[Azure Logic App
with the diagnostic setting] end subgraph "Event Hub" A1-->|1. Nodinite JSON Log Event| AA(EH1) C1-->|1. Nodinite JSON Log Event| AA C1 -.- CC(EH2) B-->|2. Azure Diagnostics log| BB(EH3) end
  1. Logging from Azure functions can share the same Eventy Hub if the content logged is a Nodinite JSON Log Event
  2. Logging from Azure Logic Apps requires one or more OTHER event hubs as the target since the content is in a different format.

Using a shared Event Hub for a Nodinite JSON Log Event has the advantage of less administration (i.e. you must configure the Nodinite Pickup LogEvents Service Logging Agent to fetch these events)
The disadvantage can be performance-related, or other factors should be considered like pricing (logging from multiple regions), scalability, retention settings, number of partitions, ...

Make sure to use separate event hub entities for different purposes and needs, i.e. multiple Logic App Logging, multiple API management Services, multiple APIs, multiple Azure functions, all according to your architecture

Event Hub Logger

You must first add one or more Event Hub loggers to create the Event Hub Logger used by the Policy. This Event Hub Logger enables logging to Nodinite using an Azure Event Hub as the intermediate storage

You can use the Nodinite Azure Monitoring Agent to manage and monitor these Event Hub Loggers.

The Event Hub Logger can also be created by performing a PUT method with a Body, template provided below:

Method URL
PUT https://management.azure.com/subscriptions/{{subscriptionId}}/resourceGroups/{{resourceGroup}}/providers/Microsoft.ApiManagement/service/{{APIMGMTServiceName}}/loggers/{{loggerName}}?api-version=2019-12-01
DELETE https://management.azure.com/subscriptions/{{subscriptionId}}/resourceGroups/{{resourceGroup}}/providers/Microsoft.ApiManagement/service/{{APIMGMTServiceName}}/loggers/{{loggerName}}?api-version=2019-12-01
  • {{subscriptionId}} -

The {loggerName} is the unique identifier for this logger in your API Management Service ({{APIMGMTServiceName}}). This name is later used in the API Policy.
Replace {loggerName} with the name of the logger, this name should be unique per Event Hub used as the target for logging. Use at least one Event Hub per Environment (Prod, QA, Test, ...)

{
  "properties": {
    "loggerType": "azureEventHub",
    "description": "Sample Event Hub Logger enabling logging from a Policy in Azure API Services to Nodinite",
    "credentials": {
      "name": "{loggerName}",
      "connectionString": "Endpoint=sb://{namespace}.servicebus.windows.net/;SharedAccessKeyName={sharedAccessKeyName};SharedAccessKey={sharedAccessKey};EntityPath={entityPath}"
    }
  }
}

Use this Body as a template and modify it according to your preferences

Property Description
{loggerName} The name of the Event Logger. Created using this Azure REST API, and then it is to be used in policy for the API mgmt Service APIs
{namespace} The namespace with your Event Hub to log to
{sharedAccessKeyName} The name of the shared access key
{sharedAccessKey} The shared access key for the provided key name
{entityPath} The name of the event hub to use as the intermediate destination for Nodinite JSON Log Event, created by the API Management Service Policy

API Policy Configuration

Depending on what you want to log (Request and/or response), you must add a logging configuration into your policy either in the inbound, outbound or in both configurations.

Direction Type
Inbound Request
Outbound Response

The template policy configuration below logs the following items:

  • Mandatory properties for a Nodinite Log Event
    • Feel free to modify content as required by your business requirements
  • HTTP Headers are retained as context properties (Note Some properties are excluded in the sample code)
  • Body

Replace the '{loggerName}' in both API management logger and the code snippets below for the Policy configuration

The example below creates a basic Nodinite JSON Log Event with some common properties, required and optional.

  • Context is optional
  • Body is optional
<log-to-eventhub logger-id='{loggerName}'>
    @{
    var body =  context.Request.Body.As<string>(preserveContent: true);

    var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

    var headers = context.Request.Headers
                .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");

    Dictionary<string, string> contextProperties = new Dictionary<string, string>();
    foreach (var h in headers) {
        contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
    }
    
    var requestLogMessage = new {
        LogAgentValueId = 15,
        EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
        EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
        EndPointDirection = 10,
        EndPointTypeId = 71,
        OriginalMessageTypeName = "Nodinite.Schemas/Nodinite.Demo/5.0#Orders",
        LogDateTime = DateTime.UtcNow,
        Context = contextProperties,
        Body = bodyToLog
    };

    return JsonConvert.SerializeObject(requestLogMessage);
    }
</log-to-eventhub>

Example Azure Event Hub Policy

This complete policy includes a configuration to log both the inbound (Request) and the outbound (Response) information.

In this example, we will add a clientTrackingId (guid) if it was not provided in the x-ms-client-tracking-id HTTP header and some other optional properties for a Nodinite Log Event.

x-ms-client-tracking-id
<policies>
    <inbound>
        <!-- creating a tracking/correlation id -->
        <set-variable name="clientTrackingId" value="@{ 
            return Guid.NewGuid().ToString();
        }" />
        <!-- if x-ms-client-tracking-id exists use it, if not, use clientTrackingId -->
        <set-header name="x-ms-client-tracking-id" exists-action="skip">
            <value>@(context.Variables.GetValueOrDefault<string>("clientTrackingId"))</value>
        </set-header>
        <!-- Put on logger -->
        <log-to-eventhub logger-id="{loggerName}">
            @{
                var body =  context.Request.Body.As<string>(preserveContent: true); // in case we need it later...

                var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

                var headers = context.Request.Headers
                            .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");

                var correlationId = headers.Any(h=>h.Key == "x-ms-client-tracking-id") ? headers.First(h => h.Key == "x-ms-client-tracking-id").Value[0] : context.Variables.GetValueOrDefault<string>("clientTrackingId"); 

                Dictionary<string, string> contextProperties = new Dictionary<string, string>();
                contextProperties.Add("Correlation Id", correlationId);
            
                foreach (var h in headers) {
                    contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
                }
                
                var requestLogMessage = new {
                    LogAgentValueId = 15,
                    EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                    EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                    EndPointDirection = 10,
                    EndPointTypeId = 71,
                    EventDirection=21,
                    OriginalMessageTypeName = "Nodinite.Schemas/Nodinite.Demo/5.0#Orders",
                    LogDateTime = DateTime.UtcNow,
                    LogStatus = 0,
                    ApplicationInterchangeId = correlationId,
                    Context = contextProperties,
                    LogText = "Hello World from Nodinite API mgmt logging policy",
                    Body = bodyToLog
                };

                return JsonConvert.SerializeObject(requestLogMessage);
            }
        </log-to-eventhub>
    </inbound>
    <backend>
        <forward-request />
    </backend>
    <outbound>
        <!-- if x-ms-client-tracking-id exists use it, if not, use clientTrackingId -->
        <set-header name="x-ms-client-tracking-id" exists-action="skip">
            <value>@(context.Variables.GetValueOrDefault<string>("Correlation Id"))</value>
        </set-header>
        <log-to-eventhub logger-id="{loggerName}">
            @{
                var body =  context.Response.Body.As<string>(preserveContent: true); // in case we need it later...

                var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

                var headers = context.Response.Headers
                            .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");
                
                var correlationId = headers.Any(h=>h.Key == "x-ms-client-tracking-id") ? headers.First(h => h.Key == "x-ms-client-tracking-id").Value[0] : context.Variables.GetValueOrDefault<string>("clientTrackingId"); 

                Dictionary<string, string> contextProperties = new Dictionary<string, string>();
                contextProperties.Add("Correlation Id", correlationId);
                foreach (var h in headers) {
                    contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
                }
                string logText = "";

                if (context.Response.StatusCode > 300)
                {
                    logText = "Guru meditation";
                }
                
                var requestLogMessage = new {
                    LogAgentValueId = 15,
                    EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                    EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                    EndPointDirection = 10,
                    EndPointTypeId = 71,
                    EventDirection=25,
                    OriginalMessageTypeName = "Nodinite.Schemas/Nodinite.Demo/5.0#OrderResponses",
                    LogDateTime = DateTime.UtcNow,
                    ApplicationInterchangeId = correlationId,
                    Context = contextProperties,
                    Body = bodyToLog,
                    LogStatus = context.Response.StatusCode,
                    LogText = logText
                };
                return JsonConvert.SerializeObject(requestLogMessage);
            }
        </log-to-eventhub>
    </outbound>
    <on-error />
</policies>

Logging using Rest API Call

This approach is typically used when you are using a consumption-based tier

Before logging into the Nodinite Log API, make sure that it is available from your Azure API Management. It could be part of the VNET or the Log API is available on a public HTTP/HTTPS endpoint.

graph LR A[Azure API Management] -->|Log Event| B(Nodinite's Log API)
<send-request mode="new" response-variable-name="nodiniteRequestResponse" timeout="60" ignore-error="true">
    <set-url>@("{Your public Nodinite log api url")</set-url>
    <set-method>POST</set-method>
    <set-header name="content-type" exists-action="override">
        <value>application/json</value>
    </set-header>
    <set-body>@{

    var body =  context.Request.Body.As<string>(preserveContent: true); // in case we need it later...

    var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

    var headers = context.Request.Headers
                .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");

    Dictionary<string, string> contextProperties = new Dictionary<string, string>();
    contextProperties.Add("CorrelationId", context.Variables.GetValueOrDefault<string>("clientTrackingId"));
    foreach (var h in headers) {
        contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
    }
    
    var requestLogMessage = new {
        LogAgentValueId = 15,
        EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
        EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
        EndPointDirection = 0,
        EndPointTypeId = 71,
        OriginalMessageTypeName = "Azure.ApiManagement.Request",
        LogDateTime = DateTime.UtcNow,
        ApplicationInterchangeId = context.Variables.GetValueOrDefault<string>("clientTrackingId"),
        //Context = new {
            //CorrelationId = context.Variables.GetValueOrDefault<string>("clientTrackingId")
        //},
        Context = contextProperties,
        Body = bodyToLog
    };

    
    return JsonConvert.SerializeObject(requestLogMessage);
    }</set-body>
</send-request>

Example API call

<policies>
    <inbound>
        <!-- creating a tracking/correlation id -->
          <set-variable name="clientTrackingId" value="@{ 
            return Guid.NewGuid().ToString();
        }" />
        <!-- if x-ms-client-tracking-id exists use it, if not, use clientTrackingId -->
        <set-header name="x-ms-client-tracking-id" exists-action="skip">
            <value>@(context.Variables.GetValueOrDefault<string>("clientTrackingId"))</value>
        </set-header>
        <!-- Send request -->
        <send-request mode="new" response-variable-name="nodiniteRequestResponse" timeout="60" ignore-error="true">
            <set-url>@("{Your public Nodinite log api url")</set-url>
            <set-method>POST</set-method>
            <set-header name="content-type" exists-action="override">
                <value>application/json</value>
            </set-header>
            <set-body>@{

            var body =  context.Request.Body.As<string>(preserveContent: true); // in case we need it later...

            var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

            var headers = context.Request.Headers
                        .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");

            Dictionary<string, string> contextProperties = new Dictionary<string, string>();
            contextProperties.Add("CorrelationId", context.Variables.GetValueOrDefault<string>("clientTrackingId"));
            foreach (var h in headers) {
                contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
            }
            
            var requestLogMessage = new {
                LogAgentValueId = 15,
                EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointDirection = 0,
                EndPointTypeId = 71,
                EventDirection=21,
                OriginalMessageTypeName = "SupplyChain.Schemas/1.0#Orders",
                LogDateTime = DateTime.UtcNow,
                ApplicationInterchangeId = correlationId,
                Context = contextProperties,
                LogText = "Forward the API Request to the Logic App", 
                Body = bodyToLog
            };

            
            return JsonConvert.SerializeObject(requestLogMessage);
            }</set-body>
        </send-request>
    </inbound>
    <backend>
        <forward-request />
    </backend>
    <outbound>
        <set-header name="x-ms-client-tracking-id" exists-action="override">
            <value>@(context.Variables.GetValueOrDefault<string>("clientTrackingId"))</value>
        </set-header>
        <send-request mode="new" response-variable-name="nodiniteRequestResponse" timeout="60" ignore-error="true">
            <set-url>@("{Your public Nodinite log api url")</set-url>
            <set-method>POST</set-method>
            <set-header name="content-type" exists-action="override">
                <value>application/json</value>
            </set-header>
            <set-body>@{
            var body =  context.Response.Body.As<string>(preserveContent: true); // in case we need it later...

            var bodyToLog = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(body));

            var headers = context.Response.Headers
                        .Where(h => h.Key != "Authorization" && h.Key != "Ocp-Apim-Subscription-Key");

            Dictionary<string, string> contextProperties = new Dictionary<string, string>();
            contextProperties.Add("CorrelationId", context.Variables.GetValueOrDefault<string>("clientTrackingId"));
            contextProperties.Add("StatusCode", context.Response.StatusCode.ToString());
            foreach (var h in headers) {
                contextProperties.Add(string.Format("Header#{0}", h.Key), String.Join(", ", h.Value));
            }
            
            string logText = "";

            if (context.Response.StatusCode > 300)
            {
                logText = "Guru meditation";
            }
            var requestLogMessage = new {
                LogAgentValueId = 15,
                EndPointName = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointUri = context.Deployment.ServiceName + "/" + context.Api.Name + "/" + context.Operation.Name,
                EndPointDirection = 1,
                EndPointTypeId = 71,
                EventDirection=25,
                OriginalMessageTypeName = "Azure.ApiManagement.Response",
                LogDateTime = DateTime.UtcNow,                                                                                        
                ApplicationInterchangeId = correlationId,
                Context = contextProperties,
                LogStatus = context.Response.StatusCode,
                Body = bodyToLog,
                LogText = logText
            };

            return JsonConvert.SerializeObject(requestLogMessage);
            }</set-body>
        </send-request>
    </outbound>
    <on-error />
</policies>

Troubleshooting

The Event Hub currently has a 1MB limit. If you log larger messages than 1 MB, there will be no payload logged at all(!). The code in the Policy must limit the size of the payload.

Next Step

Azure Application Access
JSON Log Event

Pickup LogEvents Service Logging Agent
Non Events Agent
Monitoring
Managing

Interested in logging from other Azure related services?