- 15 minutes to read

Configuration of Nodinite Logging Agent - Pickup Service

In this guide, you will learn how to configure the Nodinite Pickup LogEvents Service Logging Agent so it can pick up JSON Log Events from any of the supported sources.

Configuration file overview

Below is a complete list for all fields/properties and their purpose

The configuration file (Settings.json), is in the JSON format.

The default path is:

C:\Program Files\Nodinite\Logging Agent - Pickup Service\Settings.json

You must restart the Pickup LogEvents Service Logging Agent for any changes to take effect.

General

The general section holds some global information that affects mostly the internal behaviour of the Pickup LogEvents Service Logging Agent.

Property Description Value Example Comment
Environment The name for the type of environment this agent is running in Test, Prod, QA
Debug A flag that determines the amount of diagnostics information to write true or false (default) In a production environment, this value is ordinarily false
Version The internal version number for the configuration file 1.2.3.4 READ ONLY, this is internal information do NOT change this value manually!!!
CultureInfo format the output of date and so on SV-SE, EN-us Currently not implemented/honoured

Folders

The Folders section is where you manage file folders. These can either be local folders or SMB enabled shares.

Folders is an array of a Folder object. Each Folder points to one root (a file folder or SMB share) where the Log Agent should look for log files with JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false This feature means that you don't have to remove the configuration to disable the pickup service temporarily
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
Folder The Folder where to look for log files for a network share, this means \\server\share will be \\\\server\\share in the JSON configuration file Use \\ to escape \ in the JSON notation.
BackoutFolder The Folder where to put invalid log files for a network share, this means \\server\theothershare will be \\\\server\\theothershare in the JSON configuration file Use \\ to escape \ in the JSON notation. CAUTION This should NOT be a child folder if the IncludeChildFolders option is enabled, unless, you have a proper exclude configuration for these
RemoveEmptyFiles When true, then empty files will be removed by the system true, false Empty files should not have been here anyway...
Filter A file filter to only look for file matching pattern. *.log or *.json, ... Can contain a combination of valid literal path and wildcard (* and ?)
IncludeChildFolders A flag used to search for JSON Log Event in subfolders true or false
ExcludeChildFolders An array of child folders to exclude if the IncludeChildFolders property is set to true Note: The full path must be provided, not only the name of the subfolder
ReplaceRules Review the shared ReplaceRules section for additional information

ActiveMQs

The ActiveMQs section is where you manage Apache ActiveMQ related sources.

ActiveMQs is an array of an ActiveMQ queues. Each configuration is valid for one ActiveMQ queue. This configuration tells Log Agent where to look for log messages with the Nodinite JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false This feature means that you don't have to remove the configuration to disable the pickup service temporarily
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
ActiveConsumers Number of consumers that the agent will use to collect messages 1-n Our recommendation is a positive integer between 1 and 4
ConnectionString The connection string used to the ActiveMQ server, can also be a fail-over address tcp://127.0.0.1:61616 or failover:(tcp://127.0.0.1:61616,tcp://host2:61616)?randomize=false&timeout=1000&startupMaxReconnectAttempts=2&|maxReconnectDelay=2000
Username The user name used to connect with ActiveMQ System
Password The password for the user Manager
Queue The name of the ActiveMQ Queue from where to pickup JSON Log Event from
DeadLetterQueue The name of the Dead letter queue where the pickup service puts invalid events and/or events that fails to log
ReplaceRules Review the shared ReplaceRules section for additional information

MSMQs

The MSMQs section is where you manage Microsoft MSMQ related sources.

MSMQs is an array of MSMQ queues. Each configuration is valid for one MSMQ queue. This configuration tells Log Agent where to look for log messages with the Nodinite JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false This feature means that you don't have to remove the configuration to disable the pickup service temporarily
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
ActiveConsumers Number of consumers that the agent will use to collect messages 1-n Use any positive integer, between 1 and 4 is recommended
Server The name of the Windows server with MSMQ
Queue The name of the MSMQ Queue from where to pickup JSON Log Event from Note: Only the name of the queue
DeadLetterQueue The name of the specific Dead letter queue where the pickup service will put invalid events and/or events that can't be logged
ReplaceRules Review the shared ReplaceRules section for additional information

ServiceBusQueues

The ServiceBusQueues section is where you manage Microsoft Azure Service Bus Queue related sources.

ServiceBusQueues is an array of a ServiceBus queue. Each configuration is valid for one Service Bus queue. This configuration tells Log Agent where to look for log messages with the Nodinite JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false This feature means that you don't have to remove the configuration to disable the pickup service temporarily
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
ConnectionString Service Bus Connection string Endpoint=sb://nodinite.com/servicebus.windows.net/;SharedAccessKeyName=yoursecretkey;SharedAccessKey=ib2+ofZaFyEhbwzgn1nAhw2XDY3UVc1JefdlGgLGIbb=" Copy from the Azure portal
Queue The name of the Service Bus Queue from where to pickup JSON Log Event from Note: Only the name of the queue
ReplaceRules Review the shared ReplaceRules section for additional information

SQLServers

The SQLServers section is where you manage Microsoft SQL Server related sources.

SQLServers is an array of a SQLServer object. Each SQLServer object points to one Nodinite specific SQL Server table where the Log Agent should go look for log rows with the JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false This feature means that you don't have to remove the configuration to disable the pickup service temporarily
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
ConnectionString SQL Server Database Connection string Server=myServerAddress;Database=myDataBase;Integrated Security=SSPI;Connection Timeout=60
MaxRowsLimit The maximum number of rows to fetch on each loop 500 (default) Using larger values will impact the memory used
ReplaceRules Review the shared ReplaceRules section for additional information

SQL Server table

If you are using a PostgreSQL database to store Log Events then your custom logging solution must have the following LogEvents table defined as:

Column Datatype Purpose
LogEvent json Your JSON encoded Log Event goes here
Id bigint Automatically created identity, the PickupService must know which row to flag as invalid if value in LogEvent column is not valid
ValidationFailed boolean Automatically set to false when inserted and might later be changed to true during processing if any errors are detected
ValidationFailedText text Column data is Updated if errors occur validating the LogEvent during processing
Created timestamp Automatically created during insert, might come in handy when troubleshooting logged but invalid log events

Invalid Log Events will get the ValidationFailed column set to True and you must rid these entries manually.

-- Table: [LogEvents], used by Nodinite Logging

-- DROP TABLE [dbo].[LogEvents];

CREATE TABLE [dbo].[LogEvents](
	[Id] [bigint] IDENTITY(1,1) NOT NULL,
	[LogEvent] [nvarchar](max) NOT NULL,
	[ValidationFailed] [bit] NOT NULL,
	[ValidationFailedText] [nvarchar](max) NULL,
	[Created] [datetimeoffset](7) NOT NULL,
 CONSTRAINT [PK_LogEvents] PRIMARY KEY NONCLUSTERED 
(
	[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 7) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]

GO

ALTER TABLE [dbo].[LogEvents] ADD  CONSTRAINT [DF_LogEvents_ValidationFailed]  DEFAULT ((0)) FOR [ValidationFailed]
GO

ALTER TABLE [dbo].[LogEvents] ADD  CONSTRAINT [DF_LogEvents_Created]  DEFAULT (sysutcdatetime()) FOR [Created]
GO

-- Do not forget to assign user access rights

Remember to assign the service account for the Pickup Service the following rights:

  • db_datareader
  • db_datawriter

PostgreSQLs

The PostgreSQLs section is where you manage PostgreSQL related sources.

PostgreSQLs is an array of a PostgreSQL object. Each PostgreSQL object points to one Nodinite specific postgreSQL table where the Log Agent should go look for log rows with JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false This feature means that you don't have to remove the configuration to disable the pickup service temporarily
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
ConnectionString PostgreSQL Connection string Server=replaceme.databases.appdomain.cloud;Port=31902;Database=ibmclouddb;User Id=ibm_cloud_id;Password=yoursecretpassword;SSLMode=Prefer;Trust Server Certificate=true
MaxRowsLimit The maximum number of rows to fetch on each loop 500 (default) Using larger values will impact the memory used
ReplaceRules Review the shared ReplaceRules section for additional information

PostgreSQL table

If you are using a PostgreSQL database to store Log Events then your custom logging solution must have the following LogEvents table defined as:

Column Datatype Purpose
LogEvent json Your JSON encoded Log Event goes here
Id bigint Automatically created identity, the PickupService must know which row to flag as invalid if value in LogEvent column is not valid
ValidationFailed boolean Automatically set to false when inserted and might later be changed to true during processing if any errors are detected
ValidationFailedText text Column data is Updated if errors occur validating the LogEvent during processing
Created timestamp Automatically created during insert, might come in handy when troubleshooting logged but invalid log events

Invalid Log Events will get the ValidationFailed column set to True and you must rid these entries manually.

-- Table: public."LogEvents"

-- DROP TABLE public."LogEvents";

CREATE TABLE public."LogEvents"
(
    "LogEvent" json NOT NULL,
    "Id" bigint NOT NULL GENERATED BY DEFAULT AS IDENTITY ( INCREMENT 1 START 1 MINVALUE 1 MAXVALUE 9223372036854775807 CACHE 1 ),
    "ValidationFailed" boolean NOT NULL DEFAULT false,
    "ValidationFailedText" text NULL,    
    "Created" timestamp with time zone NOT NULL DEFAULT now(),
    CONSTRAINT "LogEvents_pkey" PRIMARY KEY ("Id")
)
WITH (
    OIDS = FALSE
)
TABLESPACE pg_default;

ALTER TABLE public."LogEvents"
    OWNER to admin;

GRANT ALL ON TABLE public."LogEvents" TO usergoeshere;

-- Repeat grants as needed

Make sure to change the value for the 'usergoeshere'

EventHubs

The EventHubs section is where you manage Azure EventHub related sources. The JSON Log Events on the event hub may come from logging using a Policy from your API management entry points, please review the 'LOG AZURE API MANAGEMENT REQUESTS' user guide.

EventHubs is an array of a EventHub object. Each EventHub object points to one EventHub where the Log Agent should look for log rows with JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false You don't have to remove the whole EventHub object to temporarily disable logging
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
EventHubConnectionString Connection string for EventHub Endpoint=sb://replaceme.servicebus.windows.net/;SharedAccessKeyName=[KeyName];SharedAccessKey=[Key];EntityPath=[EntityPathName]
EnableAmqpLinkRedirect A flag used to control the EnableAmqpLinkRedirect property true (default) - currently not honored. Default is True Review more information here
EventHubName Name of the EventHub If EntityPath=[EntityPathName] is not provided in the connection string then you must provide the name using this entry
StorageAccountName Storage Acccount Name Please review the Storage Account Name section below
StorageAccountKey The access key to the storage account Please review the Storage Account Keys section below
StorageContainerName Name of the storage Checkpoints are being used and these are written to the storage
ReplaceRules Review the shared ReplaceRules section for additional information

Note about EnableAmqpLinkRedirect

Clients that use AMQP connections over TCP require ports 5671 and 5672 to be opened in the local firewall. Along with these ports, it might be necessary to open additional ports if the EnableLinkRedirect feature is enabled. EnableLinkRedirect is a new messaging feature that helps skip one-hop while receiving messages, thus helping to boost throughput. The client would start communicating directly with the back-end service over port range 104XX.

A .NET client would fail with a SocketException ("An attempt was made to access a socket in a way forbidden by its access permissions") if these ports are blocked by the firewall. The feature can be disabled by setting EnableAmqpLinkRedirect=false in the connectiong string, which forces the clients to communicate with the remote service over port 5671.

Storage Account Name

The StorageAccountName can be found as pictured below in the Azure Portal:
Storage Account Name

Storage Account Keys

The StorageAccountKey can be found as pictured below in the Azure Portal:
Access Keys

AnypointMQs

The AnypointMQs section is where you manage Mulesoft Cloudhub AnypointMQ related sources.

AnypointMQs is an array of an AnypointMQ object. Each AnypointMQ object points to one AnypointMQ queue where the Log Agent should look for utf-8 encoded JSON Log Events.

Property Description Value Example Comment
Enabled A flag used to disable or enable logging true or false This feature means that you don't have to remove the configuration to disable the pickup service temporarily
LogAPI The address to the Nodinite LogAPI https://localhost/Nodinite/LogAPI/ Change according to your setup
UseConfigurationDatabase A flag used to bypass the LogAPI true or false Use this option only for high throughput solutions. Enabling this feature may increase the overall load on the system
ConfigurationDatabaseConnectionString Connection string to Configuration database Server=localhost;Database=NodiniteConfig_Dev;Integrated Security=SSPI;Connection Timeout=60
ActiveConsumers Number of threads to run 1 Currently limited to 1 thread
Username The name of the user to logon with nodinite-reader
Password Password for user 1337
Environment The name of the environment Test
EnvironmentId Identifier for environment f7a16b7b-1337-4cb5-93ba-1e5289f707ec Guid
OrganizationId Identifier for organization 65ec82b6-f930-4ea2-1337-7c56ee24c9d3 Guid
RegionId Name of region North America Northern Virginia (us-east-1), Oregon (us-west-2), Central Canada (ca-central-1),
Europe Ireland (eu-west-1), London (eu-west-2)
APAC Singapore (ap-southeast-1), Sydney (ap-southeast-2)
According to your run-time, for example eu-west-1
Queue Name of the queue with JSON Log Events Nodinite-LogEvents
DeadLetterQueue Name of queue for invalid Log Events Nodinite-LogEvents.DLQ
ReplaceRules Review the shared ReplaceRules section for additional information

Replace Rules

This is a shared section for many of the configuration options above ReplaceRules is an array, of Rules to update provided End Points with a new Name or Uri if they are dynamic with for example id:s

Property Description Value Example Comment
Name The name of the rule to know what they are for
ReplaceName A flag to indicate whether to fix the End Point Name true (default) or false
ReplaceUri A flag to indicate whether to fix the End Point Uri true (default) or false
Pattern A RegEx pattern to match the value to replace note! only value in the first group is replaced /([0-9]{4,})$ Find numbers
Group1Value The new value to write instead of the match {customerId} To replace matched value with

Example Original URI was logged as: (actual id differs on most unique calls)

https://www.nodinite.com/int042/getCustomer/12345

With the example configuration the logged URI value is logged as:

https://www.nodinite.com/int042/getCustomer/{CustomerId}

If you did not configure ReplaceRules before deploying your solution you may end up with an awful lot of Endpoints, contact our support to get help tidying up.

Example configuration file

In the Troubleshooting section, there is an example of a fully populated configuration file.

Next Step

JSON Log Event

Install Log Agent - Pickup Service
Log Agents
Logging Service