Log Deduplication
Overview
The Log deduplication processor is used to identify and remove duplicate log entries from a pipeline. When enabled, logs will be deduplicated based on the defined fields.
Supported types: Logs
Configuration
| Parameter | Type | Default Value | Required | Description |
|---|---|---|---|---|
| interval | duration | 10s | Yes | The interval at which logs are aggregated. The counter will reset after each interval. |
| log_count_attribute | string | log_count | No | The name of the count attribute of deduplicated logs that will be added to the emitted aggregated log. |
| timezone | string | UTC | Yes | The timezone of the first_observed_timestamp and last_observed_timestamp timestamps on the emitted aggregated log. The available locations depend on the local IANA Time Zone database. See the List of tz database time zones for available values. |
| enable_dedup_strategy | bool | false | No | If enabled, logs will be deduplicated based on the defined fields. |
Note: The entire body cannot be excluded. If the body is a map then fields within it can be excluded.
Deduplication Settings
When Enable Deduplication Strategy is set to true, the following settings are available:
Choose how to determine a log's unique identity
EXCLUDE Mode
| Parameter | Type | Default Value | Required | Description |
|---|---|---|---|---|
| exclude_fields | list | No | Fields to exclude from duplication matching. Fields can be excluded from the log body or attributes. These fields will not be present in the emitted aggregated log. Nested fields must be . delimited. This option is mutually exclusive with include_fields. |
INCLUDE Mode
| Parameter | Type | Default Value | Required | Description |
|---|---|---|---|---|
| include_fields | list | No | Fields to include in duplication matching. Fields can be from the log body or attributes. Nested fields must be . delimited. If a field contains a . it can be escaped by using a . This option is mutually exclusive with exclude_fields. |
Example Configuration
{
"interval": "10s", // default: "10s"
"timezone": "UTC", // default: "UTC"
"log_count_attribute": "log_count", // default: "log_count"
"enable_dedup_strategy": false, // default: false
"dedup_strategy_mode": "EXCLUDE", // default: "EXCLUDE"
"exclude_fields": [], // default: none
"include_fields": [], // default: none
}