Filelog

Overview

Filelog reads log data directly from files on the filesystem and sends the collected logs into the telemetry pipeline.

Supported platforms

• Linux: Logs
• Windows: Logs
• macOS: Logs

Basic Configuration

Basic configuration defines which log files should be monitored and how they are read.

Parameter	Type	Default Value	Required	Description
include_paths	array[string]	none	Yes	Array of file paths or glob patterns that should be monitored for log reading.
exclude_paths	array[string]	none	No	Array of file paths that should be excluded.
multiline_parsing	string	disabled	No	Enables multiline log parsing using pattern matching. Supported values: disabled, line_start_pattern, line_end_pattern.
start_at	string	end	No	Determines where the source starts reading the log file when it starts. Options: beginning or end.

Note

• line_start_pattern
  Regex pattern that matches the beginning of a log entry for handling multiline logs.
  Example: line_start_pattern: '^\d{4}-\d{2}-\d{2}'
• line_end_pattern
  Regex pattern that matches the end of a log entry, used to determine when a multiline log entry should terminate.
  Example: line_end_pattern: '\.$'

Advanced Configuration

Parameter	Type	Default Value	Description
include_file_name_attr	bool	false	Whether to add the file name as the attribute log.file.name.
include_file_path_attr	bool	false	Whether to add the file path as the attribute log.file.path.
include_file_name_resolved_attr	bool	false	Whether to add the file name after symlinks resolution as the attribute log.file.name_resolved.
include_file_path_resolved_attr	bool	false	Whether to add the file path after symlinks resolution as the attribute log.file.path_resolved.
delete_after_read	bool	false	Deletes files after they are fully read and processed. (Only valid in combination with start_at: beginning.)
encoding	string	utf-8	The encoding of the files being read. Valid values: nop, utf-8, utf-8-raw,utf-16le, utf-16be, ascii, big5.
compression	string	none	Compression format of input files. Supported values: none, gzip, auto.
poll_interval	duration	200ms	The duration between filesystem polls.
force_flush_period	duration	500ms	Time since last time new data was found in the file, after which a partial log at the end of the file may be emitted.
fingerprint_size	string	1kb	Number of bytes used to uniquely identify a file.
max_log_size	string	1MiB	Maximum size of a single log entry that will be read.
max_concurrent_files	int	1024	Maximum number of log files that can be read concurrently.
persist_offsets	bool	true	Persists file reading offsets so logs are not reread after restart.
retry_on_failure	bool	false	Enables automatic retry when sending logs fails.
ordering_criteria	bool	false	Enables file ordering criteria when reading multiple files.

Note: This source offers a delete_after_read option. When this option is combined with file globbing, it will delete every file that matches the globbing pattern. Use with caution and care.

Retry On Failure Configuration

The Retry On Failure option enables automatic retry logic when Filelog fails to send logs. When enabled, the integration temporarily retains the log batch and attempts to resend it based on the configured retry intervals.

Parameter	Type	Default Value	Description
retry_on_failure_initial_interval	duration	1s	Time to wait after the first failure before retrying.
retry_on_failure_max_interval	duration	30s	Upper bound on retry backoff interval. Once this value is reached the delay between consecutive retries will remain constant at the specified value.
retry_on_failure_max_elapsed_time	duration	5m	Maximum amount of time (including retries) spent trying to send a logs batch to a downstream consumer. Once this value is reached, the data is discarded. Retrying never stops if set to 0.

Ordering Criteria Configuration

The Ordering Criteria option controls the order in which log files are processed when multiple files match the configured include_paths.

By default, files are processed in the order they are discovered by the filesystem. When Ordering Criteria is enabled, the source can sort and prioritize files using values extracted from file names, file metadata, or file contents.

Parameter	Type	Default Value	Description
ordering_criteria_regex	string	none	Regular expression used for sorting, should contain named capture groups that are to be used in ordering_criteria_sort_by_regex_key.
ordering_criteria_group_by	string	none	Regular expression used for grouping, which is done pre-sorting. Should contain named capture groups.
ordering_criteria_top_n	int	1	The number of files to track when using file ordering. The top N files are tracked after applying the ordering criteria.
ordering_criteria_sort_by_regex_key	string	none	Regular expression named capture group defined in ordering_criteria_regex to use for sorting.
ordering_criteria_sort_by_sort_type	string	none	Sorting method. Supported values: numeric, alphabetical, timestamp, mtime.
ordering_criteria_sort_by_ascending	bool	true	Determines whether sorting is performed in ascending order.

Example Configuration

{
  "include_paths": "/var/log/system.log", // required

  "exclude_paths": "/var/log/auth.log",

  "multiline_parsing": "disabled", // default: "disabled"
  "line_start_pattern": "", // default: none
  "line_end_pattern": "", // default: none

  "start_at": "end", // default: "end"

  "include_file_name_attr": false, // default: false
  "include_file_path_attr": false, // default: false
  "include_file_name_resolved_attr": false, // default: false
  "include_file_path_resolved_attr": false, // default: false

  "delete_after_read": false, // default: false

  "encoding": "utf-8", // default: "utf-8"
  "compression": "", // default: ""

  "poll_interval": "200ms", // default: "200ms"
  "force_flush_period": "500ms", // default: "500ms"

  "fingerprint_size": "1kb", // default: "1kb"
  "max_log_size": "1MiB", // default: "1MiB"
  "max_concurrent_files": 1024, // default: 1024

  "persist_offsets": true, // default: true

  "retry_on_failure": false, // default: false
  "retry_on_failure_initial_interval": "1s", // default: "1s"
  "retry_on_failure_max_interval": "30s", // default: "30s"
  "retry_on_failure_max_elapsed_time": "5m", // default: "5m"

  "ordering_criteria": false, // default: false
  "ordering_criteria_regex": "", // default: none
  "ordering_criteria_group_by": "", // default: none
  "ordering_criteria_top_n": 1, // default: 1
  "ordering_criteria_sort_by_regex_key": "", // default: none
  "ordering_criteria_sort_by_sort_type": "", // default: none
  "ordering_criteria_sort_by_ascending": true, // default: true
}