AWS S3

Overview

AWS S3 downloads and processes log files from an S3 bucket using the AWS SDK for S3.

Supported platforms

• Linux: Logs
• Windows: Logs
• macOS: Logs

Authentication

AWS S3 supports AWS credentials.

Parameter	Type	Description
credential_type	option	aws or awsimds.
access_key_id	string	AWS access key (required if using aws).
secret_access_key	string	AWS secret key (required if using aws).
session_token	string	Optional session token for temporary credentials.

Note: The Credential Type field is required if not using default AWS credentials.

S3 Downloader

Parameter	Type	Default Value	Required	Description
region	string	none	Yes	The AWS region where your S3 bucket is located.
s3_bucket	string	none	Yes	The name of the S3 bucket to download from.
s3_prefix	string	none	No	The prefix path in the S3 bucket to search for objects.
s3_partition_format	string	year=%Y/month=%m/day=%d/hour=%H/minute=%M	No	Format for the partition key.
s3_partition_timezone	string	UTC	No	The timezone for partitioning (IANA timezone name applied when formatting the partition key).
file_prefix	string	none	No	Optional prefix to filter files in the S3 bucket.
file_prefix_include_telemetry_type	boolean	false	No	Whether to include telemetry type in the file prefix.

Data Selection

Parameter	Type	Default Value	Required	Description
value	integer	1	Yes	Number of units to look back. Max allowed is 4 weeks (1 month).
unit	string	week	Yes	Time unit for the lookback window.

Polling

Parameter	Type	Default Value	Required	Description
poll_interval	string	10s	No	Time between polls for new objects when using time-based polling.
encoding	string	none	No	Optional OpenTelemetry encoding extension wired into the S3 receiver: none, text_encoding, aws_logs_encoding, or json_log_encoding.
aws_logs_format	string	none	When encoding is aws_logs_encoding	format value for the AWS logs encoding extension (for example s3access, cloudtrail, vpcflow). The rendered collector registers the extension as aws_logs_encoding/<format> with an encodings suffix chosen from this format: vpcflow and elbaccess use suffix .log (matches standard S3 keys such as ...log / ...log.gz after decompress); cloudtrail, waf, cloudwatch, networkfirewall, and s3access use suffix .json.
json_log_encoding_mode	string	body	When encoding is json_log_encoding	mode for the JSON log encoding extension: body or body_with_inline_attributes.
json_log_encoding_array_mode	boolean	true	No	Maps to the extension array_mode field (default true).
logs_payload_format	string	default	No	When set to json_array, maps to the awss3 receiver logs_payload_format field (for example AWS Config snapshots stored as a JSON object with a single array of records).
logs_json_array_field	string	none	When logs_payload_format is json_array	Maps to logs_json_array_field in the generated collector config (for example configurationItems).
decompress	string	auto	No	Whether to decompress log files if they are compressed.
max_objects_per_scan	integer	10	No	Maximum number of objects to process in a single poll cycle.
max_bytes_per_object	integer	1048576	No	Maximum number of bytes to read from a single object.
max_log_size	integer	1048576	No	Maximum size of a single log entry in bytes.

State Management

Parameter	Type	Default Value	Required	Description
persist_offsets	boolean	true	No	Persist S3 object reading offsets to resume from where left off.

Example Configuration

{
  "s3downloader": {
    "region": "us-west-2", // required
    "s3_bucket": "my-logs-bucket", // required
    "s3_prefix": "logs/", // default: none
    "s3_partition_format": "year=%Y/month=%m/day=%d/hour=%H/minute=%M", // default
    "s3_partition_timezone": "UTC", // default
  },

  "lookback": {
    "value": 1, // required, default: 1
    "unit": "week", // required, default: "week"
  },

  "poll_interval": "10s", // default: "10s"
  "encoding": "none", // none | text_encoding | aws_logs_encoding | json_log_encoding
  // When encoding is aws_logs_encoding:
  // "aws_logs_format": "s3access",
  // When encoding is json_log_encoding:
  // "json_log_encoding_mode": "body",
  // "json_log_encoding_array_mode": true,
  // Optional JSON-array snapshot style (AWS Config configurationItems, etc.):
  // "logs_payload_format": "json_array",
  // "logs_json_array_field": "configurationItems",
  "decompress": "auto", // default: "auto"
  "max_objects_per_scan": 10,
  "max_bytes_per_object": 1048576,
  "max_log_size": 1048576,

  "persist_offsets": true, // default: true
}