AWS S3

Overview

AWS S3 downloads and processes log files from an S3 bucket using the AWS SDK for S3.

Supported platforms

• Linux: Logs
• Windows: Logs
• macOS: Logs

Authentication

AWS S3 supports AWS credentials.

Parameter	Type	Description
credential_type	option	aws or awsimds.
access_key_id	string	AWS access key (required if using aws).
secret_access_key	string	AWS secret key (required if using aws).
session_token	string	Optional session token for temporary credentials.

Note: The Credential Type field is required if not using default AWS credentials.

S3 Downloader

Parameter	Type	Default Value	Required	Description
region	string	none	Yes	The AWS region where your S3 bucket is located.
s3_bucket	string	none	Yes	The name of the S3 bucket to download from.
s3_prefix	string	none	No	The prefix path in the S3 bucket to search for objects. A path segment containing * matches folder names at that level (* never crosses a /), so one source can read many sibling folders, e.g. o-abc/AWSLogs/o-abc/*/CloudTrail/us-east-1 reads every account. Matching folders are re-discovered every 5 minutes; each keeps its own progress and a new folder is read from the lookback start. Up to 1000 folders, and max_objects_per_scan applies per folder. Requires a collector build with wildcard support.
s3_partition_format	string	none (empty)	No	Format for the partition key, e.g. %Y/%m/%d for AWS service delivery or year=%Y/month=%m/day=%d/hour=%H/minute=%M. Leave it empty (or set none) for objects that are not stored under date folders (e.g. CloudWatch export tasks): every object under the prefix, at any depth, is read once, oldest first by S3 last-modified time. Each poll lists the whole prefix, so prefer a longer poll_interval for large buckets. Requires a collector build with flat-mode support.
s3_partition_timezone	string	UTC	No	The timezone for partitioning (IANA timezone name applied when formatting the partition key).
file_prefix	string	none	No	Optional prefix to filter files in the S3 bucket.
file_prefix_include_telemetry_type	boolean	false	No	Whether to include telemetry type in the file prefix.

Data Selection

Parameter	Type	Default Value	Required	Description
value	integer	1	Yes	Number of units to look back. Max allowed is 4 weeks (1 month).
unit	string	week	Yes	Time unit for the lookback window.

Polling

Parameter	Type	Default Value	Required	Description
poll_interval	string	10s	No	Time between polls for new objects when using time-based polling.
encoding	string	none	No	Optional OpenTelemetry encoding extension wired into the S3 receiver: none, text_encoding, aws_logs_encoding, or json_log_encoding.
aws_logs_format	string	none	When encoding is aws_logs_encoding	format value for the AWS logs encoding extension (for example s3access, cloudtrail, vpcflow). The rendered collector registers the extension as aws_logs_encoding/<format> with an encodings suffix chosen from this format: vpcflow, elbaccess, and waf use suffix .log (matches standard S3 keys such as ...log / ...log.gz after decompress); cloudtrail, cloudwatch, networkfirewall, and s3access use suffix .json.
json_log_encoding_mode	string	body	When encoding is json_log_encoding	mode for the JSON log encoding extension: body or body_with_inline_attributes.
json_log_encoding_array_mode	boolean	true	No	Maps to the extension array_mode field (default true). Set to false for JSON Lines (one object per line), such as GuardDuty exports.
file_suffixes	list	none	No	Object key endings the selected encoding is applied to, compared after .gz is removed (a typed .jsonl.gz is treated as .jsonl unless decompress is none). Empty uses the encoding defaults: .json and .jsonl for json_log_encoding, .txt for text_encoding, and the per-format suffix above for aws_logs_encoding. * applies the encoding to every object under the prefix. Objects that fail to decode are skipped and counted as parse errors. Ignored when encoding is none.
logs_payload_format	string	default	No	When set to json_array, maps to the awss3 receiver logs_payload_format field (for example AWS Config snapshots stored as a JSON object with a single array of records).
logs_json_array_field	string	none	When logs_payload_format is json_array	Maps to logs_json_array_field in the generated collector config (for example configurationItems). Ignored for any other payload format.
decompress	string	auto	No	Whether to decompress log files if they are compressed.
max_objects_per_scan	integer	10	No	Maximum number of objects to process in a single poll cycle.
max_bytes_per_object	integer	1048576	No	Maximum number of bytes to read from a single object.
max_log_size	integer	1048576	No	Maximum size of a single log entry in bytes.
late_arrival_window	string	15m	No	How long a partition folder stays open for re-scanning after its nominal time. S3 delivery (WAF, ALB, CloudTrail, Firehose) writes objects under a path derived from event time, often minutes later, so a folder that lists empty now can gain objects afterwards. Polling never advances past now - late_arrival_window, which is what keeps those objects reachable. Set to 0s to disable the hold.
max_tracked_keys_per_window	integer	10000	No	Upper bound on the per-folder set of already-read object keys that makes late arrivals detectable. A folder exceeding this collapses to a single ordering cursor, after which a late object sorting below it is skipped. Raise it for very high-volume partitions; 0 means unbounded.

State Management

Parameter	Type	Default Value	Required	Description
persist_offsets	boolean	true	No	Persist S3 object reading offsets to resume from where left off.

Example Configuration

{
  "s3downloader": {
    "region": "us-west-2", // required
    "s3_bucket": "my-logs-bucket", // required
    "s3_prefix": "logs/", // default: none
    "s3_partition_format": "year=%Y/month=%m/day=%d/hour=%H/minute=%M", // default
    "s3_partition_timezone": "UTC", // default
  },

  "lookback": {
    "value": 1, // required, default: 1
    "unit": "week", // required, default: "week"
  },

  "poll_interval": "10s", // default: "10s"
  "encoding": "none", // none | text_encoding | aws_logs_encoding | json_log_encoding
  // When encoding is aws_logs_encoding:
  // "aws_logs_format": "s3access",
  // When encoding is json_log_encoding:
  // "json_log_encoding_mode": "body",
  // "json_log_encoding_array_mode": true,
  // Optional JSON-array snapshot style (AWS Config configurationItems, etc.):
  // "logs_payload_format": "json_array",
  // "logs_json_array_field": "configurationItems",
  "decompress": "auto", // default: "auto"
  "max_objects_per_scan": 10,
  "max_bytes_per_object": 1048576,
  "max_log_size": 1048576,
  // Keeps recent partition folders re-scannable so objects S3 delivers late are
  // not missed. Raise it if your source's delivery lag exceeds 15 minutes.
  "late_arrival_window": "15m", // default: "15m"
  "max_tracked_keys_per_window": 10000, // default: 10000

  "persist_offsets": true, // default: true
}