AWS S3

Overview

AWS S3 uploads log batches to an S3 bucket.

Supported types: Logs · Metrics · Traces

Authentication Configuration

AWS S3 supports AWS access-key, assume-role, and IMDS credentials.

Parameter	Type	Required	Description
credential_type	option	true	aws for access keys or assume role; awsimds for EC2 metadata credentials.
auth_method	option	false	Use access_keys or iam_role for aws credentials.
access_key_id	string	false	AWS access key. Required for access_keys; optional source creds for role.
secret_access_key	string	false	AWS secret key. Required for access_keys; optional source creds for role.
session_token	string	false	Session token for temporary credentials.
role_arn	string	false	IAM role ARN to assume when auth_method is iam_role.
role_session_name	string	false	Optional session name for the assumed role.
profile	string	false	Optional AWS shared config profile.
imds_endpoint	string	false	Optional IMDS endpoint for awsimds credentials.

Note: The Credential Type field is required if not using default AWS credentials.

Configuration

Parameter	Type	Default Value	Required	Description
region	string	none	Yes	The AWS region where the S3 bucket is located.
s3_bucket	string	none	Yes	The name of the S3 bucket to upload to.
s3_prefix	string	none	Yes	The prefix path in the S3 bucket for uploaded objects.
s3_partition_format	string	%Y/%m/%d/%H/%M	No	Partition format for uploaded objects.
timeout	string	5s	No	Timeout for upload operations.

Sending Queue Settings

Parameter	Type	Default Value	Description
queue_size	int64	1000	Maximum number of log entries to hold in the queue.
num_consumers	int	10	Number of parallel workers processing logs from the queue.
enable_disk_backed_queue	bool	true	Saves the queue to disk so data can survive restarts.

Note: Sending Queue fields available if enabled.

Example Configuration

{
  "s3uploader": {
    "region": "us-west-2", // required
    "s3_bucket": "my-output-bucket", // required
    "s3_prefix": "output/", // required
    "s3_partition_format": "%Y/%m/%d/%H/%M",
  },

  "timeout": "30s",

  "sending_queue": {
    // conditional
    "queue_size": 5000,
    "num_consumers": 10,
    "enable_disk_backed_queue": true,
  },
}