Documentation Grafana Loki Set up Migrate Migrate to Thanos storage clients

Migrate to Thanos storage clients

Loki has object storage clients based on the Thanos Object Storage Client Go module. These clients were added in Loki 3.4.

One of the reasons for making this change is to have a consistent storage configuration across Grafana Loki, Mimir and other telemetry databases from Grafana Labs. If you are already using Grafana Mimir or Pyroscope, you can reuse the storage configuration for setting up Loki.

Loki 4.0 uses the Thanos-based clients by default, because the use_thanos_objstore setting defaults to true. The legacy storage clients are deprecated. Use this guide to convert your storage configuration to the new format. If you are not ready to migrate, set use_thanos_objstore to false to keep using the legacy clients.

In Loki 3.4 and later 3.x releases, the Thanos-based clients are optional. To use them in those releases, set use_thanos_objstore to true.

Configure the new storage clients

1. Configure your object storage under storage_config.object_store. When the Thanos-based clients are in use, Loki reads this section instead of the legacy storage configuration. In Loki 3.x, you must also set use_thanos_objstore to true in the storage_config section, or set the -use-thanos-objstore flag to true. The following examples set this option explicitly, which is also valid in Loki 4.0.

   YAML Copy

   # Uses the new storage clients for connecting to gcs backend
   storage_config:
     use_thanos_objstore: true # use the Thanos-based storage clients
     object_store:
       gcs:
         bucket_name: "example-bucket"

2. As an alternative, you can also configure the new clients in the common storage section if you prefer to use the common config section.

   YAML Copy

   storage_config:
      use_thanos_objstore: true # use the Thanos-based storage clients
   common:
     storage:
       object_store:
         gcs:
           bucket_name: "example-bucket"

3. Ruler storage should be configured under the ruler_storage section when using the new storage clients.

   YAML Copy

   storage_config:
      use_thanos_objstore: true # use the Thanos-based storage clients
   ruler_storage:
      backend: gcs
      gcs:
         bucket_name: "example-bucket"

4. If you are using store.object-prefix flag or the corresponding object_prefix YAML setting, you’ll need to update your configuration to use the new object_store.storage-prefix flag or the corresponding storage_prefix YAML setting.

   YAML Copy

   # Example configuration to prefix all objects with "prefix"
   storage_config:
      use_thanos_objstore: true # use the Thanos-based storage clients
      object_store:
         storage_prefix: "prefix"

GCS Storage Migration

When migrating from the existing Google Cloud Storage (GCS) storage client to the new Thanos-based client, you’ll need to update your configuration parameters as follows:

Example configuration migration (GCS):

Existing configuration:

YAML Copy

storage_config:
  gcs:
    bucket_name: example-bucket
    chunk_buffer_size: 10MB
    enable_retries: true

New configuration (Thanos-based):

YAML Copy

storage_config:
  use_thanos_objstore: true
  object_store:
    gcs:
      bucket_name: example-bucket
      chunk_buffer_size: 10MB
      max_retries: 5

Amazon S3 Storage Migration

When migrating from the existing Amazon S3 storage client to the new Thanos-based client, update or remove parameters as follows:

Example configuration migration (S3):

Existing configuration

YAML Copy

storage_config:
  aws:
    bucket_names: my-bucket1,my-bucket2   # multiple buckets no longer supported
    endpoint: s3.amazonaws.com
    region: us-west-2
    access_key_id: example-key
    secret_access_key: example-secret
    signature_version: v4
    disable_dualstack: true
    storage_class: STANDARD
    http_config:
      timeout: 1m
      insecure_skip_verify: false
    # ...
    backoff_config:
      max_retries: 5
    sse:
      type: SSE-KMS
      kms_key_id: mySSEKey

New configuration (Thanos-based)

YAML Copy

storage_config:
  use_thanos_objstore: true
  object_store:
    s3:
      bucket_name: my-bucket1                       # single bucket
      endpoint: s3.amazonaws.com
      region: us-west-2
      access_key_id: example-key
      secret_access_key: example-secret
      dualstack_enabled: false                        # was disable_dualstack: true
      storage_class: STANDARD
      max_retries: 5
      http:
        insecure_skip_verify: false
      sse:
        type: SSE-KMS
        kms_key_id: mySSEKey

For more advanced configuration options (such as list_objects_version, bucket_lookup_type, etc.), see the Thanos S3 configuration reference.

Azure Storage Migration

When migrating from the existing Azure storage client to the new Thanos-based client, no changes are required if you are using the following parameters:

If you are using an authentication method other than storage account key or user-assigned managed identity, you’ll have to pass the necessary credentials using environment variables. For more details, refer to Azure Identity Client Module for Go.

Filesystem Storage Migration

When migrating from the existing Filesystem storage client to the new Thanos-based client, update or remove parameters as follows:

Example configuration migration (Filesystem):

Existing configuration (FSConfig)

YAML Copy

storage_config:
  filesystem:
    directory: /var/loki/chunks

New configuration (Thanos-based)

YAML Copy

storage_config:
  use_thanos_objstore: true
  object_store:
    filesystem:
      dir: /var/loki/chunks