github.com/yankunsam/loki/v2@v2.6.3-0.20220817130409-389df5235c27/docs/sources/operations/storage/_index.md (about)

     1  ---
     2  title: Storage
     3  weight: 40
     4  ---
     5  # Grafana Loki Storage
     6  
     7  [High level storage overview here]({{< relref "../../storage/_index.md" >}})
     8  
     9  Grafana Loki needs to store two different types of data: **chunks** and **indexes**.
    10  
    11  Loki receives logs in separate streams, where each stream is uniquely identified
    12  by its tenant ID and its set of labels. As log entries from a stream arrive,
    13  they are compressed as "chunks" and saved in the chunks store. See [chunk
    14  format](#chunk-format) for how chunks are stored internally.
    15  
    16  The **index** stores each stream's label set and links them to the individual
    17  chunks.
    18  
    19  Refer to Loki's [configuration](../../configuration/) for details on
    20  how to configure the storage and the index.
    21  
    22  For more information:
    23  
    24  1. [Table Manager](table-manager/)
    25  1. [Retention](retention/)
    26  1. [Logs Deletion](logs-deletion/)
    27  
    28  ## Supported Stores
    29  
    30  The following are supported for the index:
    31  
    32  - [Single Store (boltdb-shipper) - Recommended for 2.0 and newer](boltdb-shipper/) index store which stores boltdb index files in the object store
    33  - [Amazon DynamoDB](https://aws.amazon.com/dynamodb)
    34  - [Google Bigtable](https://cloud.google.com/bigtable)
    35  - [Apache Cassandra](https://cassandra.apache.org)
    36  - [BoltDB](https://github.com/boltdb/bolt) (doesn't work when clustering Loki)
    37  
    38  The following are supported for the chunks:
    39  
    40  - [Amazon DynamoDB](https://aws.amazon.com/dynamodb)
    41  - [Google Bigtable](https://cloud.google.com/bigtable)
    42  - [Apache Cassandra](https://cassandra.apache.org)
    43  - [Amazon S3](https://aws.amazon.com/s3)
    44  - [Google Cloud Storage](https://cloud.google.com/storage/)
    45  - [Filesystem](filesystem/) (please read more about the filesystem to understand the pros/cons before using with production data)
    46  - [Baidu Object Storage](https://cloud.baidu.com/product/bos.html)
    47  
    48  ## Cloud Storage Permissions
    49  
    50  ### S3
    51  
    52  When using S3 as object storage, the following permissions are needed:
    53  
    54  - `s3:ListBucket`
    55  - `s3:PutObject`
    56  - `s3:GetObject`
    57  - `s3:DeleteObject` (if running the Single Store (boltdb-shipper) compactor)
    58  
    59  Resources: `arn:aws:s3:::<bucket_name>`, `arn:aws:s3:::<bucket_name>/*`
    60  
    61  ### DynamoDB
    62  
    63  When using DynamoDB for the index, the following permissions are needed:
    64  
    65  - `dynamodb:BatchGetItem`
    66  - `dynamodb:BatchWriteItem`
    67  - `dynamodb:DeleteItem`
    68  - `dynamodb:DescribeTable`
    69  - `dynamodb:GetItem`
    70  - `dynamodb:ListTagsOfResource`
    71  - `dynamodb:PutItem`
    72  - `dynamodb:Query`
    73  - `dynamodb:TagResource`
    74  - `dynamodb:UntagResource`
    75  - `dynamodb:UpdateItem`
    76  - `dynamodb:UpdateTable`
    77  - `dynamodb:CreateTable`
    78  - `dynamodb:DeleteTable` (if `table_manager.retention_period` is more than 0s)
    79  
    80  Resources: `arn:aws:dynamodb:<aws_region>:<aws_account_id>:table/<prefix>*`
    81  
    82  - `dynamodb:ListTables`
    83  
    84  Resources: `*`
    85  
    86  #### AutoScaling
    87  
    88  If you enable autoscaling from table manager, the following permissions are needed:
    89  
    90  ##### Application Autoscaling
    91  
    92  - `application-autoscaling:DescribeScalableTargets`
    93  - `application-autoscaling:DescribeScalingPolicies`
    94  - `application-autoscaling:RegisterScalableTarget`
    95  - `application-autoscaling:DeregisterScalableTarget`
    96  - `application-autoscaling:PutScalingPolicy`
    97  - `application-autoscaling:DeleteScalingPolicy`
    98  
    99  Resources: `*`
   100  
   101  ##### IAM
   102  
   103  - `iam:GetRole`
   104  - `iam:PassRole`
   105  
   106  Resources: `arn:aws:iam::<aws_account_id>:role/<role_name>`
   107  
   108  
   109  ## Chunk Format
   110  
   111  ```
   112    -------------------------------------------------------------------
   113    |                               |                                 |
   114    |        MagicNumber(4b)        |           version(1b)           |
   115    |                               |                                 |
   116    -------------------------------------------------------------------
   117    |         block-1 bytes         |          checksum (4b)          |
   118    -------------------------------------------------------------------
   119    |         block-2 bytes         |          checksum (4b)          |
   120    -------------------------------------------------------------------
   121    |         block-n bytes         |          checksum (4b)          |
   122    -------------------------------------------------------------------
   123    |                        #blocks (uvarint)                        |
   124    -------------------------------------------------------------------
   125    | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
   126    -------------------------------------------------------------------
   127    | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
   128    -------------------------------------------------------------------
   129    | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
   130    -------------------------------------------------------------------
   131    | #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) |
   132    -------------------------------------------------------------------
   133    |                      checksum(from #blocks)                     |
   134    -------------------------------------------------------------------
   135    |           metasOffset - offset to the point with #blocks        |
   136    -------------------------------------------------------------------
   137  ```
   138