storj.io/minio@v0.0.0-20230509071714-0cbc90f649b1/docs/erasure/storage-class/README.md (about)

     1  # MinIO Storage Class Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io)
     2  
     3  MinIO server supports storage class in erasure coding mode. This allows configurable data and parity disks per object.
     4  
     5  This page is intended as a summary of MinIO Erasure Coding. For a more complete explanation, see https://docs.min.io/minio/baremetal/concepts/erasure-coding.html. 
     6  
     7  ## Overview
     8  
     9  MinIO supports two storage classes, Reduced Redundancy class and Standard class. These classes can be defined using environment variables
    10  set before starting MinIO server. After the data and parity disks for each storage class are defined using environment variables,
    11  you can set the storage class of an object via request metadata field `x-amz-storage-class`. MinIO server then honors the storage class by
    12  saving the object in specific number of data and parity disks.
    13  
    14  ## Storage usage
    15  
    16  The selection of varying data and parity drives has a direct impact on the drive space usage. With storage class, you can optimize for high
    17  redundancy or better drive space utilization.
    18  
    19  To get an idea of how various combinations of data and parity drives affect the storage usage, let’s take an example of a 100 MiB file stored
    20  on 16 drive MinIO deployment. If you use eight data and eight parity drives, the file space usage will be approximately twice, i.e. 100 MiB
    21  file will take 200 MiB space. But, if you use ten data and six parity drives, same 100 MiB file takes around 160 MiB. If you use 14 data and
    22  two parity drives, 100 MiB file takes only approximately 114 MiB.
    23  
    24  Below is a list of data/parity drives and corresponding _approximate_ storage space usage on a 16 drive MinIO deployment. The field _storage
    25  usage ratio_ is simply the drive space used by the file after erasure-encoding, divided by actual file size.
    26  
    27  | Total Drives (N) | Data Drives (D) | Parity Drives (P) | Storage Usage Ratio |
    28  |------------------|-----------------|-------------------|---------------------|
    29  |               16 |               8 |                 8 |                2.00 |
    30  |               16 |               9 |                 7 |                1.79 |
    31  |               16 |              10 |                 6 |                1.60 |
    32  |               16 |              11 |                 5 |                1.45 |
    33  |               16 |              12 |                 4 |                1.34 |
    34  |               16 |              13 |                 3 |                1.23 |
    35  |               16 |              14 |                 2 |                1.14 |
    36  
    37  You can calculate _approximate_ storage usage ratio using the formula - total drives (N) / data drives (D).
    38  
    39  ### Allowed values for STANDARD storage class
    40  
    41  `STANDARD` storage class implies more parity than `REDUCED_REDUNDANCY` class. So, `STANDARD` parity disks should be
    42  
    43  - Greater than or equal to 2, if `REDUCED_REDUNDANCY` parity is not set.
    44  - Greater than `REDUCED_REDUNDANCY` parity, if it is set.
    45  
    46  Parity blocks can not be higher than data blocks, so `STANDARD` storage class parity can not be higher than N/2. (N being total number of disks)
    47  
    48  The default value for the `STANDARD` storage class depends on the number of volumes in the erasure set:
    49  
    50  | Erasure Set Size | Default Parity (EC:N) |
    51  |------------------|-----------------------|
    52  | 5 or fewer       |                 EC:2  |
    53  | 6-7              |                 EC:3  |
    54  | 8 or more        |                 EC:4  |
    55  
    56  Prior to the ``RELEASE.2021-01-30T00-20-58Z`` MinIO release, the default `STANDARD` value was `EC(N/2)` where `N` was the number of erasure set drives.
    57  For more complete documentation on Erasure Set sizing, see the [MinIO Documentation on Erasure Sets](https://docs.min.io/minio/baremetal/concepts/erasure-coding.html#erasure-sets).
    58  
    59  ### Allowed values for REDUCED_REDUNDANCY storage class
    60  
    61  `REDUCED_REDUNDANCY` implies lesser parity than `STANDARD` class. So,`REDUCED_REDUNDANCY` parity disks should be
    62  
    63  - Less than N/2, if `STANDARD` parity is not set.
    64  - Less than `STANDARD` Parity, if it is set.
    65  
    66  As parity below 2 is not recommended, `REDUCED_REDUNDANCY` storage class is not supported for 4 disks erasure coding setup.
    67  
    68  Default value for `REDUCED_REDUNDANCY` storage class is `2`.
    69  
    70  ## Get started with Storage Class
    71  
    72  ### Set storage class
    73  
    74  The format to set storage class environment variables is as follows
    75  
    76  `MINIO_STORAGE_CLASS_STANDARD=EC:parity`
    77  `MINIO_STORAGE_CLASS_RRS=EC:parity`
    78  
    79  For example, set `MINIO_STORAGE_CLASS_RRS` parity 2 and `MINIO_STORAGE_CLASS_STANDARD` parity 3
    80  
    81  ```sh
    82  export MINIO_STORAGE_CLASS_STANDARD=EC:3
    83  export MINIO_STORAGE_CLASS_RRS=EC:2
    84  ```
    85  
    86  Storage class can also be set via `mc admin config` get/set commands to update the configuration. Refer [storage class](https://github.com/minio/minio/tree/master/docs/config#storage-class) for
    87  more details.
    88  
    89  *Note*
    90  
    91  - If `STANDARD` storage class is set via environment variables or `mc admin config` get/set commands, and `x-amz-storage-class` is not present in request metadata, MinIO server will
    92  apply `STANDARD` storage class to the object. This means the data and parity disks will be used as set in `STANDARD` storage class.
    93  
    94  - If storage class is not defined before starting MinIO server, and subsequent PutObject metadata field has `x-amz-storage-class` present
    95  with values `REDUCED_REDUNDANCY` or `STANDARD`, MinIO server uses default parity values.
    96  
    97  ### Set metadata
    98  
    99  In below example `minio-go` is used to set the storage class to `REDUCED_REDUNDANCY`. This means this object will be split across 6 data disks and 2 parity disks (as per the storage class set in previous step).
   100  
   101  ```go
   102  s3Client, err := minio.New("localhost:9000", "YOUR-ACCESSKEYID", "YOUR-SECRETACCESSKEY", true)
   103  if err != nil {
   104  	log.Fatalln(err)
   105  }
   106  
   107  object, err := os.Open("my-testfile")
   108  if err != nil {
   109  	log.Fatalln(err)
   110  }
   111  defer object.Close()
   112  objectStat, err := object.Stat()
   113  if err != nil {
   114  	log.Fatalln(err)
   115  }
   116  
   117  n, err := s3Client.PutObject("my-bucketname", "my-objectname", object, objectStat.Size(), minio.PutObjectOptions{ContentType: "application/octet-stream", StorageClass: "REDUCED_REDUNDANCY"})
   118  if err != nil {
   119  	log.Fatalln(err)
   120  }
   121  log.Println("Uploaded", "my-objectname", " of size: ", n, "Successfully.")
   122  ```