github.com/minio/minio@v0.0.0-20240328213742-3f72439b8a27/docs/compression/README.md (about) 1 # Compression Guide [](https://slack.min.io) 2 3 MinIO server allows streaming compression to ensure efficient disk space usage. 4 Compression happens inflight, i.e objects are compressed before being written to disk(s). 5 MinIO uses [`klauspost/compress/s2`](https://github.com/klauspost/compress/tree/master/s2) 6 streaming compression due to its stability and performance. 7 8 This algorithm is specifically optimized for machine generated content. 9 Write throughput is typically at least 500MB/s per CPU core, 10 and scales with the number of available CPU cores. 11 Decompression speed is typically at least 1GB/s. 12 13 This means that in cases where raw IO is below these numbers 14 compression will not only reduce disk usage but also help increase system throughput. 15 Typically, enabling compression on spinning disk systems 16 will increase speed when the content can be compressed. 17 18 ## Get Started 19 20 ### 1. Prerequisites 21 22 Install MinIO - [MinIO Quickstart Guide](https://min.io/docs/minio/linux/index.html#quickstart-for-linux). 23 24 ### 2. Run MinIO with compression 25 26 Compression can be enabled by updating the `compress` config settings for MinIO server config. 27 Config `compress` settings take extensions and mime-types to be compressed. 28 29 ```bash 30 ~ mc admin config get myminio compression 31 compression extensions=".txt,.log,.csv,.json,.tar,.xml,.bin" mime_types="text/*,application/json,application/xml" 32 ``` 33 34 Default config includes most common highly compressible content extensions and mime-types. 35 36 ```bash 37 ~ mc admin config set myminio compression extensions=".pdf" mime_types="application/pdf" 38 ``` 39 40 To show help on setting compression config values. 41 42 ```bash 43 ~ mc admin config set myminio compression 44 ``` 45 46 To enable compression for all content, no matter the extension and content type 47 (except for the default excluded types) set BOTH extensions and mime types to empty. 48 49 ```bash 50 ~ mc admin config set myminio compression enable="on" extensions="" mime_types="" 51 ``` 52 53 The compression settings may also be set through environment variables. 54 When set, environment variables override the defined `compress` config settings in the server config. 55 56 ```bash 57 export MINIO_COMPRESSION_ENABLE="on" 58 export MINIO_COMPRESSION_EXTENSIONS=".txt,.log,.csv,.json,.tar,.xml,.bin" 59 export MINIO_COMPRESSION_MIME_TYPES="text/*,application/json,application/xml" 60 ``` 61 62 > [!NOTE] 63 > To enable compression for all content when using environment variables, set either or both of the extensions and MIME types to `*` instead of an empty string: 64 > ```bash 65 > export MINIO_COMPRESSION_ENABLE="on" 66 > export MINIO_COMPRESSION_EXTENSIONS="*" 67 > export MINIO_COMPRESSION_MIME_TYPES="*" 68 > ``` 69 70 ### 3. Compression + Encryption 71 72 Combining encryption and compression is not safe in all setups. 73 This is particularly so if the compression ratio of your content reveals information about it. 74 See [CRIME TLS](https://en.wikipedia.org/wiki/CRIME) as an example of this. 75 76 Therefore, compression is disabled when encrypting by default, and must be enabled separately. 77 78 Consult our security experts on [SUBNET](https://min.io/pricing) to help you evaluate if 79 your setup can use this feature combination safely. 80 81 To enable compression+encryption use: 82 83 ```bash 84 ~ mc admin config set myminio compression allow_encryption=on 85 ``` 86 87 Or alternatively through the environment variable `MINIO_COMPRESSION_ALLOW_ENCRYPTION=on`. 88 89 ### 4. Excluded Types 90 91 - Already compressed objects are not fit for compression since they do not have compressible patterns. 92 Such objects do not produce efficient [`LZ compression`](https://en.wikipedia.org/wiki/LZ77_and_LZ78) 93 which is a fitness factor for a lossless data compression. 94 95 Pre-compressed input typically compresses in excess of 2GiB/s per core, 96 so performance impact should be minimal even if precompressed data is re-compressed. 97 Decompressing incompressible data has no significant performance impact. 98 99 Below is a list of common files and content-types which are typically not suitable for compression. 100 101 - Extensions 102 103 | `gz` | (GZIP) | 104 | `bz2` | (BZIP2) | 105 | `rar` | (WinRAR) | 106 | `zip` | (ZIP) | 107 | `7z` | (7-Zip) | 108 | `xz` | (LZMA) | 109 | `mp4` | (MP4) | 110 | `mkv` | (MKV media) | 111 | `mov` | (MOV) | 112 113 - Content-Types 114 115 | `video/*` | 116 | `audio/*` | 117 | `application/zip` | 118 | `application/x-gzip` | 119 | `application/zip` | 120 | `application/x-bz2` | 121 | `application/x-compress` | 122 | `application/x-xz` | 123 124 All files with these extensions and mime types are excluded from compression, 125 even if compression is enabled for all types. 126 127 ## To test the setup 128 129 To test this setup, practice put calls to the server using `mc` and use `mc ls` on 130 the data directory to view the size of the object. 131 132 ## Explore Further 133 134 - [Use `mc` with MinIO Server](https://min.io/docs/minio/linux/reference/minio-mc.html) 135 - [Use `aws-cli` with MinIO Server](https://min.io/docs/minio/linux/integrations/aws-cli-with-minio.html) 136 - [Use `minio-go` SDK with MinIO Server](https://min.io/docs/minio/linux/developers/go/minio-go.html) 137 - [The MinIO documentation website](https://min.io/docs/minio/linux/index.html)