github.com/minio/minio@v0.0.0-20240328213742-3f72439b8a27/docs/debugging/README.md (about)

     1  # MinIO Server Debugging Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
     2  
     3  ## HTTP Trace
     4  
     5  HTTP tracing can be enabled by using [`mc admin trace`](https://github.com/minio/mc/blob/master/docs/minio-admin-complete-guide.md#command-trace---display-minio-server-http-trace) command.
     6  
     7  Example:
     8  
     9  ```sh
    10  minio server /data
    11  ```
    12  
    13  Default trace is succinct only to indicate the API operations being called and the HTTP response status.
    14  
    15  ```sh
    16  mc admin trace myminio
    17  ```
    18  
    19  To trace entire HTTP request
    20  
    21  ```sh
    22  mc admin trace --verbose myminio
    23  ```
    24  
    25  To trace entire HTTP request and also internode communication
    26  
    27  ```sh
    28  mc admin trace --all --verbose myminio
    29  ```
    30  
    31  ## Subnet Health
    32  
    33  Subnet Health diagnostics help ensure that the underlying infrastructure that runs MinIO is configured correctly, and is functioning properly. This test is one-shot long running one, that is recommended to be run as soon as the cluster is first provisioned, and each time a failure scenario is encountered. Note that the test incurs majority of the available resources on the system. Care must be taken when using this to debug failure scenario, so as to prevent larger outages. Health tests can be triggered using `mc support diagnostics` command.
    34  
    35  Example:
    36  
    37  ```sh
    38  minio server /data{1...4}
    39  ```
    40  
    41  The command takes no flags
    42  
    43  ```sh
    44  mc support diagnostics myminio/
    45  ```
    46  
    47  The output printed will be of the form
    48  
    49  ```sh
    50  ● Admin Info ... ✔ 
    51  ● CPU ... ✔ 
    52  ● Disk Hardware ... ✔ 
    53  ● Os Info ... ✔ 
    54  ● Mem Info ... ✔ 
    55  ● Process Info ... ✔ 
    56  ● Config ... ✔ 
    57  ● Drive ... ✔ 
    58  ● Net ... ✔ 
    59  *********************************************************************************
    60                                     WARNING!!
    61       ** THIS FILE MAY CONTAIN SENSITIVE INFORMATION ABOUT YOUR ENVIRONMENT ** 
    62       ** PLEASE INSPECT CONTENTS BEFORE SHARING IT ON ANY PUBLIC FORUM **
    63  *********************************************************************************
    64  mc: Health data saved to dc-11-health_20200321053323.json.gz
    65  ```
    66  
    67  The gzipped output contains debugging information for your system
    68  
    69  ## Decoding Metadata
    70  
    71  Metadata is stored in `xl.meta` files for erasure coded objects. Each disk in the set containing the object has this file. The file format is a binary format and therefore requires tools to view values.
    72  
    73  ### Installing xl-meta
    74  
    75  To install, [Go](https://golang.org/dl/) must be installed. Once installed, execute this to install the binary:
    76  
    77  ```bash
    78  go install github.com/minio/minio/docs/debugging/xl-meta@latest
    79  ```
    80  
    81  ### Using xl-meta
    82  
    83  Executing `xl-meta` will look for an `xl.meta` in the current folder and decode it to JSON. It is also possible to specify multiple files or wildcards, for example `xl-meta ./**/xl.meta` will output decoded metadata recursively. It is possible to view what inline data is stored inline in the metadata using `--data` parameter `xl-meta -data xl.json` will display an id -> data size. To export inline data to a file use the `--export` option.
    84  
    85  ### Remotely Inspecting backend data
    86  
    87  `mc support inspect` allows collecting files based on *path* from all backend drives. Matching files will be collected in a zip file with their respective host+drive+path. A MinIO host from October 2021 or later is required for full functionality. Syntax is `mc support inspect ALIAS/path/to/files`. This can for example be used to collect `xl.meta` from objects that are misbehaving. To collect `xl.meta` from a specific object, for example placed at `ALIAS/bucket/path/to/file.txt` append `/xl.meta`, for instance `mc support inspect ALIAS/bucket/path/to/file.txt/xl.meta`. All files can be collected, so this can also be used to retrieve `part.*` files, etc.
    88  
    89  Wildcards can be used, for example `mc support inspect ALIAS/bucket/path/**/xl.meta` will collect all `xl.meta` recursively. `mc support inspect ALIAS/bucket/path/to/file.txt/*/part.*` will collect parts for all versions for the object located at `bucket/path/to/file.txt`.
    90  
    91  `xl-meta` accepts zip files as input and will output all `xl.meta` files found within the archive. For example:
    92  
    93  ```
    94  $ mc support inspect play/test123/test*/xl.meta
    95  mc: File data successfully downloaded as inspect.6f96b336.zip
    96  $ xl-meta inspect.6f96b336.zip
    97  {
    98          "bf6178f9-4014-4008-9699-86f2fac62226/test123/testw3c.pdf/xl.meta": {"Versions":[{"Type":1,"V2Obj":{"ID":"aGEA/ZUOR4ueRIZsAgfDqA==","DDir":"9MMwM47bS+K6KvQqN3hlDw==","EcAlgo":1,"EcM":2,"EcN":2,"EcBSize":1048576,"EcIndex":4,"EcDist":[4,1,2,3],"CSumAlgo":1,"PartNums":[1],"PartETags":[""],"PartSizes":[101974],"PartASizes":[176837],"Size":101974,"MTime":1634106631319256439,"MetaSys":{"X-Minio-Internal-compression":"a2xhdXNwb3N0L2NvbXByZXNzL3My","X-Minio-Internal-actual-size":"MTc2ODM3","x-minio-internal-objectlock-legalhold-timestamp":"MjAyMS0xMC0xOVQyMjozNTo0Ni4zNTE4MDU3NTda"},"MetaUsr":{"x-amz-object-lock-mode":"COMPLIANCE","x-amz-object-lock-retain-until-date":"2022-10-13T06:30:31.319Z","etag":"67ed8f49b7137cb957858ce468f2e79e","content-type":"application/pdf","x-amz-object-lock-legal-hold":"OFF"}}}]},
    99          "fe012443-6ba9-4ef2-bb94-b729d2060c78/test123/testw3c.pdf/xl.meta": {"Versions":[{"Type":1,"V2Obj":{"ID":"aGEA/ZUOR4ueRIZsAgfDqA==","DDir":"9MMwM47bS+K6KvQqN3hlDw==","EcAlgo":1,"EcM":2,"EcN":2,"EcBSize":1048576,"EcIndex":1,"EcDist":[4,1,2,3],"CSumAlgo":1,"PartNums":[1],"PartETags":[""],"PartSizes":[101974],"PartASizes":[176837],"Size":101974,"MTime":1634106631319256439,"MetaSys":{"X-Minio-Internal-compression":"a2xhdXNwb3N0L2NvbXByZXNzL3My","X-Minio-Internal-actual-size":"MTc2ODM3","x-minio-internal-objectlock-legalhold-timestamp":"MjAyMS0xMC0xOVQyMjozNTo0Ni4zNTE4MDU3NTda"},"MetaUsr":{"content-type":"application/pdf","x-amz-object-lock-legal-hold":"OFF","x-amz-object-lock-mode":"COMPLIANCE","x-amz-object-lock-retain-until-date":"2022-10-13T06:30:31.319Z","etag":"67ed8f49b7137cb957858ce468f2e79e"}}}]},
   100          "5dcb9f38-08ea-4728-bb64-5cecc7102436/test123/testw3c.pdf/xl.meta": {"Versions":[{"Type":1,"V2Obj":{"ID":"aGEA/ZUOR4ueRIZsAgfDqA==","DDir":"9MMwM47bS+K6KvQqN3hlDw==","EcAlgo":1,"EcM":2,"EcN":2,"EcBSize":1048576,"EcIndex":2,"EcDist":[4,1,2,3],"CSumAlgo":1,"PartNums":[1],"PartETags":[""],"PartSizes":[101974],"PartASizes":[176837],"Size":101974,"MTime":1634106631319256439,"MetaSys":{"X-Minio-Internal-compression":"a2xhdXNwb3N0L2NvbXByZXNzL3My","X-Minio-Internal-actual-size":"MTc2ODM3","x-minio-internal-objectlock-legalhold-timestamp":"MjAyMS0xMC0xOVQyMjozNTo0Ni4zNTE4MDU3NTda"},"MetaUsr":{"content-type":"application/pdf","x-amz-object-lock-legal-hold":"OFF","x-amz-object-lock-mode":"COMPLIANCE","x-amz-object-lock-retain-until-date":"2022-10-13T06:30:31.319Z","etag":"67ed8f49b7137cb957858ce468f2e79e"}}}]},
   101          "48beacc7-4be0-4660-9026-4eceaf147504/test123/testw3c.pdf/xl.meta": {"Versions":[{"Type":1,"V2Obj":{"ID":"aGEA/ZUOR4ueRIZsAgfDqA==","DDir":"9MMwM47bS+K6KvQqN3hlDw==","EcAlgo":1,"EcM":2,"EcN":2,"EcBSize":1048576,"EcIndex":3,"EcDist":[4,1,2,3],"CSumAlgo":1,"PartNums":[1],"PartETags":[""],"PartSizes":[101974],"PartASizes":[176837],"Size":101974,"MTime":1634106631319256439,"MetaSys":{"X-Minio-Internal-compression":"a2xhdXNwb3N0L2NvbXByZXNzL3My","X-Minio-Internal-actual-size":"MTc2ODM3","x-minio-internal-objectlock-legalhold-timestamp":"MjAyMS0xMC0xOVQyMjozNTo0Ni4zNTE4MDU3NTda"},"MetaUsr":{"x-amz-object-lock-retain-until-date":"2022-10-13T06:30:31.319Z","x-amz-object-lock-legal-hold":"OFF","etag":"67ed8f49b7137cb957858ce468f2e79e","content-type":"application/pdf","x-amz-object-lock-mode":"COMPLIANCE"}}}]}
   102  }
   103  ```
   104  
   105  Optionally `--encrypt` can be specified. This will output an encrypted file and a decryption key:
   106  
   107  ```
   108  $ mc support inspect --encrypt play/test123/test*/*/part.*
   109  mc: Encrypted file data successfully downloaded as inspect.ad2b43d8.enc
   110  mc: Decryption key: ad2b43d847fdb14e54c5836200177f7158b3f745433525f5d23c0e0208e50c9948540b54
   111  
   112  mc: The decryption key will ONLY be shown here. It cannot be recovered.
   113  mc: The encrypted file can safely be shared without the decryption key.
   114  mc: Even with the decryption key, data stored with encryption cannot be accessed.
   115  ```
   116  
   117  This file can be decrypted using the decryption tool below:
   118  
   119  ### Installing decryption tool
   120  
   121  To install, [Go](https://golang.org/dl/) must be installed.
   122  
   123  Once installed, execute this to install the binary:
   124  
   125  ```bash
   126  go install github.com/minio/minio/docs/debugging/inspect@latest
   127  ```
   128  
   129  ### Usage
   130  
   131  To decrypt the file above:
   132  
   133  ```
   134  $ inspect -key=ad2b43d847fdb14e54c5836200177f7158b3f745433525f5d23c0e0208e50c9948540b54 inspect.ad2b43d8.enc
   135  Output decrypted to inspect.ad2b43d8.zip
   136  ```
   137  
   138  If `--key` is not specified an interactive prompt will ask for it. The file name will contain the beginning of the key. This can be used to verify that the key is for the encrypted file.