github.com/pachyderm/pachyderm@v1.13.4/doc/docs/1.9.x/enterprise/stats.md (about)

     1  # Advanced Statistics
     2  
     3  To use the advanced statistics features in Pachyderm Enterprise Edition, you need to:
     4  
     5  1. Run your pipelines on a Pachyderm cluster that has activated Enterprise
     6     features. See [Deploying Enterprise Edition](deployment.md).
     7  2. Enable stats collection in your pipelines by including
     8     `"enable_stats": true` in your [pipeline specification](https://docs.pachyderm.com/latest/reference/pipeline_spec/#enable-stats-optional).
     9  
    10  Advanced statistics provides the following information for any jobs
    11  corresponding to your pipelines:
    12  
    13  - The amount of data that was uploaded and downloaded during the job and on a per-datum
    14    level.
    15  - The time spend uploading and downloading data on a per-datum level.
    16  - The amount of data uploaded and downloaded on a per-datum level.
    17  - The total time spend processing on a per-datum level.
    18  - Success/failure information on a per-datum level.
    19  - The directory structure of input data that was seen by the job.
    20  
    21  The primary and recommended way to view this information is via the
    22  Pachyderm Enterprise dashboard. However, the same information is
    23  available through the `pachctl inspect datum` and `pachctl list datum`
    24  commands or through their language client equivalents.
    25  
    26  !!! note
    27      Pachyderm recommends that you enable stats for all of your pipelines
    28      and only disabling the feature for very stable, long-running pipelines.
    29      In most cases, the debugging and maintenance benefits of the stats data
    30      outweigh any disadvantages of storing the extra data associated with
    31      the stats. Also note, none of your data is duplicated in producing the stats.
    32  
    33  ## Enabling Stats for a Pipeline
    34  
    35  As mentioned above, enabling stats collection for a pipeline is as simple as
    36  adding the `"enable_stats": true` field to a pipeline specification. For
    37  example, to enable stats collection for the [OpenCV demo pipeline](../getting_started/beginner_tutorial.md),
    38  modify the pipeline specification as follows:
    39  
    40  !!! example
    41  
    42      ```shell
    43      {
    44        "pipeline": {
    45          "name": "edges"
    46        },
    47        "input": {
    48          "pfs": {
    49            "glob": "/*",
    50            "repo": "images"
    51          }
    52        },
    53        "transform": {
    54          "cmd": [ "python3", "/edges.py" ],
    55          "image": "pachyderm/opencv"
    56        },
    57        "enable_stats": true
    58      }
    59      ```
    60  
    61  Once the pipeline has been created and you have use it to process data,
    62  you can confirm that stats are being collected with `list file`. There
    63  should now be stats data in the output repo of the pipeline under a
    64  branch called `stats`:
    65  
    66  !!! example
    67  
    68      ```shell
    69      $ pachctl list file edges@stats
    70      NAME                                                               TYPE                SIZE
    71      002f991aa9db9f0c44a92a30dff8ab22e788f86cc851bec80d5a74e05ad12868   dir                 342.7KiB
    72      0597f2df3f37f1bb5b9bcd6397841f30c62b2b009e79653f9a97f5f13432cf09   dir                 1.177MiB
    73      068fac9c3165421b4e54b358630acd2c29f23ebf293e04be5aa52c6750d3374e   dir                 270.3KiB
    74      0909461500ce508c330ca643f3103f964a383479097319dbf4954de99f92f9d9   dir                 109.6KiB
    75      ...
    76      ```
    77  
    78  ## Accessing Stats Through the Dashboard
    79  
    80  If you have deployed and activated the Pachyderm Enterprise
    81  dashboard, you can explore advanced statistics. For example, if you
    82  navigate to the `edges` pipeline, you might see something similar to this:
    83  
    84  ![alt tag](../assets/images/stats1.png)
    85  
    86  In this example case, you can see that the pipeline has 1 recent successful
    87  job and 2 recent job failures. Pachyderm advanced stats can be very helpful
    88  in debugging these job failures. When you click on one of the job failures,
    89  can see general stats about the failed job, such as total time, total data
    90  upload/download, and so on:
    91  
    92  ![alt tag](../assets/images/stats2.png)
    93  
    94  To get more granular per-datum stats, click on the `41 datums total`, to get
    95  the following information:
    96  
    97  ![alt tag](../assets/images/stats3.png)
    98  
    99  You can identify the exact datums that caused the pipeline to fail, as well
   100  as the associated stats:
   101  
   102  - Total time
   103  - Time spent downloading data
   104  - Time spent processing
   105  - Time spent uploading data
   106  - Amount of data downloaded
   107  - Amount of data uploaded
   108  
   109  If we need to, you can even go a level deeper and explore the exact details
   110  of a failed datum. Clicking on one of the failed datums reveals the logs
   111  that corresponds to the datum processing failure along with the exact input
   112  files of the datum:
   113  
   114  ![alt tag](../assets/images/stats4.png)