github.com/pachyderm/pachyderm@v1.13.4/doc/docs/1.9.x/how-tos/err_cmd.md (about)

     1  # Skip Failed Datums
     2  
     3  !!! note "TL;DR"
     4      The `err_cmd` parameter enables you to fail a datum without failing the
     5      whole job.
     6  
     7  !!! note
     8      Before you read this section, make sure that you understand such
     9      concepts as [Datum](../../concepts/pipeline-concepts/datum/) and
    10      [Pipeline](../../concepts/pipeline-concepts/pipeline/).
    11  
    12  When Pachyderm processes your data, it breaks it up into units of
    13  computation called datums. Each datum is processed separately.
    14  In a basic pipeline configuration, a failed datum results in a failed
    15  job. However, in some cases, you might not need all datums
    16  to consider a job successful. If your downstream pipelines can be run
    17  on only the successful datums instead of needing all the datums to be
    18  successful, Pachyderm can mark some datums as *recovered* which means
    19  that they failed with a non-critical error, but the successful datums
    20  will be processed.
    21  
    22  To configure a condition under which you want your failed datums not
    23  to fail the whole job, you can add your custom error code in
    24  `err_cmd` and `err_stdin` fields in your pipeline specification.
    25  
    26  For example, your DAG consists of two pipelines:
    27  
    28  * The pipeline 1 cleans the data.
    29  * The pipeline 2 trains your model by using the data from the first pipeline.
    30  
    31  That means that the second pipeline takes the results of the first pipeline
    32  from its output repository and uses that data to train a model. In some cases,
    33  you might not need all the datums in the first pipeline to be successful
    34  to run the second pipeline.
    35  
    36  The following diagram describes how Pachyderm transformation and error
    37  code work:
    38  
    39  ![err_cmd logic](../assets/images/err_cmd_workflow.svg)
    40  
    41  Here is what is happening in the diagram above:
    42  
    43  1. Pachyderm executes the transformation code that you defined in
    44  the `cmd` field against your datums.
    45  1. If a datum is processed without errors, Pachyderm marks it as
    46  `processed`.
    47  1. If a datum fails, Pachyderm marks executes your
    48  error code (`err_cmd`) on that datum.
    49  1. If the code in `err_cmd` successfully runs on the *skipped* datum,
    50  Pachyderm marks the skipped datum as `recovered`. The datum is in a
    51  failed state and, therefore, the pipeline does not put it into the output
    52  repository, but successful datums continue onto the next step in your DAG.
    53  1. If the `err_cmd` code fails on the skipped datum, the datum is marked
    54  as failed, and, consequently, the job is marked as failed.
    55  
    56  You can view the processed, skipped, and recovered datums in the `PROGRESS`
    57  field in the output of the `pachctl list job` command:
    58  
    59  ![Datums in progress](../assets/images/datums_in_progress.svg)
    60  
    61  Pachyderm writes only processed datums of successful jobs to the output
    62  commit so that these datums can be processed by downstream pipelines.
    63  For example, in your first pipeline, Pachyderm processes three datums.
    64  If one of the datums is marked as *recovered* and two others are
    65  successfully processed, only these two successful datums are used in
    66  the next pipeline.
    67  
    68  !!! note "See Also:"
    69      [Example err_cmd pipeline](https://github.com/pachyderm/pachyderm/tree/master/examples/err_cmd/)