github.com/pachyderm/pachyderm@v1.13.4/doc/docs/1.10.x/how-tos/updating_pipelines.md (about)

     1  # Update a Pipeline
     2  
     3  While working with your data, you often need to modify an existing
     4  pipeline with new transformation code or pipeline
     5  parameters.
     6  For example, when you develop a machine learning model, you might need
     7  to try different versions of your model while your training data
     8  stays relatively constant. To make changes to a pipeline, you need to use
     9  the `pachctl update pipeline` command.
    10  
    11  ## Update Your Pipeline Specification
    12  
    13  If you need to update your
    14  [pipeline specification](../reference/pipeline_spec.md), such as change the
    15  parallelism settings, add an input repository, or other, you need to update your
    16  JSON file and then run the `pachctl update pipeline`command.
    17  By default, when you update your code, the new pipeline specification
    18  does not reprocess the data that has already been processed. Instead,
    19  it processes only the new data that you submit to the input repo.
    20  If you want to run the changes in your pipeline against the data in
    21  the `HEAD` commit of your input repo, use the `--reprocess` flag.
    22  After that, the updated pipeline continues to process new input data.
    23  Previous results remain accessible through the corresponding commit IDs.
    24  
    25  To update a pipeline specification, complete the following steps:
    26  
    27  1. Make the changes in your pipeline specification JSON file.
    28  
    29  1. Update the pipeline with the new configuration:
    30  
    31     ```shell
    32     pachctl update pipeline -f pipeline.json
    33     ```
    34  
    35  Similar to `create pipeline`, `update pipeline` with the `-f` flag can also
    36  take a URL if your JSON manifest is hosted on GitHub or other
    37  remote location.
    38  
    39  ## Update the Code in a Pipeline
    40  
    41  The `pachctl update pipeline` updates the code that you use in one or
    42  more of your pipelines. To apply your code changes, you need to
    43  build a new Docker image and push it to your Docker image registry.
    44  
    45  You can either use your registry instructions to build and push your
    46  new image or push the new image by using the flags built into
    47  the `pachctl update pipeline` command. Both procedures achieve the same goal,
    48  and it is entirely a matter of a personal preference which one of them
    49  to follow. If you do not have a build-push process that you
    50  already follow, you might prefer to use Pachyderm's built-in functionality.
    51  
    52  To create a new image by using the Pachyderm commands, you need
    53  to use the `--build` flag with the `pachctl update pipeline`
    54  command. By default, if you do not specify a registry with the `--registry`
    55  flag, Pachyderm uses [DockerHub](https://hub.docker.com).
    56  When you build your image with Pachyderm, it assigns a random
    57  tag to your new image.
    58  
    59  If you use a private registry or any other registry that is different
    60  from the default value, use the `--registry` flag to specify it.
    61  Make sure that you specify the private registry in the [pipeline
    62  specification](../reference/pipeline_spec.md).
    63  
    64  For example, if you want to push a `pachyderm/opencv` image to a
    65  registry located at `localhost:5000`, you need to add this in
    66  your pipeline spec:
    67  
    68   ```shell
    69   "image": "localhost:5000/pachyderm/opencv"
    70   ```
    71  
    72  Also, to be able to build and push images, you need to make sure that
    73  the Docker daemon is running. Depending on your operating system and
    74  the Docker distribution that you use, steps for enabling it might
    75  vary.
    76  
    77  To update the code in your pipeline, complete the following steps:
    78  
    79  1. Make the code changes.
    80  1. Verify that the Docker daemon is running:
    81  
    82     ```shell
    83     docker ps
    84     ```
    85  
    86     * If you get an error message similar to the following:
    87  
    88       ```shell
    89       Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
    90       ```
    91  
    92       enable the Docker daemon. To enable the Docker daemon,
    93       see the Docker documentation for your operating system and platform.
    94       For example, if you use `minikube` on  macOS, run the following
    95       command:
    96  
    97       ```shell
    98       eval $(minikube docker-env)
    99       ```
   100  
   101  1. Build, tag, and push the new image to your image registry:
   102  
   103     * If you prefer to use Pachyderm commands:
   104  
   105       1. Run the following command:
   106  
   107          ```shell
   108          pachctl update pipeline -f <pipeline name> --build --registry <registry> --username <registry user>
   109          ```
   110  
   111          If you use DockerHub, omit the `--registry` flag.
   112  
   113          **Example:**
   114  
   115          ```shell
   116          pachctl update pipeline -f edges.json --build --username testuser
   117          ```
   118  
   119       1. When prompted, type your image registry password:
   120  
   121          **Example:**
   122  
   123          ```
   124          Password for docker.io/testuser: Building pachyderm/opencv:f1e0239fce5441c483b09de425f06b40, this may take a while.
   125          ```
   126  
   127     * If you prefer to use instructions for your image registry:
   128  
   129       1. Build, tag, and push a new image as described in the
   130       image registry documentation. For example, if you use
   131       DockerHub, see [Docker Documentation](https://docs.docker.com/docker-hub/).
   132  
   133       1. Update the pipeline:
   134  
   135          ```shell
   136          pachctl update pipeline -f <pipeline.json>
   137          ```