github.com/pachyderm/pachyderm@v1.13.4/doc/docs/1.11.x/how-tos/developer-workflow/index.md (about)

     1  # Developer Workflow
     2  
     3  In general, the developer workflow for Pachyderm involves adding 
     4  data to versioned data repositories, creating pipelines to 
     5  read from those repositories, executing the pipeline's code, and writing the pipeline's output to other data repositories.
     6  Both the data and pipeline can be iterated on independently with Pachyderm
     7  handling the code execution according to the pipeline specfication.
     8  The workflow steps are shown below.
     9  
    10  ![Developer workflow](../../assets/images/d_steps_analysis_pipeline.svg)
    11  
    12  ## Data Workflow - Load Your Data into Pachyderm
    13  
    14  You need to add your data to Pachyderm so that your pipeline runs your code
    15  against it. You can do so by using one of the following methods:
    16  
    17  * By using the `pachctl put file` command
    18  * By using a special type of pipeline, such as a [spout](../../concepts/pipeline-concepts/pipeline/spout/) or [cron](../../concepts/pipeline-concepts/pipeline/cron/) 
    19  * By using one of the Pachyderm's [language clients](../../../reference/clients/)
    20  * By using a compatible S3 client
    21  
    22  For more information, see [Load Your Data Into Pachyderm](../load-data-into-pachyderm/).
    23  
    24  ## Pipeline Workflow - Processing Data in Pachyderm
    25  
    26  The fundamental concepts of Pachyderm are very powerful, but the manual build steps mentioned in the [pipeline workflow](working-with-pipelines.md) can become cumbersome during rapid-iteration development cycles. We've created a few helpful developer workflows and tools to automate steps that are error-prone or repetitive:
    27  
    28  * [Build Pipelines](build-pipelines.md) map code changes into a the pipeline using a default base Docker image without rebuilding it. They are most useful when iterating on the code, with few changes to the Docker image.
    29  * The [build flag](build-flag.md) or `--build` is a optional flag that can be passed to the `create` or `update` pipeline command. This option is most useful when you need to customize your Docker image or are iterating on the Docker image and code together, since it rebuilds and pushes the image before updating the pipeline. 
    30  * [CI/CD Integration](ci-cd-integration.md) provides a way to incorporate Pachyderm functions into the CI process. This is most useful when working with a complex project or for code collaboration. 
    31  * [create_python_pipeline](https://pachyderm.github.io/python-pachyderm/python_pachyderm.m.html#python_pachyderm.create_python_pipeline) is Python-specific way to quickly update pipelines and was the predecessor to Build Pipelines. They are only available for Python via the [Python Pachyderm](https://github.com/pachyderm/python-pachyderm) package. This tool can be useful when using the [Pachyderm IDE](../use-pachyderm-ide).