github.com/pachyderm/pachyderm@v1.13.4/doc/docs/1.11.x/how-tos/use-pachyderm-ide/using-pachyderm-ide.md (about)

     1  # Using `python-pachyderm` with Pachyderm IDE
     2  
     3  If you deployed Pachyderm IDE,
     4  the `python-pachyderm` client is preinstalled in your Pachyderm IDE instance
     5  This section describes a few basic operations that you can execute from Pachyderm IDE
     6  to interact with Pachyderm.
     7  
     8  After you log in, use the [python-pachyderm](https://pachyderm.github.io/python-pachyderm/python_pachyderm.m.html#header-functions)
     9  client API to manage Pachyderm directly from your Jupyter notebook.
    10  
    11  You need to create a new Notebook and add your code in a new cell.
    12  To run your code, click the **Run** button.
    13  
    14  The following code initializes the Python Pachyderm client in Pachyderm IDE:
    15  
    16  ```python
    17  import python_pachyderm
    18  client = python_pachyderm.Client.new_in_cluster()
    19  ```
    20  
    21  !!! note
    22      This function is different from the function you'd call locally.
    23  
    24  For example, you can check the current user by
    25  running the following code:
    26  
    27  ```python
    28  import python_pachyderm
    29  client = python_pachyderm.Client.new_in_cluster()
    30  print(client.who_am_i())
    31  ```
    32  
    33  The following screenshot demonstrates how this looks in Pachyderm IDE:
    34  ![JupyterHub whoami](../../assets/images/s_jupyterhub_whoami.png)
    35  
    36  !!! note
    37      If you have not enabled Pachyderm authentication, this
    38      code returns an error.
    39  
    40  ## Create a Pipeline
    41  
    42  As discussed in [Difference in Pipeline Creation Methods](../../use-pachyderm-ide/#difference-in-pipeline-creation-methods),
    43  you can use the standard `create_pipeline` method or `create_python_pipeline`
    44  function to create Pachyderm pipelines in Pachyderm IDE. Depending on your
    45  choice, use one of the following examples to create a pipeline:
    46  
    47  * By using the `create_python_pipeline` method:
    48  
    49    ```python
    50    import python_pachyderm
    51    client = python_pachyderm.Client.new_in_cluster()
    52  
    53    def relpath(path):
    54        return os.path.join(os.path.dirname(os.path.abspath(__file__)), path)
    55  
    56    python_pachyderm.create_python_pipeline(
    57        client,
    58        relpath("test"),
    59        python_pachyderm.Input(pfs=python_pachyderm.PFSInput(glob="/*", repo="input_repo")),
    60    )
    61    client.list_pipeline()
    62    ```
    63  
    64    This code likely will not work as is. To run a pipeline, you need to specify
    65    `main.py` and `requirements.txt` files in the root directory of your
    66    JupyterHub notebook. For more information, see the
    67    [OpenCV example](https://github.com/pachyderm/python-pachyderm/blob/master/examples/opencv/opencv.py).
    68  
    69  * By using the `create_pipeline` method:
    70  
    71    !!! note
    72        The input repository must exist. Therefore, in the
    73        code below we first create the repository and then
    74        create the pipeline.
    75  
    76    ```python
    77    import python_pachyderm
    78    client = python_pachyderm.Client.new_in_cluster()
    79    client.create_repo('test')
    80  
    81    client.create_pipeline(
    82        "test",
    83        transform=python_pachyderm.Transform(cmd=["python3", "/test.py"], image="mytest/testimage"),
    84        input=python_pachyderm.Input(pfs=python_pachyderm.PFSInput(glob="/", repo="input_repo")),
    85    )
    86    client.list_pipeline()
    87    ```
    88  
    89    **System response:**
    90  
    91     ```
    92     pipeline_info {
    93      pipeline {
    94        name: "test"
    95      }
    96      transform {
    97        image: "mytest/testimage"
    98        cmd: "python3"
    99        cmd: "/test.py"
   100      }
   101      created_at {
   102        seconds: 1578520420
   103        nanos: 789017291
   104      }
   105      version: 1
   106      output_branch: "master"
   107      resource_requests {
   108        memory: "64M"
   109      }
   110      input {
   111        pfs {
   112          name: "input_repo"
   113          repo: "input_repo"
   114          branch: "master"
   115          glob: "/"
   116        }
   117      }
   118      cache_size: "64M"
   119      salt: "8e57267114c24419a685e250e0fd491b"
   120      max_queue_size: 1
   121      spec_commit {
   122        repo {
   123          name: "__spec__"
   124        }
   125        id: "c83631246a2142cdb93c7a6e4d16fcd2"
   126      }
   127      datum_tries: 3
   128     }
   129     ```
   130  
   131  For more information, see the
   132  [OpenCV example](https://github.com/pachyderm/python-pachyderm/blob/master/examples/opencv/opencv.py).
   133  
   134  ## Create a Repository
   135  
   136  To create a repository, run the following code:
   137  
   138  ```python
   139  import python_pachyderm
   140  client = python_pachyderm.Client.new_in_cluster()
   141  client.create_repo('<repo-name>')
   142  client.list_repo()
   143  ```
   144  
   145  **Example:**
   146  
   147  ```python
   148  import python_pachyderm
   149  client = python_pachyderm.Client.new_in_cluster()
   150  client.create_repo('test')
   151  client.list_repo()
   152  ```
   153  
   154  **System Response:**
   155  
   156  ```
   157  [repo {
   158  name: "test"
   159  }
   160  created {
   161    seconds: 1576869000
   162    nanos: 886123695
   163  }
   164  auth_info {
   165    access_level: OWNER
   166  }
   167  ]
   168  ```
   169  
   170  ## Add Files to a Repository
   171  
   172  To add a file to a repository, run the following code:
   173  
   174  ```python
   175  client.put_file_url("<repo-name>/<branch>", "<filename>", "<path-to-file>")
   176  ```
   177  
   178  **Example:**
   179  
   180  ```python
   181  client.put_file_url("images/master", "46Q8nDz.jpg", "http://imgur.com/46Q8nDz.jpg")
   182  ```
   183  
   184  ## Delete a Repository
   185  
   186  To delete a repository, run the following code:
   187  
   188  ```python
   189  import python_pachyderm
   190  client = python_pachyderm.Client.new_in_cluster()
   191  client.delete_repo('test')
   192  client.list_repo()
   193  ```
   194  
   195  **System Response:**
   196  
   197  ```
   198  []
   199  ```
   200  
   201  ## Update Your Pipeline
   202  
   203  When you need to update your pipeline, you can do so directly in the
   204  Pachyderm IDE by modifying the corresponding notebook and running it
   205  again. If you use the `create_python_pipeline` function that uses the
   206  code stored in a local directory, you can update the pipeline directly
   207  in the Pachyderm IDE by adding the `update=True` parameter to your
   208  code into a new Jupyter notebook cell and running it.
   209  
   210  **Example:**
   211  
   212  ```python hl_lines="8"
   213  import os
   214  import python_pachyderm
   215  client = python_pachyderm.Client.new_in_cluster()
   216  python_pachyderm.create_python_pipeline(
   217      client,
   218      "./edges",
   219      python_pachyderm.Input(pfs=python_pachyderm.PFSInput(glob="/*", repo="images")),
   220      update=True
   221  )
   222  ```
   223  
   224  If you are using the standard `create_pipeline` method,
   225  you need to rebuild and push your Docker container to your image
   226  registry. Then, you need to update the image tag in your pipeline
   227  creation code and run `create_pipeline` with `update=True`.
   228  
   229  ## Example
   230  
   231  To get started with Pachyderm IDE, try the
   232  [OpenCV example for JupyterHub](https://github.com/pachyderm/jupyterhub-pachyderm/blob/master/doc/opencv.md).
   233  
   234  This example walks you through the same steps as in the
   235  [Beginner Tutorial](../../../getting_started/beginner_tutorial/) but using the
   236  `python-pachyderm` client instead of `pachctl` or the Pachyderm UI.