github.com/pachyderm/pachyderm@v1.13.4/doc/mkdocs-master.yml (about) 1 site_name: Pachyderm Documentation 2 site_description: A Pachyderm documentation website that provides information on how to use Pachyderm data version control products. 3 site_author: Pachyderm 4 site_url: https://docs.pachyderm.com/ 5 docs_dir: docs/master/ 6 site_dir: site/master/ 7 8 repo_name: pachyderm/pachyderm 9 repo_url: https://github.com/pachyderm/pachyderm 10 edit_uri: "" 11 12 # Copyright 13 copyright: 'Copyright Pachyderm Inc, 2020' 14 pach_latest_version: '1.12.0' 15 16 # Configuration 17 theme: 18 name: pachydermdocs 19 custom_dir: overrides 20 21 # 404 page 22 static_templates: 23 - 404.html 24 include_search_page: false 25 search_index_only: true 26 language: en 27 features: 28 - tabs 29 palette: 30 primary: #4B2A5C 31 accent: #4B2A5C 32 font: 33 text: Robotol 34 code: Roboto Mono 35 favicon: assets/images/favicon-32x32.png 36 icon: 37 logo: material/school 38 39 # Plugins 40 plugins: 41 - search 42 - minify: 43 minify_html: true 44 - git-revision-date-localized 45 - macros 46 47 # Customization 48 extra: 49 social: 50 - icon: fontawesome/brands/github 51 link: https://pachdm.com/github 52 - icon: fontawesome/brands/twitter 53 link: https://pachdm.com/twitter 54 - icon: fontawesome/brands/linkedin 55 link: https://pachdm.com/linkedin 56 57 extra_css: 58 - stylesheets/extra.css 59 60 # Extensions 61 markdown_extensions: 62 - markdown.extensions.admonition 63 - markdown.extensions.codehilite: 64 guess_lang: false 65 - markdown.extensions.def_list 66 - markdown.extensions.footnotes 67 - markdown.extensions.meta 68 - markdown.extensions.toc: 69 permalink: true 70 - pymdownx.arithmatex 71 - pymdownx.betterem: 72 smart_enable: all 73 - pymdownx.caret 74 - pymdownx.critic 75 - pymdownx.details 76 - pymdownx.emoji: 77 emoji_index: !!python/name:materialx.emoji.twemoji 78 emoji_generator: !!python/name:materialx.emoji.to_svg 79 - pymdownx.inlinehilite 80 - pymdownx.keys 81 - pymdownx.magiclink 82 - pymdownx.mark 83 - pymdownx.smartsymbols 84 - pymdownx.superfences 85 - pymdownx.tabbed 86 - pymdownx.tasklist: 87 custom_checkbox: true 88 - pymdownx.tilde 89 90 # Page tree 91 nav: 92 - Home: 93 - Welcome: index.md 94 - Contributing: 95 - Setup for Contributors: contributing/setup.md 96 - Gcloud Cluster Setup: contributing/gcloud-setup.md 97 - Repo Layout: contributing/repo-layout.md 98 - Coding Conventions: contributing/coding-conventions.md 99 - Documentation Style Guide: contributing/docs-style-guide.md 100 - Developing Pachyderm in Windows with VSCode: contributing/windows.md 101 - Getting Started: 102 - Overview: getting_started/index.md 103 - Local Installation: getting_started/local_installation.md 104 - Install pachctl Autocompletion: getting_started/install-pachctl-completion.md 105 - Deploy Pachyderm on Windows: getting_started/wsl-deploy.md 106 - Beginner Tutorial: getting_started/beginner_tutorial.md 107 - Concepts: 108 - Overview: concepts/index.md 109 - Versioned Data Concepts: 110 - Overview: concepts/data-concepts/index.md 111 - Repository: concepts/data-concepts/repo.md 112 - Commit: concepts/data-concepts/commit.md 113 - Branch: concepts/data-concepts/branch.md 114 - File: concepts/data-concepts/file.md 115 - Provenance: concepts/data-concepts/provenance.md 116 - History: concepts/data-concepts/history.md 117 - Pipeline Concepts: 118 - Overview: concepts/pipeline-concepts/index.md 119 - Pipeline: 120 - Overview: concepts/pipeline-concepts/pipeline/index.md 121 - Cron: concepts/pipeline-concepts/pipeline/cron.md 122 - Service: concepts/pipeline-concepts/pipeline/service.md 123 - Spout: concepts/pipeline-concepts/pipeline/spout.md 124 - Job: concepts/pipeline-concepts/job.md 125 - Datum: 126 - Overview: concepts/pipeline-concepts/datum/index.md 127 - Glob Patterns: concepts/pipeline-concepts/datum/glob-pattern.md 128 - Datum Processing: concepts/pipeline-concepts/datum/relationship-between-datums.md 129 - Cross and Union: concepts/pipeline-concepts/datum/cross-union.md 130 - Join: concepts/pipeline-concepts/datum/join.md 131 - Group: concepts/pipeline-concepts/datum/group.md 132 - Datum Processing States: concepts/pipeline-concepts/datum/datum-processing-states.md 133 - Advanced Concepts: 134 - Deferred Processing of Data: concepts/advanced-concepts/deferred_processing.md 135 - Distributed Computing: concepts/advanced-concepts/distributed_computing.md 136 - How-Tos: 137 - Overview: how-tos/index.md 138 - Workflow: 139 - Overview: how-tos/developer-workflow/index.md 140 - Working with Pipelines: how-tos/developer-workflow/working-with-pipelines.md 141 - Build Pipelines: how-tos/developer-workflow/build-pipelines.md 142 - Build Flag: how-tos/developer-workflow/build-flag.md 143 - CI/CD Integration: how-tos/developer-workflow/ci-cd-integration.md 144 - Load Your Data Into Pachyderm: how-tos/load-data-into-pachyderm.md 145 - Export Your Data From Pachyderm: 146 - Overview: how-tos/export-data-out-pachyderm/index.md 147 - Export Your Data with pachctl: how-tos/export-data-out-pachyderm/export-data-pachctl.md 148 - Export Your Data with egress: how-tos/export-data-out-pachyderm/export-data-egress.md 149 - Mount a Repo to a Local Computer: how-tos/export-data-out-pachyderm/mount-repo-to-local-computer.md 150 - Split Data: 151 - Overview: how-tos/splitting-data/index.md 152 - Splitting Data for Distributed Processing: how-tos/splitting-data/splitting.md 153 - Adjusting Data Processing by Splitting Data: how-tos/splitting-data/adjusting_data_processing_w_split.md 154 - Create a Machine Learning Workflow: how-tos/create-ml-workflow.md 155 - Use Pachyderm IDE: 156 - Overview: how-tos/use-pachyderm-ide/index.md 157 - Using the Pachyderm IDE with python-pachyderm: how-tos/use-pachyderm-ide/using-pachyderm-ide.md 158 - Mount a Volume: how-tos/mount-volume.md 159 - Pipeline Operations: 160 - Test your datums: concepts/pipeline-concepts/datum/glob-pattern/#test-your-datums 161 - Create a Pipeline: how-tos/create-pipeline.md 162 - Run a Pipeline on a Specific Commit: how-tos/run_pipeline.md 163 - Update a Pipeline: how-tos/updating_pipelines.md 164 - Delete a Pipeline: how-tos/delete-pipeline.md 165 - Monitor Job Progress: how-tos/monitor-job-progress.md 166 - Processing Time-Windowed Data: how-tos/time_windows.md 167 - Use Transactions: how-tos/use-transactions-to-run-multiple-commands.md 168 - Delete Data: how-tos/removing_data_from_pachyderm.md 169 - Ingress and Egress Data from an External Object Store: how-tos/ingressing_from_diff_cloud.md 170 - Skip Failed Datums: how-tos/err_cmd.md 171 - Deploy and Manage: 172 - Overview: deploy-manage/index.md 173 - Deploy Pachyderm: 174 - Overview: deploy-manage/deploy/index.md 175 - Getting Started with Pachyderm Hub: pachub/pachub_getting_started.md 176 - Deploy on GKE: deploy-manage/deploy/google_cloud_platform.md 177 - Deploy on AWS: 178 - Overview: deploy-manage/deploy/amazon_web_services/index.md 179 - Deploy Pachyderm on Amazon EKS: deploy-manage/deploy/amazon_web_services/deploy-eks.md 180 - Deploy Kubernetes with kops: deploy-manage/deploy/amazon_web_services/aws-deploy-kubernetes-kops.md 181 - Deploy Pachyderm on AWS: deploy-manage/deploy/amazon_web_services/aws-deploy-pachyderm.md 182 - Deploy a Pachyderm Cluster with CloudFront: deploy-manage/deploy/amazon_web_services/aws_cloudfront.md 183 - Deploy on Azure: deploy-manage/deploy/azure.md 184 - Deploy on IBM CLoud: deploy-manage/deploy/ibmcloud-openshift.md 185 - Deploy on OpenShift: deploy-manage/deploy/openshift.md 186 - Deploy On-Premises: deploy-manage/deploy/on_premises.md 187 - Deploy Pachyderm IDE: deploy-manage/deploy/deploy-pachyderm-ide.md 188 - Create a Custom Deployment: 189 - Overview: deploy-manage/deploy/deploy_custom/index.md 190 - Before You Begin: deploy-manage/deploy/deploy_custom/deploy_custom_before_you_begin.md 191 - Pachyderm Deployment Manifest: deploy-manage/deploy/deploy_custom/deploy_custom_pachyderm_deployment_manifest.md 192 - Configuring Persistent Disk Parameters: deploy-manage/deploy/deploy_custom/deploy_custom_configuring_persistent_disk_parameters.md 193 - Configuring Object Store: deploy-manage/deploy/deploy_custom/deploy_custom_configuring_object_store.md 194 - Create a Complete Configuration: deploy-manage/deploy/deploy_custom/deploy_custom_complete_example_invocation.md 195 - Additional flags: deploy-manage/deploy/deploy_custom/deploy_custom_additional_flags.md 196 - Additional Customizations: 197 - Import a Kubernetes Context: deploy-manage/deploy/import-kubernetes-context.md 198 - Deploy Pachyderm with TLS: deploy-manage/deploy/deploy_w_tls.md 199 - Deploy in a Custom Namespace: deploy-manage/deploy/namespaces.md 200 - Deploy a Custom Object Store: deploy-manage/deploy/non-cloud-object-stores.md 201 - Configure RBAC: deploy-manage/deploy/rbac.md 202 - Configure Tracing with Jaeger: deploy-manage/deploy/tracing.md 203 - Enable logs aggregation with Loki: deploy-manage/deploy/loki.md 204 - Connect to a Pachyderm cluster: deploy-manage/deploy/connect-to-cluster.md 205 - Configure Environment Variables: deploy-manage/deploy/environment-variables.md 206 - Manage Pachyderm: 207 - Overview: deploy-manage/manage/index.md 208 - Manage Cluster Access: deploy-manage/manage/cluster-access.md 209 - Configure External Access: 210 - Overview: deploy-manage/manage/configure-external-access.md 211 - deploy-manage/manage/expose-pach-ui-ingress.md 212 - Using Pachctl Shell: deploy-manage/manage/pachctl_shell.md 213 - Autoscale your Cluster: deploy-manage/manage/autoscaling.md 214 - Backup and Restore: deploy-manage/manage/backup_restore.md 215 - Storage Use Optimization: deploy-manage/manage/data_management.md 216 - Use GPUs: deploy-manage/manage/gpus.md 217 - Sharing GPU Resources: deploy-manage/manage/sharing_gpu_resources.md 218 - Upgrades and Migrations: 219 - Overview: deploy-manage/manage/upgrades_migrations.md 220 - Migrate to a Major Version: deploy-manage/manage/migrations.md 221 - Upgrade your Cluster: deploy-manage/manage/upgrades.md 222 - Disable Usage Metrics: deploy-manage/manage/disable-metrics.md 223 - Using the S3 Gateway: 224 - Overview: deploy-manage/manage/s3gateway/index.md 225 - Configure the S3 Gateway: deploy-manage/manage/s3gateway/configure-s3client.md 226 - Create an S3-enabled Pipeline: deploy-manage/manage/s3gateway/deploy-s3gateway-sidecar.md 227 - Supported Operations: deploy-manage/manage/s3gateway/supported-operations.md 228 - Unsupported Operations: deploy-manage/manage/s3gateway/unsupported-operations.md 229 - Reference: 230 - Overview: reference/index.md 231 - Pipeline Specification: reference/pipeline_spec.md 232 - Pachyderm Config Specification: reference/config_spec.md 233 - Pachyderm Language Clients: reference/clients.md 234 - S3 Gateway API Reference: reference/s3gateway_api.md 235 - Pachctl Reference: 236 - reference/pachctl/pachctl.md 237 - reference/pachctl/pachctl_auth.md 238 - reference/pachctl/pachctl_auth_activate.md 239 - reference/pachctl/pachctl_auth_check.md 240 - reference/pachctl/pachctl_auth_deactivate.md 241 - reference/pachctl/pachctl_auth_get-auth-token.md 242 - reference/pachctl/pachctl_auth_get-config.md 243 - reference/pachctl/pachctl_auth_get-otp.md 244 - reference/pachctl/pachctl_auth_get.md 245 - reference/pachctl/pachctl_auth_list-admins.md 246 - reference/pachctl/pachctl_auth_login.md 247 - reference/pachctl/pachctl_auth_logout.md 248 - reference/pachctl/pachctl_auth_modify-admins.md 249 - reference/pachctl/pachctl_auth_set-config.md 250 - reference/pachctl/pachctl_auth_set.md 251 - reference/pachctl/pachctl_auth_use-auth-token.md 252 - reference/pachctl/pachctl_auth_whoami.md 253 - reference/pachctl/pachctl_completion.md 254 - reference/pachctl/pachctl_completion_bash.md 255 - reference/pachctl/pachctl_completion_zsh.md 256 - reference/pachctl/pachctl_config.md 257 - reference/pachctl/pachctl_config_delete.md 258 - reference/pachctl/pachctl_config_delete_context.md 259 - reference/pachctl/pachctl_config_get.md 260 - reference/pachctl/pachctl_config_get_active-context.md 261 - reference/pachctl/pachctl_config_get_context.md 262 - reference/pachctl/pachctl_config_get_metrics.md 263 - reference/pachctl/pachctl_config_list.md 264 - reference/pachctl/pachctl_config_list_context.md 265 - reference/pachctl/pachctl_config_set.md 266 - reference/pachctl/pachctl_config_set_active-context.md 267 - reference/pachctl/pachctl_config_set_context.md 268 - reference/pachctl/pachctl_config_set_metrics.md 269 - reference/pachctl/pachctl_config_update.md 270 - reference/pachctl/pachctl_config_update_context.md 271 - reference/pachctl/pachctl_copy.md 272 - reference/pachctl/pachctl_copy_file.md 273 - reference/pachctl/pachctl_create.md 274 - reference/pachctl/pachctl_create_branch.md 275 - reference/pachctl/pachctl_create_pipeline.md 276 - reference/pachctl/pachctl_create_repo.md 277 - reference/pachctl/pachctl_create_secret.md 278 - reference/pachctl/pachctl_debug.md 279 - reference/pachctl/pachctl_debug_binary.md 280 - reference/pachctl/pachctl_debug_dump.md 281 - reference/pachctl/pachctl_debug_pprof.md 282 - reference/pachctl/pachctl_debug_profile.md 283 - reference/pachctl/pachctl_delete.md 284 - reference/pachctl/pachctl_delete_all.md 285 - reference/pachctl/pachctl_delete_branch.md 286 - reference/pachctl/pachctl_delete_commit.md 287 - reference/pachctl/pachctl_delete_file.md 288 - reference/pachctl/pachctl_delete_job.md 289 - reference/pachctl/pachctl_delete_pipeline.md 290 - reference/pachctl/pachctl_delete_repo.md 291 - reference/pachctl/pachctl_delete_secret.md 292 - reference/pachctl/pachctl_delete_transaction.md 293 - reference/pachctl/pachctl_deploy.md 294 - reference/pachctl/pachctl_deploy_amazon.md 295 - reference/pachctl/pachctl_deploy_custom.md 296 - reference/pachctl/pachctl_deploy_export-images.md 297 - reference/pachctl/pachctl_deploy_google.md 298 - reference/pachctl/pachctl_deploy_ide.md 299 - reference/pachctl/pachctl_deploy_import-images.md 300 - reference/pachctl/pachctl_deploy_list-images.md 301 - reference/pachctl/pachctl_deploy_local.md 302 - reference/pachctl/pachctl_deploy_microsoft.md 303 - reference/pachctl/pachctl_deploy_storage.md 304 - reference/pachctl/pachctl_deploy_storage_amazon.md 305 - reference/pachctl/pachctl_deploy_storage_google.md 306 - reference/pachctl/pachctl_deploy_storage_microsoft.md 307 - reference/pachctl/pachctl_diff.md 308 - reference/pachctl/pachctl_diff_file.md 309 - reference/pachctl/pachctl_edit.md 310 - reference/pachctl/pachctl_edit_pipeline.md 311 - reference/pachctl/pachctl_enterprise.md 312 - reference/pachctl/pachctl_enterprise_activate.md 313 - reference/pachctl/pachctl_enterprise_get-state.md 314 - reference/pachctl/pachctl_exit.md 315 - reference/pachctl/pachctl_extract.md 316 - reference/pachctl/pachctl_extract_pipeline.md 317 - reference/pachctl/pachctl_finish.md 318 - reference/pachctl/pachctl_finish_commit.md 319 - reference/pachctl/pachctl_finish_transaction.md 320 - reference/pachctl/pachctl_flush.md 321 - reference/pachctl/pachctl_flush_commit.md 322 - reference/pachctl/pachctl_flush_job.md 323 - reference/pachctl/pachctl_fsck.md 324 - reference/pachctl/pachctl_garbage-collect.md 325 - reference/pachctl/pachctl_get.md 326 - reference/pachctl/pachctl_get_file.md 327 - reference/pachctl/pachctl_get_object.md 328 - reference/pachctl/pachctl_get_tag.md 329 - reference/pachctl/pachctl_glob.md 330 - reference/pachctl/pachctl_glob_file.md 331 - reference/pachctl/pachctl_inspect.md 332 - reference/pachctl/pachctl_inspect_branch.md 333 - reference/pachctl/pachctl_inspect_cluster.md 334 - reference/pachctl/pachctl_inspect_commit.md 335 - reference/pachctl/pachctl_inspect_datum.md 336 - reference/pachctl/pachctl_inspect_file.md 337 - reference/pachctl/pachctl_inspect_job.md 338 - reference/pachctl/pachctl_inspect_pipeline.md 339 - reference/pachctl/pachctl_inspect_repo.md 340 - reference/pachctl/pachctl_inspect_secret.md 341 - reference/pachctl/pachctl_inspect_transaction.md 342 - reference/pachctl/pachctl_list.md 343 - reference/pachctl/pachctl_list_branch.md 344 - reference/pachctl/pachctl_list_commit.md 345 - reference/pachctl/pachctl_list_datum.md 346 - reference/pachctl/pachctl_list_file.md 347 - reference/pachctl/pachctl_list_job.md 348 - reference/pachctl/pachctl_list_pipeline.md 349 - reference/pachctl/pachctl_list_repo.md 350 - reference/pachctl/pachctl_list_secret.md 351 - reference/pachctl/pachctl_list_transaction.md 352 - reference/pachctl/pachctl_logs.md 353 - reference/pachctl/pachctl_mount.md 354 - reference/pachctl/pachctl_port-forward.md 355 - reference/pachctl/pachctl_put.md 356 - reference/pachctl/pachctl_put_file.md 357 - reference/pachctl/pachctl_restart.md 358 - reference/pachctl/pachctl_restart_datum.md 359 - reference/pachctl/pachctl_restore.md 360 - reference/pachctl/pachctl_resume.md 361 - reference/pachctl/pachctl_resume_transaction.md 362 - reference/pachctl/pachctl_run.md 363 - reference/pachctl/pachctl_run_cron.md 364 - reference/pachctl/pachctl_run_pipeline.md 365 - reference/pachctl/pachctl_shell.md 366 - reference/pachctl/pachctl_start.md 367 - reference/pachctl/pachctl_start_commit.md 368 - reference/pachctl/pachctl_start_pipeline.md 369 - reference/pachctl/pachctl_start_transaction.md 370 - reference/pachctl/pachctl_stop.md 371 - reference/pachctl/pachctl_stop_job.md 372 - reference/pachctl/pachctl_stop_pipeline.md 373 - reference/pachctl/pachctl_stop_transaction.md 374 - reference/pachctl/pachctl_subscribe.md 375 - reference/pachctl/pachctl_subscribe_commit.md 376 - reference/pachctl/pachctl_undeploy.md 377 - reference/pachctl/pachctl_unmount.md 378 - reference/pachctl/pachctl_update-dash.md 379 - reference/pachctl/pachctl_update.md 380 - reference/pachctl/pachctl_update_pipeline.md 381 - reference/pachctl/pachctl_update_repo.md 382 - reference/pachctl/pachctl_version.md 383 - Examples: examples/examples.md 384 - Enterprise: 385 - Overview: enterprise/index.md 386 - Features: enterprise/overview.md 387 - Deploy Enterprise Edition: enterprise/deployment.md 388 - Configure Access Controls: 389 - Overview: enterprise/auth/index.md 390 - Enable Access Controls: enterprise/auth/enable-auth.md 391 - Manage Users and Groups: enterprise/auth/manage-users-groups.md 392 - Manage Authentication Config: enterprise/auth/auth-config.md 393 - Configure a SAML Provider: 394 - Overview: enterprise/auth/saml/saml.md 395 - Configure SAML: enterprise/auth/saml/saml_setup.md 396 - Use SAML: enterprise/auth/saml/saml_usage.md 397 - Configure an OIDC Provider: 398 - Overview: enterprise/auth/oidc/index.md 399 - Configure Pachyderm with Keycloak: enterprise/auth/oidc/configure-keycloak.md 400 - Configure Pachyderm with Auth0: enterprise/auth/oidc/configure-auth0.md 401 - Configure Pachyderm with Google OAuth 2.0: enterprise/auth/oidc/configure-google-oidc.md 402 - Advanced Statistics: enterprise/stats.md 403 - Troubleshooting: 404 - Overview: troubleshooting/index.md 405 - General Troubleshooting: troubleshooting/general_troubleshooting.md 406 - Deployment Troubleshooting: troubleshooting/deploy_troubleshooting.md 407 - Pipeline Troubleshooting: troubleshooting/pipeline_troubleshooting.md 408 409 # Google Analytics 410 #google_analytics: 411 # - !!python/object/apply:os.getenv ["UA-158483597-1"] 412 # - auto