github.com/pachyderm/pachyderm@v1.13.4/doc/docs/archived/existing_vpc.md (about)

     1  # Deploy Into an Existing VPC
     2  
     3  ## Prereqs
     4  
     5  - Terraform
     6  - An existing AWS VPC deployed
     7  
     8  ## How to generate terraform k8s cluster deployment manifest
     9  
    10  This how to is based off of [this guide](https://ryaneschinger.com/blog/kubernetes-aws-vpc-kops-terraform/)
    11  
    12  1) Collect the following info / set the following env vars
    13  
    14  ```
    15  VPC_ID=vpc-2345
    16  ZONE_ID=34l5kj34l5kj
    17  ZONES=us-east-1a,us-east-1b,us-east-1c
    18  # the cluster name will also be its domain
    19  # and needs to be a valid subdomain on the hosted zone
    20  NAME=prod.sourceai.io 
    21  ```
    22  
    23  Collect your list of subnets, which should look like this:
    24  
    25  ```
    26    subnets:
    27    - egress: nat-sdfgsdfgsdfgsdfg
    28      id: subnet-2345bc2345b
    29      name: us-east-1a
    30      type: Private
    31      zone: us-east-1a
    32    - egress: nat-sdfgsdfgsdfgsdfg
    33      id: subnet-57b3575b375b
    34      name: us-east-1b
    35      type: Private
    36      zone: us-east-1b
    37    - egress: nat-sdfgsdfgsdfgsdfg
    38      id: subnet-0879ef078ef087
    39      name: us-east-1c
    40      type: Private
    41      zone: us-east-1c
    42    - id: subnet-2263be6e26be62
    43      name: Public Subnet 1
    44      type: Utility
    45      zone: us-east-1a
    46    - id: subnet-3444b5425b5
    47      name: Public Subnet 2
    48      type: Utility
    49      zone: us-east-1b
    50    - id: subnet-2314c334c43
    51      name: Public Subnet 3
    52      type: Utility
    53      zone: us-east-1c
    54  ```
    55  
    56  Note: For some silly reason, the private subnet names need to match the zone.
    57  This seems to be a requirement for the change to be accepted by kops.
    58  
    59  2) Create a kops state store bucket
    60  
    61  We need to do this because we're going to use kops to stage the changes, then
    62  emit them as a terraform manifest. To do that kops needs a state store.
    63  
    64  You can use TF to generate an s3 bucket. [For example](https://github.com/ryane/kubernetes-aws-vpc-kops-terraform/blob/master/main.tf#L44). Otherwise, here's one
    65  way to do it w some basic error handling:
    66  
    67  ```
    68  create_s3_bucket() {
    69    if [[ "$#" -lt 1 ]]; then
    70      echo "Error: create_s3_bucket needs a bucket name"
    71      return 1
    72    fi
    73    BUCKET="${1#s3://}"
    74  
    75    # For some weird reason, s3 emits an error if you pass a location constraint when location is "us-east-1"
    76    if [[ "${AWS_REGION}" == "us-east-1" ]]; then
    77      aws s3api create-bucket --bucket ${BUCKET} --region ${AWS_REGION}
    78    else
    79      aws s3api create-bucket --bucket ${BUCKET} --region ${AWS_REGION} --create-bucket-configuration LocationConstraint=${AWS_REGION}
    80    fi
    81  }
    82  
    83  export AWS_REGION="us-east-1"
    84  
    85  create_s3_bucket some_bucket_name
    86  ```
    87  
    88  3) Create the kops cluster
    89  
    90  ```
    91  kops create cluster \
    92       --master-zones $ZONES \
    93       --zones $ZONES \
    94       --topology private \
    95       --dns-zone $ZONE_ID \
    96       --networking calico \
    97       --vpc $VPC_ID \
    98       --target=terraform \
    99       --out=. \
   100       ${NAME}
   101  ```
   102  
   103  4) Update the kops cluster
   104  
   105  First edit the deployment to specify your VPC, CIDR, and subnets:
   106  
   107  ```
   108  kops edit cluster $NAME
   109  ```
   110  
   111  You can find the CIDR listed on the AWS console.
   112  
   113  
   114  Then update the cluster:
   115  
   116  ```
   117  kops update cluster \
   118     --out=. \
   119     --target=terraform \
   120     ${NAME}
   121  ```
   122  
   123  Which applies the changes to the kops state store and stages them there.
   124  
   125  5) Deploy using terraform
   126  
   127  ```
   128  terraform plan
   129  terraform apply
   130  ```
   131  
   132  6) Tear down
   133  
   134  To tear down, do:
   135  
   136  ```
   137  terraform destroy
   138  kops delete cluster $NAME
   139  ```
   140  
   141  ## How to generate k8s Pachyderm cluster manifest
   142  
   143  The [Deploy Pachyderm on Amazon AWS](https://docs.pachyderm.com/latest/deploy-manage/deploy/amazon_web_services/)
   144  section provides an overview of Pachyderm cluster manifest generation.
   145  
   146  But it boils down to this.
   147  
   148  1) Create an s3 bucket for the data store
   149  2) Set the `BUCKET_NAME`, `STORAGE_SIZE`, `AWS_REGION`, and AWS credentials env
   150  vars
   151  3) Run the `pachctl deploy amazon ...` command w the `--dry-run` flag to emit
   152  the yaml k8s manifest
   153  4) Store that manifest in the infra repo
   154  5) Deploy via `kubectl create -f pachyderm.yaml`
   155  
   156  
   157  ## Next Steps
   158  
   159  
   160  [Connect to your Pachyderm Cluster](connecting_to_your_cluster.html)