github.com/1aal/kubeblocks@v0.0.0-20231107070852-e1c03e598921/docs/user_docs/kubeblocks-for-vector-database/manage-vector-databases.md (about)

     1  ---
     2  title: Manage Vector Databases with KubeBlocks
     3  description: How to manage vector databases on KubeBlocks
     4  keywords: [qdrant, milvus,weaviate]
     5  sidebar_position: 1
     6  sidebar_label: Manage Vector Databases with KubeBlocks
     7  ---
     8  # Manage Vector Databases with KubeBlocks
     9  
    10  The popularity of generative AI (Generative AI) has aroused widespread attention and completely ignited the vector database (Vector Database) market. KubeBlocks supports the management of vector databases, such as Qdrant, Milvus, and Weaviate.
    11  In this chapter, we take Qdrant as an example to show how to manage vector databases with KubeBlocks.
    12  Before you start, [Install KubeBlocks](./../installation/install-with-helm/) and `kbcli`.
    13  
    14  ## Create a cluster
    15  
    16  ***Steps***
    17  
    18  1. Execute the following command to create a Qdrant cluster. You can change the `cluster-definition` value as any other databases supported.
    19  
    20     ```bash
    21     kbcli cluster create qdrant --cluster-definition=qdrant
    22     ```
    23  
    24     If you want to create a Qdrant cluster with multiple replicas. Use the following command and set the replica numbers.
    25  
    26     ```bash
    27     kbcli cluster create qdrant --cluster-definition=qdrant --set replicas=3
    28     ```
    29  
    30  2. Check whether the cluster is created.
    31  
    32     ```bash
    33     kbcli cluster list
    34     >
    35     NAME     NAMESPACE   CLUSTER-DEFINITION   VERSION        TERMINATION-POLICY   STATUS    CREATED-TIME
    36     qdrant   default     qdrant               qdrant-1.1.0   Delete               Running   Aug 15,2023 23:03 UTC+0800
    37     ```
    38  
    39  3. Check the cluster information.
    40  
    41     ```bash
    42     kbcli cluster describe qdrant
    43     >
    44     Name: qdrant         Created Time: Aug 15,2023 23:03 UTC+0800
    45     NAMESPACE   CLUSTER-DEFINITION   VERSION        STATUS    TERMINATION-POLICY
    46     default     qdrant               qdrant-1.1.0   Running   Delete
    47  
    48     Endpoints:
    49     COMPONENT   MODE        INTERNAL                                       EXTERNAL
    50     qdrant      ReadWrite   qdrant-qdrant.default.svc.cluster.local:6333   <none>
    51                             qdrant-qdrant.default.svc.cluster.local:6334
    52  
    53     Topology:
    54     COMPONENT   INSTANCE          ROLE     STATUS    AZ       NODE                   CREATED-TIME
    55     qdrant      qdrant-qdrant-0   <none>   Running   <none>   x-worker3/172.20.0.3   Aug 15,2023 23:03 UTC+0800
    56     qdrant      qdrant-qdrant-1   <none>   Running   <none>   x-worker2/172.20.0.5   Aug 15,2023 23:03 UTC+0800
    57     qdrant      qdrant-qdrant-2   <none>   Running   <none>   x-worker/172.20.0.2    Aug 15,2023 23:04 UTC+0800
    58  
    59     Resources Allocation:
    60     COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
    61     qdrant      false       1 / 1                1Gi / 1Gi               data:20Gi      standard
    62  
    63     Images:
    64     COMPONENT   TYPE     IMAGE
    65     qdrant      qdrant   docker.io/qdrant/qdrant:latest
    66  
    67     Data Protection:
    68     AUTO-BACKUP   BACKUP-SCHEDULE   TYPE     BACKUP-TTL   LAST-SCHEDULE   RECOVERABLE-TIME
    69     Disabled      <none>            <none>   7d           <none>          <none>
    70  
    71     Show cluster events: kbcli cluster list-events -n default qdrant
    72     ```
    73  
    74  ## Connect to a vector database cluster
    75  
    76  Qdrant provides both HTTP and gRPC protocols for client access on ports 6333 and 6334 respectively. Depending on where the client is, different connection options are offered to connect to the Qdrant cluster.
    77  
    78  :::note
    79  
    80  If your cluster is on AWS, install the AWS Load Balancer Controller first.
    81  
    82  :::
    83  
    84  - If your client is inside a K8s cluster, run `kbcli cluster describe qdrant` to get the ClusterIP address of the cluster or the corresponding K8s cluster domain name.
    85  - If your client is outside the K8s cluster but in the same VPC as the server, run `kbcli cluster expose qdant --enable=true --type=vpc` to get a VPC load balancer address for the database cluster.
    86  - If your client is outside the VPC, run `kbcli cluster expose qdant --enable=true --type=internet` to open a public network reachable address for the database cluster.
    87  
    88  ## Monitor the vector database
    89  
    90  Open the grafana monitor web page.
    91  
    92  ```bash
    93  kbcli dashboard open kubeblocks-grafana
    94  ```
    95  
    96  When executing this command, browser is opened and you can see the dashboard.
    97  
    98  ## Scaling
    99  
   100  Scaling function for vector databases is also supported.
   101  
   102  ### Scale horizontally
   103  
   104  Use the following command to perform horizontal scaling.
   105  
   106  ```bash
   107  kbcli cluster hscale qdrant --replicas=5 --components=qdrant
   108  ```
   109  
   110  Please wait a few seconds until the scaling process is over.
   111  
   112  The `kbcli cluster hscale` command print the `opsname`, to check the progress of horizontal scaling, you can use the following command with the `opsname`.
   113  
   114  ```bash
   115  kubectl get ops qdrant-horizontalscaling-xpdwz
   116  >
   117  NAME                             TYPE                CLUSTER   STATUS    PROGRESS   AGE
   118  qdrant-horizontalscaling-xpdwz   HorizontalScaling   qdrant    Running   0/2        16s
   119  ```
   120  
   121  To check whether the scaling is done, use the following command.
   122  
   123  ```bash
   124  kbcli cluster describe qdrant
   125  ```
   126  
   127  ### Scale vertically
   128  
   129  Use the following command to perform vertical scaling.
   130  
   131  ```bash
   132  kbcli cluster vscale qdrant --cpu=0.5 --memory=512Mi --components=qdrant 
   133  ```
   134  
   135  Please wait a few seconds until the scaling process is over.
   136  The `kbcli cluster vscale` command print the `opsname`, to check the progress of scaling, you can use the following command with the `opsname`.
   137  
   138  ```bash
   139  kubectl get ops qdrant-verticalscaling-rpw2l
   140  >
   141  NAME                           TYPE              CLUSTER   STATUS    PROGRESS   AGE
   142  qdrant-verticalscaling-rpw2l   VerticalScaling   qdrant    Running   1/5        44s
   143  ```
   144  
   145  To check whether the scaling is done, use the following command.
   146  
   147  ```bash
   148  kbcli cluster describe qdrant
   149  ```
   150  
   151  ## Volume Expanding
   152  
   153  ***Steps:***
   154  
   155  ```bash
   156  kbcli cluster volume-expand qdrant --storage=40Gi --components=qdrant -t data
   157  ```
   158  
   159  The volume expanding may take a few minutes.
   160  
   161  The `kbcli cluster volume-expand` command print the `opsname`, to check the progress of volume expanding, you can use the following command with the `opsname`.
   162  
   163  ```bash
   164  kubectl get ops qdrant-volumeexpansion-5pbd2
   165  >
   166  NAME                           TYPE              CLUSTER   STATUS   PROGRESS   AGE
   167  qdrant-volumeexpansion-5pbd2   VolumeExpansion   qdrant    Running  -/-        67s
   168  ```
   169  
   170  To check whether the expanding is done, use the following command.
   171  
   172  ```bash
   173  kbcli cluster describe qdrant
   174  ```