github.com/treeverse/lakefs@v1.24.1-0.20240520134607-95648127bfb0/docs/integrations/kafka.md (about)

     1  ---
     2  title: Apache Kafka
     3  description: This section explains how you can start using lakeFS with Kafka using Confluent’s S3 Sink Connector.
     4  parent: Integrations
     5  redirect_from: 
     6      - /using/kakfa.html
     7      - /integrations/kakfa.html
     8  ---
     9  
    10  # Using lakeFS with Apache Kafka
    11  
    12  [Apache Kafka](https://kafka.apache.org/) provides a unified, high-throughput, low-latency platform for handling real-time data feeds.
    13  
    14  Different distributions of Kafka offer different methods for exporting data to S3 called Kafka Sink Connectors.
    15  
    16  The most commonly used Connector for S3 is [Confluent's S3 Sink Connector](https://docs.confluent.io/current/connect/kafka-connect-s3/index.html).
    17  
    18  Add the following to `connector.properties` file for lakeFS support:
    19  
    20  ```properties
    21  # Your lakeFS repository
    22  s3.bucket.name=example-repo
    23  
    24  # Your lakeFS S3 endpoint and credentials
    25  store.url=https://lakefs.example.com
    26  aws.access.key.id=AKIAIOSFODNN7EXAMPLE
    27  aws.secret.access.key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    28  
    29  # main being the branch we want to write to
    30  topics.dir=main/topics 
    31  ```