github.com/wfusion/gofusion@v1.1.14/common/infra/watermill/docs/content/pubsubs/kafka.md (about)

     1  +++
     2  title = "Kafka"
     3  description = "A distributed streaming platform from Apache"
     4  date = 2019-07-06T22:30:00+02:00
     5  bref = "A distributed streaming platform from Apache"
     6  weight = -80
     7  type = "docs"
     8  toc = false
     9  +++
    10  
    11  ### Kafka
    12  
    13  Apache Kafka is one of the most popular Pub/Subs. We are providing Pub/Sub implementation based on [Shopify's Sarama](https://github.com/Shopify/sarama).
    14  
    15  ### Installation
    16  
    17      go get github.com/ThreeDotsLabs/watermill-kafka/v2
    18  
    19  #### Characteristics
    20  
    21  | Feature | Implements | Note |
    22  | ------- | ---------- | ---- |
    23  | ConsumerGroups | yes | |
    24  | ExactlyOnceDelivery | no | in theory can be achieved with [Transactions](https://www.confluent.io/blog/transactions-apache-kafka/), currently no support for any Golang client  |
    25  | GuaranteedOrder | yes | require [partition key usage](#partitioning)  |
    26  | Persistent | yes| |
    27  
    28  #### Configuration
    29  
    30  {{% render-md %}}
    31  {{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/subscriber.go" first_line_contains="type SubscriberConfig struct" last_line_contains="// Subscribe" %}}
    32  {{% /render-md %}}
    33  
    34  ##### Passing custom `Sarama` config
    35  
    36  You can pass [custom config](https://github.com/Shopify/sarama/blob/master/config.go#L20) parameters via `overwriteSaramaConfig *sarama.Config` in `NewSubscriber` and `NewPublisher`.
    37  When `nil` is passed, default config is used (`DefaultSaramaSubscriberConfig`).
    38  
    39  {{% render-md %}}
    40  {{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/subscriber.go" first_line_contains="// DefaultSaramaSubscriberConfig" last_line_contains="return config" padding_after="1" %}}
    41  {{% /render-md %}}
    42  
    43  #### Connecting
    44  
    45  ##### Publisher
    46  {{% render-md %}}
    47  {{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/publisher.go" first_line_contains="// NewPublisher" last_line_contains="(*Publisher, error)" padding_after="0" %}}
    48  
    49  Example:
    50  {{% load-snippet-partial file="src-link/_examples/pubsubs/kafka/main.go" first_line_contains="saramaSubscriberConfig :=" last_line_contains="panic(err)" padding_after="1" %}}
    51  
    52  {{% /render-md %}}
    53  
    54  ##### Subscriber
    55  {{% render-md %}}
    56  {{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/subscriber.go" first_line_contains="// NewSubscriber" last_line_contains="(*Subscriber, error)" padding_after="0" %}}
    57  
    58  Example:
    59  {{% load-snippet-partial file="src-link/_examples/pubsubs/kafka/main.go" first_line_contains="publisher, err := kafka.NewPublisher" last_line_contains="panic(err)" padding_after="1" %}}
    60  {{% /render-md %}}
    61  
    62  #### Publishing
    63  
    64  {{% render-md %}}
    65  {{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/publisher.go" first_line_contains="// Publish" last_line_contains="func (p *Publisher) Publish" %}}
    66  {{% /render-md %}}
    67  
    68  #### Subscribing
    69  
    70  {{% render-md %}}
    71  {{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/subscriber.go" first_line_contains="// Subscribe" last_line_contains="func (s *Subscriber) Subscribe" %}}
    72  {{% /render-md %}}
    73  
    74  #### Marshaler
    75  
    76  Watermill's messages cannot be directly sent to Kafka - they need to be marshaled. You can implement your marshaler or use default implementation.
    77  
    78  {{% render-md %}}
    79  {{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/marshaler.go" first_line_contains="// Marshaler" last_line_contains="func (DefaultMarshaler)" padding_after="0" %}}
    80  {{% /render-md %}}
    81  
    82  #### Partitioning
    83  
    84  Our Publisher has support for the partitioning mechanism.
    85  
    86  It can be done with special Marshaler implementation:
    87  
    88  {{% render-md %}}
    89  {{% load-snippet-partial file="src-link/watermill-kafka/pkg/kafka/marshaler.go" first_line_contains="type kafkaJsonWithPartitioning" last_line_contains="func (j kafkaJsonWithPartitioning) Marshal" padding_after="0" %}}
    90  {{% /render-md %}}
    91  
    92  When using, you need to pass your function to generate partition key.
    93  It's a good idea to pass this partition key with metadata to not unmarshal entire message.
    94  
    95  {{< highlight >}}
    96  marshaler := kafka.NewWithPartitioningMarshaler(func(topic string, msg *message.Message) (string, error) {
    97      return msg.Metadata.Get("partition"), nil
    98  })
    99  {{< /highlight >}}
   100