github.com/yankunsam/loki/v2@v2.6.3-0.20220817130409-389df5235c27/tools/kafka/README.md (about)

     1  # Kafka tools
     2  
     3  This folder contains tools for testing promtail <-> kafka integration.
     4  
     5  **Requirements:**
     6  
     7  - docker and docker-compose
     8  - make
     9  - go > 1.15 for consuming using code source of promtail
    10  
    11  ## Running kafka locally
    12  
    13  To start kafka use `make start-kafka` this should start `kafka` and a `zookeeper` docker compose stack.
    14  To discover available brokers you can use the `make print-brokers`.
    15  
    16  Finally to stop the compose stack use `make stop-kafka`. This will result in all topics being lost with their messages.
    17  
    18  ## Running secure kafka locally
    19  
    20  To test authentication, you need to start the Kafka container which is configured with authentication.
    21  
    22  You can also use `make start-kafka` in appropriate directory like `sasl-scram` you need.
    23  
    24  In addition, you need to create certificates using `make create-certs` when using SSL/TLS.
    25  
    26  If you don't need to authenticate, you should use the tools in `plain` directory.
    27  
    28  ## Working with Topic
    29  
    30  In Kafka before sending messages you need to create and select the topic you want to use for the exchange.
    31  
    32  To create a new topic use: `make create-topic`, by default the topic name will be `promtail` if you wish to overrides it you can use the `TOPIC` variable as shown below:
    33  
    34  ```bash
    35  TOPIC=new-topic make create-topic
    36  ```
    37  
    38  To list all available topics use the `make list-topics` target.
    39  
    40  ```bash
    41   make list-topics
    42  __consumer_offsets
    43  promtail
    44  ```
    45  
    46  To describe a topic use the `make describe-topic` target.
    47  
    48  ```bash
    49   TOPIC=new-topic make describe-topic
    50  
    51  Topic: new-topic        PartitionCount: 3       ReplicationFactor: 1    Configs: segment.bytes=1073741824
    52          Topic: new-topic        Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001
    53          Topic: new-topic        Partition: 1    Leader: 1001    Replicas: 1001  Isr: 1001
    54          Topic: new-topic        Partition: 2    Leader: 1001    Replicas: 1001  Isr: 1001
    55  ```
    56  
    57  As you can see by default each topic is assigned by default  3 partitions and a replication factor of 1. You can change those default using respectively `PARTS` and `RF` variable, for example the example below will create a topic with 4 partitions:
    58  
    59  ```bash
    60  PARTS=4 TOPIC=new-topic-part make create-topic
    61  ```
    62  
    63  Partitions in kafka are used for scaling reads and writes. Each partitions is replicated on multiple machines.
    64  
    65  > WARNING: Order of messages is guaranteed on a single partition but not across a single topic.
    66  
    67  ## Producing messages
    68  
    69  You can start sending message using `make producer` target, it will start reading newlines and send them to the default topic (`promtail`):
    70  
    71  ```bash
    72  TOPIC=new-topic make producer
    73  
    74  Producing messages to topic new-topic...
    75  Write a message and press Enter
    76  >hello world !
    77  >
    78  ```
    79  
    80  ## Consuming with promtail
    81  
    82  You can run promtail in `dry-run` mode and connect it to the local kafka to try out the integration.
    83  
    84  Before doing so make sure the brokers list (`make print-broker`) is correctly configured in your promtail configuration file, see for example the [kafka example](../../clients/cmd/promtail/promtail-kafka.yaml).
    85  
    86  ```bash
    87  go run ../../clients/cmd/promtail/main.go --dry-run --config.file ../../clients/cmd/promtail/promtail-kafka.yaml
    88  
    89  Clients configured:
    90  ----------------------
    91  url: http://localhost:3100/loki/api/v1/push
    92  batchwait: 1s
    93  batchsize: 1048576
    94  follow_redirects: false
    95  backoff_config:
    96    min_period: 500ms
    97    max_period: 5m0s
    98    max_retries: 10
    99  timeout: 10s
   100  tenant_id: ""
   101  stream_lag_labels: filename
   102  
   103  level=info ts=2021-11-02T10:44:14.137894Z caller=server.go:260 http=[::]:9080 grpc=[::]:59237 msg="server listening on addresses"
   104  level=info ts=2021-11-02T10:44:14.138059Z caller=main.go:119 msg="Starting Promtail" version="(version=, branch=, revision=)"
   105  level=info ts=2021-11-02T10:44:14.139308Z caller=target_syncer.go:133 msg="new topics received" topics=[promtail]
   106  level=info ts=2021-11-02T10:44:14.139337Z caller=consumer.go:50 msg="starting consumer" topics=[promtail]
   107  level=info ts=2021-11-02T10:44:14.153164Z caller=consumer.go:92 msg="consuming topic" details="member_id=sarama-8cfa484d-2a04-458a-a0c0-4506c7a0969f generation_id=5 topic=promtail partition=1 initial_offset=12"
   108  level=info ts=2021-11-02T10:44:14.153673Z caller=consumer.go:92 msg="consuming topic" details="member_id=sarama-8cfa484d-2a04-458a-a0c0-4506c7a0969f generation_id=5 topic=promtail partition=0 initial_offset=13"
   109  level=info ts=2021-11-02T10:44:14.153927Z caller=consumer.go:92 msg="consuming topic" details="member_id=sarama-8cfa484d-2a04-458a-a0c0-4506c7a0969f generation_id=5 topic=promtail partition=2 initial_offset=10"
   110  
   111  2021-11-02T11:47:08.849115+0100{group="some_group", job="kafka", partition="1", topic="promtail"}       hello world !
   112  ```
   113  
   114  > Alternatively you can use the binary or docker version of promtail.