github.com/grafana/pyroscope@v1.18.0/docs/sources/reference-pyroscope-architecture/components/ingester.md (about)

     1  ---
     2  title: "Pyroscope ingester"
     3  menuTitle: "Ingester"
     4  description: "The ingester writes incoming profiles to long-term storage."
     5  weight: 30
     6  aliases:
     7    - /docs/phlare/latest/reference-phlare-architecture/components/ingester/
     8  ---
     9  
    10  # Pyroscope ingester
    11  
    12  The ingester is a stateful component that writes incoming profiles first to [on disk storage](../../about-grafana-pyroscope-architecture/#long-term-storage) on the write path and returns series samples for queries on the read path.
    13  
    14  Incoming profiles from [distributors](../distributor/) are not immediately written to the long-term storage but are either kept in the ingester's memory or offloaded to the ingester's disk.
    15  Eventually, all profiles are written to disk and periodically uploaded to the long-term storage.
    16  For this reason, the [queriers](../querier/) might need to fetch samples from both ingesters and long-term storage while executing a query on the read path.
    17  
    18  Any Pyroscope component that calls the ingesters starts by first looking up ingesters registered in the [hash ring](../../hash-ring/) to determine which ingesters are available.
    19  Each ingester could be in one of the following states:
    20  
    21  - `PENDING`<br />
    22    The ingester has just started. While in this state, the ingester does not receive write or read requests.
    23  - `JOINING`<br />
    24    The ingester starts up and joins the ring. While in this state, the ingester does not receive write or read requests.
    25    The ingester loads tokens from disk (if `-ingester.ring.tokens-file-path` is configured) or generates a set of new random tokens.
    26    Finally, the ingester optionally observes the ring for token conflicts, and once resolved, moves to the `ACTIVE` state.
    27  - `ACTIVE`<br />
    28    The ingester is up and running. While in this state, the ingester can receive both write and read requests.
    29  - `LEAVING`<br />
    30    The ingester is shutting down and leaving the ring. While in this state, the ingester doesn't receive write requests, but can still receive read requests.
    31  - `UNHEALTHY`<br />
    32    The ingester has failed to heartbeat to the hash ring. While in this state, distributors bypass the ingester, which means that the ingester does not receive write or read requests.
    33  
    34  To configure the ingesters' hash ring, refer to [configuring memberlist](../../../configure-server/configuring-memberlist/).
    35  
    36  ## Ingesters write de-amplification
    37  
    38  Ingesters store recently received samples in-memory in order to perform write de-amplification.
    39  If the ingesters immediately write received samples to the long-term storage, the system would have difficulty scaling due to the high pressure on the long-term storage.
    40  For this reason, the ingesters batch and compress samples in-memory and periodically upload them to the long-term storage.
    41  
    42  Write de-amplification is the main source of Pyroscope's low total cost of ownership (TCO).
    43  
    44  ## Ingesters failure and data loss
    45  
    46  If an ingester process crashes or exits abruptly, all the in-memory profiles
    47  that have not yet been uploaded to the long-term storage could be lost. There
    48  are the following ways to mitigate this failure mode:
    49  
    50  - Replication
    51  
    52  ### Replication
    53  
    54  By default, each profile series is replicated to three ingesters. Writes to the
    55  Pyroscope cluster are successful if a quorum of ingesters received the data, which
    56  is a minimum of 2 with a replication factor of 3. If the Pyroscope cluster loses an
    57  ingester, the in-memory profiles held by the head block of the lost ingester
    58  are available at least in one other ingester. In the event of a single ingester
    59  failure, no profiles are lost. If multiple ingesters fail, profiles might be
    60  lost if the failure affects all the ingesters holding the replicas of a
    61  specific profile series.