github.com/waldiirawan/apm-agent-go/v2@v2.2.2/docs/log-correlation.asciidoc (about)

     1  [[log-correlation]]
     2  == Log Correlation
     3  
     4  The Go agent provides integrations for popular logging frameworks that
     5  inject trace ID fields into the application's log records. You can find a list of
     6  the supported logging frameworks under <<supported-tech-logging, supported technologies>>.
     7  
     8  If your favorite logging framework is not already supported, there are two other options:
     9  
    10  * Open a feature request, or contribute code, for additional support as described in <<contributing>>.
    11  * Manually inject trace IDs into log records, as described below in <<log-correlation-manual>>.
    12  
    13  Regardless of how you integrate APM with logging, use {filebeat-ref}[Filebeat] to
    14  send your logs to Elasticsearch, in order to correlate your traces and logs and link from
    15  APM to the {observability-guide}/monitor-logs.html[Logs app].
    16  
    17  [float]
    18  [[log-correlation-manual]]
    19  === Manual log correlation
    20  
    21  If the agent-provided logging integrations are not suitable or not available for your
    22  application, then you can use the agent's <<api, API>> to inject trace IDs manually.
    23  There are two main approaches you can take, depending on whether you are using structured
    24  or unstructured logging.
    25  
    26  [float]
    27  [[log-correlation-manual-structured]]
    28  ==== Manual log correlation (structured)
    29  
    30  For correlating structured logs with traces, the following fields should be added to your logs:
    31  
    32   - `trace.id`
    33   - `transaction.id`
    34   - `span.id` (if logging in the context of a span)
    35  
    36  Given a transaction object, you can obtain its trace ID and transaction ID using
    37  the <<transaction-tracecontext, apm.Transaction.TraceContext>> method. Similarly,
    38  given a span object, you can obtain its span ID using <<span-tracecontext, apm.Span.TraceContext>>.
    39  
    40  If you use the context APIs to start transactions and spans, then you can obtain
    41  the context's current transaction using <<apm-transaction-from-context, apm.TransactionFromContext>>,
    42  and current span using <<apm-span-from-context, apm.SpanFromContext>>. Note that if
    43  a transaction is not sampled, `apm.TransactionFromContext` will return `nil`.
    44  Similarly, spans may be dropped by the agent, so `apm.SpanFromContext` may also return `nil`.
    45  
    46  [source,go]
    47  ----
    48  labels := make(map[string]string)
    49  tx := apm.TransactionFromContext(ctx)
    50  if tx != nil {
    51  	traceContext := tx.TraceContext()
    52  	labels["trace.id"] = traceContext.Trace.String()
    53  	labels["transaction.id"] = traceContext.Span.String()
    54  	if span := apm.SpanFromContext(ctx); span != nil {
    55  		labels["span.id"] = span.TraceContext().Span
    56  	}
    57  }
    58  ----
    59  
    60  Follow this article to ingest JSON-encoded logs with Filebeat:
    61  {blog-ref}how-to-instrument-your-go-app-with-the-elastic-apm-go-agent#logs[How to instrument your Go app with the Elastic APM Go agent].
    62  
    63  [float]
    64  [[log-correlation-manual-unstructured]]
    65  ==== Manual log correlation (unstructured)
    66  
    67  For correlating unstructured logs (e.g. basic printf-style logging, like the standard library's
    68  `log` package), then you will need to need to include the trace IDs in your log message. Then,
    69  extract them using Filebeat.
    70  
    71  If you already have a transaction or span object, use the
    72  <<transaction-tracecontext, Transaction.TraceContext>> or <<span-tracecontext, Span.TraceContext>>
    73  methods. The trace, transaction, and span ID types all provide `String` methods that yield
    74  their canonical hex-encoded string representation.
    75  
    76  [source,go]
    77  ----
    78  traceContext := tx.TraceContext()
    79  spanID := span.TraceContext().Span
    80  log.Printf("ERROR [trace.id=%s transaction.id=%s span.id=%s] an error occurred", traceContext.Trace, traceContext.Span, spanID)
    81  ----
    82  
    83  
    84  If instead you are dealing with context objects, you may prefer to use the
    85  <<apm-traceformatter, TraceFormatter>> function. For example, you could supply it as an argument
    86  to `log.Printf` as follows:
    87  
    88  [source,go]
    89  ----
    90  log.Printf("ERROR [%+v] an error occurred", apm.TraceFormatter(ctx))
    91  ----
    92  
    93  This would print a log message along the lines of:
    94  
    95      2019/09/17 14:48:02 ERROR [trace.id=cd04f33b9c0c35ae8abe77e799f126b7 transaction.id=cd04f33b9c0c35ae span.id=960834f4538880a4] an error occurred
    96  
    97  For log correlation to work, the trace IDs must be extracted from the log message and
    98  stored in separate fields in the Elasticsearch document. This can be achieved by
    99  {filebeat-ref}/configuring-ingest-node.html[using an ingest pipeline to parse the data], in particular
   100  by using {ref}/grok-processor.html[the grok processor].
   101  
   102  [source,json]
   103  ----
   104  {
   105    "description": "...",
   106    "processors": [
   107      {
   108        "grok": {
   109          "field": "message",
   110          "patterns": ["%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME} %{LOGLEVEL:log.level} \\[trace.id=%{TRACE_ID:trace.id}(?: transaction.id=%{SPAN_ID:transaction.id})?(?: span.id=%{SPAN_ID:span.id})?\\] %{GREEDYDATA:message}"],
   111          "pattern_definitions": {
   112            "TRACE_ID": "[0-9A-Fa-f]{32}",
   113            "SPAN_ID": "[0-9A-Fa-f]{16}"
   114          }
   115        }
   116      }
   117    ]
   118  }
   119  ----