go.chromium.org/luci@v0.0.0-20240309015107-7cdc2e660f33/logdog/client/butler/bundler/doc.go (about)

     1  // Copyright 2015 The LUCI Authors.
     2  //
     3  // Licensed under the Apache License, Version 2.0 (the "License");
     4  // you may not use this file except in compliance with the License.
     5  // You may obtain a copy of the License at
     6  //
     7  //      http://www.apache.org/licenses/LICENSE-2.0
     8  //
     9  // Unless required by applicable law or agreed to in writing, software
    10  // distributed under the License is distributed on an "AS IS" BASIS,
    11  // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    12  // See the License for the specific language governing permissions and
    13  // limitations under the License.
    14  
    15  // Package bundler is responsible for efficiently transforming aggregate stream
    16  // data into Butler messages for export.
    17  //
    18  // A process will instantiate a single Bundler instance. The Bundler manages an
    19  // elastic set of Stream instances, each of which contains state for a single
    20  // log Stream.
    21  //
    22  // Each Stream instance will have sequential stream binary data appended to it
    23  // via Append, which it will collect and organize for export as a series of
    24  // ButlerLogBundle_Entry protobufs. Streams operate independently and buffer
    25  // data until it is consumed by their Bundler instance. If a Stream's buffer is
    26  // full, the Stream will block on appending data, which will, in turn, block its
    27  // data source.
    28  //
    29  // The Bundler owns the various Stream instances. When its Next() method is
    30  // called, it will sort through the stream instances to prepare an
    31  // optimally-sized ButlerLogBundle protobuf for export. The construction of this
    32  // bundle may block pending data, and may be subject to various data urgency
    33  // requests.
    34  //
    35  // The Bundler acknowledges the following constraints:
    36  //   - Data enqueued into a Stream should be exported within a specific period
    37  //     of time from its introduction
    38  //   - The exported ButlerLogBundle protobuf must not exceed a maximum bundle
    39  //     size constraint.
    40  //   - Stream data may be added during the bundling process, and should be
    41  //     acknowledged if possible.
    42  //
    43  // When a Stream is finished, its Close method should be called. This alerts the
    44  // Stream that it will receive no more data, causing it to export a terminal
    45  // ButlerLogBundle and unregister from the Bundler.
    46  //
    47  // The Bundler may block via its CloseAndFinish() method until all Streams are
    48  // drained and cleared.
    49  package bundler