github.com/openshift/installer@v1.4.17/docs/dev/bootstrap_services.md (about)

     1  #### Bootstrap Service Records ####
     2  
     3  For the purposes of diagnosing installation failures that occur during bootstrapping, the
     4  progresses of services running on the bootstrap machines are tracked in json files in the
     5  /var/log/openshift directory. The progress for each service is tracked in its own file. For
     6  example, the bootkube service progress is tracked in the /var/log/openshift/bootkube.json
     7  file. The following progress events are tracked.
     8  * A service adds an entry when the service starts.
     9  * A service adds an entry when the service ends. The entry includes the result of the service
    10  invocation, either success or failure. If the invocation failed, then the entry includes
    11  the line number of the error and the last three lines from the service's journal log.
    12  * A service adds an entry when a stage of the service starts. An example of a service stage
    13  is the cvo-bootstrap stage of the bootkube service. During that stage, the
    14  cluster-version-operator renders its manifests.
    15  * A service adds an entry when a stage of the service ends. This is similar to the entry
    16  added when a service ends, including having a result and error information if applicable.
    17  * A service adds an entry when a pre- or post-command starts.
    18  * A service adds en entry when a pre- or post-command ends. This is similar to the entry
    19  added when a service ends, including having a result and error information if applicable.
    20  
    21  ##### Managing Service Records in a Service #####
    22  
    23  To track its progress, a service should source the /usr/local/bin/bootstrap-service-record.sh
    24  script. When a service sources the script, the script will add an entry to the json file for
    25  the service indicating that the service started. When the service ends, either successfully
    26  or due to an error, the script will add an entry to the json file for the service indicating that
    27  the service ended. The script will consider whether the last command executed was successful or
    28  not in order to determine whether the service was successful.
    29  
    30  For tracking stages, the service should call functions from the sourced script.
    31  * The service should call the `record_service_stage_start` function when a stage of the service
    32  starts. The function takes the name of the stage as its single argument.
    33  * The service should call the `record_service_stage_success` function when a stage of the service
    34  ends successfully.
    35  * The service should call the `record_service_stage_failure` function when a stage of the service
    36  ends due to a failure. The script will automatically record an entry for a stage failure if the
    37  service ends during the execution of a stage.
    38  
    39  ###### Pre- and Post-Commands ######
    40  
    41  If a service has pre- or post-commands that could either run for significant periods or could
    42  potentially fail, then those commands should add to the json file as well. Such a command should
    43  source the same /usr/local/bin/bootstrap-service-record.sh script. It should also set either the
    44  `PRE_COMMAND` or `POST_COMMAND` environment variable with a value that identifies the command.
    45  For example, kubelet.service has a pre-command of /usr/local/bin/kubelet-pause-image.sh. The
    46  kubelet-pause-image.sh script sets the `PRE_COMMAND` environment variable to "kubelet-pause-image"
    47  before sourcing the bootstrap-service-record.sh script. The entries for the pre-command will
    48  contain a `preCommand` field with the "kubelet-pause-image" value.
    49  
    50  ###### Sample Script #######
    51  
    52  ```shell script
    53  #!/usr/bin/env bash
    54  set -euoE pipefail
    55  
    56  # Source the script to record service entries.
    57  # This will create en entry for the start of the service.
    58  . /usr/local/bin/bootstrap-service-record.sh
    59  
    60  # Record the start of the "first" stage.
    61  record_service_stage_start "first"
    62  
    63  # Record the successful end of the "first" stage.
    64  record_service_stage_success
    65  
    66  while true
    67  do
    68      # Record the start of the "second" stage.
    69      record_service_stage_start "second-stage"
    70      
    71      if [ some_check ]
    72      then
    73          # Record the successful end of the "second" stage.
    74          record_service_stage_success
    75          break
    76      else
    77          # Record the failing end of the "second" stage.
    78          record_service_stage_failure  
    79      fi
    80  done
    81  
    82  # Record the start of the third stage.
    83  record_service_stage_start "third"
    84  
    85  # If the command fails, then an entry will be recorded for the failing end
    86  # of the third stage and an entry will be recorded for the failing end of
    87  # the service.
    88  some_command_that_may_fail
    89  
    90  # Record the end of the third stage.
    91  record_service_stage_success
    92  
    93  # Since this is the end of the script, an entry will be recorded for the
    94  # successful end of the service.
    95  ```