github.com/nginxinc/kubernetes-ingress@v1.12.5/docs-web/app-protect/troubleshooting.md (about)

     1  # Troubleshoot the Ingress Controller with App Protect Integration
     2  
     3  This document describes how to troubleshoot problems with the Ingress Controller with the [App Protect](/nginx-app-protect/) module enabled.
     4  
     5  For general troubleshooting of the Ingress Controller, check the general [troubleshooting](/nginx-ingress-controller/troubleshooting/) documentation.
     6  
     7  For additional troubleshooting of the App Protect module itself, check the [troubleshooting](/nginx-app-protect/troubleshooting/) guide in the App Protect module documentation. 
     8  
     9  ## Potential Problems
    10  
    11  The table below categorizes some potential problems with the Ingress Controller when App Protect module is enabled. It suggests how to troubleshoot those problems, using one or more methods from the next section.
    12  
    13  ```eval_rst
    14  .. list-table::
    15     :header-rows: 1
    16  
    17     * - Problem area
    18       - Symptom
    19       - Troubleshooting method
    20       - Common cause
    21     * - Start.
    22       - The Ingress Controller fails to start.
    23       - Check the logs.
    24       - Misconfigured APLogConf or APPolicy.
    25     * - APLogConf, APPolicy or Ingress Resource.
    26       - The configuration is not applied.
    27       - Check the events of the APLogConf, APPolicy and Ingress Resource, check the logs, replace the policy.
    28       - APLogConf or APPolicy is invalid.
    29     * - NGINX.
    30       - The Ingress Controller NGINX verification timeouts while starting for the first time or while reloading after a change.
    31       - Check the logs for ``Unable to fetch version: X`` message. Check the Availability of APPolicy External References.
    32       - Too many Ingress Resources with App Protect enabled. Check the `NGINX fails to start/reload section <#nginx-fails-to-start-or-reload>`_ of the Known Issues.
    33  ```
    34  
    35  ## Troubleshooting Methods
    36  
    37  ### Check the Ingress Controller and App Protect logs
    38  
    39  App Protect logs are part of the Ingress Controller logs when the module is enabled. To check the Ingress Controller logs, follow the steps of [Checking the Ingress Controller Logs](/nginx-ingress-controller/troubleshooting/#checking-the-ingress-controller-logs) of the Troubleshooting guide.
    40  
    41  For App Protect specific logs, look for messages starting with `APP_PROTECT`, for example:
    42  ```
    43  2020/07/10 11:13:20 [notice] 17#17: APP_PROTECT { "event": "configuration_load_success", "software_version": "2.52.1", "completed_successfully":true,"attack_signatures_package":{"revision_datetime":"2020-06-18T10:11:32Z","version":"2020.06.18"}}
    44  ```
    45  
    46  ### Check events of an Ingress Resource
    47  
    48  Follow the steps of [Checking the Events of an Ingress Resource](/troubleshooting/#checking-the-events-of-an-ingress-resource).
    49  
    50  ### Check events of APLogConf
    51  
    52  After you create or update an APLogConf, you can immediately check if the NGINX configuration was successfully applied by NGINX:
    53  ```
    54  $ kubectl describe aplogconf logconf
    55  Name:         logconf
    56  Namespace:    default
    57  . . . 
    58  Events:
    59    Type    Reason          Age   From                      Message
    60    ----    ------          ----  ----                      -------
    61    Normal  AddedOrUpdated  11s   nginx-ingress-controller  AppProtectLogConfig  default/logconf was added or updated
    62  ```
    63  Note that in the events section, we have a `Normal` event with the `AddedOrUpdated` reason, which informs us that the configuration was successfully applied.
    64  
    65  ### Check events of APPolicy
    66  
    67  After you create or update an APPolicy, you can immediately check if the NGINX configuration was successfully applied by NGINX:
    68  ```
    69  $ kubectl describe appolicy dataguard-alarm
    70  Name:         dataguard-alarm
    71  Namespace:    default
    72  . . . 
    73  Events:
    74    Type    Reason          Age    From                      Message
    75    ----    ------          ----   ----                      -------
    76    Normal  AddedOrUpdated  2m25s  nginx-ingress-controller  AppProtectPolicy default/dataguard-alarm was added or updated
    77  ```
    78  Note that in the events section, we have a `Normal` event with the `AddedOrUpdated` reason, which informs us that the configuration was successfully applied.
    79  
    80  ### Replace the policy
    81  
    82  NOTE: This method only applies if using [external references](https://docs.nginx.com/nginx-app-protect/configuration/#external-references) 
    83  If items on the external reference change but the spec of the APPolicy remains unchanged (even when re-applying the policy), kubernetes will not detect the update.
    84  In this case you can force-replace the resource. This will remove the resource and add it again, triggering a reload. For example:
    85  
    86  ```
    87  kubectl replace appolicy -f your-policy-manifest.yaml --force
    88  ```
    89  
    90  ### Check the Availability of APPolicy External References.
    91  
    92  NOTE: This method only applies if you're using [external references](https://docs.nginx.com/nginx-app-protect/configuration/#external-references) in NGINX App Protect policies.
    93  
    94  To check what servers host the external references of a policy: 
    95  ```
    96  kubectl get appolicy mypolicy -o jsonpath='{.items[*].spec.policy.*.link}' | tr ' ' '\n'
    97  
    98  http://192.168.100.100/resources/headersettings.txt
    99  ```
   100  
   101  You can check the total time a http request takes, in multiple ways eg. using curl:
   102  ```
   103  curl -w '%{time_total}' http://192.168.100.100/resources/headersettings.txt
   104  ```
   105  
   106  ## Run App Protect in Debug Mode
   107  
   108  When you set the Ingress Controller to use debug mode, the setting also applies to the App Protect module.  See  [Running NGINX in the Debug Mode](/nginx-ingress-controller/troubleshooting/#running-nginx-in-the-debug-mode) for instructions.
   109  
   110  ## Known Issues
   111  
   112  When using the Ingress Controller with the App Protect module, the following issues have been reported. The occurrence of these issues is commonly related to a higher number of Ingress Resources with App Protect being enabled in a cluster.
   113  
   114  When you make a change that requires NGINX to apply a new configuration, the Ingress Controller reloads NGINX automatically. Without the App Protect module enabled, usual reload times are around 150ms. If App Protect module is enabled and is being used by any number of Ingress Resources, these reloads might take a few seconds instead. 
   115  
   116  ### NGINX Configuration Skew
   117  
   118  If you are running more than one instance of the Ingress Controller, the extended reload time may cause the NGINX configuration of your instances to be out of sync. This can occur because there is no order imposed on how the Ingress Controller processes the Kubernetes Resources. The configurations will be the same after all instances have completed the reload.
   119  
   120  In order to reduce these inconsistencies, we advise that you do not apply changes to multiple resources handled by the Ingress Controller at the same time. 
   121  
   122  ### NGINX Fails to Start or Reload
   123  
   124  The first time the Ingress Controller starts, or whenever there is a change that requires reloading NGINX, the Ingress Controller will verify if the reload was successful. The timeout for this verification is normally 4 seconds. When App Protect is enabled, this timeout is 20 seconds. 
   125  
   126  This timeout should be more than enough to verify configurations. However, when numerous Ingress Resources with App Protect enabled are handled by the Ingress Controller at the same time, you may find that you need to extend the timeout further.  Examples of when this might be necessary include:
   127  
   128  - You need to apply a large amount of Ingress Resources at once.
   129  - You are running the Ingress Controller for the first time in a cluster where the Ingress Resources with App Protect enabled are already present.
   130  
   131  You can increase this timeout by setting the `nginx-reload-timeout` [cli-argument](/nginx-ingress-controller/configuration/global-configuration/command-line-arguments/#cmdoption-nginx-reload-timeout).
   132  
   133  When using the User Defined Signature feature, an update to an `APUserSig` requires more reload time from NGINX Plus compared with the other AppProtect resources. As a consequence, we recommend increasing the `nginx-reload-timeout` to 30 seconds if you're planning to use this feature.
   134  
   135  If you are using external references in your Nginx App Protect policies, verify if the servers hosting the referenced resources are available and that their response time is as short as possible (see the Check the Availability of APPolicy External References section). If the references are not available during the Ingress Controller startup, the pod will fail to start. In case the resources are not available during a reload, the reload will fail, and NGINX Plus will use the previous correct configuration.