github.com/nginxinc/kubernetes-ingress@v1.12.5/docs-web/troubleshooting.md (about)

     1  # Troubleshooting
     2  
     3  This document describes how to troubleshoot problems with the Ingress Controller.
     4  
     5  ## Potential Problems
     6  
     7  The table below categorizes some potential problems with the Ingress Controller you may encounter and suggests how to troubleshoot those problems using one or more methods from the next section.
     8  
     9  ```eval_rst
    10  .. list-table::
    11     :header-rows: 1
    12  
    13     * - Problem area
    14       - Symptom
    15       - Troubleshooting method
    16       - Common cause
    17     * - Start
    18       - The Ingress Controller fails to start.
    19       - Check the logs.
    20       - Misconfigured RBAC, a missing default server TLS Secret.
    21     * - Ingress Resource and Annotations
    22       - The configuration is not applied.
    23       - Check the events of the Ingress resource, check the logs, check the generated config.
    24       - Invalid values of annotations.
    25     * - VirtualServer and VirtualServerRoute Resources
    26       - The configuration is not applied.
    27       - Check the events of the VirtualServer and VirtualServerRoutes, check the logs, check the generated config.
    28       - VirtualServer or VirtualServerRoute is invalid.
    29     * - Policy Resource
    30       - The configuration is not applied.
    31       - Check the events of the Policy resource as well as the events of the VirtualServers that reference that policy, check the logs, check the generated config.
    32       - Policy is invalid.
    33     * - ConfigMap Keys
    34       - The configuration is not applied.
    35       - Check the events of the ConfigMap, check the logs, check the generated config. 
    36       - Invalid values of ConfigMap keys.
    37     * - NGINX
    38       - NGINX responds with unexpected responses.
    39       - Check the logs, check the generated config, check the live activity dashboard (NGINX Plus only), run NGINX in the debug mode.
    40       - Unhealthy backend pods, a misconfigured backend service. 
    41  ```
    42  
    43  ## Troubleshooting Methods
    44  
    45  Note that the commands in the next sections make the following assumptions:
    46  * The Ingress Controller is deployed in the namespace `nginx-ingress`.
    47  * `<nginx-ingress-pod>` is the name of one of the Ingress Controller pods.
    48  
    49  ### Checking the Ingress Controller Logs
    50  
    51  To check the Ingress Controller logs -- both of the Ingress Controller software and the NGINX access and error logs -- run:
    52  ```
    53  $ kubectl logs <nginx-ingress-pod> -n nginx-ingress
    54  ```
    55  
    56  Controlling the verbosity and format:
    57  * To control the verbosity of the Ingress Controller software logs (from 1 to 4), use the `-v` [command-line argument](/nginx-ingress-controller/configuration/global-configuration/command-line-arguments). For example, with `-v=3` you will get more information and the content of any new or updated configuration file will be printed in the logs.
    58  * To control the verbosity and the format of the NGINX logs, configure the corresponding [ConfigMap keys](/nginx-ingress-controller/configuration/global-configuration/configmap-resource).
    59  
    60  ### Checking the Events of an Ingress Resource
    61  
    62  After you create or update an Ingress resource, you can immediately check if the NGINX configuration for that Ingress resource was successfully applied by NGINX:
    63  ```
    64  $ kubectl describe ing cafe-ingress
    65  Name:             cafe-ingress
    66  Namespace:        default
    67  . . .
    68  Events:
    69    Type    Reason          Age   From                      Message
    70    ----    ------          ----  ----                      -------
    71    Normal  AddedOrUpdated  12s   nginx-ingress-controller  Configuration for default/cafe-ingress was added or updated
    72  ```
    73  Note that in the events section, we have a `Normal` event with the `AddedOrUpdated` reason, which informs us that the configuration was successfully applied.
    74  
    75  ### Checking the Events of a VirtualServer and VirtualServerRoute Resources
    76  
    77  After you create or update a VirtualServer resource, you can immediately check if the NGINX configuration for that  resource was successfully applied by NGINX:
    78  ```
    79  $ kubectl describe vs cafe
    80  . . .
    81  Events:
    82    Type    Reason          Age   From                      Message
    83    ----    ------          ----  ----                      -------
    84    Normal  AddedOrUpdated  16s   nginx-ingress-controller  Configuration for default/cafe was added or updated
    85  ```
    86  Note that in the events section, we have a `Normal` event with the `AddedOrUpdated` reason, which informs us that the configuration was successfully applied.
    87  
    88  Checking the events of a VirtualServerRoute is similar:
    89  ```
    90  $ kubectl describe vsr coffee 
    91  . . .
    92  Events:
    93    Type     Reason                 Age   From                      Message
    94    ----     ------                 ----  ----                      -------
    95    Normal   AddedOrUpdated         1m    nginx-ingress-controller  Configuration for default/coffee was added or updated
    96  ```
    97  
    98  ### Checking the Events of a Policy Resource
    99  
   100  After you create or update a Policy resource, you can use `kubectl describe` to check whether or not the Ingress Controller accepted the Policy:
   101  ```
   102  $ kubectl describe pol webapp-policy
   103  . . .
   104  Events:
   105    Type    Reason          Age   From                      Message
   106    ----    ------          ----  ----                      -------
   107    Normal  AddedOrUpdated  11s   nginx-ingress-controller  Policy default/webapp-policy was added or updated
   108  ```
   109  Note that in the events section, we have a `Normal` event with the `AddedOrUpdated` reason, which informs us that the policy was successfully accepted.
   110  
   111  However, the fact that a policy was accepted doesn't guarantee that the NGINX configuration was successfully applied. To confirm that, check the events of the VirtualServer and VirtualServerRoute resources that reference that policy.
   112  
   113  ### Checking the Events of the ConfigMap Resource
   114  
   115  After you update the [ConfigMap](/nginx-ingress-controller/configuration/global-configuration/configmap-resource) resource, you can immediately check if the configuration was successfully applied by NGINX:
   116  ```
   117  $ kubectl describe configmap nginx-config -n nginx-ingress
   118  Name:         nginx-config
   119  Namespace:    nginx-ingress
   120  Labels:       <none>
   121  . . .
   122  Events:
   123    Type    Reason   Age                From                      Message
   124    ----    ------   ----               ----                      -------
   125    Normal  Updated  11s (x2 over 26m)  nginx-ingress-controller  Configuration from nginx-ingress/nginx-config was updated
   126  ```
   127  Note that in the events section, we have a `Normal` event with the `Updated` reason, which informs us that the configuration was successfully applied.
   128  
   129  ### Checking the Generated Config
   130  
   131  For each Ingress/VirtualServer resource, the Ingress Controller generates a corresponding NGINX configuration file in the `/etc/nginx/conf.d` folder. Additionally, the Ingress Controller generates the main configuration file `/etc/nginx/nginx.conf`, which includes all the configurations files from `/etc/nginx/conf.d`. The config of a VirtualServerRoute resource is located in the configuration file of the VirtualServer that references the resource.
   132  
   133  You can view the content of the main configuration file by running:
   134  ```
   135  $ kubectl exec <nginx-ingress-pod> -n nginx-ingress -- cat /etc/nginx/nginx.conf
   136  ```
   137  
   138  Similarly, you can view the content of any generated configuration file in the `/etc/nginx/conf.d` folder. 
   139  
   140  You can also print all NGINX configuration files together:
   141  ```
   142  $ kubectl exec <nginx-ingress-pod> -n nginx-ingress -- nginx -T
   143  ```
   144  However, this command will fail if any of the configuration files is not valid.
   145  
   146  ### Checking the Live Activity Monitoring Dashboard
   147  
   148  The live activity monitoring dashboard shows the real-time information about NGINX Plus and the applications it is load balancing, which is helpful for troubleshooting. To access the dashboard, follow the steps from [here](/nginx-ingress-controller/logging-and-monitoring/status-page).
   149  
   150  ### Running NGINX in the Debug Mode
   151  
   152  Running NGINX in the [debug mode](https://docs.nginx.com/nginx/admin-guide/monitoring/debugging/) allows us to enable its debug logs, which can help to troubleshoot problems in NGINX. Note that it is highly unlikely that a problem you encounter with the Ingress Controller is caused by a bug in the NGINX code, but it is rather caused by NGINX misconfiguration. Thus, this method is rarely needed.
   153  
   154  To enable the debug mode, set the `error-log-level` to `debug` in the [ConfigMap](/nginx-ingress-controller/configuration/global-configuration/configmap-resource) and use the `-nginx-debug` [command-line argument](/nginx-ingress-controller/configuration/global-configuration/command-line-arguments) when running the Ingress Controller.