github.com/cilium/cilium@v1.16.2/Documentation/contributing/docs/docsframework.rst (about) 1 .. only:: not (epub or latex or html) 2 3 WARNING: You are looking at unreleased Cilium documentation. 4 Please use the official rendered version released here: 5 https://docs.cilium.io 6 7 .. _docs_framework: 8 9 *********************** 10 Documentation framework 11 *********************** 12 13 This page contains notes on the framework in use for Cilium documentation. Its 14 objective is to help contributors understand the tools and build process for 15 the documentation, and to help maintain it. 16 17 Alas, this sort of document goes quickly out of date. When in doubt of 18 accuracy, double-check the codebase to verify information. If you find 19 discrepancies, please update this page. 20 21 Sphinx 22 ====== 23 24 Cilium relies on `Sphinx`_ to generate its documentation. 25 26 .. _Sphinx: https://www.sphinx-doc.org 27 28 Sphinx usage 29 ------------ 30 31 Contributors do not usually call Sphinx directly, but rather use the Makefile 32 targets defined in ``Documentation/Makefile``. For instructions on how to 33 quickly render the documentation, see :ref:`testing documentation 34 <testing-documentation>`. 35 36 Sphinx features 37 --------------- 38 39 Here are some specific Sphinx features used in Cilium's documentation: 40 41 - `Tab groups`_ 42 43 - `OpenAPI`_ documentation generation 44 45 - Mark-up languages: reStructuredText (rST) and Markdown (`MyST`_ flavor) 46 47 - Substitutions, for example: 48 49 - ``|SCM_WEB|`` 50 - ``|CHART_VERSION|`` 51 52 - Multiple versions (for all supported branches, plus two aliases: ``stable`` 53 and ``latest``) 54 55 .. _OpenAPI: https://github.com/sphinx-contrib/openapi 56 .. _Tab groups: https://github.com/executablebooks/sphinx-tabs/ 57 .. _MyST: https://myst-parser.readthedocs.io 58 59 Sphinx version 60 -------------- 61 62 The version of Sphinx in use is defined in 63 ``Documentation/requirements-min/requirements.txt``. For more details, see the 64 :ref:`section on requirements <docs_requirements>`. 65 66 Auto-generated contents 67 ======================= 68 69 Some contents are automatically generated at build time. File 70 ``Documentation/Makefile`` contains the following target, shown here in a 71 simplified version, which regenerates a number of documents and then checks 72 that they are all up-to-date: 73 74 .. code-block:: makefile 75 76 check: builder-image api-flaggen update-cmdref update-crdlist update-helm-values update-codeowners 77 ./check-cmdref.sh 78 ./check-helmvalues.sh 79 $(DOCKER_RUN) ./check-examples.sh # Runs "cilium policy validate" and "yamllint" 80 ./check-codeowners.sh 81 ./check-flaggen.sh 82 ./check-crdlist.sh 83 84 Regeneration happens when the different dependency targets for ``check`` are 85 run. They are: 86 87 - ``api-flaggen`` 88 89 - Runs ``go run tools/apiflaggen`` 90 - Generates ``Documentation/configuration/api-restrictions-table.rst`` 91 92 - ``update-cmdref`` 93 94 - Runs ``./update-cmdref.sh`` 95 - Includes running various binaries with ``--cmdref`` 96 - Generates ``Documentation/cmdref/\*`` 97 98 - ``update-crdlist`` 99 100 - ``make -C ../ generate-crd-docs`` 101 - Runs ``tools/crdlistgen/main.go`` 102 - Parses docs to list CRDs 103 - Generates ``Documentation/crdlist.rst`` 104 105 - ``update-helm-values`` 106 107 - Generates from ``install/kubernetes`` 108 - Generates ``Documentation/helm-values.rst`` 109 110 - ``update-codeowners`` 111 112 - ``./update-codeowners.sh`` 113 - Synchronizes teams description from ``CODEOWNERS`` 114 - Generates ``Documentation/codeowners.rst`` 115 116 Other auto-generated contents include: 117 118 - OpenAPI reference 119 120 - YAML generated from the ``Makefile`` at the root of the repository 121 - Relies on the contents of ``api``, linked as ``Documentation/_api`` 122 - Processed and included via a dedicated add-on, from 123 ``Documentation/api.rst``: ``.. openapi:: ../api/v1/openapi.yaml`` 124 125 - gRPC API reference 126 127 - Markdown generated from the main ``Makefile`` at the root of the repository 128 - Relies on the contents of ``api``, linked as ``Documentation/_api`` 129 - Included from ``Documentation/grpcapi.rst`` 130 131 Build system 132 ============ 133 134 Makefile targets 135 ---------------- 136 137 Here are the main ``Makefile`` targets related to documentation to run from the 138 root of the Cilium repository, as well as some indications on what they call: 139 140 - ``make`` -> ``all: ... postcheck`` -> ``make -C Documentation check``: 141 Build Cilium and validate the documentation via the ``postcheck`` target 142 - ``make -C Documentation html``: 143 Render the documentation as HTML 144 - ``make test-docs`` -> ``make -C Documentation html``: 145 Render the documentation as HTML 146 - ``make -C Documentation live-preview``: 147 Build the documentation and start a server for local preview 148 - ``make render-docs`` -> ``make -C Documentation live-preview``: 149 Build the documentation and start a server for local preview 150 151 Generating documentation 152 ------------------------ 153 154 - The ``Makefile`` builds the documentation using the ``docs-builder`` Docker 155 image. 156 157 - The build includes running ``check-build.sh``. This script: 158 159 a. Runs the linter (``rstcheck``), unless the environment variable 160 ``SKIP_LINT`` is set 161 b. Runs the spell checker 162 c. Builds the HTML version of the documentation 163 d. Exits with an error if any unexpected warning or error is found 164 165 Tweaks and tools 166 ================ 167 168 See also file ``Documentation/conf.py``. 169 170 Spell checker 171 ------------- 172 173 The build system relies on Sphinx's `spell-checker module`_ (considered a 174 `builder`_ in Sphinx). 175 176 The spell checker uses a list of known exceptions contained in 177 ``Documentation/spelling_wordlist.txt``. Words in the list that are written 178 with lowercase exclusively, or uppercase exclusively, are case-insensitive 179 exceptions for spell-checking. Words with mixed case are case-sensitive. Keep 180 this file sorted alphabetically. 181 182 To add new entries to the list, run ``Documentation/update-spelling_wordlist.sh``. 183 184 To clean-up obsolete entries, first make sure the spell checker reports no 185 issue on the current version of the documentation. Then remove all obsolete 186 entries from the file, run the spell checker, and re-add all reported 187 exceptions. 188 189 Cilium's build framework uses a custom filter for the spell checker, for 190 spelling ``WireGuard`` correctly as ``WireGuard``, or ``wireguard`` in some 191 contexts, but never as ``Wireguard``. This filter is implemented in 192 ``Documentation/_exts/cilium_spellfilters.py`` and registered in 193 ``Documentation/conf.py``. 194 195 .. _spell-checker module: https://github.com/sphinx-contrib/spelling 196 .. _builder: https://www.sphinx-doc.org/en/master/usage/builders 197 198 :spelling:word:`rstcheck` 199 ------------------------- 200 201 The documentation framework relies on `rstcheck`_ to validate the rST 202 formatting. There is a list of warnings to ignore, in part because the linter 203 has bugs. The call to the tool, and this list of exceptions, are configured in 204 ``Documentation/check-build.sh``. 205 206 .. _rstcheck: https://rstcheck.readthedocs.io 207 208 Link checker 209 ------------ 210 211 The documentation framework has a link checker under 212 ``Documentation/check-links.sh``. However, due to some unsolved issues, it does 213 not run in CI. See :gh-issue:`27116` for details. 214 215 Web server for local preview 216 ---------------------------- 217 218 Launch a web server to preview the generated documentation locally with ``make 219 render-docs``. 220 221 For more information on this topic, see :ref:`testing documentation 222 <testing-documentation>`. 223 224 Custom Sphinx roles 225 ------------------- 226 227 The documentation defines several custom roles: 228 229 - ``git-tree`` 230 - ``github-project`` 231 - ``github-backport`` 232 - ``gh-issue`` 233 - ``prev-docs`` 234 235 Calling these roles helps insert links based on specific URL templates, via the 236 `extlinks`_ extension. They are all configured in ``Documentation/conf.py``. 237 They should be used wherever relevant, to ensure that formatting for all links 238 to the related resources remain consistent. 239 240 .. _extlinks: https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html 241 242 Custom Sphinx directives 243 ------------------------ 244 245 Cilium's documentation does not implement custom directives as of this writing. 246 247 Custom extensions 248 ----------------- 249 250 Cilium's documentation uses custom extensions for Sphinx, implemented under 251 ``Documentation/_exts``. 252 253 - One defines the custom filters for the spell checker. 254 - One patches Sphinx's HTML translator to open all external links in new tabs. 255 256 Google Analytics 257 ---------------- 258 259 The documentation uses Google Analytics to collect metrics. This is configured 260 in ``Documentation/conf.py``. 261 262 Customization 263 ------------- 264 265 Here are additional elements of customization for Cilium's documentation 266 defined in the main repository: 267 268 - Some custom CSS; see also class ``wrapped-table`` in the related CSS file 269 ``Documentation/_static/wrapped-table.css`` 270 271 - A "Copy" button, including a button to copy only commands from console-code 272 blocks, implemented in ``Documentation/_static/copybutton.js`` and 273 ``Documentation/_static/copybutton.css`` 274 275 - Custom header and footer definitions, for example to make link to Slack 276 target available on all pages 277 278 - Warning banner on older branches, telling to check out the latest version 279 (these may be handled directly in the ReadTheDocs configuration in the 280 future, see also :gh-issue:`29969`) 281 282 Algolia search engine 283 --------------------- 284 285 - :spelling:word:`Algolia` provides a search engine for the documentation website. See also the 286 repository for the `DocSearch scraper`_. 287 288 .. _DocSearch scraper: https://github.com/cilium/docsearch-scraper-webhook 289 290 Build set up 291 ============ 292 293 .. _docs_requirements: 294 295 Requirements (dependencies) 296 --------------------------- 297 298 The repository contains two files for requirements: one that declares and pins 299 the core dependencies for the documentation build system, and that maintainers 300 use to generate a second requirement files that includes all sub-dependencies, 301 via a dedicated Makefile target. 302 303 - The base requirements are defined in 304 ``Documentation/requirements-min/requirements.txt``. 305 - Running ``make -C Documentation update-requirements`` uses this file as a 306 base to generate ``Documentation/requirements.txt``. 307 308 Dependencies defined in ``Documentation/requirements-min/requirements.txt`` 309 should never be updated in ``Documentation/requirements.txt`` directly. 310 Instead, update the former and regenerate the latter. 311 312 File ``Documentation/requirements.txt`` is used to build the ``docs-builder`` 313 Docker image. 314 315 Dependencies defined in these requirements files include the documentation's 316 custom theme. 317 318 Docker set-up 319 ------------- 320 321 The documentation build system relies on a Docker image, ``docs-builder``, to 322 ensure the build environment is consistent across different systems. Resources 323 related to this image include ``Documentation/Dockerfile`` and the requirement 324 files. 325 326 Versions of this image are automatically built and published to a registry when 327 the Dockerfile or the list of dependencies is updated. This is handled in CI 328 workflow ``.github/workflows/build-images-docs-builder.yaml``. 329 330 If a Pull Request updates the Dockerfile or its dependencies, have someone run 331 the two-steps deployment described in this workflow to ensure that the CI picks 332 up an updated image. 333 334 ReadTheDocs 335 ----------- 336 337 Cilium's documentation is hosted on ReadTheDocs. The main configuration options 338 are defined in ``Documentation/.readthedocs.yaml``. 339 340 Some options, however, are only configurable in the ReadTheDocs web interface. 341 For example: 342 343 - The location of the configuration file in the repository 344 - Redirects 345 - Triggers for deployment 346 347 Custom theme 348 ============ 349 350 The online documentation uses a custom theme based on `the ReadTheDocs theme`_. 351 This theme is defined in its `dedicated sphinx_rtd_theme fork repository`_. 352 353 .. _the ReadTheDocs theme: https://github.com/readthedocs/sphinx_rtd_theme 354 .. _dedicated sphinx_rtd_theme fork repository: 355 https://github.com/cilium/sphinx_rtd_theme/ 356 357 Do not use the ``master`` branch of this repository. The commit or branch to 358 use is referenced in ``Documentation/requirements.txt``, generated from 359 ``Documentation/requirements-min/requirements.txt``, in the Cilium repository. 360 361 CI checks 362 ========= 363 364 There are several workflows relating to the documentation in CI: 365 366 - Documentation workflow: 367 368 - Defined in ``.github/workflows/documentation.yaml`` 369 - Tests the build, runs the linter, checks the spelling, ensures auto-generated 370 contents are up-to-date 371 - Runs ``./Documentation/check-builds.sh html`` from the ``docs-builder`` 372 image 373 374 - Netlify preview: 375 376 - Hook defined at Netlify, configured in Netlify's web interface 377 - Checks the build 378 - Used for previews on Pull Requests, but *not* for deploying the 379 documentation 380 - Uses a separate Makefile target (``html-netlify``), runs ``check-build.sh`` 381 with ``SKIP_LINT=1`` 382 383 - Runtime tests: 384 385 - In the absence of updates to the Dockerfile or documentation dependencies, 386 runtime tests are the only workflow that always rebuilds the 387 ``docs-builder`` image before generating the docs. 388 389 - Image update workflow: 390 391 - Rebuilds the ``docs-builder`` image, pushes it to Quay.io, and updates the 392 image reference with the new one in the documentation workflow 393 - Triggers when requirements or ``Documentation/Dockerfile`` are updated 394 - Needs approval from one of the ``docs-structure`` team members 395 396 Redirects 397 ========= 398 399 Some pages change location or name over time. To improve user experience, there 400 is a set of redirects in place. These redirects are configured from the 401 ReadTheDocs interface. They are a pain to maintain. 402 403 Redirects could possibly be configured from existing, dedicated Sphinx 404 extensions, but this option would require research to analyze and implement.