github.com/web-platform-tests/wpt.fyi@v0.0.0-20240530210107-70cf978996f1/api/query/cache/README.md (about)

     1  # Searchcache
     2  
     3  This directory contains the implementation of the searchcache for wpt.fyi. The
     4  searchcache runs as a separate service from the main dashboard, providing a
     5  caching layer between the dashboard and the backing test runs datastore for
     6  complex search queries.
     7  
     8  ## Developing locally
     9  
    10  It is possible to run the searchcache locally when developing, though it is
    11  difficult to also have it talk to a local build of the dashboard. However
    12  running the searchcache and then communicating with it via `curl` serves most
    13  development needs.
    14  
    15  ### Prerequisites
    16  
    17  Outside of the standard setup for developing wpt.fyi (see the main
    18  CONTRIBUTING.md file), searchcache development requires:
    19  
    20  1. At least 4GB of RAM (8GB+ preferable). The searchcache loves RAM more than
    21     Chrome does.
    22  1. A GCP [service account](https://cloud.google.com/iam/docs/understanding-service-accounts)
    23     with the role 'Cloud Datastore User' for the `wptdashboard-staging` project.
    24     The key JSON file for this account must be available locally (called
    25     `gcp.json` in this README).
    26  
    27  ### Building
    28  
    29  From the `service/` subdirectory, run:
    30  
    31  ```sh
    32  go build
    33  ```
    34  
    35  This should create a new binary, `service`.
    36  
    37  ### Running
    38  
    39  **Do not run the searchcache binary without flags!**. It is configured to grab
    40  pretty much all of the RAM on the machine it runs on - great for a VM, bad for
    41  your development machine.
    42  
    43  The following are required flags when running `service`:
    44  
    45  * `--project_id=wptdashboard-staging` - set the project to the staging instance
    46    of [wpt.fyi](https://wpt.fyi).
    47  * `--gcp_credentials_file=gcp.json` - provide the necessary credentials to
    48    access the datastore. See [Prerequisites](#prerequisites) above.
    49  * `--max_heap_bytes=256000000` - sets a soft limit on memory usage. Emphasis on
    50    'soft' - with a 256MB heap set here, the author had their searchcache
    51    stabilize at approximately 2 **GB** of memory.
    52  
    53  Putting it together:
    54  
    55  ```sh
    56  ./service \
    57      --project_id=wptdashboard-staging \
    58      --gcp_credentials_file=gcp.json \
    59      --max_heap_bytes=256000000
    60  ```
    61  
    62  Other flags can be found by passing `--help` to the `service` binary.
    63  
    64  **Tip**: The `service` binary is quite noisy (as it slurps up test runs from
    65  the datastore); it can be useful to pipe its output via `tee` and store the log
    66  locally to let you easily grep through it.
    67  
    68  ### Interacting with searchcache
    69  
    70  By default, searchcache is hosted on `localhost:8080`. It can be communicated
    71  with using HTTP requests. For example, using `curl`:
    72  
    73  ```sh
    74  curl -H "Content-Type: application/json" \
    75      -X POST \
    76      -d '{"run_ids":[267810084, 255750007],"query":{"exists":[{"is":"different"}]}}' \
    77      http://localhost:8080/api/search/cache
    78  ```