github.com/koneal2013/storymetadatagenerator@v0.0.0-20230222212341-b170f254daa5/README.md (about)

     1  # StoryMetadataGenerator
     2  
     3  A Go service that generates metadata about stories pulled from the Axios Story Stream.
     4  
     5  ### Set Up and Run
     6  
     7  You can start `storymetadatagenerator` in a terminal window or by building and running the required docker containers. See [docker-compose.yaml](./docker-compose.yaml) for a list of these contaniers and their configuration.
     8  
     9  To start the service run:
    10  
    11  ```
    12  make docker-start
    13  ```
    14  
    15  `storymetadatagenerator` will start an http server on port 8080 and grpc server on port 8081. Optionally, you can override the default configuration via the [config.json](./config.json) file. The config file ([config.json](./config.json)) is automatically passed as an argument to the `make start` and `make docker-start` commands (see [setup and run](#set-up-and-run)) and must be present in the project root to build.  See [config.example.json](./config.example.json) for an example of what values can be overridden.
    16  For a description of each config value, build an executable (see [Build](#Build)) and run:
    17  
    18  ```
    19  ./storymetadatagenerator -h
    20  ```
    21  
    22  Run `make docker-down` to stop all containers.
    23  
    24  ### Build
    25  
    26  To build an executable for your target architecture, run:
    27  
    28  ```
    29  make build
    30  ```
    31  The build command will run the tests and place an executable labeled `storymetadatagenerator` in the project root. This executable must be provided with the path to the [config.json](./config.json) file to run properly. See example below:
    32  
    33  ```
    34  ./storymetadatagenerator --config-file=./config.json
    35  ```
    36  
    37  Please note that running the service this way (or via `make start`) requires that the supporting docker containers (otel-collector, jaeger-all-in-one, zipkin-all-in-one, and prometheus) are running for the service to start. Use `make docker-start` to start all the containers then manually stop the `storymetadatagenerator-storymetadatagenerator-server-1` container so there are no port conflicts.
    38  
    39  ### Test
    40  
    41  The `make start` command will automatically run the tests. If you need to run the tests outside of this command, run `make docker-start` before executing the individual tests. Run `make docker-down` to teardown the containers after your tests have completed.
    42  
    43  A code coverage report can be generated by running the following command:
    44  
    45  ```
    46  make cover
    47  ```
    48  
    49  Additionally, you can use `make covero` to generate a code coverage report and immediatly open the report in a web browser.
    50  
    51  Code coverage output is stored [here](/.coverage)
    52  
    53  ### Http Server
    54  
    55  The http server exposes two endpoints:
    56  
    57  | Method | URI                | Name                                            | Input                                                                              | Summary                                                                                                                                                                        |
    58  | ------ | ------------------ | ----------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
    59  | GET    | /status            | [get status](./internal/server/http.go)         | None.                                                                              | Returns 20 OK if server is ready to accept requests.                                                                                                                           |
    60  | GET    | /v1/story-metadata | [get story-metadata](./internal/server/http.go) | Source: Body. Number of story pages to retrieve. Defaults to 1 if '0' is provided. | Generates Story metadata for 'N' requested story stream pages. Returns story metadata for each story that completed sucessfully and a list of errors for stories that did not. |
    61  
    62  ### Grpc Server
    63  
    64  The protocol buffers for the Grpc server can be found [here](./api/v1/grpc/storymetadata.proto)
    65  
    66  ### JaegerUI
    67  
    68  Traces are collected via an opentelemetry collector and sent to an instance of JagerUI running on port 16681.
    69  
    70  ![image.png](./assets/jaeger.png)
    71  
    72  ![image.png](./assets/jaeger2.png)
    73  
    74  ### ZipkinUI
    75  
    76  Traces are collected via an opentelemetry collector and sent to an instance of ZipkinUI running on port 9411.
    77  
    78  ![image.png](./assets/zipkin.png)
    79  
    80  ### TODO
    81  
    82  A list of tasks that I would have liked to incorporate in my solution but did not have the time to can be found [here](./TODO.md).
    83  
    84  ### Assumptions
    85  
    86  I thought it would be more efficient to continue processing the story stream pages even if one story encountered an error. Therefore, the response from both the Http server and the Grpc server return the following JSON:
    87  
    88  ```
    89  {
    90  "stories": {
    91      "03aa8c72-7007-46c0-b35c-d3cefdb5aee9": {
    92        "word_count": 235,
    93        "reading_time": "<1",
    94        "headline": "Manchin on 2024 political ambitions:",
    95        "permalink": "https://www.axios.com/2023/01/22/joe-manchin-2024-senate-democrats"
    96      },
    97    },
    98  "errors": {},
    99  }
   100  ```
   101  
   102  Errors are ommited form the response if the `errors` field is empty. I chose this approach to provide more flexibility for the comsuming service to determine how individual story errors should be handeled.
   103  
   104  Please feel free to email me at [koneal2013@gmail.com](mailto:koneal2013@gmail.com) with questions regarding my solution.
   105