github.com/ipni/storetheindex@v0.8.30/README.md (about)

     1  # StoreTheIndex 🗂️
     2  [![](https://img.shields.io/badge/made%20by-Protocol%20Labs-blue.svg?style=flat-square)](https://protocol.ai)
     3  [![Go Reference](https://pkg.go.dev/badge/github.com/ipni/storetheindex.svg)](https://pkg.go.dev/github.com/ipni/storetheindex)
     4  [![Coverage Status](https://codecov.io/gh/ipni/storetheindex/branch/main/graph/badge.svg)](https://codecov.io/gh/ipni/storetheindex/branch/main)
     5  > The first place to go in order to find a CID stored in Filecoin
     6  
     7  This repo provides an indexer implementation that can be used to index data stored by a range of participating storage providers.
     8  
     9  ## Design
    10  - [IPNI: InterPlanetary Network Indexer](https://github.com/ipni/specs/blob/main/IPNI.md#ipni-interplanetary-network-indexer)
    11  
    12  ## Current Status
    13  Released for production: The current production release is running at https://cid.contact 
    14  
    15  This project and is currently under active development 🚧.  
    16  
    17  ## Install
    18  This assumes go is already installed.
    19  
    20  Install storetheindex:
    21  ```sh
    22  go install github.com/ipni/storetheindex@latest
    23  ```
    24  
    25  Initialize the storetheindex repository and configuration:
    26  ```sh
    27  storetheindex init
    28  ```
    29  
    30  Optionally, edit the configuration
    31  ```sh
    32  edit ~/.storetheindex/config 
    33  ```
    34  
    35  ## Running the Indexer Service
    36  To run storetheindex as a service, run the `daemon` command. The service watches for providers to index, and exposes a query / content routing client interface.
    37  
    38  The daemon is configured by the config file in the storetheindex repository. The config file and repo are created when storetheindex is initialized, using the `init` command. This repo is located in the local file system. By default, the repo is located at ~/.storetheindex. To change the repo location, set the `$STORETHEINDEX_PATH` environmental variable.
    39  
    40  ## Provider Removal Policy
    41  After a configured amount of time without any updates from a provider (`PollInterval`), the indexer will poll the provider at its last know publisher address, for any index updates. If there is no response from the provider after at least one attempt to poll, then the provider is considered inactive and is not returned in any find results. The indexer will continue polling on an interval (`PollRetryAfter`) until a time limit (`PollStopAfter`) is reached. If there is still no response to the poll attempts after this time limit is reached, then the provider is removed from the indexer and its records are garbage-collected and will need to be refetched. 
    42  
    43  The configuration values that control this are documented [here](https://pkg.go.dev/github.com/ipni/storetheindex/config#Discovery), and their default values are specified [here](https://github.com/ipni/storetheindex/blob/main/doc/config.md#discovery). A custom polling configuration may be applied for specific providers using the `PollOverrides` configuration value to specify per-provider [Polling configuration](https://pkg.go.dev/github.com/ipni/storetheindex/config#Polling).
    44  
    45  ## Indexer CLI Commands
    46  There are a number of client commands included with storetheindex. Their purpose is to perform simple indexing and lookup actions against a running daemon.  These can be helpful to test that an indexer is working. These include the following commands:
    47  
    48  Informational:
    49  
    50  - `find` Find value by CID or multihash in indexer
    51  - `providers` Show information about providers known to the indexer
    52    - `get` Get information about a specified provider
    53    - `list` List the known providers
    54  
    55  Administrative:
    56  
    57  - `admin` Perform admin activities with an indexer
    58    - `allow` Allow advertisements and content from peer
    59    - `block` Block advertisements and content from peer
    60    - `import-providers` Import provider information from another indexer
    61    - `reload-config` Reload various settings from the configuration file
    62    - `sync` Sync indexer with provider
    63  - `init` Initialize or upgrade indexer node config file
    64  
    65  Testing:
    66  
    67  - `import` Imports data to indexer from different sources
    68  - `register` Register provider information with an indexer
    69  - `synthetic` Generate synthetic load to import in indexer
    70  
    71  ## Help
    72  To see a list of available commands, see `storetheindex --help`. For help with command usage, see `storetheindex <command> --help`.
    73  
    74  
    75  ## Configuration
    76  The storetheindex config file [documentation](https://github.com/ipni/storetheindex/blob/main/doc/config.md#the-storetheindex-config-file)
    77  
    78  ## License
    79  [SPDX-License-Identifier: Apache-2.0 OR MIT](LICENSE.md)