github.com/ipni/storetheindex@v0.8.30/README.md (about) 1 # StoreTheIndex 🗂️ 2 [![](https://img.shields.io/badge/made%20by-Protocol%20Labs-blue.svg?style=flat-square)](https://protocol.ai) 3 [![Go Reference](https://pkg.go.dev/badge/github.com/ipni/storetheindex.svg)](https://pkg.go.dev/github.com/ipni/storetheindex) 4 [![Coverage Status](https://codecov.io/gh/ipni/storetheindex/branch/main/graph/badge.svg)](https://codecov.io/gh/ipni/storetheindex/branch/main) 5 > The first place to go in order to find a CID stored in Filecoin 6 7 This repo provides an indexer implementation that can be used to index data stored by a range of participating storage providers. 8 9 ## Design 10 - [IPNI: InterPlanetary Network Indexer](https://github.com/ipni/specs/blob/main/IPNI.md#ipni-interplanetary-network-indexer) 11 12 ## Current Status 13 Released for production: The current production release is running at https://cid.contact 14 15 This project and is currently under active development 🚧. 16 17 ## Install 18 This assumes go is already installed. 19 20 Install storetheindex: 21 ```sh 22 go install github.com/ipni/storetheindex@latest 23 ``` 24 25 Initialize the storetheindex repository and configuration: 26 ```sh 27 storetheindex init 28 ``` 29 30 Optionally, edit the configuration 31 ```sh 32 edit ~/.storetheindex/config 33 ``` 34 35 ## Running the Indexer Service 36 To run storetheindex as a service, run the `daemon` command. The service watches for providers to index, and exposes a query / content routing client interface. 37 38 The daemon is configured by the config file in the storetheindex repository. The config file and repo are created when storetheindex is initialized, using the `init` command. This repo is located in the local file system. By default, the repo is located at ~/.storetheindex. To change the repo location, set the `$STORETHEINDEX_PATH` environmental variable. 39 40 ## Provider Removal Policy 41 After a configured amount of time without any updates from a provider (`PollInterval`), the indexer will poll the provider at its last know publisher address, for any index updates. If there is no response from the provider after at least one attempt to poll, then the provider is considered inactive and is not returned in any find results. The indexer will continue polling on an interval (`PollRetryAfter`) until a time limit (`PollStopAfter`) is reached. If there is still no response to the poll attempts after this time limit is reached, then the provider is removed from the indexer and its records are garbage-collected and will need to be refetched. 42 43 The configuration values that control this are documented [here](https://pkg.go.dev/github.com/ipni/storetheindex/config#Discovery), and their default values are specified [here](https://github.com/ipni/storetheindex/blob/main/doc/config.md#discovery). A custom polling configuration may be applied for specific providers using the `PollOverrides` configuration value to specify per-provider [Polling configuration](https://pkg.go.dev/github.com/ipni/storetheindex/config#Polling). 44 45 ## Indexer CLI Commands 46 There are a number of client commands included with storetheindex. Their purpose is to perform simple indexing and lookup actions against a running daemon. These can be helpful to test that an indexer is working. These include the following commands: 47 48 Informational: 49 50 - `find` Find value by CID or multihash in indexer 51 - `providers` Show information about providers known to the indexer 52 - `get` Get information about a specified provider 53 - `list` List the known providers 54 55 Administrative: 56 57 - `admin` Perform admin activities with an indexer 58 - `allow` Allow advertisements and content from peer 59 - `block` Block advertisements and content from peer 60 - `import-providers` Import provider information from another indexer 61 - `reload-config` Reload various settings from the configuration file 62 - `sync` Sync indexer with provider 63 - `init` Initialize or upgrade indexer node config file 64 65 Testing: 66 67 - `import` Imports data to indexer from different sources 68 - `register` Register provider information with an indexer 69 - `synthetic` Generate synthetic load to import in indexer 70 71 ## Help 72 To see a list of available commands, see `storetheindex --help`. For help with command usage, see `storetheindex <command> --help`. 73 74 75 ## Configuration 76 The storetheindex config file [documentation](https://github.com/ipni/storetheindex/blob/main/doc/config.md#the-storetheindex-config-file) 77 78 ## License 79 [SPDX-License-Identifier: Apache-2.0 OR MIT](LICENSE.md)