github.com/google/trillian-examples@v0.0.0-20240520080811-0d40d35cef0e/clone/cmd/ctclone/README.md (about)

     1  # `ctclone`
     2  
     3  This tool clones a Certificate Transparency (RFC6962) log.
     4  See background and database setup in the [parent docs](../../README.md).
     5  
     6  ## Cloning
     7  
     8  Assuming the database is provisioned, the log can be downloaded with:
     9  
    10  ```
    11  go run ./clone/cmd/ctclone --alsologtostderr --v=1 --log_url https://ct.googleapis.com/logs/xenon2022/ --mysql_uri 'clonetool:letmein@tcp(localhost)/google_xenon2022'
    12  ```
    13  
    14  ## Tuning
    15  
    16  In addition the general tuning flags (`workers` and `write_batch_size`) mentioned in the parents docs, the CT clone tool also has `fetch_batch_size`. As a rule of thumb, this should be set to the maximum size that the log supports for a single batch.
    17  
    18  ## Docker
    19  
    20  ### Docker Compose
    21  
    22  `docker-compose.yaml` is provided as an example that will clone a log into a database hosted in a local container.
    23  The benefit of this is that it can be run as a single command:
    24  
    25  ```bash
    26  docker compose up -d
    27  ```
    28  
    29  For Raspberry Pi users, there is a slight change in order to override the DB:
    30  
    31  ```bash
    32  docker compose -f docker-compose.yaml -f docker-compose.rpi.yaml up -d
    33  ```
    34  
    35  This will bring up two containers: `ctclone-db-1` and `ctclone-clone-xenon2022-1`.
    36  It is expected that users will write their own tools to query the DB, but the following command demonstrates the leaves being queried from the command line:
    37  
    38  ```bash
    39  docker exec -i ctclone-db-1 /usr/bin/mysql -uctclone -pletmein -Dxenon2022 <<< "select * from leaves where id < 5;"
    40  ```
    41  
    42  The sample files here clone xenon2022, though this can be changed to another log by updating the `docker-compose.yaml` and the `.env` file.
    43  For users that would like to clone multiple logs doing this, creating multiple databases inside the db container is possible but [more complicated](https://stackoverflow.com/questions/39204142/docker-compose-with-multiple-databases).
    44  
    45  ### Direct
    46  
    47  To build a docker image, run the following from the `trillian-examples` root directory:
    48  
    49  ```
    50  docker build . -t ctclone -f ./clone/cmd/ctclone/Dockerfile
    51  ```
    52  
    53  This can be pointed at a local MySQL instance running outside of docker using:
    54  
    55  ```
    56  docker run --name clone_xenon2022 -d ctclone --alsologtostderr --v=1 --log_url https://ct.googleapis.com/logs/xenon2022/ --mysql_uri 'clonetool:letmein@tcp(host.docker.internal)/google_xenon2022'
    57  ```