github.com/google/syzkaller@v0.0.0-20251211124644-a066d2bc4b02/pkg/coveragedb/README.md (about)

     1  # Prepare coverage aggregation pipeline
     2  
     3  Assuming you have the coverage `*.jsonl` files in some bucket:
     4  1. Create BigQuery table.
     5  2. Start data transfers from the bucket to BigQuery table.
     6  
     7  Coverage merger job is consuming data from BQ table and store aggregations
     8  in the Spanner DB.
     9  
    10  The dots in BQ table name are not supported, thus:
    11  1. For the namespace "upstream" the expected BQ table name is "upstream".
    12  2. For the namespace "android-6.12" the expected BQ table name is "android-6-12".
    13  
    14  
    15  ### Create new BigQuery table for coverage data
    16  ```bash
    17  bq mk \
    18    --table \
    19    --description "android 6.12"
    20    --time_partitioning_field timestamp \
    21    --time_partitioning_type DAY \
    22    --require_partition_filter=true \
    23    --clustering_fields file_path,kernel_commit,hit_count \
    24    syzkaller:syzbot_coverage.android-6-12 \
    25    ./pkg/coveragedb/bq-schema.json
    26  ```
    27  
    28  ### Add new data transfer
    29  ```bash
    30  bq mk \
    31    --transfer_config \
    32    --display_name=ci-android-6-12-bucket-to-syzbot_coverage \
    33    --params='{"destination_table_name_template":"android-6-12",
    34    "data_path_template": "gs://$COVERAGE_STREAM_BUCKET/ci-android-6.12/*.jsonl",
    35    "allow_jagged_rows": false,
    36    "allow_quoted_newlines": false,
    37    "delete_source_files": true,
    38    "encoding": "UTF8",
    39    "field_delimiter": ",",
    40    "file_format": "JSON",
    41    "ignore_unknown_values": false,
    42    "max_bad_records": "0",
    43    "parquet_enable_list_inference": false,
    44    "parquet_enum_as_string": false,
    45    "preserve_ascii_control_characters": false,
    46    "skip_leading_rows": "0",
    47    "use_avro_logical_types": false,
    48    "write_disposition": "APPEND"
    49    }' \
    50    --project_id=syzkaller \
    51    --data_source=google_cloud_storage \
    52    --target_dataset=syzbot_coverage
    53  ```
    54  
    55  ### List BigQuery data transfers
    56  ```bash
    57  bq ls \
    58    --transfer_config \
    59    --transfer_location=us-central1 \
    60    --project_id=syzkaller
    61  ```