github.com/treeverse/lakefs@v1.24.1-0.20240520134607-95648127bfb0/design/open/gc-on-parquet.md (about) 1 ## Garbage collection using Parquet format 2 3 ### Motivation 4 5 Garbage collection in lakeFS is struggling at large scale. 6 7 ### Why Parquet? 8 9 Parquet is more than a file format. It's a breathing open-source project aiming to provide exactly what we're looking for here: efficient data retrieval. GC is essentially a huge anti-join, and it may be very difficult to achieve a similar optimization in our dedicated format. 10 11 12 ### Steps 13 14 #### 1. Sync 15 Ranges of the repository are translated into Parquet files. These are stored in a new metadata location under the repository's storage namespace. 16 17 #### 2. Sanity 18 Verify all the ranges required for this run are present, and that the number of entries in each range is correct. 19 20 #### 3. Start 21 Do an anti-join extracting all the addresses that need to be deleted. 22 23 ### Considerations 24 1. Storage: this requires using more of the user's storage. I don't think that's an issue since metadata is usually very small compared to the data size. In the future we can also optimize the set of ranges we actually bring there. 25 1. Risk: we are using a copy of the metadata instead of the actual one. We need to be careful not to miss anything, since missing data may potentially cause innocent objects to be deleted. This is a risk that any GC solution needs to take into account. 26 1. Ops burden: we will have to maintain and monitor the copying of the metadata. In case it fails, we need to recover it.