How SVGZO validates and deduplicates 3.3M vector files

One pipeline, run on the files that ship

The same ingest code that produces the previews on this site produces the sample packs and scores the catalogue. That matters because a validation report generated by a different tool from the one that packaged the files is a report about something else.

Validation

Every file is parsed and rendered with resvg, and cross-checked against a second renderer so that a file which happens to suit one engine is not mistaken for a file that renders correctly. Dimensions and viewBox must be present and consistent. Anything that fails is set aside rather than shipped, and the failure is recorded.

Files carrying an embedded bitmap are excluded from delivery, samples and previews. So are files with scripts, external references or foreign objects, because a self-contained document is the point.

Deduplication

Two passes, because they catch different things. A SHA-256 over the canonicalised document catches files that are structurally identical however their source text differs. A perceptual hash over the render catches files that differ in the source but look the same.

Both thresholds are published with the rate. A duplicate rate quoted without its threshold is not a measurement.

Fonts and notices

Embedded fonts are read from the file itself: the copyright and licence fields in the font's own name table, falling back to the published listing for the family. A file whose embedded font cannot be resolved to a libre licence is excluded rather than shipped, and every notice that does ship records where its copyright and licence came from.