fix(benchmark): rank by sheet cost so engines can't game the score

The benchmark is about to be used as the objective for LLM-designed
engines, and several gaps would have rewarded the wrong behavior:

- Ranking was utilization-first, so dropping awkward parts raised the
  score. Rank valid > fully placed > cost > plates, where cost is
  salvage-credited sheet area plus a largest-sheet penalty per unplaced
  part; placing a part is never scored worse than omitting it.
- Salvage rate was ignored in scoring; cost now uses EstimateNetArea,
  recomputed from job geometry rather than trusted from the engine.
- Rotation constraints were never validated. Add RotationPolicy.Allows
  (shared with NestJobPlacementValidator) and check every placement.
- Returned sheets were trusted, so an engine could loosen spacing or
  invent a size. Sheets must now match offered stock.
- Part-in-part placements were flagged as overlaps; spacing now accounts
  for cutouts, with an X-sorted sweep to prune distant pairs.
- Summary averaged per-job percentages; it now sums areas and cost.
- --spacing and sheet sizes parsed with the current culture.
- Warn when .nest jobs offer only their original sheet sizes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
aj
2026-09-22 15:05:43 -04:00
co-authored by Claude Opus 5.5
parent 98c4929d14
commit 57e9f625b6
13 changed files with 669 additions and 84 deletions
+2 -2
View File
@@ -84,8 +84,8 @@ Compares registered `INestingEngine` implementations against each other on real
- `DxfManifestLoader` builds a `BenchmarkJob` from a JSON manifest (`sheetSizes`, `spacing`, `edgeSpacing`, `quadrant`, `parts[] { dxf, quantity, allowRotation }`) instead of a `.nest`, importing each DXF with `CadImporter.ImportDrawing`. DXF paths resolve relative to the manifest; sheet sizes are required (manifest or `--sheet-sizes`, which overrides). `allowRotation: false` locks rotation the same way `NestRunner` does. `JobLoader.Load` routes `*.json` inputs to it, and folder scans pick up `*.nest` plus `*.manifest.json` (plain `*.json` is ignored so `--output` reports are never read as manifests). Invalid manifests throw rather than being skipped.
- `BenchmarkJob.BuildNestJob(maxPlates)` converts the job into a `NestJob`: one `NestJobPart` per requested drawing (via `DrawingJobMapper.FromDrawing`) and one `NestPlateStock` per candidate sheet size (unlimited quantity — the engine decides how many of each size it uses).
- `BenchmarkRunner` fans the (job × engine) pairs out with `Parallel.ForEach` (`NoBuffering`, `MaxDegreeOfParallelism` from `--parallel`, CLI default 3, `Run`'s own default 1) and writes results by index so report order stays job-then-engine. Each solve builds its own `NestJob` snapshot and materialized drawings, so solves share no mutable drawing state. Concurrent solves compete for cores, so `Time(ms)` is only clean at `--parallel 1`. It calls each engine's `INestingEngine.Solve(NestJob)` once per job, under a wall-clock timeout so a runaway or hanging engine can't stall the whole benchmark run, then materializes the result back into legacy `Plate`/`Part` objects via `NestResultMaterializer` for scoring.
- `NestValidator` checks the returned layout: every part inside `Plate.WorkArea()`, every pair at least `Plate.PartSpacing` apart (checked geometrically via each part's own world-space polygon, inflated by the spacing — works on arbitrary concave/holed shapes, not just bounding boxes), and no drawing over its requested quantity. An invalid, throwing, or timed-out run scores zero for that job.
- Scoring matches `Plate.Utilization()` (placed drawing area / full sheet area, `Plate.Area()`). If an engine placed every requested part, ties are broken by fewer plates used (`Report`'s ranking rule) — using fewer sheets to do the same job wastes less material.
- `NestValidator` checks the returned layout: every part inside `Plate.WorkArea()`, every pair at least `Plate.PartSpacing` apart (checked geometrically: each part's perimeter inflated and cutouts shrunk by the spacing, tested against the other part's raw material with holes subtracted, so part-in-part inside a cutout is legal; an X-sorted bounding-box sweep prunes distant pairs), and no drawing over its requested quantity. `ValidateAgainstJob` also checks the raw `NestJobResult`: every sheet must match a stock entry the job offered (size, spacing, edge spacing, quadrant; finite quantity not overdrawn), and every placement rotation must satisfy its part's `RotationPolicy.Allows`. An invalid, throwing, or timed-out run places nothing for scoring.
- Ranking (`Report.Compare`): valid > invalid, fully placed > not, then lower `JobResult.Cost`, then fewer plates. Cost = salvage-credited sheet area (`StockLadderNestingEngine.EstimateNetArea` per plate, recomputed from job geometry) + `BenchmarkJob.UnplacedPartPenalty` (largest candidate sheet area) per unplaced part, so dropping hard parts never improves the score. The summary sums cost and areas across jobs (area-weighted, not a mean of per-job percentages). Without `--sheet-sizes`, `.nest` jobs only offer their original sizes, and the CLI warns that this hints engines. Numeric CLI and manifest sheet sizes parse with the invariant culture (`JobLoader.TryParseSheetSize`).
- `--engines Name1,Name2` filters to specific registered engines (default: all); `--csv <path>` writes a flat per-job CSV alongside the console report.
### OpenNest.Mcp (console app, depends on Core + Engine + IO)