diff --git a/AGENTS.md b/AGENTS.md index 239ab797..220fcdc8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,7 +30,7 @@ dotnet test OpenNest.FrontEnd.Tests/OpenNest.FrontEnd.Tests.csproj -c Release dotnet test OpenNest.WinForms.Tests/OpenNest.WinForms.Tests.csproj ``` -Only Plan Cutting changes need the [problem-nest corpus](docs/problem-nests.md) (`Category=ProblemNests`, opt-in via `OPENNEST_PROBLEM_NESTS_SERVER`); never run it for nesting engines, fills, UI, posts or file formats. Limit it to the areas the change touches with `OPENNEST_PROBLEM_NESTS_CHECKS` (table in that doc); no plate may rank below its baseline. Keep desktop-dependent tests in `OpenNest.WinForms.Tests`, never add a WinForms reference to `OpenNest.Tests`. Optional CHR fixtures use local `OpenNest.Tests/test-config.json` and skip when absent. On Linux, build Windows projects with `-p:EnableWindowsTargeting=true`; this is not Windows runtime verification. The headless console builds independently with `dotnet build OpenNest.Console/OpenNest.Console.csproj`. +Only Plan Cutting changes need the [problem-nest corpus](docs/problem-nests.md) (`Category=ProblemNests`, opt-in via `OPENNEST_PROBLEM_NESTS_SERVER`); never run it for nesting engines, fills, UI, posts or file formats. Limit it to the areas the change touches with `OPENNEST_PROBLEM_NESTS_CHECKS` (table in that doc); `part-order` and `entries` plan every nest. No plate may rank below its baseline, and none may join or climb the near-budget list compared with master's run. Keep desktop-dependent tests in `OpenNest.WinForms.Tests`, never add a WinForms reference to `OpenNest.Tests`. Optional CHR fixtures use local `OpenNest.Tests/test-config.json` and skip when absent. On Linux, build Windows projects with `-p:EnableWindowsTargeting=true`; this is not Windows runtime verification. The headless console builds independently with `dotnet build OpenNest.Console/OpenNest.Console.csproj`. Releases: follow [the release procedure](docs/releasing.md) and `scripts/Publish-Windows.ps1`; workflow artifacts are candidates, not published releases. GitHub (`ajisaacs/OpenNest`) is the primary repository; Gitea is a read-only backup mirror. diff --git a/OpenNest.Tests/CuttingPlanning/ProblemNestRegressionTests.cs b/OpenNest.Tests/CuttingPlanning/ProblemNestRegressionTests.cs index 568b2791..2b4538d6 100644 --- a/OpenNest.Tests/CuttingPlanning/ProblemNestRegressionTests.cs +++ b/OpenNest.Tests/CuttingPlanning/ProblemNestRegressionTests.cs @@ -14,7 +14,10 @@ namespace OpenNest.Tests.CuttingPlanning; /// worse than its recorded baseline. A nest with a cut-off baseline is planned a second time with /// its applied lead-ins and cut-offs cleared and default scrap cut-offs added. A "checks:" comment /// line names the planner areas a nest guards; a run limited to some areas skips nests that guard -/// none of them, and a nest without the line is always planned. Opt-in and private: +/// none of them, and a nest without the line is always planned. A run touching the part order or +/// entries plans every nest, since those change the search on every plate. Each plate's share of +/// its expansion budget is reported, and plates near it are listed last: they are the ones a change +/// to the search order tips into . Opt-in and private: /// the nests and their baselines stay on the server, never in the repository. See docs/problem-nests.md. /// [Trait("Category", "ProblemNests")] @@ -31,6 +34,15 @@ public partial class ProblemNestRegressionTests(ITestOutputHelper output) /// The planner areas a nest can guard; docs/problem-nests.md says which changes touch each. internal static readonly string[] Areas = ["part-order", "entries", "contours", "geometry-checks", "cut-offs", "grid"]; + + /// + /// Areas whose changes reorder the search on every plate, not just the nests tagged with them: + /// a run naming one plans the whole corpus. + /// + internal static readonly string[] WholeCorpusAreas = ["part-order", "entries"]; + + /// A plate whose strict search spent at least this share of its budget is near it. + internal const double NearBudget = 0.8; // A corpus is a handful of nests; more means the variable names a production archive. internal const int MaximumNests = 200; @@ -44,9 +56,14 @@ public partial class ProblemNestRegressionTests(ITestOutputHelper output) var limit = TimeSpan.FromSeconds(int.TryParse(Environment.GetEnvironmentVariable(TimeoutVariable), out var seconds) && seconds > 0 ? seconds : 120); Assert.True(records.Count > 0, $"No nests on {server}."); - var wanted = Checks(Environment.GetEnvironmentVariable(ChecksVariable)); + var named = Checks(Environment.GetEnvironmentVariable(ChecksVariable)); + var wanted = Wanted(named); + if (wanted.Length != named.Length) + output.WriteLine($"{string.Join(", ", named.Intersect(WholeCorpusAreas))} change the search on every plate, " + + "so every nest is planned."); var failures = new List(); + var usage = new List<(string Plate, double Share)>(); foreach (var record in records) { var guards = Checks(CommentLine(record.Comments, ChecksLine)); @@ -60,20 +77,32 @@ public partial class ProblemNestRegressionTests(ITestOutputHelper output) var bytes = await repository.GetFileAsync(record.Id) ?? throw new InvalidOperationException($"'{record.Name}' has no file."); var baseline = Baseline(record.Comments); - var observed = PlanAll(record.Name, bytes, false, baseline, limit, failures); + var observed = PlanAll(record.Name, bytes, false, baseline, limit, failures, usage); if (baseline.Count == 0) failures.Add($"{record.Name} has no baseline. Add this line to its comments: {Line(PlanningLine, observed)}"); if (Environment.GetEnvironmentVariable(CutOffsVariable) == "1") - PlanAll(record.Name, bytes, true, [], limit, failures); + PlanAll(record.Name, bytes, true, [], limit, failures, usage); else if (Baseline(record.Comments, CutOffPlanningLine) is { Count: > 0 } withCutOffs) - PlanAll(record.Name, bytes, true, withCutOffs, limit, failures); + PlanAll(record.Name, bytes, true, withCutOffs, limit, failures, usage); } + ReportNearBudget(usage); Assert.True(failures.Count == 0, string.Join(Environment.NewLine, failures)); } + // Plates near their budget pass today but are the first a reordered search pushes over it. + private void ReportNearBudget(List<(string Plate, double Share)> usage) + { + var near = usage.Where(u => u.Share >= NearBudget).OrderByDescending(u => u.Share).ToList(); + output.WriteLine(near.Count == 0 + ? $"No plate's search used {NearBudget:P0} of its budget." + : $"Near the budget ({near.Count}; a change that adds plates here or raises their share is a regression):"); + foreach (var (plate, share) in near) + output.WriteLine($" {plate}: {share:P0}"); + } + // Plans each plate with parts and compares it with the baseline; plate number to outcome. private SortedDictionary PlanAll(string name, byte[] bytes, bool cutOffs, - Dictionary baseline, TimeSpan limit, List failures) + Dictionary baseline, TimeSpan limit, List failures, List<(string Plate, double Share)> usage) { using var stream = new MemoryStream(bytes); var nest = new NestReader(stream).Read(); @@ -88,11 +117,18 @@ public partial class ProblemNestRegressionTests(ITestOutputHelper output) var watch = Stopwatch.StartNew(); if (cutOffs) AddScrapCutOffs(plate, number, nest.Units); + var budget = CuttingPlanBatch.PlateBudget(plate.Parts.Count); var (outcome, planned) = Plan(plate, number, limit); observed[number] = outcome; var expected = baseline.GetValueOrDefault(number); + var share = BudgetShare(planned, budget); + if (share is { } used) + usage.Add(($"{label} plate {number}", used)); output.WriteLine($"{label} plate {number}: {outcome} (baseline {expected ?? "none"}), " - + $"{planned?.Result.Expansions ?? 0} expansions, {watch.Elapsed.TotalSeconds:0.0} s"); + + $"{planned?.Result.Expansions ?? 0} of {budget} expansions" + + (share is { } shown ? $" ({shown:P0})" : "") + + (planned?.KeptCurrentOrder == true ? ", new order ran out" : "") + + $", {watch.Elapsed.TotalSeconds:0.0} s"); if (expected == null) continue; if (Rank(outcome) < Rank(expected)) @@ -183,10 +219,23 @@ public partial class ProblemNestRegressionTests(ITestOutputHelper output) return areas; } + /// The areas a run plans: none, meaning every nest, when one changes the search on every plate. + internal static string[] Wanted(string[] named) => named.Intersect(WholeCorpusAreas).Any() ? [] : named; + // Every nest when no area is wanted; otherwise one guarding a wanted area, or one naming none. internal static bool Selected(IReadOnlyCollection guards, IReadOnlyCollection wanted) => wanted.Count == 0 || guards.Count == 0 || guards.Intersect(wanted).Any(); + /// + /// The share of the plate's budget its applied strict search spent; null when no strict search + /// found the route (a refusal, the safest-route fallback or the best-effort plan), since then + /// there is no margin left to measure. + /// + internal static double? BudgetShare(CuttingPlanPlateResult planned, int budget) => + planned?.Result is { IsSafestRouteFallback: false, Expansions: > 0 } result && budget > 0 + && result.Status is CuttingPlanStatus.Ready or CuttingPlanStatus.BestEffort + ? (double)result.Expansions / budget : null; + // Ready means replayed and overlap-clear; a ready route with overlapping parts is still blocked. internal static string Outcome(CuttingPlanPlateResult planned) => planned.IsReady ? "Ready" : planned.IsRouteReady ? "Overlap" : planned.Result.Status.ToString(); @@ -249,6 +298,14 @@ public class ProblemNestBaselineTests Assert.Throws(() => ProblemNestRegressionTests.Checks("part-order, lead-ins")); } + [Fact] + public void Wanted_PlansEveryNestForAreasThatReorderEveryPlate() + { + Assert.Equal(["cut-offs", "grid"], ProblemNestRegressionTests.Wanted(["cut-offs", "grid"])); + Assert.Empty(ProblemNestRegressionTests.Wanted(["cut-offs", "part-order"])); + Assert.Empty(ProblemNestRegressionTests.Wanted(["entries"])); + } + [Fact] public void Rank_OrdersReadyAboveBestEffortAboveBudgetAboveBlockedAboveTimeout() { diff --git a/docs/problem-nests.md b/docs/problem-nests.md index d17c7f34..5b3b3f2f 100644 --- a/docs/problem-nests.md +++ b/docs/problem-nests.md @@ -9,7 +9,9 @@ posts and file formats never need it. ## Areas Each nest names the areas it guards in a `checks:` comment line. Pick every area a change touches; -when a change spans the planner or you cannot tell, run the whole corpus. +when a change spans the planner or you cannot tell, run the whole corpus. `part-order` and +`entries` change the order the search tries things on every plate, so naming either plans every +nest: a plate tagged with another area can still run out of budget after such a change. | Area | Change touches | |------|----------------| @@ -52,6 +54,21 @@ baseline, when a nest has no baseline, or when the baseline's plate numbers diff nest's. A different outcome at the same or a better rank is only reported. When it is intended, update the baseline. +## Budget headroom + +The planner's budget counts search expansions, not time, so a plate's outcome is repeatable, but +a plate that finds its route just inside the budget is fragile: any change to the order the +search tries parts or entries can push it over, into `NoSolutionWithinBudget`, without touching +the problem it guards. Each plate's line reports the expansions its applied strict search spent +against the plate's budget (`12000 of 20000 expansions (60%)`), and whether the new order ran out +before the current order was kept. Plates without a strict-search route (refusals, the +safest-route fallback, best-effort plans) report no share. + +The run ends with the plates that spent at least 80% of their budget. They pass today and fail +next. Compare that list with the same run on master: a change that adds plates to it or raises +their share made the search worse, even when every plate still meets its baseline. Say so in the +change, and fix it, or explain why the plate's search grew. + A nest whose comments also carry a `planning with cut-offs: 1=Ready 2=BestEffort` line is planned a second time, as Plan Cutting does after **Clear Lead-ins & Cut-offs** and **Add scrap cut-offs** at their defaults, and each plate is held to that line the same way. Set