mirror of
https://github.com/ajisaacs/OpenNest.git
synced 2026-10-10 14:24:13 -04:00
ci(windows): retain fail-closed desktop acceptance evidence
This commit is contained in:
1 parent
bb43118451
commit
b591528aa2
5 files changed
+1391
No files matched your search
@@ -97,6 +97,11 @@ jobs:
|
||||
if ($LASTEXITCODE -ne 0) { $failed += $project }
|
||||
}
|
||||
if ($failed.Count -gt 0) { throw "Failed: $($failed -join ', ')" }
|
||||
- name: Test acceptance evidence checker
|
||||
run: python scripts/test_windows_acceptance.py -v
|
||||
- name: Verify named Windows acceptance coverage
|
||||
if: always()
|
||||
run: python scripts/windows_acceptance.py --results TestResults
|
||||
- name: Upload test results
|
||||
if: always()
|
||||
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# Automated Windows acceptance
|
||||
|
||||
The `windows-desktop` job in `.github/workflows/ci.yml` runs on GitHub-hosted
|
||||
`windows-2022` for every pull request and `master` push. It executes the full
|
||||
WinForms and FrontEnd test projects in Release, including shown-form STA tests
|
||||
and Windows child-process lifetime tests. No production service, customer
|
||||
fixture, physical GPU, or CNC controller is required.
|
||||
|
||||
Each test host has a five-minute hang watchdog with a mini dump; the job has a
|
||||
20-minute limit. A failure in either suite fails the step, without suppressing
|
||||
the other suite. This job is separate from the existing Linux `tests` aggregate;
|
||||
require `windows-desktop` as well when assessing acceptance.
|
||||
|
||||
## Results and coverage contract
|
||||
|
||||
Download the `windows-desktop-test-results` artifact from the workflow run.
|
||||
It contains both TRX files, any hang diagnostics, and:
|
||||
|
||||
- `windows-acceptance.json`: commit/tree, runner platform, workflow identifiers,
|
||||
TRX hashes, enumerated results, failures/skips and per-area test evidence.
|
||||
- `windows-acceptance.md`: a readable summary, also shown in the Actions job
|
||||
summary, with remaining acceptance beside the automated result.
|
||||
|
||||
Artifacts are retained for 14 days. Save evidence needed for longer-lived task
|
||||
verification before expiration. On pull requests the checked-out commit can be
|
||||
GitHub's test merge commit; use the report's commit/tree and workflow link, not
|
||||
just the branch name.
|
||||
|
||||
`scripts/windows-acceptance.json` is the committed coverage contract. It names
|
||||
required test methods and the minimum number of executed cases for each
|
||||
(including theory rows). The report fails if either TRX is absent/malformed,
|
||||
empty or inconsistent; if a required method/case is missing, failed or skipped;
|
||||
if any other test fails; or if a run was aborted. Skips outside the named contract
|
||||
are retained explicitly, never counted as passes. Added cases are allowed;
|
||||
removing or renaming required coverage requires an intentional manifest change.
|
||||
The whole suites still run, not just this manifest's subset.
|
||||
|
||||
The areas are BOM import, Nest Info material preservation, saved-nest browsing,
|
||||
nesting/commit policy, PDF export, the cutting dialog, automatic cutoffs,
|
||||
overlap/posting consent, UI lifetime, import/export failures, and MCP processes.
|
||||
The manifest is public test coverage only. Private tracker IDs, plans, customer
|
||||
data, and acceptance notes stay out of the repository and CI artifacts.
|
||||
|
||||
The report is evidence, not automatic task closure. A passing area does not
|
||||
establish absent implementation, real-service/two-PC behavior, DPI or physical
|
||||
printing quality, packaged-build behavior, or machine safety. In particular,
|
||||
fake-repository browser tests do not verify the HTTP/SQLite save/paging lifecycle.
|
||||
Keep those remaining gates in the tracker; link exact test names and the workflow
|
||||
run when updating an individual task.
|
||||
|
||||
## Repeat locally on Windows
|
||||
|
||||
Use a clean, committed checkout, .NET 8 and Python 3. Run from the repository root
|
||||
in PowerShell. Always use a new results directory so stale TRX files cannot stand
|
||||
in for a failed build. Do not reuse downloaded CI results as a new local run.
|
||||
|
||||
```powershell
|
||||
$results = Join-Path 'TestResults' ([guid]::NewGuid().ToString())
|
||||
$failed = @()
|
||||
foreach ($project in @('OpenNest.WinForms.Tests', 'OpenNest.FrontEnd.Tests')) {
|
||||
dotnet test "$project/$project.csproj" -c Release --blame-hang-timeout 5m --blame-hang-dump-type mini --logger "trx;LogFileName=$project.trx" --results-directory $results
|
||||
if ($LASTEXITCODE -ne 0) { $failed += $project }
|
||||
}
|
||||
python scripts/windows_acceptance.py --results $results
|
||||
if ($LASTEXITCODE -ne 0 -or $failed.Count -gt 0) { throw 'Windows acceptance failed; inspect results.' }
|
||||
```
|
||||
|
||||
The CLI refuses Linux/macOS as Windows runtime evidence and refuses tracked
|
||||
source edits. The source revision is provenance of the checkout running the
|
||||
command, not a cryptographic attestation of arbitrary supplied TRX files; the
|
||||
fresh-checkout CI run and the fresh-directory local procedure establish that
|
||||
relationship. The JSON retains hashes so the saved files can be matched later.
|
||||
|
||||
## Maintain the gate
|
||||
|
||||
Run `python scripts/test_windows_acceptance.py -v` on any platform after changing
|
||||
the parser or manifest. Its generated TRX fixtures test the reporter only, not
|
||||
the application. Exercise a draft PR on Windows before integrating new form
|
||||
cases. Inspect actual TRX failures and skip reasons; a Linux cross-build is not
|
||||
a Windows pass. Keep visual/operator limitations alongside the named cases.
|
||||
@@ -0,0 +1,146 @@
|
||||
"""Synthetic parser controls, not Windows application acceptance evidence."""
|
||||
|
||||
import copy
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
import windows_acceptance as acceptance
|
||||
|
||||
|
||||
class WindowsAcceptanceTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.path = Path(self.temp.name)
|
||||
self.manifest = {
|
||||
"schema_version": 1,
|
||||
"projects": ["Example.Tests"],
|
||||
"groups": [{
|
||||
"id": "example", "title": "Example",
|
||||
"required_tests": [{"project": "Example.Tests", "method": "Example.Tests.Form.Accept", "minimum_cases": 2}],
|
||||
"remaining_acceptance": ["Visual/operator check"],
|
||||
}],
|
||||
}
|
||||
self.rows = [("Example.Tests.Form.Accept", "Passed"), ("Example.Tests.Form.Accept", "Passed")]
|
||||
|
||||
def write_trx(self, rows=None, total=None, summary="Completed"):
|
||||
rows = self.rows if rows is None else rows
|
||||
root = ET.Element("TestRun", xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010")
|
||||
definitions = ET.SubElement(root, "TestDefinitions")
|
||||
results = ET.SubElement(root, "Results")
|
||||
for index, (method, outcome) in enumerate(rows):
|
||||
test = ET.SubElement(definitions, "UnitTest", id=str(index))
|
||||
class_name, name = method.rsplit(".", 1)
|
||||
ET.SubElement(test, "TestMethod", className=class_name, name=name)
|
||||
result = ET.SubElement(results, "UnitTestResult", testId=str(index), executionId=str(index),
|
||||
testName=f"{method}(case: {index})", outcome=outcome)
|
||||
if outcome != "Passed":
|
||||
ET.SubElement(ET.SubElement(result, "Output"), "StdOut").text = "fixture reason"
|
||||
result_summary = ET.SubElement(root, "ResultSummary", outcome=summary)
|
||||
ET.SubElement(result_summary, "Counters", total=str(len(rows) if total is None else total),
|
||||
passed=str(sum(r[1] == "Passed" for r in rows)),
|
||||
failed=str(sum(r[1] == "Failed" for r in rows)), notExecuted="0")
|
||||
path = self.path / "Example.Tests.trx"
|
||||
ET.ElementTree(root).write(path, encoding="utf-8", xml_declaration=True)
|
||||
return path
|
||||
|
||||
def evaluate(self):
|
||||
return acceptance.evaluate(self.manifest, self.path)
|
||||
|
||||
def test_complete_theory_passes_and_retains_manual_gate(self):
|
||||
self.write_trx()
|
||||
report = self.evaluate()
|
||||
self.assertEqual("passed", report["automated_status"])
|
||||
self.assertEqual(2, report["groups"][0]["result_count"])
|
||||
self.assertEqual(["Visual/operator check"], report["groups"][0]["remaining_acceptance"])
|
||||
self.assertEqual(2, report["suites"]["Example.Tests"]["total"])
|
||||
|
||||
def test_missing_or_malformed_file_fails(self):
|
||||
self.assertEqual("failed", self.evaluate()["automated_status"])
|
||||
(self.path / "Example.Tests.trx").write_text("<broken", encoding="utf-8")
|
||||
self.assertEqual("failed", self.evaluate()["automated_status"])
|
||||
|
||||
def test_missing_method_or_theory_row_fails(self):
|
||||
for rows in ([self.rows[0]], [("Example.Tests.Other.Accept", "Passed")] * 2):
|
||||
with self.subTest(rows=rows):
|
||||
self.write_trx(rows)
|
||||
self.assertEqual("failed", self.evaluate()["automated_status"])
|
||||
|
||||
def test_skip_failed_aborted_and_unknown_required_results_fail(self):
|
||||
for outcome in ("NotExecuted", "Failed", "Aborted", "TimedOut", "Unknown"):
|
||||
with self.subTest(outcome=outcome):
|
||||
self.write_trx([self.rows[0], (self.rows[1][0], outcome)])
|
||||
report = self.evaluate()
|
||||
self.assertEqual("failed", report["automated_status"])
|
||||
self.assertEqual("fixture reason", report["groups"][0]["tests"][1]["detail"])
|
||||
|
||||
def test_unmapped_failed_test_still_fails_gate(self):
|
||||
self.write_trx(self.rows + [("Example.Tests.Other.Failure", "Failed")])
|
||||
self.assertEqual("failed", self.evaluate()["automated_status"])
|
||||
|
||||
def test_unmapped_skip_is_explicit_not_a_pass(self):
|
||||
self.write_trx(self.rows + [("Example.Tests.Other.Optional", "NotExecuted")])
|
||||
report = self.evaluate()
|
||||
self.assertEqual("passed", report["automated_status"])
|
||||
suite = report["suites"]["Example.Tests"]
|
||||
self.assertEqual({"NotExecuted": 1, "Passed": 2}, suite["outcomes"])
|
||||
self.assertEqual("fixture reason", suite["results"][2]["detail"])
|
||||
|
||||
def test_empty_or_inconsistent_counters_fail(self):
|
||||
for rows, total in (([], 0), (self.rows, 3)):
|
||||
self.write_trx(rows, total)
|
||||
self.assertEqual("failed", self.evaluate()["automated_status"])
|
||||
path = self.write_trx()
|
||||
path.write_text(path.read_text().replace('passed="2"', 'passed="1"'))
|
||||
self.assertEqual("failed", self.evaluate()["automated_status"])
|
||||
|
||||
def test_aborted_run_fails_even_with_passed_rows(self):
|
||||
self.write_trx(summary="Aborted")
|
||||
self.assertEqual("failed", self.evaluate()["automated_status"])
|
||||
|
||||
def test_duplicate_or_unmapped_execution_fails(self):
|
||||
for old, new in (('executionId="1"', 'executionId="0"'),
|
||||
('testId="1"', 'testId="not-defined"'), ('id="1"', 'id="0"')):
|
||||
path = self.write_trx()
|
||||
path.write_text(path.read_text().replace(old, new))
|
||||
self.assertEqual("failed", self.evaluate()["automated_status"])
|
||||
|
||||
def test_invalid_manifests_are_rejected(self):
|
||||
original = copy.deepcopy(self.manifest)
|
||||
for change in (
|
||||
lambda m: m.update(projects=[]),
|
||||
lambda m: m.update(groups=[]),
|
||||
lambda m: m["groups"].append(copy.deepcopy(m["groups"][0])),
|
||||
lambda m: m["groups"][0].update(required_tests=[]),
|
||||
lambda m: m["groups"][0]["required_tests"][0].update(minimum_cases=0),
|
||||
lambda m: m["groups"][0]["required_tests"][0].update(project="Unknown.Tests"),
|
||||
):
|
||||
self.manifest = copy.deepcopy(original)
|
||||
change(self.manifest)
|
||||
with self.assertRaises(ValueError):
|
||||
self.evaluate()
|
||||
|
||||
def test_cli_refuses_linux_runtime_and_dirty_source(self):
|
||||
self.write_trx()
|
||||
manifest = self.path / "manifest.json"
|
||||
manifest.write_text(json.dumps(self.manifest), encoding="utf-8")
|
||||
args = ["--results", str(self.path), "--manifest", str(manifest)]
|
||||
for system, dirty in (("Linux", ""), ("Windows", " M source.cs")):
|
||||
with self.subTest(system=system, dirty=dirty), \
|
||||
patch.object(acceptance.platform, "system", return_value=system), \
|
||||
patch.object(acceptance.platform, "platform", return_value="Synthetic parser fixture"), \
|
||||
patch.object(acceptance.subprocess, "check_output", side_effect=["a" * 40, "b" * 40, dirty]), \
|
||||
patch.dict(acceptance.os.environ, {"GITHUB_STEP_SUMMARY": ""}), \
|
||||
patch("builtins.print"):
|
||||
self.assertEqual(1, acceptance.main(args))
|
||||
report = json.loads((self.path / "windows-acceptance.json").read_text())
|
||||
self.assertEqual("failed", report["automated_status"])
|
||||
self.assertTrue(report["errors"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,986 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"projects": [
|
||||
"OpenNest.FrontEnd.Tests",
|
||||
"OpenNest.WinForms.Tests"
|
||||
],
|
||||
"groups": [
|
||||
{
|
||||
"id": "bom-import",
|
||||
"title": "BOM import and editable row identity",
|
||||
"required_tests": [
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.BomImportFormTests.BlankingMaterialOrThickness_MarksTheRowAndDropsItFromTheGroups",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.BomImportFormTests.EditingQty_ChangesThatPartAndTheGroupTotal",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.BomImportFormTests.Edits_ChangeThePartShownInThatRow_AfterTheRowsAreReloadedInAnotherOrder",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.BomImportFormTests.InvalidQty_IsRefusedAndTheOldValueKept",
|
||||
"minimum_cases": 5
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.BomImportFormTests.PartColumns_KeepBomOrderAndOnlyMaterialThicknessAndQtyAreEditable",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.BomImportFormTests.ProgrammaticCommitOfInvalidText_KeepsTheRowsValue",
|
||||
"minimum_cases": 4
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.BomImportFormTests.RowsWithoutADrawing_CannotEnterEditMode",
|
||||
"minimum_cases": 1
|
||||
}
|
||||
],
|
||||
"remaining_acceptance": [
|
||||
"Visual appearance of disabled rows; additional material-dropdown behavior is not covered."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "nest-material",
|
||||
"title": "Nest Info material preservation",
|
||||
"required_tests": [
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.EditNestInfoFormTests.SaveWithChangedMaterial_GetsNameOnly",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.EditNestInfoFormTests.SaveWithUnchangedMaterial_KeepsGradeAndDensity",
|
||||
"minimum_cases": 1
|
||||
}
|
||||
],
|
||||
"remaining_acceptance": [
|
||||
"Shared-material selection requires its separate implementation."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "database-browser",
|
||||
"title": "Saved-nest browser",
|
||||
"required_tests": [
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.SavedNestsFormTests.HighlightedNestShowsItsPlatesAndDrawings_AndEnterOpensIt",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.SavedNestsFormTests.MissingArchive_ShowsTheErrorInTheDetailsLine_AndEscapeCancels",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.SavedNestsFormTests.OpensAsALargeNormalWindowInsideTheOwnersScreen",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.SavedNestsFormTests.PlatePreview_StepsThroughThePlates_AndFollowsThePlatesTable",
|
||||
"minimum_cases": 1
|
||||
}
|
||||
],
|
||||
"remaining_acceptance": [
|
||||
"Real HTTP/SQLite save/update/copy, server switching, outage/recovery, paging/sort/search/debounce, empty-last-page deletion, and two-PC checks are not established by the fake-repository form tests.",
|
||||
"Multi-monitor and DPI appearance need operator acceptance."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "nesting-pipeline",
|
||||
"title": "Nesting controls and front-end commit policy",
|
||||
"required_tests": [
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.DefaultEngineSelectionTests.ConsoleAutonestWithoutAnEngineUsesDefault",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.DefaultEngineSelectionTests.McpAutonestEngineDescriptionNamesEveryBuiltInEngine",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.DefaultEngineSelectionTests.McpAutonestWithoutAnEngineUsesDefault",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.DefaultEngineSelectionTests.McpFillToolsWithoutAnEngineUseFill",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.NestingPipelineTests.Api_PluginSelectionDerivesCountsPreservesIdsAndReportsValidation",
|
||||
"minimum_cases": 5
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.NestingPipelineTests.Api_UnknownEngineAndCancellationPropagate",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.NestingPipelineTests.Console_CancelledOrdinaryFillDoesNotOverwriteOutput",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.NestingPipelineTests.Console_CancelledSolveDoesNotSave",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.NestingPipelineTests.Console_EmptyResultDoesNotOverwriteOutputOrOriginal",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.NestingPipelineTests.Console_KeepPartsOnOccupiedSheetRejectsBeforeSolve",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.NestingPipelineTests.Console_RefusesInvalidWithoutOverwritingUnlessFaithfullyKeepable",
|
||||
"minimum_cases": 5
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.NestingPipelineTests.Console_ValidCommitPreservesSettingsAndUsesOnePhysicalSheet",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.NestingPipelineTests.Mcp_CancellationDoesNotCommit",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.NestingPipelineTests.Mcp_OccupiedSheetRejectsBeforeSolve",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.NestingPipelineTests.Mcp_OnlyCommitsFaithfulSingleSheet",
|
||||
"minimum_cases": 6
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.NestingPipelineTests.Mcp_SessionGateSerializesCallsBeforeCheckingOccupiedTarget",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.NestingPipelineTests.UnknownEngineDoesNotWriteOrCommit",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.ActionSelectAreaCutOffTests.SelectedAreaAndFillStayOnSelectedSideOfCutOff",
|
||||
"minimum_cases": 16
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutoNestFormStockOptionsTests.BlankAndCancelledRowsDoNotBecomeStockOptions",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutoNestFormStockOptionsTests.FractionalStockSizesRoundTripUnderCommaDecimalCulture",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutoNestFormStockOptionsTests.InvalidPendingEditCannotPublishOldBoundValue",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutoNestFormStockOptionsTests.NonblankInvalidRowRejectsWholeCollection",
|
||||
"minimum_cases": 5
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutoNestFormStockOptionsTests.StockOptionsKeepAnAddRowAfterAddingAPlate",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutoNestFormStockOptionsTests.StockSizeRegexHasExplicitTimeout",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutoNestFormStockOptionsTests.TimeoutAfterValidRowPublishesNoPrefixAndDoesNotAcceptDialog",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.NestValidationFormTests.ClosingWholeJobProgressWaitsForWorkerAndRequestsCancellation",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.NestValidationFormTests.CompletionClosesWithoutCancellingSuccessfulJob",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.NestValidationFormTests.DiscardIsDefaultAndMalformedOutputCannotBeKept",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateViewFillLifetimeTests.ClosedProgressWindowStaysBusyThroughCommitAndCleanup",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateViewFillLifetimeTests.QueuedProgressCannotOverwriteNewPreviewEvenDuringAnotherFill",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateViewFillLifetimeTests.SetupFailureReleasesGuardAndClosesProgressWindow",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateViewFillLifetimeTests.WorkerFailureReleasesGuardAndDiscardsPreview",
|
||||
"minimum_cases": 1
|
||||
}
|
||||
],
|
||||
"remaining_acceptance": [
|
||||
"Interactive engine preference/fallback and every group/area-fill path still need their specific operator checks."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "pdf-report",
|
||||
"title": "Desktop PDF export",
|
||||
"required_tests": [
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.NestReportExportTests.BusyAfterDialogRejectsWriteWithoutTouchingDestination",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.NestReportExportTests.BusyPlateActionOnAnyViewOfTheNestRejects",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.NestReportExportTests.CancelWritesNothingAndNotifiesNothing",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.NestReportExportTests.InteractiveFillOnAnyViewOfTheNestRejects",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.NestReportExportTests.InvalidNestNameRejectsBeforeAnyDialog",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.NestReportExportTests.JobBusyRejectsBeforeAnyDialog",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.NestReportExportTests.MenuFollowsActiveDocument",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.NestReportExportTests.RenderFailureLeavesExistingDestinationUnchanged",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.NestReportExportTests.SuccessfulExportWritesPdfAndLeavesJobStateUntouched",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.NestReportExportTests.WriteRejectsAStaleTargetFromAnotherDocument",
|
||||
"minimum_cases": 1
|
||||
}
|
||||
],
|
||||
"remaining_acceptance": [
|
||||
"Packaged-build fonts/notices, physical print output, and report visual readability are not verified by these development-build tests."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "cutting-planner",
|
||||
"title": "Current/all-plate cutting dialog",
|
||||
"required_tests": [
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Controls.PlateViewReorderTests.DisposedViewIgnoresItsRetainedPlate",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Controls.PlateViewReorderTests.FollowsAnInPlaceReorderOfItsCurrentPlateOnly",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Controls.PlateViewReorderTests.KeepsOneLayoutPerOccurrenceOfARepeatedPart",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.CuttingPlanFormTests.AllPlates_SkipsEmptyPlatesAndRefusesTheBatchWhenOnePlateIsBlocked",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.CuttingPlanFormTests.ApplyAfterALiveEdit_ChangesNothingAndReplanRecovers",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.CuttingPlanFormTests.BlockedPlate_KeepsApplyDisabledAndShowsTheCurrentParts",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.CuttingPlanFormTests.BusyEditor_DoesNotStartPlanningUntilItIsFree",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.CuttingPlanFormTests.ClosingWhilePlanning_CancelsAndWaitsForTheWorker",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.CuttingPlanFormTests.Constructor_PlansWithItsOwnCopyOfTheSettings",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.CuttingPlanFormTests.PlanThenApply_InstallsThePlanAndRedrawsTheEditorInPlateOrder",
|
||||
"minimum_cases": 1
|
||||
}
|
||||
],
|
||||
"remaining_acceptance": [
|
||||
"Selected-part side-panel migration and legacy automatic-path retirement are separate unfinished work.",
|
||||
"Actual job and NC/controller safety are not certified."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "automatic-cutoffs",
|
||||
"title": "Automatic cutoff dialogs",
|
||||
"required_tests": [
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutomaticCutOffFormTests.ApplyRecomputesWithCurrentPartsAndSettingsAndRegeneratesOnce",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutomaticCutOffFormTests.ApplyRefusesEmptyPlate",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutomaticCutOffFormTests.ApplyRefusesNewBlockingConflictSincePreview",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutomaticCutOffFormTests.CloseCallbackClearsPreviewWithoutApplying",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutomaticCutOffFormTests.FailedRegenerationRestoresDefinitionsProgramsAndPartSequence",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutomaticCutOffFormTests.InitialSpacingUsesNestUnitsAndActualSheetWidth",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutomaticCutOffFormTests.InvalidMinimumTailBlocksApply",
|
||||
"minimum_cases": 3
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutomaticCutOffFormTests.MinimumTailChangeClearsPreviewAndApplyUsesCurrentValue",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutomaticCutOffFormTests.NestApply_LaterConflictBlocksEntireNest",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutomaticCutOffFormTests.NestApply_PreviewsWithoutMutationAndAppliesEveryPlate",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutomaticCutOffFormTests.NestApply_RechecksOperationGuardAfterPreview",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutomaticCutOffFormTests.NestPreview_DisposeLeavesEveryPlateUntouched",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutomaticCutOffFormTests.PreviewIsDetachedAndDisposeClearsIt",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.AutomaticCutOffFormTests.SpacingChangeAndInvalidPreviewClearOldOverlay",
|
||||
"minimum_cases": 1
|
||||
}
|
||||
],
|
||||
"remaining_acceptance": [
|
||||
"Full saved-job cutting-state persistence and operator corner-lead checks remain separate."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "overlap-posting",
|
||||
"title": "Overlap display and posting consent",
|
||||
"required_tests": [
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.LeadInSidePanelTests.PanelClosesWithoutEscapeReopeningIt",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PostVerificationFormTests.ClearReportIsDisplayedAndCancelIsDefault",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PostVerificationFormTests.ClosingDuringVerificationWaitsForWorkerAndDiscardsLateResult",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PostVerificationFormTests.DisposalIsIdempotent",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PostVerificationFormTests.RealModalCancellationKeepsOwnerDisabledUntilWorkerStops",
|
||||
"minimum_cases": 3
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PostVerificationFormTests.UnexpectedFailureCannotBeOverridden",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PostVerificationFormTests.WarningsRequireFreshAcknowledgmentAndUncheckingRevokesPermission",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.AppendingPartRetainsTheExistingPairsPathButInvalidatesTheFullReport",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.AutoCheckKeepsDisplayOffButManualCheckStillShowsAreasAndReportsToStatusBar",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.AutoCheckRunsOnceTheLayoutSettlesAndRechecksAfterADrag",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.AutoCheckSupersedesARunningCheckWhenTheLayoutChanges",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.AutoCheckWaitsForInteractionsAndRespectsCancel",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.BothRetainsAreaPathAndCoincidentHoverListsEveryPairWithoutChangingSelection",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.CentroidMarkerUsesGraphCoordinatesAndFixedScreenSizeAcrossZoom",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.CheckCommandCapturesUnitsAndNamesUntilTheNextCheck",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.CleanupReleasesDequeuedWorkersEvenWithoutCompleteOrFail",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.CompletedPathIsReleasedOnHandleRecreationAndCannotReappear",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.CrowdedHoverPagesStayInViewportAndReachLastPairWithoutSelectionOrWheelCapture",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.DefaultAnalyzerRechecksIncrementallyAndEditorsDropTheCache",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.DisplayPanZoomAndPreviewDoNotAnalyzeOrChangeCommittedParts",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.EditNestFormEnablesAutoCheck",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.EditOrPlateSwitchRejectsPendingReport",
|
||||
"minimum_cases": 7
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.FailureAndIncompleteAreVisibleAndNeverReportedAsClear",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.HandleDestructionOrDisposalNeverResurrectsRequests",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.HoverClearsBeforeStaleDetailsCanBeShown",
|
||||
"minimum_cases": 11
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.LongNameAndTinyViewNeverSilentlyElideTheOnlyPair",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.MarkerAndHoverUsePairMaterialCentroidRatherThanBoundsCenter",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.MenusFollowMdiActivationDocumentDisplayAndRunningState",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.MovingDirectlyToAnotherCoincidentGroupStartsItsFirstPage",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.PagingResetsWithHoverLifetimeAndInactiveKeysAreNotConsumed",
|
||||
"minimum_cases": 9
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.PaintChecksFreshnessEvenWhenDisplayIsOff",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.PairDetailsOnlySuppressNormalTooltipWhileVisible",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.RecheckPreservesEachVisibleModeAndDisplayOnlyRepaints",
|
||||
"minimum_cases": 3
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.RestartAndCancelRejectLateCompletionsOnTheUiThread",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.RetainedPathClearsOnHardInvalidationOrRequestFailure",
|
||||
"minimum_cases": 3
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.SlotChangingCollectionEventsImmediatelyDiscardCurrentReportAndPath",
|
||||
"minimum_cases": 3
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.SnapshotKeepsOriginalLabelsWhenLiveNamesChange",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.TripleOverlapUsesOneFillAndGraphCoordinatesDoNotDoublePan",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.PlateOverlapOverlayTests.UnchangedPairStaysVisibleDuringMotionAndPendingRecheck",
|
||||
"minimum_cases": 1
|
||||
}
|
||||
],
|
||||
"remaining_acceptance": [
|
||||
"DPI/visual appearance, full lead/tab/lock save-reopen-edit-post lifecycle, and actual machine qualification remain separate."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "ui-lifetime",
|
||||
"title": "UI lifetime, focus and disposal",
|
||||
"required_tests": [
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Controls.SeparatorPenLifetimeTests.DisposeReleasesBothPensAndDisposesTheControlOnlyOnce",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Controls.SeparatorPenLifetimeTests.PaintingKeepsOrientationColorsStylesAndResizeBehavior",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.BestFitViewerOperationLifetimeTests.CloseOrDisposalRejectsLateResultsAndErrorsWithoutTouchingControls",
|
||||
"minimum_cases": 4
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.BestFitViewerOperationLifetimeTests.CurrentComputeOrRenderFailureReportsOnceAndClearsPartialAndPreviousResults",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.BestFitViewerOperationLifetimeTests.DispatcherDrainsPendingCallbacksWhenCompletionAlreadyHolds",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.BestFitViewerOperationLifetimeTests.DispatcherObservesAsyncVoidFailureAfterOperationCompletes",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.BestFitViewerOperationLifetimeTests.OldCompletionCannotPublishReportOrResetNewerOperation",
|
||||
"minimum_cases": 4
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.BestFitViewerOperationLifetimeTests.OnlyRequestedCancellationIsExpectedAndNonmodal",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.BestFitViewerOperationLifetimeTests.ShownAndSelectionPublishCompleteResultsWithPagingAndSourceSelection",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.EditNestFocusTraversalTests.NestedActiveContainersReturnTheirLeaf",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.EditNestFocusTraversalTests.NestedContainerWithoutActiveControlIsReturned",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.EditNestFocusTraversalTests.NoActiveControlReturnsTheForm",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.EditNestFocusTraversalTests.OrdinaryActiveControlIsReturnedUnchanged",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.EditNestFocusTraversalTests.PlateListRefreshKeepsFocusOnThePlateList",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.EditNestFocusTraversalTests.PlateListRefreshRestoresTheOriginalNestedLeaf",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.StaTestThreadTests.QueuedCallbackExceptionFailsTheTestInsteadOfOpeningADialog",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.StaTestThreadTests.WithoutThrowExceptionModeTheCallbackExceptionNeverReachesTheTestBody",
|
||||
"minimum_cases": 1
|
||||
}
|
||||
],
|
||||
"remaining_acceptance": []
|
||||
},
|
||||
{
|
||||
"id": "import-export",
|
||||
"title": "Import/export failures and highlighting",
|
||||
"required_tests": [
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.RegexImportFailureTests.Api_TimeoutRetainsInnerExceptionAndNeverSolvesOrReturnsResponse",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.RegexImportFailureTests.Console_TimeoutExitsOneWithoutOutputEvenAfterEarlierImport",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.RegexImportFailureTests.Mcp_TimeoutReturnsErrorWithoutAddingDrawingOrChangingEarlierDrawing",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.ArchUnitsTests.ArchitecturalRegexHasExplicitTimeout",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.ArchUnitsTests.InvalidFieldIsRedAndNaNAndCorrectionRestoresIt",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.ArchUnitsTests.OrdinaryArchitecturalInputsKeepTheirMeaning",
|
||||
"minimum_cases": 5
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Controls.ProgramEditorControlTests.OrdinaryHighlightingPreservesAllColorsAndUnmatchedCoordinateText",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Controls.ProgramEditorControlTests.ReverseAndReorderFinishModelPreviewAndEventsWithOrWithoutTimeout",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Controls.ProgramEditorControlTests.SuccessfulRetryAndClearRemoveTheNonmodalTimeoutIndication",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Controls.ProgramEditorControlTests.TimeoutDuringInitialLoadKeepsCompleteTextAndBaseColorWithoutAParentHandle",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Controls.ProgramEditorControlTests.UnexpectedApplicationFailureStillRestoresSelectionAndResumesLayout",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Controls.ProgramEditorControlTests.UnexpectedMatchingFailureIsNotSwallowed",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.CadConverterImportFailureTests.ExtractionTimeoutPublishesNeitherRawTextNorPartialFile",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.CadConverterImportFailureTests.MTextRegexHasExplicitTimeout",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.CadConverterImportFailureTests.MTextStrippingKeepsItsExistingRules",
|
||||
"minimum_cases": 4
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.CadConverterImportFailureTests.SuccessfulExtractionPreservesTextAndControlCodeSemantics",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateExportTests.CancelWritesNothingAndNotifiesNothing",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateExportTests.CoordinateExportLeavesDocumentAndJobStateUntouched",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateExportTests.CoordinatesPreserveFormatAndReleaseFile",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateExportTests.ExportAllStopsAtFirstWriteFailure",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateExportTests.ExportAllStopsWhenSecondDialogIsCanceled",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateExportTests.ExportAllWritesEveryPlateOnceAndAdvancesToLast",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateExportTests.FailureAfterPartialWriteIsNotSuccess",
|
||||
"minimum_cases": 3
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateExportTests.FlushFailureDuringWriterDisposalIsNotSuccess",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateExportTests.JpegRendersShownPlateAndReleasesFile",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateExportTests.MissingParentDirectoryReportsOneFailure",
|
||||
"minimum_cases": 3
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.PlateExportTests.UnsupportedFilterReportsFailureWithoutCreatingFile",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.ShapeLibraryFormTests.FriendlyNameRegexHasExplicitTimeout",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.ShapeLibraryFormTests.FriendlyNameTimeoutDiscardsPartialFormatting",
|
||||
"minimum_cases": 2
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.ShapeLibraryFormTests.FriendlyNamesKeepSuffixAndWordBoundaryBehavior",
|
||||
"minimum_cases": 4
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.WinForms.Tests",
|
||||
"method": "OpenNest.WinForms.Tests.Forms.ShapeLibraryFormTests.InvalidShapeInputClearsOldPreviewAndBlocksAddUntilCorrected",
|
||||
"minimum_cases": 1
|
||||
}
|
||||
],
|
||||
"remaining_acceptance": [
|
||||
"Atomic replacement of existing plate-export destinations is a separate hardening task, not a guarantee of truthful failure reporting."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "mcp-process",
|
||||
"title": "Windows child-process lifetime",
|
||||
"required_tests": [
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.McpEngineHarnessProcessTests.AlreadyCancelled_DoesNotStartChild",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.McpEngineHarnessProcessTests.Arguments_PreserveSpacesAndQuotes",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.McpEngineHarnessProcessTests.Cancellation_StopsOwnedLiveTreeWithoutKillingUnrelatedProcess",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.McpEngineHarnessProcessTests.Completed_PreservesStderrAndNonzeroStatus",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.McpEngineHarnessProcessTests.ConcurrentDrains_PreserveFullOutputBeyondPipeCapacity",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.McpEngineHarnessProcessTests.Deadline_BoundsHungChildAndReapsIt",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.McpEngineHarnessProcessTests.Deadline_BoundsPipeDrainAfterParentExits",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.McpEngineHarnessProcessTests.InvalidConfiguration_FailsClearlyWithoutStartingChild",
|
||||
"minimum_cases": 5
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.McpEngineHarnessProcessTests.MissingNest_FailsClearly",
|
||||
"minimum_cases": 1
|
||||
},
|
||||
{
|
||||
"project": "OpenNest.FrontEnd.Tests",
|
||||
"method": "OpenNest.FrontEnd.Tests.McpEngineHarnessProcessTests.Normal_ReachesPrivateChild",
|
||||
"minimum_cases": 1
|
||||
}
|
||||
],
|
||||
"remaining_acceptance": []
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,174 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed evidence for named Windows tests, not automatic feature sign-off."""
|
||||
|
||||
import argparse
|
||||
from collections import Counter
|
||||
from datetime import datetime, timezone
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import subprocess
|
||||
import sys
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
|
||||
def read_suite(path):
|
||||
"""Read actual result rows; TRX notExecuted counters can omit xUnit skips."""
|
||||
root = ET.parse(path).getroot()
|
||||
if root.tag != "{http://microsoft.com/schemas/VisualStudio/TeamTest/2010}TestRun":
|
||||
raise ValueError(f"{path.name}: not a TRX TestRun")
|
||||
definitions = {}
|
||||
for test in root.findall("./{*}TestDefinitions/{*}UnitTest"):
|
||||
method = test.find("{*}TestMethod")
|
||||
identity = test.attrib["id"]
|
||||
if identity in definitions or method is None:
|
||||
raise ValueError(f"{path.name}: duplicate or incomplete test definition")
|
||||
definitions[identity] = method.attrib["className"] + "." + method.attrib["name"]
|
||||
results = []
|
||||
executions = set()
|
||||
for result in root.findall("./{*}Results/{*}UnitTestResult"):
|
||||
execution = result.attrib["executionId"]
|
||||
if execution in executions:
|
||||
raise ValueError(f"{path.name}: duplicate test execution")
|
||||
executions.add(execution)
|
||||
output = result.find("{*}Output")
|
||||
results.append({
|
||||
"method": definitions[result.attrib["testId"]],
|
||||
"name": result.attrib["testName"],
|
||||
"outcome": result.attrib["outcome"],
|
||||
"detail": "\n".join(output.itertext()).strip() if output is not None else "",
|
||||
})
|
||||
summary = root.find("{*}ResultSummary")
|
||||
if summary is None or summary.attrib.get("outcome") not in ("Completed", "Passed"):
|
||||
raise ValueError(f"{path.name}: missing or unsuccessful run summary")
|
||||
counters = summary.find("{*}Counters")
|
||||
if counters is None or not results or int(counters.attrib["total"]) != len(results):
|
||||
raise ValueError(f"{path.name}: empty results or total does not match enumerated results")
|
||||
outcomes = Counter(r["outcome"] for r in results)
|
||||
for key, outcome in (("passed", "Passed"), ("failed", "Failed")):
|
||||
if int(counters.attrib[key]) != outcomes[outcome]:
|
||||
raise ValueError(f"{path.name}: {key} counter does not match result rows")
|
||||
return {
|
||||
"file": path.name,
|
||||
"sha256": hashlib.sha256(path.read_bytes()).hexdigest(),
|
||||
"total": len(results),
|
||||
"outcomes": dict(sorted(outcomes.items())),
|
||||
"results": results,
|
||||
}
|
||||
|
||||
|
||||
def evaluate(manifest, results_dir):
|
||||
if manifest.get("schema_version") != 1:
|
||||
raise ValueError("Unsupported acceptance manifest schema")
|
||||
projects = manifest["projects"]
|
||||
groups = manifest["groups"]
|
||||
if not projects or len(projects) != len(set(projects)) or not groups:
|
||||
raise ValueError("Manifest must contain unique projects and nonempty groups")
|
||||
if any(Path(p).name != p or not p.endswith(".Tests") for p in projects):
|
||||
raise ValueError("Invalid test project name")
|
||||
ids = [g["id"] for g in groups]
|
||||
if len(ids) != len(set(ids)):
|
||||
raise ValueError("Duplicate acceptance group")
|
||||
suites, errors = {}, []
|
||||
for project in projects:
|
||||
try:
|
||||
suites[project] = read_suite(results_dir / f"{project}.trx")
|
||||
except (OSError, ET.ParseError, ValueError, KeyError) as exc:
|
||||
errors.append(f"{project}: {exc}")
|
||||
for project, suite in suites.items():
|
||||
for result in suite["results"]:
|
||||
if result["outcome"] not in ("Passed", "NotExecuted"):
|
||||
errors.append(f"{project}: {result['name']} => {result['outcome']}")
|
||||
reports = []
|
||||
for group in groups:
|
||||
requirements = group["required_tests"]
|
||||
if not requirements:
|
||||
raise ValueError(f"{group['id']}: empty required test list")
|
||||
failures, matched = [], []
|
||||
seen = set()
|
||||
for requirement in requirements:
|
||||
project, method, minimum = (
|
||||
requirement["project"], requirement["method"], requirement["minimum_cases"]
|
||||
)
|
||||
if project not in projects or not method.startswith(project + "."):
|
||||
raise ValueError(f"{group['id']}: invalid project/method")
|
||||
if type(minimum) is not int or minimum < 1 or (project, method) in seen:
|
||||
raise ValueError(f"{group['id']}: duplicate test or invalid minimum_cases")
|
||||
seen.add((project, method))
|
||||
rows = [r for r in suites.get(project, {}).get("results", []) if r["method"] == method]
|
||||
if len(rows) < minimum:
|
||||
failures.append(f"{method}: expected at least {minimum} cases, found {len(rows)}")
|
||||
for row in rows:
|
||||
if row["outcome"] != "Passed":
|
||||
failures.append(f"{row['name']}: {row['outcome']}")
|
||||
matched.extend(rows)
|
||||
reports.append({
|
||||
"id": group["id"],
|
||||
"title": group["title"],
|
||||
"automated_status": "failed" if failures else "passed",
|
||||
"result_count": len(matched),
|
||||
"failures": failures,
|
||||
"remaining_acceptance": group["remaining_acceptance"],
|
||||
"tests": matched,
|
||||
})
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"automated_status": "failed" if errors or any(g["failures"] for g in reports) else "passed",
|
||||
"errors": errors,
|
||||
"suites": suites,
|
||||
"groups": reports,
|
||||
"notice": "Automated evidence only. Remaining acceptance is not waived; no tracker status is changed.",
|
||||
}
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--results", type=Path, default=Path("TestResults"))
|
||||
parser.add_argument("--manifest", type=Path, default=Path(__file__).with_name("windows-acceptance.json"))
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
report = evaluate(json.loads(args.manifest.read_text(encoding="utf-8")), args.results)
|
||||
except (OSError, ValueError, KeyError, TypeError) as exc:
|
||||
report = {"schema_version": 1, "automated_status": "failed", "errors": [str(exc)], "suites": {}, "groups": []}
|
||||
source_root = Path(__file__).resolve().parent.parent
|
||||
try:
|
||||
def git(*arguments):
|
||||
return subprocess.check_output(["git", *arguments], cwd=source_root, text=True).strip()
|
||||
report["source"] = {"commit": git("rev-parse", "HEAD"), "tree": git("rev-parse", "HEAD^{tree}")}
|
||||
if git("status", "--porcelain", "--untracked-files=no"):
|
||||
raise ValueError("Tracked source differs from the committed tree")
|
||||
except (OSError, subprocess.CalledProcessError, ValueError) as exc:
|
||||
report["errors"].append(str(exc))
|
||||
report["automated_status"] = "failed"
|
||||
report["runtime"] = platform.platform()
|
||||
report["created_at"] = datetime.now(timezone.utc).isoformat()
|
||||
report["workflow"] = {key: os.environ.get(key) for key in (
|
||||
"GITHUB_REPOSITORY", "GITHUB_RUN_ID", "GITHUB_RUN_ATTEMPT", "GITHUB_SHA"
|
||||
)}
|
||||
if platform.system() != "Windows":
|
||||
report["errors"].append("Windows acceptance must execute on Windows; cross-builds are not runtime evidence")
|
||||
report["automated_status"] = "failed"
|
||||
args.results.mkdir(parents=True, exist_ok=True)
|
||||
(args.results / "windows-acceptance.json").write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
|
||||
lines = ["# Windows automated acceptance", "", report["automated_status"].upper(), "",
|
||||
"Automated evidence only; this does not close tracker tasks or waive remaining acceptance.", ""]
|
||||
lines.extend(f"- {error}" for error in report["errors"])
|
||||
for project, suite in report["suites"].items():
|
||||
lines.append(f"- {project}: {suite['total']} results, {suite['outcomes']}")
|
||||
for group in report["groups"]:
|
||||
lines.extend(["", f"## {group['title']}: {group['automated_status']} ({group['result_count']} results)"])
|
||||
lines.extend(f"- FAIL: {f}" for f in group["failures"])
|
||||
lines.extend(f"- Remaining: {r}" for r in group["remaining_acceptance"])
|
||||
summary = "\n".join(lines) + "\n"
|
||||
(args.results / "windows-acceptance.md").write_text(summary, encoding="utf-8")
|
||||
if os.environ.get("GITHUB_STEP_SUMMARY"):
|
||||
with open(os.environ["GITHUB_STEP_SUMMARY"], "a", encoding="utf-8") as output:
|
||||
output.write(summary)
|
||||
print(summary)
|
||||
return 0 if report["automated_status"] == "passed" else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user