From e6512f9b7f112c79c786e70083ff60611b2816b7 Mon Sep 17 00:00:00 2001 From: AJ Isaacs Date: Sun, 15 Feb 2026 19:14:35 -0500 Subject: [PATCH] Docs: Update ARCHITECTURE.md for tool-use, parse queue, and bulk upload Document AI tool-use framework, receipt parse queue system, background worker, bulk upload, ParseStatus enum, and updated AIReceiptParser flow with tool-aware vs enriched-prompt paths. Co-Authored-By: Claude Opus 4.6 --- ARCHITECTURE.md | 213 ++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 189 insertions(+), 24 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 9394483..2992343 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -41,6 +41,11 @@ MoneyMap follows a clean, service-oriented architecture: │ - ReceiptMatchingService (NEW) │ │ - ReceiptManager │ │ - AIReceiptParser │ +│ - ReceiptParseQueue (singleton) │ +│ - ReceiptParseWorkerService (hosted) │ +│ AI Tool Use: │ +│ - AIToolExecutor (DB query tools) │ +│ - AIToolRegistry (tool definitions) │ │ Reference & Dashboard: │ │ - ReferenceDataService (NEW) │ │ - DashboardService │ @@ -134,6 +139,9 @@ Stores uploaded receipt files (images/PDFs) linked to transactions. - `FileHashSha256` (string, 64) - SHA256 hash for deduplication - `UploadedAtUtc` (DateTime) - Upload timestamp +**Parse Queue Status:** +- `ParseStatus` (ReceiptParseStatus enum) - NotRequested(0), Queued(1), Parsing(2), Completed(3), Failed(4) + **Parsed Fields (populated by AI parser):** - `Merchant` (string, 200) - Merchant name extracted from receipt - `ReceiptDate` (DateTime?) - Date on receipt @@ -629,6 +637,11 @@ Represents a spending budget for a category or total spending. - Same as UploadReceiptAsync but for receipts without initial transaction mapping - TransactionId is null until later mapped +- `UploadManyUnmappedReceiptsAsync(IReadOnlyList files)` + - Bulk upload multiple receipt files + - Calls UploadReceiptInternalAsync for each file, collecting results + - Returns `BulkUploadResult` with lists of uploaded items and failures + - `MapReceiptToTransactionAsync(long receiptId, long transactionId)` - Links an unmapped receipt to a transaction - Returns success boolean @@ -654,35 +667,62 @@ Represents a spending budget for a category or total spending. **Location:** Services/ReceiptManager.cs:23-199 -### OpenAIReceiptParser (Services/OpenAIReceiptParser.cs) +### AIReceiptParser (Services/AIReceiptParser.cs) **Interface:** `IReceiptParser` -**Responsibility:** Parse receipts using OpenAI GPT-4o-mini Vision API. +**Responsibility:** Parse receipts using AI vision APIs with tool-use support for database-aware parsing. **Key Methods:** -- `ParseReceiptAsync(long receiptId)` +- `ParseReceiptAsync(long receiptId, string? model, string? notes)` - Loads receipt file from disk - Converts PDFs to PNG images using ImageMagick (220 DPI) - - Calls OpenAI Vision API with structured prompt - - Parses JSON response (merchant, date, due date, amounts, line items) + - Resolves vision client based on model prefix (openai, claude-, llamacpp:, ollama:) + - **Tool-aware clients (OpenAI, Claude, LlamaCpp):** Uses function calling to let the AI query the database during parsing + - **Non-tool clients (Ollama):** Pre-fetches database context and injects it into the prompt + - Parses JSON response (merchant, date, due date, amounts, line items, suggestedCategory, suggestedTransactionId) - Updates Receipt entity with extracted data - - Replaces existing line items + - Populates `ReceiptLineItem.Category` from AI response + - If AI suggested a transaction ID, attempts direct mapping before falling back to ReceiptAutoMapper - Logs parse attempt in ReceiptParseLog - - Attempts auto-mapping if receipt is unmapped - Returns `ReceiptParseResult` -**API Configuration:** -- Model: `gpt-4o-mini` -- Temperature: 0.1 (deterministic) -- Max tokens: 2000 -- API key: Environment variable `OPENAI_API_KEY` or config `OpenAI:ApiKey` +**Tool-Use Flow (OpenAI, Claude, LlamaCpp):** +``` +AI sees receipt image + prompt with tool instructions + ↓ +AI calls search_categories → gets existing categories from DB +AI calls search_transactions → finds matching bank transactions +AI calls search_merchants → normalizes merchant name + ↓ (up to 5 tool rounds) +AI returns final JSON with: + - Standard fields (merchant, date, total, lineItems) + - suggestedCategory (from existing categories) + - suggestedTransactionId (matched transaction) + - Per-line-item category +``` -**Prompt Strategy:** -- Structured JSON request with schema example -- Extracts: merchant, date, dueDate (for bills), subtotal, tax, total, confidence -- Line items with: description, quantity, unitPrice, lineTotal -- Special handling: Services/fees have null quantity (not products) -- Due date extraction: For bills (utility, credit card, etc.), extracts payment due date +**Ollama Fallback (enriched prompt):** +``` +Pre-fetch all categories, matching merchants, candidate transactions + ↓ +Inject as text block in prompt + ↓ +AI returns JSON using the provided context +``` + +**Supported Providers:** +| Provider | Model Prefix | Tool Use | Wire Format | +|----------|-------------|----------|-------------| +| OpenAI | (default) | Native | OpenAI /v1/chat/completions | +| Anthropic | claude- | Native | Anthropic /v1/messages | +| LlamaCpp | llamacpp: | Native | OpenAI-compatible /v1/chat/completions | +| Ollama | ollama: | Enriched prompt fallback | /api/generate | + +**Response Fields (ParsedReceiptData):** +- `Merchant`, `ReceiptDate`, `DueDate`, `Subtotal`, `Tax`, `Total`, `Confidence` - Standard fields +- `SuggestedCategory` (NEW) - AI's best category for the overall receipt +- `SuggestedTransactionId` (NEW) - Transaction ID the AI thinks matches this receipt +- `LineItems[].Category` (NEW) - Per-line-item category **PDF Handling:** - ImageMagick converts first page to PNG at 220 DPI @@ -690,11 +730,47 @@ Represents a spending budget for a category or total spending. - TrueColor 8-bit RGB output **Auto-Mapping Integration:** -- After successful parse of unmapped receipts, triggers ReceiptAutoMapper -- Attempts to automatically link receipt to matching transaction +- If AI suggests a specific transaction, attempts direct mapping first +- Falls back to ReceiptAutoMapper if AI mapping fails or no suggestion - Silently fails if auto-mapping unsuccessful (parsing still successful) -**Location:** Services/OpenAIReceiptParser.cs:23-342 +**Location:** Services/AIReceiptParser.cs + +### AIToolExecutor (Services/AITools/AIToolExecutor.cs) +**Interface:** `IAIToolExecutor` + +**Responsibility:** Execute AI tool calls against the database during receipt parsing. + +**Tools Available:** +| Tool | Purpose | Parameters | +|------|---------|------------| +| `search_categories` | Find existing categories with patterns and merchants | `query?` (optional filter) | +| `search_transactions` | Find unmapped bank transactions | `merchant?`, `minDate?`, `maxDate?`, `minAmount?`, `maxAmount?`, `limit?` | +| `search_merchants` | Look up known merchants | `query` (required) | + +**Key Methods:** +- `ExecuteAsync(AIToolCall)` - Dispatches to the correct handler, runs EF Core query, returns JSON result +- `GetEnrichedContextAsync(receiptDate?, total?, merchantHint?)` - Pre-fetches all data as text block for Ollama fallback + +**Design Constraints:** +- All tools are **read-only** database queries +- Results capped at 20 items +- Tool rounds capped at 5 per parse request +- Transactions already mapped to receipts are excluded from search results + +**Location:** Services/AITools/AIToolExecutor.cs + +### AIToolRegistry (Services/AITools/AIToolDefinitions.cs) + +**Responsibility:** Define tool schemas in a provider-agnostic format. + +**Key Types:** +- `AIToolDefinition` - Tool name, description, parameters +- `AIToolParameter` - Parameter name, type, description, required flag +- `AIToolCall` - Incoming tool call with arguments +- `AIToolResult` - Tool execution result returned to the AI + +**Location:** Services/AITools/AIToolDefinitions.cs ### ReceiptAutoMapper (Services/ReceiptAutoMapper.cs) **Interface:** `IReceiptAutoMapper` @@ -743,6 +819,30 @@ Represents a spending budget for a category or total spending. **Location:** Services/ReceiptAutoMapper.cs +### ReceiptParseQueue (Services/ReceiptParseQueue.cs) +**Interface:** `IReceiptParseQueue` +**Lifetime:** Singleton + +**Responsibility:** Thread-safe queue for receipt parsing jobs using `System.Threading.Channels`. + +**Key Members:** +- `EnqueueAsync(long receiptId)` - Add a receipt to the parse queue +- `EnqueueManyAsync(IEnumerable receiptIds)` - Bulk enqueue +- `DequeueAsync(CancellationToken ct)` - Wait for and dequeue the next receipt +- `QueueLength` - Current number of items waiting +- `CurrentlyProcessingId` - Receipt ID currently being parsed (thread-safe via `Interlocked`) + +### ReceiptParseWorkerService (Services/ReceiptParseWorkerService.cs) +**Type:** `BackgroundService` (hosted service) + +**Responsibility:** Continuously process receipts from the parse queue using the AI parser. + +**Behavior:** +1. **Startup Recovery:** Queries DB for receipts with `ParseStatus == Queued || Parsing`, re-enqueues them in upload order +2. **Processing Loop:** Dequeues one receipt at a time, sets status to `Parsing`, calls `IReceiptParser.ParseReceiptAsync`, updates status to `Completed` or `Failed` +3. **Status Updates:** Uses separate DB scopes for status writes to guarantee persistence even if parsing throws +4. **Graceful Shutdown:** Respects `CancellationToken` from host + ### FinancialAuditService (Services/FinancialAuditService.cs) **Interface:** `IFinancialAuditService` @@ -952,6 +1052,29 @@ EF Core DbContext managing all database entities. **Location:** Pages/Receipts.cshtml.cs +### BulkReceiptUpload.cshtml / BulkReceiptUploadModel +**Route:** `/BulkReceiptUpload` + +**Purpose:** Upload multiple receipt files at once with a queue dashboard showing parse progress. + +**Features:** +- Multi-file upload form with file list preview and upload spinner +- Queue dashboard with tabs: Queued (with position), Completed (merchant/total/confidence/line items), Failed (error + retry button) +- Currently-processing indicator with spinner +- AJAX polling every 3 seconds while items are active (auto-stops when idle) +- Each receipt links to `/ViewReceipt/{id}` +- Retry button for failed receipts re-queues them + +**Handlers:** +- `OnGetAsync()` - Loads queue dashboard data +- `OnPostUploadAsync(List files)` - Calls `UploadManyUnmappedReceiptsAsync` +- `OnGetQueueStatusAsync()` - Returns JSON for AJAX polling +- `OnPostRetryAsync(long receiptId)` - Re-queues a failed receipt + +**Dependencies:** `IReceiptManager`, `IReceiptParseQueue` + +**Location:** Pages/BulkReceiptUpload.cshtml.cs + ### EditTransaction.cshtml / EditTransactionModel **Route:** `/EditTransaction/{id}` @@ -1170,8 +1293,16 @@ builder.Services.AddScoped(); -builder.Services.AddHttpClient(); builder.Services.AddScoped(); + +// AI Vision Clients and Tool Use +builder.Services.AddHttpClient(); +builder.Services.AddHttpClient(); +builder.Services.AddHttpClient(); +builder.Services.AddHttpClient(); +builder.Services.AddScoped(); +builder.Services.AddScoped(); +builder.Services.AddScoped(); builder.Services.AddScoped(); ``` @@ -1240,7 +1371,9 @@ Subtotal (decimal(18,2)) Tax (decimal(18,2)) Total (decimal(18,2)) Currency (nvarchar(8)) +ParseStatus (int, NOT NULL, DEFAULT 0) -- 0=NotRequested, 1=Queued, 2=Parsing, 3=Completed, 4=Failed UNIQUE INDEX: (TransactionId, FileHashSha256) WHERE TransactionId IS NOT NULL +INDEX: ParseStatus ``` ### ReceiptParseLogs Table @@ -1683,10 +1816,31 @@ MoneyMap demonstrates a well-architected ASP.NET Core application with clear sep --- -**Last Updated:** 2025-12-15 -**Version:** 1.4 +**Last Updated:** 2026-02-11 +**Version:** 1.5 **Framework:** ASP.NET Core 8.0 / EF Core 9.0 +## Recent Changes (v1.5) + +### AI Tool Use for Receipt Parsing +- **Tool-Use Support**: AI can now query the database during receipt parsing via function calling +- **Three Tools**: `search_categories`, `search_transactions`, `search_merchants` - all read-only DB queries +- **Multi-Provider**: OpenAI, Claude, and LlamaCpp all support native tool calling; Ollama uses enriched prompt fallback +- **New Response Fields**: `suggestedCategory` (overall receipt), `suggestedTransactionId` (matched transaction), per-line-item `category` +- **AI-Suggested Mapping**: If the AI identifies a matching transaction, it's mapped directly before falling back to the scoring-based ReceiptAutoMapper +- **Category Population**: `ReceiptLineItem.Category` is now populated from AI responses (was previously unused) +- **Shared Tool-Use Helper**: `OpenAIToolUseHelper` implements the OpenAI-compatible tool-use loop shared by OpenAI and LlamaCpp clients +- **Anthropic Tool Use**: `ClaudeVisionClient` implements Anthropic-specific `tool_use`/`tool_result` content block format + +### New Files +- `Services/AITools/AIToolDefinitions.cs` - Provider-agnostic tool schema models and registry +- `Services/AITools/AIToolExecutor.cs` - Tool executor with database query handlers + +### Modified Files +- `Services/AIVisionClient.cs` - Added `IAIToolAwareVisionClient` interface, `OpenAIToolUseHelper`, tool-use implementations +- `Services/AIReceiptParser.cs` - Integrated tool executor, new response fields, enriched prompt fallback +- `Program.cs` - Registered `IAIToolExecutor` and `IAIVisionClientResolver` + ## Recent Changes (v1.4) ### Financial Audit API @@ -1744,3 +1898,14 @@ MoneyMap demonstrates a well-architected ASP.NET Core application with clear sep - Implemented `ReceiptAutoMapper` service with intelligent matching algorithm - Updated `ReceiptManager` with unmapped receipt support and duplicate detection - Added `MerchantService` for merchant management + +## Recent Changes (v1.3) + +### Bulk Receipt Upload & Parse Queue +- **ReceiptParseStatus**: New enum on `Receipt` model tracking parse lifecycle (NotRequested → Queued → Parsing → Completed/Failed) +- **ReceiptParseQueue**: Singleton `Channel`-based queue service replacing fire-and-forget `Task.Run` parsing +- **ReceiptParseWorkerService**: `BackgroundService` that processes parse queue sequentially, with startup recovery for interrupted items +- **Bulk Upload Page**: New `/BulkReceiptUpload` page with multi-file upload, queue dashboard (tabbed: Queued/Completed/Failed), AJAX polling, and retry for failed items +- **ReceiptManager.UploadManyUnmappedReceiptsAsync**: New bulk upload method with per-file error handling +- **ViewReceipt LLM Response**: Collapsible raw LLM response payload in parse log history +- **Unified Queue**: Both single and bulk receipt uploads now go through the same parse queue