From Rules to Hybrid ML: SAG Auto-Tuning and Optimizer Practice

SAG retrieval has 7 hyperparameters — manual tuning is not feasible. We solved this with a three-layer progressive optimization approach.

1. SAG Auto-Tuning: Three-Layer Architecture

Layer 1: Rule-based Search (Base Layer)
│  Grid Search / Random Search
│  Pure program, no LLM required
│  Traverse parameter space → Evaluate → Output optimal
│
Layer 2: Bayesian Optimization (Advanced Layer)
│  Optuna Bayesian search
│  Adaptively adjust search direction based on historical evaluation results
│  ~50 trials to find near-optimal solution
│
Layer 3: LLM Agent (High-Level Layer)
   LLM analyzes tuning history → generates insights → proposes new parameter combinations
   "Last time vector_weight=0.6 caused a recall drop, possibly because..."
   → Generate new suggestion → Evaluate → Loop

Evaluation Metrics

def evaluate_config(config: dict, max_cases: int = 100) -> dict:
    """Run eval cases for a single parameter configuration and compute recall."""
    params = SAGParams.from_dict(config)
    hits = 0
    for case_id in case_ids:
        results = sag_retrieve(query, top_k=10, params=params)
        retrieved = {r.chunk_id for r in results}
        if retrieved & expected_ids:
            hits += 1
    return {"recall": hits / len(case_ids), "config": config, ...}

Tuning History Persistence

// reports/sag_tuning/history.jsonl
{"trial": 1, "config": {"vector_weight": 0.5, ...}, "recall": 0.72, "ts": "..."}
{"trial": 2, "config": {"vector_weight": 0.6, ...}, "recall": 0.68, "ts": "..."}
{"trial": 3, "config": {"structural_base": 0.7, ...}, "recall": 0.78, "ts": "..."}

2. TicketPilot Optimizer: From Rules to Hybrid ML

A concrete application scenario — automated customer service ticket classification (intent / severity / risk):

Initial State: Pure Rule-Based Classification
├─ intent: Keyword matching (8 intent classes)
├─ severity: Derived from risk flag count (LOW/MEDIUM/HIGH)
├─ risk: 5 keywords × 6 risk flags
└─ composite score: 0.624

Optimization Path:
├─ Phase 1: Rule Optimization
│   ├─ Expand risk keywords (21% → 50%+)
│   ├─ Direct severity classification (54% → 65%+)
│   ├─ Confidence threshold fixes
│   └─ Batch optimizer (multiple fixes per round)
│
├─ Phase 2: Lightweight ML
│   ├─ FastText hybrid classifier (intent 61% → 70%+)
│   ├─ External data augmentation (JDDC/COLDataset)
│   └─ NSGA-II multi-objective optimization
│
└─ Phase 3: LLM-Guided
    └─ LLM analyzes error patterns → generates rule variants

Hybrid Classifier Architecture

Input (user text)
    │
    ├─→ Rule Classifier (keyword matching)
    │   └─ confidence_score (based on matched keyword count/weight)
    │
    ├─→ FastText Classifier (subword n-gram)
    │   └─ probability (trained on 400 samples)
    │
    └─→ Ensemble
        ├─ rule_conf > 0.8 → use rule
        ├─ fasttext_prob > 0.7 → use fasttext
        └─ both uncertain → manual review

NSGA-II Multi-Objective Optimization

Simultaneously optimize F1 for 8 intents with no regression constraints:

class TicketPilotRuleOpt(ElementwiseProblem):
    # 8 objectives: 1-F1 for each intent (minimization)
    # Constraints: F1 for each intent ≥ baseline
    # Decision variables: keyword inclusion/exclusion per intent (0/1)
    
    # Results: Pareto front
    # Solution A: intent_1 F1=0.85, intent_2 F1=0.72
    # Solution B: intent_1 F1=0.80, intent_2 F1=0.78
    # → Manual trade-off selection