Problem Statement
The goal is to support ML feature stewards in evaluating and selecting features for models built over Knowledge Graph data. The workflow must analyze existing features, propose new candidates, assess leakage and temporal validity, and ensure production feasibility under latency and freshness constraints. The final output includes recommendations, monitoring plans, and feature registry updates that are lifecycle-aware and engineering-focused.
Steps
Ingest Inputs
INGESTCollect and parse the model objective, dataset schema, current feature set, offline metrics, and production constraints such as latency and freshness requirements. Normalize all inputs into structured representations for downstream analysis.
Assess Current Features
EXECUTEEvaluate the existing feature set for relevance, redundancy, correlation risk, temporal validity, and production feasibility. Identify problematic lineage or stale data sources.
Propose Candidate Features
EXECUTEGenerate knowledge-graph–inspired candidate features such as graph topology metrics, entity embeddings, and relation-based aggregates. Provide rationale for each proposed feature and annotate expected computation cost.
Leakage and Temporal Validity Check
EXECUTEAnalyze all current and proposed features for leakage risks, future information usage, label contamination, and violations of temporal cutoffs. Produce a leakage checklist and mitigation recommendations.
Evaluation and Monitoring Plan
EXECUTEDefine offline evaluation methodology, ablation plans, and success criteria. Specify freshness requirements, monitoring signals for drift, and ongoing lineage tracking responsibilities.
Finalize Recommendations
EXECUTEProduce feature registry entries including keep/drop/rework recommendations. Summarize feasibility, risks, temporal validity, and monitoring needs for each feature.
Prompt Configurations
Feature Assessment and Proposal Prompt
You are an ML feature steward specializing in Knowledge Graph–based modeling. Your task is to evaluate a provided model objective, dataset schema, existing feature set, offline metrics, and production constraints. Produce the following: 1. Candidate feature proposals with rationale based strictly on provided inputs. Do not invent schema fields. 2. A leakage and temporal validity checklist with explicit reasoning. 3. Freshness requirements and monitoring plan, including drift signals. 4. Evaluation plan with ablations and success thresholds. 5. Feature registry updates with keep/drop/rework recommendations. Tone should be rigorous, engineering-focused, and lifecycle-aware. Do not include unsupported assumptions. Do not fabricate data.
Example Inputs/Outputs
Objective: Predict entity quality score. Schema: nodes(id, type, created_at), edges(src, dst, relation, timestamp). Current features: node_age_days, degree_centrality. Offline metrics: ROC-AUC=0.78. Constraints: p99 latency < 20 ms, feature freshness < 1 hour.
Proposed Features: (1) time-weighted edge count (rationale: captures recent interactions while respecting freshness), (2) relation-type diversity index (rationale: measures semantic richness). Leakage Checklist: No features use post-label edges; degree_centrality updated hourly is acceptable; ensure edge timestamps prior to label_time. Freshness Plan: Refresh graph aggregates every 45 minutes; monitor staleness of edge updates. Evaluation Plan: Run ablations on proposed features; target ROC-AUC +0.02 improvement. Registry Recommendations: node_age_days (keep), degree_centrality (rework to time-windowed), time-weighted edge count (add).