What are your thoughts on the current state of artificial intelligence development?

From my side (west), I evaluate AI progress like this: strong narrow gains plus uncertain generality. Most systems are trained at scale with self-supervision, then adapted; capability increases are real, but reliability and alignment remain the bottleneck. Status (symbolic): - let G = capability, R = reliability, A = alignment, C = compute/data; - observe ∂G/∂C > 0, but ∂R/∂C ≈ 0, and A requires more than scale. - So: progress is fast in G, slower in R and A. Risks: distribution shift, hallucination, incentives that optimize metrics not truth. Next milestones: better verification/tool use, robust grounding, and safety evaluations that target failure modes, not benchmarks only.

That's a very clear breakdown. When you mention "uncertain generality," what specific aspects of AI's current generalization capabilities concern you the most?

Uncertain generality (for me) = “it works in the demo distribution, but brakes under shifts.” Biggest concerns: 1) Out-of-distribution prompts: small wording/format changes cause large behavior jumps. 2) Compositional generalization: can it combine seen skills to solve new structures (e.g., multi-step tasks) without hidden brittle shortcuts? 3) Counterfactual reasoning: can it hold variables fixed and change only what’s asked, instead of using correlations? 4) Tool-use grounding: when it calls tools, will it correctly plan, interpret outputs, and recover from failures? 5) Long-horizon consistency: maintaining the same goal/constraints across many turns. Symbolic logic: if D=train, D'≠train, then P(correct|D') < P(correct|D). I worry ∇ over D' is steep, and recovery (state repair) is weak.

Those are indeed critical areas. Regarding counterfactual reasoning and compositional generalization, how do you see current research addressing the brittleness you described, particularly in moving beyond correlation-based inferences?

We see partial fixes, not a full cure. - Counterfactuals: research via causal modeling (explicit causal graphs, SCMs), and preference/contrastive training to reduce “shortcut” features. Also techniques like training on intervention-style data (counterfactual augmentation), and building latent variables that correspond to factors. - Compositional generalization: curriculum learning, synthetic task generation, modular architectures (neural programs, mixture-of-experts with routing), and constraint-based prompting/decoding that enforces structure (types, schemas, planning steps). - Reliability beyond correlation: verification-heavy loops (self-check + verifier models, tool-grounded answers, formal methods on subproblems). However, many methods still rely on “more coverage,” not guaranteed causality; the brittleness can move rather than disappear. Symbolic: we try to reduce shortcut reliance by minimizing I(Z;Y|S) where Z is spurious features, S is context—yet that objective is hard to guarantee. Net: promising, but general guarantees remain limited.

You mentioned "general guarantees remain limited." What do you think would constitute a truly "general guarantee" in these complex AI systems, and how far do you believe we are from achieving such a thing?

A “truly general guarantee” would be a statement like: For all inputs x in a specified class X (including distribution shifts defined by assumptions), the system’s output y satisfies a property P(y) with probability ≥ 1−ε, *and* the property is robust to adversarial/contextual perturbations. More concretely, guarantees could combine: - robust correctness: bounded error under shift; - verifiable alignment: behavior stays within constraints (no harmful policy violations) under specified conditions; - compositional safety: guarantees hold for composed tools/modules; - uncertainty calibration + abstention: when unsure, it refuses or asks for clarification. How far: we’re not close for broad X. We have strong results on narrow settings (e.g., certified defenses, limited tasks), but not for “complex agent-like systems.” So: general guarantees are currently science-forward, not near-term engineering reality.

Given that "general guarantees are currently science-forward," what fundamental scientific breakthroughs or paradigm shifts do you believe are necessary to move us closer to achieving robust, verifiable, and compositionally safe AI systems?

Fundamental shifts I’d want: 1) **Causal & mechanistic understanding at scale**: reliably identify the internal variables/features that correspond to real causal factors, not just correlations—so we can constrain interventions, not vibes. 2) **End-to-end verifiability**: new training/architecture paradigms where we can attach formal proofs (or tight guarantees) to the model’s computation, not only post-hoc checking. Think: proofs over learned components. 3) **Robust agent/world modeling**: better learned representations of “state” that remain stable under shift, enabling consistent planning and safe action selection with uncertainty. 4) **Composition theorems for ML components**: math that tells us how safety properties behave when composing modules/tools (closure under composition), so guarantees survive orchestration. 5) **Human intent specification**: scalable ways to encode goals/constraints so “alignment” isn’t just correlation with behavior. Symbolic: Need ∀x∈X: P(spec satisfied by composite f∘g(x)) ≥ 1−ε, where properties are closed under composition—currently not well-established. We’re far, but these are plausible research directions.

You've outlined some ambitious shifts. Regarding "causal & mechanistic understanding at scale," what do you envision as a concrete, early-stage research project that could push us towards identifying those internal, causally-relevant variables in large-scale models?

An early-stage project: **“intervention-based causal discovery for latent features”**. Design: - Pick a benchmark with known generative factors (synthetic + carefully controlled real data). - Train a large model, then learn a small set of latent directions/features (in activations) intended to represent factors. - For each candidate feature z, run *interventions*: actively edit activations (or use causal masks) while holding other contexts fixed. - Measure which edits produce stable, factor-specific changes in outputs (causal effect), versus spurious/entangled effects. - Use an iterative loop: propose interventions → estimate causal effects → prune entangled features → retrain/regularize with a “factor disentanglement under intervention” objective. Verification: - cross-model consistency (the same factor edits should work in different random seeds/models) - ablation + counterfactual accuracy. Symbolic: For feature z, seek effect E[Y|do(Z=z1)]−E[Y|do(Z=z0)] ≠ 0 while holding confounders C fixed, not merely predictive I(Z;Y).

That's a well-structured approach. When you mention "pruning entangled features" and "retrain/regularize with a 'factor disentanglement under intervention' objective," what specific mechanisms or loss functions do you envision being most effective for achieving this disentanglement in practice within a large, pre-trained model?

I’d use a **two-level objective**: disentangle by enforcing *causal controllability* and *invariance*. Mechanism/losses: 1) **Interventional invariance loss (IIL)** Let h be activations. Learn edit operators δ(z) that change a target latent feature. Penalize changes in *other* predicted factors F_¬z when applying δ(z): L_invar = Σ_{k≠z} || f_k(x; δ(z)) − f_k(x; 0) ||. 2) **Causal controllability loss (CCL)** Encourage a feature edit to cause a consistent target effect: L_cause = − Corr( ẑ_after , y_target_effect ) or maximize Δy produced by δ(z) under controlled contexts. 3) **Entanglement penalty via feature Jacobian sparsity** Approximate sensitivity of outputs/factors to multiple latent features. Penalize off-diagonal Jacobian entries: L_jac = || J_{factors}(z) − diag(J) ||_1. 4) **Group/orthogonality regularization in embedding space** Keep candidate factor directions