Controlla: Learning Controllability via Graph-Constrained Latent Geometry
Under submission to a CORE A* venue · arXiv preprint, 2026
Controllable multimodal generation is commonly formulated as an inference-time conditioning problem — prompts, guidance, or auxiliary modules — which does not explicitly structure how semantic attributes evolve and can lead to identity drift and inconsistent cross-modal behavior. Controlla is a modular factorized-control framework that treats controllability as a property of structured latent geometry. It maps image, reference image, text, and audio into factorized identity and attribute spaces, and aligns these factors with identity and affective graph priors using graph-constrained optimal transport, so attribute edits follow graph-consistent trajectories while the reference identity is held stable.
To evaluate this setting the paper introduces AffectHuman-43K, a leakage-aware, identity-disjoint multimodal benchmark for reference-grounded affective control, together with geometry-aware metrics for trajectory smoothness, graph consistency, and identity–attribute disentanglement. Experiments show consistent improvements in controllability, identity preservation, and cross-modal alignment.