Central idea. Most controllable generation methods impose control at inference time through prompts, guidance, instructions, or auxiliary modules. Controlla instead asks whether controllability can be learned as a structured representation property: a latent space where semantic changes follow graph-consistent trajectories while identity-related factors remain preserved.
Controllable multimodal generation is commonly formulated as an inference-time conditioning problem: models receive prompts, instructions, guidance signals, or auxiliary controls and are expected to produce outputs satisfying the requested edit. Although effective, this paradigm does not explicitly structure how semantic attributes should evolve inside the model's representation space. As a result, semantic edits may be endpoint-correct but trajectory-unstable, causing identity drift, inconsistent intermediate states, or disagreement between modalities.
We propose Controlla, a modular factorized-control framework that treats controllability as a property of structured latent geometry. Controlla learns identity and attribute factors from multimodal inputs and aligns them with graph priors using graph-constrained optimal transport. This encourages controllable attributes to move along graph-consistent semantic paths while preserving a stable, reference-grounded identity factor.
We evaluate this framework using affective identity-preserving multimodal control as a challenging testbed: emotion requires fine-grained semantic change, identity preservation, and cross-modal consistency between image, text, and audio. To support this evaluation, we construct AffectHuman-43K, a leakage-aware multimodal benchmark with identity-disjoint splits, and introduce geometry-aware diagnostics for trajectory consistency and latent disentanglement. Experiments show consistent improvements in controllability, identity preservation, trajectory smoothness, and cross-modal alignment over editing, personalization, and control-based baselines.
The full manuscript includes the latent-geometry formulation, graph-constrained optimal transport objective, AffectHuman-43K construction protocol, evaluation metrics, main results, ablations, robustness studies, and extended qualitative analysis.
Open PaperControlla starts from a simple observation: controllability is not only about whether a model can reach a requested endpoint. A controllable model should also know how to move between semantic states without losing identity, introducing unintended changes, or producing inconsistent intermediate representations.
Instead of treating control as an external instruction applied to an otherwise implicit latent space, Controlla learns a factorized latent representation. One factor captures reference-grounded identity, while another captures controllable semantic attributes. A graph prior defines how attributes are related, and graph-constrained optimal transport aligns the learned attribute space with this structure. Generation then proceeds by traversing the attribute factor along graph-consistent paths while keeping identity stable.
Encodes the reference identity that should remain stable during semantic transformations.
Encodes controllable semantic attributes such as emotion, pose, lighting, style, or other graph-defined factors.
Aligns latent movement with semantic graph structure so edits become smooth, interpretable, and less prone to drift.
Controlla reframes controllability as a property of latent geometry rather than only an inference-time conditioning problem. The goal is not simply to add another control signal, but to structure the latent directions along which semantic change occurs.
The framework uses graph-constrained optimal transport to align learned identity and attribute factors with graph priors. This encourages semantic transformations to preserve relational structure rather than follow arbitrary interpolation paths.
AffectHuman-43K evaluates reference-grounded multimodal control with identity-disjoint splits, full modality coverage, and explicit leakage checks. The benchmark is designed to test whether a model can preserve visual identity while following heterogeneous affective control signals.
Controlla is evaluated against no-graph, random-graph, fully-connected graph, contrastive-only, metric-only, FGW/GW removal, modality ablation, and traversal variants. These controls test whether gains come from meaningful graph geometry rather than additional conditioning capacity alone.
Controlla is compared against prompt-based editing, instruction-guided editing, personalization, ControlNet-style conditioning, FLUX-style in-context editing, and hybrid identity-preserving pipelines. The qualitative comparisons show that graph-constrained latent traversal improves identity stability and semantic coherence under matched affective controls.
The graph is not used as a post-hoc visualization. It defines a semantic structure that guides latent organization and traversal. For affective control, the graph captures relationships among emotion states using class-level affect structure and fine-grained valence-arousal variation. The same mechanism can be instantiated with other graph-defined control spaces.
During training, graph-constrained optimal transport aligns the learned attribute factor with this graph structure, while identity factors are regularized to remain stable under semantic traversal. During generation, Controlla moves along graph-consistent paths rather than unconstrained linear interpolation.
AffectHuman-43K is a leakage-aware benchmark for reference-grounded affective control. It is not designed as a naturally co-recorded multimodal identity dataset. Instead, identity is specified by a visual reference image, while text and audio provide affective control signals. This deliberately separates identity preservation from controllable semantic change.
We use affective control as the primary testbed because it stresses several aspects of controllability at once: semantic accuracy, identity preservation, smooth transition behavior, multimodal consistency, and robustness to subtle expression differences.
Multi-identity evaluation under shared target conditions.
Class-wise view with additional affective variation examples.
A central question is whether Controlla's gains come from graph-constrained latent geometry or simply from additional fusion, factorization, or adapter capacity. We address this through controlled ablations that remove or replace the graph structure while keeping the overall framework comparable.
Removing graph alignment weakens controllability and trajectory consistency. Random and fully-connected graphs perform worse than meaningful affective graphs, showing that the improvement is not caused by arbitrary regularization. Removing FGW or GW degrades attribute controllability or identity stability, indicating that both semantic graph alignment and identity-structure preservation contribute to the final behavior.
Geometry-aware scores such as latent disentanglement and geodesic consistency are treated as mechanistic diagnostics: they test whether the learned representation exhibits the intended structure. The broader controllability claim is supported jointly by independent endpoint accuracy, identity preservation, trajectory smoothness, human preference, cross-dataset transfer, and ablation results.
Tests whether factorization alone is sufficient without graph-constrained alignment.
Tests whether any graph regularizer helps, or whether meaningful semantic topology is required.
Compares graph-consistent traversal against linear, spline, random-path, and fully-connected path alternatives.
Controllability becomes difficult when semantic classes are visually similar, cues are ambiguous, or identity and attribute changes compete. These examples highlight cases where prior editing methods often drift, over-edit, or collapse subtle distinctions, while graph-constrained traversal encourages smoother and more stable transitions.
Controlla is evaluated primarily on affective transformations because affect provides a challenging and measurable setting for identity-preserving multimodal control. However, the framework is not tied to emotion labels. The same formulation can be instantiated with any control factor that admits a relational structure, including pose, expression intensity, lighting, age, style, object attributes, or task-specific semantic states.
In this view, the affect graph is one instance of a broader design principle: define a graph over the semantic attribute space, align the learned attribute factor with that graph, and traverse the factor along graph-consistent paths while preserving identity or content factors. The plug-in pose and lighting studies demonstrate this mechanism beyond affective control.
A plug-in pose graph enables ordered pose traversal while preserving reference identity.
A plug-in lighting graph enables structured illumination changes while preserving identity and content.
Controlla improves endpoint emotion accuracy, trajectory smoothness, and human preference on AffectHuman-43K. The largest gains are especially visible in trajectory-sensitive metrics, supporting the central claim that controllability should be evaluated not only by endpoint correctness, but also by the stability and coherence of semantic movement.
Geometry-aware diagnostics show stronger latent disentanglement and graph consistency, while independent metrics such as accuracy, identity preservation, human preference, and cross-dataset transfer provide complementary evidence that the learned geometry improves controllable generation behavior.
| Method | Val Acc | Val TS | Val H | Test Acc | Test TS | Test H |
|---|---|---|---|---|---|---|
| ControlNet | 67.4 | 0.58 | 2.91 | 65.8 | 0.56 | 2.82 |
| ICEdit | 73.1 | 0.66 | 3.40 | 71.8 | 0.64 | 3.32 |
| FLUX.1 Kontext | 74.3 | 0.67 | 3.52 | 73.0 | 0.65 | 3.43 |
| DreamBooth + CNet++ | 75.1 | 0.67 | 3.58 | 73.6 | 0.65 | 3.47 |
| Controlla | 77.6 | 0.73 | 3.91 | 76.4 | 0.71 | 3.82 |
Controlla builds on advances in controllable diffusion, instruction-guided editing, personalization, multimodal representation learning, graph priors, and optimal transport. Unlike methods that primarily impose external control at inference time, Controlla focuses on shaping the latent space in which control occurs.
Related systems include ControlNet, DreamBooth, CLIP, and ImageBind.
@article{murthy2026controlla,
author = {Murthy, Jamuna S. and Monsefi, Amin Karimi and Ramnath, Rajiv},
title = {Controlla: Learning Controllability via Graph-Constrained Latent Geometry},
year = {2026},
}