GALA: Geometry-Aware Language Model for Controllable Object Arrangement

Reasoning about fine-grained placement constraints, such as support and containment, is essential for realistic 3D scene creation. It remains a significant challenge for existing methods that rely on simplified object representations like oriented bounding box (OBB), which are blind to the detailed geometry. To address this limitation, we introduce a geometry-aware language model with explicit point cloud perception for controllable object placement. Our model performs autoregressively by predicting the 6D pose of one target object at a time conditioned on the scene context and placement instruction. We utilize a two-stage training strategy: the first stage aligns geometric and textual modalities through pretraining tasks, and the second stage fine-tunes the model for placement. To facilitate this task, we also curate a multi-source scene dataset covering diverse room types and fine-grained furniture arrangements, which will be open-sourced for community benefit. Experiments on our benchmark show our model significantly outperforms baselines without explicit geometric reasoning. We further develop a multi-stage plan–place–verify agent pipeline that uses the model for text-driven scene generation.

GALA instantiates a multimodal LLM whose text backbone is a pretrained causal language model and whose geometric branch is a 3D point cloud encoder followed by a lightweight projector. Given the scene boundary, the existing objects (with poses, sizes, categories, and point clouds), the new object to be inserted (with size, category, and point cloud), and a natural-language placement instruction, the model predicts a numeric 6D pose for the target object. Placement proceeds autoregressively: each predicted object is added back into the scene context before the next object is placed.

Figure 2. Overview of the geometry-aware placement model. Given the scene boundary, the already-placed objects (with poses, sizes, semantic categories, and point clouds), the new object (with size, semantic category, and point cloud), and a placement instruction, the multimodal LLM outputs a numeric 6D pose. The model performs placement autoregressively, adding each predicted object to the scene context before predicting the next one.

To make point cloud tokens readable to the LLM, we adopt a two-stage training recipe. In Stage 1, geometric–language alignment, the point cloud encoder and LLM backbone are frozen while only the projector is trained on auxiliary geometry question-answering tasks (boundary descriptions, object captions, centers, sizes, rotations, and pairwise distances). In Stage 2, autoregressive placement training, the projector and LLM are jointly fine-tuned to predict a serialized 6D pose conditioned on the textual instruction and the interleaved point-cloud tokens, while the point encoder stays frozen.

To train the geometry-aware placement model, we construct a high-quality scene dataset combining four complementary sources: the public 3D-FRONT and Imaginarium datasets, a procedurally generated SpatialLM-PCG collection built from SpatialLM scene annotations, and an artist-crafted scene collection. All sources are passed through a unified filtering pipeline: physics simulation and geometric validation remove floating objects, interpenetrations, and out-of-bound placements, and a VLM further scores placement plausibility, scene coherence, and visual quality before each scene is converted into single-step autoregressive placement samples with derived spatial relations and 6D pose supervision.

Figure 3. Dataset construction pipeline. Multi-source scenes are first cleaned by physical plausibility checks and VLM-based quality assessment. Each retained scene is parsed into a scene graph with absolute room-location edges, dependency edges (on, within, under), and proximity edges (next to). A floor-rooted scene tree is built from the dependency edges, and a breadth-first traversal yields the placement order. For each placement step, the tree edge and nearby proximity relations are verbalized into an instruction paired with per-object point clouds and the target 6D pose.

Figure 5. Overview of the handcrafted high-quality scene dataset. The gallery illustrates the diversity of room types, object categories, and fine-grained support and containment configurations.

To make GALA easier to use and to support end-to-end scene generation, we design a multi-stage agent pipeline that wraps GALA in a plan–place–verify loop. A planner analyzes the user prompt, maintains the object list and scene state, and asks an instruction generator to describe the next placement step. GALA then predicts the pose of the target object, after which a verifier checks the updated scene and sends any failure back to the instruction generator for the next iteration.

Figure 6. Agent pipeline for automated scene generation, where a planner turns a user prompt into object-level steps, GALA places each object, and a verifier feeds back failures for correction.

Figure 7. Gallery results from the automated scene generation agent. Given text prompts, the planner decomposes each request into object-level placement steps, GALA predicts poses for the target objects, and the verifier supports iterative correction to produce complete 3D scene arrangements.

We compare against ATISS and ReSpace as OBB-based references conditioned only on object category, a controlled LayoutVLM adaptation, and GPT-OSS-120B as a zero-shot large-parameter baseline, using a single-object insertion protocol. Our 0.7B model achieves the strongest instruction-conditioned performance while maintaining competitive physical plausibility.

Figure 8. Qualitative comparisons of instruction-conditioned single-object placement results across instruction-aware baselines and our method.

Figure 9. Qualitative comparison with baselines on whole-scene autoregressive inference results.

Figure 10. More autoregressive inference results of our model on held-out test scenes. Starting from scene-level object sets and placement instructions, our model sequentially predicts object poses and produces complete arrangements across different room types and layout complexity.

We isolate the contribution of point cloud input, Stage 1 alignment pretraining, and high-quality training scenes through controlled ablations on the same test dataset. Removing point cloud input or Stage 1 alignment pretraining most directly weakens the within-valid metric, where instruction following must be grounded in local containment geometry, and increases floating failures. Removing the Imaginarium and artist-crafted high-quality scenes produces the largest drop in within-valid rate, indicating these scenes primarily improve fine-grained geometric grounding for containment relations.

Figure 11. Qualitative ablation results on fine-grained placement. Each column shows one instruction, the predicted scene, and a zoomed crop around the target placement. Blue boxes mark inserted target objects.