Context-Dependent Affordance Computation in Vision-Language Models

Working Paper DAI-2518

Murad Farzulla — Dissensus AI / King's College London

arXiv: 2603.04419 (v2) · DOI: 10.5281/zenodo.18091089

Abstract

We characterize the phenomenon of context-dependent affordance computation in vision-language models (VLMs). Our primary study uses Qwen3-VL-30B-A3B (n=3,213 scene-context pairs from COCO-2017) subject to systematic context priming across 7 agentic personas, with a cross-model replication on LLaVA-1.5-13B. We demonstrate massive affordance drift: in the Qwen3-VL data, mean Jaccard similarity between context conditions is 0.095 (95% CI [0.092, 0.097] across N=479 images; 9,244 prime pairs; p < 0.0001), indicating that >90% of lexical scene description is context-dependent; the LLaVA replication reproduces the effect (mean J = 0.160, 84% context-dependent). Sentence-level cosine similarity confirms substantial drift at the semantic level (mean = 0.415, 58.5% context-dependent). Stochastic baseline experiments (2,384 inference runs across 4 temperatures and 5 seeds) confirm this drift reflects genuine context effects rather than generation noise. Tucker decomposition with bootstrap stability analysis (n=1,000 resamples) reveals stable orthogonal latent factors. These findings establish that VLMs compute affordances in a substantially context-dependent manner and suggest a direction for robotics research: dynamic, query-dependent ontological projection (JIT Ontology) rather than static world modeling. We do not claim to establish processing order or architectural primacy; such claims require internal representational analysis beyond output behavior.

Repository Structure

semantic-vision/
├── paper/
│   ├── semantic-vision-v2.tex        # LaTeX source (v2, canonical; matches arXiv v2)
│   ├── semantic-vision-v2.pdf        # Compiled paper (arXiv v2 / Zenodo v2.1.0)
│   ├── references.bib                # Bibliography (46 entries)
│   ├── figures/                      # Figure 3 (Jaccard distribution)
│   └── semantic-vision-v2r3-arxiv.tar.gz  # arXiv v2 submission bundle
├── code/
│   ├── README.md                     # Pipeline documentation + known gaps
│   ├── requirements.txt              # Pinned environment for the reported results
│   ├── run_pilot.py                  # Pilot experiment runner
│   ├── semantic_vision_pilot_results_qwen3_30b.jsonl  # Raw pilot outputs (479 images × 7 primes)
│   ├── analyze_full.py               # Full analysis pipeline
│   ├── analyze_prelim.py             # Preliminary analysis
│   ├── analyze_with_significance.py  # Statistical significance testing
│   ├── image_level_inference.py      # Cluster-robust (image-level, N=479) t-tests for Table 2
│   ├── make_jaccard_figure.py        # Regenerates Figure 3 from committed data
│   ├── analyze_stochastic_baseline.py # Stochastic baseline analysis
│   ├── improved_similarity.py        # Alternative similarity metrics
│   ├── tucker_stability.py           # Tucker decomposition + bootstrap stability
│   ├── stochastic_baseline.py        # Stochastic baseline runner (model 1)
│   ├── stochastic_baseline_model2.py # Stochastic baseline runner (model 2)
│   ├── run_cross_model_replication.py # Cross-model replication (LLaVA)
│   ├── run_cross_model_ollama.py     # Cross-model replication via Ollama
│   ├── figures/                      # Figure 3 outputs
│   ├── analysis/                     # Generated analysis outputs (figures, tables, JSON)
│   │   ├── improved_similarity/      # Alternative metric comparisons (incl. pairwise_similarities.csv)
│   │   ├── stochastic_baseline/      # Stochastic baseline figures
│   │   └── tucker_stability/         # Tucker decomposition figures
│   └── results/                      # Raw cross-model results (JSONL)
└── CITATION.cff

Code Description

The code/ directory contains the full experimental pipeline (see code/README.md for run order and known gaps):

• Core experiment: run_pilot.py runs VLM inference across 7 agentic personas (chef, child, architect, etc.) on COCO-2017 images, collecting scene descriptions under each context prime. The raw outputs are committed as semantic_vision_pilot_results_qwen3_30b.jsonl.
• Statistical analysis: analyze_with_significance.py computes Jaccard and cosine similarity across conditions; image_level_inference.py reproduces the manuscript's cluster-robust Table 2 statistics (image-level N=479: word-level t = -351.30, object-level t = -84.13), superseding the pooled-pairs statistics retained in analysis_results.json.
• Figure 3: make_jaccard_figure.py regenerates figures/jaccard_distribution.pdf from the committed pairwise CSV and raw pilot JSONL.
• Stochastic baselines: stochastic_baseline.py and stochastic_baseline_model2.py run 2,384 inference passes across 4 temperatures and 5 seeds to separate context effects from generation noise.
• Tucker decomposition: tucker_stability.py performs tensor decomposition on the image-context-feature array with 1,000 bootstrap resamples for stability analysis.
• Alternative metrics: improved_similarity.py tests robustness across multiple similarity measures (raw Jaccard, lemmatized Jaccard, sentence-level cosine).
• Cross-model replication: run_cross_model_replication.py replicates the main experiment on LLaVA-1.5-13B via Ollama.

Note: The COCO-2017 validation images (~787MB) are not included. Download from cocodataset.org and place in code/data/coco/val2017/.

Citation

@misc{farzulla2026affordance,
  author        = {Farzulla, Murad},
  title         = {Context-Dependent Affordance Computation in Vision-Language Models},
  year          = {2026},
  eprint        = {2603.04419},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  doi           = {10.5281/zenodo.18091089}
}

Preprint

• arXiv (v2, July 2026): https://arxiv.org/abs/2603.04419
• Zenodo (v2.1.0, concept DOI resolves to latest): https://doi.org/10.5281/zenodo.18091089

License

• Paper: CC-BY-4.0
• Code: MIT

Links

• Paper (arXiv): https://arxiv.org/abs/2603.04419
• Paper (Zenodo): https://doi.org/10.5281/zenodo.18091089
• Code (GitHub): https://github.com/studiofarzulla/semantic-vision
• ASCRI programme page: https://systems.ac/5/DAI-2518
• Dissensus AI: https://dissensus.ai