Graduate student at SJTU, working at the intersection of speech, vision, and language. I'm mostly interested in how to make large multimodal models smaller, faster, and more useful — which usually means fighting with tokenizers, adapters, and whatever the latest efficiency trick is.
My current focus is on speech-vision alignment (getting a VLM to actually understand audio as more than just transcribed text) and efficient VLM fine-tuning (you don't need to update 7 billion parameters to teach a model new tricks).
Before all this, I spent a lot of time doing audio signal processing. Old habits die hard — I still think in spectrograms.
What I've been building
| speech-vlm-fusion | Cross-modal adapter toolkit: Whisper + LLaVA alignment |
| LightAdap | Dynamic visual token pruning + LoRA, 2-3× speedup |
| audio-cap-bench | Evaluation toolkit for audio captioning, with audio-specific metrics |
Stack
Currently
- Finishing experiments on audio-visual cross-modal alignment
- Reading about speculative decoding (it's actually clever)
- Trying to make audio-cap-bench useful for people who aren't me