Multimodal Fine-Tuning for Chinese Chart VQA

Two halves

case6/
├── data_create/             # React + Vite UI + FastAPI backend that GENERATES the training data
└── llamafactory_train.jsonl + llamafactory_val.jsonl  # the actual dataset used to fine-tune

The dataset format is LlamaFactory-native: each row is a multi-turn conversation with an image reference, all bundled into JSONL ready for the trainer.

Why not just SFT GPT-4o

Three reasons:

1. Vocabulary specificity — 中文 chart labels (营业收入 / 同比增长 / 占比) need to land in the model's vocabulary as single tokens, not be fragmented. A small fine-tune fixes this fast.
2. Cost at scale — once it works, you can run inference on your own GPU; no per-call API spend.
3. Domain familiarity — your charts have a house style (color palette, font, axis convention). The fine-tune learns that style and stops asking "is this a bar or a column?"

Dataset construction (data_create)

Mirrors case 7's NL2SQL data_create design, but for VLM:

• pick a chart template (柱状 / 折线 / 饼图 / 堆叠柱)
• generate synthetic data with realistic Chinese labels (industries / regions / time periods)
• render the chart (matplotlib / plotly) and save as PNG
• LLM generates 5-10 Q&A pairs per chart ("营收最高的是哪个？" "同比增长率多少？")
• export as LlamaFactory JSONL

Training (LlamaFactory)

llamafactory-cli train \
    --stage sft \
    --model_name_or_path /home/ubuntu/Qwen2.5-VL-7B-Instruct \
    --finetuning_type lora \
    --template qwen2_vl \
    --dataset_dir data \
    --dataset chart_vqa_train \
    --cutoff_len 4096 \
    --image_resolution 448 \
    --num_train_epochs 3.0 \
    --per_device_train_batch_size 2 \
    --gradient_accumulation_steps 8 \
    --lora_rank 16 \
    --lora_alpha 32 \
    --bf16 True

What this signals

• You can run vertical-domain multimodal fine-tuning end to end — not just call the API
• You understand the data generation half matters as much as the trainer — and you can build it
• You stay in the LlamaFactory ecosystem so it composes with the rest of the lineup (NL2SQL / function-calling / Qwen-VL RL)