使用 LLaMA-Factory 微调 Qwen2.5 模型，并转换为 GGUF 格式部署

优质文章学习记录

— 25 min read

在开源大模型领域，Qwen 系列凭借强大的中文能力和友好的协议受到广泛欢迎。然而，直接使用基座模型往往无法满足特定业务场景的需求，需要通过微调来注入领域知识。微调后的模型如何高效部署？GGUF 格式是目前 llama.cpp 等推理后端广泛支持的格式，具有跨平台、内存映射等优点。本文将完整记录使用 LLaMA-Factory 微调 Qwen2.5-7B-Instruct 模型，并通过 llama.cpp 将微调后的模型转换为 GGUF 格式的全过程，并分享在转换过程中遇到的经典错误及其解决方案。

1.环境准备
我们在一台 Linux 服务器上操作，安装了 Conda 用于环境隔离。需要准备以下组件：
Python 3.10
LLaMA-Factory（用于微调）
llama.cpp（用于格式转换）
transformers、peft、accelerate 等依赖库

1.1 创建 Conda 环境
conda create -n llama_factory python=3.10 -y
conda activate llama_factory
1.2 安装 LLaMA-Factory
LLaMA-Factory 是一个高效的微调框架，支持多种模型和算法。我们通过源码安装：
git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e “.[torch,metrics]”
安装过程中如果遇到依赖冲突，可适当调整 transformers 版本，但建议保持最新。

1.3 安装 llama.cpp
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
pip install -r requirements.txt
注意：转换脚本 convert_hf_to_gguf.py 依赖 transformers，需要保证其版本兼容。

2.使用 LLaMA-Factory 微调 Qwen2.5-7B-Instruct
以 Qwen2.5-7B-Instruct 为基座，使用自定义数据集进行指令微调。假设数据已准备为 JSON 格式，每条包含 instruction 和 output 字段。

2.1 准备数据
将数据集放在 LLaMA-Factory/data 目录下，并创建数据集配置文件 dataset_info.json，示例如下：

{ "my_dataset": { "file_name": "my_dataset.json", "columns": { "prompt": "instruction", "response": "output" } } }

2.2 配置微调参数
LLaMA-Factory 支持通过命令行或 YAML 文件配置。这里我们使用命令行进行 LoRA 微调：

CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \ --stage sft \ --model_name_or_path Qwen/Qwen2.5-7B-Instruct \ --dataset my_dataset \ --dataset_dir ./data \ --finetuning_type lora \ --lora_target q_proj,v_proj \ --output_dir ./output/qwen2.5-lora \ --overwrite_cache \ --per_device_train_batch_size 4 \ --gradient_accumulation_steps 4 \ --lr_scheduler_type cosine \ --logging_steps 10 \ --save_steps 500 \ --learning_rate 1e-4 \ --num_train_epochs 3 \ --fp16

训练完成后，微调的 LoRA 权重保存在 ./output/qwen2.5-lora 目录中。

2.3 合并 LoRA 权重（如果需要导出完整模型）
如果希望得到一个完整的 HuggingFace 格式模型（而非仅 LoRA 适配器），可以使用 export_model.py 脚本合并：

python src/export_model.py \ --model_name_or_path Qwen/Qwen2.5-7B-Instruct \ --adapter_name_or_path ./output/qwen2.5-lora \ --template default \ --finetuning_type lora \ --export_dir ./output/qwen2.5-merged

合并后的完整模型将保存在 ./output/qwen2.5-merged 中，包含所有必要的配置文件、分词器和权重文件。

3.将微调后的模型转换为 GGUF 格式
3.1 准备转换环境
为了转换，我们需要一个独立的 conda 环境，以免与 LLaMA-Factory 的依赖冲突。创建一个新环境并安装必要工具：

conda create -n llama.cpp python=3.10 -y conda activate llama.cpp pip install torch transformers sentencepiece protobuf

3.2 使用 llama.cpp 的转换脚本
进入 llama.cpp 目录，执行转换命令（假设合并后的模型位于 /mnt/workspace/output/qwen2.5-merged）：

cd /path/to/llama.cpp python convert_hf_to_gguf.py /mnt/workspace/output/qwen2.5-merged \ --outtype f16 \ --verbose \ --outfile /mnt/workspace/qwen2.5-7B-instruct.gguf

3.3 遇到的经典错误及解决
在执行上述命令时，遇到了以下错误：

python llama.cpp/convert_hf_to_gguf.py /mnt/workspace/.cache/modelscope/models/Qwen/Qwen2.5-7B-Instruct-lora --outtype f16 --verbose --outfile /mnt/workspace/Meta-Llama-3-8B-Instruct-gguf.gguf INFO:hf-to-gguf:Loading model: Qwen2.5-7B-Instruct-lora INFO:hf-to-gguf:Model architecture: Qwen2ForCausalLM INFO:hf-to-gguf:gguf: loading model weight map from 'model.safetensors.index.json' INFO:hf-to-gguf:gguf: indexing model part 'model-00001-of-00004.safetensors' INFO:hf-to-gguf:gguf: indexing model part 'model-00002-of-00004.safetensors' INFO:hf-to-gguf:gguf: indexing model part 'model-00003-of-00004.safetensors' INFO:hf-to-gguf:gguf: indexing model part 'model-00004-of-00004.safetensors' INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only INFO:hf-to-gguf:Exporting model... INFO:hf-to-gguf:output.weight, torch.bfloat16 --> F16, shape = {3584, 152064} INFO:hf-to-gguf:token_embd.weight, torch.bfloat16 --> F16, shape = {3584, 152064} INFO:hf-to-gguf:blk.0.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.0.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.0.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.0.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.0.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.0.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.0.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.0.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.0.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.0.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.0.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.0.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.1.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.1.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.1.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.1.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.1.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.1.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.1.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.1.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.1.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.1.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.1.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.1.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.2.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.2.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.2.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.2.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.2.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.2.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.2.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.2.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.2.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.2.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.2.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.2.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.3.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.3.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.3.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.3.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.3.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.3.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.3.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.3.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.3.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.3.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.3.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.3.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.4.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.4.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.4.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.4.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.4.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.4.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.4.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.4.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.4.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.4.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.4.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.4.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.5.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.5.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.5.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.5.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.5.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.5.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.5.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.5.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.5.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.5.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.5.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.5.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.6.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.10.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.10.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.10.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.10.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.10.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.10.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.10.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.10.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.10.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.10.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.10.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.10.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.11.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.11.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.11.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.11.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.11.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.11.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.11.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.11.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.11.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.11.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.11.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.11.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.12.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.12.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.12.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.12.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.12.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.12.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.12.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.12.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.12.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.12.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.12.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.12.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.13.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.13.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.13.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.13.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.13.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.13.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.13.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.13.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.13.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.13.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.13.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.13.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.14.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.14.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.14.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.14.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.14.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.14.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.14.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.14.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.14.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.14.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.14.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.14.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.15.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.15.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.15.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.15.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.15.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.15.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.15.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.15.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.15.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.15.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.15.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.15.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.16.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.16.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.16.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.6.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.6.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.6.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.6.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.6.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.6.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.6.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.6.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.6.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.6.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.6.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.7.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.7.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.7.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.7.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.7.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.7.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.7.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.7.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.7.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.7.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.7.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.7.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.8.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.8.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.8.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.8.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.8.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.8.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.8.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.8.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.8.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.8.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.8.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.8.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.9.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.9.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.9.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.9.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.9.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.9.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.9.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.9.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.9.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.9.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.9.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.9.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.16.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.16.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.16.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.16.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.16.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.16.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.16.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.16.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.16.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.17.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.17.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.17.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.17.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.17.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.17.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.17.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.17.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.17.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.17.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.17.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.17.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.18.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.18.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.18.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.18.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.18.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.18.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.18.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.18.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.18.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.18.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.18.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.18.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.19.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.19.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.19.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.19.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.19.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.19.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.19.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.19.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.19.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.19.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.19.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.19.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.20.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.20.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.20.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.20.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.20.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.20.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.20.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.20.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.20.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.20.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.20.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.20.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.21.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.21.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.21.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.21.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.21.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.21.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.21.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.21.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.21.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.21.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.21.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.21.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.22.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.22.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.22.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.22.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.22.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.22.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.22.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.22.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.22.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.22.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.22.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.22.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.23.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.23.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.23.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.23.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.23.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.23.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.23.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.23.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.23.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.23.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.23.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.23.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.24.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.24.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.24.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.24.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.24.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.24.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.24.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.24.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.24.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.24.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.24.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.24.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.25.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.25.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.25.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.25.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.25.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.25.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.25.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.25.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.25.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.25.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.25.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.25.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.26.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.26.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.26.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.26.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.26.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.26.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.26.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.26.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.26.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.26.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.26.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.26.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.27.attn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.27.ffn_down.weight, torch.bfloat16 --> F16, shape = {18944, 3584} INFO:hf-to-gguf:blk.27.ffn_gate.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.27.ffn_up.weight, torch.bfloat16 --> F16, shape = {3584, 18944} INFO:hf-to-gguf:blk.27.ffn_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.27.attn_k.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.27.attn_k.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:blk.27.attn_output.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.27.attn_q.bias, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:blk.27.attn_q.weight, torch.bfloat16 --> F16, shape = {3584, 3584} INFO:hf-to-gguf:blk.27.attn_v.bias, torch.bfloat16 --> F32, shape = {512} INFO:hf-to-gguf:blk.27.attn_v.weight, torch.bfloat16 --> F16, shape = {3584, 512} INFO:hf-to-gguf:output_norm.weight, torch.bfloat16 --> F32, shape = {3584} INFO:hf-to-gguf:Set meta model INFO:hf-to-gguf:Set model parameters INFO:hf-to-gguf:gguf: context length = 32768 INFO:hf-to-gguf:gguf: embedding length = 3584 INFO:hf-to-gguf:gguf: feed forward length = 18944 INFO:hf-to-gguf:gguf: head count = 28 INFO:hf-to-gguf:gguf: key-value head count = 4 WARNING:hf-to-gguf:Unknown RoPE type: default INFO:hf-to-gguf:gguf: rope scaling type = NONE INFO:hf-to-gguf:gguf: rope theta = 1000000.0 INFO:hf-to-gguf:gguf: rms norm epsilon = 1e-06 INFO:hf-to-gguf:gguf: file type = 1 INFO:hf-to-gguf:Set model quantization version INFO:hf-to-gguf:Set model tokenizer Traceback (most recent call last): File "/mnt/workspace/llama.cpp/convert_hf_to_gguf.py", line 3534, in set_vocab self._set_vocab_sentencepiece() File "/mnt/workspace/llama.cpp/convert_hf_to_gguf.py", line 1358, in _set_vocab_sentencepiece tokens, scores, toktypes = self._create_vocab_sentencepiece() File "/mnt/workspace/llama.cpp/convert_hf_to_gguf.py", line 1375, in _create_vocab_sentencepiece raise FileNotFoundError(f"File not found: {tokenizer_path}") FileNotFoundError: File not found: /mnt/workspace/.cache/modelscope/models/Qwen/Qwen2.5-7B-Instruct-lora/tokenizer.model During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/mnt/workspace/llama.cpp/convert_hf_to_gguf.py", line 11934, in <module> main() File "/mnt/workspace/llama.cpp/convert_hf_to_gguf.py", line 11928, in main model_instance.write() File "/mnt/workspace/llama.cpp/convert_hf_to_gguf.py", line 689, in write self.prepare_metadata(vocab_only=False) File "/mnt/workspace/llama.cpp/convert_hf_to_gguf.py", line 830, in prepare_metadata self.set_vocab() File "/mnt/workspace/llama.cpp/convert_hf_to_gguf.py", line 3536, in set_vocab self._set_vocab_gpt2() File "/mnt/workspace/llama.cpp/convert_hf_to_gguf.py", line 1294, in _set_vocab_gpt2 tokens, toktypes, tokpre = self.get_vocab_base() File "/mnt/workspace/llama.cpp/convert_hf_to_gguf.py", line 978, in get_vocab_base tokenizer = AutoTokenizer.from_pretrained(self.dir_model) File "/root/miniconda3/envs/llama.cpp/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 814, in from_pretrained return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) File "/root/miniconda3/envs/llama.cpp/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2029, in from_pretrained return cls._from_pretrained( File "/root/miniconda3/envs/llama.cpp/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2261, in _from_pretrained tokenizer = cls(*init_inputs, **init_kwargs) File "/root/miniconda3/envs/llama.cpp/lib/python3.10/site-packages/transformers/models/qwen2/tokenization_qwen2_fast.py", line 129, in __init__ super().__init__( File "/root/miniconda3/envs/llama.cpp/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 111, in __init__ fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file) Exception: data did not match any variant of untagged enum ModelWrapper at line 757443 column 3 (llama.cpp) root@dsw-1662938-774cbc5758-kd9bv:/mnt/workspace#

错误原因分析
该错误发生在转换脚本加载 tokenizer.json 文件时，提示 JSON 结构不符合预期。通常由两个原因引起：

tokenizer.json 文件损坏：可能是下载不完整或微调过程中被意外修改。

transformers 版本不兼容：某些较新的 tokenizer.json 格式需要特定版本的 transformers 才能正确解析。

解决方案
经过排查，发现是 transformers 版本问题。当前环境中的 transformers 为旧版（如 4.36.0），而 Qwen2.5 的 tokenizer 需要更新版本支持。我们通过强制安装 transformers==4.45.0 解决了问题：

bash
pip install --force-reinstall transformers==4.45.0
重新运行转换命令，成功导出 GGUF 文件！

注意：如果模型目录中的 tokenizer.json 确实损坏，可以从 HuggingFace 官方仓库重新下载覆盖。

3.4 验证转换结果
转换完成后，检查输出文件：

ls -lh /mnt/workspace/qwen2.5-7B-instruct.gguf
可以使用 llama.cpp 提供的简单测试工具验证模型加载：

./main -m /mnt/workspace/qwen2.5-7B-instruct.gguf -p “你好，请介绍一下你自己。” -n 100
如果正常输出，说明转换成功。

4.总结
通过本文的实践，完成了以下工作：
使用 LLaMA-Factory 对 Qwen2.5-7B-Instruct 进行了 LoRA 微调，并合并为完整模型。
利用 llama.cpp 的转换工具将微调后的模型转换为 GGUF 格式，以便高效部署。
解决了转换过程中遇到的 tokenizer.json 解析错误，关键在于确保 transformers 版本与模型兼容。

关键点总结：
版本兼容性：转换脚本对 transformers 版本敏感，建议使用较新稳定版（如 4.45.0）。
文件完整性：微调后务必检查 tokenizer.json 是否完好，必要时从官方源补充。
路径命名：转换命令中的输出文件名建议与模型对应，避免混淆（如本文示例中应避免误写成 Llama 相关名称）。
GGUF 格式的模型可以轻松在 llama.cpp、Ollama、LM Studio 等推理后端运行，极大地方便了本地部署。

Read more

AIGC（生成式AI）试用 45 -- DocsGPT 与 Python开发 1

一切从python调用本地DocsGPT完成python开发开始。 遗留问题：如何验证AI开发提交的结果？ * 提问 1： 使用python+Tkinter进行GUI程序编码 1. 界面分为左右两部分     - 左侧为python代码编辑区：       左上部为代码多行输入框，嵌入python idle，浅灰色底色；       左下部为 Run 按钮     - 右侧为GPT调用区：       右上部为tab，名称 Question，嵌入多行文本，输入提问问题；       中部为Show Answer按钮，海蓝色；       下部为2个tab：tab1，名称 Answer，嵌入多行文本，显示GPT处理结果；                                tab2，名称History，显示提问历史，answer + question，数据来自名为pyai的sqlite的数据库  2. 优化界面  3. 优化代码 * DeepSeek 回复 1： - 1 次调用界面
本地部署 Kimi K2 全指南（llama.cpp、vLLM、Docker 三法）

本地部署 Kimi K2 全指南（llama.cpp、vLLM、Docker 三法）

Kimi K2 是 Moonshot AI 于2025年7月11日发布的高性能多专家语言模型（MoE），支持最大 128K 上下文，激活参数规模为 32B，具备极强的推理、代码生成与多轮对话能力。自从其权重以多种格式开源以来，许多开发者希望将其部署在本地，以获得更高的私密性和灵活性。 本文将详细介绍三种主流本地部署路径，并提供完整的配置步骤和使用建议。 📦 准备工作（通用部分） 在进行部署前，请准备如下环境与资源： ✅ 最低硬件配置建议： 项目要求存储空间≥ 250 GB（用于量化模型，若使用 FP8 请预留 1 TB）内存≥ 128 GB RAM（越大越流畅）GPU≥ 24 GB 显存，推荐多卡（如 2×A100、H100）操作系统Linux（Ubuntu 推荐）
Copilot vs Claude Code终极对决哪个会更好用呢？

Copilot vs Claude Code终极对决哪个会更好用呢？

📊 核心差异：一句话概括 * GitHub Copilot：你的智能代码补全器 * Claude Code：你的全栈AI开发伙伴 🎯 一、产品定位对比 GitHub Copilot：专注代码补全 <TEXT> 定位：AI结对编程助手 核心理念：让你写代码更快 核心功能：基于上下文的代码建议和补全 收费模式：个人$10/月，企业$19/用户/月 Claude Code：全栈开发加速器 <TEXT> 定位：AI驱动的开发平台 核心理念：提升整个开发流程效率 核心功能：代码生成+架构设计+调试+部署 收费模式：按token计费，灵活弹性 ⚡ 二、核心技术对比

dify平台集成OCR：低代码+AI模型打造智能表单识别系统

dify平台集成OCR：低代码+AI模型打造智能表单识别系统 📖 项目背景与技术选型动因 在企业数字化转型过程中，大量纸质表单、发票、合同等非结构化文档需要转化为可处理的结构化数据。传统人工录入方式效率低、成本高、易出错，而通用OCR服务往往对中文支持不完善，尤其在复杂背景或手写体场景下识别准确率骤降。 为此，我们基于 dify 低代码平台，集成了一套轻量级但高精度的 OCR 文字识别系统。该系统采用经典的 CRNN（Convolutional Recurrent Neural Network）模型架构，专为中英文混合文本识别优化，在无GPU依赖的前提下实现 <1秒 的平均响应时间，真正做到了“开箱即用”的工业级OCR能力。 本方案的核心价值在于： - 低代码集成：通过dify平台快速接入AI能力，无需深度开发即可构建智能表单应用 - 高识别精度：相比传统轻量模型，CRNN在中文长文本、模糊图像、倾斜排版等复杂场景下表现更优 - 双模输出支持：同时提供可视化Web界面和标准REST API，