You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Feedback on Training Parameters for Fine-Tuning a 7B RAG LLM, slow training:/ #191
Hardware: 22GB GPU
Input Length: MAX_LENGTH=10154 (because the model takes query, answer, and chunks as input).
Dataset: ~2K pairs.
Problem:
Training is extremely slow—around 1 minute per step, meaning 1000 steps take ~16 hours.
I expected slowness due to large input lengths, but this seems excessive.
Am I overlooking something? Any tips on improving training speed without exceeding memory limits?
Batch Size: Had to reduce per_device_train_batch_size=1 due to CUDA OOM errors. Also reduced LoRA Settings to r=64, lora_alpha=16.
By now you may have solved the issue but there is something wrong with input length in your design as it is very large. The better way of building RAG i think is by giving relevant chunks and let LLM do its job but you are feeding everything. Try reducing input length and use r less than alpha lora so that you effective scaling is higher!!
Activity
prijall commentedon Apr 8, 2026
By now you may have solved the issue but there is something wrong with input length in your design as it is very large. The better way of building RAG i think is by giving relevant chunks and let LLM do its job but you are feeding everything. Try reducing input length and use r less than alpha lora so that you effective scaling is higher!!