Azure Machine Learning
An Azure machine learning service for building and deploying models.
3,065 questions
What rank is used in LoRA fine-tuning?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 26%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENM%3C/text%3E%3C/svg%3E)
Nick Matton
0
Reputation points Microsoft Employee
I previously fine-tuned a few instances of gpt-4 using the standard Azure ML training flow. I am seeing online that Azure ML uses Low rank approximation in order to optimize fine tuning.
I am wondering what rank is used in the LoRA fine-tuning process so that I might determine the memory/flops saved by using Azure ML LoRA instead of running a full backprop on the model.
Thanks!
Sign in to answer