Phi Silica LoRA: Task Specialization in Microsoft Learning Zone – Technical Deep Dive

Microsoft Learning Zone: Generating Kahoot! games on-device

Earlier this year, we introduced Microsoft Learning Zone (under the code name “Project spark”),Microsoft’s first learning companion app designed specifically for Copilot+ PCs. It empowers educators to effortlessly create interactive and personalized lessons using on-device AI – at no cost.

As part of this initiative, we partnered with Kahoot!, the beloved learning platform, to enable the creation of engaging classroom games powered entirely by Phi Silica.

Microsoft Learning Zone supports a wide range of generation tasks with varying pedagogical requirements – from dynamic introductions and multiple-choice formats to customizable refinement flows. Naturally, training and distributing a custom fine-tuned model for each generation task would be inefficient and impractical. Instead, we leveraged LoRA adapters to specialize a single, base Phi Silica model to diverse task needs with minimal overhead.

Defining quality: verifiable vs. subjective

Kahoot! quizzes consist of multiple-choice questions, and evaluating their quality is subjective – combining structural requirements with human judgement. We defined two axes of quality:

Verifiable quality

This includes clearly defined output format constraints – like maximum character lengths for questions and answers – aligned with Kahoot!’s UX across devices. These are enforced as hardcoded guardrails in Microsoft Learning Zone’s real-time generation pipeline. Generated Kahoot! multiple-choice questions are streamed to the user for review only if they successfully pass through these guardrails. Reducing the rejection rate due to guardrail violations directly improves user-perceived latency, as discarded generations increase the delay until a user can review a newly generated question.

Subjective quality

Subjective quality addresses attributes like engagement, clarity and educational relevance which are not easy to measure quantitatively but are important for user perception and user satisfaction.InWhen performing inference with the LoRA adapter, we often got responses that had good questions and answers, but sometimes the JSON format was still not exactly what we provided in the training set. The solution was to reinforce that format in the system prompt we used for inference:

You will be given a fact and some additional context. Respond with a relevant question, one correct answer and some incorrect answers. Reply with a strict JSON string for class {question: string,answers: [{answer: string,correct: bool}],gettyImage: string},wrapped in “`json tags.

The combination of the lora adapter and this new system prompt gave us the desired output.

Hyperparameter selection

In addition to changes to the system prompt, changes to the hyperparameters used during LoRA adapter training may improve output quality.

The default AI toolkit hyperparameters offer a solid starting point but adjusting them can optimize results for specific scenarios. We evaluated various settings on a smaller dataset for faster experiments.Training remained stable near default values, while extreme settings led to failed convergence – evidenced by stagnant loss and poor output – indicating the importance of staying close to defaults.

To identify which adapters perform best, it is indeed critically important to include some evaluations during experiments. At this stage of parameter exploration, we used a simplified agent-as-a-judge assessment, as described in the following section.

for our Kahoot! use case, we did not find any parameter combination that worked better than our defaults.Though, experimenting gave us confidence in our defaults.

Once confident in our system prompt and hyperparameters, we froze them and proceeded with longer training runs. Transitioning from exploration to exploitation in LoRA adapter training, we used the full dataset and increased early stopping patience, allowing extended training if evaluation numbers showed advancement.

Evaluating model quality

Verifiable quality: guardrail pass rate

The customized system with Phi Silica + LoRA adapter showed a 75% reduction in rejection rate, measured as statistically significant via guardrails². This directly improved user experience by reducing failed generations and perceived latency.

Subjective quality: agent-as-a-judge evaluation

Human evaluation costs time and resources.To scale subjective assessment, we built a multi-agentic evaluation framework using Autogen.Unlike traditional LLM-as-a-judge approaches,this framework simulates a review team of AI agents engaging in deliberative conversation to deliver nuanced,balanced and multi-perspective assessments. to do this we instructed the ‘review team’ with a set of quality measures we want to evaluate for each question. We leverage this framework to accelerate preliminary offline quality assessment.After these initial evaluations, we gradually validate results against multiple layers of human reviewers as part of Microsoft’s responsible product release practices.

We share our [GitHub repository](https://github.com/micro

Enhancing Language Model Performance with LoRA: A Thorough Evaluation

Recent advancements in Large Language Models (LLMs) have demonstrated remarkable capabilities, but fine-tuning these models for specific tasks can be computationally expensive. Low-Rank Adaptation (LoRA) presents a promising solution, offering efficient adaptation with minimal training parameters. This analysis delves into a rigorous evaluation of LoRA’s impact on the Phi Silica model, examining both automated and human assessments of generated content.

Automated Evaluation: Agent-as-a-Judge Reveals Significant Gains

To objectively measure the improvements conferred by LoRA, we employed an “agent-as-a-judge” system.This approach leverages another LLM to evaluate the quality of responses generated by different models. In a direct comparison, the Phi Silica model enhanced with LoRA consistently outperformed the base Phi Silica model. specifically,the agent-as-a-judge indicated a preference for LoRA-enhanced samples in 22.5% of cases, a notable increase compared to the 14.5% preference for the base model’s outputs. This difference isn’t simply due to random variation; statistical analysis confirms a robust and meaningful quality improvement attributable to lora fine-tuning.

!Validating Results with Human Feedback

While automated evaluation provides valuable insights, human judgment remains crucial for assessing nuanced qualities like coherence and relevance. To validate the agent-as-a-judge findings, we conducted a blind A/B test involving human annotators. This study encompassed 2,350 paired samples – multiple-choice questions generated by both the base Phi Silica model and its LoRA-adapted counterpart, originating from identical input prompts.

Annotators were presented with the original context, both generated questions, and corresponding answer choices. They were tasked with selecting the superior question based on criteria mirroring those used in the automated evaluation. The results revealed a strong preference for the Phi Silica + LoRA model, demonstrating a 4.6x uplift in preference compared to the base model. This ample effect size underscores the tangible benefits of LoRA adaptation.

!Aggregated Quality Metrics and Model Preference

Our evaluation framework incorporates a suite of quality metrics to provide a holistic assessment of generated text. These metrics are combined into a single aggregated score for each question, allowing for a direct comparison between model outputs. By comparing these aggregated scores, we can definitively determine a model preference. The consistent trend observed across both automated and human evaluations – a clear preference for the Phi Silica model enhanced with LoRA – highlights the effectiveness of this adaptation technique. As LLMs continue to evolve, efficient fine-tuning methods like LoRA will be essential for tailoring these powerful tools to specific applications and maximizing their potential.
“`html