Responsible AI: How PowerSchool safeguards millions of students with AI-powered content filtering using Amazon SageMaker AI

This post is cowritten with Gayathri Rengarajan and Harshit Kumar Nyati from PowerSchool.

PowerSchool is a leading provider of cloud-based software for K-12 education, serving over 60 million students in more than 90 countries and over 18,000 customers, including more than 90 of the top 100 districts by student enrollment in the United States. When we launched PowerBuddy, our AI assistant integrated across our multiple educational platforms, we faced a critical challenge: implementing content filtering sophisticated enough to distinguish between legitimate academic discussions and harmful content in educational contexts.

In this post, we demonstrate how we built and deployed a custom content filtering solution using Amazon SageMaker AI that achieved better accuracy while maintaining low false positive rates. We walk through our technical approach to fine tuning Llama 3.1 8B, our deployment architecture, and the performance results from internal validations.

PowerSchool’s PowerBuddy

PowerBuddy is an AI assistant that provides personalized insights, fosters engagement, and provides support throughout the educational journey. Educational leaders benefit from PowerBuddy being brought to their data and their users’ most common workflows within the PowerSchool ecosystem – such as Schoology Learning, Naviance CCLR, PowerSchool SIS, Performance Matters, and more – to ensure a consistent experience for students and their network of support providers at school and at home.

The PowerBuddy suite includes several AI solutions: PowerBuddy for Learning functions as a virtual tutor; PowerBuddy for College and Career provides insights for career exploration; PowerBuddy for Community simplifies access to district and school information, and others. The solution includes built-in accessibility features such as speech-to-text and text-to-speech functionality.

Content filtering for PowerBuddy

As an education technology provider serving millions of students—many of whom are minors—student safety is our highest priority. National data shows that approximately 20% of students ages 12–17 experience bullying, and 16% of high school students have reported seriously considering suicide. With PowerBuddy’s widespread adoption across K-12 schools, we needed robust guardrails specifically calibrated for educational environments.

The out-of-the-box content filtering and safety guardrails solutions available on the market didn’t fully meet PowerBuddy’s requirements, primarily because of the need for domain-specific awareness and fine-tuning within the education context. For example, when a high school student is learning about sensitive historical topics such as World War II or the Holocaust, it’s important that educational discussions aren’t mistakenly flagged for violent content. At the same time, the system must be able to detect and immediately alert school administrators to indications of potential harm or threats. Achieving this nuanced balance requires deep contextual understanding, which can only be enabled through targeted fine-tuning.

We needed to implement a sophisticated content filtering system that could intelligently differentiate between legitimate academic inquiries and truly harmful content—detecting and blocking prompts indicating bullying, self-harm, hate speech, inappropriate sexual content, violence, or harmful material not suitable for educational settings. Our challenge was finding a cloud solution to train and host a custom model that could reliably protect students while maintaining the educational functionality of PowerBuddy.

After evaluating multiple AI providers and cloud services that allow model customization and fine-tuning, we selected Amazon SageMaker AI as the most suitable platform based on these critical requirements:

• Platform stability: As a mission-critical service supporting millions of students daily, we require an enterprise-grade infrastructure with high availability and reliability.
• Autoscaling capabilities: Student usage patterns in education are highly cyclical, with significant traffic spikes during school hours. Our solution needed to handle these fluctuations without degrading performance.
• Control of model weights after fine-tuning: We needed control over our fine-tuned models to enable continuous refinement of our safety guardrails, enabling us to quickly respond to new types of harmful content that might emerge in educational settings.
• Incremental training capability: The ability to continually improve our content filtering model with new examples of problematic content was essential.
• Cost-effectiveness: We needed a solution that would allow us to protect students without creating prohibitive costs that would limit schools’ access to our educational tools.
• Granular control and transparency: Student safety demands visibility into how our filtering decisions are made, requiring a solution that isn’t a black box but provides transparency into model behavior and performance.
• Mature managed service: Our team needed to focus on educational applications rather than infrastructure management, making a comprehensive managed service with production-ready capabilities essential.

Solution overview

Our content filtering system architecture, shown in the preceding figure, consists of several key components:

1. Data preparation pipeline:
   • Curated datasets of safe and unsafe content examples specific to educational contexts
   • Data preprocessing and augmentation to ensure robust model training
   • Secure storage in Amazon S3 buckets with appropriate encryption and access controls
     Note: All training data was fully anonymized and did not include personally identifiable student information

2. Model training infrastructure:
   • SageMaker training jobs for fine-tuning Llama 3.1 8B

3. Inference architecture:
   • Deployment on SageMaker managed endpoints with auto-scaling configured
   • Integration with PowerBuddy through Amazon API Gateway for real-time content filtering
   • Monitoring and logging through Amazon CloudWatch for continuous quality assessment

4. Continuous improvement loop:
   • Feedback collection mechanism for false positives/negatives
   • Scheduled retraining cycles to incorporate new data and improve performance
   • A/B testing framework to evaluate model improvements before full deployment

Development process

After exploring multiple approaches to content filtering, we decided to fine-tune Llama 3.1 8B using Amazon SageMaker JumpStart. This decision followed our initial attempts to develop a content filtering model from scratch, which proved challenging to optimize for consistency across various types of harmful content.

SageMaker JumpStart significantly accelerated our development process by providing pre-configured environments and optimized hyperparameters for fine-tuning foundation models. The platform’s streamlined workflow allowed our team to focus on curating high-quality training data specific to educational safety concerns rather than spending time on infrastructure setup and hyperparameter tuning.

We fine-tuned Llama 3.1 8B model using Low Rank Adaptation (LoRA) technique on Amazon SageMaker AI training jobs, which allowed us to maintain full control over the training process.

After the fine-tuning was done, we deployed the model on SageMaker AI managed endpoint and integrated it as a critical safety component within our PowerBuddy architecture.

For our production deployment, we selected NVIDIA A10G GPUs available through ml.g5.12xlarge instances, which offered the ideal balance of performance and cost-effectiveness for our model size. The AWS team provided crucial guidance on selecting optimal model serving configuration for our use case. This advice helped us optimize both performance and cost by ensuring we weren’t over-provisioning resources.

Technical implementation

Below is the code snippet to fine-tune the model on the pre-processed dataset. Instruction tuning dataset is first converted into domain adaptation dataset format and scripts utilize Fully Sharded Data Parallel (FSDP) as well as Low Rank Adaptation (LoRA) method for fine-tuning the model.

We define an estimator object first. By default, these models train via domain adaptation, so you must indicate instruction tuning by setting the instruction_tuned hyperparameter to True.

estimator = JumpStartEstimator(
    model_id=model_id,
    environment={"accept_eula": "true"},  
    disable_output_compression=True,
    hyperparameters={
        "instruction_tuned": "True",
        "epoch": "5",
        "max_input_length": "1024",
        "chat_dataset": "False"
    },
    sagemaker_session=session,
    base_job_name = "CF-M-0219251"
)

After we define the estimator, we are ready to start training:

estimator.fit({"training": train_data_location})

After training, we created a model using the artifacts stored in S3 and deployed the model to a real-time endpoint for evaluation. We tested the model using our test dataset that covers key scenarios to validate performance and behavior. We calculated recall, F1, confusion matrix and inspected misclassifications. If needed, adjust hyperparameters/prompt template and retrain; otherwise proceed with production deployment.

You can also check out the sample notebook for fine tuning Llama 3 models on SageMaker JumpStart in SageMaker examples.

We used the Faster autoscaling on Amazon SageMaker realtime endpoints notebook to set up autoscaling on SageMaker AI endpoints.

Validation of solution

To validate our content filtering solution, we conducted extensive testing across multiple dimensions:

• Accuracy testing: In our internal validation testing, the model achieved ~93% accuracy in identifying harmful content across a diverse test set representing various forms of inappropriate material.
• False positive analysis: We worked to minimize instances where legitimate educational content was incorrectly flagged as harmful, achieving a false positive rate of less than 3.75% in test environments; results may vary by school context.
• Performance testing: Our solution maintained response times averaging 1.5 seconds. Even during peak usage periods simulating real classroom environments, the system consistently delivered seamless user experience with no failed transactions.
• Scalability and reliability validation:
  • Comprehensive load testing achieved 100% transaction success rate with consistent performance distribution, validating system reliability under sustained educational workload conditions.
  • Transactions completed successfully without degradation in performance or accuracy, demonstrating the system’s ability to scale effectively for classroom-sized concurrent usage scenarios.
• Production deployment: Initial rollout to a select group of schools showed consistent performance in real-world educational environments.
• Student safety outcomes: Schools reported a significant reduction in reported incidents of AI-enabled bullying or inappropriate content generation compared to other AI systems without specialized content filtering.

Fine-tuned model metrics compared to out-of-the-box content filtering solutions

The fine-tuned content filtering model demonstrated higher performance than generic, out-of-the-box filtering solutions in key safety metrics. It achieved a higher accuracy (0.93 compared to 0.89), and better F1-scores for both the safe (0.95 compared to 0.91) and unsafe (0.90 compared to 0.87) classes. The fine-tuned model also demonstrated a more balanced trade-off between precision and recall, indicating more consistent performance across classes. Importantly, it makes fewer false positive errors by misclassifying only 6 safe cases as unsafe, compared to 19 original responses in a test set of 160— a significant advantage in safety-sensitive applications. Overall, our fine-tuned content filtering model proved to be more reliable and effective.

Future plans

As the PowerBuddy suite evolves and is integrated into other PowerSchool products and agent flows, the content filter model will be continuously adapted and improved with fine tuning for other products with specific needs.

We plan to implement additional specialized adapters using the SageMaker AI multi-adapter inference feature alongside our content filtering model subject to feasibility and compliance consideration. The idea is to deploy fine-tuned small language models (SLMs) for specific problem solving in cases where large language models (LLMs) are huge and generic and don’t meet the need for narrower problem domains. For example:

• Decision making agents specific to the Education domain
• Data domain identification in cases of text to SQL queries

This approach will deliver significant cost savings by eliminating the need for separate model deployments while maintaining the specialized performance of each adapter.

The goal is to create an AI learning environment that is not only safe but also inclusive and responsive to diverse student needs across our global implementations, ultimately empowering students to learn effectively while being protected from harmful content.

Conclusion

The implementation of our specialized content filtering system on Amazon SageMaker AI has been transformative for PowerSchool’s ability to deliver safe AI experiences in educational settings. By building robust guardrails, we’ve addressed one of the primary concerns educators and parents have about introducing AI into classrooms—helping to ensure student safety.

As Shivani Stumpf, our Chief Product Officer, explains: “We’re now tracking around 500 school districts who’ve either purchased PowerBuddy or activated included features, reaching over 4.2 million students approximately. Our content filtering technology ensures students can benefit from AI-powered learning support without exposure to harmful content, creating a safe space for academic growth and exploration.”

The impact extends beyond just blocking harmful content. By establishing trust in our AI systems, we’ve enabled schools to embrace PowerBuddy as a valuable educational tool. Teachers report spending less time monitoring student interactions with technology and more time on personalized instruction. Students benefit from 24/7 learning support without the risks that might otherwise come with AI access.

For organizations requiring domain-specific safety guardrails, consider how the fine-tuning capabilities and managed endpoints of SageMaker AI can be adapted to your use case.

As we continue to expand PowerBuddy’s capabilities with the multi-adapter inference of SageMaker, we remain committed to maintaining the perfect balance between educational innovation and student safety—helping to ensure that AI becomes a positive force in education that parents, teachers, and students can trust.

About the authors

Gayathri Rengarajan is the Associate Director of Data Science at PowerSchool, leading the PowerBuddy initiative. Known for bridging deep technical expertise with strategic business needs, Gayathri has a proven track record of delivering enterprise-grade generative AI solutions from concept to production.

Harshit Kumar Nyati is a Lead Software Engineer at PowerSchool with 10+ years of experience in software engineering and analytics. He specializes in building enterprise-grade Generative AI applications using Amazon SageMaker AI, Amazon Bedrock, and other cloud services. His expertise includes fine-tuning LLMs, training ML models, hosting them in production, and designing MLOps pipelines to support the full lifecycle of AI applications.

Anjali Vijayakumar is a Senior Solutions Architect at AWS with over 9 years of experience helping customers build reliable and scalable cloud solutions. Based in Seattle, she specializes in architectural guidance for EdTech solutions, working closely with Education Technology companies to transform learning experiences through cloud innovation. Outside of work, Anjali enjoys exploring the Pacific Northwest through hiking.

Dmitry Soldatkin is a Senior AI/ML Solutions Architect at Amazon Web Services (AWS), helping customers design and build AI/ML solutions. Dmitry’s work covers a wide range of ML use cases, with a primary interest in Generative AI, deep learning, and scaling ML across the enterprise. He has helped companies in many industries, including insurance, financial services, utilities, and telecommunications. You can connect with Dmitry on LinkedIn.

Karan Jain is a Senior Machine Learning Specialist at AWS, where he leads the worldwide Go-To-Market strategy for Amazon SageMaker Inference. He helps customers accelerate their generative AI and ML journey on AWS by providing guidance on deployment, cost-optimization, and GTM strategy. He has led product, marketing, and business development efforts across industries for over 10 years, and is passionate about mapping complex service features to customer solutions.