For youth, AI is starting to feel as natural as using a search engine, like Google, for their day to day inquiries. In Kids Help Phone’s 2025 study exploring how Canadian youth engage with mental health content online, they found that:
69% of youth said AI-generated mental health content was helpful
17% had used tools like ChatGPT for mental health advice
While AI responses can seem helpful in the moment, these interactions often lack the specific context required to keep youth safe. To address this, Kids Help Phone set out to explore what a generative AI system might look like when it is designed to detect nuance in language during these conversations with youth.
For over a month, we had co-built a prototype called the AI Starter Kit: a modular, multi-agent system designed to help organizations experiment with AI safely in high-trust environments.
Exploring “Proof of Care”
The goal was to create a safe environment where AI could support youth mental health conversations and provide the right support.
This meant building a system capable of:
Interpreting the emotional context of a message
Generating responses aligned with clinically informed guidance
Identifying moments where human support is necessary
Operating within safety rules shaped by experts
Rather than a single chatbot, we built a layered system of specialized AI agents. This approach ensures that every response is checked before it ever reaches a young person.
Inside the AI Starter Kit prototype
1. How messages are interpreted
In the prototype, messages enter through a secured system where identifiable information is removed before processing. We then explored a Classification Agent to interpret emotional tone, surface key topics like anxiety, and flag potential safety risks. This analysis informs which clinical guidelines should apply to the response.
2. How responses are formed
Once the intent is understood, the Chat Agent generates a response based on the following:
The latest message
The conversation history (if there is one)
Kids Help Phone approved guidelines: tone, clinical rules, and escalation protocols
Using Cohere’s Command-A, the model handles the flow of the conversation while staying strictly within defined safety boundaries.
Only the essentials are passed to the model, which keeps the system fast, efficient, and cost-effective.
3. A review layer for safety and tone
Before a message is delivered, a Judge Agent reviews the response. If the “judge” flags a concern regarding tone or accuracy, the message is revised. If it fails a second time, the system pauses and encourages the youth to connect with a human counselor.
4. How the system routes to human support
If a user expresses distress, suicidal thoughts, panic, or anything else that needs urgent care, the system would follow predefined safety protocols from Kids Help Phone.
Rather than trying to solve the situation itself, the AI recommends human support. If the young person declines, the system switches to safety planning while continuing to encourage connection.
Transparency and Privacy
The prototype was designed with logging and auditability in mind. Reviewers and prompt engineers track model performance against safety and satisfaction metrics, which would allow the system to improve over time.
No identifiable or protected youth data was used in the development or testing of this prototype. To further protect privacy, each conversation is designed to automatically expire after 15 minutes. This ensures the AI retains no ongoing memory, preventing the formation of long-term “AI relationships” and protecting user anonymity. Through a secure admin panel, Kids Help Phone staff can monitor flagged messages and adjust prompts in real-time to ensure the highest standards of care.
Shifts from conventional chatbot design
Every decision, from the tone of voice to the escalation rules, was shaped in collaboration with clinicians, counselors, and youth.
Toolkit highlights:
Explainable AI: Every model decision can be tested and explained.
Dynamic Guidelines: Different topics trigger different clinical rules.
Privacy First: Conversations are not stored long-term for protection.
Thinking beyond a single use case
While built with Kids Help Phone, this architecture is a prototype for any context where trust is essential, including healthcare, finance, and education.
The system was built in modular form, with components intended to be flexible and open to iteration rather than fixed implementation.
What this work surfaced about trust
Youth are increasingly turning to AI for answers. This shift raises a critical question: how can the systems they encounter might be designed more responsibly?
The AI Starter Kit begins to explore what more thoughtful AI interactions with youth could look like in practice. For Kids Help Phone, this project served as a strategic space to reflect on and experiment with new approaches to digital support. This collaboration offered early signals around:
A possible framework for responsible AI design
A controlled environment to test emerging ideas safely
Ways to translate experimentation into future product direction
It is important to state that this work represents the development and testing of a prototype. Rather than a finished solution, this work sits in an ongoing exploration of how human expertise, design thinking, and layered safeguards might come together to build products that people can trust. As Kids Help Phone continues to explore the potential of these tools, the goals and parameters within this AI Starter Kit will continue to evolve.
