Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
The ability to build intelligent AI experiences on, and with, Windows is developing rapidly. Windows Copilot Runtime offers AI-backed features and APIs on Copilot+ PCs. These features are in active development and run locally in the background at all times. Learn more about Windows Copilot Runtime.
Windows Copilot Runtime includes several components that can enable unique AI experiences:
- Windows Copilot Runtime APIs: you can use built-in Windows AI APIs to integrate AI services for users on Copilot+ PCs. A few of these ready-to-use AI features that you can tap into from your Windows app include:
- Phi Silica: a local, ready-to-use language model.
- AI Imaging: use AI to scale and sharpen an image (Image Super Resolution), generate text that describes an image (Image Description), identify objects within an image (Image Segmentation), and remove objects from an image (Object Erase).
- Text Recognition: use AI to detect and extract text within images and convert it into machine readable character streams
- Windows ML: a way for you to build and bring your own models using machine learning on Windows
- Dev Tools: tooling such as Visual Studio AI Toolkit and AI Dev Gallery that will help you be successful building AI capabilities
How can you use AI in your Windows app?
A few ways that Windows apps can leverage Machine Learning (ML) models to enhance their functionality and user experience with AI, include:
- Apps can use Generative AI models to understand complex topics to summarize, rewrite, report on, or expand.
- Apps can use models that transform free-form content into a structured format that your app can understand.
- Apps can use Semantic Search models that allow users to search for content by meaning and quickly find related content.
- Apps can use natural language processing models to reason over complex natural language requirements, and plan and execute actions to accomplish the user's ask.
- Apps can use image manipulation models to intelligently modify images, erase or add subjects, upscale, or generate new content.
- Apps can use predictive diagnostic models to help identify and predict issues and help guide the user or do it for them.
Using Windows Copilot Runtime APIs versus bringing your own models
Use Windows Copilot Runtime APIs
When a local AI model is the right solution, you can use Windows Copilot Runtime APIs to integrate AI services for users on Copilot+ PCs. These APIs are built-in on your PC and enable unique AI-powered features with relatively little overhead.
Train your own model
If you have the ability to train your own model using your own private data with platforms like TensorFlow or PyTorch. You can integrate that custom model into your Windows application by running it locally on the device hardware using ONNX Runtime and AI Toolkit for Visual Studio Code.
AI Toolkit for Visual Studio Code is a VS Code Extension that enables you to download and run AI models locally, including access to hardware-acceleration for better performance and scale through DirectML. The AI Tookit can also help you with:
- Testing models in an intuitive playground or in your application with a REST API.
- Fine-tuning your AI model, both locally or in the cloud (on a virtual machine) to create new skills, improve reliability of responses, set the tone and format of the response.
- Fine-tuning popular small-language models (SLMs), like Phi-3 and Mistral.
- Deploy your AI feature either to the cloud or with an application that runs on a device.
- Leverage hardware acceleration for better performance with AI features using DirectML. DirectML is a low-level API that enables your Windows device hardware to accelerate the performance of ML models using the device GPU or NPU. Pairing DirectML with the ONNX Runtime is typically the most straightforward way for developers to bring hardware-accelerated AI to their users at scale. Learn more: DirectML Overview.
You may also want to look into these model fine-tuning concepts to adjust a pre-trained model to better fit your data.
Using Cloud AI Models
If using local AI features isn't the right path for you, using Cloud AI models and resources can be a solution.
Use Responsible AI practices
Whenever you are incorporating AI features in your Windows app, we highly recommend following the Developing Responsible Generative AI Applications and Features on Windows guidance.
