From the source
GitHub Copilot can now interact with desktop applications on macOS and Windows in public preview, reading accessible content and visual context, clicking controls, entering text, pressing keys, scrolling, dragging, and navigating workflows across apps.
- The feature works in Copilot CLI and the GitHub Copilot app, requires user approval before controlling an app, and can be disabled by organization-managed settings.




