What We Learned Building a macOS AI Agent in Swift (ScreenCaptureKit, Accessibility APIs, Async Pipelines)

We have been building Fazm for about six months - a macOS desktop agent that uses voice input to control your computer. Along the way we ran into a lot of Swift-specific challenges that we did not see documented anywhere. Here is what we learned. ScreenCaptureKit for Real-Time Screen Capture The first problem is capturing what is on the screen. We needed something fast enough for real-time use but lightweight enough to run continuously without killing battery life. ScreenCaptureKit was the answer. Apple introduced it in macOS 12.3 as a modern replacement for the older CGWindowListCreateImage approach. The key advantages for an AI agent: Per-window filtering. You can capture specific windows instead of the entire screen, which means less data to process and better privacy. Hardware-accelerated. The capture pipeline runs on the GPU, so CPU overhead stays minimal even at high frame rates. CMSampleBuffer output. You get raw pixel buffers that you can feed directly to vision models without