visual-intelligence
starsforks
How to install visual-intelligence?
rshankras/claude-code-apple-skills/visual-intelligence
$npx -y skills add rshankras/claude-code-apple-skills --skill visual-intelligenceIntegrate your app with iOS Visual Intelligence for camera-based search and object recognition. Use when adding visual search capabilities.
SKILL.md
| 1 | # Visual Intelligence |
| 2 | |
| 3 | Integrate your app with iOS Visual Intelligence to let users find app content by pointing their camera at objects. |
| 4 | |
| 5 | ## When This Skill Activates |
| 6 | |
| 7 | - User wants camera-based search in their app |
| 8 | - User asks about visual search integration |
| 9 | - User wants to surface app content in system searches |
| 10 | - User needs to handle visual intelligence queries |
| 11 | |
| 12 | ## Overview |
| 13 | |
| 14 | Visual Intelligence lets users: |
| 15 | 1. Point camera at objects or use screenshots |
| 16 | 2. System identifies what they're looking at |
| 17 | 3. Your app provides matching content |
| 18 | 4. Results appear in system UI |
| 19 | |
| 20 | Your app implements: |
| 21 | - `IntentValueQuery` to receive search requests |
| 22 | - `AppEntity` types for searchable content |
| 23 | - Display representations for results |
| 24 | |
| 25 | ## Platform Availability (WWDC26 297) |
| 26 | |
| 27 | - Visual Intelligence runs on iOS, iPadOS, and macOS — the same entities, query, and OpenIntent code works unchanged on all three. Handle both **camera captures of physical objects** (iOS) and **screenshots of digital media** (iPad/Mac) as input. |
| 28 | - On Mac, the input pixel buffer can be **much larger** than what you'd encounter on iPhone — consider whether resizing is necessary before matching. |
| 29 | |
| 30 | ## Quick Start |
| 31 | |
| 32 | ### 1. Import Frameworks |
| 33 | |
| 34 | ```swift |
| 35 | import VisualIntelligence |
| 36 | import AppIntents |
| 37 | ``` |
| 38 | |
| 39 | ### 2. Create App Entity |
| 40 | |
| 41 | ```swift |
| 42 | struct ProductEntity: AppEntity { |
| 43 | var id: String |
| 44 | var name: String |
| 45 | var price: String |
| 46 | var imageName: String |
| 47 | |
| 48 | static var typeDisplayRepresentation: TypeDisplayRepresentation { |
| 49 | TypeDisplayRepresentation( |
| 50 | name: LocalizedStringResource("Product"), |
| 51 | numericFormat: "\(placeholder: .int) products" |
| 52 | ) |
| 53 | } |
| 54 | |
| 55 | var displayRepresentation: DisplayRepresentation { |
| 56 | DisplayRepresentation( |
| 57 | title: "\(name)", |
| 58 | subtitle: "\(price)", |
| 59 | image: .init(named: imageName) |
| 60 | ) |
| 61 | } |
| 62 | |
| 63 | // Deep link URL |
| 64 | var appLinkURL: URL? { |
| 65 | URL(string: "myapp://product/\(id)") |
| 66 | } |
| 67 | } |
| 68 | ``` |
| 69 | |
| 70 | ### 3. Create Intent Value Query |
| 71 | |
| 72 | ```swift |
| 73 | struct ProductIntentValueQuery: IntentValueQuery { |
| 74 | func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] { |
| 75 | // Search using labels |
| 76 | if !input.labels.isEmpty { |
| 77 | return await searchProducts(matching: input.labels) |
| 78 | } |
| 79 | |
| 80 | // Search using image |
| 81 | if let pixelBuffer = input.pixelBuffer { |
| 82 | return await searchProducts(from: pixelBuffer) |
| 83 | } |
| 84 | |
| 85 | return [] |
| 86 | } |
| 87 | |
| 88 | private func searchProducts(matching labels: [String]) async -> [ProductEntity] { |
| 89 | // Search your database using provided labels |
| 90 | // Return matching products |
| 91 | } |
| 92 | |
| 93 | private func searchProducts(from pixelBuffer: CVReadOnlyPixelBuffer) async -> [ProductEntity] { |
| 94 | // Use image recognition on the pixel buffer |
| 95 | // Return matching products |
| 96 | } |
| 97 | } |
| 98 | ``` |
| 99 | |
| 100 | ## SemanticContentDescriptor |
| 101 | |
| 102 | The system provides this object with information about what the user is looking at. |
| 103 | |
| 104 | ### Properties |
| 105 | |
| 106 | | Property | Type | Description | |
| 107 | |----------|------|-------------| |
| 108 | | `labels` | `[String]` | Classification labels from Visual Intelligence | |
| 109 | | `pixelBuffer` | `CVReadOnlyPixelBuffer?` | Raw image data | |
| 110 | |
| 111 | ### Usage Patterns |
| 112 | |
| 113 | **Label-based Search:** |
| 114 | ```swift |
| 115 | func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] { |
| 116 | // Labels like "shoe", "sneaker", "Nike" etc. |
| 117 | let labels = input.labels |
| 118 | |
| 119 | // Search your content using these labels |
| 120 | return products.filter { product in |
| 121 | labels.contains { label in |
| 122 | product.tags.contains(label.lowercased()) |
| 123 | } |
| 124 | } |
| 125 | } |
| 126 | ``` |
| 127 | |
| 128 | **Image-based Search:** |
| 129 | ```swift |
| 130 | func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] { |
| 131 | guard let pixelBuffer = input.pixelBuffer else { |
| 132 | return [] |
| 133 | } |
| 134 | |
| 135 | // Convert to CGImage for processing |
| 136 | let ciImage = CIImage(cvPixelBuffer: pixelBuffer) |
| 137 | let context = CIContext() |
| 138 | |
| 139 | guard let cgImage = context.createCGImage(ciImage, from: ciImage.extent) else { |
| 140 | return [] |
| 141 | } |
| 142 | |
| 143 | // Use your ML model or image matching logic |
| 144 | return await imageSearch.findMatches(for: cgImage) |
| 145 | } |
| 146 | ``` |
| 147 | |
| 148 | ## Multiple Result Types |
| 149 | |
| 150 | Use `@UnionValue` when your app has different content types. |
| 151 | |
| 152 | Rules (WWDC26 297): |
| 153 | |
| 154 | - **An app can have only ONE `IntentValueQuery` that accepts a `SemanticContentDescriptor`.** All result types must flow through that single query — a `@UnionValue` enum with one case per entity type. |
| 155 | - **Every entity type in the union needs its own `OpenIntent`** — without one, results of that type can't appear in image search. |
| 156 | - Think beyond pixel matching: an album matched by image similarity can also surface the artist's **nearby co |
People who liked this also viewed
6 picksType
- hyperframes-cliUse the HyperFrames CLI development loop: init, add, catalog, capture, lint, check, snapshot, compare, grade-compare, preview, play, present, beats, keyframes, single or batch render, publish, cloud, SkillsJul 2026
- request-refactor-planSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026
- analyze-projectRigor Analyze / Rigor Audit read-only skill for deep learning research repositories. Use when the user wants to read and understand a repository, inspect model structure and training or inference entrSkillsMay 2026
- explore-codeRigor Improve implementation leaf skill for auditable candidate implementation in deep learning research repositories. Use when the researcher explicitly authorizes exploratory work on an isolated braSkillsMay 2026
- ubiquitous-languageSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026
- scaffold-exercisesSkills for Real Engineers. Straight from my .agents directory.SkillsJul 2026
