Master computer vision with AI APIs: image recognition, object detection, OCR text extraction, image analysis, and content moderation. Give your applications the ability to see and understand images.
Computer vision enables applications to understand and analyze visual information - from identifying objects in photos to extracting text from documents. With AI-powered vision APIs, you can add sophisticated image understanding capabilities to your applications without needing computer vision expertise.
Modern computer vision APIs make it incredibly easy to build features like automatic image tagging, document digitization, visual search, accessibility improvements, and content moderation. These capabilities are essential for modern applications handling user-generated content, documents, and media.
In this lesson, you'll learn to use OpenAI's Vision API (GPT-4 Vision) and other computer vision services to analyze images, extract information, and build intelligent visual features for your applications.
Image recognition identifies what's in an image - objects, scenes, actions, and more. Essential for automatic tagging, search, and content organization.
What is Image Recognition?
Image recognition (image classification) identifies and categorizes the content of images:
OpenAI Vision API (GPT-4 Vision):
1import OpenAI from 'openai';
2
3const openai = new OpenAI({
4 apiKey: process.env.OPENAI_API_KEY
5});
6
7async function analyzeImage(imageUrl) {
8 const response = await openai.chat.completions.create({
9 model: "gpt-4-vision-preview",
10 messages: [{
11 role: "user",
12 content: [
13 {
14 type: "text",
15 text: "What's in this image? Describe what you see."
16 },
17 {
18 type: "image_url",
19 image_url: {
20 url: imageUrl
21 }
22 }
23 ]
24 }],
25 max_tokens: 300
26 });
27
28 return response.choices[0].message.content;
29}
30
31// Usage
32const description = await analyzeImage(
33 "https://example.com/photo.jpg"
34);
35console.log(description);
36// "This image shows a golden retriever dog playing in a park..."Using Base64 Images:
1import fs from 'fs';
2
3async function analyzeLocalImage(imagePath) {
4 // Read image and convert to base64
5 const imageBuffer = fs.readFileSync(imagePath);
6 const base64Image = imageBuffer.toString('base64');
7 const mimeType = 'image/jpeg'; // or 'image/png', etc.
8
9 const response = await openai.chat.completions.create({
10 model: "gpt-4-vision-preview",
11 messages: [{
12 role: "user",
13 content: [
14 { type: "text", text: "Describe this image in detail." },
15 {
16 type: "image_url",
17 image_url: {
18 url: `data:${mimeType};base64,${base64Image}`
19 }
20 }
21 ]
22 }],
23 max_tokens: 300
24 });
25
26 return response.choices[0].message.content;
27}Structured Image Analysis:
1async function analyzeImageStructured(imageUrl) {
2 const response = await openai.chat.completions.create({
3 model: "gpt-4-vision-preview",
4 messages: [{
5 role: "user",
6 content: [
7 {
8 type: "text",
9 text: `Analyze this image and return JSON:
10{
11 "objects": ["list of objects"],
12 "scene": "scene description",
13 "colors": ["dominant colors"],
14 "mood": "mood or atmosphere",
15 "text_visible": true/false,
16 "people_count": number,
17 "quality": "high/medium/low",
18 "tags": ["relevant tags"]
19}`
20 },
21 {
22 type: "image_url",
23 image_url: { url: imageUrl }
24 }
25 ]
26 }],
27 response_format: { type: "json_object" },
28 max_tokens: 500
29 });
30
31 return JSON.parse(response.choices[0].message.content);
32}
33
34// Usage
35const analysis = await analyzeImageStructured(imageUrl);
36console.log(analysis);
37// {
38// objects: ["laptop", "coffee cup", "notebook"],
39// scene: "modern workspace",
40// colors: ["white", "gray", "brown"],
41// mood: "professional and focused",
42// text_visible: false,
43// people_count: 0,
44// quality: "high",
45// tags: ["workspace", "technology", "productivity"]
46// }Multi-Image Analysis:
1async function compareImages(imageUrls) {
2 const imageContent = imageUrls.map(url => ({
3 type: "image_url",
4 image_url: { url }
5 }));
6
7 const response = await openai.chat.completions.create({
8 model: "gpt-4-vision-preview",
9 messages: [{
10 role: "user",
11 content: [
12 {
13 type: "text",
14 text: "Compare these images. What are the similarities and differences?"
15 },
16 ...imageContent
17 ]
18 }],
19 max_tokens: 500
20 });
21
22 return response.choices[0].message.content;
23}Use Cases:
1. E-commerce:
2. Social Media:
3. Content Management:
4. Security & Compliance:
Best Practices:
1. Image Quality:
2. Prompt Engineering:
1// Bad prompt
2"What is this?"
3
4// Good prompt
5"Describe this image in detail, including:
6- Main objects and their positions
7- Colors and lighting
8- Setting/environment
9- Any text visible
10- Overall mood or atmosphere"3. Error Handling:
1async function analyzeImageSafe(imageUrl) {
2 try {
3 return await analyzeImage(imageUrl);
4 } catch (error) {
5 if (error.code === 'invalid_image_url') {
6 return { error: "Invalid or inaccessible image URL" };
7 } else if (error.code === 'content_policy_violation') {
8 return { error: "Image violates content policy" };
9 }
10 throw error;
11 }
12}4. Cost Optimization:
1// Vision API costs by image size
2function calculateVisionCost(imagePixels, detail = "auto") {
3 // Base cost: 512x512 = $0.01275
4 // High detail adds more tokens
5 const baseCost = 0.01275;
6 const scaleFactor = imagePixels / (512 * 512);
7 return baseCost * scaleFactor;
8}OCR (Optical Character Recognition) extracts text from images - essential for document processing, receipt scanning, and accessibility.
Text Extraction with Vision API:
1async function extractText(imageUrl) {
2 const response = await openai.chat.completions.create({
3 model: "gpt-4-vision-preview",
4 messages: [{
5 role: "user",
6 content: [
7 {
8 type: "text",
9 text: "Extract all text visible in this image. Return the text exactly as it appears."
10 },
11 {
12 type: "image_url",
13 image_url: { url: imageUrl }
14 }
15 ]
16 }],
17 max_tokens: 500
18 });
19
20 return response.choices[0].message.content;
21}
22
23// Usage - receipt scanning
24const receiptText = await extractText("receipt-image-url");
25console.log(receiptText);
26// "Grocery Store
27// Date: 2024-01-15
28// Items:
29// - Milk: $3.99
30// - Bread: $2.50
31// Total: $6.49"Structured Document Processing:
1async function parseInvoice(imageUrl) {
2 const response = await openai.chat.completions.create({
3 model: "gpt-4-vision-preview",
4 messages: [{
5 role: "user",
6 content: [
7 {
8 type: "text",
9 text: `Extract invoice data and return JSON:
10{
11 "invoice_number": "string",
12 "date": "YYYY-MM-DD",
13 "vendor": "vendor name",
14 "items": [
15 {"description": "item", "quantity": number, "price": number}
16 ],
17 "subtotal": number,
18 "tax": number,
19 "total": number,
20 "currency": "USD/EUR/etc"
21}`
22 },
23 {
24 type: "image_url",
25 image_url: { url: imageUrl }
26 }
27 ]
28 }],
29 response_format: { type: "json_object" },
30 max_tokens: 1000
31 });
32
33 return JSON.parse(response.choices[0].message.content);
34}
35
36// Usage
37const invoice = await parseInvoice(invoiceImageUrl);
38console.log(invoice);
39// {
40// invoice_number: "INV-2024-001",
41// date: "2024-01-15",
42// vendor: "ABC Supplies Inc.",
43// items: [
44// { description: "Office Chair", quantity: 2, price: 150.00 },
45// { description: "Desk Lamp", quantity: 3, price: 45.00 }
46// ],
47// subtotal: 435.00,
48// tax: 43.50,
49// total: 478.50,
50// currency: "USD"
51// }Business Card Parser:
1async function parseBusinessCard(imageUrl) {
2 const response = await openai.chat.completions.create({
3 model: "gpt-4-vision-preview",
4 messages: [{
5 role: "user",
6 content: [
7 {
8 type: "text",
9 text: `Extract business card information as JSON:
10{
11 "name": "full name",
12 "title": "job title",
13 "company": "company name",
14 "email": "email address",
15 "phone": "phone number",
16 "address": "physical address",
17 "website": "website URL"
18}`
19 },
20 {
21 type: "image_url",
22 image_url: { url: imageUrl }
23 }
24 ]
25 }],
26 response_format: { type: "json_object" },
27 max_tokens: 500
28 });
29
30 return JSON.parse(response.choices[0].message.content);
31}Form Processing:
1async function processForm(imageUrl) {
2 const response = await openai.chat.completions.create({
3 model: "gpt-4-vision-preview",
4 messages: [{
5 role: "user",
6 content: [
7 {
8 type: "text",
9 text: `Extract form field values as JSON. Include:
10- Field labels
11- Field values (or "empty" if blank)
12- Checkboxes (checked/unchecked)
13- Signatures (present/absent)`
14 },
15 {
16 type: "image_url",
17 image_url: { url: imageUrl }
18 }
19 ]
20 }],
21 response_format: { type: "json_object" },
22 max_tokens: 1000
23 });
24
25 return JSON.parse(response.choices[0].message.content);
26}Table Extraction:
1async function extractTable(imageUrl) {
2 const response = await openai.chat.completions.create({
3 model: "gpt-4-vision-preview",
4 messages: [{
5 role: "user",
6 content: [
7 {
8 type: "text",
9 text: `Extract the table from this image as JSON array:
10{
11 "headers": ["column names"],
12 "rows": [
13 ["row 1 values"],
14 ["row 2 values"]
15 ]
16}`
17 },
18 {
19 type: "image_url",
20 image_url: { url: imageUrl }
21 }
22 ]
23 }],
24 response_format: { type: "json_object" },
25 max_tokens: 1500
26 });
27
28 return JSON.parse(response.choices[0].message.content);
29}Use Cases:
1. Document Management:
2. Accessibility:
3. Data Entry:
4. Archival:
Best Practices:
1. Image Preprocessing:
2. Validation:
1function validateExtractedData(data) {
2 // Validate email format
3 if (data.email && !isValidEmail(data.email)) {
4 console.warn("Invalid email format");
5 }
6
7 // Validate phone number
8 if (data.phone && !isValidPhone(data.phone)) {
9 console.warn("Invalid phone format");
10 }
11
12 // Validate dates
13 if (data.date && !isValidDate(data.date)) {
14 console.warn("Invalid date format");
15 }
16
17 return data;
18}3. Error Handling:
1async function extractTextSafe(imageUrl) {
2 try {
3 const text = await extractText(imageUrl);
4
5 if (!text || text.trim().length === 0) {
6 return { error: "No text found in image" };
7 }
8
9 return { text, success: true };
10 } catch (error) {
11 return {
12 error: error.message,
13 success: false
14 };
15 }
16}Object detection identifies and locates specific objects within images, while image captioning generates natural language descriptions of images.
Object Detection:
1async function detectObjects(imageUrl) {
2 const response = await openai.chat.completions.create({
3 model: "gpt-4-vision-preview",
4 messages: [{
5 role: "user",
6 content: [
7 {
8 type: "text",
9 text: `Detect and list all objects in this image as JSON:
10{
11 "objects": [
12 {
13 "name": "object name",
14 "confidence": 0.0-1.0,
15 "position": "location description",
16 "count": number
17 }
18 ],
19 "total_objects": number
20}`
21 },
22 {
23 type: "image_url",
24 image_url: { url: imageUrl }
25 }
26 ]
27 }],
28 response_format: { type: "json_object" },
29 max_tokens: 500
30 });
31
32 return JSON.parse(response.choices[0].message.content);
33}
34
35// Usage
36const detection = await detectObjects(imageUrl);
37console.log(detection);
38// {
39// objects: [
40// { name: "person", confidence: 0.95, position: "center", count: 2 },
41// { name: "laptop", confidence: 0.92, position: "left side", count: 1 },
42// { name: "coffee cup", confidence: 0.88, position: "right side", count: 1 }
43// ],
44// total_objects: 4
45// }Image Captioning:
1async function generateCaption(imageUrl, style = "descriptive") {
2 const stylePrompts = {
3 descriptive: "Generate a detailed, descriptive caption",
4 concise: "Generate a brief, one-sentence caption",
5 seo: "Generate an SEO-friendly caption with relevant keywords",
6 accessible: "Generate an accessibility-focused alt text description",
7 creative: "Generate a creative, engaging caption"
8 };
9
10 const response = await openai.chat.completions.create({
11 model: "gpt-4-vision-preview",
12 messages: [{
13 role: "user",
14 content: [
15 {
16 type: "text",
17 text: stylePrompts[style] || stylePrompts.descriptive
18 },
19 {
20 type: "image_url",
21 image_url: { url: imageUrl }
22 }
23 ]
24 }],
25 max_tokens: 200
26 });
27
28 return response.choices[0].message.content;
29}
30
31// Usage
32const descriptive = await generateCaption(imageUrl, "descriptive");
33// "A modern workspace featuring a silver laptop, wireless mouse, and coffee mug on a wooden desk near a window with natural lighting"
34
35const seo = await generateCaption(imageUrl, "seo");
36// "Modern home office workspace with laptop and coffee - remote work setup"
37
38const accessible = await generateCaption(imageUrl, "accessible");
39// "Photo of a desk workspace containing a laptop computer, computer mouse, and coffee mug"Scene Understanding:
1async function analyzeScene(imageUrl) {
2 const response = await openai.chat.completions.create({
3 model: "gpt-4-vision-preview",
4 messages: [{
5 role: "user",
6 content: [
7 {
8 type: "text",
9 text: `Analyze the scene and return JSON:
10{
11 "scene_type": "indoor/outdoor/etc",
12 "setting": "specific location type",
13 "time_of_day": "morning/afternoon/evening/night",
14 "weather": "sunny/cloudy/rainy/etc (if visible)",
15 "activity": "what's happening",
16 "mood": "atmosphere or feeling",
17 "style": "photography style or aesthetic"
18}`
19 },
20 {
21 type: "image_url",
22 image_url: { url: imageUrl }
23 }
24 ]
25 }],
26 response_format: { type: "json_object" },
27 max_tokens: 300
28 });
29
30 return JSON.parse(response.choices[0].message.content);
31}Face Detection (Without Identification):
1async function detectFaces(imageUrl) {
2 const response = await openai.chat.completions.create({
3 model: "gpt-4-vision-preview",
4 messages: [{
5 role: "user",
6 content: [
7 {
8 type: "text",
9 text: `Detect faces in image and return JSON (no identification):
10{
11 "faces_detected": number,
12 "faces": [
13 {
14 "position": "location in image",
15 "estimated_age_range": "age range",
16 "gender_presentation": "presentation",
17 "emotion": "apparent emotion",
18 "accessories": ["glasses", "hat", etc]
19 }
20 ]
21}`
22 },
23 {
24 type: "image_url",
25 image_url: { url: imageUrl }
26 }
27 ]
28 }],
29 response_format: { type: "json_object" },
30 max_tokens: 500
31 });
32
33 return JSON.parse(response.choices[0].message.content);
34}Product Detection (E-commerce):
1async function analyzeProduct(imageUrl) {
2 const response = await openai.chat.completions.create({
3 model: "gpt-4-vision-preview",
4 messages: [{
5 role: "user",
6 content: [
7 {
8 type: "text",
9 text: `Analyze this product image and return JSON:
10{
11 "product_type": "category",
12 "brand": "brand if visible",
13 "color": "primary color",
14 "style": "style description",
15 "condition": "new/used/damaged",
16 "features": ["notable features"],
17 "suggested_price_range": "price estimate",
18 "similar_to": ["comparable products"]
19}`
20 },
21 {
22 type: "image_url",
23 image_url: { url: imageUrl }
24 }
25 ]
26 }],
27 response_format: { type: "json_object" },
28 max_tokens: 400
29 });
30
31 return JSON.parse(response.choices[0].message.content);
32}Use Cases:
1. Content Creation:
2. E-commerce:
3. Security:
4. Accessibility:
Content moderation uses AI to detect inappropriate, unsafe, or policy-violating images - essential for user-generated content platforms.
Content Moderation with Vision API:
1async function moderateImage(imageUrl) {
2 const response = await openai.chat.completions.create({
3 model: "gpt-4-vision-preview",
4 messages: [{
5 role: "user",
6 content: [
7 {
8 type: "text",
9 text: `Analyze this image for content safety and return JSON:
10{
11 "safe": true/false,
12 "issues": [
13 {
14 "category": "violence/adult/hate/etc",
15 "severity": "low/medium/high",
16 "confidence": 0.0-1.0,
17 "description": "what was detected"
18 }
19 ],
20 "action": "approve/review/reject",
21 "explanation": "brief explanation"
22}`
23 },
24 {
25 type: "image_url",
26 image_url: { url: imageUrl }
27 }
28 ]
29 }],
30 response_format: { type: "json_object" },
31 max_tokens: 400
32 });
33
34 return JSON.parse(response.choices[0].message.content);
35}
36
37// Usage
38const moderation = await moderateImage(imageUrl);
39
40if (!moderation.safe) {
41 console.log("Content flagged:", moderation.issues);
42 // Take appropriate action
43}Brand Safety Check:
1async function checkBrandSafety(imageUrl) {
2 const response = await openai.chat.completions.create({
3 model: "gpt-4-vision-preview",
4 messages: [{
5 role: "user",
6 content: [
7 {
8 type: "text",
9 text: `Check brand safety for advertising and return JSON:
10{
11 "brand_safe": true/false,
12 "concerns": [
13 {
14 "type": "controversial/sensitive/inappropriate",
15 "description": "specific concern",
16 "severity": "low/medium/high"
17 }
18 ],
19 "suitable_for": ["general/teen/adult audiences"],
20 "recommendations": ["action items"]
21}`
22 },
23 {
24 type: "image_url",
25 image_url: { url: imageUrl }
26 }
27 ]
28 }],
29 response_format: { type: "json_object" },
30 max_tokens: 400
31 });
32
33 return JSON.parse(response.choices[0].message.content);
34}Age-Appropriate Content Check:
1async function checkAgeAppropriate(imageUrl, targetAge) {
2 const response = await openai.chat.completions.create({
3 model: "gpt-4-vision-preview",
4 messages: [{
5 role: "user",
6 content: [
7 {
8 type: "text",
9 text: `Check if image is appropriate for age ${targetAge} and return JSON:
10{
11 "appropriate": true/false,
12 "age_rating": "G/PG/PG-13/R",
13 "concerns": ["list any issues"],
14 "explanation": "why appropriate or not"
15}`
16 },
17 {
18 type: "image_url",
19 image_url: { url: imageUrl }
20 }
21 ]
22 }],
23 response_format: { type: "json_object" },
24 max_tokens: 300
25 });
26
27 return JSON.parse(response.choices[0].message.content);
28}Quality and Authenticity Check:
1async function checkImageQuality(imageUrl) {
2 const response = await openai.chat.completions.create({
3 model: "gpt-4-vision-preview",
4 messages: [{
5 role: "user",
6 content: [
7 {
8 type: "text",
9 text: `Assess image quality and authenticity:
10{
11 "quality": "high/medium/low",
12 "issues": ["blurry", "poor lighting", etc],
13 "likely_edited": true/false,
14 "editing_indicators": ["what suggests editing"],
15 "ai_generated_likelihood": "low/medium/high",
16 "recommendation": "accept/reject/review"
17}`
18 },
19 {
20 type: "image_url",
21 image_url: { url: imageUrl }
22 }
23 ]
24 }],
25 response_format: { type: "json_object" },
26 max_tokens: 400
27 });
28
29 return JSON.parse(response.choices[0].message.content);
30}Complete Moderation Pipeline:
1class ContentModerationService {
2 async moderateImage(imageUrl) {
3 const results = {
4 url: imageUrl,
5 timestamp: new Date().toISOString(),
6 checks: {}
7 };
8
9 try {
10 // Safety check
11 results.checks.safety = await moderateImage(imageUrl);
12
13 // Only proceed if safe
14 if (results.checks.safety.safe) {
15 // Brand safety
16 results.checks.brandSafety = await checkBrandSafety(imageUrl);
17
18 // Quality check
19 results.checks.quality = await checkImageQuality(imageUrl);
20 }
21
22 // Determine final decision
23 results.decision = this.makeDecision(results.checks);
24
25 return results;
26 } catch (error) {
27 results.error = error.message;
28 results.decision = "error";
29 return results;
30 }
31 }
32
33 makeDecision(checks) {
34 if (!checks.safety?.safe) {
35 return "reject";
36 }
37
38 if (checks.brandSafety && !checks.brandSafety.brand_safe) {
39 return "review";
40 }
41
42 if (checks.quality?.quality === "low") {
43 return "review";
44 }
45
46 return "approve";
47 }
48}
49
50// Usage
51const moderator = new ContentModerationService();
52const result = await moderator.moderateImage(imageUrl);
53
54if (result.decision === "reject") {
55 console.log("Content rejected:", result.checks.safety.issues);
56} else if (result.decision === "review") {
57 console.log("Flagged for manual review");
58} else {
59 console.log("Content approved");
60}Best Practices:
1. Multi-Layer Approach:
2. Privacy Considerations:
3. False Positive Handling:
4. Performance:
A comprehensive computer vision application demonstrating image recognition, OCR, object detection, captioning, and content moderation using AI vision APIs.
1// Computer Vision with GPT-4 Vision API
2async function analyzeImage(imageUrl, analysisType) {
3 const prompts = {
4 describe: "Describe this image in detail.",
5 ocr: "Extract all text from this image.",
6 objects: "List all objects visible in this image.",
7 caption: "Generate a descriptive caption for this image.",
8 moderate: "Check if this image is safe and appropriate."
9 };
10
11 const response = await openai.chat.completions.create({
12 model: "gpt-4-vision-preview",
13 messages: [{
14 role: "user",
15 content: [
16 { type: "text", text: prompts[analysisType] },
17 { type: "image_url", image_url: { url: imageUrl } }
18 ]
19 }],
20 max_tokens: 500
21 });
22
23 return response.choices[0].message.content;
24}
25
26// Multi-function image processor
27class ImageProcessor {
28 async processImage(imageUrl) {
29 return {
30 description: await this.describe(imageUrl),
31 objects: await this.detectObjects(imageUrl),
32 text: await this.extractText(imageUrl),
33 caption: await this.generateCaption(imageUrl),
34 safety: await this.checkSafety(imageUrl)
35 };
36 }
37
38 async describe(imageUrl) {
39 return await analyzeImage(imageUrl, 'describe');
40 }
41
42 async detectObjects(imageUrl) {
43 return await analyzeImage(imageUrl, 'objects');
44 }
45
46 async extractText(imageUrl) {
47 return await analyzeImage(imageUrl, 'ocr');
48 }
49
50 async generateCaption(imageUrl) {
51 return await analyzeImage(imageUrl, 'caption');
52 }
53
54 async checkSafety(imageUrl) {
55 return await analyzeImage(imageUrl, 'moderate');
56 }
57}This computer vision suite demonstrates:
Core Capabilities:
Real-World Applications:
Excellent work mastering computer vision with AI APIs! You can now analyze images, extract text, detect objects, and moderate content. In our next lesson, "DALL-E and Image Generation," you'll learn the flip side - creating images from text descriptions, editing images with AI, and generating variations. Get ready to become an image generation expert!
Your progress is saved in this browser, on this device only.