Openai on David Lang

Multi-Modal AI: Working with Images and Text

Tue, 05 Nov 2024 00:00:00 +0000

Multi-modal models accept images and text in one request-enabling document OCR, UI screenshot analysis, and visual Q&A.

Vision API Example

const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'What error is shown in this screenshot?' },
        { type: 'image_url', image_url: { url: imageDataUrl } },
      ],
    },
  ],
});

Use Cases

Receipt parsing, diagram explanation, accessibility alt-text generation, and visual regression triage.

Claude API vs OpenAI API: Choosing the Right LLM

Wed, 14 Aug 2024 00:00:00 +0000

Anthropic’s Claude and OpenAI’s GPT families both offer strong APIs. Choosing between them depends on task, context length, cost, and compliance-not benchmark hype alone.

Strengths at a Glance

Claude - Long context windows, careful refusals, strong long-document analysis and coding reviews.

OpenAI - Broad ecosystem, function calling maturity, image and audio modalities, largest third-party integration surface.

Integration Pattern

Abstract the provider behind an interface:

interface LLMProvider {
  chat(messages: Message[]): Promise<string>;
}

Swap implementations per route (cheap model for classification, premium for generation).

Building AI Agents with LangChain and OpenAI

Sun, 28 Jul 2024 00:00:00 +0000

AI agents loop: observe, plan, act with tools, observe again. They handle multi-step tasks like research, booking, or code changes.

Tool Definition

from langchain.tools import tool

@tool
def search_docs(query: str) -> str:
    '''Search internal documentation.'''
    return vector_store.similarity_search(query, k=5)

Bind tools to the model; the model returns tool calls you execute and feed back as observations.

Control and Safety

Set max iterations. Require human approval for destructive tools. Log every step for audit.

Conclusion

Agents are powerful and unpredictable. Start with a fixed workflow (chain); graduate to agents when the task path genuinely varies per request.

Streaming AI Responses with OpenAI API in Next.js

Sat, 30 Mar 2024 00:00:00 +0000

Streaming improves chat UX by showing tokens as they are generated. Next.js Route Handlers make it straightforward to proxy streams securely.

Route Handler

// app/api/chat/route.ts
import OpenAI from 'openai';

export async function POST(req: Request) {
  const { messages } = await req.json();
  const openai = new OpenAI();

  const stream = await openai.chat.completions.create({
    model: 'gpt-4',
    messages,
    stream: true,
  });

  const encoder = new TextEncoder();
  const readable = new ReadableStream({
    async start(controller) {
      for await (const chunk of stream) {
        const text = chunk.choices[0]?.delta?.content || '';
        if (text) controller.enqueue(encoder.encode(text));
      }
      controller.close();
    },
  });

  return new Response(readable, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
}

Client Consumption

Use fetch with a reader loop or libraries like Vercel AI SDK’s useChat for React state management.

Getting Started with the OpenAI API in Node.js

Thu, 12 Jan 2023 00:00:00 +0000

The OpenAI API brought large language models to application developers through a simple HTTP interface. Node.js remains a natural fit for BFF layers that call LLMs.

Installation and First Request

npm install openai

import OpenAI from 'openai';

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

const completion = await client.chat.completions.create({
  model: 'gpt-4',
  messages: [
    { role: 'system', content: 'You are a helpful coding assistant.' },
    { role: 'user', content: 'Explain async/await in JavaScript.' },
  ],
});

console.log(completion.choices[0].message.content);

Production Considerations

Never expose API keys in frontend bundles. Proxy requests through your backend. Set max_tokens, timeouts, and retry policies. Log token usage for cost control.