AI Infrastructure & Tools9 min read

Multimodal AI

Text + images + audio + video — AI that sees, hears, and reads
scope:Foundationaldifficulty:Beginner

What Does Multimodal Mean?

A "modality" is a type of information — text is one modality, images another, audio another. Early AI models only worked with one modality — GPT-3 could only understand text.

Multimodal AI is a model that can understand and generate multiple modalities — text, images, audio, and video — all at once. Just like humans can simultaneously see, hear, read, and speak, multimodal AI can do the same.

Types of Modalities

  • Text: Understanding and generating language — answering questions, translation, summarization.
  • Image: Understanding images (Vision) and generating images (Generation).
  • Audio: Understanding speech (Speech-to-Text) and generating speech (Text-to-Speech).
  • Video: Understanding video content and generating video.
Note: The 'o' in GPT-4o stands for 'Omni': OpenAI's GPT-4o processes text, images, and audio — all three modalities in a single model. Previously, different modalities required separate models — GPT-4o unified everything.

Comparing Multimodal Capabilities

  • GPT-4o (OpenAI): Text, image, and audio input/output. Real-time voice conversations. Image generation via DALL-E.
  • Claude (Anthropic): Text and image input. Excellent at PDF and document analysis. Doesn't generate images.
  • Gemini (Google): Text, image, audio, and video — everything. Its 1 million token context window can process entire videos.

Practical Applications

  • Document analysis: Extracting information from scanned documents, charts, and graphs.
  • Accessibility: Image descriptions for visually impaired users, automatic subtitles for deaf users.
  • Content creation: Turn a blog post into illustrated social media posts, video scripts, and audio narration.
  • Education: Solving math from photos, digitizing handwritten notes.

Using the Multimodal API — Image Analysis

from openai import OpenAI
client = OpenAI()


# Analyze an image (Vision)
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What do you see in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://example.com/photo.jpg"
                    }
                }
            ]
        }
    ]
)
print(response.choices[0].message.content)


# Audio to text (Whisper)
audio_file = open("podcast.mp3", "rb")
transcript = client.audio.transcriptions.create(
    model="whisper-1",
    file=audio_file,
    language="en"
)
print(transcript.text)
Output
# Vision output:
# "The image shows a red balloon floating over a green field.
# In the background, there's a blue sky with white clouds."

# Whisper output:
# "In today's podcast, we'll be discussing
# the future of artificial intelligence..."
Challenge

Quick check

What's the most important feature of multimodal AI?

Continue reading