Multimodal: sending images to the model

Gemini reads images, not just text. Learn how parts carry inline data, how images are billed as tokens, and how to ask questions about a picture.

Lesson 14 of 1910 min+30 XPHands-on

Every request so far has had a single text part. The parts array exists because one message can mix different kinds of content. Gemini is : the same endpoint accepts images, audio and video alongside your words.

An image is just another part

Small files go directly in the request, encoded as text, next to the question about them. Put the image first and the question second; that tends to work best.

one turn, two parts

await gemini.generateContent({
  contents: [{
    role: "user",
    parts: [
      { inlineData: { mimeType: "image/png", data: base64String } },
      { text: "What colors appear in this image?" },
    ],
  }],
});

data is raw base64 — no data:image/png;base64, prefix. Including the prefix is the single most common mistake here.
mimeType (the ) must match the actual file: image/png, image/jpeg or image/webp.
Inline data is capped at roughly 20MB per request, counting everything in it. Larger files are uploaded with the and referred to by a link instead.

Under the hood — Why base64, and why does it make files bigger?

A JSON request body is text. An image is arbitrary bytes, many of which are not valid text at all and some of which (quotes, backslashes) would break the JSON if inserted raw.

Base64 solves that by re-encoding the bytes using only 64 safe characters. It takes 3 bytes at a time and represents them as 4 characters — which is why a base64 string is roughly 33% larger than the file it encodes. A 6MB photo becomes about 8MB of request body.

That overhead is the reason the Files API exists for anything substantial: you upload the raw bytes once, then reference the result by URI instead of re-encoding and re-sending the whole thing on every request.

Images cost tokens too

A picture isn't free. Gemini converts an image into and charges you for them: a small image costs a few hundred, a large one considerably more. The usageMetadata on the response tells you exactly what you spent, and countTokens works on image parts as well.

Prompting with an image

Everything from the prompting lessons still applies, and pays off even more. "Describe this image" gets you a paragraph your code then has to pick apart. Combining an image with a responseSchema gets you { colors: ["red", "blue"], layout: "horizontal split" } — the same constrained-output trick from the structured output lesson, now applied to images.

Your exercise uses a tiny generated image so nothing depends on an external file: 32×32 pixels, red on the top half, blue on the bottom.

Key takeaways

An image is just another part of the request: base64 data plus its MIME type, with no data: prefix.
Images cost tokens too, and large images cost a lot more.
Vision models are weak at small text and exact counts, so check anything that must be exact.

Sign in to run the exercise

Reading is free. Writing code here needs an account so we have somewhere to keep your Gemini key and the +30 XP you are about to earn.

Continue with Google