Google Vertex AI API: Generate Content

Non-streaming:

POST https://zenmux.ai/api/vertex-ai/v1/publishers/{provider}/models/{model}:generateContent

Streaming:

POST https://zenmux.ai/api/anthropic/v1/publishers/{provider}/models/{model}:streamGenerateContent

ZenMux supports the Google Vertex AI API via the Gen AI SDK. For detailed request parameters and response schemas, see the official Google Vertex AI documentation.

Path parameters

provider string

Model provider (e.g., google).

model string

Model name (e.g., gemini-2.5-pro).

Request headers

Authorization string

Bearer token authentication.

Content-Type string

Default: application/json

Request body

The request body is JSON.

contents array<Content>

Current conversation content (single turn / multi-turn history + the current input).

role string

The content producer (defaults to user).

• user: Indicates the message is sent by a human, typically user-generated.
• model: Indicates the message is generated by the model.

parts Part[]

At least 1 Part.

part

• text string

  Text prompt or code snippet.

• inlineData Blob

  Inline data as raw bytes.

  • mimeType string

    The media type of the file specified in the data or fileUri field. Acceptable values include:
    

    • application/pdf
    • audio/mpeg
    • audio/mp3
    • audio/wav
    • image/png
    • image/jpeg
    • image/webp
    • text/plain
    • video/mov
    • video/mpeg
    • video/mp4
    • video/mpg
    • video/avi
    • video/wmv
    • video/mpegps
    • video/flv
    • video/x-ms-wmv

  • data bytes

    Inline data as raw bytes.

• fileData FileData

  Data stored in a file.

  • mimeType string

  • fileUri string

    The URI or URL of the file to include in the prompt.

• functionCall FunctionCall

  Contains a string representing the FunctionDeclaration.name field, plus a structured JSON object with all parameters of the function call predicted by the model.

  • name string

    The name of the function to call.

  • args Record<string,any>

    Function parameters and values represented as a JSON object.

• functionResponse FunctionResponse

  The output of a FunctionCall. Contains a string representing the FunctionDeclaration.name field and a structured JSON object containing any output of the function call. It is used as context for the model.

  • name string

    The name of the function to call.

  • response Record<string,any>

    Function response represented as a JSON object.

• videoMetadata VideoMetadata

  For video inputs: the start and end offsets (duration format), and the frame rate.

  • startOffset number

    Video start offset (duration format).

  • endOffset number

    Video end offset (duration format).

  • fps number

    Video frame rate.

• mediaResolution enum

  Controls how input media is processed. If specified, this configuration overrides the mediaResolution setting in generationConfig. LOW reduces the number of tokens per image/video, which may cause loss of detail, but allows longer videos to be included in the context. Supported values: HIGH, MEDIUM, LOW.

cachedContent string

Cached content resource name (used as context):

• projects/{project}/locations/{location}/cachedContents/{cachedContent}

tools array<Tool>

List of tools (e.g., function calling, retrieval, search, code execution, etc.).

toolConfig ToolConfig

Tool configuration (shared across all tools in this request).

safetySettings array<SafetySetting>

Per-request safety settings (applies to candidates).

category string

The safety category to configure a threshold for.

• HARM_CATEGORY_UNSPECIFIED: Unspecified harm category.
• HARM_CATEGORY_HATE_SPEECH: Harm category is hate speech.
• HARM_CATEGORY_HARASSMENT: Harm category is harassment.
• HARM_CATEGORY_SEXUALLY_EXPLICIT: Harm category is sexually explicit content.
• HARM_CATEGORY_DANGEROUS_CONTENT: Harm category is dangerous content.

threshold string

Threshold for blocking responses that fall into the specified safety category based on probability.

• OFF: Turn off safety settings when all categories are disabled.
• BLOCK_NONE: Block nothing.
• BLOCK_ONLY_HIGH: Block only high-threshold content (i.e., block less).
• BLOCK_MEDIUM_AND_ABOVE: Block medium-threshold and above content.
• BLOCK_LOW_AND_ABOVE: Block low-threshold and above content (i.e., block more).
• HARM_BLOCK_THRESHOLD_UNSPECIFIED: Unspecified harm block threshold.

method string

Specifies whether the threshold applies to probability scores or severity scores. If not specified, the threshold applies to probability scores.

• HARM_BLOCK_METHOD_UNSPECIFIED: Unspecified harm block method.
• SEVERITY: Harm block method uses both probability and severity scores.
• PROBABILITY: Harm block method uses probability scores.

generationConfig GenerationConfig

Generation parameters (controls sampling, length, stop conditions, structured output, logprobs, audio timestamps, thinking, media processing quality, etc.).

temperature number

Controls randomness/diversity in the output. Lower values are more deterministic and “test-like”; higher values are more creative/diverse. 0 tends to always pick the highest-probability token, so it is closer to deterministic (but small variations can still occur). If replies are too templated/too short, try increasing it; if you see anomalies such as “runaway generation,” try raising it to at least 0.1. Ranges/defaults vary by model (for example, some Gemini Flash models commonly use 0.0~2.0 with a default of 1.0).

topP number

Nucleus sampling threshold: the model samples only from the smallest set of tokens whose cumulative probability reaches topP. Lower values are more conservative/less random; higher values are more diverse. Range 0.0~1.0 (model defaults may vary). In general, it is recommended to primarily tune either temperature or topP, not both significantly.

topK integer

Top-K sampling threshold: the model samples only from the topK tokens with the highest probability. For example, topK=40 means each step selects the next token only from the top 40 candidates. Smaller values are more conservative; larger values are more diverse. Model defaults may vary.

candidateCount integer

Number of returned candidates (response variations). Output tokens for all candidates are billed (inputs are typically billed once). Multi-candidate is usually a preview capability and typically only supported by generateContent (not streamGenerateContent), and different models constrain the range/max (e.g., some support 1~8).

maxOutputTokens integer

Maximum number of output tokens to limit response length; tokens can be roughly understood as ~4 characters in English. Smaller values produce shorter output; larger values allow longer output.

stopSequences array<string>

Stop sequence list: generation stops immediately when any stop sequence is encountered in the output, and is truncated at the first occurrence; case-sensitive. The list can contain up to 5 elements.

presencePenalty number

Presence penalty: penalizes tokens that have already appeared in the “generated text,” increasing the probability of generating new/different content. Range -2.0 ~ <2.0.

frequencyPenalty number

Frequency penalty: penalizes tokens that are repeated, reducing the probability of repetitive output. Range -2.0 ~ <2.0.

seed integer

Random seed: with a fixed seed, the model will “try” to return the same result for repeated requests, but does not guarantee full determinism; changes in model version or parameters (e.g., temperature) may also cause differences. If omitted, a random seed is used by default.

responseMimeType string

Specifies the MIME type of candidate outputs. Common supported values:

• text/plain (default): plain text output
• application/json: JSON output (for structured output / JSON mode)
• text/x.enum: for classification tasks, outputs enum values defined by responseSchema

Note: If you want to constrain structured output using responseSchema, you must set responseMimeType to a value other than text/plain (e.g., application/json).

responseSchema Schema

Schema for structured output: constrains candidate text to conform to this schema (for “controlled generation output / JSON Schema” scenarios). When using this field, you must set responseMimeType to a supported non-text/plain type (e.g., application/json).

Schema object fields

• type enum (Type)

  Data type. Supported values:

  • STRING: String type; supports constraints such as enum, format (e.g., date-time, email, byte), minLength, maxLength, pattern, etc.
  • INTEGER: Integer type; supports format (e.g., int32, int64), minimum, maximum, etc.
  • NUMBER: Floating-point number type; supports format (e.g., float, double), minimum, maximum, etc.
  • BOOLEAN: Boolean type.
  • ARRAY: Array type; supports items, minItems, maxItems, etc.
  • OBJECT: Object type; supports properties, required, propertyOrdering, nullable, etc.

• format string

  Additional format info. For NUMBER: float, double; for INTEGER: int32, int64; for STRING: email, byte, date, date-time, password, etc.

• description string

  Text description of the property/field to help the model understand what to generate.

• enum array<string>

  List of allowed enum values; the model can only choose one. Typically used with type: STRING.

• items Schema

  When type is ARRAY, specifies the schema for array elements.

• properties map<string, Schema>

  When type is OBJECT, defines each property and its schema.

• required array<string>

  List of required properties in the object.

• propertyOrdering array<string>

  Specifies the output order of object properties.

• nullable boolean

  Whether null is allowed.

• minimum number

  When type is INTEGER or NUMBER, the minimum allowed value.

• maximum number

  When type is INTEGER or NUMBER, the maximum allowed value.

• minItems integer

  When type is ARRAY, the minimum number of elements.

• maxItems integer

  When type is ARRAY, the maximum number of elements.

• minLength integer

  When type is STRING, the minimum string length.

• maxLength integer

  When type is STRING, the maximum string length.

• pattern string

  When type is STRING, a regex constraint. Example: "^[\\w\\s,.-]+$".

• title string

  Title for the schema.

• anyOf array<Schema>

  Union/conditional types: the value must satisfy at least one schema in anyOf. oneOf is also interpreted with the semantics of anyOf.

Note: Excessive schema complexity (very long property names, too many enum values, overly deep nesting, etc.) may cause InvalidArgument: 400 errors. Keep schemas as simple as possible. Circular references are not supported (only limited expansion is allowed in non-required properties).

responseJsonSchema object

A JSON Schema alternative to responseSchema. When this field is set, you must omit responseSchema, and set responseMimeType to application/json. Accepts standard JSON Schema syntax directly.

responseLogprobs boolean

Whether to return log probabilities for output tokens. When set to true, the response includes per-token log probability details in logprobsResult. You must enable this parameter before using the logprobs field.

logprobs integer

Returns log probabilities for the top candidate tokens at each generation step. Range 1~20. Requires responseLogprobs=true to use this field, and the token selected by the model is not necessarily the top candidate token.

responseModalities array<enum>

Specifies which modalities to return in the response. Only some Gemini models support multimodal output. If not set, defaults to text only. Supported values:

• TEXT: Text output (default).
• IMAGE: Image output; when used, must also include TEXT, i.e., ["TEXT", "IMAGE"]. Only supported by some models (e.g., gemini-2.5-flash-image, gemini-3-pro-image-preview).
• AUDIO: Audio output; mainly for Live API (real-time streaming) scenarios.

audioTimestamp boolean

Audio timestamp understanding: timestamp understanding for audio-only files (preview). Only supported by some models (e.g., some Gemini Flash models).

thinkingConfig ThinkingConfig

“Thinking” (internal reasoning) configuration for Gemini 2.5 and later. Setting this field on models that do not support thinking will return an error.

• thinkingBudget integer

  Token budget for thinking (applies to Gemini 2.5). Model-specific ranges:

  • Gemini 2.5 Pro: 128 ~ 32768 (thinking cannot be disabled)
  • Gemini 2.5 Flash: 0 ~ 24576 (0 = disable thinking)
  • Gemini 2.5 Flash Lite: 512 ~ 24576

  Set to -1 for dynamic thinking (the model adjusts budget based on request complexity). If omitted, the model controls it automatically.

  Note: Cannot be used together with thinkingLevel. thinkingBudget applies only to Gemini 2.5.

• thinkingLevel enum (ThinkingLevel)

  Controls internal reasoning intensity (recommended for Gemini 3). Supported values:

  • THINKING_LEVEL_UNSPECIFIED: Unspecified; use the model’s default dynamic behavior.
  • MINIMAL: Use as few thinking tokens as possible (Gemini 3 Flash only).
  • LOW: Low reasoning intensity; suitable for simple tasks.
  • MEDIUM: Medium reasoning intensity (Gemini 3 Flash only).
  • HIGH: High reasoning intensity (Gemini 3 default); suitable for complex tasks like math, code, and logical analysis.

  Note: Cannot be used together with thinkingBudget. thinkingLevel applies only to Gemini 3.

• includeThoughts boolean

  Whether to return a summary of the thinking process in the response. When set to true, a thought summary is returned when available. This is best-effort: even if enabled, thoughts are not guaranteed to be returned.

mediaResolution enum (MediaResolution)

Controls the token resolution when processing input media (images/videos/PDFs), balancing response quality and token usage. Higher resolution lets the model perceive more detail but consumes more tokens. Supported values:

• MEDIA_RESOLUTION_UNSPECIFIED: Unspecified; use the model default (token counts differ across model generations).
• MEDIA_RESOLUTION_LOW: Low resolution; fewer tokens, faster, cheaper.
• MEDIA_RESOLUTION_MEDIUM: Medium resolution; balance between quality and cost.
• MEDIA_RESOLUTION_HIGH: High resolution; more tokens, finer detail.
• MEDIA_RESOLUTION_ULTRA_HIGH: Ultra-high resolution (Gemini 3 only).

Gemini 3 reference image token counts: ULTRA_HIGH ≈ 2240, HIGH ≈ 1120, MEDIUM ≈ 560.

speechConfig SpeechConfig

Speech generation configuration; used when responseModalities includes AUDIO.

• voiceConfig VoiceConfig

  Single-voice configuration (mutually exclusive with multiSpeakerVoiceConfig).

  • prebuiltVoiceConfig PrebuiltVoiceConfig

    Prebuilt voice configuration.

    • voiceName string

      Prebuilt voice name (e.g., Kore, Puck, Charon, etc.).

  • replicatedVoiceConfig ReplicatedVoiceConfig

    Replicated voice configuration (clone a voice from an audio sample).

    • mimeType string

      MIME type of the voice sample. Currently only audio/wav is supported (16-bit signed little-endian, 24kHz sample rate).

    • voiceSample string(bytes)

      Custom voice sample (base64-encoded).

• multiSpeakerVoiceConfig MultiSpeakerVoiceConfig

  Multi-speaker voice configuration (mutually exclusive with voiceConfig); for multi-role TTS scenarios.

  • speakerVoiceConfigs array<SpeakerVoiceConfig>

    List of voice configurations for each speaker.

    • speaker string: Speaker name.
    • voiceConfig VoiceConfig: Voice configuration used by this speaker (same structure as above).

• languageCode string

  Language code for speech output (e.g., en-US).

routingConfig RoutingConfig

Routing configuration (Vertex AI): routes requests to a specific model. autoMode and manualMode are mutually exclusive.

Note: This field is deprecated. Google recommends using modelConfig instead.

• autoMode AutoRoutingMode

  Auto routing: routing is determined by a pretrained routing model plus user preferences.

  • modelRoutingPreference enum (ModelRoutingPreference)

    Routing preference. Supported values include BALANCED, etc.

• manualMode ManualRoutingMode

  Manual routing: explicitly specify the target model.

  • modelName string

    Target model name (e.g., gemini-1.5-pro-001).

imageConfig ImageConfig

Image generation configuration; used when responseModalities includes IMAGE.

• aspectRatio string

  Aspect ratio of generated images. Supported values: 1:1, 2:3, 3:2, 3:4, 4:3, 9:16, 16:9, 21:9; some models also support 4:5, 5:4.

• imageSize string

  Size of generated images. Supported values: 1K, 2K, 4K. Default is 1K.

• outputCompressionQuality number

  Output compression quality for generated images (applies only to image/jpeg).

• outputMimeType string

  MIME type of generated images.

enableAffectiveDialog boolean

Whether to enable affective dialogue: when enabled, the model detects the user’s emotion and adjusts the response style accordingly.

systemInstruction Content

System instruction (guides the model’s overall behavior; recommended to use only text in parts, with each part as a separate paragraph).

Response (non-streaming)

The official response structure is as follows:

{
  "candidates": [
    {
      "content": {
        "parts": [
          {
            "text": string
          }
        ]
      },
      "finishReason": enum (FinishReason),
      "safetyRatings": [
        {
          "category": enum (HarmCategory),
          "probability": enum (HarmProbability),
          "blocked": boolean
        }
      ],
      "citationMetadata": {
        "citations": [
          {
            "startIndex": integer,
            "endIndex": integer,
            "uri": string,
            "title": string,
            "license": string,
            "publicationDate": {
              "year": integer,
              "month": integer,
              "day": integer
            }
          }
        ]
      },
      "avgLogprobs": double,
      "logprobsResult": {
        "topCandidates": [
          {
            "candidates": [
              {
                "token": string,
                "logProbability": float
              }
            ]
          }
        ],
        "chosenCandidates": [
          {
            "token": string,
            "logProbability": float
          }
        ]
      }
    }
  ],
  "usageMetadata": {
    "promptTokenCount": integer,
    "candidatesTokenCount": integer,
    "totalTokenCount": integer

    // (possible extended stats)
    // "cachedContentTokenCount": integer,
    // "thoughtsTokenCount": integer,
    // "toolUsePromptTokenCount": integer,
    // "promptTokensDetails": [...],
    // "candidatesTokensDetails": [...],
    // "toolUsePromptTokensDetails": [...]
  },
  "modelVersion": string,
  "createTime": string,
  "responseId": string
}

candidates array<Candidate>

List of candidate results returned for this generation.

Candidate.content object

Candidate content.

• content.parts array
  Array of content parts.
  • parts[].text string
    Generated text.

Candidate.finishReason enum (FinishReason)

Why the model stopped generating tokens; if empty, it indicates generation has not stopped yet.

Common values (as listed officially):

• FINISH_REASON_STOP: Natural stopping point or matched a stop sequence
• FINISH_REASON_MAX_TOKENS: Reached the requested max token limit
• FINISH_REASON_SAFETY: Stopped for safety reasons (if output is blocked by the filter, Candidate.content is empty)
• FINISH_REASON_RECITATION: Stopped due to flagged unauthorized recitation
• FINISH_REASON_BLOCKLIST: Contains blocked terms
• FINISH_REASON_PROHIBITED_CONTENT: Contains prohibited content (e.g., CSAM)
• FINISH_REASON_IMAGE_PROHIBITED_CONTENT: An image in the prompt contains prohibited content
• FINISH_REASON_NO_IMAGE: The prompt should include an image but none was provided
• FINISH_REASON_SPII: Contains sensitive personally identifiable information (SPII)
• FINISH_REASON_MALFORMED_FUNCTION_CALL: Function call is malformed and cannot be parsed
• FINISH_REASON_OTHER: Other reasons
• FINISH_REASON_UNSPECIFIED: Unspecified

Candidate.safetyRatings array<SafetyRating>

Array of safety ratings.

• safetyRatings[].category enum (HarmCategory)
  Safety category (e.g., HARM_CATEGORY_SEXUALLY_EXPLICIT, HARM_CATEGORY_HATE_SPEECH, HARM_CATEGORY_HARASSMENT, HARM_CATEGORY_DANGEROUS_CONTENT).
• safetyRatings[].probability enum (HarmProbability)
  Harm probability level: NEGLIGIBLE / LOW / MEDIUM / HIGH, etc.
• safetyRatings[].blocked boolean
  Indicates whether the model input or output was blocked.

Candidate.citationMetadata object

Citation information (when the output includes citations).

• citationMetadata.citations array<Citation>
  • citations[].startIndex integer
    Start position of the citation in content, measured in bytes of the UTF-8 response.
  • citations[].endIndex integer
    End position of the citation in content, also measured in bytes.
  • citations[].uri string
    Source URL (official docs describe it as url/URL; in the example structure, the field name is uri).
  • citations[].title string
    Source title.
  • citations[].license string
    Associated license.
  • citations[].publicationDate object
    Publication date (valid formats: YYYY / YYYY-MM / YYYY-MM-DD).
    • publicationDate.year integer
    • publicationDate.month integer
    • publicationDate.day integer

Candidate.avgLogprobs double

Average log probability for the candidate.

Candidate.logprobsResult object

Returns top candidate tokens (topCandidates) and the actually selected tokens (chosenCandidates) for each step.

• logprobsResult.topCandidates array
  • topCandidates[].candidates array
    • candidates[].token string: token (character/word/phrase, etc.)
    • candidates[].logProbability float: log probability (confidence) of the token
• logprobsResult.chosenCandidates array
  • chosenCandidates[].token string
  • chosenCandidates[].logProbability float

usageMetadata object

Token usage statistics.

• usageMetadata.promptTokenCount integer
  Number of tokens in the request.
• usageMetadata.candidatesTokenCount integer
  Number of tokens in the response.
• usageMetadata.totalTokenCount integer
  Total tokens for request + response.
• (May appear) thoughtsTokenCount / toolUsePromptTokenCount / cachedContentTokenCount and per-modality details.

Note: The official docs add that for billing purposes, in Gemini 3 Pro and later models, tokens consumed for processing “document inputs” are billed as image tokens.

modelVersion string

Model and version used for generation (example: gemini-2.0-flash-lite-001). Below is the Vertex AI streamGenerateContent (SSE streaming) response body for each chunk.
Key point: In streaming mode, multiple chunks are returned, and each chunk’s JSON structure is still GenerateContentResponse. Intermediate chunks often have an empty finishReason; only the final chunk provides termination info such as finishReason/finishMessage.

createTime string

Server receive time (RFC3339 Timestamp).

responseId string

Response identifier.

Response (streaming: response body of each stream chunk)

{
  "candidates": [
    {
      "index": integer,
      "content": {
        "role": string,
        "parts": [
          {
            "thought": boolean,
            "thoughtSignature": string, // bytes(base64)
            "mediaResolution": {
              "level": enum,
              "numTokens": integer
            },

            // Union field data (only one of the following fields appears at a time)
            "text": string,
            "inlineData": { "mimeType": string, "data": string, "displayName": string },
            "fileData": { "mimeType": string, "fileUri": string, "displayName": string },
            "functionCall": {
              "id": string,
              "name": string,
              "args": object,
              "partialArgs": [
                {
                  "jsonPath": string,
                  "stringValue": string,
                  "numberValue": number,
                  "boolValue": boolean,
                  "nullValue": string,
                  "willContinue": boolean
                }
              ],
              "willContinue": boolean
            },
            "functionResponse": {
              "id": string,
              "name": string,
              "response": object,
              "parts": [
                {
                  "inlineData": { /* bytes blob */ },
                  "fileData": { /* file ref */ }
                }
              ],
              "scheduling": enum,
              "willContinue": boolean
            },
            "executableCode": { "language": enum, "code": string },
            "codeExecutionResult": { "outcome": enum, "output": string },

            // Union field metadata (only when inlineData/fileData is a video)
            "videoMetadata": {
              "startOffset": string,
              "endOffset": string,
              "fps": number
            }
          }
        ]
      },
      "avgLogprobs": number,
      "logprobsResult": {
        "topCandidates": [
          {
            "candidates": [
              { "token": string, "tokenId": integer, "logProbability": number }
            ]
          }
        ],
        "chosenCandidates": [
          { "token": string, "tokenId": integer, "logProbability": number }
        ]
      },
      "finishReason": enum,
      "safetyRatings": [
        { "category": enum, "probability": enum, "blocked": boolean }
      ],
      "citationMetadata": {
        "citations": [
          {
            "startIndex": integer,
            "endIndex": integer,
            "uri": string,
            "title": string,
            "license": string,
            "publicationDate": { "year": integer, "month": integer, "day": integer }
          }
        ]
      },
      "groundingMetadata": {
        "webSearchQueries": [ string ],
        "retrievalQueries": [ string ],
        "groundingChunks": [
          {
            "web": { "uri": string, "title": string, "domain": string },
            "retrievedContext": object,
            "maps": object
          }
        ],
        "groundingSupports": [ object ],
        "sourceFlaggingUris": [ object ],
        "searchEntryPoint": { "renderedContent": string, "sdkBlob": string },
        "retrievalMetadata": { "googleSearchDynamicRetrievalScore": number },
        "googleMapsWidgetContextToken": string
      },
      "urlContextMetadata": {
        "urlMetadata": [
          { "retrievedUrl": string, "urlRetrievalStatus": enum }
        ]
      },
      "finishMessage": string
    }
  ],
  "modelVersion": string,
  "createTime": string,
  "responseId": string,
  "promptFeedback": {
    "blockReason": enum,
    "blockReasonMessage": string,
    "safetyRatings": [
      { "category": enum, "probability": enum, "blocked": boolean }
    ]
  },
  "usageMetadata": {
    "promptTokenCount": integer,
    "candidatesTokenCount": integer,
    "totalTokenCount": integer

    // (possible extended stats)
    // "cachedContentTokenCount": integer,
    // "thoughtsTokenCount": integer,
    // "toolUsePromptTokenCount": integer,
    // "promptTokensDetails": [...],
    // "candidatesTokensDetails": [...],
    // "toolUsePromptTokensDetails": [...]
  }
}

promptFeedback: Returned only in the first stream chunk, and appears only when no candidates are generated due to policy violations.
finishMessage: Returned only when finishReason is present.

candidates array<Candidate>

List of candidate results in this chunk.

Candidate.index integer

Candidate index (starting from 0).

Candidate.content object

Candidate content (multiple parts).

content.role string

Producer role: typically 'user' or 'model'.

content.parts array<Part>

Array of content parts; each part is a “single-type” data block (text / inlineData / functionCall …).

Part object（content.parts[]）

Part.thought boolean

Whether this is a “thought/reasoning” part.

Part.thoughtSignature string(bytes)

Reusable signature for the thought (base64).

Part.mediaResolution PartMediaResolution

Input media resolution (affects media tokenization).

• mediaResolution.level enum (PartMediaResolutionLevel): LOW / MEDIUM / HIGH / ULTRA_HIGH / UNSPECIFIED.
• mediaResolution.numTokens integer: Expected media token sequence length.

Part.data (Union)

Part.text string

Text content (the most common location for streaming incremental output).

Part.inlineData Blob

Inline binary data (base64).

• inlineData.mimeType string: IANA MIME Type.
• inlineData.data string(bytes): base64 bytes.
• inlineData.displayName string: Optional display name (returned only in some scenarios).

Part.fileData FileData

Reference to an external file (e.g., GCS).

• fileData.mimeType string: IANA MIME Type.
• fileData.fileUri string: File URI.
• fileData.displayName string: Optional display name.

Part.functionCall FunctionCall

A function call predicted by the model.

• functionCall.id string: Function call id (used to match with functionResponse).
• functionCall.name string: Function name.
• functionCall.args object: Function arguments (JSON object).
• functionCall.partialArgs array<PartialArg>: Streaming function-argument deltas (available in some APIs/modes).
• functionCall.willContinue boolean: Whether additional deltas for this FunctionCall will follow.

PartialArg (for functionCall.partialArgs)

• jsonPath string: Path to the parameter being streamed incrementally (RFC 9535).
• stringValue / numberValue / boolValue / nullValue: The delta value for this chunk (one of four).
• willContinue boolean: Whether more deltas will follow for this jsonPath.

Part.functionResponse FunctionResponse

Structure used when you send tool execution results back to the model (may also appear in responses in some modes).

• functionResponse.id string: Corresponding functionCall.id.
• functionResponse.name string: Function name (matches functionCall.name).
• functionResponse.response object: Function result (JSON object; conventionally output/error).
• functionResponse.parts array<FunctionResponsePart>: Multi-part function response (can include files/inline data).
• functionResponse.scheduling enum (FunctionResponseScheduling): SILENT / WHEN_IDLE / INTERRUPT / …
• functionResponse.willContinue boolean: Whether more response fragments will follow.

Part.executableCode ExecutableCode

Code generated by the model for a code-execution tool.

• executableCode.language enum (Language): e.g., PYTHON.
• executableCode.code string: Code string.

Part.codeExecutionResult CodeExecutionResult

Code execution result.

• codeExecutionResult.outcome enum (Outcome): OUTCOME_OK / OUTCOME_FAILED / OUTCOME_DEADLINE_EXCEEDED.
• codeExecutionResult.output string: stdout or error message.

Part.metadata (Union)

Part.videoMetadata VideoMetadata

Metadata used only when the part carries video data.

• videoMetadata.startOffset string: Start offset.
• videoMetadata.endOffset string: End offset.
• videoMetadata.fps number: Frame rate.

Candidate.avgLogprobs number

Candidate average logprob (length-normalized).

Candidate.logprobsResult LogprobsResult

Logprobs details.

• logprobsResult.topCandidates array<LogprobsResultTopCandidates>: Per-step top-token lists.
  • topCandidates[].candidates array<LogprobsResultCandidate>: Sorted by logProbability descending.
• logprobsResult.chosenCandidates array<LogprobsResultCandidate>: The final sampled/selected token per step.
• LogprobsResultCandidate.token / tokenId / logProbability: token, tokenId, log probability.

Candidate.finishReason enum (FinishReason)

Why generation stopped; if empty, it means “not stopped yet”.

Common values (example enums):

• STOP / MAX_TOKENS / SAFETY / RECITATION / BLOCKLIST / PROHIBITED_CONTENT / SPII / MALFORMED_FUNCTION_CALL / OTHER / FINISH_REASON_UNSPECIFIED …

Candidate.safetyRatings array<SafetyRating>

Safety ratings for the candidate output (at most one entry per category).

• safetyRatings[].category enum (HarmCategory): e.g., HATE_SPEECH / SEXUALLY_EXPLICIT / DANGEROUS_CONTENT / HARASSMENT / CIVIC_INTEGRITY …
• safetyRatings[].probability enum (HarmProbability): NEGLIGIBLE / LOW / MEDIUM / HIGH …
• safetyRatings[].blocked boolean: Whether it was filtered due to this rating.

Some APIs/SDKs may also provide finer-grained fields such as probabilityScore / severity / severityScore, which may not appear in all Vertex REST outputs.

Candidate.citationMetadata CitationMetadata

Citation information.

• citationMetadata.citations array<Citation>
  • citations[].startIndex integer: Citation start position
  • citations[].endIndex integer: Citation end position
  • citations[].uri string: Source URL/URI
  • citations[].title string: Source title
  • citations[].license string: License
  • citations[].publicationDate {year,month,day}: Publication date

Candidate.groundingMetadata GroundingMetadata

Retrieval/evidence source metadata returned when grounding is enabled.

• groundingMetadata.webSearchQueries string[]: Queries used for Google Search.
• groundingMetadata.retrievalQueries string[]: Queries actually executed by the retrieval tool.
• groundingMetadata.groundingChunks array<GroundingChunk>: Evidence chunks.
  • groundingChunks[].web {uri,title,domain}: Web evidence.
  • groundingChunks[].retrievedContext / maps: Other evidence sources (object structure depends on the source).
• groundingMetadata.searchEntryPoint {renderedContent,sdkBlob}: Search entry-point info.
• groundingMetadata.retrievalMetadata {googleSearchDynamicRetrievalScore}: Retrieval-related metadata.
• Plus sourceFlaggingUris / googleMapsWidgetContextToken, etc. (when using Google Maps grounding).

Candidate.urlContextMetadata UrlContextMetadata

URL retrieval metadata returned when the model uses the urlContext tool.

• urlContextMetadata.urlMetadata array<UrlMetadata>: URL list.
  • urlMetadata[].retrievedUrl string: The URL that was actually retrieved.
  • urlMetadata[].urlRetrievalStatus enum (UrlRetrievalStatus): SUCCESS / ERROR / PAYWALL / UNSAFE / UNSPECIFIED.

Candidate.finishMessage string

A more detailed explanation of finishReason (returned only when finishReason is present).

modelVersion string

Model version used for this generation.

createTime string

Server receive time (RFC3339 Timestamp).

responseId string

Response identifier.

promptFeedback object

Prompt content-filtering result: only appears in the first stream chunk and only when there are no candidates due to violations.

• promptFeedback.blockReason enum (BlockedReason): Blocking reason.
• promptFeedback.blockReasonMessage string: Human-readable reason (may not be supported in all environments).
• promptFeedback.safetyRatings array<SafetyRating>: Safety ratings for the prompt.

usageMetadata object

Token usage.

• usageMetadata.promptTokenCount integer: Prompt token count.
• usageMetadata.candidatesTokenCount integer: Total candidate output token count.
• usageMetadata.totalTokenCount integer: Total token count.
• (May appear) thoughtsTokenCount / toolUsePromptTokenCount / cachedContentTokenCount and per-modality details.

TypeScript

import { GoogleGenAI } from "@google/genai";

const client = GoogleGenAI({
  apiKey: "$ZENMUX_API_KEY",
  vertexai: true,
  httpOptions: {
    baseUrl: "https://zenmux.ai/api/vertex-ai",
    apiVersion: "v1",
  },
});

const response = await client.models.generateContent({
  model: "google/gemini-2.5-pro",
  contents: "How does AI work?",
});
console.log(response);

Python

from google import genai
from google.genai import types

client = genai.Client(
    api_key="$ZENMUX_API_KEY",
    vertexai=True,
    http_options=types.HttpOptions(
        api_version='v1',
        base_url='https://zenmux.ai/api/vertex-ai'
    ),
)

response = client.models.generate_content(
    model="google/gemini-2.5-pro",
    contents="How does AI work?"
)
print(response.text)

json

{
  "candidates": [
    {
      "content": {
        "role": "model",
        "parts": [
          {
            "text": "Of course. This is a fantastic question. Let's break down how AI works using a simple analogy and then add the technical details.\n\n### The Simple Analogy: Teaching a Child to Recognize a Cat\n\nImagine you're teaching a very young child what a \"cat\" is. You don't write down a long list of rules like \"a cat has pointy ears, four legs, a tail, and whiskers.\" Why? Because some cats have folded ears, some might be missing a leg, and a dog also fits that description.\n\nInstead, you do this:\n\n1.  **Show Examples:** You show the child hundreds of pictures. You point and say, \"That's a cat.\" \"That's also a cat.\" \"This is *not* a cat; it's a dog.\"\n2.  **Let Them Guess:** You show them a new picture and ask, \"Is this a cat?\"\n3.  **Give Feedback:** If they're right, you say \"Yes, good job!\" If they're wrong, you say \"No, that's a fox.\"\n\nOver time, the child's brain, without being told the specific rules, starts to recognize the *patterns* that make a cat a cat. They build an internal, intuitive understanding.\n\n**AI works in almost the exact same way.** It's a system designed to learn patterns from data without being explicitly programmed with rules.\n\n---\n\n### The Core Components of How AI Works\n\nNow, let's replace the child with a computer program. The process has three key ingredients:\n\n#### 1. Data (The Pictures)\n\nThis is the most critical ingredient. AI is fueled by data. For our example, this would be a massive dataset of thousands or millions of images, each one labeled by a human: \"cat,\" \"dog,\" \"hamster,\" etc.\n\n*   **More Data is Better:** The more examples the AI sees, the better it gets at identifying the patterns.\n*   **Good Data is Crucial:** The data must be accurate and diverse. If you only show it pictures of black cats, it will struggle to recognize a white cat.\n\n#### 2. Model / Algorithm (The Child's Brain)\n\nThis is the mathematical framework that learns from the data. Think of it as the \"engine\" that finds the patterns. When you hear terms like **\"Neural Network,\"** this is what they're referring to.\n\nA neural network is inspired by the human brain. It's made of interconnected digital \"neurons\" organized in layers.\n\n*   **Input Layer:** Takes in the raw data (e.g., the pixels of an image).\n*   **Hidden Layers:** This is where the magic happens. Each layer recognizes increasingly complex patterns. The first layer might learn to spot simple edges and colors. The next might combine those to recognize shapes like ears and tails. A deeper layer might combine those shapes to recognize a \"cat face.\"\n*   **Output Layer:** Gives the final answer (e.g., a probability score: \"95% chance this is a cat, 3% dog, 2% fox\").\n\n#### 3. The Training Process (Learning from Feedback)\n\nThis is where the **Model** learns from the **Data**. It's an automated version of showing pictures and giving feedback.\n\n1.  **Prediction (The Guess):** The model is given an input (an image of a cat) and makes a random guess. Early on, its internal settings are all random, so its guess will be wild—it might say \"50% car, 50% dog.\"\n2.  **Compare (Check the Answer):** The program compares its prediction to the correct label (\"cat\"). It then calculates its \"error\" or \"loss\"—a measure of how wrong it was.\n3.  **Adjust (Learn):** This is the key step. The algorithm uses a mathematical process (often called **\"backpropagation\"** and **\"gradient descent\"**) to slightly adjust the millions of internal connections in the neural network. The adjustments are tiny, but they are specifically designed to make the model's guess *less wrong* the next time it sees that same image.\n4.  **Repeat:** This process is repeated **millions or billions of times** with all the data. Each time, the model gets a little less wrong. Over many cycles, these tiny adjustments cause the network to get incredibly accurate at recognizing the patterns it's being shown.\n\nAfter training is complete, you have a **\"trained model.\"** You can now give it brand new data it has never seen before, and it will be able to make accurate predictions.\n\n---\n\n### Major Types of AI Learning\n\nWhile the above is the most common method, there are three main ways AI learns:\n\n**1. Supervised Learning (Learning with an Answer Key)**\nThis is the \"cat\" example we just used. The AI is \"supervised\" because it's trained on data that is already labeled with the correct answers.\n*   **Examples:** Spam filters (emails labeled \"spam\" or \"not spam\"), predicting house prices (houses with known prices), language translation.\n\n**2. Unsupervised Learning (Finding Patterns on its Own)**\nThis is like giving the AI a giant pile of data with *no labels* and asking it to \"find interesting patterns.\" The AI might group the data into clusters based on hidden similarities.\n*   **Examples:** Customer segmentation (finding groups of customers with similar buying habits), identifying anomalies in a computer network.\n\n**3. Reinforcement Learning (Learning through Trial and Error)**\nThis is how you train an AI to play a game or control a robot. The AI takes an action in an environment and receives a reward or a penalty. Its goal is to maximize its total reward over time.\n*   **Examples:** An AI learning to play chess (it gets a reward for winning the game), a robot learning to walk (it gets a reward for moving forward without falling), self-driving car simulations.\n\n### Summary\n\nSo, \"How does AI work?\"\n\n**At its core, modern AI is a system that learns to recognize incredibly complex patterns by processing vast amounts of data, making guesses, and correcting its errors over and over again until it becomes highly accurate.**\n\nIt's less about being \"intelligent\" in a human sense and more about being a phenomenally powerful pattern-matching machine."
          }
        ]
      },
      "finishReason": "STOP",
      "avgLogprobs": -0.4167558059635994
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 5,
    "candidatesTokenCount": 1353,
    "totalTokenCount": 2794,
    "trafficType": "ON_DEMAND",
    "promptTokensDetails": [
      {
        "modality": "TEXT",
        "tokenCount": 5
      }
    ],
    "candidatesTokensDetails": [
      {
        "modality": "TEXT",
        "tokenCount": 1353
      }
    ],
    "thoughtsTokenCount": 1436
  },
  "modelVersion": "google/gemini-2.5-pro",
  "createTime": "2026-01-29T08:40:38.791866Z",
  "responseId": "Bh17abqqMOSS4_UPqqeqoAc"
}