Create Eval Completions Run Data Source

• CreateEvalCompletionsRunDataSource object \{ source, type, input_messages, 2 more \}

  A CompletionsRunDataSource object describing a model sampling configuration.

  • source: object \{ content, type \} or object \{ id, type \} or object \{ type, created_after, created_before, 3 more \}

    Determines what populates the item namespace in this run's data source.

    • EvalJSONLFileContentSource object \{ content, type \}

      • content: array of object \{ item, sample \}

        The content of the jsonl file.

        • item: map[unknown]

        • sample: optional map[unknown]

      • type: "file_content"

        The type of jsonl source. Always file_content.

        • "file_content"

    • EvalJSONLFileIDSource object \{ id, type \}

      • id: string

        The identifier of the file.

      • type: "file_id"

        The type of jsonl source. Always file_id.

        • "file_id"

    • StoredCompletionsRunDataSource object \{ type, created_after, created_before, 3 more \}

      A StoredCompletionsRunDataSource configuration describing a set of filters

      • type: "stored_completions"

        The type of source. Always stored_completions.

        • "stored_completions"

      • created_after: optional number or null

        An optional Unix timestamp to filter items created after this time.

      • created_before: optional number or null

        An optional Unix timestamp to filter items created before this time.

      • limit: optional number or null

        An optional maximum number of items to return.

      • metadata: optional Metadata or null

        Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format, and querying for objects via API or the dashboard.

        Keys are strings with a maximum length of 64 characters. Values are strings with a maximum length of 512 characters.

      • model: optional string or null

        An optional model to filter by (e.g., 'gpt-6-astra').

  • type: "completions"

    The type of run data source. Always completions.

    • "completions"

  • input_messages: optional object \{ template, type \} or object \{ item_reference, type \}

    Used when sampling from a model. Dictates the structure of the messages passed into the model. Can either be a reference to a prebuilt trajectory (ie, item.input_trajectory), or a template with variable references to the item namespace.

    • TemplateInputMessages object \{ template, type \}

      • template: array of EasyInputMessage or object \{ content, role, type \}

        A list of chat messages forming the prompt or context. May include variable references to the item namespace, ie {{item.name}}.

        • EasyInputMessage object \{ content, role, phase, type \}

          A message input to the model with a role indicating instruction following hierarchy. Instructions given with the developer or system role take precedence over instructions given with the user role. Messages with the assistant role are presumed to have been generated by the model in previous interactions.

          • content: string or ResponseInputMessageContentList

            Text, image, or audio input to the model, used to generate a response. Can also contain previous assistant responses.

            • TextInput = string

              A text input to the model.

            • ResponseInputMessageContentList = array of ResponseInputContent

              A list of one or many input items to the model, containing different content types.

              • ResponseInputText object \{ text, type, prompt_cache_breakpoint \}

                A text input to the model.

                • text: string

                  The text input to the model.

                • type: "input_text"

                  The type of the input item. Always input_text.

                  • "input_text"

                • prompt_cache_breakpoint: optional object \{ mode \}

                  Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.

                  • mode: "explicit"

                    The breakpoint mode. Always explicit.

                    • "explicit"

              • ResponseInputImage object \{ detail, type, file_id, 2 more \}

                An image input to the model. Learn about image inputs.

                • detail: ImageDetail

                  The detail level of the image to be sent to the model. One of high, low, auto, or original. Defaults to auto.

                  • "low"

                  • "high"

                  • "auto"

                  • "original"

                • type: "input_image"

                  The type of the input item. Always input_image.

                  • "input_image"

                • file_id: optional string or null

                  The ID of the file to be sent to the model.

                • image_url: optional string or null

                  The URL of the image to be sent to the model. A fully qualified URL or base64 encoded image in a data URL.

                • prompt_cache_breakpoint: optional object \{ mode \}

                  Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.

                  • mode: "explicit"

                    The breakpoint mode. Always explicit.

                    • "explicit"

              • ResponseInputFile object \{ type, detail, file_data, 4 more \}

                A file input to the model.

                • type: "input_file"

                  The type of the input item. Always input_file.

                  • "input_file"

                • detail: optional "auto" or "low" or "high"

                  The detail level of the file to be sent to the model. Use auto to let the system select the detail level; for GPT-5.6 and later models, auto uses high-quality rendering, which may increase input token usage. Use low for lower-cost rendering, or high to render the file at higher quality. Defaults to auto.

                  • "auto"

                  • "low"

                  • "high"

                • file_data: optional string

                  The content of the file to be sent to the model.

                • file_id: optional string or null

                  The ID of the file to be sent to the model.

                • file_url: optional string

                  The URL of the file to be sent to the model.

                • filename: optional string

                  The name of the file to be sent to the model.

                • prompt_cache_breakpoint: optional object \{ mode \}

                  Marks the exact end of a reusable prompt prefix. The breakpoint inherits its TTL from the request's prompt_cache_options.ttl; the boundary is not rounded to a token block.

                  • mode: "explicit"

                    The breakpoint mode. Always explicit.

                    • "explicit"

          • role: "user" or "assistant" or "system" or "developer"

            The role of the message input. One of user, assistant, system, or developer.

            • "user"

            • "assistant"

            • "system"

            • "developer"

          • phase: optional "commentary" or "final_answer" or null

            Labels an assistant message as intermediate commentary (commentary) or the final answer (final_answer). For models like gpt-5.3-codex and beyond, when sending follow-up requests, preserve and resend phase on all assistant messages — dropping it can degrade performance. Not used for user messages.

            • "commentary"

            • "final_answer"

          • type: optional "message"

            The type of the message input. Always message.

            • "message"

        • EvalMessageObject object \{ content, role, type \}

          A message input to the model with a role indicating instruction following hierarchy. Instructions given with the developer or system role take precedence over instructions given with the user role. Messages with the assistant role are presumed to have been generated by the model in previous interactions.

          • content: string or ResponseInputText or object \{ text, type \} or 3 more

            Inputs to the model - can contain template strings. Supports text, output text, input images, and input audio, either as a single item or an array of items.

            • TextInput = string

              A text input to the model.

            • ResponseInputText object \{ text, type, prompt_cache_breakpoint \}

              A text input to the model.

            • OutputText object \{ text, type \}

              A text output from the model.

              • text: string

                The text output from the model.

              • type: "output_text"

                The type of the output text. Always output_text.

                • "output_text"

            • InputImage object \{ image_url, type, detail \}

              An image input block used within EvalItem content arrays.

              • image_url: string

                The URL of the image input.

              • type: "input_image"

                The type of the image input. Always input_image.

                • "input_image"

              • detail: optional string

                The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

            • ResponseInputAudio object \{ input_audio, type \}

              An audio input to the model.

              • input_audio: object \{ data, format \}

                • data: string

                  Base64-encoded audio data.

                • format: "mp3" or "wav"

                  The format of the audio data. Currently supported formats are mp3 and wav.

                  • "mp3"

                  • "wav"

              • type: "input_audio"

                The type of the input item. Always input_audio.

                • "input_audio"

            • GraderInputs = array of string or ResponseInputText or object \{ text, type \} or 2 more

              A list of inputs, each of which may be either an input text, output text, input image, or input audio object.

              • TextInput = string

                A text input to the model.

              • ResponseInputText object \{ text, type, prompt_cache_breakpoint \}

                A text input to the model.

              • OutputText object \{ text, type \}

                A text output from the model.

                • text: string

                  The text output from the model.

                • type: "output_text"

                  The type of the output text. Always output_text.

                  • "output_text"

              • InputImage object \{ image_url, type, detail \}

                An image input block used within EvalItem content arrays.

                • image_url: string

                  The URL of the image input.

                • type: "input_image"

                  The type of the image input. Always input_image.

                  • "input_image"

                • detail: optional string

                  The detail level of the image to be sent to the model. One of high, low, or auto. Defaults to auto.

              • ResponseInputAudio object \{ input_audio, type \}

                An audio input to the model.

          • role: "user" or "assistant" or "system" or "developer"

            The role of the message input. One of user, assistant, system, or developer.

            • "user"

            • "assistant"

            • "system"

            • "developer"

          • type: optional "message"

            The type of the message input. Always message.

            • "message"

      • type: "template"

        The type of input messages. Always template.

        • "template"

    • ItemReferenceInputMessages object \{ item_reference, type \}

      • item_reference: string

        A reference to a variable in the item namespace. Ie, "item.input_trajectory"

      • type: "item_reference"

        The type of input messages. Always item_reference.

        • "item_reference"

  • model: optional string

    The name of the model to use for generating completions (e.g. "o3-mini").

  • sampling_params: optional object \{ max_completion_tokens, reasoning_effort, response_format, 4 more \}

    • max_completion_tokens: optional number

      The maximum number of tokens in the generated output.

    • reasoning_effort: optional ReasoningEffort or null

      Constrains effort on reasoning for reasoning models. Currently supported values are none, minimal, low, medium, high, xhigh, and max. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response. Not all reasoning models support every value. See the reasoning guide for model-specific support.

      • "none"

      • "minimal"

      • "low"

      • "medium"

      • "high"

      • "xhigh"

      • "max"

    • response_format: optional ResponseFormatText or ResponseFormatJSONSchema or ResponseFormatJSONObject

      An object specifying the format that the model must output.

      Setting to \{ "type": "json_schema", "json_schema": \{...\} \} enables Structured Outputs which ensures the model will match your supplied JSON schema. Learn more in the Structured Outputs guide.

      Setting to \{ "type": "json_object" \} enables the older JSON mode, which ensures the message the model generates is valid JSON. Using json_schema is preferred for models that support it.

      • ResponseFormatText object \{ type \}

        Default response format. Used to generate text responses.

        • type: "text"

          The type of response format being defined. Always text.

          • "text"

      • ResponseFormatJSONSchema object \{ json_schema, type \}

        JSON Schema response format. Used to generate structured JSON responses. Learn more about Structured Outputs.

        • json_schema: object \{ name, description, schema, strict \}

          Structured Outputs configuration options, including a JSON Schema.

          • name: string

            The name of the response format. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

          • description: optional string

            A description of what the response format is for, used by the model to determine how to respond in the format.

          • schema: optional map[unknown]

            The schema for the response format, described as a JSON Schema object. Learn how to build JSON schemas here.

          • strict: optional boolean or null

            Whether to enable strict schema adherence when generating the output. If set to true, the model will always follow the exact schema defined in the schema field. Only a subset of JSON Schema is supported when strict is true. To learn more, read the Structured Outputs guide.

        • type: "json_schema"

          The type of response format being defined. Always json_schema.

          • "json_schema"

      • ResponseFormatJSONObject object \{ type \}

        JSON object response format. An older method of generating JSON responses. Using json_schema is recommended for models that support it. Note that the model will not generate JSON without a system or user message instructing it to do so.

        • type: "json_object"

          The type of response format being defined. Always json_object.

          • "json_object"

    • seed: optional number

      A seed value to initialize the randomness, during sampling.

    • temperature: optional number

      A higher temperature increases randomness in the outputs.

    • tools: optional array of ChatCompletionFunctionTool

      A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for. A max of 128 functions are supported.

      • function: FunctionDefinition

        • name: string

          The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.

        • description: optional string

          A description of what the function does, used by the model to choose when and how to call the function.

        • parameters: optional FunctionParameters

          The parameters the functions accepts, described as a JSON Schema object. See the guide for examples, and the JSON Schema reference for documentation about the format.

          Omitting parameters defines a function with an empty parameter list.

        • strict: optional boolean or null

          Whether to enable strict schema adherence when generating the function call. If set to true, the model will follow the exact schema defined in the parameters field. Only a subset of JSON Schema is supported when strict is true. Learn more about Structured Outputs in the function calling guide.

      • type: "function"

        The type of the tool. Currently, only function is supported.

        • "function"

    • top_p: optional number

      An alternative to temperature for nucleus sampling; 1.0 includes all tokens.