Get Metrics

POST 
/get_metrics

Get the metrics given all problem results in a dataset (partially supported)

Request

application/json

Body

required

dataset Dataset (string)required

config

object

required

custom_extract_logic: a piece of python code that calls submit_code_blocks(cbs) to extract custom code cbs: List[CodeBlock], CodeBlock(priority=40, code='xxx', language='xxx') priority: fenced = 30, incomplete fenced = 20, heuristic = 10

dataset_type

object

the dataset class used to process, only works when the dataset id is not registered.

anyOf

MOD1

string

language

object

anyOf

MOD1

string

Possible values: [python, cpp, nodejs, go, go_test, java, php, csharp, bash, typescript, sql, rust, cuda, lua, R, perl, D_ut, ruby, scala, julia, pytest, junit, kotlin_script, jest, verilog, python_gpu, lean, swift, racket]

locale

object

anyOf

MOD1

string

is_fewshot

object

anyOf

MOD1

boolean

compile_timeout

object

anyOf

MOD1

number

run_timeout

object

anyOf

MOD1

number

custom_extract_logic

object

anyOf

MOD1

string

provided_data

object

anyOf

MOD1

Array [

property name*

object

anyOf

MOD1

string

MOD2

integer

]

MOD2

property name*

object

anyOf

MOD1

string

MOD2

integer

extra object

Default value: {}

results

object[]

required

Array [

id

object

required

anyOf

MOD1

integer

MOD2

string

accepted Accepted (boolean)required

extracted_code Extracted Code (string)required

full_code

object

anyOf

MOD1

string

test_code

object

anyOf

MOD1

string

tests

object[]

required

Array [

passed Passed (boolean)required

exec_info

object

required

status RunStatus (string)required

Possible values: [Success, Failed, SandboxError]

message Message (string)required

compile_result

object

anyOf

CommandRunResult

status CommandRunStatus (string)required

Possible values: [Finished, Error, TimeLimitExceeded]

execution_time

object

anyOf

MOD1

number

return_code

object

anyOf

MOD1

integer

stdout

object

anyOf

MOD1

string

stderr

object

anyOf

MOD1

string

run_result

object

anyOf

CommandRunResult

status CommandRunStatus (string)required

Possible values: [Finished, Error, TimeLimitExceeded]

execution_time

object

anyOf

MOD1

number

return_code

object

anyOf

MOD1

integer

stdout

object

anyOf

MOD1

string

stderr

object

anyOf

MOD1

string

executor_pod_name

object

anyOf

MOD1

string

files

object

property name* string

test_info

object

anyOf

MOD1

object

]

extracted_type

object

anyOf

MOD1

string

Possible values: [fenced, incomplete_fenced, heuristic, empty]

extra

object

anyOf

MOD1

object

]

Responses

200

Successful Response

application/json

Schema

Schema

object

Example (from schema)

{}

422

Validation Error

application/json

Schema

Schema

detail

object[]

Array [

loc

object[]

required

Array [

anyOf

MOD1

string

MOD2

integer

]

msg Message (string)required

type Error Type (string)required

]

Example (from schema)

{
  "detail": [
    {
      "loc": [
        "string",
        0
      ],
      "msg": "string",
      "type": "string"
    }
  ]
}

Loading...