MBPP

This is the test set of Mostly Basic Python Problems Dataset, with id in [11, 510].

Configuration

Field	Value	Description
is_fewshot		Append fewshot examples before problem
run_timeout	​	Execution timeout in seconds

Usage

from datasets import load_dataset
import requests

config = {
  'is_fewshot': True,
  'run_timeout': 20,
  'dataset_type': "MBPPDataset"
}

# Get dataset data in sandbox format
data = list(load_dataset("laylarsssss/FusedMBPP", split="test"))

config['provided_data'] = data
prompts = requests.post('http://localhost:8080/get_prompts', json={
  'dataset': 'mbpp',
  'config': config
}).json()

print('please perform model inference on these prompts:')
print('\n'.join([p['prompt'] for p in prompts[:3]]))
print('...')

# your model inference code here
completions = ['' for _ in prompts]

for completion, sample in zip(completions, data):
    config['provided_data'] = sample
    res = requests.post('http://localhost:8080/submit', json={
        'dataset': 'mbpp',
        'id': '',
        'completion': completion,
        'config': config
    })

    print(f'result: {res.json()}')
    break

Note: always put raw completion in the request, Sandbox will handle the extraction of code according to different modes.