📜 SECTION 1: MISSION
What is the competition?
SimulacraBench — a UN competition to improve the accuracy of sociological surveys through AI.
Task: Predict the probabilities of respondents' answers to questions they were not asked or did not answer.
Format:
- 3 datasets: UNICEF, World Bank, UNHCR
- Each dataset is a "respondent × question" table
- We only see part of the answers (GIVEN), we must predict the hidden ones (PREDICT)
- Metric: Skill (0 = random guessing, 1 = perfect)
Prizes: Not specified in the README, but this is a prestigious competition from the UN + Stanford.
Why does it matter?
It's not just about "filling in the blanks." It's about making surveys cheaper and more accurate by predicting the answers of those who did not respond or who were not asked a question due to survey logic (gate).
🗺️ SECTION 2: WORLD MAP
Project structure
SituatedEvals/public/
├── data/
│ ├── sample.json # Test schema (400 respondents)
│ ├── unicef.json # Real UNICEF schema
│ ├── world_bank.json # Real World Bank schema
│ └── unhcr.json # Real UNHCR schema
├── baseline/
│ ├── marginal_counts/ # Baseline: marginal frequencies
│ └── bundled_artifact/ # Example of loading weights
├── tutorials/
│ ├── ru.ipynb # Tutorial in Russian
│ └── en.ipynb # Tutorial in English
├── tools/
│ └── check_submission_zip.py # Archive validator
├── make_sandbox.py # Sandbox generator
├── score.py # Local scoring
├── config.yml # Configuration (limits, phases)
└── requirements.txt # DependenciesKey files (study priority)
config.yml— time limits, memory, phase rulesdata/*.json— survey schemas (question structure, gate logic)baseline/marginal_counts/main.py— minimal working templatetutorials/en.ipynb— explanation of mechanics by example
Competition phases
Phase 1 (Development):
- Available: TRAIN (with answers), DEV (GIVEN only)
- Leaderboard: noisy (Laplace noise)
- Attempts: 1 per day
- Timer: 900 seconds for all 3 datasets
Phase 2 (Final):
- Available: TRAIN + DEV (both with answers), TEST (GIVEN only)
- Leaderboard: exact
- Attempts: 1 for all time
- Timer: 3600 seconds
📍 SECTION 3: CURRENT POINT (Day 3)
Where we are now:
- ✅ Repository cloned
- ✅ Dependencies installed
- ✅ Sandbox generated
- ⏳ Next step: First baseline submission to Codabench
What we already know:
- Data structure studied (schemas
data/*.json) - We understand the contract
predict(frame, schema) - We know about gate logic (skip patterns)
- We understand the Skill metric
What we DON'T know:
- Real quality of the baseline on the leaderboard
- How noisy the leaderboard is in Phase 1
- Which gate rules give the maximum boost
🎮 SECTION 4: HERO'S PATH (Main plan)
LEVEL 0: "First Steps" (Days 3-5)
Goal: Get a guaranteed working submission
Stage 0.1: Local baseline validation
Actions:
# Generate sandbox for all 3 datasets
python make_sandbox.py --schema data/unicef.json --out _sandbox/unicef
python make_sandbox.py --schema data/world_bank.json --out _sandbox/world_bank
python make_sandbox.py --schema data/unhcr.json --out _sandbox/unhcr
# Check the baseline on all 3 datasets
python score.py --submission baseline/marginal_counts --data _sandbox/unicef --schema data/unicef.json --phase 1
python score.py --submission baseline/marginal_counts --data _sandbox/world_bank --schema data/world_bank.json --phase 1
python score.py --submission baseline/marginal_counts --data _sandbox/unhcr --schema data/unhcr.json --phase 1Artifact: 3 PASS outputs with local skill scores
Reinforcement: We understand the baseline level we will build upon
Success criterion: All 3 commands return PASS
Stage 0.2: First submission to Codabench
Actions:
# Copy the baseline
cp -R baseline/marginal_counts sub_01_baseline
cd sub_01_baseline
# Package (IMPORTANT: from inside the folder!)
zip -r ../sub_01_baseline.zip .
cd ..
# Check the archive
python tools/check_submission_zip.py sub_01_baseline.zip
# Upload to Codabench via VPN
# https://codabench.org/competitions/[competition_ID]Artifact: First line on the leaderboard
Reinforcement:
- We understand the real baseline level
- We see how the leaderboard works
- We gain confidence that everything works
Success criterion: Leaderboard shows skill > 0
Stage 0.3: Data schema reconnaissance
Actions:
Ask Qwen: "Analyze data/unicef.json, data/world_bank.json, data/unhcr.json. For each dataset, output:
- Number of GIVEN, PREDICT, EXCLUDE questions
- Number of PREDICT cells (how many to predict)
- All gate rules (parent → observed_if)
- Typical probability vector length (len(values) + 1)"
Artifact: Table with characteristics of all 3 datasets
Reinforcement: We understand the scale of the task and where to look for "easy money"
Success criterion: You can answer: How many PREDICT cells need to be predicted? How many gate rules are in each dataset?
LEVEL 1: "Hunting for easy money" (Days 6-12)
Goal: Find and use deterministic rules
Stage 1.1: Gate logic analysis
Actions:
Ask Qwen: "Write a script that:
- Loads all 3 schemas from
data/*.json - For each gate rule outputs: parent question name,
observed_ifcondition, child question name - Saves the result to
gate_rules_analysis.txt"
Artifact: gate_rules_analysis.txt with all rules
Reinforcement: You see which rules can be used as deterministic
Success criterion: List of N gate rules where observed_if is a specific value
Stage 1.2: Deterministic gate rules
Actions:
Ask Qwen: "Based on gate_rules_analysis.txt, write a function apply_gate_rules(frame, schema) that:
- For each PREDICT question with a gate, checks the parent question value
- If the parent has a value from
observed_if, returns probability 1.0 for gated_value (last slot) - Otherwise returns None (to use the regular model)"
Integration prompt:
"Now integrate apply_gate_rules into baseline/marginal_counts/main.py. If apply_gate_rules returned a vector, use it. Otherwise use marginal frequencies."
Artifact: New version of main.py with gate logic
Reinforcement: Deterministic rules give a huge boost (often +10-20% to skill)
Success criterion: Local score.py shows skill > baseline
Stage 1.3: Submission with gate logic
Actions:
cp -R baseline/marginal_counts sub_02_gate_rules
cd sub_02_gate_rules
zip -r ../sub_02_gate_rules.zip .
cd ..
python tools/check_submission_zip.py sub_02_gate_rules.zipArtifact: Second line on the leaderboard
Reinforcement: You see the real gain from gate logic, understand how noisy the leaderboard is
Success criterion: skill(sub_02) > skill(sub_01)
LEVEL 2: "First model" (Days 13-25)
Goal: Build a model that uses respondent features
Stage 2.1: Analysis of marginal distributions
Actions:
Ask Qwen: "Write a script analyze_marginals.py that:
- Loads TRAIN data from the sandbox
- For each PREDICT question, calculates the distribution of answers
- Outputs the top-3 most "confident" questions (one option >60%)
- Outputs the top-3 most "uniform" questions (all options <20%)
- Saves marginal frequencies to
marginals.json"
Artifact: marginals.json + understanding where the model can win
Reinforcement: You understand which questions are "easy" and which are "hard"
Success criterion: The script works and outputs statistics
Stage 2.2: Simple model (CatBoost/XGBoost)
Actions:
Ask Qwen: "Write a script train_catboost.py that:
- Loads TRAIN data from the sandbox
- For each PREDICT question, trains CatBoost where: features are all GIVEN columns, target is the answer to the PREDICT question
- Saves models to
models/catboost_{question_name}.cbm - Outputs accuracy on validation"
Artifact: Trained models for each question
Reinforcement: The model takes into account individual respondent features
Success criterion: Models train and accuracy > marginal baseline
Stage 2.3: Integrating the model into main.py
Actions:
Ask Qwen: "Write main.py that:
- At the module level loads all models from
models/ - In
predict()for each PREDICT question: if there is a gate rule and it triggered → use it, otherwise use CatBoost - Last slot (gated_value) = probability from gate rule or 0.0
- Returns vectors in the correct order"
Artifact: Working main.py with CatBoost
Reinforcement: The model uses respondent features + gate logic
Success criterion: Local score.py shows skill > gate rules
