Roadmap for participation in the UN AI competition 1 - 2.1

📜 SECTION 1: MISSION

What is the competition?

SimulacraBench — a UN competition to improve the accuracy of sociological surveys through AI.

Task: Predict the probabilities of respondents' answers to questions they were not asked or did not answer.

Format:

• 3 datasets: UNICEF, World Bank, UNHCR
• Each dataset is a "respondent × question" table
• We only see part of the answers (GIVEN), we must predict the hidden ones (PREDICT)
• Metric: Skill (0 = random guessing, 1 = perfect)

Prizes: Not specified in the README, but this is a prestigious competition from the UN + Stanford.

Why does it matter?

It's not just about "filling in the blanks." It's about making surveys cheaper and more accurate by predicting the answers of those who did not respond or who were not asked a question due to survey logic (gate).

---

🗺️ SECTION 2: WORLD MAP

Project structure

SituatedEvals/public/
├── data/
│   ├── sample.json          # Test schema (400 respondents)
│   ├── unicef.json          # Real UNICEF schema
│   ├── world_bank.json      # Real World Bank schema
│   └── unhcr.json           # Real UNHCR schema
├── baseline/
│   ├── marginal_counts/     # Baseline: marginal frequencies
│   └── bundled_artifact/    # Example of loading weights
├── tutorials/
│   ├── ru.ipynb             # Tutorial in Russian
│   └── en.ipynb             # Tutorial in English
├── tools/
│   └── check_submission_zip.py  # Archive validator
├── make_sandbox.py          # Sandbox generator
├── score.py                 # Local scoring
├── config.yml               # Configuration (limits, phases)
└── requirements.txt         # Dependencies

Key files (study priority)

1. config.yml — time limits, memory, phase rules
2. data/*.json — survey schemas (question structure, gate logic)
3. baseline/marginal_counts/main.py — minimal working template
4. tutorials/en.ipynb — explanation of mechanics by example

Competition phases

Phase 1 (Development):

• Available: TRAIN (with answers), DEV (GIVEN only)
• Leaderboard: noisy (Laplace noise)
• Attempts: 1 per day
• Timer: 900 seconds for all 3 datasets

Phase 2 (Final):

• Available: TRAIN + DEV (both with answers), TEST (GIVEN only)
• Leaderboard: exact
• Attempts: 1 for all time
• Timer: 3600 seconds

---

📍 SECTION 3: CURRENT POINT (Day 3)

Where we are now:

• ✅ Repository cloned
• ✅ Dependencies installed
• ✅ Sandbox generated
• ⏳ Next step: First baseline submission to Codabench

What we already know:

• Data structure studied (schemas data/*.json)
• We understand the contract predict(frame, schema)
• We know about gate logic (skip patterns)
• We understand the Skill metric

What we DON'T know:

• Real quality of the baseline on the leaderboard
• How noisy the leaderboard is in Phase 1
• Which gate rules give the maximum boost

---

🎮 SECTION 4: HERO'S PATH (Main plan)

LEVEL 0: "First Steps" (Days 3-5)

Goal: Get a guaranteed working submission

Stage 0.1: Local baseline validation

Actions:

# Generate sandbox for all 3 datasets
python make_sandbox.py --schema data/unicef.json --out _sandbox/unicef
python make_sandbox.py --schema data/world_bank.json --out _sandbox/world_bank
python make_sandbox.py --schema data/unhcr.json --out _sandbox/unhcr

# Check the baseline on all 3 datasets
python score.py --submission baseline/marginal_counts --data _sandbox/unicef --schema data/unicef.json --phase 1
python score.py --submission baseline/marginal_counts --data _sandbox/world_bank --schema data/world_bank.json --phase 1
python score.py --submission baseline/marginal_counts --data _sandbox/unhcr --schema data/unhcr.json --phase 1

Artifact: 3 PASS outputs with local skill scores

Reinforcement: We understand the baseline level we will build upon

Success criterion: All 3 commands return PASS

---

Stage 0.2: First submission to Codabench

Actions:

# Copy the baseline
cp -R baseline/marginal_counts sub_01_baseline
cd sub_01_baseline

# Package (IMPORTANT: from inside the folder!)
zip -r ../sub_01_baseline.zip .
cd ..

# Check the archive
python tools/check_submission_zip.py sub_01_baseline.zip

# Upload to Codabench via VPN
# https://codabench.org/competitions/[competition_ID]

Artifact: First line on the leaderboard

Reinforcement:

• We understand the real baseline level
• We see how the leaderboard works
• We gain confidence that everything works

Success criterion: Leaderboard shows skill > 0

---

Stage 0.3: Data schema reconnaissance

Actions:

Ask Qwen: "Analyze data/unicef.json, data/world_bank.json, data/unhcr.json. For each dataset, output:

1. Number of GIVEN, PREDICT, EXCLUDE questions
2. Number of PREDICT cells (how many to predict)
3. All gate rules (parent → observed_if)
4. Typical probability vector length (len(values) + 1)"

Artifact: Table with characteristics of all 3 datasets

Reinforcement: We understand the scale of the task and where to look for "easy money"

Success criterion: You can answer: How many PREDICT cells need to be predicted? How many gate rules are in each dataset?

---

LEVEL 1: "Hunting for easy money" (Days 6-12)

Goal: Find and use deterministic rules

Stage 1.1: Gate logic analysis

Actions:

Ask Qwen: "Write a script that:

1. Loads all 3 schemas from data/*.json
2. For each gate rule outputs: parent question name, observed_if condition, child question name
3. Saves the result to gate_rules_analysis.txt"

Artifact: gate_rules_analysis.txt with all rules

Reinforcement: You see which rules can be used as deterministic

Success criterion: List of N gate rules where observed_if is a specific value

---

Stage 1.2: Deterministic gate rules

Actions:

Ask Qwen: "Based on gate_rules_analysis.txt, write a function apply_gate_rules(frame, schema) that:

1. For each PREDICT question with a gate, checks the parent question value
2. If the parent has a value from observed_if, returns probability 1.0 for gated_value (last slot)
3. Otherwise returns None (to use the regular model)"

Integration prompt:

"Now integrate apply_gate_rules into baseline/marginal_counts/main.py. If apply_gate_rules returned a vector, use it. Otherwise use marginal frequencies."

Artifact: New version of main.py with gate logic

Reinforcement: Deterministic rules give a huge boost (often +10-20% to skill)

Success criterion: Local score.py shows skill > baseline

---

Stage 1.3: Submission with gate logic

Actions:

cp -R baseline/marginal_counts sub_02_gate_rules
cd sub_02_gate_rules
zip -r ../sub_02_gate_rules.zip .
cd ..
python tools/check_submission_zip.py sub_02_gate_rules.zip

Artifact: Second line on the leaderboard

Reinforcement: You see the real gain from gate logic, understand how noisy the leaderboard is

Success criterion: skill(sub_02) > skill(sub_01)

---

LEVEL 2: "First model" (Days 13-25)

Goal: Build a model that uses respondent features

Stage 2.1: Analysis of marginal distributions

Actions:

Ask Qwen: "Write a script analyze_marginals.py that:

1. Loads TRAIN data from the sandbox
2. For each PREDICT question, calculates the distribution of answers
3. Outputs the top-3 most "confident" questions (one option >60%)
4. Outputs the top-3 most "uniform" questions (all options <20%)
5. Saves marginal frequencies to marginals.json"

Artifact: marginals.json + understanding where the model can win

Reinforcement: You understand which questions are "easy" and which are "hard"

Success criterion: The script works and outputs statistics

---

Stage 2.2: Simple model (CatBoost/XGBoost)

Actions:

Ask Qwen: "Write a script train_catboost.py that:

1. Loads TRAIN data from the sandbox
2. For each PREDICT question, trains CatBoost where: features are all GIVEN columns, target is the answer to the PREDICT question
3. Saves models to models/catboost_{question_name}.cbm
4. Outputs accuracy on validation"

Artifact: Trained models for each question

Reinforcement: The model takes into account individual respondent features

Success criterion: Models train and accuracy > marginal baseline

---

Stage 2.3: Integrating the model into main.py

Actions:

Ask Qwen: "Write main.py that:

1. At the module level loads all models from models/
2. In predict() for each PREDICT question: if there is a gate rule and it triggered → use it, otherwise use CatBoost
3. Last slot (gated_value) = probability from gate rule or 0.0
4. Returns vectors in the correct order"

Artifact: Working main.py with CatBoost

Reinforcement: The model uses respondent features + gate logic

Success criterion: Local score.py shows skill > gate rules