Inference over Training
The meta has shifted. Instead of spending millions of dollars pre-training larger models, labs are scaling Test-Time Compute.
If an LLM gets a complex math or logic problem wrong on the first autoregressive pass, we don't need a 100B parameter model to fix it. We just need a 7B model hooked up to a verifier, utilizing Monte Carlo Tree Search (MCTS) to explore multiple reasoning paths.
Implementing MCTS for LLMs
We implemented an AlphaGo-style search tree over the LLM's token generation space. The model generates 5 different possible "next steps" in a reasoning chain. A separate value network evaluates each step. The model explores the most promising branches.
def mcts_step(node, llm, value_network):
if not node.is_expanded:
# Generate N possible reasoning steps
children_texts = llm.generate_n_continuations(node.text, n=5)
for child_text in children_texts:
score = value_network.evaluate(child_text)
node.add_child(child_text, prior_prob=score)
return
# Select best child based on UCB (Upper Confidence Bound)
best_child = select_child(node)
mcts_step(best_child, llm, value_network)It takes 10x longer to generate an answer because the model is internally backtracking and verifying its own logic, but the accuracy on complex reasoning tasks jumped from 45% to 82%. This is the architecture of the future.