From Resembling Humans to Predicting Humans: Insights from Three Papers on User Behavior Simulation

Predicting human behavior is super valuable research. In business, it can help companies optimize interfaces and products by spotting user behavior patterns early on to optimize the product. In social sciences, it helps researchers estimate how different crowds will react to policies and social issues, providing a reference for government policymaking.

The rise of Large Language Models (LLMs) has really pushed this field forward. Traditional models are better at handling pre-defined labels, like click or no click, buy or no buy. But LLMs can understand long texts, webpage structures, and multi-turn histories. They can even generate non-fixed labels like search queries and form contents, giving them a real shot at simulating open-ended human behaviors that traditional models struggle to cover.

Previously, the biggest breakout paper in this space was the 2023 paper by the Stanford team, Generative Agents: Interactive Simulacra of Human Behavior. That paper revolved around the memory mechanism of Agents, and its concepts of memory streams, retrieval, reflection, and planning are no longer strangers to most people today.

But the Stanford town mainly answered one question: Do the character behaviors generated by the model look like real people? The three papers I want to share today take this a step further: Can the model accurately predict what a real person will actually do?

The Three Papers

1. Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data — Yuxuan Lu, Jing Huang, Yan Han, Bingsheng Yao, Sisong Bei, Jiri Gesi, Yaochen Xie, Yisi Sang, Zheshen Wang, Qi He, Dakuo Wang (Northeastern University, Amazon)
2. Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning — Yimeng Zhang, Tian Wang, Jiri Gesi, Ziyi Wang, Yuxuan Lu, Jiacheng Lin, Sinong Zhan, Vianne Gao, Ruochen Jiao, Junze Liu, Kun Qian, Yuxin Tang, Ran Xue, Houyu Zhang, Qingjun Cui, Yufan Guo, Dakuo Wang (Michigan State University, Amazon, Northeastern University, UIUC, Northwestern University)
3. Finetuning LLMs for Human Behavior Prediction in Social Science Experiments — Akaash Kolluri, Shengguang Wu, Joon Sung Park, Michael S. Bernstein (Simile.ai, Stanford University)

The first two papers share several co-authors and both study users' sequential behavior in online shopping. The third paper shares co-authors with the Stanford town paper and looks at what kind of answers different user personas would give in social science experiments.

Just looking at a few words in the titles: LLM Agents, R1, and Finetuning LLMs. These pretty much sum up the routes these three papers took: directly using LLMs to simulate humans, using supervised fine-tuning (SFT) to make models learn from real human records, and using preference optimization or reinforcement learning (RL) to further tweak the models.

Before diving into the methods, though, we have to talk about the data. Because when doing experiments with LLMs, the easiest way to give yourself the illusion that "my model is doing great" is to mistake the model's vivid and seemingly highly logical answers for the model actually being able to make accurate predictions.

Two Completely Different Types of User Behavior Data

ShopCART Dataset

The first two papers study online shopping using the ShopCART dataset. This is Amazon's web trajectory data. A shopping trajectory (session) is made up of multiple steps of web behavior. Users usually type in a search query first, then might view a product, flip the page, adjust filters, tweak their search terms, and finally buy a product or close the page. Each step includes the HTML page the user was looking at right at that moment, as well as the actual action that took place.

And all actions are standardized into three formats:

• Click means clicking on a specific product. Filters, pagination buttons, or buy buttons: basically any clickable part of the page.
• Type_and_submit is typing open text into a search box and hitting submit.
• Terminate means ending the current shopping behavior and leaving or closing the page.

These three actions seem simple on the surface, but their difficulties are completely different. Terminate is obviously hard too, because sometimes you just don't know when a person will leave a webpage unless you see clear signals like checking out, but it's still relatively simple because it only has one fixed answer. For click, you have to find the exact element the real person clicked among dozens or even hundreds of elements on the current page. And type_and_submit is the hardest, because not only does the model have to judge whether the user wants to search, but it also has to generate the specific search text itself.

The data reported in the first paper includes 31,865 shopping sessions, 3,526 users, and 230,965 actions, with 4,432 sessions ultimately ending in a purchase and 27,433 sessions ending in termination.

The second paper, Shop-R1, uses the same shopping behavior data, but reports about 52,137 sessions, and constructed 176,949 training samples and 1,879 testing samples. Therefore, even though Shop-R1 and the first paper study similar tasks, you can't just put their numbers side by side to compare. Their data versions, training scales, and evaluation scopes aren't exactly the same.

SocSci210 Dataset

The third paper's data is totally different. It built its own SocSci210 dataset, and the construction of this dataset was a major piece of work in the paper itself. The final data includes 210 social science experiments, 400,491 participants, and about 2.9 million individual responses.

The authors downloaded 443 research projects from the public repository of Time-sharing Experiments for the Social Sciences, and after deduplication, 321 independent studies remained, ultimately keeping 210. The hard part here is that the data organization methods for different studies were completely different. Some raw data was in CSV, some in PDF or Word docs. Experimental conditions, survey questions, participant demographics, and answers could also be scattered across papers, codebooks, and multiple data files. The authors first standardized the tables into standard CSVs, converted PDFs and Word docs to text, and then handed each study's paper description, data files, codebook, and experimental materials over to a data-building Agent driven by o4-mini-high. This Agent identified the experimental conditions and outcome questions in the research, then automatically wrote and ran parsing code to merge and clean the raw files.

After going through this whole process, they finally got participant profiles (P), experimental conditions (c), outcome questions (o), and real human answers (r). So what the model ultimately learns is to predict r from P, c, and o: meaning it uses the participant's demographic profile, the experimental materials they saw, and the question they needed to answer. The output is a closed option, like an attitude score from 1 to 7, or Yes and No.

For example, a data point might be the profile of a 42-year-old female, conservative, high-income participant, who then sees a piece of news about the economic outlook, and finally, the task is to predict her level of concern about U.S. economic stability, scoring it from 1 to 5.

Key Distinction: ShopCART is multi-step, open-ended behavior prediction constrained by a webpage environment. SocSci210 is single-step, closed-ended answer prediction driven by demographic profiles and experimental conditions. The former has behavioral history but lacks corresponding user profiles. The latter has profiles, but lacks the continuously changing behavioral history of a person over a long session.

The First Paper: Answers That Look Human Don't Equal Accurate Predictions

Can LLM Agents Simulate Multi-Turn Human Behavior? first did something very simple but super important: it stopped letting people subjectively judge whether an Agent's trajectory was believable, and instead compared the model's generated next step with the real human's actual next step, item by item.

The model's input is the current webpage, previously viewed webpages, the history of trajectory actions, and synthetic rationales in the history. The model needs to output the next rationale and action.

The result was that the un-finetuned DeepSeek-R1 model only had an 11.86% accuracy rate for predicting the next action, and Claude 3.5 Sonnet v2 was at 11.69%. These models can generate a shopping trajectory that looks reasonable, but they have a really hard time restoring the exact step a real person actually took at that moment.

When you compare them, this difference is glaringly obvious. Un-finetuned models generally show an overly goal-oriented, task-completion style of behavior. They are more likely to stick to the initial search query, use filters, click on a product, and keep marching toward a purchase. But real human behavior patterns are often inefficient and filled with hesitation, repeated searches, clicking back and forth, and so on.

The paper gave two very intuitive examples. After a user searches for "disney gift," the real person's next step isn't clicking an existing product on the page, but changing the query to "disney gift card." After a user misspells "tee connector" as "tee conector," the real person's next step is to correct the spelling and search again. But in both of these cases, Claude chose to just directly click on a product on the results page instead of searching for the content again. From the perspective of completing the shopping task, these clicks aren't entirely unreasonable, and even seem more direct. But from the perspective of predicting what this real person will do next, these predictions are just wrong.

However, after finetuning Qwen2.5-7B on real behavior data, the accuracy jumped to 16.67%. When adding the rationales synthesized by Claude 3.5 Sonnet, the accuracy further reached 17.26%. This shows that when it comes to simulating real human behavior, finetuning actually works, and it tells us that LLMs have a systematic bias in their thinking patterns compared to real people. What we are after is reproducing the process of human behavior, not completing the final behavioral goal on behalf of humans.

The Second Paper: Why RL Rewarding Only Correct Behavior Isn't Enough

The second paper, Shop-R1, continues from the results of the first one. The paper found that SFT can already teach the model basic behavioral structures, but it still struggles to learn those high-entropy, long-tail actions. "High entropy" here refers to situations where there are a ton of possible reasonable answers, and "long-tail" means those very rare types of actions.

The standard move after SFT is to do reinforcement learning (RL) training. In RL training, if we only reward the steps predicted correctly, it's not enough. The reason is that predicting terminate is easy because the output is fixed. But predicting click means finding the precise target among a massive amount of page elements, so it's relatively hard. And the hardest one, search, requires generating open text. Since the difficulty of these three behaviors is completely disproportionate, in the early stages of training, almost all clicks and searches get no reward. The model has a really hard time knowing whether it messed up on the action type, the target element, or the search text.

Because of this, Shop-R1 splits the reinforcement learning reward into multiple tiers:

• If the output format is correct, it gets a format reward.
• If the action type is correct, it gets an action type reward.
• If the required fields for click or search are present, it gets a field reward.
• If the click target or search text is close to the real human's answer, it can get a partial reward based on text similarity.
• For harder long-text targets, it uses difficulty scaling to boost the reward, preventing the model from just learning to repeatedly output the easiest action, terminate.

Using Qwen2.5-3B to predict directly without any training yields an accuracy of only 0.32%. Doing binary reward RL straight from the base model gets 1.01%. SFT reaches 16.76%. Doing standard binary reward RL after SFT actually drops slightly to 16.55%. But Shop-R1's tiered rewards hit 27.72%.

These results tell us three things:

First, RL cannot replace SFT. The model must first learn the webpage structure, correct output formats, and the general shape of open text through real trajectories.

Second, rewards must tell the model where it went wrong. Sparse rewards based only on ultimate right or wrong cannot effectively train long-tail actions.

Third, the reward design itself changes the behavior the model learns. When rewards are unbalanced, the model will find the easiest shortcut to score points, instead of acting more like a human.

A Personal Note on Reward Design

My own understanding is that SFT relative to RL can be seen as giving RL a good starting point. Pure RL might have reward signals, but in complex tasks, it still has to explore among a massive number of possible actions. It might go a long time without encountering a single valid result, or it might exploit loopholes in the reward function. Using demonstration data for SFT or behavior cloning first allows the model to grasp the foundational skills that can be clearly described and demonstrated; then, RL can start from this initial policy and continue exploring and optimizing around the real goal. What SFT provides is a usable initial policy and behavioral prior, not completing the final learning on behalf of RL.

Speaking of reward design, I've recently been trying to get a reinforcement learning model to actually learn how to play a game called Rocket League. Before this, I knew a bit about the principles of RL and always naively thought that reinforcement learning was just about designing a reward function and then running tons of simulated training. But only after really getting my hands dirty did I realize that RL training is largely the art of task decomposition.

Take making the little cars hit the ball in Rocket League as an example. If you only provide a reward for scoring a goal right from the start, the model learns almost nothing. A randomly moving car is highly unlikely to accidentally complete the entire behavior chain of "finding the ball on the field, getting close to it, touching the ball, hitting it in the right direction, and continuously bumping it into the goal," so the vast majority of attempts will get zero reward. Not only does the model not know what the right thing to do is, it doesn't even know which previous step was closer to success than before.

The actually effective training method in real practice is more like real soccer training. You can first let the model learn to approach the ball quickly, then learn to touch the ball steadily, and then learn to push the ball toward the goal. Only after it masters these basic moves do you gradually introduce getting around opponents, passing, and complex physical play. This kind of curriculum learning, from simple to complex, can translate an initially extremely sparse ultimate goal into signals the model can learn step-by-step.

But task decomposition isn't just endlessly stacking partial rewards. If "touching the ball" always yields a high score, the model might just learn to repeatedly touch the ball to farm points instead of trying to score a goal. If the reward for "getting close to the ball" is too strong, it might even just hover around the ball doing nothing. Some rewards that are effective in early training might need to be swapped out later on; these rewards are only responsible for helping the model learn basic moves and quickly grasp some skills early on, and their influence as "scaffolding" rewards must be gradually reduced throughout the process.

Therefore, doing an RL training well requires understanding the algorithm, but also designing the rewards. More importantly, it requires understanding the task itself: knowing what sequence to break a skill down into, when to change the learning difficulty, how to spot the model exploiting reward loopholes, and how to confirm that improvements in intermediate metrics actually serve the ultimate goal. Shop-R1 breaking down the full action reward into tiers like format, action type, target, and text similarity is essentially solving the same problem: when the model can't yet complete the whole thing, first let it know exactly which little step it got right.

The Third Paper: Accurately Predicting a Single Behavior Isn't the Same as Restoring the Population Distribution

First, we have to remember that Finetuning LLMs for Human Behavior Prediction in Social Science Experiments isn't studying web operations, but rather questionnaire responses in social science experiments.

It trained Llama3-8B and Qwen2.5-14B, and compared three methods:

• The first is standard SFT, directly using the demographic profile, experimental materials, and question to predict the real human's answer.
• The second is SFT with synthetic reasoning. GPT-4o-mini generates an explanation that looks like pre-response thinking after seeing the real answer, and then that explanation and the answer are used together for training.
• The third is DPO. For the same experimental condition and question, it picks two real participants who gave different answers. When training on one profile, it uses that person's real answer as chosen, and the other person's real answer as rejected.

The results were that the Qwen2.5-14B base model had an individual accuracy of 72.9%, standard SFT was 69.5%, SFT with reasoning was 67.6%, and DPO was 74.0%.

But when you look at the population distribution, the conclusion flips. The Wasserstein distance for the base model was 0.205, standard SFT improved it to 0.151, while DPO was 0.181. The lower this metric is, the closer the model's generated response distribution is to real humans.

So we get this seemingly contradictory result: DPO scores higher on average individual accuracy, but SFT is better at restoring the response proportions of the whole population.

Why This Happens

Here we have to differentiate between what it means to predict proportions versus correctly predicting individual answers. For a multiple-choice question with only two options, let's say out of 10,000 people, 6,000 choose A and 4,000 choose B. This is the real human choice distribution.

If we have a completely random model that randomly picks 60% of people to answer A, and the remaining 40% to answer B, then we've got a model that can perfectly predict the choice distribution of the real population.

But the catch is, if we calculate the prediction accuracy for each individual, it's 52%:

60% x 60% + 40% x 40% = 0.52

However, if we have another model that just purely predicts everyone will choose A, its accuracy is 60%, which is higher than the previous model's 52%. But its predicted distribution is 100% for A and 0% for B, which is way off from the real 60/40 distribution.

Based on the experiments, we observed that while SFT performs worse than DPO in terms of accuracy, it's closer to the real population in terms of distribution. Actually, this result is mathematically expected, because DPO optimizes for one answer being ranked ahead of another, without requiring the model to preserve real frequencies (unless the sampling frequency of preference pairs happens to equal the real frequency). Meanwhile, the maximum likelihood objective of SFT is much closer to learning the complete conditional distribution in the training data.

What Conclusions Can We Draw By Putting These Three Papers Together?

1. The convincing stories spun by LLMs do not equal reliable predictions. LLMs are incredibly good at explaining a behavior in hindsight and are great at generating a seemingly reasonable explanation. But a behavior can have many reasonable explanations, and a single webpage could correspond to many reasonable next steps. A true prediction has to be benchmarked against actual behavior.
2. What SFT learns first is the behavioral prior of the population. Handing massive amounts of real human records to a model primarily serves to correct the general LLM's task-completion bias, letting it see that people will search, hesitate, go back, and quit. SFT is especially suited for learning basic behavioral frequencies, language formats, and group distributions.
3. DPO and GRPO can improve discriminability, but neither guarantees comprehensive correctness. DPO makes models more sensitive to differentiating between distinct user profiles. GRPO can break complex actions into learnable rewards. But the very nature of reinforcement learning might mean that the rewards we boost won't smoothly bring us the correct behavioral predictions and probability distributions.
4. The intrinsic value of external statistics. When predicting user behavior, it's super easy to get trapped in an internal perspective. The model sees a search term and a page of products, and starts analyzing which product this user ought to like. It might sound incredibly logical in its analysis, yet totally miss the most basic question: what do people historically in a similar position usually do?
5. Predict probability distributions, not deterministic certainty. The current crop of papers all try to directly predict a user's exact next action. But the reality is, actual human behavior is almost entirely unpredictable due to huge amounts of objective or subjective noise and random fluctuations. A much more reliable way to abstract real human behavior is as a probability distribution. Don't ask broadly whether a user will click this button or search this content. Instead, we can borrow from "Fermi estimation" to break down a massive, directly un-estimable problem into a series of small variables that can be estimated within the realm of common sense, and then combine them. For us, we need to break it down into a probability distribution over the actionable range of a page. Rather than saying we confirm a person's next step is to click the buy button, it's better to say we predict there's a 67% chance they'll click, a 20% chance they'll search again, and a 13% chance they might just quit directly.

Paper Links

• Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data — https://aclanthology.org/2026.acl-long.2034/
• Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning — https://openreview.net/forum?id=iS9tzosTtI
• Finetuning LLMs for Human Behavior Prediction in Social Science Experiments — https://aclanthology.org/2025.emnlp-main.1530/
• Generative Agents: Interactive Simulacra of Human Behavior — https://arxiv.org/abs/2304.03442