ScamWorld

An interactive environment for evaluating web agents against online fraud

Web agents now shop, pay and log in for their users. ScamWorld rebuilds 363 reported scam websites and 53 legitimate ones, across 16 scenarios, 38 fraud mechanisms and 26 languages, so you can watch what an agent does when the website in front of it is a scam.

Paper Code Docker image Browse all trajectories

Trajectories

Hand-checked episodes. Each tile replays what the agent saw, one screenshot at a time. Open one to read every step.

Same site, different choices

Two agents, the same website, the same warning sign. One stopped, the other went ahead.

Results

Deception and detection rates from the paper (Table 3)

On scam websites, six frontier-LLM agents took the risky final step in 11% to 35% of tasks. Read the paper.

Get started

Everything runs from one Docker image, offline.

docker pull wuyoscar666/scamworld:1.1

docker run --rm wuyoscar666/scamworld:1.1 tasks --scene shop

docker run --rm --network host -v "$PWD/results:/runs" wuyoscar666/scamworld:1.1 run shop-01 --model M --base-url http://localhost:8000/v1

Custom tasks Custom agents Model APIs Evaluators

Cite

@article{wu2026scamworld,
  title   = {ScamWorld: An Interactive Environment for Evaluating Web Agents Against Online Fraud},
  author  = {Wu, Yutao and Liu, Xiao and Wen, Ming and Huang, Hanxun and Wang, Jun and Ma, Xingjun},
  journal = {arXiv preprint arXiv:2611.04879},
  year    = {2026}
}