ToolEmu

Name: ToolEmu
Author: ryoungj

214stars22forks

How to install ToolEmu?

ryoungj/toolemu

$git clone https://github.com/ryoungj/ToolEmu

Installs into the current project.

Summary

<br>

Files · 1

View on GitHub

README.md

1	# ToolEmu: Identifying the Risks of LM Agents with an LM-Emulated Sandbox
2
3	<div align="center">
4	<img src="assets/figures/ToolEmu.png" width="50%">
5	</div>
6
7
8	<div align="center" style="font-size: 28px;">
9	[<a href="https://arxiv.org/abs/2309.15817">📃 Paper</a>]
10	[<a href="https://toolemu.com">🌐 Website</a>]
11	[<a href="https://demo.toolemu.com">🎮 Demo</a>]
12	[<a href="https://twitter.com/YangjunR/status/1708880142649676056">🐦 Twitter</a>]
13	</div>
14
15	<br>
16
17
18
19	Recent advances in Language Model (LM) agents and tool use, exemplified by applications like ChatGPT Plugins, enable a rich set of capabilities but also amplify potential risks—such as leaking private data or causing financial losses. Identifying these risks is labor-intensive, necessitating implementing the tools, manually setting up the environment for each test scenario, and finding risky cases. As tools and agents become more complex, the high cost of testing these agents will make it increasingly difficult to find high-stakes, long-tailed risks.
20	ToolEmu is an LM-based emulation framework that enables identifying and assessing such risks at scale, facilitating the development of safter LM agents.
21
22
23	This repo contains the code for:
24	* [testing LM agents with specific test cases in emulation](#running-specific-test-cases-in-emulation)
25	* [evaluating LM agents with our automatic evaluators and curated benchmark](#evaluating-lm-agents-with-our-benchmark)
26	* [curating your own toolkits and test cases](#curating-your-toolkits-and-test-cases)
27
28
29	The flexibility of ToolEmu makes it easy to curate new toolkits and test cases for testing LM agents.
30	We warmly invite you to enhance our benchmark by [contributing](#contributing) your curated toolkits and test cases!
31
32
33	## ToolEmu
34
35
36	https://github.com/ryoungj/ToolEmu/assets/12556773/149071ee-53c4-4697-9070-cc047a2deeae
37
38
39
40	ToolEmu assists in rapidly identifying realistic failures of LM agents across various tools and scenarios within an LM-emulated environment and facilitates the development of safer LM agents with LM-automated evaluations. It consists of 3 main components:
41
42	- Tool Emulators: ToolEmu uses a strong LM (e.g. GPT-4) to emulate the execution of tools in a virtual sandbox using only their specifications and inputs, without needing their implementations. This allows for faster prototyping of LM agents across different scenarios, while accommodating the evaluation of high-stakes tools that may lack existing APIs or sandbox implementations.
43	- Safety & Helpfulness Evaluators: To support scalable and quantitative risk assessments, ToolEmu includes an LM-based safety evaluator to automate the identification of potential failures caused by LM agents and quantifies the associated risk severities. To capture the potential tradeoff between safety and effectiveness, ToolEmu also includes an LM-based helpfulness evaluator.
44	- Curated Benchmark: ToolEmu ships with an initial benchmark covering 36 toolkits (311 tools) and 144 test cases for an quantitative evaluation of LM agents across various tools and scenarios. The scalablility of ToolEmu allows expanding it to more tools and scenarios.
45
46
47
48	<div align="center">
49	<img src="assets/figures/Framework.jpg" width="95%">
50	</div>
51
52	<!-- [Scalable Risk Identification of LM Agents with an LM-Emulated Sandbox](https://arxiv.org/)
53	<br />
54	[Yangjun Ruan](https://www.cs.toronto.edu/~yjruan/)\*,
55	[Honghua Dong](http://dhh1995.github.io)\*,
56	[Andrew Wang](https://scholar.google.com/citations?user=cxqJgC8AAAAJ&hl=en),
57	[Silviu Pitis](https://silviupitis.com/),
58	[Yongchao Zhou](https://scholar.google.com/citations?user=35M6rhsAAAAJ&hl=en),
59	[Jimmy Ba](https://jimmylba.github.io/),
60	[Yann Dubois](https://yanndubs.github.io/),
61	[Chris J. Maddison](https://www.cs.toronto.edu/~cmaddis/) and
62	[Tatsunori Hashimoto](https://thashim.github.io/)
63	<br />
64	(\*: indicates equal contribution.)
65	<br /> -->
66
67	## Setup
68
69	### Installation
70	To run our code

People who liked this also viewed

6 picks

Type

Resk-LLM Resk is a robust Python library designed to enhance security and manage context when interacting with LLMs. It provides a protective layer for API calls, safeguarding against common vulnerabilities and ensuring optimal performance. And safe layer again Prompt Injection. PromptsJun 202650022
agentic-ai-prompt-researchResearch into how agentic AI coding assistants work. Reconstructed prompt patterns, agent coordination, and security classificationPromptsMar 2026—2.5k
promptmapa security scanner for custom LLM applicationsPromptsDec 2025—1.2k
finite-monkey-engineAI engine for smart contract auditPromptsJan 2026—356
prompt-hacker-collectionsprompt attack-defense, prompt Injection, reverse engineering notes and examples | 提示词对抗、破解例子与笔记 PromptsFeb 2025—346
acipThe Advanced Cognitive Inoculation PromptPromptsMar 2026—329

ToolEmu

214stars22forks

View on GitHub

How to install ToolEmu?

ryoungj/toolemu

$git clone https://github.com/ryoungj/ToolEmu

Installs into the current project.

Summary

<br>

Files · 1

View on GitHub

README.md

1	# ToolEmu: Identifying the Risks of LM Agents with an LM-Emulated Sandbox
2
3	<div align="center">
4	<img src="assets/figures/ToolEmu.png" width="50%">
5	</div>
6
7
8	<div align="center" style="font-size: 28px;">
9	[<a href="https://arxiv.org/abs/2309.15817">📃 Paper</a>]
10	[<a href="https://toolemu.com">🌐 Website</a>]
11	[<a href="https://demo.toolemu.com">🎮 Demo</a>]
12	[<a href="https://twitter.com/YangjunR/status/1708880142649676056">🐦 Twitter</a>]
13	</div>
14
15	<br>
16
17
18
19	Recent advances in Language Model (LM) agents and tool use, exemplified by applications like ChatGPT Plugins, enable a rich set of capabilities but also amplify potential risks—such as leaking private data or causing financial losses. Identifying these risks is labor-intensive, necessitating implementing the tools, manually setting up the environment for each test scenario, and finding risky cases. As tools and agents become more complex, the high cost of testing these agents will make it increasingly difficult to find high-stakes, long-tailed risks.
20	ToolEmu is an LM-based emulation framework that enables identifying and assessing such risks at scale, facilitating the development of safter LM agents.
21
22
23	This repo contains the code for:
24	* [testing LM agents with specific test cases in emulation](#running-specific-test-cases-in-emulation)
25	* [evaluating LM agents with our automatic evaluators and curated benchmark](#evaluating-lm-agents-with-our-benchmark)
26	* [curating your own toolkits and test cases](#curating-your-toolkits-and-test-cases)
27
28
29	The flexibility of ToolEmu makes it easy to curate new toolkits and test cases for testing LM agents.
30	We warmly invite you to enhance our benchmark by [contributing](#contributing) your curated toolkits and test cases!
31
32
33	## ToolEmu
34
35
36	https://github.com/ryoungj/ToolEmu/assets/12556773/149071ee-53c4-4697-9070-cc047a2deeae
37
38
39
40	ToolEmu assists in rapidly identifying realistic failures of LM agents across various tools and scenarios within an LM-emulated environment and facilitates the development of safer LM agents with LM-automated evaluations. It consists of 3 main components:
41
42	- Tool Emulators: ToolEmu uses a strong LM (e.g. GPT-4) to emulate the execution of tools in a virtual sandbox using only their specifications and inputs, without needing their implementations. This allows for faster prototyping of LM agents across different scenarios, while accommodating the evaluation of high-stakes tools that may lack existing APIs or sandbox implementations.
43	- Safety & Helpfulness Evaluators: To support scalable and quantitative risk assessments, ToolEmu includes an LM-based safety evaluator to automate the identification of potential failures caused by LM agents and quantifies the associated risk severities. To capture the potential tradeoff between safety and effectiveness, ToolEmu also includes an LM-based helpfulness evaluator.
44	- Curated Benchmark: ToolEmu ships with an initial benchmark covering 36 toolkits (311 tools) and 144 test cases for an quantitative evaluation of LM agents across various tools and scenarios. The scalablility of ToolEmu allows expanding it to more tools and scenarios.
45
46
47
48	<div align="center">
49	<img src="assets/figures/Framework.jpg" width="95%">
50	</div>
51
52	<!-- [Scalable Risk Identification of LM Agents with an LM-Emulated Sandbox](https://arxiv.org/)
53	<br />
54	[Yangjun Ruan](https://www.cs.toronto.edu/~yjruan/)\*,
55	[Honghua Dong](http://dhh1995.github.io)\*,
56	[Andrew Wang](https://scholar.google.com/citations?user=cxqJgC8AAAAJ&hl=en),
57	[Silviu Pitis](https://silviupitis.com/),
58	[Yongchao Zhou](https://scholar.google.com/citations?user=35M6rhsAAAAJ&hl=en),
59	[Jimmy Ba](https://jimmylba.github.io/),
60	[Yann Dubois](https://yanndubs.github.io/),
61	[Chris J. Maddison](https://www.cs.toronto.edu/~cmaddis/) and
62	[Tatsunori Hashimoto](https://thashim.github.io/)
63	<br />
64	(\*: indicates equal contribution.)
65	<br /> -->
66
67	## Setup
68
69	### Installation
70	To run our code

People who liked this also viewed

6 picks

Type