PromptChainer: Chaining Prompts with Visual Programming

https://cbarkinozer.medium.com/promptchainer-i%CC%87stemlerin-g%C3%B6rsel-programlama-ile-zincirlenmesi-5ec2dcd6d914

We review and summarize the article “PromptChainer: Chaining Large Language Model Prompts through Visual Programming”.

Abstract

Although LLMs make it possible to quickly prototype new machine learning functions, many real-world applications involve complex tasks that cannot be easily handled by the work of a single LLM. Recent studies have found that having multiple LLMs working together (with the output of one step as the input of another) can help users perceive these more complex tasks as more transparent and controllable. However, it is not yet fully understood what users need when writing their own LLM chains; This is an important step in reducing barriers for non-AI experts to prototype AI-infused applications.

In this study, we investigate the LLM chain writing process. Findings from pilot studies show that users need debugging support at multiple levels of detail in the chain, as well as data conversion between steps of a chain. To meet these needs, we designed PromptChainer, an interactive interface for visual programming chains. Case studies with four designers and developers show that PromptChainer supports prototyping for a variety of applications and addresses the questions we left open about supporting low-fi chain prototyping along with scaling chains to more complex tasks.

Summary

The use of large language models (LLMs) to prototype AI functions and the challenges encountered when combining multiple LLM studies are discussed. The authors propose to program LLM chains visually through an interactive interface called PromptChainer to meet the needs of users when writing their own chains. Findings from case studies with designers and developers using PromptChainer are highlighted, raising open questions about scaling chains to more complex tasks and supporting low-fi chain prototyping.

The challenges faced by researchers when prototyping complex applications using LLM chains are discussed. These challenges highlight the need for data transformation, the instability of LLM function signatures, and the possibility of cascading errors.

The authors propose an interface called PromptChainer, which includes a Chain View for creating and viewing chains, a Node View for writing individual steps in the chain, and support for chain debugging. The interface provides various types of nodes, including LLM nodes, helper nodes for data transformation and evaluation, custom JavaScript nodes, and communication nodes. PromptChainer provides examples of frequently created chains to help users develop a mental model of useful capabilities.

Node View allows users to review, implement and test individual nodes, with automatic updates to ensure consistency with quick edits. Interactive debugging functions are available at different levels of detail. The text describes a tool called PromptChainer, which allows users to chain multiple prompts together. It supports breakpoint debugging and allows users to feed a node's output before editing it. In the study, participants with prompt writing experience were assigned and allowed to create a chain using PromptChainer. Participants successfully create the chains they want, with an average of 5.5 nodes per chain. It reflects different patterns, such as chains, parallel branches of logic, and increased duplication in content. Participants used chaining to address single prompt limitations and make their prototypes more generalizable. PromptChainer supported various chaining strategies and enabled multi-level debugging. Participants took different approaches when creating their chains and met most of the chaining needs of predefined helpers. PromptChainer has been effective in helping users iteratively write and improve their chains. The text discusses PromptChainer, a tool designed to help users create chains of language models for a variety of tasks.

PromptChainer helps generate prompts and debug the chain when LLM steps have interactive effects. The study identifies challenges in LLM chain authoring, such as ensuring consistency between interdependent subtasks and tracking chains with complex logic. Future directions include supporting more complex chains and streamlining “half-baked” chain structures for rapid testing of different request structures.

Images

Figure 1: PromptChainer interface. (A) Chain View visualizes the chain structure with node edge diagrams (seen in detail in Figure 2) and allows users to edit the chain by adding, removing, or reconnecting nodes. (B) The Node View supports implementation, refinement, and testing of each node, for example, editing claims for LLM nodes. PromptChainer also supports running the chain end-to-end.

 Figure 2: An example chain for prototyping a music chatbot created by a pilot user (overview in Figure 1). We provide primary input-output examples and annotate node functions inline.

 Figure 3: A summary of node types, including core LLM nodes, slave nodes for data transformation and evaluation, and communication nodes for exchanging LLM data with external users or services.

Figure 4: An extension of Figure 2, related to music: (A) Node visualization: the node has a status icon (𝑎1), a list of named input (𝑎2) and output handles (𝑎3), as well as detailed data previews (𝑎4). (B) Implementation: handle names are synchronized with the base prompt template (𝑏1). We can debug nodes at multiple levels.

 Figure 5: Four different chains created by user research participants. Chains of P1 and P2 use parallel branching logic, while chains of P3 and P4 depict iterative content processing. Full details are in Figure 6, Appendix A.

Conclusion

We have identified three unique challenges for LLM chain writing that come with the versatile and open-ended capabilities of LLMs. To overcome these challenges, we designed PromptChainer and found that this tool helps users transform intermediate LLM output and debug the chain when LLM steps are interactive. Our work also revealed interesting future directions, including support for more complex chains as well as more explicit support for “half-baked” chain structures. So users were able to easily sketch a chain structure without spending too much time on pre-routing.

Resource

[1] Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, Carrie J Cai, (13 Mar 2022), PromptChainer: Chaining Large Language Model Prompts through Visual Programming, https://doi.org/10.48550/arXiv.2203.06566

[https://arxiv.org/abs/2203.06566]