Core Concepts

​

Organizations

Each organization is typically associated with an individual company. Organizations each have their own billing configuration. Each organization can have multiple users, with the following roles: Admin, Write, or Read. An organization often has multiple apps within it.

​

Apps

Many apps can exist within an organization. Each app can be synced to a GitHub repository. Apps are isolated from each other, and each contain their own prompt templates, traces, metrics, api keys, etc. These can be used as staging/production/dev environments as well.

​

Branches

Each app corresponds to a default branch where prompt templates and other data gets saved to. This branch is also be used for reading configuration files. Enterprise users can lock the default branch, and opt for a PR workflow with preview branches instead of direct commits.

​

API Keys

Multiple api keys can exist within an App. API keys allow you to safely communicate with Puzzlet’s API for fetching prompt templates, tracing your workflows, etc.

​

Prompts

Prompts are where your prompts, components, and evals get stored in a seriliazed file format. These files get stored as AgentMark files. Instead of retrieving these as files from your repository, you can fetch them from our secure CDN. This allows you to iterate on your prompts separately from your code.

​

Traces

Traces are all of the steps a user goes through to go from input to output. Each individual step is counted as a span. For example, if a user invokes a prompt chain with 3 tool calls, then all of those steps would be part of the same trace.

​

Datasets

Datasets are a collection of data that you can use to improve your prompts. You can create datasets from a variety of sources, including your own data, public datasets, or traces.

​

Metrics

Metrics allow you to see at a high-level how users are using your application. This allows you to answer questions about cost, latency, model usage, users, etc. Metrics can also be filtered for a specific time period, model, etc.

​

Evals

Evals are a way to evaluate the quality of your prompts. These are coming soon to Puzzlet.