Skip to content
IQ

IQ Vexla

IQ Vexla lets your workspace try a model change on its own examples before you switch. You run a study, and the report shows what each model gains and loses against your current setup, in right answers and cost.

Use it when you are thinking about a cheaper model, a newer model or a new prompt, and you want evidence from your own work first. A study only produces a report. Your routing stays as it is.

Who can use it

| | | |---|---| | Plans | Team and Enterprise. | | Roles | Owners and admins can read results. Owners can run studies, and so can admins unless an owner turns off their Vexla tests permission. Members do not see studies. | | Admin permissions | By default, admins can create, start and stop studies and raise study budgets. An owner can turn off Vexla tests (create, start and stop studies) or Budgets and limits (raise a study budget) for an admin. | | Availability | Early access. Owners ask to join from the dashboard. | | Not available | Zero data retention workspaces. |

To change what an admin can do, an owner opens Settings, then Members and seats, and presses Permissions on the admin's row. The two switches are in the Budgets, Steward and Vexla section.

Early access

IQ Vexla is in early access. Workspace owners can ask to join from the dashboard.

  1. Open the dashboard and choose IQ Vexla in the sidebar. Only owners see this entry.
  2. On a Team or Enterprise workspace, press Ask for early access.
  3. The page confirms: "Request sent. We will email you at {email}." The email is the owner's sign-in email.

Asking shares your workspace name and your email with IQ. No prompts or model data are sent.

On a Free, trial or Personal workspace, the page says "Early access is for Team and Enterprise workspaces." with a See plans link. Admins and members do not see the IQ Vexla page.

If the request does not go through, the page says "That did not go through. Try again, or email sales@iq-routing.com."

Turn it on

Once your workspace is admitted, owners and admins use the same IQ Vexla sidebar entry. It opens Studies, with Saved evaluations alongside when saved evaluations are available.

If storage is not turned on, owners and admins with Vexla tests permission can turn it on in one step on the Studies list or New study page. Under "Storage is not turned on", press Allow evaluation data to be stored for 30 days. This setting also applies to saved evaluations. Zero data retention workspaces never see this control and cannot use studies.

Other study pages show "Storage is not turned on" with a Go to Saved evaluations link. See Data handling.

The study pages show this storage notice:

Access to study content ends within 30 days of study creation, or earlier if storage consent is withdrawn. Some operational records remain.

Studies

A study compares your current model with other models on the same examples. It moves through four steps: your examples, the plan, the run and the results.

1. Add your prompt and examples

On the Studies tab, press New study. Fill in:

  • Your current prompt: the prompt you use today.
  • What would you like to test?: describe what a good answer should do.
  • Your examples: paste JSONL, one example per line, or choose a JSONL file.
  • Answer format (optional, under Show details): the format an answer should follow. Only helper steps read it. The tested models do not see it, so put the format in your prompt too.

Each example needs a prompt and an expected_answer. You can add language and domain to each example, so the report can show how each group did.

Under Show details, Answer check says how answers are checked: "Answers are compared with your expected answers as exact text. Spaces and line breaks at the start and end are ignored. Capital letters and punctuation still count. Prompts are sent with no other instructions, so ask for the exact answer format in your prompt." Every new study uses this check. It ignores spaces and line breaks at the start and end of an answer, so "billing " with a space at the end is right when the expected answer is "billing". Capital letters and punctuation still count, so "Billing" or "billing." is wrong. Each study keeps the check it was created with, so an older study may still count those spaces. Write each expected answer exactly as a right answer should read. Grading is rule-based. No model grades the answers.

{"prompt": "Classify this ticket: My invoice shows the wrong amount.", "expected_answer": "billing", "language": "en", "domain": "support"}
{"prompt": "Classify this ticket: I cannot sign in on my phone.", "expected_answer": "account", "language": "en", "domain": "support"}
{"prompt": "Clasifica este ticket: Quiero cancelar mi plan.", "expected_answer": "cancellation", "language": "es", "domain": "support"}

A study takes 1 to 1,000 examples, up to 8 MiB in total. The file must be UTF-8. An example cannot carry its own answer check: every example is checked with the study's one check.

Press Prepare my study.

2. Review the plan

Choose what to compare:

  • Your current model: the model you use today.
  • Other models to test (optional): up to six other model IDs, separated by commas. Do not repeat your current model. Leave this blank to use the suggested list.

Press Review the plan. IQ Vexla checks which models your workspace can run with its own provider keys. The plan shows:

  • the examples every model will use;
  • your current model and the alternatives, with any Models left out and why;
  • the Cost at maximum answer length (it assumes every answer is as long as the Output limit in tokens allows) and the Maximum reservation;
  • Plan checks, including anything that blocks the plan from running.

You can also turn on Helper steps: Ask a helper to suggest alternatives (the shortlist guess) and Try prompt rewrites (the prompt rewrite). The plan names the helper model and the workspace key it uses. Data handling says what a helper step sends.

Price limit per 1,000 requests in dollars is optional. Leave it blank to use the current model's conservative maximum price quote.

Set Your spending limit: a limit for the whole study, including retries. It must be more than $0 and at most $100,000. If the plan has no helper steps, the limit must be at least its Maximum reservation. The workspace's monthly limit, $25 by default, also applies (see Limits).

3. Start and follow the run

Before starting, the plan shows this notice:

Study content is kept for 30 days. Prompt rewrites and shortlist guesses are written by a model, and results can be wrong. Study calls run on this workspace's own provider keys and are billed by your providers. See the Terms, section 8, and the Privacy Policy.

To start, check the box: "I approve this plan, its spending limit and sending the content described here to the model providers." Then press Start study.

When you run a study, the providers of the models used for its tests and any selected helper steps receive the study data needed for those steps under the workspace's provider credentials. Those providers bill your workspace for these calls as usual.

While the study runs, the Progress section shows how many planned checks are done. The study first compares the models, then checks the selected setup on the examples kept aside for the final check. While this page is open on your screen, the study moves on by itself when a stage ends, within the study budget, and gets the results after the last stage. To make it wait after each stage, check Wait after each stage until you press Continue study. Then press Continue study to start the next stage, or to get the results after the last. The box is not saved: when you open the page again, it starts unchecked, and a study that is waiting moves on at once. If you leave the page, a running stage still finishes, and the study waits there until you come back.

  • Stop study stops the run.
  • When the study cannot fund its next step, it pauses. Raise the budget with Raise budget, then Continue study. A study budget can only go up.

4. Read the results

When the study finishes, the Results section opens with one plain answer, for example:

  • "{model} stayed within your 10-point loss limit and is worth considering for this task."
  • "Keep your current setup for now."
  • "More testing is needed before choosing."
  • "This study does not support a model change."

Every result applies only to the task and examples in that study.

A model is worth considering only if the study shows, at your chosen confidence, that it is no more than your loss limit behind your current setup in right answers. You set the loss limit in the plan, under Show details, as Allowed difference in percentage points. The results call it Allowed loss. It is 10 points by default, and 10 is the most you can set. So a model that passes may still be up to that many percentage points behind your current setup in right answers.

IQ Vexla compares the answers on your examples. Under Show details, the Leaderboard lists every tested setup, with your current setup at the top. The columns include:

| Column | What it shows | |---|---| | Right answers (exact match) | How many of your examples each setup answered with an exact match to the expected answer. | | Change vs current | The difference from your current setup, in percentage points. | | Cost per 1,000 requests | The setup's estimated cost for 1,000 requests like your examples, at this study's settings. It does not include helper steps. | | Cost per right answer | Cost divided by right answers. | | Time to a valid answer, p50 / p95 | Speed. Studies do not measure timing, so this reads "Not measured". |

The report also says where the reference answers came from, for example "Reference answers come from your dataset."

Group checks show how each setup did for each language and domain in your examples. A group with too few examples is shown for inspection only.

Each note in the report is labeled Blocker, Warning, Coverage, Limit or Context, so you can see what held a result back.

Strictness preview shows how the result reads at another loss limit or confidence. It uses the saved results and sends nothing.

Export downloads the saved report. Choose Report, or Report and evidence. Both files include the system prompt. The evidence file also includes model answers and reference answers, so keep exports somewhere private.

A study does not change your routing. To act on a result, change your model or prompt yourself.

Data handling

If your workspace uses IQ Vexla, IQ stores the prompts, examples, reference answers, model outputs and reports used or produced in a study.

IQ Vexla requires workspace consent to store study content and is unavailable while zero data retention is enabled for the workspace.

When a study runs, IQ sends the system prompt and example input for each test to the provider of the model being tested, using the workspace's provider credentials. If the study uses a shortlist guess or prompt rewrite, IQ sends the helper model's provider the prompt and selected development examples, which may include reference answers, using the workspace's provider credentials. A prompt rewrite may also send the helper model's provider outputs from failed development examples. For models whose provider must count input tokens, IQ sends that provider the system prompt and example input for counting during the study, using the workspace's provider credentials.

IQ Vexla's prompt rewrites and shortlist guesses are written by models, and its scores and verdicts are estimates that can be wrong.

Study content expires 30 days after the study is created, including content from later runs or reassessments. After expiry, study content can no longer be read or used for new calls, and IQ removes it from the live store through a cleanup process that can be delayed. Deleting a study's data or withdrawing storage consent also starts removal of its content from the live store.

The dashboard has no control to delete a single study.

After study content is removed, IQ keeps study records such as model identifiers, costs, token usage, timing, status and right-or-wrong results for the life of the workspace. For a completed study, and for a stopped study when a report is produced, the workspace audit log also keeps the verdict label and a digest of the report under the audit-log retention rule.

Encrypted backups may still contain study content after it has been removed from the live store.

Active workspace owners and administrators can view study lists, status, reports and exports. Owners and administrators can also export study content, including answers, reference answers and the system prompt.

See the Privacy policy and Terms of service for data handling and provider billing.

Limits

| Limit | Value | |---|---| | Examples per study | 1 to 1,000 | | Upload size | 8 MiB per study | | Other models per study | Up to 6, besides your current model | | Study spending limit | More than $0, up to $100,000. It can be raised but not lowered. | | Workspace monthly limit | $25 by default per workspace per calendar month (UTC), for studies and saved evaluations together. | | Saved strictness previews | 200 per study | | Group labels | At most 100 values per label and 300 groups. A study whose report would be too large is refused before it starts. |

Errors

These are the messages the dashboard shows, and what to do.

| Message | What to do | |---|---| | "Studies need a Team plan." | Move the workspace to Team or Enterprise. See Billing. | | "Your role cannot do this. Ask an owner or admin." | Ask an owner. If you are an admin, ask an owner to turn Vexla tests back on for you, or Budgets and limits to raise a budget. | | "Storage is not turned on" | On the Studies list or New study page, an owner or admin with Vexla tests permission presses Allow evaluation data to be stored for 30 days. Zero data retention workspaces do not see this control. | | "That upload is too large. The limit is 8 MiB." | Use fewer or shorter examples. | | "Every case is checked with the study's one answer check. Remove the check from individual cases." | Remove the check field from your examples. | | "Confirm the notice about sending your examples to the model providers." | Check the approval box before you press Start study. | | "The plan changed. Review it again before you start." | Press Review the plan again, then start. | | "The budget is too small for this plan. Raise it and try again." | Set a higher spending limit. | | "The budget can only be raised." | Enter a limit at or above the current one. | | "The study budget is used up. Raise the budget to continue." | Use Raise budget, then Continue study. | | "Your workspace's monthly spending limit is reached." | Wait for the next calendar month. | | "Studies are busy right now. Try again in a moment." | Try again in a moment. If a saved evaluation run is active in the workspace, wait for it or stop it. | | "This study has reached its limit of saved previews." | Use a preview you already saved. | | "This study was not found." | Open the study again from the Studies tab. | | "Sign in again to continue." | Sign in again. | | "This view is unavailable right now. Try again." | Try again in a moment. |

See also