wandb-best-score

---
name: workspace-filesystem
description: The virtual filesystem layout of this workspace.
---

# Virtual Filesystem

You work in a terminal over a virtual filesystem. Read files with shell
commands; service operations are listed in SKILL.md.

| Path | Description |
|---|---|
| `/workspace` | Agent workspace; current directory and home (`~`). |
| `/tmp` | Temporary files. |
| `/dev` | Device files: `null` and `zero` (mounted automatically). |
| `/wandb` | W&B experiment metadata and captured histories, read-only. |

---
name: workspace-commands
description: The shell commands and task tools available in this workspace.
---

# Available commands

Standard Unix commands, among them:

    cat  head  tail  ls  find  grep  wc  sort  uniq  cut  tr  sed  awk
    diff  tee  xargs  jq  unzip  curl

with pipes, `>` redirection, `$(...)` substitution and `$'...'`.

The task CLIs below are registered with this shell rather than installed
as files; do not inspect them with `which`, `file` or `/usr/bin`.

## wandb

Use `wandb <command> '<JSON arguments>'`. Run `wandb <command> --help` to
get more information.

| Command | Arguments | Description |
|---|---|---|
| `wandb query-weave-traces-tool` | `{entity_name: str, project_name: str, filters?: dict = {}, sort_by?: str = "started_at", sort_direction?: str = "desc", limit?: int = 10000000, include_costs?: bool = true, include_feedback?: bool = true, columns?: list = [], expand_columns?: list = [], truncate_length?: int = 200, return_full_data?: bool = false, metadata_only?: bool = false}` | Query Weave traces, trace metadata, and trace costs with filtering and sorting options. --- **Cost Calculation and Sorting Enhancements:** - For each model in the `costs` dictionary, a new field `total_cost` is computed as the sum of `completion_tokens_total_cost` and `prompt_tokens_total_cost`. - You can post-hoc sort traces by any of: `total_cost`, `completion_cost`, or `prompt_cost` (across all models, summed if multiple). --- <wandb_vs_weave_product_distinction> **IMPORTANT PRODUCT DISTINCTION:** W&B offers two distinct products with different purposes: 1. W&B Models: A system for ML experiment tracking, hyperparameter optimization, and model lifecycle management. Use `query_wandb_tool` for questions about: - Experiment runs, metrics, and performance comparisons - Artifact management and model registry - Hyperparameter optimization and sweeps - Project dashboards and reports 2. W&B Weave: A toolkit for LLM and GenAI application observability and evaluation. Use `query_weave_traces_tool` (this tool) for questions about: - Execution traces and paths of LLM operations - LLM inputs, outputs, and intermediate results - Chain of thought visualization and debugging - LLM evaluation results and feedback FYI: The Weigths & Biases platform is owned by Coreweave. If there are queries related to W&B, wandb or weave and Coreweave, they might be related to W&B products or features that leverage Coreweave's GPU or compute infrastructure. </wandb_vs_weave_product_distinction> <use_case_selector> **USE CASE SELECTOR - READ FIRST:** - For runs, metrics, experiments, artifacts, sweeps etc → use query_wandb_tool - For traces, LLM calls, chain-of-thought, LLM evaluations, AI agent traces, AI apps etc → use query_weave_traces_tool ===================================================================== ⚠️ TOOL SELECTION WARNING ⚠️ This tool is ONLY for WEAVE TRACES (LLM operations), NOT for run metrics or experiments! ===================================================================== **KEYWORD GUIDE:** If user question contains: - "runs", "experiments", "metrics" → Use query_wandb_tool - "traces", "LLM calls" etc → Use this tool **COMMON MISUSE CASES:** ❌ "Looking at metrics of my latest runs" - Do NOT use this tool, use query_wandb_tool instead ❌ "Compare performance across experiments" - Do NOT use this tool, use query_wandb_tool instead </use_case_selector> If the users asks for data about "runs" or "experiments" or anything about "experiment tracking" then use the `query_wandb_tool` instead. </use_case_selector> <usage_tips> query_traces_tool can return a lot of data, below are some usage tips for this function in order to avoid overwhelming a LLM's context window with too much data. <managing_llm_context_window> Returning all weave trace data can possibly result in overwhelming the LLM context window if there are 100s or 1000s of logged weave traces (depending on how many child traces each has) as well as resulting in a lot of data from or calls to the weave API. So, depending on the user query, consider doing the following to return enough data to answer the user query but not too much data that it overwhelms the LLM context window: - return only the root traces using the `trace_roots_only` boolean filter if you only need the top-level/parent traces and don't need the data from all child traces. For example, if a user wants to know the number of successful traces in a project but doesn't need the data from all child traces. Or if a user wants to visualise the number of parent traces over time. - return only the truncated values of the trace data keys in order to first give a preview of the data that can then inform more targeted weave trace queries from the user. in the extreme you can set `truncate_length` to 0 in order to only return keys but not the values of the trace data. - return only the metadata for all the traces (set `metadata_only = True`) if the query doesn't need to know anything about the structure or content of the individual weave traces. Note that this still requires requesting all the raw traces data from the weave API so can still result in a lot of data and/or a lot of calls being made to the weave API. - return only the columns needed using the `columns` parameter. In weave, the `inputs` and `output` columns of a trace can contain a lot of data, so avoiding returning these columns can help. Note you have to explicitly specify the columns you want to return if there are certain columns you don't want to return. Its almost always a good idea to specficy the columns needed. <returning_metadata_only> If `metadata_only = True` this returns only metadata of the traces such as trace counts, token counts, trace types, time range, status counts and distribution of op names. if `metadata_only = False` the trace data is returned either in full or truncated to `truncate_length` characters depending if `return_full_data = True` or `False` respectively. </returning_metadata_only> <truncating_trace_data_values> If `return_full_data = False` the trace data is truncated to `truncate_length` characters, default 200 characters. Otherwise the trace data is returned in full. </truncating_trace_data_values> Remember, LLM context window is precious, only return the minimum amount of data needed to complete an analysis. </managing_llm_context_window> <usage_guidance> - Exploratory queries: For generic exploratory or initial queries about a set of weave traces in a project it can be a good idea to start with just returning metadata or truncated data. Consider asking the user for clarification and warn them that returning a lot of weave traces data might overwhelm the LLM context window. No need to warn them multiple times, just once is enough. - Project size: Consider using the count_traces_tool to get an estimate of the number of traces in a project before querying for them as query_trace_tool can return a lot of data. - Partial op name matching: Use the `op_name_contains` filter if a users has only given a partial op name or if they are unsure of the exact op name. - Weave Evaluations: If asked about weave evaluations or evals traces: - Evals are complicated to query, prompt the user with follow up questions if needed. - First, always try and oritent yourself - pull a summary of the evaluation, get all of the top level column names in the eval and always get a count of the total number of child traces in this eval by filtering by parent_id and using the count_traces tool. - As part of orienting yourself, just pull a subset of child traces from the eval, maybe 3 to 5, to understand the column structure and values. - Always be explicit about the amount of data returned and limits used in your query - return to the user the count of traces analysed. - Always stay filterd on the evaluation id (filter by `parent_id`) unless specifically asked questions across different evaulations, e.g. if a parent id (or parentId) is provided then ensure to use that filter in the query. - filter for traces with `op_name_contains = "Evaluation.evaluate"` as a first step. These ops are parent traces that contain aggregated stats and scores about the evaluation. The child traces of these ops are the actual evaluation results for each sample in an evaluation dataset. If asked about individual rows in an evaluation then use the parent_ids filter to return the child traces. - for questions where both a child call name of an evaluation and an evaluation id or name are provided, always ensure that you first correctly get the evaluation id, and then use it as the parent_id in the query for the child traces. Otherwise there is a risk of returning traces that do not belong to the evaluation that was given. - Weave nomenclature: Note that users might refer to weave ops as "traces" or "calls" or "traces" as "ops". </usage_guidance> Parameters ---------- entity_name : str The Weights & Biases entity name (team or username) project_name : str The Weights & Biases project name filters : dict Dict of filter conditions, supporting: - display_name : str or regex pattern Filter by display name seen in the Weave UI - op_name : str or regex pattern Filter by weave op name, a long URI starting with 'weave:///' - op_name_contains : str Filter for op_name containing this substring (easier than regex) - trace_roots_only : bool Boolean to filter for only top-level/parent traces. Useful when you don't need to return the data from all child traces. - trace_id : str Filter by a specific `trace_id` (e.g., "01958ab9-3c67-7c72-92bf-d023fa5a0d4d"). A `trace_id` groups multiple calls/spans. Use if the user explicitly say they provided a "trace_id" for a group of operations. Always first try to filter by `call_ids` if a user provides an ID, before trying to filter by `trace_id`. - call_ids : str or list of str Filter by specific `call_id`s (also known as Span IDs) (string or list of strings, e.g., ["01958ab9-3c68-7c23-8ccd-c135c7037769"]). **GUIDANCE**: `call_id` (Span ID) identifies a *single* operation/span and is typically found in Weave UI URLs. If a user provides an ID for a specific item they're viewing, **prefer `call_ids`**. Format as a list: `{"call_ids": ["user_provided_id"]}`. - parent_ids : str or list of str Return traces that are children of the given parent trace ids (string or list of strings). Ensure you use this if given an evaluation trace id or name. - status : str Filter by trace status, defined as whether or not the trace had an exception or not. Can be `success` or `error`. NOTE: When users ask for "failed", "wrong", or "incorrect" traces, use `status:'error'` or `has_exception:True` as the filter. - time_range : dict Dict with "start" and "end" datetime strings. Datetime strings should be in ISO format (e.g. `2024-01-01T00:00:00Z`) - attributes : dict Dict of the weave attributes of the trace. Supports nested paths (e.g., "metadata.model_name") via dot notation. Value can be: * A literal for exact equality (e.g., `"status": "success"`) * A dictionary with a comparison operator: `$gt`, `$lt`, `$eq`, `$gte`, `$lte` (e.g., `{"token_count": {"$gt": 100}}`) * A dictionary with the `$contains` operator for substring matching on string attributes (e.g., `{"model_name": {"$contains": "gpt-3"}}`) **Warning:** The `$contains` operator performs simple substring matching only, full regular expression matching (e.g., via `$regex`) is **not supported** for attributes. Do not attempt to use `$regex`. - has_exception : bool, optional Optional[bool] to filter traces by exception status: - None (or key not present): Show all traces regardless of exception status - True: Show only traces that have exceptions (exception field is not null) - False: Show only traces without exceptions (exception field is null) sort_by : str, optional Field to sort by (started_at, ended_at, op_name, etc.). Defaults to 'started_at' sort_direction : str, optional Sort direction ('asc' or 'desc'). Defaults to 'desc' limit : int, optional Maximum number of results to return. Defaults to None include_costs : bool, optional Include tracked api cost information in the results. Defaults to True include_feedback : bool, optional Include weave annotations (human labels/feedback). Defaults to True columns : list of str, optional List of specific columns to include in the results. Its almost always a good idea to specficy the columns needed. Defaults to None (all columns). Available columns are: id: <class 'str'> project_id: <class 'str'> op_name: <class 'str'> display_name: typing.Optional[str] trace_id: <class 'str'> parent_id: typing.Optional[str] started_at: <class 'datetime.datetime'> attributes: dict[str, typing.Any] inputs: dict[str, typing.Any] ended_at: typing.Optional[datetime.datetime] exception: typing.Optional[str] output: typing.Optional[typing.Any] summary: typing.Optional[SummaryMap] # Contains nested data like 'summary.weave.status' and 'summary.weave.latency_ms' status: typing.Optional[str] # Synthesized from summary.weave.status if requested latency_ms: typing.Optional[int] # Synthesized from summary.weave.latency_ms if requested wb_user_id: typing.Optional[str] wb_run_id: typing.Optional[str] deleted_at: typing.Optional[datetime.datetime] expand_columns : list of str, optional List of columns to expand in the results. Defaults to None truncate_length : int, optional Maximum length for string values in weave traces. Defaults to 200 return_full_data : bool, optional Whether to include full untruncated trace data. If True, the `truncate_length` parameter is ignored. If `False` returns truncation_length = 0, no values for the column keys are returned. Defaults to True. metadata_only : bool, optional Return only metadata without traces. Defaults to False Returns ------- str JSON string containing either full trace data or metadata only, depending on parameters <examples> ```python<br>    # Get an overview of the traces in a project<br>    query_traces_tool(<br>        entity_name="my-team",<br>        project_name="my-project",<br>        filters={"root_traces_only": True},<br>        metadata_only=True,<br>        return_full_data=False<br>    )<br><br>    # Get failed traces with costs and feedback<br>    query_traces_tool(<br>        entity_name="my-team",<br>        project_name="my-project",<br>        filters={"status": "error"},<br>        include_costs=True,<br>        include_feedback=True<br>    )<br><br>    # Get specific columns for traces who's op name (i.e. trace name) contains a specific substring<br>    query_traces_tool(<br>        entity_name="my-team",<br>        project_name="my-project",<br>        filters={"op_name_contains": "Evaluation.summarize"},<br>        columns=["id", "op_name", "started_at", "costs"]<br>    )<br>    ``` </examples> |
| `wandb count-weave-traces-tool` | `{entity_name: str, project_name: str, filters?: dict = null}` | count Weave traces and return the total storage size in bytes for the given filters. Use this tool to query data from Weights & Biases Weave, an observability product for tracing and evaluating LLMs and GenAI apps. This tool only provides COUNT information and STORAGE SIZE (bytes) about traces, not actual logged traces data, metrics or run data. <tool_choice_guidance> <wandb_vs_weave_product_distinction> **IMPORTANT PRODUCT DISTINCTION:** W&B offers two distinct products with different purposes: 1. W&B Models: A system for ML experiment tracking, hyperparameter optimization, and model lifecycle management. Use `query_wandb_tool` for questions about: - Experiment runs, metrics, and performance comparisons - Artifact management and model registry - Hyperparameter optimization and sweeps - Project dashboards and reports 2. W&B Weave: A toolkit for LLM and GenAI application observability and evaluation. Use `query_weave_traces_tool` (this tool) for questions about: - Execution traces and paths of LLM operations - LLM inputs, outputs, and intermediate results - Chain of thought visualization and debugging - LLM evaluation results and feedback </wandb_vs_weave_product_distinction> <use_case_selector> **USE CASE SELECTOR - READ FIRST:** - For runs, metrics, experiments, artifacts, sweeps etc → use query_wandb_tool - For traces, LLM calls, chain-of-thought, LLM evaluations, AI agent traces, AI apps etc → use query_weave_traces_tool ===================================================================== ⚠️ TOOL SELECTION WARNING ⚠️ This tool is ONLY for WEAVE TRACES (LLM operations), NOT for run metrics or experiments! ===================================================================== **KEYWORD GUIDE:** If user question contains: - "runs", "experiments", "metrics" → Use query_wandb_tool - "traces", "LLM calls" etc → Use this tool **COMMON MISUSE CASES:** ❌ "Looking at metrics of my latest runs" - Do NOT use this tool, use query_wandb_tool instead ❌ "Compare performance across experiments" - Do NOT use this tool, use query_wandb_tool instead </use_case_selector> </tool_choice_guidance> Returns the total number of traces in a project and the number of root (i.e. "parent" or top-level) traces. This is more efficient than query_trace_tool when you only need the count. This can be useful to understand how many traces are in a project before querying for them as query_trace_tool can return a lot of data. Parameters ---------- entity_name : str The Weights & Biases entity name (team or username). project_name : str The Weights & Biases project name. filters : Dict[str, Any], optional Dict of filter conditions, supporting: - display_name: Filter by display name (string or regex pattern) - op_name_contains: Filter for op_name containing a substring. Not a good idea to use in conjunction with trace_roots_only. - trace_id: Filter by specific trace ID - status: Filter by trace status ('success', 'error', etc.) - time_range: Dict with "start" and "end" datetime strings - latency: Filter by latency in milliseconds (summary.weave.latency_ms). Use a nested dict with operators: $gt, $lt, $eq, $gte, $lte. ($lt and $lte are implemented via logical negation on the backend). e.g., {"latency": {"$gt": 5000}} - attributes: Dict of attribute path and value/operator to match. Supports nested paths (e.g., "metadata.model_name") via dot notation. Value can be literal for equality or a dict with operator ($gt, $lt, $eq, $gte, $lte) for comparison (e.g., {"token_count": {"$gt": 100}}). - has_exception: Boolean to filter traces with/without exceptions - trace_roots_only: Boolean to filter for only top-level (aka parent) traces Returns ------- int The number of traces matching the query parameters. Examples -------- >>> # Count failed traces >>> count = count_traces( ... entity_name="my-team", ... project_name="my-project", ... filters={"status": "error"} ... ) >>> # Count traces faster than 500ms >>> count = count_traces( ... entity_name="my-team", ... project_name="my-project", ... filters={"latency": {"$lt": 500}} ... ) |
| `wandb query-wandb-tool` | `{query: str, variables?: dict = null, max_items?: int = 100, items_per_page?: int = 20}` | Execute an arbitrary GraphQL query against the Weights & Biases (W&B) Models API. Use this tool to query data from Weights & Biases Models features, including experiment tracking runs, model registry, reports, artifacts, sweeps. <wandb_vs_weave_product_distinction> **IMPORTANT PRODUCT DISTINCTION:** W&B offers two distinct products with different purposes: 1. W&B Models: A system for ML experiment tracking, hyperparameter optimization, and model lifecycle management. Use `query_wandb_tool` for questions about: - Experiment runs, metrics, and performance comparisons - Artifact management and model registry - Hyperparameter optimization and sweeps - Project dashboards and reports 2. W&B Weave: A toolkit for LLM and GenAI application observability and evaluation. Use `query_weave_traces_tool` for questions about: - Execution traces and paths of LLM operations - LLM inputs, outputs, and intermediate results - Chain of thought visualization and debugging - LLM evaluation results and feedback FYI: The Weigths & Biases platform is owned by Coreweave. If there are queries related to W&B, wandb or weave and Coreweave, they might be related to W&B products or features that leverage Coreweave's GPU or compute infrastructure. </wandb_vs_weave_product_distinction> <use_case_selector> **USE CASE SELECTOR - READ FIRST:** - For runs, metrics, experiments, artifacts, sweeps etc → use query_wandb_tool (this tool) - For traces, LLM calls, chain-of-thought, LLM evaluations, AI agent traces, AI apps etc → use query_weave_traces_tool ===================================================================== ⚠️ TOOL SELECTION WARNING ⚠️ This tool is ONLY for WANDB MODELS DATA (MLOps), NOT for LLM TRACES or GENAI APPS! ===================================================================== **KEYWORD GUIDE:** If user question contains: - "runs", "experiments", "metrics" → Use query_wandb_tool (this tool) - "traces", "LLM calls" etc → Use query_weave_traces_tool **COMMON MISUSE CASES:** ❌ "Looking at performance of my latest weave evals" - Use query_weave_traces_tool ❌ "what system prompt was used for my openai call" - Use query_weave_traces_tool ❌ "Show me the traces for my weave evals" - Use query_weave_traces_tool <query_analysis_step> **STEP 1: ANALYZE THE USER QUERY FIRST!** Before constructing the GraphQL query, determine how the user is referring to W&B entities, especially runs: - Is the user providing a short, 8-character **Run ID** (e.g., `gtng2y4l`, `h0fm5qp5`)? - Or are they providing a longer, human-readable **Display Name** (e.g., `transformer_train_run_123`, `eval_on_benchmark_v2`)? Your choice of query structure depends heavily on this analysis (see Key Concepts and Examples below). </query_analysis_step> <key_concepts> **KEY CONCEPTS - READ CAREFULLY:** * **Run ID vs. Display Name:** * To fetch a **single, specific run** using its unique 8-character ID (e.g., `gtng2y4l`), use the `run(name: $runId)` field. The variable `$runId` MUST be the ID, not the display name. * To **find runs based on their human-readable `displayName`** (e.g., `my-cool-experiment-1`), use the `runs` collection field with a `filters` argument like: `runs(filters: "{\"displayName\":{\"$eq\":\"my-cool-experiment-1\"}}")`. This might return multiple runs if display names are not unique. * **Filters require JSON Strings:** When using the `filters` argument (e.g., for `runs`, `artifacts`), the value provided in the `variables` dictionary MUST be a JSON formatted *string*. Use `json.dumps()` in Python to create it. * **Collections Require Pagination Structure:** Queries fetching lists/collections (like `project.runs`, `artifact.files`) MUST include the `edges { node { ... } } pageInfo { endCursor hasNextPage }` pattern. * **Summary Metrics:** Use the `summaryMetrics` field (returns a JSON string) to access a run's summary dictionary, not the deprecated `summary` field. </key_concepts> This function allows interaction with W&B data (Projects, Runs, Artifacts, Sweeps, Reports, etc.) using the GraphQL query language. Parameters ---------- query : str he GraphQL query string. This defines the operation (query/mutation), the data to fetch (selection set), and any variables used. variables : dict[str, Any] \| None, optional A dictionary of variables to pass to the query. Keys should match variable names defined in the query (e.g., $entity, $project). Values should match the expected types (String, Int, Float, Boolean, ID, JSONString). **Crucially, complex arguments like `filters` MUST be provided as a JSON formatted *string*. Use `json.dumps()` in Python to create this string.** max_items : int, optional Maximum number of items to fetch across all pages. Default is 100. items_per_page : int, optional Number of items to request per page. Default is 50. Returns ------- Dict[str, Any] The aggregated GraphQL response dictionary. <critical_warning> **⚠️ CRITICAL WARNING: Run ID vs. Display Name ⚠️** If the user query mentions a run using its **long, human-readable name** (Display Name), you **MUST** use the `runs(filters: ...)` approach shown in the examples. **DO NOT** use `run(name: ...)` with a Display Name; it will fail because `name` expects the short Run ID. Use `run(name: ...)` **ONLY** when the user provides the 8-character Run ID. Review the "Minimal Example: Run ID vs Display Name" and "Get Run by Display Name" examples carefully. </critical_warning> <required_pagination_structure> **⚠️ REQUIRED PAGINATION STRUCTURE ⚠️** All collection queries MUST include the complete W&B connection pattern with these elements: 1. `edges` array containing nodes 2. `node` objects inside edges containing your data fields 3. `pageInfo` object with: - `endCursor` field (to enable pagination) - `hasNextPage` field (to determine if more data exists) This is a strict requirement enforced by the pagination system. Queries without this structure will fail with the error "Query doesn't follow the W&B connection pattern." Example of required pagination structure for any collection: ```graphql<br>runs(first: 10) {  # or artifacts, files, etc.<br>    edges {<br>    node {<br>        id<br>        name<br>        # ... other fields you need<br>    }<br>    # cursor # Optional: include cursor if needed for specific pagination logic<br>    }<br>    pageInfo {<br>    endCursor<br>    hasNextPage<br>    }<br>}<br>``` </required_pagination_structure> <llm_context_window_management> **LLM CONTEXT WINDOW MANAGEMENT** The results of this tool are returned to a LLM. Be mindful of the context window of the LLM! <warning_about_open_ended_queries> **WARNING: AVOID OPEN-ENDED QUERIES!** Open-ended queries should be strictly avoided when: - There are a lot of runs in the project (e.g., hundreds or thousands) - There are runs with large amounts of data (e.g., many metrics, large configs, etc.) Examples of problematic open-ended queries: - Requesting all runs in a project without limits - Requesting complete run histories without filtering specific metrics - Requesting all files from artifacts without specifying names/types Instead, always: - Use the `first` parameter to limit the number of items returned (start small, e.g., 5-10) - Apply specific filters to narrow down results (e.g., state, creation time, metrics) - Request only the specific fields needed, avoid selecting everything - Consider paginating results if necessary (don't request everything at once) Bad: ```graphql<br>query AllRuns($entity: String!, $project: String!) {<br>    project(name: $project, entityName: $entity) {<br>    # Potentially huge response: requests all fields for all runs<br>    runs { edges { node { id name state history summaryMetrics config files { edges { node { name size }}}}}}}<br>    }<br>}<br>``` Good: ```graphql<br>query LimitedRuns($entity: String!, $project: String!) {<br>    project(name: $project, entityName: $entity) {<br>    # Limits runs, specifies filters, and selects only necessary fields<br>    runs(first: 5, filters: "{\"state\":\"finished\"}") {<br>        edges { <br>        node { <br>            id <br>            name <br>            createdAt <br>            summaryMetrics # Get summary JSON, parse later if needed<br>        } <br>        }<br>        pageInfo { endCursor hasNextPage } # Always include pageInfo for collections<br>    }<br>    }<br>}<br>``` </warning_about_open_ended_queries> Some tactics to consider to avoid exceeding the context window of the LLM when using this tool: - First return just metadata about the wandb project or run you will be returning. - Select only a subset of the data such as just particular columns or rows. - If you need to return a large amount of data consider using the `query_wandb_tool` in a loop - Break up the query into smaller chunks. If you are returning just a sample subset of the data warn the user that this is a sample and that they should use the tool again with additional filters or pagination to get a more complete view. </llm_context_window_management> **Constructing GraphQL Queries:** 1. **Operation Type:** Start with `query` for fetching data or `mutation` for modifying data. 2. **Operation Name:** (Optional but recommended) A descriptive name (e.g., `ProjectInfo`). 3. **Variables Definition:** Define variables used in the query with their types (e.g., `($entity: String!, $project: String!)`). `!` means required. 4. **Selection Set:** Specify the fields you want to retrieve, nesting as needed based on the W&B schema. **W&B Schema Overview:** * **Core Types:** `Entity`, `Project`, `Run`, `Artifact`, `Sweep`, `Report`, `User`, `Team`. * **Relationships:** Entities contain Projects. Projects contain Runs, Sweeps, Artifacts. Runs use/are used by Artifacts. Sweeps contain Runs. * **Common Fields:** `id`, `name`, `description`, `createdAt`, `config` (JSONString), `summaryMetrics` (JSONString - **Note:** use this field, not `summary`, to access the run's summary dictionary as a JSON string), `historyKeys` (List of String), etc. * **Connections (Lists):** Many lists (like `project.runs`, `artifact.files`) use a connection pattern: ```graphql<br>    runs(first: Int, after: String, filters: JSONString, order: String) {<br>        edges { node { id name ... } cursor }<br>        pageInfo { hasNextPage endCursor }<br>    }<br>    ``` Use `first` for limit, `after` with `pageInfo.endCursor` for pagination, `filters` (as a JSON string) for complex filtering, and `order` for sorting. * **Field Type Handling:** - Some fields require subfield selection (e.g., `tags { name }`) while others are scalar (e.g., `historyKeys`). - Check the schema if you get errors like "must have a selection of subfields" or "must not have a selection". **Query Examples:** <!-- WANDB_GQL_EXAMPLE_START name=MinimalRunIdVsDisplayName --> * **Minimal Example: Run ID vs Display Name:** * **A) User provides Run ID (e.g., "get info for run h0fm5qp5"):** ```graphql<br>        query GetRunById($entity: String!, $project: String!, $runId: String!) {<br>          project(name: $project, entityName: $entity) {<br>            # Use run(name: ...) with the Run ID<br>            run(name: $runId) {<br>              id<br>              name # This will be the Run ID<br>              displayName # This is the human-readable name<br>            }<br>          }<br>        }<br>        ``` ```python<br>        variables = {"entity": "...", "project": "...", "runId": "h0fm5qp5"}<br>        ``` * **B) User provides Display Name (e.g., "get info for run transformer_train_123"):** ```graphql<br>        # Note: Querying *runs* collection and filtering<br>        query GetRunByDisplayNameMinimal($project: String!, $entity: String!, $displayNameFilter: JSONString) {<br>          project(name: $project, entityName: $entity) {<br>            # Use runs(filters: ...) with the Display Name<br>            runs(first: 1, filters: $displayNameFilter) {<br>              edges {<br>                node {<br>                  id<br>                  name # Run ID<br>                  displayName # Display Name provided by user<br>                }<br>              }<br>              pageInfo { endCursor hasNextPage } # Required for collections<br>            }<br>          }<br>        }<br>        ``` ```python<br>        import json<br>        variables = {<br>            "entity": "...",<br>            "project": "...",<br>            "displayNameFilter": json.dumps({"displayName": {"$eq": "transformer_train_123"}})<br>        }<br>        ``` <!-- WANDB_GQL_EXAMPLE_END name=MinimalRunIdVsDisplayName --> <!-- WANDB_GQL_EXAMPLE_START name=GetProjectInfo --> * **Get Project Info:** (Doesn't retrieve a collection, no pagination needed) ```graphql<br>    query ProjectInfo($entity: String!, $project: String!) {<br>        project(name: $project, entityName: $entity) {<br>        id<br>        name<br>        entityName<br>        description<br>        runCount<br>        }<br>    }<br>    ``` ```python<br>    variables = {"entity": "my-entity", "project": "my-project"}<br>    ``` <!-- WANDB_GQL_EXAMPLE_END name=GetProjectInfo --> <!-- WANDB_GQL_EXAMPLE_START name=GetSortedRuns --> * **Get Sorted Runs:** (Retrieves a collection, requires pagination structure) ```graphql<br>    query SortedRuns($project: String!, $entity: String!, $limit: Int, $order: String) {<br>        project(name: $project, entityName: $entity) {<br>        runs(first: $limit, order: $order) {<br>            edges {<br>            node { id name displayName state createdAt summaryMetrics }<br>            cursor # Optional cursor<br>            }<br>            pageInfo { # Required for collections<br>            hasNextPage<br>            endCursor<br>            }<br>        }<br>        }<br>    }<br>    ``` ```python<br>    variables = {<br>        "entity": "my-entity",<br>        "project": "my-project",<br>        "limit": 10,<br>        "order": "+summary_metrics.accuracy"  # Ascending order by accuracy<br>        # Use "-createdAt" for newest first (default if order omitted)<br>        # Use "+createdAt" for oldest first<br>    }<br>    ``` <!-- WANDB_GQL_EXAMPLE_END name=GetSortedRuns --> <!-- WANDB_GQL_EXAMPLE_START name=GetFilteredRuns --> * **Get Runs with Pagination and Filtering:** (Requires pagination structure) ```graphql<br>    query FilteredRuns($project: String!, $entity: String!, $limit: Int, $cursor: String, $filters: JSONString, $order: String) {<br>        project(name: $project, entityName: $entity) {<br>        runs(first: $limit, after: $cursor, filters: $filters, order: $order) {<br>            edges {<br>            node { id name state createdAt summaryMetrics }<br>            cursor # Optional cursor<br>            }<br>            pageInfo { endCursor hasNextPage } # Required<br>        }<br>        }<br>    }<br>    ``` ```python<br>    # Corrected: Show filters as the required escaped JSON string<br>    variables = {<br>        "entity": "my-entity",<br>        "project": "my-project",<br>        "limit": 10,<br>        "order": "-summary_metrics.accuracy",  # Optional: sort<br>        "filters": "{"state": "finished", "summary_metrics.accuracy": {"$gt": 0.9}}", # Escaped JSON string<br>        # "cursor": previous_pageInfo_endCursor # Optional for next page<br>    }<br>    # Note: The *content* of the `filters` JSON string must adhere to the specific <br>    # filtering syntax supported by the W&B API (e.g., using operators like `$gt`, `$eq`, `$in`). <br>    # Refer to W&B documentation for the full filter specification.<br>    ``` <!-- WANDB_GQL_EXAMPLE_END name=GetFilteredRuns --> <!-- WANDB_GQL_EXAMPLE_START name=GetRunHistoryKeys --> * **Get Run History Keys:** (Run is not a collection, historyKeys is scalar) ```graphql<br>    query RunHistoryKeys($entity: String!, $project: String!, $runName: String!) {<br>        project(name: $project, entityName: $entity) {<br>        run(name: $runName) {<br>            id<br>            name<br>            historyKeys # Returns ["metric1", "metric2", ...]<br>        }<br>        }<br>    }<br>    ``` ```python<br>    variables = {"entity": "my-entity", "project": "my-project", "runName": "run-abc"}<br>    ``` <!-- WANDB_GQL_EXAMPLE_END name=GetRunHistoryKeys --> <!-- WANDB_GQL_EXAMPLE_START name=GetRunHistorySampled --> * **Get Specific Run History Data:** (Uses `sampledHistory` for specific keys) ```graphql<br>    # Corrected: Use specs argument<br>    query RunHistorySampled($entity: String!, $project: String!, $runName: String!, $specs: [JSONString!]!) {<br>        project(name: $project, entityName: $entity) {<br>        run(name: $runName) {<br>            id<br>            name<br>            # Use sampledHistory with specs to get actual values for specific keys<br>            sampledHistory(specs: $specs)<br>        }<br>        }<br>    }<br>    ``` ```python<br>    # Corrected: Define specs variable with escaped JSON string literal for keys<br>    variables = {<br>        "entity": "my-entity", <br>        "project": "my-project", <br>        "runName": "run-abc", <br>        "specs": ["{"keys": ["loss", "val_accuracy"]}}"] # List containing escaped JSON string<br>    }<br>    # Note: sampledHistory returns rows where *at least one* of the specified keys was logged.<br>    # The 'item' field is a JSON string, you'll need to parse it (e.g., json.loads(row['item'])) <br>    # to get the actual key-value pairs for that step. It might not contain all requested keys<br>    # if they weren't logged together at that specific step.<br>    ``` <!-- WANDB_GQL_EXAMPLE_END name=GetRunHistorySampled --> <!-- WANDB_GQL_EXAMPLE_START name=GetRunByDisplayName --> * **Get Run by Display Name:** (Requires filtering and pagination structure) ```graphql<br>    # Note: Querying *runs* collection and filtering, not the singular run(name:...) field<br>    query GetRunByDisplayName($project: String!, $entity: String!, $displayNameFilter: JSONString) {<br>        project(name: $project, entityName: $entity) {<br>        # Filter the runs collection by displayName<br>        runs(first: 1, filters: $displayNameFilter) {<br>            edges {<br>            # Select desired fields from the node (the run)<br>            node { id name displayName state createdAt summaryMetrics }<br>            }<br>            # Required pageInfo for collections<br>            pageInfo { endCursor hasNextPage }<br>        }<br>        }<br>    }<br>    ``` ```python<br>    # Use json.dumps for the filters argument<br>    import json<br>    target_display_name = "my-experiment-run-123"<br>    variables = {<br>        "entity": "my-entity",<br>        "project": "my-project",<br>        # Filter for the specific display name<br>        "displayNameFilter": json.dumps({"displayName": {"$eq": target_display_name}})<br>        # W&B filter syntax might vary slightly, check docs if needed. Common is {"field": "value"} or {"field": {"$operator": "value"}}<br>    }<br>    # Note: This finds runs where displayName *exactly* matches.<br>    # It might return multiple runs if display names are not unique.<br>    # The `name` field (often the run ID like 'gtng2y4l') is guaranteed unique per project.<br>    # Use `run(name: $runId)` if you know the unique run ID ('name').<br>    ``` <!-- WANDB_GQL_EXAMPLE_END name=GetRunByDisplayName --> <!-- WANDB_GQL_EXAMPLE_START name=GetArtifactDetails --> * **Get Artifact Details:** (Artifact is not a collection, but `files` is) ```graphql<br>    query ArtifactDetails($entity: String!, $project: String!, $artifactName: String!) {<br>        project(name: $project, entityName: $entity) {<br>        artifact(name: $artifactName) { # Name format often 'artifact-name:version' or 'artifact-name:alias'<br>            id<br>            digest<br>            description<br>            state<br>            size<br>            createdAt<br>            metadata # JSON String<br>            aliases { alias } # Corrected: Use 'alias' field instead of 'name'<br>            files { # Files is a collection, requires pagination structure<br>            edges { <br>                node { name url digest } # Corrected: Removed 'size' from File fields<br>            } <br>            pageInfo { endCursor hasNextPage } # Required for files collection<br>            } <br>        }<br>        }<br>    }<br>    ``` ```python<br>    variables = {"entity": "my-entity", "project": "my-project", "artifactName": "my-dataset:v3"}<br>    ``` <!-- WANDB_GQL_EXAMPLE_END name=GetArtifactDetails --> <!-- WANDB_GQL_EXAMPLE_START name=GetViewerInfo --> * **Get Current User Info (Viewer):** (No variables needed) ```graphql<br>    query GetViewerInfo {<br>        viewer {<br>        id<br>        username<br>        email<br>        entity<br>        }<br>    }<br>    ``` ```python<br>    # No variables needed for this query<br>    variables = {}<br>    ``` <!-- WANDB_GQL_EXAMPLE_END name=GetViewerInfo --> **Troubleshooting Common Errors:** * `"Cannot query field 'summary' on type 'Run'"`: Use the `summaryMetrics` field instead of `summary`. It returns a JSON string containing the summary dictionary. * `"Argument 'filters' has invalid value ... Expected type 'JSONString'"`: Ensure the `filters` argument in your `variables` is a JSON formatted *string*, likely created using `json.dumps()`. Also check the *content* of the filter string for valid W&B filter syntax. * `"400 Client Error: Bad Request"` (especially when using filters): Double-check the *syntax* inside your `filters` JSON string. Ensure operators (`$eq`, `$gt`, etc.) and structure are valid for the W&B API. Invalid field names or operators within the filter string can cause this. * `"Unknown argument 'direction' on field 'runs'"`: Control sort direction using `+` (ascending) or `-` (descending) prefixes in the `order` argument string (e.g., `order: "-createdAt"`), not with a separate `direction` argument. * Errors related to `history` (e.g., `"Unknown argument 'keys' on field 'history'"` or `"Field 'history' must not have a selection..."`): To get *available* metric keys, query the `historyKeys` field (returns `[String!]`). To get *time-series data* for specific keys, use the `sampledHistory(keys: [...])` field as shown in the examples; it returns structured data points. The simple `history` field might return raw data unsuitable for direct querying or is deprecated. * `"Query doesn't follow the W&B connection pattern"`: Ensure any field returning a list/collection (like `runs`, `files`, `artifacts`, etc.) includes the full `edges { node { ... } } pageInfo { endCursor hasNextPage }` structure. This is mandatory for pagination. * `"Field must not have a selection"` / `"Field must have a selection"`: Check if the field you are querying is a scalar type (like `String`, `Int`, `JSONString`, `[String!]`) which cannot have sub-fields selected, or an object type which requires you to select sub-fields. * `"Cannot query field 'step' on type 'Run'"`: The `Run` type does not have a direct `step` field. To find the maximum step count or total steps logged, query the `summaryMetrics` field (look for a key like `_step` or similar in the returned JSON string) or use the `historyLineCount` field which indicates the total number of history rows logged (often corresponding to steps). **Notes:** * Refer to the official W&B GraphQL schema (via introspection or documentation) for the most up-to-date field names, types, and available filters/arguments. * Structure your query to request only the necessary data fields to minimize response size and improve performance. * **Sorting:** Use the `order` parameter string. Prefix with `+` for ascending, `-` for descending (default). Common sortable fields: `createdAt`, `updatedAt`, `heartbeatAt`, `config.*`, `summary_metrics.*`. * Handle potential errors in the returned dictionary (e.g., check for an 'errors' key in the response). |
| `wandb create-wandb-report-tool` | `{entity_name: str, project_name: str, title: str, description?: str = null, markdown_report_text?: str = "", plots_html?: dict \| str = null}` | Create a new Weights & Biases Report and add text and HTML-rendered charts. Useful to save/document analysis and other findings. Only call this tool if the user explicitly asks to create a report or save to wandb/weights & biases. Always provide the returned report link to the user. <plots_html_usage_guide> - If the analsis has generated plots then they can be logged to a Weights & Biases report via converting them to html. - All charts should be properly rendered in raw HTML, do not use any placeholders for any chart, render everything. - All charts should be beautiful, tasteful and well proportioned. - Plot html code should use SVG chart elements that should render properly in any modern browser. - Include interactive hover effects where it makes sense. - If the analysis contains multiple charts, break up the html into one section of html per chart. - Ensure that the axis labels are properly set and aligned for each chart. - Always use valid markdown for the report text. </plots_html_usage_guide> <plots_html_format_guide> **IMPORTANT: plots_html Parameter Format** - The plots_html parameter accepts either: 1. A dictionary where keys are chart names and values are HTML strings: {"chart1": "<html>...</html>", "chart2": "<html>...</html>"} 2. A single HTML string (will be automatically wrapped with key "chart") - Do NOT pass raw HTML as a JSON string - pass it directly as an HTML string - If you have multiple charts, use the dictionary format for better organization - The tool will provide feedback about how your input was processed </plots_html_format_guide> Args: entity_name: str, The W&B entity (team or username) - required project_name: str, The W&B project name - required title: str, Title of the W&B Report - required description: str, Optional description of the W&B Report markdown_report_text: str, beuatifully formatted markdown text for the report body plots_html: str, Optional dict of plot name and html string of any charts created as part of an analysis Returns: str, The url to the report Example: ```python<br>    # Create a simple report<br>    report = create_report(<br>        entity_name="my-team",<br>        project_name="my-project",<br>        title="Model Analysis Report",<br>        description="Analysis of our latest model performance",<br>        markdown_report_text='''<br>            # Model Analysis Report<br>            [TOC]<br>            ## Performance Summary<br>            Our model achieved 95% accuracy on the test set.<br>            ### Key Metrics<br>            Precision: 0.92<br>            Recall: 0.89<br>        '''<br>    )<br>    ``` |
| `wandb query-wandb-entity-projects` | `{entity?: str = null}` | Fetch all projects for a specific wandb or weave entity. Useful to use when the user hasn't specified a project name or queries are failing due to a missing or incorrect Weights & Biases project name. If no entity is provided, the tool will fetch all projects for the current user as well as all the project in the teams they are part of. <critical_info> **Important:** Do not use this tool if the user has not specified a W&BB entity name. Instead ask the user to provide either their W&B username or W&B team name. </critical_info> <debugging_tips> **Error Handling:** If this function throws an error, it's likely because the W&B entity name is incorrect. If this is the case, ask the user to double check the W&B entity name given by the user, either their personal user or their W&B Team name. **Expected Project Name Not Found:** If the user doesn't see the project they're looking for in the list of projects, ask them to double check the W&B entity name, either their personal W&B username or their W&B Team name. </debugging_tips> Args: entity (str): The wandb entity (username or team name) Returns: List[Dict[str, Any]]: List of project dictionaries containing: - name: Project name - entity: Entity name - description: Project description - visibility: Project visibility (public/private) - created_at: Creation timestamp - updated_at: Last update timestamp - tags: List of project tags |
| `wandb query-wandb-support-bot` | `{question: str}` | Query the Weights & Biases support bot api for help with questions about the Weights & Biases platform and how to use W&B Models and W&B Weave. W&B features mentioned could include: - Experiment tracking with Runs and Sweeps - Model management with Models - Model management and Data versioning with Artifacts and Registry - Collaboration with Teams, Organizations and Reports - Visualization with Tables and Charts - Tracing and logging with Weave - Evaluation and Scorers with Weave Evaluations - Weave Datasets FYI: The Weigths & Biases platform is owned by Coreweave. If there are queries related to W&B, wandb or weave and Coreweave, they might be related to W&B products or features that leverage Coreweave's GPU or compute infrastructure. Parameters ---------- question : str Users question about a Weights & Biases product or feature Returns ------- str newer to the user's question |

## git

Use `git --help` for Git operations on workspace repositories.

## python3

Execute Python code directly under the agent workspace and return stdout
and stderr. Code can be pasted into a `.py` file and run with `python3`.

Python cannot launch subprocesses or make direct network requests. Run the
documented service commands directly in the shell. Use Python for local
file processing and generating JSON arguments.

cat  head  tail  ls  find  grep  wc  sort  uniq  cut  tr  sed  awk
diff  tee  xargs  jq  unzip  curl