pleasefixai/oracle-ai-data-platform-workbench-engineer-agent

Discover, author, deploy, and run AIDP agent flows. Use when the user wants to list/inspect agent flows, or create/update/deploy/run a flow, manage sessions/guardrails, attach compute, or attach a remote MCP server to a flow (an MCP_TOOL node — AIDP's "Native MCP Client Support" LA feature, where the flow connects OUT to OAC/ADW/OIC/any MCP-compatible service). Everything runs over the Limited-Availability AgentFlows REST API via oci raw-request; verify live first. (An aidp MCP list_agent_flows, if configured, is an optional read accelerator only.)

Build AIDP agents in high-code Python with aidputils + LangGraph (the code-first alternative to the low-code agent-flow canvas). Use when the user wants to write an agent in Python, use LangGraph / create_react_agent / StateGraph, call aidputils (OCIAIConf, AIDPToolConf, init_oci_llm, create_langgraph_tool), build a custom or multi-agent supervisor flow in code, or asks "how do I code an AIDP agent". For the low-code/REST node-graph path use aidp-agent-flows.

Run LLM functions inside Spark SQL on AIDP via ai_generate(). Use when the user wants to summarize/classify/extract/enrich rows with an LLM directly in SQL, generate narratives over aggregated results, or do grounded RAG-style analysis in the lakehouse. Signature is model-first; available models must be confirmed live before relying on it.

Answer business questions over the AIDP lakehouse with Spark SQL. Use when the user asks a data question ("how many…", "top N…", "show me…", "trend of…", "revenue by…") or wants to run ad-hoc Spark SQL. Grounds in .aidp/catalog.md + .aidp/semantic.md and reuses validated verified queries before generating SQL, then executes via the bundled aidp_sql.py helper.

Manage and search AIDP audit logs — enable/disable auditing, set retention, and query audit-log entries for a DataLake. Use when the user asks about audit logging, who did what, compliance/retention of AIDP activity, or wants to search audit events. Self-contained — official aidp audit CLI preferred, oci raw-request fallback.

Deploy AIDP resources as a bundle — create a bundle, deploy all its assets atomically, query deployment status, and purge. Use when the user wants to promote/deploy a set of AIDP resources together, manage a deployment bundle, check bundle deployment status, or tear one down. Preview API via oci raw-request; verify live first.

Browse the AIDP data catalog live — list catalogs, schemas, tables, and volumes, inspect a table's columns/properties, and resolve a human name to its catalog key. Use when the user asks "what catalogs/schemas/tables exist", "describe table X", "what columns does Y have", or needs a catalog/schema/table key for another operation.

One-time AIDP catalog discovery that writes a cached, version-controllable .aidp/catalog.md grounding file (tables, columns, FK/join hints, value dictionaries). Use when the user says "oracle-ai-data-platform-workbench-engineer-agent:aidp-catalog-init", asks to "map/discover my lakehouse", set up data discovery, or before answering data questions when no catalog cache exists. Re-run with --refresh when the schema changes.

Manage AIDP Spark compute clusters — list, status, start/stop/restart, installed libraries (JARs/Python), provision/scale a new cluster (driver/worker shapes, autoscale, GPU/RAPIDS, AI Compute), and connect external BI tools (JDBC/ODBC). Use when the user asks about clusters, needs to start/stop compute, create or scale a cluster, install libraries, set up a GPU cluster, use AI Compute for agent flows, connect Tableau/Power BI/DBeaver, or pick a cluster before running data work.

Manage the AIDP credential store (secrets) — list, get, create, update, delete credentials used by AIDP workflows. Use when the user wants to store/rotate a secret centrally instead of embedding it, or manage connection credentials. Primary engine is the official aidp CLI (aidp credentials …); the same Preview REST API via oci raw-request is the no-CLI fallback. Verify the endpoint live before relying on it.

Run data-quality rule checks on AIDP tables — not-null, uniqueness, allowed ranges/sets, referential integrity, and freshness. Use when the user wants to validate data, check for nulls/duplicates/orphans, assert a column's domain, or gate a pipeline on quality. Expresses each rule as bounded Spark SQL and reports pass/fail with offending counts.

Share AIDP data via Delta Sharing — create/manage shares, add data assets, manage recipients, and grant access/permissions. Use when the user wants to share a table/schema with another team or org, set up Delta Sharing, manage recipients, or control share permissions/token expiry. Primary engine is the official aidp CLI (aidp delta-share …); the same GA REST API via oci raw-request is the no-CLI fallback.

First-run setup for the self-contained AIDP plugin. Verifies the oci CLI + a DEFAULT api_key profile, installs the bundled Python deps, discovers region + DataLake OCID + workspace, then smoke-tests both engines (a control-plane GET /catalogs via oci raw-request and a trivial SELECT 1 cell via scripts/aidp_sql.py). Use on first run, when the user asks to "set up / configure / install / bootstrap AIDP", or when control-plane or SQL calls fail with auth/config errors. No MCP install required.

Router + setup/auth troubleshooting for Oracle AI Data Platform (AIDP) data-engineering work. Use when the user mentions AIDP and isn't sure which skill applies, asks "what can you do with AIDP", describes a task by capability, or hits an AIDP auth/workspace/cluster error. Routes to the right aidp-* skill and carries the shared environment + auth-ladder rules.

Federate across multiple data sources in one AIDP Spark session — read from several connectors (Oracle ADB/ExaCS, Fusion, Snowflake, S3, lakehouse tables, …) and join them in a single notebook. Use when the user wants to combine/join data from more than one source, blend an external system with the lakehouse, or do cross-source analysis. Composes the spark-connectors plugin; does not duplicate connectors.

Operate native Git in an AIDP workspace — branches, checkout, commit/push, pull, merge, rebase, reset, diff, and conflict resolution on workspace git repositories. Use when the user wants to version workspace notebooks/code in Git, manage branches, commit/push changes, or resolve conflicts inside AIDP. Preview GitService API via oci raw-request; verify live first.

Load a data file (CSV/JSON/Parquet/etc.) into a managed AIDP Delta table. Use when the user wants to ingest a file into a table, create a table from a file, or land raw data in the lakehouse. Supports the 1-step path and the 3-step upload→infer→create path. Control-plane via the official aidp CLI.

Build and manage AIDP Knowledge Bases for RAG — create a KB over catalog data, pick an embedding model + chunking, build an HNSW/IVF vector index, add data sources (volume/table), run ingestion jobs, and manage KB permissions. Use when the user wants RAG, a knowledge base, document/vector search, embeddings, semantic retrieval, or to feed a RAG_TOOL agent-flow node. This is the corpus/index layer; the agent that queries it is authored in aidp-agent-flows (RAG_TOOL) or aidp-agent-highcode.

Guide a migration of notebooks/jobs from another platform (e.g. Databricks) into AIDP. Use when the user wants to port Databricks notebooks/jobs to AIDP, move workloads onto the AIDP lakehouse, or plan a migration. Orchestration-only — it composes the other self-contained aidp-* skills; it adds no new API surface.

Track ML work in AIDP's MLflow-compatible MLOps — experiments, runs, metrics/params, registered models, and model versions. Use when the user wants to log/track experiments, list runs or metrics, register a model, transition a model-version stage, or query the model registry. Primary engine is the official aidp CLI (aidp mlops …); the same Preview MLflow REST API via oci raw-request is the no-CLI fallback. Verify live first.

List the models available/installed in the AIDP DataLake and inspect their parameters, and browse the MLOps model registry. Use when the user asks "what models are available", needs a model name for ai_generate()/agent flows, wants a model's parameters, or wants to list registered models / versions. Registry browse uses the official aidp CLI (folded into the mlops group); the LA Models platform-catalog API via oci raw-request is verified and the no-CLI fallback.

Create, edit, and execute AIDP notebooks and manage their kernel sessions. Use when the user wants to author a notebook, run cells/code on a cluster, attach/detach a kernel session, or build an interactive notebook (e.g. with widgets + Spark SQL + Plotly). Notebook FILE ops run via the Notebook contents REST API; CELL EXECUTION runs via scripts/aidp_sql.py.

Surface AIDP operational state — recently accessed resources and long-running async operations (status, completion, waiting). Use when the user asks "what happened recently", "what's running", "is that operation done yet", or needs to track/await an async AIDP operation (e.g. provisioning, large commits). Runs over the official aidp CLI.

Author, schedule, run, and monitor AIDP Jobs (task DAGs of notebooks/python with cron). Use when the user wants to build a pipeline, create/update/run a Job, schedule a recurring run, check a run's status, read a task's output, or cancel a run.

Profile an AIDP table — row count, per-column null %, distinct count, min/max/mean, and top-K values. Use when the user asks to profile a table, wants column statistics or a data-quality snapshot, or needs to understand a dataset's shape before using it. Runs bounded Spark SQL via the bundled aidp_sql.py helper.

Manage AIDP roles and access — list roles, view permissions, create roles, add/remove members, and grant/revoke per-resource permissions on catalogs, schemas, tables, views, volumes, workspaces, workspace objects, and clusters. Also covers column masking/classification (restricted views + ontology sensitivity — no masking REST API exists). Use when the user asks about roles/RBAC, who can access what, granting/revoking access on any resource, adding someone to a role, or masking/classifying columns. Primary engine is the official aidp CLI; the same REST API via oci raw-request is the no-CLI fallback.

Maintain a semantic grounding layer (.aidp/semantic.md) for AIDP — logical entity names, SQL-defined metrics, joins with cardinality, synonyms, and value dictionaries. Use when the user wants to define metrics/business terms, improve NL-to-SQL accuracy, standardize "revenue/customers/etc.", or set up a semantic model. Read by analyzing-data, verified-queries, profiling, and data-quality.

Diagnose slow or failed AIDP Spark work using cluster logs, metrics, and the Spark UI REST API. Use when a job/query is slow or failed, the user asks "why did this fail / why is it slow", or you need stage/task timings, skew, shuffle/spill, executor, or SQL-execution details. Lightweight triage — for deep performance tuning (skew/spill/shuffle/joins/AQE/Delta) use the aidp-spark-optimization skill.

Use when a Spark job/notebook is slow, missing an SLA, spilling, OOMing, generating too many small files, or shuffling/skewing heavily; when reviewing Spark/PySpark code or a Spark UI for performance; or before running a large Spark workload. Covers open-source Apache Spark 3.5.0 (+ Delta Lake) tuning -- partitions, shuffle, joins, skew, file layout, memory, codegen, caching, AQE, compression, and configuration.

Write-side Spark SQL on the AIDP lakehouse — DDL (CREATE/ALTER/DROP table·view·schema), DML (INSERT/UPDATE/DELETE/MERGE upsert), and Delta maintenance (OPTIMIZE, VACUUM, time travel, RESTORE, schema evolution, liquid clustering). Use when the user wants to create/alter/drop a table or view, insert/update/delete/merge rows, upsert, compact/optimize/vacuum a Delta table, query an old version (time travel), restore, evolve a schema, or run any non-SELECT SQL. For read-only analytics use aidp-analyzing-data; for control-plane (CLI/REST) table lifecycle use aidp-table-management.

Control-plane lifecycle for AIDP catalog objects — create/update/delete/refresh tables, views, schemas, and catalogs, and register an EXTERNAL catalog/connection (ALH/ADW/object-storage) as a persistent catalog object. Use when the user wants to create or drop a table/view/schema/catalog via the control plane (not a SELECT), register an external data source as a catalog, refresh catalog metadata, or test a connection. For SQL-native CREATE/ALTER/DROP use aidp-sql-ddl; for file→table ingestion use aidp-ingest-file-to-table; for browsing use aidp-catalog-explore.

Create and manage standalone reusable AIDP agent Tools — SQL, Prompt, RAG, HTTP, Custom, and MCP tools that can be shared across agent flows by key. Use when the user wants to define a reusable tool (not inline in one flow), list/create/update/delete a tool, test a tool, or build a tool an agent/agent-flow node will reference. For attaching a tool inside a single flow as a node, see aidp-agent-flows + references/agent-flow-nodes.md.

Manage AIDP user settings / preferences — list, get, create, update, delete a user's settings for the DataLake. Use when the user wants to view or change their AIDP workbench preferences/settings, or manage stored user-setting entries. Self-contained — official aidp user-setting CLI preferred, oci raw-request fallback.

Register and validate reusable question→Spark-SQL pairs in .aidp/verified-queries.md so the agent reuses trusted SQL before generating new SQL. Use when the user wants to save a working query as canonical, build a verified-query repository, or improve answer reliability for recurring questions. Validates each pair on the cluster before marking it verified.

Work with AIDP volumes — list volumes, browse files inside a volume, upload/download via the PAR flow, and create directories. Use when the user mentions volumes, needs to stage large/binary files, or move data in/out of a volume (distinct from the workspace filesystem). Control-plane via the official aidp CLI.

Provision and inspect AIDP DataLake instances and workspaces, including private-network workspaces attached to a customer VCN/subnet. Use when the user wants to create/list/get a workspace or DataLake instance, set up a new (e.g. private) AIDP environment, or replicate a customer setup. Create/delete are guarded — confirm before any provisioning.

Manage files and notebooks in the AIDP workspace filesystem — list, upload, download, read, create, move, rename, delete files/folders/notebooks. Use when the user wants to put a script/notebook/config into the workspace, read or move workspace files, or organize Shared/ folders.