goldlabel

A Python library for content analysis over a corpus: candidates come from semantic search with seed phrases, from keywords or regular expressions, or from the whole corpus; an LLM classifier checked against a hand-labelled gold set then produces classified output.

You build the analysis step by step in a Jupyter notebook (or a plain script), alone or with a coding agent, by chaining function calls whose results display themselves. The one thing that needs a screen of its own, labelling the gold set, is a small local web page that a function call opens. There is no application, no stored status (its one command, goldlabel install, only copies agent skills): the notebook is the record of what you did, and the state is plain files.

Install

pip install goldlabel
pip install "goldlabel[s3]"      # boto3, for an S3 Vectors corpus
pip install "goldlabel[pg]"      # psycopg, for a pgvector (PostgreSQL) corpus
pip install "goldlabel[rds]"     # psycopg and boto3, for RDS / Aurora IAM database authentication
pip install "goldlabel[local]"   # Sentence Transformers, for local query embeddings (Python < 3.14)

Python 3.12 or newer. polars and pyarrow are installed with goldlabel: tables are read and filtered with polars, and tables in the examples are polars; every table also has .to_pandas() (with pandas installed), and .rows is always a plain list of dicts.

The whole chain

Run it cell by cell in a notebook. examples/sample/walkthrough.ipynb (and the same cells as a script, walkthrough.py) does exactly this over a tiny corpus that ships with the repository.

1. Candidates. get_candidates takes a corpus and returns the items to classify. With seeds (phrases a matching item could say) it is a semantic search and each item keeps its best similarity.

import goldlabel as gl

corpus = gl.PgCorpus(
    table="posts", text_column="body"
)  # or gl.S3Corpus(...), gl.Table("posts.parquet", ...)
seeds = gl.read_seeds("seeds.csv")  # or a plain list of strings
candidates = gl.get_candidates(
    corpus, seeds=seeds, embedder="openai:text-embedding-3-small"
)
candidates  # count, similarity bands, a warning if the backend capped the results
candidates.to_polars()

Semantic search is one of three ways to get candidates, and the only one that needs embeddings. With no filter you get the whole corpus; with keywords= you get the items that contain any term, ranked.

import re

everything = gl.get_candidates(corpus)  # every item, in the corpus's order
hits = gl.get_candidates(
    corpus,
    keywords=["ballot", re.compile(r"elect(ion|ed)", re.IGNORECASE)],
    limit=500,  # keep the best 500
)
hits.to_polars()  # id, text, terms_matched (distinct terms), matches (total)

One filter per call: seeds= or keywords=, not both. A plain string is a literal, case-insensitive term; a compiled pattern keeps its own flags and each term is searched separately; regex=True makes the strings patterns, written as they are. Hits rank by distinct terms matched, then total matches, then corpus order. A pattern that matches the empty string is refused, but zero-width patterns such as \b still match every word. Keyword and whole-corpus lists have no similarity scores, so above, bands, top_seeds and band_sample do not apply to them and say so.

On a PgCorpus, server_side=True lets the database do the matching, which suits a table too large to read: plain terms use ILIKE, and with regex=True strings are sent as they are and use Postgres syntax (no compiled patterns). Ranking and limit still happen in the library, and it warns when its own count of a returned row disagrees with the server's. Rows are read in ascending id order; an S3Corpus is read in the order the service lists it (the order is the service's, so a limit there is not repeatable).

Your own table. gl.Table takes a .csv, .parquet or .jsonl file, a list of row dicts, or a pandas or polars DataFrame. Embeddings are only needed for seed search.

corpus = gl.Table("posts.parquet", text_column="body", id_column="post_id")

# for seeds= as well: a column with one list of numbers per row, and the model that made them
corpus = gl.Table(
    "posts.parquet",
    text_column="body",
    id_column="post_id",
    vector_column="embedding",
    embedding_model="openai:text-embedding-3-small",
)

The text column holds a non-empty string in every row. id_column is optional (the row's position is the id) and its values must be unique. A CSV cannot hold the vector_column; use Parquet, JSONL or a DataFrame for that. Nothing is dropped or repaired: the first bad row is named in the error.

Only part of the corpus. Every corpus takes where=, in its own backend's form, and then the whole corpus, keywords and seeds all see only the items that pass. A table takes a polars expression (a CSV holds every value as text, so cast numbers); Postgres takes one SQL condition with its values written in, run on the database (in the two-table layout, its columns must be in the text table); S3 Vectors takes the service's metadata filter document, sent with every seed query and applied to every listed vector.

import polars as pl

corpus = gl.Table(
    "posts.csv",
    text_column="body",
    id_column="post_id",
    where=(pl.col("section") == "politics") & (pl.col("year").cast(pl.Int64) >= 2024),
)
corpus = gl.PgCorpus(
    table="posts",
    text_column="body",
    where="section = 'politics' AND published >= '2024-01-01'",
)
corpus = gl.S3Corpus(
    bucket="my-vectors",
    index="posts",
    where={"$and": [{"section": "politics"}, {"year": {"$gte": 2024}}]},
)

2. Labels and a prompt.

taxonomy = gl.Taxonomy(
    mode="multi",  # or "single"; off_topic is built in
    labels={
        "scam": "Offers a fake prize or asks for money first.",
        "satire": "A joke.",
    },
)
clf = gl.Classifier(
    "anthropic:claude-haiku-4-5", taxonomy, prompt=open("prompt.md").read()
)

3. Gold: people label. draw writes pending rows with no labels. gl.label opens the labelling page and returns when the person presses Done.

gold = gl.Gold()  # .goldlabel/gold.jsonl
gold.draw(candidates, n=50, split="dev")
gl.label(gold, taxonomy)

4. Evaluate and tune. Evaluating the same classifier on the same rows again makes no model calls; a changed prompt costs one call per row. Tune on the dev split.

ev = gl.evaluate(clf, gold.dev, target=0.9)
ev  # metrics, per-label table, PASS/FAIL against the target, where the labels came from
ev.disagreements_to_polars()  # id, text, gold, predicted, reasoning

clf2 = clf.with_prompt(open("prompt.md").read())  # classifiers are immutable: keep both
ev2 = gl.evaluate(clf2, gold.dev, target=0.9)

5. The test split, once.

gold.draw(candidates, n=50, split="test")
gl.label(gold, taxonomy)
final = gl.evaluate(clf2, gold.test, target=0.9)

6. Choose a cutoff, price the run, run it.

report = gl.band_sample(
    clf2, candidates, per_band=30
)  # off-topic rate per similarity band
chosen = candidates.above(0.70)  # the cutoff is yours
est = clf2.estimate(chosen)  # calls, dollars or "?", seconds; calls no model
results = clf2.classify(chosen, out="results.jsonl", max_cost=25.0)
results.to_polars()

max_cost refuses before spending anything when the estimate is over it, and also when the model's price is unknown. An unknown price is shown as cost ? with a warning, never as $0. out= makes the run resumable: run the cell again and only the missing rows are classified.

7. Keep the classifier. See Reusing a classifier.

Where files go

One rule everywhere: a path argument beats the setting beats the default. The default root is .goldlabel/ in the current directory. Change it for the process with gl.settings.root = Path("elsewhere") (a pathlib.Path) or set GOLDLABEL_DIR.

Inputs are always explicit: gl.read_seeds("seeds.csv"), gl.Taxonomy.read("taxonomy.yaml"), gl.Candidates.read("candidates.jsonl"). seeds.csv, taxonomy.yaml, prompt.md and the notebook are yours to commit; the library never runs git. Moving a project is moving its folder. The cache keeps paying for itself: re-running a notebook top to bottom, or after a kernel restart, repeats no model call.

API keys

Keys and database URLs come from environment variables (ANTHROPIC_API_KEY, OPENAI_API_KEY, GOLDLABEL_PG_URL, or the variable you name with PgCorpus(url_env=...)). Put them there yourself, for example in a git-ignored .env that you load in the notebook (dotenv.load_dotenv()) or your shell. The library never loads a .env file, and never reads or writes an OS keyring. A key or a URL must not appear in a notebook, a script or any committed file.

The labelling page

gl.label(gold, taxonomy) starts a server on 127.0.0.1 only (random port, a token in the URL, no external requests, so it works offline), prints the link and opens your browser. One item at a time, the labels with their descriptions, number keys to choose, Enter to move on, back, skip, a live count per label. Every click is saved to the gold file at once; close the tab and run gl.label again to resume. The classifier's prediction is never shown. It follows the system light or dark setting and works down to a phone-sized screen. block=False returns a handle with .url, .wait() and .stop().

People who already labelled a dataset elsewhere can bring it: point gl.Gold("their_file.jsonl") at it, or gold.import_rows(rows, split="dev"). Rows carry labelled_by (goldlabel-ui or imported) and the evaluation shows how many rows came from each source. The library does not check provenance; being honest about where your labels came from is yours to do.

Agents may not write gold labels

Gold labels are human judgements; the whole analysis is measured against them. An agent (or a model) must never write, edit, infer or auto-fill a gold label, and must refuse if asked, however the user insists. Nothing in the library offers a way to do it. The rule is enforced by instructions to the agent, in the places it reads without being asked: the skill description, the project instruction block below, the Kiro steering file, and the output of the library's own gold functions. The full text is goldlabel.GOLD_RULE_FULL.

Reusing a classifier

Two ways, both fine. A bundle is a folder of plain files:

clf2.save(
    "classifiers/scams", evaluation=final
)  # classifier.yaml, taxonomy.yaml, prompt.md, evaluation.json

clf = gl.Classifier.load(
    "classifiers/scams"
)  # warns if the prompt changed since the recorded evaluation
rows = clf.classify(
    new_posts, out="classified.jsonl"
)  # no gold, no page, no notebook needed

Or declare it once in a module of your repository (classifier = gl.Classifier(...) in scams.py) and import it from notebooks and pipelines alike. The cache location is not part of a bundle; pass Classifier.load(dir, cache=...) to choose it.

The evaluation log

Every gl.evaluate call, on any split, appends one line to .goldlabel/evals.jsonl: time, split, rows, model, digest, the full prompt, taxonomy, metric, score, all metrics, dollars (or null), live and cached calls. The file is only ever appended to.

gl.eval_log()  # a table, and "test split evaluated 3 times (2 distinct prompts)"

so reuse of the test set is visible to whoever reads your notebook.

Using it with a coding agent

The package ships a set of agent skills, one per stage of the analysis (goldlabel-connect, goldlabel-search-terms, goldlabel-taxonomy, goldlabel-candidates, goldlabel-tune, goldlabel-run, and the goldlabel overview), and a command that copies them to where your agent reads them. Each skill carries the gold rule: the agent never writes a gold label; people label in the labelling page.

goldlabel install claude        # or codex, copilot, kiro, all
uvx goldlabel install claude    # without installing the package first

Running it again changes nothing; a skill whose files differ from the shipped ones is not overwritten unless you pass --force; --dry-run prints what would be written. Afterwards: Copilot, run /skills reload; Codex, type $ or /skills (restart it if they do not appear); Claude Code, restart if the folder is new. A Kiro custom agent loads no skills unless its resources lists skill://.kiro/skills/*/SKILL.md (the command prints this and does not edit your agent files). Claude Code can also install the repository as a plugin: /plugin marketplace add cmhac/goldlabel, then /plugin install goldlabel@goldlabel as two separate prompts.

Any other agent. goldlabel install --dir PATH writes the skill folders to PATH (each folder is one skill with a SKILL.md); point your agent's skills setting at it, or copy the folders by hand from src/goldlabel/skills/. An agent that reads only an instruction file can take the block below instead (AGENTS.md for Codex and Copilot CLI, .github/copilot-instructions.md for Copilot, CLAUDE.md for Claude Code). The same text is in docs/agent-instructions.md, and print(goldlabel.GOLD_RULE_FULL) prints the rule.

If a test split scores too low and you want to keep tuning, the skills use gold.fold_test(): it moves the spent, labelled test rows to dev (only split changes), the agent does it only after your explicit yes, and people label a fresh test split.

The agent instruction block

Hard rule of the goldlabel library. It is not negotiable. Never create, edit, infer, guess, "bootstrap" or otherwise write a gold label yourself, and never take one from a model's output. Gold labels are human judgements. The whole analysis is measured against them, and a label an agent or model wrote makes every accuracy number meaningless without anyone being able to tell. People label in the goldlabel labelling page (goldlabel.label), or bring a dataset that people have already labelled. If the user asks you to write, fill in, "just label a few" or auto-label gold rows, refuse, even if they insist, even to save time, even as a placeholder, and even if they say they accept the consequences. Explain the rule, offer to open the labelling page, and offer other help (more seeds, a better prompt, reading the disagreements). Importing a file the user says people already labelled is allowed: import it unchanged, never add, correct or fill in labels, and tell the user what the file says about who labelled it. Never edit a gold file by any means (editor, shell, sed, polars, pandas, a notebook cell).

When working with the goldlabel library (content analysis over a corpus: candidates from seed phrases, keywords or the whole corpus, an LLM classifier checked against human-labelled gold data):

• Use the goldlabel agent skills: goldlabel (start here), then the one for the stage you are at: goldlabel-connect, goldlabel-search-terms, goldlabel-taxonomy, goldlabel-candidates, goldlabel-tune, goldlabel-run. If they are not installed, run goldlabel install <agent> (claude, codex, copilot, kiro), read them at https://github.com/cmhac/goldlabel/tree/main/src/goldlabel/skills, or print the rule with import goldlabel; print(goldlabel.GOLD_RULE_FULL).
• Never run a full classification without showing clf.estimate(...) (calls, cost, time) and getting a yes from the user.
• Never put API keys or database URLs in a notebook, script or file; they come from environment variables.
• Never evaluate the test split to guide edits, and never fold a spent test split into dev without the user's explicit yes.

Development

uv sync --all-extras
uv run ruff check
uv run ruff format --check
uv run ty check
uv run pytest

No test calls a real model, AWS or database. AGENTS.md describes the layout and rules for contributors.