Job Research Tool

AI-powered job market research tool for searching and analyzing job listings across German job boards.

Overview

Job Research Tool aggregates job listings from German job boards through three independent scraper backends: the python-jobspy library, the Bundesagentur für Arbeit API, and an LLM web search served by either OpenRouter (default) or the Anthropic Claude API. Results pass through a shared post-fetch pipeline (salary and seniority filtering, board interleaving, truncation) before being analyzed and displayed in an interactive Gradio web UI. Fetches are cached for 24 hours to minimize API costs on repeated queries.

Features

• Multi-Source Search: every selected source is searched in one run, each through whichever backend can reach it, running concurrently
• Three Scraper Backends: python-jobspy, the Bundesagentur für Arbeit API, and LLM web search
• Swappable LLM Provider: the LLM backend runs on OpenRouter (any model it fronts) or the Anthropic API directly — one env var, no code change
• Per-Source Health: each run reports what every source returned, so a failing source is visible rather than looking like a quiet market
• Source Registry: One place declares every board, which backends can reach it, and how to recognise its listing URLs. The UI offers only sources the active backend can actually search
• Relevance Scoring: Titles are scored against your keywords and alternatives, with a configurable threshold and a negative-keyword list
• Post-Fetch Filters: Salary by range overlap, seniority inferred from the job title
• Salary Normalization: Hourly and monthly pay annualised so filters and statistics compare like with like. Bundesagentur supplies structured amounts with an explicit period; other sources are parsed from free text
• Recency Filter: "Posted within N days", narrowing the fetch itself on Bundesagentur (veroeffentlichtseit) and jobspy (hours_old), with a local filter as backstop
• Run-over-Run Tracking: A local SQLite store remembers every posting seen, so each run can tell you what is new since last time — and show only those
• Multi-Source Interleaving: Results from different boards are round-robin mixed before truncation, so one board cannot crowd out the rest
• Cross-Source Deduplication: One rule for every backend — normalized URL, then title + company + location — merging duplicates into one entry that keeps every board's link
• Data Analysis: Salary statistics and distribution by board, location, and work type
• Multiple Export Formats: Reports in Markdown or CSV
• Smart Caching: 24h cache keyed only on what the backend actually received, so adjusting salary or seniority re-filters instead of re-fetching
• Web Interface: Gradio UI with live progress and a search log

Quick Start

Prerequisites

• Python 3.13+
• uv package manager
• An OpenRouter API key, or an Anthropic API key (only for the LLM backend; the other two need no key)

Installation

1. Install uv (if not already installed):

   curl -LsSf https://astral.sh/uv/install.sh | sh

2. Install dependencies:

   uv sync

3. Configure environment:

   cp .env.example .env
   # Edit .env — set OPENROUTER_API_KEY to enable the LLM backend

Running the Application

uv run python src/ui/app.py
# or:
uv run job-research

Opens at http://127.0.0.1:7860

Usage

Web Interface

1. Enter keywords, location, work type, seniority, and salary range
2. Click "Search Jobs" — a live status line and expandable log panel show progress
3. Results render inline as formatted Markdown (or CSV in a code block) with a download link

Configuration

Edit .env:

# LLM backend — pick a provider and set its key. Without one, the LLM backend
# is absent and the free backends still work.
LLM_PROVIDER=openrouter        # openrouter (default) or anthropic
OPENROUTER_API_KEY=sk-or-...
OPENROUTER_MODEL=anthropic/claude-haiku-4.5   # any model OpenRouter fronts
OPENROUTER_SEARCH_RESULTS=10   # results per search
OPENROUTER_SEARCH_ENGINE=      # pin to `exa` for models whose native search
                               # ignores domain filters (e.g. Gemini)
CLAUDE_API_KEY=sk-ant-...      # used when LLM_PROVIDER=anthropic
CLAUDE_MODEL=claude-haiku-4-5-20251001

# Optional
SCRAPER_BACKEND=llm            # preferred backend (`claude` still accepted)
MAX_JOBS_PER_SEARCH=200        # max results to show (default 200)
INTERLEAVE_SITES=true          # mix results from different job boards
ENABLED_SOURCES=indeed,arbeitsagentur  # which boards to search
EXCLUDED_KEYWORDS=             # drop titles containing these (comma-separated)
RELEVANCE_THRESHOLD=0.6        # phrase coverage a title must reach to be kept
POSTED_WITHIN_DAYS=            # only listings newer than N days (empty = no limit)
STORE_PATH=data/jobs.db        # where the seen-jobs store lives
JOB_KEYWORDS=AI Engineer
LOCATION=All of Germany
WORK_TYPE=Remote/Hybrid
SALARY_MINIMUM=50000
SALARY_MAXIMUM=200000

Architecture

Sources are searched together, not one backend at a time. Each enabled source is assigned to a backend that can reach it; the backends run concurrently and their results merge into a single pipeline. SCRAPER_BACKEND sets a preference, not an exclusive choice — it wins wherever it can serve a source, and free backends are preferred over the metered one for everything else.

• JobSpyScraper — queries Indeed (and LinkedIn, if enabled) via python-jobspy. Free, no key, but relies on site structure staying put.

• BundesagenturScraper — the federal job board's own search endpoint. Free, no key, the largest German corpus, and the only source here that exposes structured salary and server-side recency filtering. Unofficial endpoint — read ADR-001 before enabling.

• LLMScraper — asks a model to search each source's domain and return JSON, cross-checking every extracted URL against the search results the provider actually reported, so nothing is fabricated. Reaches boards the other two cannot; costs API spend per search. Which vendor serves it is a providers concern:

  • OpenRouter (default) — one key reaches every model it fronts, via the web plugin (include_domains, url_citation annotations). Set OPENROUTER_MODEL to switch models.
  • Anthropic — the Claude API's server-side web_search tool directly (allowed_domains, web_search_tool_result blocks).

  Domain restriction is what makes per-source search work, and not every model's native search honours it — Anthropic's and OpenAI's do, Google's does not. With a Gemini model, pin OPENROUTER_SEARCH_ENGINE=exa.

Both backends produce a common JobListing data model (Pydantic) and return the full deduplicated fetch, which is what gets cached. Everything that shapes the view happens afterwards in pipeline.py, so it applies to every backend and can change without invalidating the cache:

1. Relevance — scores each title against the search phrase and its alternatives, keeping those that cover more than half of any one phrase. Negative keywords drop titles outright.
2. Deduplicate — collapses the same posting reached through several sources: normalized URL first, then title + company + location.
3. Salary filter — keeps listings whose stated range overlaps the requested window. Listings with no stated salary are kept.
4. Recency filter — keeps listings published within N days. Listings with no parseable date are kept.
5. Seniority filter — infers a level from the job title and keeps matching listings. Heuristic; no board exposes a reliable seniority field.
6. Interleave — round-robins across boards so truncation does not favour whichever board was scraped first.
7. Truncate — cuts to MAX_JOBS_PER_SEARCH, last, so filtering never returns fewer results than requested for no reason.

Surviving listings are then recorded in the job store, which reports which of them were not present in any earlier run. JobAnalyzer generates summary statistics and a formatted report, marking new listings with 🆕.

Job identity

Every posting gets a stable id derived from its content — normalized title, company, and location — never from its URL, which differs per source:

ai-engineer-acme-gmbh-berlin-d060f925

Readable on purpose: if job records ever become files in a Markdown vault, this becomes the filename. The same posting keeps the same id across runs and across the sources that found it, which is what makes "new since last run" trustworthy.

What the store does not claim: disappearance. A posting missing from today's results has not necessarily been withdrawn — it may just not match today's query. Reporting that soundly needs per-query history, so the store reports new and returning and stays quiet rather than guessing.

Gradio UI → run_job_research() → Orchestrator
               ├── JobSpyScraper.search_jobs()        ─┐  (concurrent;
               ├── BundesagenturScraper.search_jobs() ─┤   cached: full fetch)
               └── LLMScraper.search_jobs()          ─┤
                     └── OpenRouterProvider | AnthropicProvider
                                                 ▼
                                          pipeline.apply()
       (relevance → dedup → salary → recency → seniority → interleave → truncate)
                                                 ▼
                                         JobAnalyzer → Report

Sources

Source	llm backend	jobspy backend	Default
Indeed.de	yes	yes	on
LinkedIn	yes	yes	off — see ADR-002
Google Jobs	—	yes	off — see ADR-003
StepStone.de	yes	—	off
Monster.de	yes	—	off
Honeypot.io	yes	—	off
startup.jobs	yes	—	off
Arbeitsagentur.de	yes	—	on — see ADR-001

(The bundesagentur backend serves Arbeitsagentur.de through its own API rather than the two columns above.)

Selecting a source the active backend cannot reach searches less rather than failing; selecting nothing it can reach fails loudly.

Where job status (interested / applied / rejected) should live is deliberately still open — see Phase 5b in the architecture review.

Project Structure

job-research/
├── src/
│   ├── job_research/
│   │   ├── __init__.py
│   │   ├── config.py           # Configuration management
│   │   ├── providers.py        # OpenRouterProvider / AnthropicProvider
│   │   ├── scraper.py          # LLMScraper (provider-agnostic search + extract)
│   │   ├── jobspy_scraper.py   # JobSpyScraper (python-jobspy)
│   │   ├── sources/            # Source registry + the Bundesagentur client
│   │   ├── pipeline/           # Relevance, dedup, filters, interleaving
│   │   ├── salary.py           # Salary parsing and annualisation
│   │   ├── identity.py         # Stable, readable job ids
│   │   ├── store.py            # SQLite store: what has been seen, and when
│   │   ├── orchestrator.py     # Fans a search across backends, reports health
│   │   ├── analyzer.py         # Job analysis and reporting
│   │   └── utils.py            # Caching, formatting, parsing
│   └── ui/
│       └── app.py              # Gradio application + main()
├── docs/
│   ├── architecture-review.md  # Findings and phased refactor plan
│   ├── decisions/              # Architecture decision records
│   ├── setup.md                # Detailed setup guide
│   ├── project-structure.md    # File-by-file breakdown
│   ├── scraper-design.md       # Scraper architecture decisions
│   └── changelog.md            # Version history
├── tests/                      # pytest test suite
├── data/
│   ├── results/                # Output reports
│   ├── cache/                  # Cached search results
│   └── jobs.db                 # Seen-jobs store (gitignored)
├── pyproject.toml
└── .env.example

Dependencies

• anthropic (0.49+) — Claude API with web search tool (the anthropic provider)
• requests (2.31+) — Bundesagentur Jobsuche client and the OpenRouter provider
• python-jobspy (1.1+) — Direct job board scraping
• pydantic (2.0+) — Data validation and job listing models
• gradio (4.0+) — Web interface
• pandas (2.0+) — Data analysis
• python-dotenv (1.0+) — Environment configuration

Development

# Run tests
uv run pytest tests/ -v

# Code quality
uv run black src/ tests/
uv run ruff check src/ tests/
uv run mypy src/

All four gates run in CI on every push and pull request (.github/workflows/ci.yml); they must pass before a branch merges. To keep git blame readable across the bulk reformatting commit:

git config blame.ignoreRevsFile .git-blame-ignore-revs

tests/test_characterization.py pins current behaviour — including behaviour already identified as wrong in the architecture review. A failing test_characterizes_* test usually means a planned fix landed; update the test to assert the new behaviour rather than working around it.

Docs

• Architecture Review & Refactor Plan
• Decision Records — ADR-001 Bundesagentur, ADR-002 LinkedIn, ADR-003 Google Jobs
• Setup Guide
• Project Structure
• Scraper Design
• Changelog

License

MIT License

---

Version: 0.3.0