Skip to content

CLAUDE.md

Latest commit

 

History

History
146 lines (104 loc) · 7.76 KB

CLAUDE.md

File metadata and controls

146 lines (104 loc) · 7.76 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Poolex is an Elixir library for managing pools of workers, inspired by poolboy. It provides a GenServer-based worker pool with configurable overflow workers, delayed shutdown, and pluggable implementations for managing worker and caller collections.

Development Commands

Core Development Tasks

  • mix deps.get - Install dependencies
  • mix check - Run all code analysis & testing tools (includes tests, dialyzer, credo, format check)
  • mix test - Run tests only
  • mix format - Format code
  • mix dialyzer - Run Dialyzer type checking
  • mix credo - Run code analysis

Testing

  • mix test - Run all tests
  • mix test --cover - Run tests with coverage
  • mix test test/poolex_test.exs - Run specific test file
  • mix test --only focus - Run only focused tests (tagged with @tag :focus)

Documentation

  • mix docs - Generate documentation
  • mix hex.docs open - Open generated docs in browser

Architecture

Core Components

The main pool GenServer (Poolex) manages several internal components:

  • State Management: Poolex.Private.State - Core state structure containing pool configuration and runtime state
  • Worker Collections: Pluggable implementations for managing different types of workers:
    • idle_workers - Available workers ready for checkout
    • busy_workers - Workers currently in use
    • idle_overflowed_workers - Overflow workers with delayed shutdown
  • Caller Management: waiting_callers - Queue of processes waiting for workers
  • Monitoring: Poolex.Private.Monitoring - Tracks worker and caller processes
  • Metrics: Poolex.Private.Metrics - Telemetry integration for pool metrics

Worker Lifecycle

  1. Workers are started via DynamicSupervisor under Poolex.Private.Supervisor
  2. Idle workers are stored in configurable collections (List or ErlangQueue implementations)
  3. When requested, workers move from idle → busy state
  4. After use, workers either return to idle or are shut down (if overflow)
  5. Failed workers are automatically restarted with configurable retry intervals

Pluggable Implementations

The library uses behaviour-based implementations for:

  • Workers: Poolex.Workers.Behaviour - How to store/retrieve worker PIDs
  • Callers: Poolex.Callers.Behaviour - How to queue waiting callers

Built-in implementations:

  • Poolex.Workers.Impl.List - Simple list-based storage
  • Poolex.Workers.Impl.ErlangQueue - Erlang queue-based storage
  • Poolex.Callers.Impl.ErlangQueue - Erlang queue for caller management

Key Features

  • Overflow Workers: Temporary workers beyond the base pool size
  • Worker Shutdown Delay: Configurable delay before stopping overflow workers
  • Failed Worker Retry: Automatic retry of failed worker initialization
  • Pool Metrics: Telemetry integration for monitoring pool state
  • Process Monitoring: Automatic cleanup when workers or callers crash

Configuration Options

Key pool configuration (see Poolex.poolex_option() type):

  • worker_module - Module implementing the worker (required)
  • workers_count - Base number of workers (required)
  • max_overflow - Additional workers allowed beyond base count
  • worker_shutdown_delay - Delay before stopping overflow workers
  • pool_id - Identifier for the pool (defaults to worker_module)
  • worker_args - Arguments passed to worker start function
  • *_impl options - Custom implementations for worker/caller management

Testing Patterns

  • Test files are in test/ directory
  • Test support modules in test/support/
  • Start pools with PoolHelpers.start_pool/1 (test/support/pool_helpers.ex, wraps start_supervised); the default pool_id is the worker module
  • Common test pattern: start pool → call Poolex.run/3 → verify behavior
  • Don't wait for the pool with a fixed Process.sleep/1 — that is what made the suite flaky on loaded CI runners. Wrap the assertion in PoolHelpers.eventually/2 (retries until it passes or times out), or assert_receive an explicit signal sent after the action under test. launch_long_task(s) already returns only once every caller holds a worker or is queued
  • Make sure an eventually assertion can't pass on the initial state (e.g. await the tasks that change the state first)
  • Casts sent by the test process are handled before its next :sys.get_state/1 or GenServer.call to the pool, so no wait is needed in between
  • Keep fixed sleeps only for checking that something does not happen, or where elapsed time is the point (shutdown delays); leave margins of hundreds of milliseconds
  • Timing flakes rarely show up on an idle dev machine. To shake them out locally, run the suite under CPU contention, e.g. start one yes > /dev/null per core and loop ERL_FLAGS="+S 2:2" mix test --seed $RANDOM (under such load sleeps overshoot by up to ~400 ms)
  • To reproduce races between messages in the pool's mailbox deterministically, :sys.suspend(pool) → trigger the senders → wait for them (e.g. assert_receive {:DOWN, ...}) → :sys.resume(pool). This emulates a loaded pool. Example: describe "caller dies abnormally right after release" in test/poolex_manual_acquisition_test.exs
  • Registry-based naming for test isolation

Git Workflow

  • Uses git-flow branch naming: main for releases, develop for development; PRs target develop
  • git flow tooling is not initialized in the repo — create feature branches with plain git checkout -b feature/<name> from develop
  • Always rebase, never merge when integrating upstream changes
  • Run mix check before committing. After a failed run, mix check re-runs only the failed tools (retry mode); use mix check --no-retry for a full run

Requirements

  • Elixir >= 1.17, Erlang/OTP >= 25
  • Use asdf for version management (see .tool-versions)

Recent Improvements & Known Issues

Known Issues

  1. remove_idle_workers!/2 Design Issue
    • Current behavior: Removes workers from idle list but doesn't stop them
    • Problem: Workers continue running but are inaccessible to pool (potential resource leak)
    • Root cause: Can't easily unmonitor by pid (only by reference)
    • Workaround: Would need reverse monitor map (pid → reference)
    • Better solution: See "Planned Architecture Improvements" in TODO.md

Architecture Considerations

Current Monitoring Approach:

  • Uses Process.monitor with %{reference() => kind_of_process()} map
  • Limitation: Can't unmonitor by pid, only by reference
  • Makes features like remove_idle_workers! difficult to implement correctly
  • Pool sets trap_exit: true in init/1 and handles {:EXIT, pid, reason} in handle_info (stops the pool gracefully on exit signals)
  • Every acquire/2 (and so every run/3) spawns a watcher process (start_manual_monitor/3) that reports an abnormal caller death as {:manual_caller_down, worker, monitor_pid}. The report is asynchronous and can arrive after the worker was released and handed to another caller, so the pool acts on it only while manual_monitors[worker] still points at that monitor. Any new asynchronous message about a specific worker needs the same kind of ownership check

Proposed Future Improvement (see TODO.md):

  • Replace Process.monitor with Process.link + trap_exit (full refactoring)
  • Would eliminate need for monitors map entirely
  • Would make remove_idle_workers! trivial: Process.unlink(worker) + terminate_child
  • Significant architectural simplification

Development Notes

  • When adding new worker lifecycle features, be aware of monitoring limitations
  • Consider the Process.link refactoring (in TODO.md) for major features requiring worker removal
  • Always run mix check before committing (includes tests, dialyzer, credo)