Workers & Channels

ShipQ includes a workers/channels system for background job processing and real-time WebSocket communication. It’s backed by Redis (job queue and state), Centrifugo (WebSocket hub), and generates all the plumbing — Go worker binary, typed channel definitions, TypeScript clients, and tests.

Prerequisites

Before bootstrapping workers, you need:

• shipq auth must have been run first (workers depend on the auth system)
• Redis available on your system (redis-server on PATH or accessible via URL)
• Centrifugo available on your system (centrifugo on PATH)

Bootstrapping the Workers System

shipq workers

This single command performs the full bootstrap:

1. Verifies prerequisites ([auth] configured, redis-server and centrifugo available)
2. Writes a [workers] section into shipq.ini with local defaults
3. Generates and applies a job_results migration (runs shipq migrate up internally)
4. Compiles query definitions and the handler registry
5. Generates typed channel Go code
6. Generates cmd/worker/main.go — the background worker binary
7. Generates centrifugo.json — Centrifugo configuration
8. Generates TypeScript channel clients (shipq-channels.ts) and framework helpers
9. Generates tests for the workers/channels system

After bootstrapping, tidy up dependencies:

go mod tidy

Configuration

After shipq workers, your shipq.ini will have a [workers] section:

[workers]
redis_url = redis://localhost:6379
centrifugo_url = http://localhost:8000
centrifugo_api_key = <auto-generated>
centrifugo_secret = <auto-generated>

The API key and secret are auto-generated during bootstrap. These are used for:

• centrifugo_api_key: Server-to-Centrifugo API authentication
• centrifugo_secret: HMAC signing of JWT tokens for client-to-Centrifugo authentication

Running the Services

Workers require multiple services running simultaneously. Open separate terminals for each:

# Terminal 1: Start Redis
shipq start redis

# Terminal 2: Start Centrifugo (WebSocket hub)
shipq start centrifugo

# Terminal 3: Start the background worker
shipq start worker

# Terminal 4: Start your application server
shipq start server

Each service can also be started manually:

• Redis: redis-server (or your preferred method)
• Centrifugo: uses the generated centrifugo.json config
• Worker: go run ./cmd/worker
• Server: go run ./cmd/server

Background Jobs

The workers system uses Redis-backed job queues (powered by Machinery) for background task processing.

How jobs work

1. Your application server dispatches a job by enqueueing it to Redis
2. The worker process (cmd/worker/main.go) picks up the job from the queue
3. The worker executes the job handler and records the result in the job_results table
4. Optionally, the worker can publish a real-time notification via Centrifugo when the job completes

Job results tracking

The job_results migration (generated during bootstrap) creates a table for tracking job execution status. This gives you visibility into job history, failures, and retry behavior.

Real-Time Channels

Channels provide real-time, bidirectional communication between your server and connected clients via WebSockets, using Centrifugo as the transport layer.

How channels work

1. Server-side: Your Go code publishes messages to named channels via the Centrifugo API
2. Client-side: TypeScript clients subscribe to channels and receive messages in real time
3. Authentication: Channel subscriptions are authenticated using Centrifugo-specific JWT tokens signed with the centrifugo_secret (these are separate from the cookie-based HTTP auth system — JWTs are only used for the Centrifugo WebSocket protocol)

Typed channel definitions

ShipQ generates typed channel code so that both the Go server and TypeScript client agree on:

• Channel names and naming patterns
• Message payload types
• Subscription permissions

This type safety spans the full stack — the same channel definition produces both the Go publishing code and the TypeScript subscription client.

TypeScript client

After shipq workers, a shipq-channels.ts file is generated with:

• Typed channel subscription helpers
• Automatic JWT token handling for Centrifugo authentication
• Framework-specific hooks (React or Svelte, depending on your [typescript] framework setting)

Recompiling After Changes

If you modify channel definitions after the initial bootstrap, you don’t need to run the full shipq workers again. Use the fast recompile command:

shipq workers compile

This performs only the codegen/compile steps:

• Channel discovery
• Typed channel Go code generation
• Worker main regeneration
• Centrifugo config regeneration
• TypeScript client regeneration
• Query definition compilation
• Handler registry compilation

It skips migrations, go mod tidy, prerequisite checks, and library embedding — making it much faster for iterative development.

Email Integration

If you need email sending (e.g., for email verification or password reset), the workers system provides the infrastructure. After setting up workers:

shipq email

This generates email verification and password reset flows that use the worker queue to send emails asynchronously. See the Authentication guide for details.

Architecture Overview

┌──────────────┐      ┌──────────────┐      ┌──────────────────┐
│  Your Server │─────▶│    Redis     │◀─────│  Worker Process  │
│ (cmd/server) │      │  (job queue) │      │  (cmd/worker)    │
└──────┬───────┘      └──────────────┘      └────────┬─────────┘
       │                                              │
       │  publish                          publish    │
       ▼                                              ▼
┌──────────────────────────────────────────────────────────────┐
│                      Centrifugo                              │
│                   (WebSocket hub)                             │
└──────────────────────────────┬───────────────────────────────┘
                               │
                     WebSocket │ connections
                               ▼
                  ┌────────────────────────┐
                  │   Browser / Client     │
                  │  (shipq-channels.ts)   │
                  └────────────────────────┘

Both the application server and the worker process can publish messages to Centrifugo. Clients connect to Centrifugo directly over WebSockets and receive real-time updates.

Testing

ShipQ generates tests for the workers/channels system during bootstrap. Run them alongside your other tests:

go test ./... -v

The generated tests verify:

• Job dispatch and execution
• Channel publishing and subscription
• Authentication of channel connections
• End-to-end worker flow (dispatch → execute → publish notification)

File Ownership

As with all ShipQ-generated code, files with the zz_generated_ prefix or the // Code generated by shipq. DO NOT EDIT. header are overwritten on every compile. Don’t hand-edit those.

Key generated files:

• cmd/worker/main.go — regenerated by shipq workers compile
• centrifugo.json — regenerated by shipq workers compile
• shipq-channels.ts — regenerated by shipq workers compile

Next Steps

• Authentication — Workers depend on auth; make sure it’s set up first
• TypeScript Clients — Learn about the generated channel client and framework hooks
• Deployment — Deploy your server, worker, Redis, and Centrifugo together