# Architecture
Source: https://docs.techulus.cloud/agents/architecture

State machine, drift detection, build pipeline, and leased commands.

## State Machine

The agent uses a two-state reconciliation model:

```mermaid theme={null}
stateDiagram-v2
    IDLE: IDLE (poll every 15s)
    PROCESSING: PROCESSING (no poll)

    IDLE --> PROCESSING: Drift detected
    PROCESSING --> IDLE: Done / Failed / Timeout (5 min)
```

### IDLE State

* Polls the control plane every 15 seconds for expected state.
* Compares expected state against actual state.
* Transitions to `PROCESSING` when drift is detected.

### PROCESSING State

* Uses a snapshot of expected state without re-polling.
* Applies one change at a time:
  1. Stop orphan containers with no deployment ID.
  2. Start containers in `created` or `exited` state.
  3. Deploy missing containers.
  4. Redeploy containers with the wrong image.
  5. Update DNS records.
  6. Update Traefik routes on proxy nodes.
  7. Update WireGuard peers.
* Times out after 5 minutes.
* Always reports status before returning to `IDLE`.

## Drift Detection

Drift detection is deterministic and uses hashes:

* **Containers**: missing, orphaned, wrong state, or image mismatch.
* **DNS**: hash of sorted records.
* **Traefik**: hash of sorted routes on proxy nodes.
* **WireGuard**: hash of sorted peers.

## Build System

Agents can build container images directly from GitHub sources:

1. Poll for pending builds.
2. Claim the build to prevent duplicate work.
3. Clone the repository using a GitHub App installation token.
4. Run Railpack to generate a build plan, or use the existing Dockerfile.
5. Build the image with BuildKit.
6. Push the image to the installation's user-owned Google Artifact Registry repository using the Writer credential supplied by the control plane.
7. Update build status.

Build logs stream to VictoriaLogs in real time.

## Leased Commands

Agents report status to the control plane on a short interval. The status response
can include one command for operations that cannot be modeled purely as expected
state. The command's attempt number acts as a generation guard: the agent reports
the result with that attempt, and stale completions from older attempts are
ignored. If an agent crashes or stops renewing the command through status
reports, the command can be retried up to the fixed attempt limit.

When an agent is idle, it also makes a bounded long-poll request for queued-work
wakeups. The control plane uses PostgreSQL `LISTEN/NOTIFY` to end that request as
soon as work is enqueued. The notification is only a wake hint. The durable work
queue remains authoritative, and the existing status request performs the atomic
lease before the agent starts work.

The agent continues periodic status reports while long polling. If the wait
endpoint or PostgreSQL notification listener is unavailable, the agent backs off
and queued work is still discovered through the next status report.

| Type              | Description                                   |
| ----------------- | --------------------------------------------- |
| `restart`         | Restart a specific container                  |
| `stop`            | Stop a specific container                     |
| `force_cleanup`   | Force remove containers for a service         |
| `cleanup_volumes` | Remove volume directories for a service       |
| `deploy`          | Handled through expected-state reconciliation |

## Serverless Gateway

Proxy agents run a local HTTP wake gateway on `127.0.0.1:18080`. The control
plane emits Traefik routes to this gateway only on proxy nodes that host a local
proxy deployment for that serverless service. Proxies without a local
proxy-hosted deployment do not emit a public HTTP route for that service.
Worker-only serverless services keep normal direct routes because worker
deployments do not sleep.

This means public DNS or external load balancers must route a serverless
service's traffic only to proxy nodes that own a local proxy replica for that
service.

For serverless service traffic, Traefik forwards the public request to the local
gateway. The gateway resolves the host from expected-state serverless metadata,
queues `wake_started` as a status transition, starts local proxy-hosted
containers from expected-state config, waits for ready upstreams, then proxies
the original request to the selected container over the WireGuard mesh.

The gateway keeps a short upstream cache and collapses concurrent wake requests
for the same host. Request cancellation is respected while waiting for a shared
wake result.

The gateway tracks request activity in memory. When a host stays idle for the
service's configured sleep timeout, the proxy agent removes only its local
proxy-hosted containers and queues a `sleep` status transition. The control plane
records the transition and updates dashboard/routing state; it does not run the
idle timer.

If a status report fails, pending serverless transitions stay queued in memory
and are retried. A locally slept deployment is also guarded from immediate
reconcile until a fresh expected-state fetch confirms the control plane recorded
the sleep or rejects it by continuing to advertise `desiredState: "running"`.


# Introduction
Source: https://docs.techulus.cloud/agents/introduction

What the agent does, node types, and how it communicates with the control plane.

The agent is a lightweight Go binary that runs on every server in your cluster. It polls the control plane for expected state and reconciles containers, networking, and routing automatically.

## Node Types

Each agent runs in one of two modes:

| Type   | Flag      | Traefik | Description                    |
| ------ | --------- | ------- | ------------------------------ |
| Worker | Default   | No      | Runs containers only           |
| Proxy  | `--proxy` | Yes     | Handles TLS and public traffic |

### Proxy Node

* Runs Traefik for TLS termination.
* Receives Traefik routes from the control plane.
* Handles public traffic and routes requests to containers over WireGuard.
* Collects and ships Traefik access logs.

### Worker Node

* Does not run Traefik.
* Receives empty Traefik route sets from the control plane.
* Skips Traefik-related drift detection and reconciliation.
* Keeps a lighter runtime footprint focused on container workloads.

## Container Labels

The agent tracks managed containers with Podman labels:

| Label                    | Description                         |
| ------------------------ | ----------------------------------- |
| `techulus.deployment.id` | Links the container to a deployment |
| `techulus.service.id`    | Links the container to a service    |
| `techulus.service.name`  | Human-readable service name         |

Containers without `techulus.deployment.id` are treated as orphans and cleaned up.

## Command Line Flags

| Flag              | Default  | Description                               |
| ----------------- | -------- | ----------------------------------------- |
| `--url`           | Required | Control plane URL                         |
| `--token`         | Empty    | Registration token, required on first run |
| `--logs-endpoint` | Empty    | VictoriaLogs endpoint for log shipping    |
| `--proxy`         | `false`  | Run as a proxy node                       |


# Setup
Source: https://docs.techulus.cloud/agents/setup

Install the agent and add servers to your cluster.

## Automated Setup

The setup script installs all dependencies, registers the server, and configures systemd services.

### Interactive

```bash theme={null}
curl -sSL https://your-control-plane.com/setup.sh | sudo bash
```

The script prompts for the control plane URL, registration token, and node type.

### Non-Interactive

```bash theme={null}
export CONTROL_PLANE_URL=https://your-control-plane.com
export REGISTRATION_TOKEN=your-token
export IS_PROXY=false
curl -sSL $CONTROL_PLANE_URL/setup.sh | sudo bash
```

## What Gets Installed

### All Nodes

* **WireGuard** — encrypted mesh networking
* **Podman** — container runtime
* **BuildKit** — container image builds
* **Railpack** — build plan generation

### Proxy Nodes Only

* **Traefik** — reverse proxy and TLS termination
* **CrowdSec** — automated threat detection and IP banning

The script also enables IP forwarding and configures firewall rules for ports 80, 443, and 51820 (WireGuard).

## Registration

Servers register with the control plane using a one-time token. Generate a token from the web UI, then pass it during the first run.

On registration, the agent:

1. Generates an Ed25519 signing key pair and a WireGuard key pair.
2. Sends its public keys and IP addresses to the control plane.
3. Receives a server ID, WireGuard subnet, and encryption key.
4. Saves configuration to `/var/lib/techulus-agent/config.json`.

After registration, the token is invalidated. Subsequent runs do not require a token.

## Manual Setup

### Worker Node

```bash theme={null}
sudo apt update && sudo apt upgrade -y
sudo apt install wireguard wireguard-tools podman git -y
sudo systemctl enable --now podman.socket

# Install Railpack
curl -sSL https://railpack.com/install.sh | sh
sudo ln -s ~/.railpack/bin/railpack /usr/local/bin/railpack

# Install BuildKit
curl -sSL https://github.com/moby/buildkit/releases/download/v0.26.3/buildkit-v0.26.3.linux-amd64.tar.gz \
  | sudo tar -xz -C /usr/local
```

### Proxy Node

Install everything above, plus Traefik:

```bash theme={null}
TRAEFIK_VERSION="v3.2.3"
curl -fsSL "https://github.com/traefik/traefik/releases/download/${TRAEFIK_VERSION}/traefik_${TRAEFIK_VERSION}_linux_amd64.tar.gz" \
  -o /tmp/traefik.tar.gz
sudo tar -xzf /tmp/traefik.tar.gz -C /usr/local/bin traefik
rm /tmp/traefik.tar.gz
```

### First Run

Worker node:

```bash theme={null}
sudo ./agent --url <control-plane-url> --token <registration-token>
```

Proxy node:

```bash theme={null}
sudo ./agent --url <control-plane-url> --token <registration-token> --proxy
```

## Running as a Service

### Worker Node

Create `/etc/systemd/system/techulus-agent.service`:

```ini theme={null}
[Unit]
Description=Techulus Cloud Agent
After=network.target podman.socket buildkitd.service
Wants=podman.socket

[Service]
Type=simple
ExecStart=/usr/local/bin/agent --url <control-plane-url>
Restart=always
RestartSec=5
KillMode=process

[Install]
WantedBy=multi-user.target
```

### Proxy Node

```ini theme={null}
[Unit]
Description=Techulus Cloud Agent
After=network.target podman.socket traefik.service buildkitd.service
Wants=podman.socket

[Service]
Type=simple
ExecStart=/usr/local/bin/agent --url <control-plane-url> --proxy
Restart=always
RestartSec=5
KillMode=process

[Install]
WantedBy=multi-user.target
```

Enable and start the service:

```bash theme={null}
sudo systemctl daemon-reload
sudo systemctl enable techulus-agent
sudo systemctl start techulus-agent
```

<Note>
  `KillMode=process` ensures only the agent process is stopped on restart, not the containers it manages.
  The rootful Podman API socket at `/run/podman/podman.sock` is required for container metrics collection.
</Note>

## Troubleshooting

### Agent restart kills containers

Ensure `KillMode=process` is set in the systemd service file.

### Containers stuck in "created" state

This is normal after a restart. The agent detects drift and starts them automatically.

### Checking agent logs

```bash theme={null}
sudo journalctl -u techulus-agent -f
```

### Inspecting containers

```bash theme={null}
podman ps -a --format "table {{.Names}}\t{{.State}}\t{{.Labels}}"
```


# Updating
Source: https://docs.techulus.cloud/agents/updating

Update agents from the control plane.

## Update an agent

When an update is available, open the server in the control plane and click **Update**. Confirm the target version, then click **Queue upgrade**.

The control plane sends a work item over the authenticated agent channel. The agent:

1. Downloads the target version of the agent binary from GitHub releases.
2. Verifies the SHA256 checksum.
3. Replaces the existing binary.
4. Restarts itself after installation.

No configuration or dependencies are changed during an update.

## Manual update

Download the binary for your architecture:

```bash theme={null}
# For x86_64
curl -fsSL https://github.com/techulus/cloud/releases/latest/download/agent-linux-amd64 \
  -o /usr/local/bin/techulus-agent
chmod +x /usr/local/bin/techulus-agent

# For arm64
curl -fsSL https://github.com/techulus/cloud/releases/latest/download/agent-linux-arm64 \
  -o /usr/local/bin/techulus-agent
chmod +x /usr/local/bin/techulus-agent
```

Restart the service:

```bash theme={null}
sudo systemctl restart techulus-agent
```

## Verify

Check that the agent is running after the update:

```bash theme={null}
sudo systemctl status techulus-agent
sudo journalctl -u techulus-agent -n 20
```


# Public API reference
Source: https://docs.techulus.cloud/api/public-api

Automate service configuration, deployments, and observability.

The public API is available under `/api/v1`. The `tc` CLI uses this API directly.

## Authentication

Send your CLI API key in the `X-API-Key` header on every resource request:

```bash theme={null}
curl https://cloud.example.com/api/v1/me \
  --header "X-API-Key: $TECHULUS_API_KEY"
```

Browser cookies and bearer tokens cannot access resource endpoints. A bearer token or browser session can only create an API key with `POST /api/v1/api-keys`. An API key cannot create another API key.

`tc auth login` completes the device flow, creates an API key, and stores it locally. `tc auth logout` only removes the local credentials.

### Create an API key

```http theme={null}
POST /api/v1/api-keys
Authorization: Bearer <device-access-token>
Content-Type: application/json
```

```json theme={null}
{
  "name": "CLI workstation",
  "metadata": {
    "machineName": "workstation",
    "platform": "darwin/arm64"
  }
}
```

`name` is required and can contain up to 32 characters. The response contains the secret once:

```json theme={null}
{
  "apiKey": "tcl_...",
  "keyId": "...",
  "name": "CLI workstation"
}
```

## Authorization and service identity

Roles are global for the current Techulus Cloud installation. There are no project-level permissions.

* `reader`, `developer`, and `admin` can read resources.
* `developer` and `admin` can change configuration and deploy services.

Service operations use the globally unique service ID:

```text theme={null}
/api/v1/services/{serviceId}
```

Nested project and environment paths remain available for browsing collections. Service responses include an authoritative `target` with the containing project, environment, and service labels.

## Errors

Every API error uses the same shape:

```json theme={null}
{
  "message": "Resource not found",
  "code": "NOT_FOUND"
}
```

Use `code` for automation. Do not match the human-readable `message`.

## Identity and collections

| Method | Path                                                                 | Description                                           |
| ------ | -------------------------------------------------------------------- | ----------------------------------------------------- |
| `GET`  | `/api/v1/me`                                                         | Return the API-key user, role, and key-backed session |
| `GET`  | `/api/v1/projects`                                                   | List projects                                         |
| `GET`  | `/api/v1/projects/{projectId}/environments`                          | List environments in a project                        |
| `GET`  | `/api/v1/projects/{projectId}/environments/{environmentId}/services` | List services in an environment                       |
| `GET`  | `/api/v1/services/{serviceId}`                                       | Return the target and canonical service configuration |

Collection responses use an opaque keyset cursor:

```json theme={null}
{
  "services": [
    {
      "id": "...",
      "name": "web",
      "hostname": "web",
      "source": { "type": "image", "image": "nginx:1.27" },
      "createdAt": "2026-07-20T00:00:00.000Z"
    }
  ],
  "nextCursor": null
}
```

Pass `nextCursor` as `?cursor=...`. `limit` defaults to 100 and accepts values from 1 through 100.

## Service resources

The paths in this table are relative to `/api/v1/services/{serviceId}`.

| Method       | Path suffix                  | Description                                                                            |
| ------------ | ---------------------------- | -------------------------------------------------------------------------------------- |
| `GET`, `PUT` | `/configuration`             | Read safe current/active configuration or atomically replace all managed configuration |
| `POST`       | `/configuration/plan`        | Validate and plan a complete managed configuration replacement                         |
| `GET`        | `/status`                    | Read source, latest build and rollout, and persisted deployments                       |
| `POST`       | `/deploy`                    | Queue an image rollout or GitHub build                                                 |
| `GET`        | `/logs`                      | Search logs and optionally long poll                                                   |
| `GET`        | `/rollouts`                  | List rollouts with cursor pagination                                                   |
| `GET`        | `/rollouts/{rolloutId}`      | Read a contained rollout and its deployments                                           |
| `GET`        | `/rollouts/{rolloutId}/logs` | Search bounded rollout logs                                                            |
| `GET`        | `/builds`                    | List GitHub builds with cursor pagination                                              |
| `GET`        | `/metrics`                   | Read metrics with explicit provider state                                              |
| `GET`        | `/revisions`                 | Read a redacted configuration changelog                                                |

Rollout and build collections accept `limit` from 1 through 100 and an opaque `cursor`. Their default limit is 25. Revisions accept their returned opaque `cursor` and return up to 25 items.

## Configuration

`GET /configuration` returns these distinct states:

* `current` is the mutable configuration stored for the service.
* `active` is the immutable specification used by the active deployment.
* `activeRevisionId` and `activeDeploymentId` identify that active state.
* `hasPendingChanges` and `changes` compare current and active state.
* `management.patchable` and `management.blockers` explain whether the API can safely manage the service.

A GitHub service keeps mutable repository settings in `current.source`. Its `active.source` comes from the immutable service revision used by the active deployment. The public projection includes the snapshotted repository, branch, and root directory. It omits repository IDs and authentication details.

Configuration and revision responses never include secret names, values, or ciphertext.

### Replace configuration

`POST /configuration/plan` accepts the same complete desired configuration as `PUT /configuration`. It performs no writes and returns the authoritative `target`, `action`, `currentVersion`, `desiredVersion`, and structured `changes` (`field`, `from`, and `to`). Service IDs on these flat service routes are authorized installation-globally rather than being scoped by a project ID in the URL.

`PUT /configuration` is atomic and replaces the complete managed configuration. Send the `currentVersion` returned by the plan as one quoted strong ETag (for example, `If-Match: "sha256:…"`). Missing, unquoted, multiple, weak, or otherwise invalid headers are rejected; a service change after planning returns `409 CONFIGURATION_PLAN_STALE`. A successful response returns the authoritative target and the structured change set that was applied. The request must contain exactly `name`, `source`, `hostname`, `ports`, `placement`, `healthCheck`, `startCommand`, `resources`, and `crons`. Omitted or unknown fields are rejected. `hostname` must be concrete and non-null. Use `null` to clear nullable fields, including `resources`.

Replacing the final public HTTP port with a domain automatically disables serverless in the mutable service configuration. The plan reports this side effect as a `serverless.enabled` change before apply. You do not send serverless settings in the replacement request.

```json theme={null}
{
  "name": "web",
  "source": {
    "type": "github",
    "repository": "https://github.com/techulus/cloud",
    "branch": "main",
    "rootDir": "web"
  },
  "hostname": "cloud",
  "ports": [
    {
      "containerPort": 3000,
      "public": true,
      "domain": "cloud.example.com"
    }
  ],
  "placement": {
    "mode": "automatic",
    "replicas": 3
  },
  "healthCheck": {
    "cmd": "curl --fail http://localhost:3000/health",
    "interval": 10,
    "timeout": 5,
    "retries": 3,
    "startPeriod": 30
  },
  "startCommand": null,
  "resources": {
    "cpuCores": 1,
    "memoryMb": 512
  },
  "crons": [
    {
      "path": "/api/cron/digest",
      "schedule": "0 5 * * *"
    }
  ]
}
```

`crons` is the complete desired list of UTC service cron definitions. Each item contains exactly `path` and `schedule`. Paths must be unique, origin-relative paths without a query string or fragment. Send an empty list to remove all cron definitions. `current.crons` returns definitions only; runtime status remains outside the configuration fingerprint. Cron-only changes take effect without creating a deployment revision. See [Service crons](/services/configuration#service-crons) for request and secret behavior.

The `tc apply` command sends this complete replacement. A linked `techulus.yml` stores its target identity under `target.serviceId` and desired configuration under `service`. `tc link` writes current cron definitions into the manifest, and `tc apply` treats that manifest list as authoritative. Explicit CLI targeting uses `--service <serviceId>`. During interactive `tc link`, a project with exactly one environment selects it automatically; zero environments is an error and multiple environments prompt for a choice.

The API supports these source variants:

```json theme={null}
{ "type": "image", "image": "nginx:1.27" }
```

```json theme={null}
{
  "type": "github",
  "repository": "https://github.com/owner/repository",
  "branch": "main",
  "rootDir": "services/api"
}
```

GitHub repository URLs are canonical HTTPS `github.com` URLs. `rootDir` is required and must either stay inside the repository or be `null` to clear an existing build root. Source conversion and GitHub repository switching are not supported.

Use automatic placement for stateless services:

```json theme={null}
{ "mode": "automatic", "replicas": 3 }
```

To autoscale an eligible service, replace fixed `replicas` with a range:

```json theme={null}
{
  "mode": "automatic",
  "autoscaling": { "minReplicas": 2, "maxReplicas": 8 }
}
```

Autoscaling requires automatic placement, a stateless non-serverless service
without volumes, and both CPU and memory limits. Minimum and maximum values must
be from 1 through 32, with minimum no greater than maximum. CPU and memory use
fixed 60% targets. Configuration responses expose the mutable policy in
`current.placement` and the deployed policy plus concrete active target in
`active.placement`. Policy changes participate in plan/apply fingerprints.
To disable autoscaling, send the fixed automatic-placement form with `replicas`.

Use manual placement to choose exact servers:

```json theme={null}
{
  "mode": "manual",
  "placements": [
    { "serverId": "server-id", "count": 2 }
  ]
}
```

Manual placement requires online servers with WireGuard configured. Serverless services require proxy servers, including when automatic placement is used. A replacement that removes the final public HTTP domain can move the service to worker placement because the same update disables serverless. Automatic placement is not available for stateful or volume-backed services. Submit replica changes through `placement`; the API rejects a top-level `replicas` field.

CLI manifests remain fixed-replica only in V1. Configure autoscaling through the
dashboard or public configuration API. `tc apply` continues to send the fixed
replica manifest contract.

The API only manages stateless services with HTTP ports. Existing volumes, stateful mode, TCP or UDP ports, TLS passthrough, or invalid resource limits return a conflict with an actionable code.

## Deployments and builds

`POST /deploy` uses the persisted service source. It does not apply configuration first.

* An image service queues a rollout.
* A GitHub service queues a build. A rollout starts after a successful build.

Deploy responses are always HTTP 202:

```json theme={null}
{
  "operation": "rollout",
  "status": "queued",
  "rolloutId": "...",
  "buildId": null
}
```

Image migration can return `status: "migration_started"` and `rolloutId: null`. GitHub returns:

```json theme={null}
{
  "operation": "build",
  "status": "build_queued",
  "rolloutId": null,
  "buildId": null
}
```

Build records are created asynchronously, so `buildId` is initially `null`. `/status` reports `latestBuild` and `latestRollout` independently.

Each GitHub deployment resolves an exact 40-character commit before it queues build work. The control plane creates an immutable service revision that snapshots the service configuration, source provenance, and reserved artifact identity. Every platform build references that revision and reads its repository, branch, commit, root directory, authentication mode, secrets, and final image URI from the revision.

After all platform images succeed, the control plane creates the final manifest and rolls out that same revision. Configuration changes made while a build is running do not alter its inputs. Retrying a failed or cancelled build creates a new revision with a new artifact identity. It never overwrites an artifact reserved by an earlier revision.

## Logs

Service logs accept these query parameters:

| Parameter         | Description                                                                   |
| ----------------- | ----------------------------------------------------------------------------- |
| `q`               | Literal, case-insensitive message search; maximum 200 characters              |
| `range`           | `1h`, `6h`, `24h`, or `7d`; defaults to `24h`                                 |
| `limit` or `tail` | 1 through 1,000 records; defaults to 100                                      |
| `cursor`          | Continue after the last emitted record by using the opaque `nextCursor` value |
| `wait`            | Long-poll for 0 through 20 seconds when `cursor` is present                   |

Round-trip `nextCursor` without decoding or modifying it. When `hasMore` is `true`, request the next page immediately. Otherwise, respect `pollAfterMs` before the next request. Keep the service, `q`, and `range` unchanged while reusing a cursor.

```json theme={null}
{
  "provider": "enabled",
  "logs": [
    {
      "deploymentId": "...",
      "stream": "stdout",
      "message": "server started",
      "timestamp": "2026-07-20T00:00:00.123456789Z"
    }
  ],
  "nextCursor": "eyJ2IjoxLCJ0IjoiLi4uIiwiZSI6Ii4uLiJ9",
  "hasMore": false,
  "pollAfterMs": 250
}
```

When logging is not configured, the response uses `provider: "disabled"` with an empty list. An upstream failure returns `502 LOG_PROVIDER_ERROR`. Log following returns `409 LOG_CURSOR_UNAVAILABLE` if an agent must be upgraded before it can provide deterministic cursors.

Rollout logs accept `q` and `limit`. Their response includes bounded stage messages for the contained rollout.

## Metrics

Metrics accept `range=1h|6h|24h|7d|30d`.

* A configured provider returns `{ "provider": "enabled", "metrics": ... }`. `metrics` can contain successful empty series.
* A disabled provider returns `{ "provider": "disabled", "metrics": null }`.
* An upstream failure returns `502 METRICS_PROVIDER_ERROR`.

The endpoint exposes fixed service metrics. It does not accept raw PromQL.

## CLI commands

The CLI uses the same endpoints documented above:

| Command                | Purpose                                       |
| ---------------------- | --------------------------------------------- |
| `tc config`            | Show current and active service configuration |
| `tc projects`          | List projects                                 |
| `tc environments`      | List project environments                     |
| `tc services`          | List environment services                     |
| `tc status`            | Show build, rollout, and deployment status    |
| `tc logs -q <text>`    | Search or follow service logs                 |
| `tc rollouts`          | List rollout history                          |
| `tc rollout <id>`      | Show rollout detail                           |
| `tc rollout logs <id>` | Fetch rollout logs                            |
| `tc builds`            | List builds when the source supports them     |
| `tc metrics`           | Query service metrics                         |
| `tc revisions`         | List the redacted revision changelog          |

`tc link` stores the selected `target.serviceId` and current managed configuration, including cron definitions, in `techulus.yml`. Relinking a manifest to a different service requires removing `target.serviceId` first. `tc apply` always displays the server-generated plan and prompts before writing; use `tc apply --yes` for noninteractive automation. Image and GitHub services use the same `tc link`, `tc apply`, `tc deploy`, and inspection commands.


# Architecture
Source: https://docs.techulus.cloud/architecture

System design, networking model, rollout lifecycle, and reconciliation flow.

# Techulus Cloud Architecture

## Overview

Techulus Cloud is a stateless container deployment platform built around three core principles:

1. **Workloads are disposable**: containers can be killed and recreated at any time.
2. **Two node types**: proxy nodes handle public traffic, worker nodes run containers.
3. **Networking is private-first**: services communicate over a WireGuard mesh, with public exposure routed through proxy nodes.

## Tech Stack

| Component             | Choice                   | Rationale                                                                          |
| --------------------- | ------------------------ | ---------------------------------------------------------------------------------- |
| Control Plane         | Next.js (full-stack)     | Single deployment with React frontend and API routes                               |
| Database              | Postgres + Drizzle       | Simple, low operational overhead, easy backup                                      |
| Background Jobs       | Inngest (self-hosted)    | Durable workflows, retries, event-driven orchestration                             |
| Server Agent          | Go                       | Single binary that shells out to Podman                                            |
| Container Runtime     | Podman                   | Docker-compatible, daemonless, bridge networking with static IPs                   |
| Reverse Proxy         | Traefik                  | Automatic HTTPS via Let's Encrypt, runs on proxy nodes only                        |
| Private Network       | WireGuard                | Full mesh coordinated by the control plane                                         |
| Service Discovery     | Built-in DNS             | Agent serves `.internal` domains                                                   |
| Agent Communication   | Pull-based HTTP          | Agent polls expected state, reports status, and long polls for queued-work wakeups |
| Source Image Registry | Google Artifact Registry | User-owned storage and provider-managed artifact cleanup                           |

## Node Types

| Type   | Traefik | Public Traffic          | Containers |
| ------ | ------- | ----------------------- | ---------- |
| Proxy  | Yes     | Handles TLS termination | Yes        |
| Worker | No      | None                    | Yes        |

* **Proxy nodes** handle incoming public traffic, terminate TLS using HTTP-01 ACME, and route requests to containers over WireGuard.
* **Worker nodes** run containers only and have no public exposure.

## Architecture Diagram

```mermaid theme={null}
graph TD
    Internet["Internet"] -->|"DNS"| P1

    CP["Control Plane<br/>Next.js + API Routes + Postgres"]
    GAR["User-owned Google Artifact Registry"]
    P1 -->|"HTTPS state, status, and work wait"| CP
    W1 -->|"HTTPS state, status, and work wait"| CP
    W2 -->|"HTTPS state, status, and work wait"| CP
    P1 -->|"Build push and deployment pull"| GAR
    W1 -->|"Build push and deployment pull"| GAR
    W2 -->|"Build push and deployment pull"| GAR

    subgraph Servers
        P1["Proxy Node<br/>Agent · Podman · Traefik · DNS · WireGuard<br/>WG: 10.100.1.1 · Containers: 10.200.1.2-254"]
        W1["Worker Node 1<br/>Agent · Podman · DNS · WireGuard<br/>WG: 10.100.2.1 · Containers: 10.200.2.2-254"]
        W2["Worker Node 2<br/>Agent · Podman · DNS · WireGuard<br/>WG: 10.100.3.1 · Containers: 10.200.3.2-254"]
        P1 -. "WireGuard Mesh" .- W1
        P1 -. "WireGuard Mesh" .- W2
        W1 -. "WireGuard Mesh" .- W2
    end
```

## Agent State Machine

The agent uses a two-state machine to prevent race conditions during reconciliation.

```mermaid theme={null}
stateDiagram-v2
    IDLE: IDLE (poll every 15s)
    PROCESSING: PROCESSING (no poll)

    IDLE --> PROCESSING: Drift detected
    PROCESSING --> IDLE: Done / Failed / Timeout (5 min)
```

### IDLE State

* Poll the control plane every 15 seconds for expected state.
* Compare expected state versus actual state for containers, DNS, Traefik, and WireGuard.
* Report status on a separate short interval and remain in `IDLE` if no drift exists.
* If drift is detected, snapshot expected state and transition to `PROCESSING`.

Traefik drift detection only applies on proxy nodes.

### PROCESSING State

* Stop polling and work from the expected-state snapshot.
* Apply one change at a time with verification.
* Re-check drift after every change.
* Transition back to `IDLE` once drift is resolved.
* Force a return to `IDLE` after 5 minutes if reconciliation stalls.
* Always send a status report before returning to `IDLE`.

### Drift Detection

The agent uses hash comparisons for deterministic drift detection:

* **Containers**: missing, orphaned, wrong state, or image mismatch.
* **DNS**: hash of sorted records versus current DNS config.
* **Traefik**: hash of sorted routes and certificates versus current config on proxy nodes, plus confirmation that Traefik successfully reloaded the newest dynamic files.
* **WireGuard**: hash of sorted peers versus current `wg0.conf`.

### Container Reconciliation Order

1. Stop orphan containers with no deployment ID.
2. Start containers in `created` or `exited` state.
3. Deploy missing containers.
4. Redeploy containers with wrong state or image mismatch.
5. Update DNS records.
6. Update Traefik certificates and routes on proxy nodes, then confirm a successful reload.
7. Update WireGuard peers.

## Rollout Stages

```text theme={null}
queued -> preparing -> certificates -> deploying -> health_check -> dns_sync -> completed
```

| Stage          | Description                                                                                                                                                                        |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `queued`       | Rollout is waiting for the previous rollout of the service to finish                                                                                                               |
| `preparing`    | Placements and target servers are being validated                                                                                                                                  |
| `certificates` | Certificates for public domains are being provisioned                                                                                                                              |
| `deploying`    | Agents are creating and starting candidate containers                                                                                                                              |
| `health_check` | Configured health checks are being evaluated; without one, a running container satisfies this stage                                                                                |
| `dns_sync`     | Displayed as **Routing traffic**. Every frozen workload target must have matching DNS; every frozen proxy target must also have matching routes and certificates loaded by Traefik |
| `completed`    | Routing convergence is confirmed and old deployments can be stopped                                                                                                                |

Special states:

* `unknown`: the agent stopped reporting this deployment and the container may still exist.
* `stopped`: the container was explicitly stopped.
* `failed`: the deployment failed, such as during health checks.
* `rolled_back`: rollout failed and reverted to the previous deployment.

For each rollout, the control plane freezes the required target set immediately before promotion. Private services wait for their workload servers. Public services also wait for every proxy that was online at promotion. Agents acknowledge the exact rollout ID only after their expected snapshot converges; unrelated or stale acknowledgements cannot complete another rollout.

With no configured health check, `healthy` means only that the container is running. Routing convergence prevents completion before network configuration is live, but application-level readiness remains the responsibility of a user-configured health check.

## Pull Request Preview Isolation

An enabled GitHub service represents each eligible pull request as an ordinary,
visible stateless service in the project's `previews` environment. An existing
environment with that name is reused; otherwise the control plane creates it.
The environment remains after its last preview closes and may also contain
unrelated user-created services.

Each copy has an independent service ID, revision, build, rollout, deployment
set, GAR package, and generated route while reusing the normal runtime
pipeline. It is copied from the base service only when first created, so later
user edits to the visible preview service survive pull request updates.

The control plane resolves the exact synthetic merge ref and queues an ordinary
build. The latest revision on the copy prevents superseded build and rollout
callbacks from deploying or reporting success. GitHub reports the transient
environment as ready only after health and routing convergence complete.
Closing, merging, or drafting the pull request deletes the copied service after
runtime teardown. [GAR cleanup policies](/infrastructure/registry) manage image
retention independently of service deletion, retaining the newest 10 versions
per package and deleting other versions older than 30 days.

## Networking

### IP Address Scheme

| Range            | Purpose                     |
| ---------------- | --------------------------- |
| `10.100.X.1`     | WireGuard IP for server `X` |
| `10.200.X.2-254` | Container IPs on server `X` |

`X` is the server subnet ID from `1` to `255`.

### WireGuard Mesh

Each server gets a `/24` subnet for routing:

* Server 1: `10.100.1.0/24` with WireGuard IP `10.100.1.1`
* Server 2: `10.100.2.0/24` with WireGuard IP `10.100.2.1`

Every server peers with every other server. `AllowedIPs` includes both WireGuard and container subnets:

```ini theme={null}
AllowedIPs = 10.100.2.0/24, 10.200.2.0/24
```

### Container Network

Each server has a Podman bridge network:

```bash theme={null}
podman network create \
  --driver bridge \
  --subnet 10.200.1.0/24 \
  --gateway 10.200.1.1 \
  --disable-dns \
  techulus
```

Containers receive static IPs assigned by the control plane:

```bash theme={null}
podman run -d \
  --name service-deployment \
  --network techulus \
  --ip 10.200.1.2 \
  --label techulus.deployment.id=<deployment-id> \
  --label techulus.service.id=<service-id> \
  traefik/whoami
```

### DNS Resolution

Each agent runs a built-in DNS server for `.internal` domains:

* It listens on the container gateway IP, such as `10.200.1.1`.
* It configures `systemd-resolved` to forward `.internal` queries.
* Records are pushed from the control plane through expected state.

Services resolve through `.internal` names with round-robin across replicas.

### Traefik on Proxy Nodes

Proxy nodes receive routes and certificates from the control plane:

* Routes live in `/etc/traefik/dynamic/routes.yaml`.
* Certificates live in `/etc/traefik/dynamic/tls.yaml`.
* Routes map `subdomain.example.com` to container IPs over WireGuard.
* TLS certificates are managed centrally by the control plane.
* `/.well-known/acme-challenge/*` is routed back to the control plane for ACME validation.

Worker nodes do not run Traefik.

### Serverless Containers

Public HTTP services can be configured as serverless. Serverless scale-to-zero is
proxy-local: deployments placed on proxy nodes may sleep after an idle period,
then wake on the next public request handled by that proxy. Stateless services
without volumes can use automatic placement across proxy nodes. Stateful or
volume-backed serverless services require manual placement on a proxy node.

Serverless uses the same declarative expected-state model as normal
deployments, but proxy agents own the local lifecycle decision:

* A sleeping deployment keeps `trafficState: "active"` but records
  `runtimeDesiredState: "stopped"` and `observedPhase: "sleeping"`.
* Expected state advertises proxy-hosted sleeping containers with
  `desiredState: "stopped"` so normal reconciliation does not restart them.
* The proxy gateway wakes local sleeping deployments from expected-state
  metadata and reports `wake_started` through the next status report.
* The proxy agent reports local sleep with a `sleep` status transition after it
  stops the local container.

```mermaid theme={null}
sequenceDiagram
    participant User
    participant Traefik
    participant Gateway as Agent wake gateway
    participant Agent
    participant CP as Control plane
    participant Container

    User->>Traefik: Request service domain
    Traefik->>Gateway: Route to 127.0.0.1:18080
    Gateway->>Gateway: Resolve host from expected-state metadata
    Gateway->>Agent: Queue wake_started status transition
    Agent->>Container: Start local Podman container
    Agent->>CP: Report wake_started and container status
    CP->>CP: Record waking/healthy state
    Gateway->>Container: Proxy original request
```

Proxy Traefik routes point to the local wake gateway only on proxy nodes that
host a local proxy deployment for that serverless service. Non-owner proxies do
not emit a public HTTP route for that service.

Public DNS or external load balancers must therefore send serverless traffic only
to proxy nodes that own a local proxy replica for that service. Cross-proxy wake
coordination is intentionally out of scope.

The wake gateway keeps the incoming request open while it starts local proxy
replicas. Queued requests resume when one upstream is ready. If no upstream
becomes ready and no local wake is still in progress, the gateway returns a 503.

Sleep is driven by the proxy gateway's in-memory activity timer. The control
plane does not scan request activity and does not enqueue sleep work. It accepts
validated `sleep`, `wake_started`, and `wake_failed` transitions through the
existing agent status endpoint.

Wake failures are bounded. Transient failures return a deployment to `sleeping`
so a later request can retry. After repeated failures, the deployment is parked
as `failed` with a visible failed stage until a redeploy creates fresh
deployments.

### Multiple Proxy Nodes

The platform supports multiple proxy nodes behind a shared edge hostname. A
stable external load balancer with active health checks is the ideal production
solution for proxy failure:

* Users point custom domains to one shared edge hostname.
* The load balancer sends new connections only to healthy proxy origins.
* Workload placement changes proxy routes without changing public DNS.
* All proxies share the same TLS certificates from the control plane.

Example:

```text theme={null}
Proxy US:   1.2.3.4
Proxy EU:   5.6.7.8
Proxy SYD:  9.10.11.12

External load balancer:
  example.com -> lb.techulus.cloud
  -> actively health-check proxy origins
  -> stop new traffic to an unhealthy proxy
```

Health-aware GeoDNS can provide proximity steering, but recovery remains subject
to resolver and client caching. Plain multiple A records are best-effort traffic
distribution, not reliable proxy failover.

### Proximity-Aware Load Balancing

Within a proxy node, traffic is distributed using weighted round-robin:

1. Local replicas on the same proxy server use weight `5`.
2. Remote replicas on other proxy servers use weight `1`.

That keeps the majority of traffic local whenever possible while still preserving cross-node routing.


# Compose Import
Source: https://docs.techulus.cloud/deployments/compose

Import existing Docker Compose files to create services.

You can import an existing Docker Compose file to create multiple services at once. The compose parser extracts service definitions and maps them to Techulus Cloud services.

## Supported Fields

The following Compose fields are parsed and applied:

| Field                     | Mapping                                                        |
| ------------------------- | -------------------------------------------------------------- |
| `image`                   | Container image                                                |
| `environment`             | Secrets (encrypted at rest)                                    |
| `volumes`                 | Named volumes with container paths                             |
| `ports`                   | Service ports (public/private, protocol)                       |
| `healthcheck`             | Health check command, interval, timeout, retries, start period |
| `deploy.replicas`         | Replica count                                                  |
| `deploy.resources.limits` | CPU and memory limits                                          |
| `command`                 | Start command override                                         |

When a Compose service does not define `deploy.resources.limits`, Techulus Cloud applies the default Large preset: 2 CPU cores and 1024 MB memory.

## How It Works

1. Paste or upload your `docker-compose.yml` in the project settings.
2. The parser validates the file and shows a preview of services to be created.
3. Warnings are shown for any unsupported options.
4. On confirmation, services are created with their configuration, secrets, volumes, and ports.

Each service in the compose file becomes a separate Techulus Cloud service within the same project and environment.

## Stateful Services

If a service in the compose file defines volumes, it is automatically marked as stateful. Stateful services are limited to 1 replica and pinned to a single server.

Imported stateful services use single-server local storage. Techulus Cloud does not currently provide replicated volumes or automatic failover for these services. Do not import production databases into Techulus Cloud local volumes; use an external managed or HA database instead.


# GitHub Integration
Source: https://docs.techulus.cloud/deployments/github

Automatic builds and deployments from GitHub repositories.

## Setup

Techulus Cloud integrates with GitHub through a [GitHub App](https://docs.github.com/en/apps). To enable it:

1. Create a GitHub App and configure it with your control plane URL.
2. Set the following environment variables on the control plane:

| Variable                 | Description                           |
| ------------------------ | ------------------------------------- |
| `GITHUB_APP_ID`          | Your GitHub App ID                    |
| `GITHUB_APP_PRIVATE_KEY` | Private key (base64-encoded)          |
| `GITHUB_WEBHOOK_SECRET`  | Webhook secret for verifying payloads |

3. Install the GitHub App on your GitHub account or organization.

The GitHub App needs these repository permissions:

* **Contents:** Read
* **Pull requests:** Read and write
* **Deployments:** Read and write

Subscribe the app to both the **Push** and **Pull request** webhook events.
If you increase permissions for an existing app, approve the new permissions on
each installation before enabling previews.

## Connecting a Repository

Once the GitHub App is installed, connect a repository to a service:

1. Set the service source type to `github`.
2. Select the repository from your installed GitHub accounts.
3. Choose the branch to deploy from (defaults to `main`).
4. Optionally set a root directory if your app isn't at the repository root.

## Auto-Deploy

When auto-deploy is enabled (the default), pushing to the configured branch triggers a build and deployment automatically.

The flow:

1. GitHub sends a push webhook to the control plane.
2. The control plane creates a build for the new commit.
3. An agent claims the build, clones the repository, and builds the image.
4. The agent pushes the final multi-platform image to the installation's user-owned Google Artifact Registry repository.
5. A rollout deploys the image.

GitHub deployment statuses are updated on the commit so you can track progress from pull requests.

## Pull Request Preview Deployments

Preview deployments are opt in from a GitHub-backed service's **Configuration**
page. Each eligible pull request gets one visible service in the project's
ordinary `previews` environment. An existing environment named `previews` is
reused and is left in place when previews close. If you configure an
**Automatic Subdomain Domain** and its wildcard DNS record, each preview also
gets a stable generated HTTPS URL beneath that domain. Without this setting,
the preview is still created without a public URL.

A pull request is eligible only when it:

* comes from the same repository as the base branch (forks are skipped),
* targets the service's configured deployment branch,
* is open and ready for review (drafts are skipped), and
* belongs to a stateless service.

The preview builds GitHub's synthetic merge result at
`refs/pull/<number>/merge`. This tests the change as it would merge into the
configured branch. If GitHub cannot produce that ref because of merge
conflicts, the preview service is removed rather than building the raw pull
request head. Reconciliation recreates it after GitHub can produce the merge
ref again.

When first created, preview services inherit the base service's current source
configuration, replicas, autoscaling, placement, health check, start command,
resource limits, ports, serverless settings, and complete secret set. They do
not copy volumes, backups, deployment schedules, cron jobs, production custom
domains, or public TCP/UDP routes. Serverless mode is disabled when no generated
public domain is available. Preview services use the normal service pages and
may be edited like other services. Their source repository remains tied to the
pull request, and volumes remain unavailable. Later pull request updates
preserve other edits.

New commits replace the preview revision without changing its URL. Converting
the pull request to a draft, closing it, or merging it removes the runtime and
route and marks the GitHub deployment inactive. A daily reconciliation job
rechecks previews in case a webhook was missed. Generated hosts use the normal
HTTP-01 certificate path, so installations with high pull-request volume should
monitor their certificate authority's issuance limits.

Each preview service maintains one status comment on the pull request. The
comment shows the current deployment status and preview URL. Status and commit
updates replace the same comment instead of adding new comments.

## Build Process

Agents build images using one of two methods:

| Method     | When Used                                                                              |
| ---------- | -------------------------------------------------------------------------------------- |
| Railpack   | No Dockerfile present — Railpack auto-detects the framework and generates a build plan |
| Dockerfile | A `Dockerfile` exists in the repository (or specified root directory)                  |

### Custom Dockerfile Path

To use a Dockerfile at a different path or with a different name, set a `TECHULUS_DOCKERFILE_PATH` environment variable on the service, e.g. `TECHULUS_DOCKERFILE_PATH=docker/prod.Dockerfile`.

* The path is relative to the build context (the repository root, or the service's root directory if set).
* When set, the Dockerfile build method is always used — if the file doesn't exist, the build fails instead of falling back to Railpack.

Images are built with BuildKit, assigned a unique `revision-<revision-id>` tag, and pushed to [Google Artifact Registry](/infrastructure/registry). The same repository-scoped GAR Writer credential is used for BuildKit pushes and Podman pulls. Repository cleanup policies alone control retention: keep the newest 10 versions per package and delete other versions older than 30 days, regardless of deployment status. See the registry guide for retention and rollback limitations.

Build statuses:

| Status      | Description                                   |
| ----------- | --------------------------------------------- |
| `pending`   | Waiting for an available agent                |
| `claimed`   | Agent has taken the build                     |
| `cloning`   | Cloning the repository                        |
| `building`  | Building the container image                  |
| `pushing`   | Pushing the image to Google Artifact Registry |
| `completed` | Build succeeded                               |
| `failed`    | Build failed                                  |
| `cancelled` | Build was cancelled                           |

Build logs stream to [Victoria Logs](/infrastructure/logging) in real time and are viewable from the web UI.


# Scheduled Deployments
Source: https://docs.techulus.cloud/deployments/scheduled

Cron-based automatic redeployments.

You can configure a service to redeploy automatically on a schedule. This is useful for workloads that need periodic restarts or for pulling the latest version of an image tag.

## Configuration

Set a deployment schedule using a cron expression in the service settings.

Examples:

| Schedule              | Cron Expression |
| --------------------- | --------------- |
| Every hour            | `0 * * * *`     |
| Daily at midnight UTC | `0 0 * * *`     |
| Every 6 hours         | `0 */6 * * *`   |
| Weekdays at 9 AM UTC  | `0 9 * * 1-5`   |

The platform tracks the last scheduled deployment time to prevent duplicate runs.

## How It Works

A background job checks for services with deployment schedules and triggers a new rollout when the schedule is due. The rollout follows the same process as a manual deployment — pulling the image, starting new containers, running health checks, and stopping old containers.


# Introduction
Source: https://docs.techulus.cloud/index

Techulus Cloud is an open-source, self-hosted container deployment platform with private networking.

Techulus Cloud is an open-source container deployment platform that you host on your own infrastructure. It turns any set of machines — bare metal servers, cloud VMs, or Raspberry Pis — into a private container platform with encrypted networking, automatic HTTPS, and zero-downtime deployments.

## How It Works

Every machine in your cluster runs a lightweight **agent** that pulls its expected state from a central **control plane** and reconciles containers, networking, and routing automatically. There are no master nodes or schedulers to manage — each server operates independently.

Machines are assigned one of two roles:

* **Proxy nodes** handle public traffic, terminate TLS, and route requests to containers.
* **Worker nodes** run containers with no public exposure.

All communication between nodes happens over an encrypted **WireGuard mesh network**. Services discover each other via `.internal` DNS — no hardcoded IPs or service mesh required.

## Key Features

* **Private by default** — services communicate over WireGuard. Nothing is exposed publicly unless you configure it.
* **Build from source** — push code and build with Railpack or your own Dockerfile. Or deploy pre-built images.
* **GitHub auto-deploy** — connect a repo and deploy on every push.
* **Automatic HTTPS** — TLS certificates are provisioned and renewed automatically via Let's Encrypt.
* **Persistent volumes** — attach named local volumes for stateful workloads with scheduled backups to S3-compatible storage. Replicated storage and HA failover are not yet supported.
* **Service discovery** — containers resolve each other by name using `.internal` domains.
* **Multi-environment** — run production, staging, and dev within the same project.
* **External edge compatibility** — use a stable load balancer with active health
  checks, or health-aware GeoDNS when DNS caching tradeoffs are acceptable.
* **TCP/UDP proxy** — expose non-HTTP services like game servers or custom protocols.

## Next Steps

<Card title="Architecture" icon="map" href="/architecture">
  Learn how the control plane, agents, networking, and rollout lifecycle fit together.
</Card>

<Card title="Agent" icon="server" href="/agent">
  Understand how the agent reconciles state, builds images, and manages containers.
</Card>


# Email Alerts
Source: https://docs.techulus.cloud/infrastructure/alerts

Get notified about server issues, build failures, and deployment problems.

Techulus Cloud can send email alerts when things go wrong. Alerts are delivered via SMTP to a configurable list of recipients.

## Alert Types

| Alert              | Trigger                                       |
| ------------------ | --------------------------------------------- |
| Server offline     | A server stops reporting to the control plane |
| Build failure      | A container image build fails                 |
| Deployment failure | A deployment fails during rollout             |
| Deployment moved   | A deployment is moved to a different server   |

## Configuration

Add the following environment variables to the control plane:

| Variable            | Description                                           |
| ------------------- | ----------------------------------------------------- |
| `SMTP_ENABLED`      | Set to `true` to enable alerts                        |
| `SMTP_HOST`         | SMTP server hostname                                  |
| `SMTP_PORT`         | SMTP port (e.g., `587`)                               |
| `SMTP_USERNAME`     | SMTP authentication username                          |
| `SMTP_PASSWORD`     | SMTP authentication password                          |
| `SMTP_ENCRYPTION`   | `starttls` or `tls`                                   |
| `SMTP_FROM_NAME`    | Sender display name (e.g., `Techulus Cloud`)          |
| `SMTP_FROM_ADDRESS` | Sender email address                                  |
| `SMTP_TIMEOUT`      | Connection timeout in milliseconds (default: `10000`) |
| `SMTP_ALERT_EMAILS` | Comma-separated list of recipient email addresses     |

Works with any SMTP provider — Amazon SES, Resend, Mailgun, Postmark, or your own mail server.


# Backups
Source: https://docs.techulus.cloud/infrastructure/backups

Automated volume backups to S3-compatible storage.

Techulus Cloud can automatically back up service volumes to S3-compatible storage. Backups are triggered on a schedule or manually from the web UI.

> **Backups are not high availability:** Backups provide point-in-time disaster recovery only. If a server hosting a stateful service is lost, any writes after the last successful backup may be lost, and recovery requires restoring data before the service can run elsewhere. Backups do not provide replicated storage or automatic failover.

## How It Works

The agent stops the running container, archives the volume directory as a `.tar.gz` file, uploads the archive to S3-compatible storage, then starts the container again if it was running.

Backups are volume snapshots. Techulus Cloud does not run database-native dump tools, and running production databases on local single-node volumes is not recommended.

## Configuration

Configure S3-compatible storage in the control plane environment:

| Variable                        | Description                                                |
| ------------------------------- | ---------------------------------------------------------- |
| `BACKUP_STORAGE_PROVIDER`       | Storage provider (e.g., `s3`)                              |
| `BACKUP_STORAGE_BUCKET`         | Bucket name                                                |
| `BACKUP_STORAGE_REGION`         | AWS region                                                 |
| `BACKUP_STORAGE_ENDPOINT`       | Custom endpoint for MinIO or other S3-compatible providers |
| `BACKUP_STORAGE_ACCESS_KEY`     | Access key                                                 |
| `BACKUP_STORAGE_SECRET_KEY`     | Secret key                                                 |
| `BACKUP_STORAGE_RETENTION_DAYS` | Number of days to retain backups (default: `7`)            |

Works with AWS S3, MinIO, DigitalOcean Spaces, and any S3-compatible provider.


# Logging
Source: https://docs.techulus.cloud/infrastructure/logging

Centralized log aggregation with Victoria Logs.

Techulus Cloud uses [Victoria Logs](https://docs.victoriametrics.com/victorialogs/) for centralized log aggregation. Container output, HTTP access logs, service cron history, build logs, and agent system logs are collected and searchable from the web UI.

## Log Types

| Type        | Source                            | Fields                                                                       |
| ----------- | --------------------------------- | ---------------------------------------------------------------------------- |
| `container` | Container stdout/stderr           | `deployment_id`, `service_id`, `server_id`, `stream`                         |
| `http`      | Traefik access logs (proxy nodes) | `service_id`, `host`, `method`, `path`, `status`, `duration_ms`, `client_ip` |
| `cron`      | Scheduled service HTTP requests   | `service_id`, `cron_id`, `path`, `result`, `status`, `duration_ms`           |
| `build`     | BuildKit image builds             | `build_id`, `service_id`, `project_id`                                       |
| `agent`     | Agent process output              | `server_id`, `level`                                                         |

## How It Works

Each agent ships logs directly to Victoria Logs over HTTP using the JSON Lines format.

* **Container logs** are streamed from running containers, batched in groups of 1000, and flushed every 5 seconds. Log positions are tracked per container to prevent duplicates.
* **HTTP logs** are tailed from Traefik's access log file on proxy nodes, batched in groups of 500.
* **Cron logs** are written by the control plane after each scheduled request with its result, HTTP status, and duration. They never include `CRON_SECRET`, the base URL, or response bodies.
* **Build logs** are captured during image builds and streamed in real time.
* **Agent logs** intercept the agent's own stdout/stderr with automatic log level detection.

Agent-originated log batches retry up to 3 times with exponential backoff on failure. Cron history writes are best-effort and do not retry the cron request when log ingestion fails.

## Configuration

Victoria Logs runs as a Docker container alongside the control plane.

| Variable       | Description                          |
| -------------- | ------------------------------------ |
| `VL_USERNAME`  | Authentication username              |
| `VL_PASSWORD`  | Authentication password              |
| `VL_RETENTION` | Log retention period (default: `7d`) |

The control plane exposes logs at `https://logs.<ROOT_DOMAIN>` with basic auth. Agents write to the internal endpoint at `http://victoria-logs:9428`.

## Accessing Logs

Logs are accessible from the web UI for each service, deployment, build, and server. The control plane queries Victoria Logs using LogSQL with filters for `service_id`, `deployment_id`, `server_id`, and time ranges.

Searches run against Victoria Logs rather than only the entries currently loaded in the browser. Continuous service, request, and server log views default to the last 24 hours and support 1-hour, 6-hour, 24-hour, and 7-day ranges. These query ranges do not change the separate `VL_RETENTION` storage setting.

Cron history follows `VL_RETENTION`. The latest cron result remains visible in the read-only **Crons** section of [service configuration](/services/configuration#service-crons).


# Google Artifact Registry
Source: https://docs.techulus.cloud/infrastructure/registry

Provision and operate the required user-owned image registry.

Techulus Cloud requires a user-owned [Google Artifact Registry](https://cloud.google.com/artifact-registry/docs) (GAR) Docker repository for images built from GitHub sources. Techulus does not provision or own the repository. Agents use one repository-scoped Writer credential to push and pull images. User-supplied images can still use credentials configured under **Settings → Registries**.

## Image layout

Each source-backed service has one GAR package:

```text theme={null}
<location>-docker.pkg.dev/<gcp-project>/<repository>/<techulus-project-id>/<service-id>:revision-<revision-id>
```

## Provision GAR

Create the repository and Writer service account in [Google Cloud Console](https://console.cloud.google.com/). Use a billing-enabled project and enable the Artifact Registry and IAM APIs if prompted.

The operator needs permission to enable APIs, administer Artifact Registry repositories and their IAM policies, create service accounts, and create service-account keys. The predefined roles that supply these permissions are Service Usage Admin, Artifact Registry Administrator, Service Account Admin, and Service Account Key Admin, or custom roles with equivalent permissions.

### Create the repository

Open **Artifact Registry → Repositories → Create repository**:

| Setting             | Value                                              |
| ------------------- | -------------------------------------------------- |
| Name                | `techulus-images`                                  |
| Format              | **Docker**                                         |
| Mode                | **Standard**, not remote or virtual                |
| Location            | A region close to your deployment agents           |
| Immutable tags      | **Disabled** so cleanup can delete tagged versions |
| Encryption          | Google-managed key unless you require CMEK         |
| Cleanup policy mode | **Dry run**, not **Delete artifacts** yet          |

Keep the repository private. Configure the [cleanup policy](#cleanup-policy) before activating deletion.

Alternatively, use Google Cloud Shell or an authenticated `gcloud` CLI:

```bash theme={null}
export GAR_PROJECT_ID=google-project-id
export GAR_LOCATION=us-central1
export GAR_REPOSITORY_ID=techulus-images

gcloud services enable artifactregistry.googleapis.com iam.googleapis.com \
  --project="$GAR_PROJECT_ID"
gcloud artifacts repositories create "$GAR_REPOSITORY_ID" \
  --project="$GAR_PROJECT_ID" \
  --location="$GAR_LOCATION" \
  --repository-format=docker \
  --description="Techulus Cloud source images"
```

### Create the Writer account

Open **IAM & Admin → Service Accounts** and create `techulus-agent` without granting a project-wide role. In the repository's **Permissions** panel, grant it Artifact Registry Writer (`roles/artifactregistry.writer`) on that repository only. The account pushes and pulls images; Techulus does not need repository administration access.

For the equivalent gcloud setup:

```bash theme={null}
export GAR_AGENT_SA="techulus-agent@${GAR_PROJECT_ID}.iam.gserviceaccount.com"
gcloud iam service-accounts create techulus-agent \
  --project="$GAR_PROJECT_ID" \
  --display-name="Techulus agents"
gcloud artifacts repositories add-iam-policy-binding "$GAR_REPOSITORY_ID" \
  --project="$GAR_PROJECT_ID" \
  --location="$GAR_LOCATION" \
  --member="serviceAccount:${GAR_AGENT_SA}" \
  --role="roles/artifactregistry.writer"
```

### Download and encode the JSON key

For the Writer account, open **Keys → Add key → Create new key → JSON** and download the key to a trusted workstation as `agent-key.json`. Service-account key creation can be prohibited by the `iam.disableServiceAccountKeyCreation` organization policy. If blocked, ask your organization administrator to approve an exemption. Google does not allow an existing private key to be downloaded again.

With gcloud, create the key using:

```bash theme={null}
umask 077
gcloud iam service-accounts keys create agent-key.json \
  --iam-account="$GAR_AGENT_SA" --project="$GAR_PROJECT_ID"
```

Encode the **complete JSON document** as one line. These commands work on macOS and Linux; use a new output filename if `gar-credentials.env` already exists:

```bash theme={null}
umask 077
printf 'GAR_AGENT_KEY_BASE64=%s\n' \
  "$(base64 < agent-key.json | tr -d '\r\n')" > gar-credentials.env
chmod 600 gar-credentials.env
```

`umask 077` makes newly created files private to your user; `chmod 600` explicitly restricts the output to owner read/write access. Base64 is not encryption. Treat both files as private keys: never commit them or paste them into chat.

### Configure Techulus

Securely copy the encoded value into the deployment `.env` or installer and add the repository path:

```env theme={null}
GAR_REPOSITORY=us-central1-docker.pkg.dev/google-project-id/techulus-images
GAR_AGENT_KEY_BASE64=<base64 Writer service-account JSON>
```

`GAR_REPOSITORY` must not include a scheme, image path, tag, or digest. The health endpoint returns 503 when a mandatory GAR value is missing or malformed. After securely transferring the credential, delete temporary files:

```bash theme={null}
rm agent-key.json gar-credentials.env
```

The control plane includes this credential in each agent's encrypted registry bundle. BuildKit uses it to push images and Podman uses it to pull deployments. TLS verification is always enabled. Static repository-scoped keys are intentional because builds can outlast GAR OAuth access tokens.

## Cleanup policy

The tracked [`deployment/gar-cleanup-policy.json`](https://github.com/techulus/cloud/blob/main/deployment/gar-cleanup-policy.json) defines exactly two repository-wide rules:

* keep the 10 most recent versions of every package, with no package-prefix filter;
* delete versions in any tag state once they are older than 30 days.

Keep policies take precedence over delete policies. A version outside the newest 10 remains available until it is older than 30 days. GAR evaluates age from version creation time.

<Warning>
  This policy is intentionally independent of deployments. An active or rollback image can disappear when it falls outside a package's newest 10 versions and becomes older than 30 days. Failed and undeployed builds count toward version history. Multi-platform images create multiple registry versions, so 10 versions does not mean 10 builds or rollouts. The newest versions of deleted services remain, and a seven-day service/data restore does not guarantee that its image is still available.
</Warning>

In the repository's cleanup settings, use **Dry run** and configure:

| Name                                | Action                    | Condition                             |
| ----------------------------------- | ------------------------- | ------------------------------------- |
| `keep-most-recent-ten-versions`     | Keep most recent versions | Keep count `10`, no package prefixes  |
| `delete-versions-after-thirty-days` | Delete                    | Tag state `Any`, older than `30 days` |

The operator applying policies needs Artifact Registry Administrator permissions. From a Techulus checkout, the equivalent gcloud command is:

```bash theme={null}
gcloud artifacts repositories set-cleanup-policies "$GAR_REPOSITORY_ID" \
  --project="$GAR_PROJECT_ID" \
  --location="$GAR_LOCATION" \
  --policy=deployment/gar-cleanup-policy.json \
  --dry-run
```

GAR evaluates cleanup asynchronously, generally within one day. Enable Artifact Registry Data Write audit logs, wait for a dry-run cycle, and inspect proposed `BatchDeleteVersions` operations. When correct, switch to **Delete artifacts** in Console or run the same command with `--no-dry-run`.

### Updating an existing repository

After deploying the code that no longer manages GAR tags, replace the old protected-tag Keep rule with the tracked policy; Techulus does not update cloud policies automatically. Then remove `GAR_ADMIN_KEY_BASE64` from the deployment environment and revoke/delete the unused Repository Administrator service-account key. Remove the account or its repository role if it has no other purpose. There is no compatibility or migration automation.

## Upgrading from the bundled registry

This is a clean beta cutover. Techulus does not copy old images or rewrite persisted image references. Configure GAR and the Writer key before upgrading. Existing source-built revisions then point to an unavailable old registry and every source-backed service must be rebuilt before it can deploy from GAR.

The upgrade removes the old registry container but leaves its Docker volume. After verifying the upgrade and deciding rollback data is no longer needed, inspect and explicitly remove the orphaned volume:

```bash theme={null}
docker volume ls --filter name=registry-data
docker volume rm <exact-volume-name>
```

Volume deletion is irreversible and is never performed automatically.

## Global custom registries

Administrators can add credentials for user-supplied private images under **Settings → Registries**. A host, including an explicit port when needed, is globally unique and cannot include a scheme or path. The GAR system host is reserved.

Changes are queued for every registered agent, including offline agents. Entries cannot be edited; delete and recreate one to change its username, password, or TLS setting. Use pull-only robot or service-account credentials where possible. Passwords are write-only in the UI, and an actual agent pull remains the authoritative availability check.


# Installation
Source: https://docs.techulus.cloud/installation

Deploy the Techulus Cloud control plane on your own infrastructure.

## Prerequisites

* Docker and Docker Compose
* A domain name with DNS configured
* Ports 80 and 443 available
* A user-owned Google Artifact Registry Docker repository with a repository-scoped Writer account and cleanup policy. Complete the [GAR setup](/infrastructure/registry) before installing.

You need two DNS records pointing to your server:

| Record                 | Purpose                         |
| ---------------------- | ------------------------------- |
| `your-domain.com`      | Control plane web UI            |
| `logs.your-domain.com` | Log aggregation (Victoria Logs) |

## Quick Start

Run the automated install script on a fresh server:

```bash theme={null}
curl -fsSL https://raw.githubusercontent.com/techulus/cloud/main/deployment/install.sh | bash
```

The script detects your OS, installs Docker, walks you through DNS and environment configuration, and starts all services.

By default, the installer deploys the latest GitHub release and verifies its
Compose files against the release manifest. Set `TECHULUS_CLOUD_VERSION` before
running the script only when you need a specific release.

## Manual Setup

Clone the repository and configure your environment:

```bash theme={null}
cd deployment
cp .env.example .env
openssl rand -hex 32
```

Edit `.env` with your values. Set `CONTROL_PLANE_UPDATER_TOKEN` to
the generated token and configure `GAR_REPOSITORY` and `GAR_AGENT_KEY_BASE64`
from the [GAR setup](/infrastructure/registry).

For an external PostgreSQL database, keep:

```env theme={null}
COMPOSE_FILE=compose.production.yml
```

Set `TECHULUS_CLOUD_VERSION` to the release tag you are installing:

```env theme={null}
TECHULUS_CLOUD_VERSION=vX.Y.Z
```

Then start the stack:

```bash theme={null}
docker compose -f compose.production.yml up -d --pull always --remove-orphans
```

To use the bundled PostgreSQL instead of an external database:

```bash theme={null}
sed -i.bak 's/^COMPOSE_FILE=.*/COMPOSE_FILE=compose.postgres.yml/' .env
docker compose -f compose.postgres.yml up -d --pull always --remove-orphans
```

Production hosts should also cap Docker container logs. The installer creates
`/etc/docker/daemon.json` with `json-file` rotation on fresh hosts. If Docker is
already configured, keep your existing daemon settings and add equivalent log
rotation manually.

## Manual upgrades

Use the one-click upgrade flow in **Settings** when possible. If the updater
cannot run, update the deployment from the host manually.

<Warning>
  The GAR release is a clean beta cutover. Images and persisted references from the old bundled
  registry are not migrated. Provision GAR, add both GAR variables to `.env`, and apply its
  cleanup policy before upgrading. After the upgrade, rebuild every source-backed service.
</Warning>

Back up your `.env` file and database before upgrading. Then fetch the Compose
files for the release, update `TECHULUS_CLOUD_VERSION`, pull images, and restart
the stack:

```bash theme={null}
cd /opt/techulus-cloud
TARGET_VERSION=vX.Y.Z
COMPOSE_FILE=$(grep -E '^COMPOSE_FILE=' .env | cut -d= -f2- || true)
if [ -z "$COMPOSE_FILE" ]; then
  COMPOSE_FILE=compose.production.yml
fi

cp .env ".env.backup.$(date +%Y%m%d%H%M%S)"
curl -fsSL "https://raw.githubusercontent.com/techulus/cloud/${TARGET_VERSION}/deployment/compose.production.yml" -o compose.production.yml
curl -fsSL "https://raw.githubusercontent.com/techulus/cloud/${TARGET_VERSION}/deployment/compose.postgres.yml" -o compose.postgres.yml

if grep -q '^TECHULUS_CLOUD_VERSION=' .env; then
  sed -i.bak "s/^TECHULUS_CLOUD_VERSION=.*/TECHULUS_CLOUD_VERSION=${TARGET_VERSION}/" .env
else
  echo "TECHULUS_CLOUD_VERSION=${TARGET_VERSION}" >> .env
fi

docker compose -f "$COMPOSE_FILE" pull
docker compose -f "$COMPOSE_FILE" up -d --remove-orphans
```

Rollback after migrations may require restoring the database backup before you
restart older images.

The Compose files include container health checks for visibility. Plain Docker
Compose reports unhealthy containers but does not restart them automatically, so
use the common commands below when investigating a self-hosted service.

## Environment Variables

### Required

| Variable               | Description                                                                        |
| ---------------------- | ---------------------------------------------------------------------------------- |
| `ROOT_DOMAIN`          | Your domain (e.g., `cloud.example.com`)                                            |
| `ACME_EMAIL`           | Email for Let's Encrypt certificates                                               |
| `DATABASE_URL`         | PostgreSQL connection string (e.g., `postgres://user:pass@postgres:5432/techulus`) |
| `BETTER_AUTH_SECRET`   | Secret key for authentication                                                      |
| `ENCRYPTION_KEY`       | 32 bytes as a 64-character hex string. Required unless you configure AWS KMS BYOK. |
| `GAR_REPOSITORY`       | GAR image base: `<location>-docker.pkg.dev/<gcp-project>/<repository>`             |
| `GAR_AGENT_KEY_BASE64` | Base64 JSON key for the repository-scoped Artifact Registry Writer sent to agents  |

### AWS KMS BYOK

AWS KMS BYOK is available for AWS-hosted dedicated control planes. The control plane can run in your AWS account or in the Techulus AWS account. Attach an EC2 instance profile or ECS task role to the web service. Do not store static AWS credentials in `.env`.

Set these variables instead of `ENCRYPTION_KEY` for a new installation:

```env theme={null}
ENCRYPTION_KMS_KEY_ARN=arn:aws:kms:us-east-1:123456789012:key/your-key-id
AWS_REGION=us-east-1
```

The role needs `kms:GenerateDataKey`, `kms:Encrypt`, `kms:Decrypt`, and `kms:DescribeKey` for the configured key. Restrict the cryptographic operations to the role, the exact key, and this encryption context:

```json theme={null}
{
  "StringEquals": {
    "kms:EncryptionContext:techulus:purpose": "service-secret-dek"
  }
}
```

#### Control plane in the Techulus AWS account

You can keep the KMS key and compute agents in your AWS account while running the dedicated control plane in the Techulus AWS account. Grant the Techulus control-plane role direct cross-account access to the KMS key. The application uses that role through the standard AWS credential chain. It does not call AWS STS `AssumeRole`.

Merge statements like these into the KMS key policy in your AWS account. Replace the account IDs and role name. `DescribeKey` is separate because that operation does not accept an encryption context:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowTechulusControlPlaneDescribe",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/techulus-control-plane-customer-name"
      },
      "Action": "kms:DescribeKey",
      "Resource": "*"
    },
    {
      "Sid": "AllowTechulusControlPlaneCrypto",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/techulus-control-plane-customer-name"
      },
      "Action": ["kms:GenerateDataKey", "kms:Encrypt", "kms:Decrypt"],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "kms:EncryptionContext:techulus:purpose": "service-secret-dek"
        }
      }
    }
  ]
}
```

The Techulus control-plane role also needs this identity policy. Set `Resource` to the full ARN of your KMS key:

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DescribeCustomerEncryptionKey",
      "Effect": "Allow",
      "Action": "kms:DescribeKey",
      "Resource": "arn:aws:kms:us-east-1:444455556666:key/your-key-id"
    },
    {
      "Sid": "UseCustomerEncryptionKey",
      "Effect": "Allow",
      "Action": ["kms:GenerateDataKey", "kms:Encrypt", "kms:Decrypt"],
      "Resource": "arn:aws:kms:us-east-1:444455556666:key/your-key-id",
      "Condition": {
        "StringEquals": {
          "kms:EncryptionContext:techulus:purpose": "service-secret-dek"
        }
      }
    }
  ]
}
```

Configure `ENCRYPTION_KMS_KEY_ARN` with the customer-account key ARN and set `AWS_REGION` to that key's region. Your compute agents do not need AWS credentials or KMS permissions. They continue receiving the data encryption key during registration.

You can revoke future cold-start unwraps by removing the Techulus role from the key policy or disabling the key. This does not erase a key already cached by a running control-plane process or persisted by an existing agent.

The first secret operation generates a 32-byte data encryption key. The control plane stores only its KMS-wrapped form in PostgreSQL. Each web process unwraps and caches the key when first needed. The secret ciphertext format and agent behavior remain unchanged.

To migrate an existing installation:

1. Back up PostgreSQL and `.env`.
2. Add `ENCRYPTION_KMS_KEY_ARN` and `AWS_REGION`. Keep the existing `ENCRYPTION_KEY` temporarily.
3. Restart the web service and verify that you can reveal an existing secret and register an agent.
4. Remove `ENCRYPTION_KEY` and restart all web replicas.

When a wrapped key already exists, any remaining `ENCRYPTION_KEY` must contain the same key bytes. The control plane fails closed if they differ. Do not add a newly generated `ENCRYPTION_KEY` to an existing KMS installation.

Do not change `ENCRYPTION_KMS_KEY_ARN` after initialization. Automatic rotation of the same KMS key is transparent. Moving to another KMS key requires a reviewed rewrap procedure.

If KMS is unavailable, cold-started web processes return a service-unavailable response for secret operations and agent registration. Other control-plane features remain available. Warm processes and existing agents can continue using keys already held in memory or local agent configuration.

Back up both PostgreSQL and continued access to the KMS key. Deleting the KMS key can make a database restore unrecoverable. Disabling the key blocks future unwraps, but it does not erase keys already cached by running processes or registered agents.

### Control plane deployment

| Variable                      | Description                                                                                                                               |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `COMPOSE_FILE`                | Compose file used by self-updates. Use `compose.production.yml` for external PostgreSQL or `compose.postgres.yml` for bundled PostgreSQL. |
| `TECHULUS_CLOUD_VERSION`      | Installed release tag. The installer defaults to the latest GitHub release.                                                               |
| `CONTROL_PLANE_UPDATER_TOKEN` | Random token used by the web app to call the internal updater service. Generate with `openssl rand -hex 32`.                              |

### Victoria Logs

| Variable       | Description                          |
| -------------- | ------------------------------------ |
| `VL_USERNAME`  | Logs service username                |
| `VL_PASSWORD`  | Logs service password                |
| `VL_RETENTION` | Log retention period (default: `7d`) |

### Victoria Metrics

| Variable       | Description                               |
| -------------- | ----------------------------------------- |
| `VM_USERNAME`  | Metrics service username                  |
| `VM_PASSWORD`  | Metrics service password                  |
| `VM_RETENTION` | Metrics retention period (default: `30d`) |

### Google Artifact Registry

| Variable               | Description                                                               |
| ---------------------- | ------------------------------------------------------------------------- |
| `GAR_REPOSITORY`       | Required GAR repository image base, with no scheme or trailing image path |
| `GAR_AGENT_KEY_BASE64` | Required base64 Writer service-account JSON key                           |

The Writer account must be scoped to the configured repository. See
[Google Artifact Registry](/infrastructure/registry)
for provisioning, authentication, cleanup policy, and upgrade instructions.

The policy keeps the newest 10 versions per package and deletes versions in any
tag state after 30 days. An active or rollback image can disappear. Failed
builds count, and multi-platform images mean 10 versions is not 10 builds or
rollouts. Deleted services retain their newest versions. A seven-day data
restore does not guarantee that its image remains available.

After deploying Writer-only code, existing operators must replace the old
protected-tag Keep rule with the tracked policy. Then remove
`GAR_ADMIN_KEY_BASE64` from `.env` and revoke its unused admin key. Techulus
does not automate this migration.

### Inngest

| Variable              | Description                                            |
| --------------------- | ------------------------------------------------------ |
| `INNGEST_SIGNING_KEY` | Request verification key (prefix with `signkey-prod-`) |
| `INNGEST_EVENT_KEY`   | Event API key                                          |

### Control Plane Replicas

| Variable       | Description                                                  |
| -------------- | ------------------------------------------------------------ |
| `WEB_REPLICAS` | Number of control plane web containers to run (default: `1`) |

When `WEB_REPLICAS` is greater than `1`, Traefik discovers the replicated web
containers through Docker and load balances requests for `<ROOT_DOMAIN>` across
them. Schema sync runs once from the dedicated `migrate` service before the
replicated web containers start, so scaling `WEB_REPLICAS` does not run
migrations from every replica.

### GitHub Integration (Optional)

| Variable                 | Description                             |
| ------------------------ | --------------------------------------- |
| `GITHUB_APP_ID`          | GitHub App ID                           |
| `GITHUB_APP_PRIVATE_KEY` | GitHub App private key (base64-encoded) |
| `GITHUB_WEBHOOK_SECRET`  | Webhook secret                          |

Configure the app with Contents read, Pull requests read and write, and
Deployments read and write repository permissions. Subscribe it to Push and
Pull request events. Pull request previews use the Automatic Subdomain Domain
setting and its wildcard DNS record when configured. Without it, previews are
created without public URLs.

## Generating Secrets

```bash theme={null}
# Encryption key (64 hex characters)
openssl rand -hex 32

# Auth secret
openssl rand -hex 32

# Inngest signing key
echo "signkey-prod-$(openssl rand -hex 32)"

# Inngest event key
openssl rand -hex 16
```

## Services

Once running, the following services are available:

| Service    | Endpoint                     |
| ---------- | ---------------------------- |
| Web        | `https://<ROOT_DOMAIN>`      |
| Logs       | `https://logs.<ROOT_DOMAIN>` |
| PostgreSQL | Internal only                |
| Inngest    | Internal only                |

Traefik handles TLS termination and automatic certificate renewal via Let's Encrypt.

## Database Migrations

The schema is synced automatically by the one-shot `migrate` service via `drizzle-kit push`. Non-destructive changes (adding tables, columns, indexes) are applied automatically. Destructive changes like dropping columns require manual intervention. If schema sync fails, `web` startup is blocked; inspect the failure with `docker compose logs migrate`.

## Common Commands

```bash theme={null}
# Check service status
docker compose -f compose.production.yml ps

# View logs
docker compose -f compose.production.yml logs -f

# Stop all services
docker compose -f compose.production.yml down --remove-orphans

# Update to the configured image references
docker compose -f compose.production.yml up -d --pull always --remove-orphans
```

Use versioned or digest-pinned image references for production updates.


# Service Discovery
Source: https://docs.techulus.cloud/networking/service-discovery

Internal DNS for service-to-service communication.

Services discover each other using `.internal` domain names. Every agent runs a built-in DNS server that resolves these names to container IP addresses over the [WireGuard mesh](/architecture#wireguard-mesh).

## How It Works

Each service gets a hostname like `my-service.internal`. The DNS server on every agent is configured with all service records pushed from the control plane.

When a container queries `my-service.internal`, the local DNS server resolves it to the container IPs of that service. If the service has multiple replicas, responses use round-robin across all healthy containers.

All DNS resolution happens over the private WireGuard network — no traffic leaves the mesh.

## Configuration

Service discovery works automatically. The DNS server:

* Listens on the container gateway IP (e.g., `10.200.1.1`).
* Configures `systemd-resolved` to forward `.internal` queries.
* Receives record updates from the control plane as part of expected state.

No manual configuration is needed. Services can reference each other by name immediately after deployment.

## Example

If you have an `api` service and a `web` service, the web service can connect to the API using:

```
http://api.internal:3000
```

For production databases, use an external managed database or another HA database setup. Techulus Cloud stateful volumes are single-server local storage and do not currently provide replicated storage or automatic failover.


# TCP/UDP Proxy
Source: https://docs.techulus.cloud/networking/tcp-udp-proxy

Expose non-HTTP services like game servers and custom protocols.

Not every service speaks HTTP. Techulus Cloud supports exposing raw TCP and UDP ports through proxy nodes for services like game servers or custom protocols.

> **Database exposure warning:** Do not expose database ports publicly through the TCP/UDP proxy. Public database access increases security risk, and Techulus Cloud does not provide HA storage or automatic failover for production databases.

## Configuration

When adding a port to a service, set the protocol to `tcp` or `udp`:

| Field         | Description                                       |
| ------------- | ------------------------------------------------- |
| Port          | The container port your service listens on        |
| Protocol      | `tcp` or `udp`                                    |
| External port | The port exposed on the proxy node                |
| Public        | Whether this port is accessible from the internet |

Traffic is routed from the proxy node's external port through the WireGuard mesh to the container.

## TLS Passthrough

For TCP services that handle their own TLS, enable **TLS passthrough**. This forwards the encrypted connection directly to the container without Traefik terminating TLS.

## Firewall

The agent setup script configures firewall rules to allow traffic on the TCP/UDP proxy port ranges. If you set up servers manually, ensure the relevant ports are open.


# Configuration
Source: https://docs.techulus.cloud/services/configuration

Environment variables, secrets, resource limits, and start commands.

## Source Type

Each service is deployed from one of two sources:

| Source   | Description                        |
| -------- | ---------------------------------- |
| `image`  | Deploy a pre-built container image |
| `github` | Build from a GitHub repository     |

For GitHub sources, you can specify the branch, root directory, and whether to auto-deploy on push. See [GitHub Integration](/deployments/github) for details.

## Environment Variables & Secrets

Secrets are encrypted at rest and injected into containers at runtime. They are never baked into images.

Each secret has a key and an encrypted value, scoped to a single service. You can add, update, and remove secrets from the service settings in the web UI.

Secrets are passed as environment variables to the container when it starts.

## Service crons

Service crons send scheduled HTTP `GET` requests to your application. Define them under `service.crons` in `techulus.yml`:

```yaml theme={null}
service:
  crons:
    - path: /api/cron/digest
      schedule: "0 5 * * *"
    - path: /api/cron/cleanup
      schedule: "0 * * * *"
```

Each path must be unique, start with `/`, and omit a query string or fragment. Schedules use five-field cron expressions in UTC. You can configure up to 100 crons per service.

Add these service secrets from the web UI:

| Secret          | Required | Description                                                         |
| --------------- | -------- | ------------------------------------------------------------------- |
| `CRON_BASE_URL` | Yes      | HTTP or HTTPS origin joined with each configured path               |
| `CRON_SECRET`   | No       | Sent as `Authorization: Bearer <value>`; requires an HTTPS base URL |

The destination must be reachable from the control plane. Cron requests do not follow redirects or retry, and each request has a 10-second deadline. If the control plane misses multiple intervals, it sends only the latest due occurrence instead of backfilling every missed run.

Missing or invalid configuration produces a skipped run. Redirects and non-2xx responses produce failed runs. The **Crons** section in service configuration shows the latest result. Full history is available in [service logs](/infrastructure/logging) and follows the configured log retention period.

`tc apply` treats `service.crons` as the complete desired list. Removing the field or setting it to an empty list removes all cron definitions. Cron-only changes take effect without creating a deployment revision.

## Start Command

Override the container's default entrypoint by setting a custom start command. This is useful when deploying from pre-built images that need different startup behavior.

## Resource Limits

You can set CPU and memory limits per service:

| Setting      | Description                                       |
| ------------ | ------------------------------------------------- |
| CPU limit    | Maximum CPU cores (e.g., `0.5`, `1`, `2`)         |
| Memory limit | Maximum memory in MB (e.g., `256`, `512`, `1024`) |

New services default to the Large preset: 2 CPU cores and 1024 MB memory. You can change the preset or choose No limit to let the container use whatever resources are available on the host.

Resource limits are runtime caps, not reserved capacity. Placement uses current server health and replica distribution, but it does not require the sum of configured limits to fit as guaranteed minimum allocation.

## Health Checks

Health checks verify that a container is ready to receive traffic. When configured, the platform waits for the health check to pass before routing traffic to a new deployment.

| Setting      | Default | Description                                            |
| ------------ | ------- | ------------------------------------------------------ |
| Command      | —       | Shell command to run inside the container              |
| Interval     | `10s`   | Time between checks                                    |
| Timeout      | `5s`    | Maximum time for a single check                        |
| Retries      | `3`     | Consecutive failures before marking unhealthy          |
| Start period | `30s`   | Grace period after container start before checks count |

Health check statuses:

| Status      | Description                    |
| ----------- | ------------------------------ |
| `none`      | No health check configured     |
| `starting`  | Within the start period        |
| `healthy`   | Check is passing               |
| `unhealthy` | Check has failed after retries |

If no health check command is set, deployment readiness only confirms that the container started. The platform still waits for DNS, proxy routes, certificates, and a successful Traefik configuration reload before marking the rollout complete, but it does not verify that the application inside the container accepts requests. Configure a health check when application readiness matters.


# Custom Domains
Source: https://docs.techulus.cloud/services/domains

Bind custom domains with automatic HTTPS.

## Adding a Domain

Custom domains are bound to a specific service port. When you add a domain:

1. Point your domain's DNS to your edge hostname.
2. Add the domain in the service port settings.
3. The platform automatically provisions a TLS certificate via Let's Encrypt.

Traffic flows: `Internet → Proxy Node → Traefik (TLS) → WireGuard → Container`.

## Automatic Subdomains

Configure **Automatic Subdomain Domain** under **Settings → Infrastructure** to
offer automatic domains in service networking settings. For example, with the
setting configured as `apps.example.com`, a service whose private hostname is
`project-api-production` can use `project-api-production.apps.example.com`.

Create a wildcard `CNAME` record for `*.apps.example.com` that points to your
edge hostname. The setting contains only the base domain, without `*.` or a
protocol. Custom domains remain available alongside automatic domains.

## TLS Certificates

Certificates are provisioned using the ACME HTTP-01 challenge. The control plane handles the challenge validation — Traefik routes `/.well-known/acme-challenge/*` requests back to the control plane.

Certificates are:

* Issued automatically when a domain is added.
* Stored in the database and distributed to all proxy nodes.
* Renewed automatically before expiration.

## Multiple Proxy Nodes

When using multiple proxy nodes for geographic distribution, all proxies share the same TLS certificates from the control plane.

A stable external load balancer with active health checks is the ideal production
solution for proxy failure. Configure every proxy as an origin, point the edge
hostname to the load balancer, and point custom domains to the edge hostname.

Health-aware GeoDNS is an alternative, but failover remains subject to DNS and
client caching. Plain multiple A records provide best-effort distribution and do
not guarantee that clients avoid an offline proxy. See the
[Architecture](/architecture#multiple-proxy-nodes) page for details.

## Protocols

Each service port specifies a protocol:

| Protocol | Description                                                      |
| -------- | ---------------------------------------------------------------- |
| `http`   | HTTP/HTTPS traffic routed through Traefik                        |
| `tcp`    | Raw TCP traffic — see [TCP/UDP Proxy](/networking/tcp-udp-proxy) |
| `udp`    | Raw UDP traffic — see [TCP/UDP Proxy](/networking/tcp-udp-proxy) |

For HTTP ports, you can optionally bind a custom domain. For TCP/UDP ports, traffic is exposed via an external port on the proxy node.


# Scaling
Source: https://docs.techulus.cloud/services/scaling

Replicas, placement, and server pinning.

## Replicas

Each service can run multiple replicas across your cluster. Configure how many replicas run on each server from the service settings.

Replica count ranges from 1 to 32 per service.

Automatic placement supports two replica-management modes:

* **Fixed** keeps the configured replica count running.
* **Autoscaled** keeps the active count within a configured minimum and maximum
  from 1 through 32. CPU and memory each use a fixed 60% target; thresholds are
  not configurable.

Autoscaling is available only for automatic-placement, stateless,
non-serverless services without volumes. Both CPU and memory limits must be
configured because utilization is measured against those limits. Saving stages
the policy; deploying snapshots it into the immutable revision and activates it.

The controller evaluates trailing three-minute CPU and memory averages every
minute. Scale-down additionally requires five minutes of complete low-usage
observations. Missing or stale metrics hold the current count. A scaling attempt
starts a ten-minute cooldown, including failed attempts.

Autoscaling V1 scales up directly to the CPU- or memory-based recommendation.
After stabilization, it scales down one replica at a time. Each change uses the
normal rolling full-fleet replacement. Existing containers remain active until
the complete replacement fleet is healthy and routing has converged, but this
can temporarily require both the old and replacement capacity and can reset
long-lived connections.

Changing the configured range clamps an active count outside the new bounds
directly to the nearest bound in one rollout.

## Serverless scaling

Public HTTP services can be configured to sleep when idle on proxy nodes. A
sleeping deployment keeps active traffic intent, records stopped runtime intent,
and stops the local container. The next public HTTP request wakes local
proxy-hosted deployments and is held until an upstream is ready or the wake
timeout is reached.

For services with proxy-hosted serverless replicas, public traffic must be sent
only to proxy nodes that host a local proxy replica for that service. Non-owner
proxies do not emit a public HTTP route for the service, so DNS or the external
load balancer must avoid those proxies for that service's domains.

Serverless settings are configured per service:

| Setting           | Default | Description                                                       |
| ----------------- | ------- | ----------------------------------------------------------------- |
| Enable serverless | Off     | Allows proxy-hosted HTTP deployments to sleep and wake on request |
| Sleep after       | `300s`  | Idle period before running containers are stopped                 |
| Wake timeout      | `300s`  | Maximum time the wake gateway waits for ready upstreams           |

The serverless settings appear only when the service has a public HTTP port with
a domain. Removing the final qualifying endpoint disables serverless in the
pending service configuration.

A cold wake starts the sleeping local proxy replicas for that host. Held
requests resume when one upstream is ready.

Serverless services require at least one configured replica.

## Placement

Stateless services support automatic and manual placement. Automatic placement
stores either a fixed count or an autoscaling range and distributes the concrete
replica target across online, configured nodes during rollout. Manual placement
selects exact target servers and replica counts.

Serverless services support automatic placement across online, configured proxy
nodes. Public ingress must use health checks to avoid proxy nodes that do not
currently own a replica for the service.

## Server Pinning

Stateful services (those with [volumes](/services/volumes)) are automatically pinned to a single server. This ensures the container always mounts the same local data path and avoids accidentally starting on a server that does not have the volume.

Pinning does not provide high availability. Volume data is not replicated across servers, and if the pinned server is lost, the service must be recovered from completed backups.

You can also manually lock any service to a specific server by setting the locked server. This is useful for workloads that need to run on a particular machine.

## Limitations

* Stateful services are limited to 1 replica.
* Stateful services are always pinned to their locked server.
* Stateful services do not automatically fail over to another server.
* Maximum 32 replicas per service.
* Serverless scaling requires a public HTTP service domain.
* Sleep and wake are proxy-local; serverless replicas must be placed on proxy nodes.
* Serverless traffic must be routed only to proxy nodes that own a local proxy replica for that service.
* Proxy agents report sleep and wake transitions through normal status reports.


# Volumes
Source: https://docs.techulus.cloud/services/volumes

Persistent storage for stateful services.

Volumes provide persistent storage that survives container restarts and redeployments.

> **Stateful storage warning:** Volumes are stored on a single server's local filesystem. Techulus Cloud does not currently provide replicated volumes, automatic storage failover, or high availability for stateful services. If the server hosting a volume is lost, data can only be recovered from completed backups. Do not use Techulus Cloud local volumes for production databases.

## Adding Volumes

Each volume has a name and a container path:

| Field          | Description                                                      |
| -------------- | ---------------------------------------------------------------- |
| Name           | Unique identifier for the volume                                 |
| Container path | Where the volume is mounted inside the container (e.g., `/data`) |

When you add a volume, the service automatically becomes **stateful**. Stateful services are locked to a single server and limited to 1 replica so the container always mounts the same local data path. When the last volume is removed, the service reverts to stateless.

Stateful services can use serverless scaling when they have a public HTTP domain
and manual placement on a proxy node. Automatic serverless placement is only
available to stateless services without volumes.

## Volume Backups

Volumes can be backed up to S3-compatible storage on a schedule or on demand.

| Setting         | Description                          |
| --------------- | ------------------------------------ |
| Backup enabled  | Toggle automatic backups             |
| Backup schedule | Cron expression for backup frequency |

Backups stop the running container, compress the volume as a `.tar.gz` archive, upload it to the configured [backup storage](/infrastructure/backups), then start the container again if it was running. Each backup tracks its size, checksum, and completion status.

Backup statuses:

| Status      | Description          |
| ----------- | -------------------- |
| `pending`   | Backup queued        |
| `uploading` | Uploading to storage |
| `completed` | Successfully stored  |
| `failed`    | Backup failed        |

## Restoring

You can restore a volume from any completed backup. The restore process downloads the backup archive from storage and extracts it to the volume path on the target server.

## Limitations

* Services with volumes are locked to a single server.
* Replica count is fixed at 1 for stateful services.
* Volume data lives on the host filesystem and is not replicated to other servers.
* If the server is lost, data is only recoverable from completed backups.
* Backups are point-in-time recovery, not high availability or automatic failover.