45 Commits
Author SHA1 Message Date
qpismont 74207da914 Merge pull request 'Postgres persistance + job queue + api with sse' (#9) from 1.3 into main
ci/woodpecker/push/tests Pipeline failed
Reviewed-on: #9
2026-10-26 21:19:31 +01:00
qpismont 426a10d5be update lock cve
ci/woodpecker/push/tests Pipeline was successful
2026-10-14 16:52:12 +00:00
qpismont cedd052a4d same migration process test and prod
ci/woodpecker/push/tests Pipeline was canceled
2026-10-14 16:50:02 +00:00
qpismont b40985528e rwlock to mutex
ci/woodpecker/push/tests Pipeline failed
2026-10-14 16:41:50 +00:00
qpismont 131f1ec84a Fix rust version
ci/woodpecker/push/tests Pipeline was successful
2026-10-13 20:09:15 +00:00
qpismont 3078ba5705 clean project structure
ci/woodpecker/push/tests Pipeline was canceled
2026-10-13 19:57:24 +00:00
qpismont 7493de6952 Add live dashboard with SSE job events
ci/woodpecker/push/tests Pipeline was successful
Expose /jobs, /dedup, and /events routes backed by a new EventBus that
broadcasts job lifecycle and per-job progress over SSE. Serve a static
dashboard from front/ that lists pending and in-progress jobs, streams
agent activity, and shows completed reviews.

Also add migrate-apply/migrate-generate scripts that load .env, update
the Containerfile to ship the front assets and use the apply script, and
refresh the roadmap.
2026-10-13 16:14:48 +00:00
qpismont 8c763786c1 Remove apalis + redis => use postgresql instead
ci/woodpecker/push/tests Pipeline was successful
2026-10-06 19:41:14 +00:00
qpismont d74db2096c Project website + use Apalis crate for task queue (with redis backend)
ci/woodpecker/push/tests Pipeline was successful
instead channel and semaphore queue + upgrade to rust 1.99
2026-10-05 20:44:57 +00:00
qpismont c546707f72 Merge pull request 'add default image docker fallback' (#8) from 1.2.1 into main
ci/woodpecker/push/tests Pipeline was successful
ci/woodpecker/tag/release Pipeline was successful
Reviewed-on: qpismont/herald#8
2026-09-29 20:50:36 +02:00
qpismont f5f0d3999e rebranding herald => steep
ci/woodpecker/push/tests Pipeline was successful
2026-09-29 18:36:01 +00:00
qpismont 9f03384bc9 Add roadmap
ci/woodpecker/push/tests Pipeline was successful
2026-09-29 17:49:24 +00:00
qpismont ece8f8d29c Fix inconsistency between doc and code for workspaceFolder destination
ci/woodpecker/push/tests Pipeline was successful
2026-09-29 16:42:37 +00:00
qpismont 4999091f32 improve cleanup if container setup fail
ci/woodpecker/push/tests Pipeline was successful
2026-09-29 16:30:34 +00:00
qpismont e7b8048992 add default image docker fallback
ci/woodpecker/push/tests Pipeline was successful
2026-09-22 20:57:21 +00:00
qpismont 589d337057 Merge pull request '1.2: Sandboxing' (#7) from 1.2 into main
ci/woodpecker/push/tests Pipeline was successful
ci/woodpecker/tag/release Pipeline was successful
Reviewed-on: qpismont/herald#7
2026-09-20 16:58:36 +02:00
qpismont fa03fd8589 add review type (perf, bug, security, ...) + fix tests
ci/woodpecker/push/tests Pipeline was successful
2026-09-20 14:42:14 +00:00
qpismont be7284c6d8 Fix file_size test
ci/woodpecker/push/tests Pipeline failed
2026-09-20 13:21:52 +00:00
qpismont 48e09aa373 Fix missing workspace when herald run in container
ci/woodpecker/push/tests Pipeline failed
2026-09-20 13:10:05 +00:00
qpismont 620ec6727e clean devcontainer-rs architecture
ci/woodpecker/push/tests Pipeline was successful
2026-09-17 20:37:54 +00:00
qpismont 624bc1e028 replace ContainerRuntime CLI to Bollard crate (default: docker socket)
ci/woodpecker/push/tests Pipeline was successful
limit tool result to open router tool response (with truncated info for
ai)

Hard kill if graceful shutdown is too long
2026-09-17 19:51:59 +00:00
qpismont 78ad2bf701 translate comment
ci/woodpecker/push/tests Pipeline was successful
2026-09-17 14:49:41 +00:00
qpismont 04cc172848 Replace openrouter-rs with in-tree client and require sandbox
Remove the openrouter-rs dependency in favor of a minimal in-tree
OpenRouter chat-completions client, and drop the BOT_NAME and
SANDBOX_ENABLED config options. Reviews now always run inside the
sandbox, and the review prompt asks the model to read files with the
available tools instead of embedding the diff.
2026-09-17 14:48:42 +00:00
qpismont ee221b0954 Update rust version in ci job
ci/woodpecker/push/tests Pipeline was successful
2026-09-17 13:22:24 +00:00
qpismont 99d1c2feef Add sandbox
ci/woodpecker/push/tests Pipeline was canceled
2026-09-17 13:16:15 +00:00
qpismont 536c55f27b update deps + fix sentry config
ci/woodpecker/push/tests Pipeline was successful
2026-09-01 10:07:39 +00:00
qpismont a29051b0e4 Fix clippy errors
ci/woodpecker/push/tests Pipeline was successful
2026-07-31 20:39:50 +00:00
qpismont 8c53bc0e20 re fix fmt lol
ci/woodpecker/push/tests Pipeline failed
2026-07-31 20:33:47 +00:00
qpismont f0e64e0c1d Fix fmt
ci/woodpecker/push/tests Pipeline failed
2026-07-31 20:32:59 +00:00
qpismont 6a21c7d6c3 Renforce woodpecker tests
ci/woodpecker/push/tests Pipeline failed
2026-07-31 20:25:37 +00:00
qpismont b3a0cb63e9 Update woodpecker rust job (1.96 => 1.97)
ci/woodpecker/push/tests Pipeline was successful
2026-07-31 20:21:05 +00:00
qpismont 5b9d870b46 Dockerfile field must be present
ci/woodpecker/push/tests Pipeline was successful
2026-07-31 20:18:29 +00:00
qpismont 15f619ccf7 Move to multi crates project
ci/woodpecker/push/tests Pipeline was successful
Starting impl devcontainer spec
2026-07-31 20:06:37 +00:00
qpismont d711153553 Merge pull request 'Observability' (#6) from 1.1 into main
ci/woodpecker/push/tests Pipeline was successful
ci/woodpecker/tag/release Pipeline was successful
Reviewed-on: qpismont/herald#6
2026-07-26 21:55:41 +02:00
qpismont b6a299ac18 bump version
ci/woodpecker/push/tests Pipeline was successful
2026-07-26 18:41:22 +00:00
qpismont c426b9b513 update readme
ci/woodpecker/push/tests Pipeline was successful
2026-07-26 11:56:58 +00:00
qpismont e1cb5d7d96 fix usd metric
ci/woodpecker/push/tests Pipeline was successful
2026-07-26 11:41:58 +00:00
qpismont 6ffd88927c add prometheus metrics
ci/woodpecker/push/tests Pipeline was successful
2026-07-26 11:02:34 +00:00
qpismont 7252bf7673 fix tests
ci/woodpecker/push/tests Pipeline was successful
2026-06-30 20:48:46 +00:00
qpismont 743b6b33c9 Switch to vscode + fetch bot_name with token
ci/woodpecker/push/tests Pipeline failed
2026-06-30 20:44:25 +00:00
qpismont 7f24d7657c Merge pull request 'Fix missing env var error' (#5) from 1.0.1 into main
ci/woodpecker/tag/release Pipeline was successful
Reviewed-on: qpismont/herald#5
2026-06-16 22:05:24 +02:00
qpismont a613fdb99e Another trace clear 2026-06-16 18:53:33 +00:00
qpismont 975581093a Add missing sentry error backtrace
Clear traces spam
2026-06-16 18:49:35 +00:00
qpismont 00d46ce968 Fix AI review
Webhook action check before user check
2026-06-12 22:01:49 +00:00
qpismont 3f6c5b5559 Fix missing env var error
ci/woodpecker/push/tests Pipeline was successful
2026-06-12 21:27:12 +00:00
87 changed files with 12261 additions and 1795 deletions

No files matched your search

+7 -7
View File
@@ -1,4 +1,4 @@
FROM debian:trixie FROM rust:1.99-trixie
ARG USERNAME=dev ARG USERNAME=dev
ARG USER_UID=1000 ARG USER_UID=1000
@@ -11,18 +11,18 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
curl \ curl \
git \ git \
build-essential \ build-essential \
libssl-dev \
cmake \ cmake \
pkg-config \ pkg-config \
clang \ clang \
postgresql \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
RUN groupadd --gid ${USER_GID:-1000} $USERNAME \ RUN groupadd --gid ${USER_GID:-1000} $USERNAME \
&& useradd --uid ${USER_UID:-1000} --gid ${USER_GID:-1000} -m $USERNAME && useradd --uid ${USER_UID:-1000} --gid ${USER_GID:-1000} -m $USERNAME \
&& rustup component add clippy \
&& rustup component add rustfmt
USER $USERNAME USER $USERNAME
WORKDIR /home/$USERNAME WORKDIR /home/$USERNAME
ENV PATH="/home/${USERNAME}/.cargo/bin:${PATH}"
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable
+11 -3
View File
@@ -1,5 +1,5 @@
{ {
"name": "herald", "name": "steep",
"build": { "build": {
"dockerfile": "Dockerfile", "dockerfile": "Dockerfile",
"args": { "args": {
@@ -12,8 +12,16 @@
"containerEnv": { "containerEnv": {
"SHELL": "/bin/bash" "SHELL": "/bin/bash"
}, },
"workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/herald,type=bind", "customizations": {
"workspaceFolder": "/workspaces/herald", "vscode": {
"extensions": ["fill-labs.dependi", "rust-lang.rust-analyzer", "tamasfe.even-better-toml"]
}
},
"workspaceMount": "source=${localWorkspaceFolder},target=/workspaces/steep,type=bind",
"workspaceFolder": "/workspaces/steep",
"mounts": [
"source=/run/user/1000/podman/podman.sock,target=/var/run/docker.sock,type=bind"
],
"runArgs": ["--userns=keep-id", "--security-opt", "label=disable"], "runArgs": ["--userns=keep-id", "--security-opt", "label=disable"],
"appPort": [3000] "appPort": [3000]
} }
-4
View File
@@ -1,4 +0,0 @@
target/
.env
.devcontainer/
docs/
+9 -1
View File
@@ -1,5 +1,4 @@
HTTP_PORT=3000 HTTP_PORT=3000
BOT_NAME=Herald
WEBHOOK_SIG_HEADER_SECRET= WEBHOOK_SIG_HEADER_SECRET=
@@ -16,3 +15,12 @@ GITEA_TIMEOUT=30
# Optional # Optional
SENTRY_DSN= SENTRY_DSN=
RUST_LOG=info RUST_LOG=info
RUST_BACKTRACE=1
METRICS_BIND_ADDR=
# Sandboxed tool execution
# DOCKER_HOST=
SANDBOX_MAX_ITERATIONS=8
DATABASE_URL=
+2 -1
View File
@@ -10,8 +10,9 @@ steps:
- /data/woodpecker-builds:/data - /data/woodpecker-builds:/data
commands: commands:
- echo $DOCKER_PASSWORD | buildah login docker.io -u $DOCKER_USERNAME --password-stdin - echo $DOCKER_PASSWORD | buildah login docker.io -u $DOCKER_USERNAME --password-stdin
- chmod +x scripts/build.sh - chmod +x scripts/build.sh scripts/build-website.sh
- bash scripts/build.sh - bash scripts/build.sh
- bash scripts/build-website.sh
environment: environment:
DOCKER_USERNAME: DOCKER_USERNAME:
from_secret: docker_username from_secret: docker_username
+40 -4
View File
@@ -2,14 +2,50 @@ when:
event: event:
- push - push
services:
- name: postgres
image: postgres:18
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: steep
steps: steps:
- name: fmt
image: rust:1.99
commands:
- rustup component add rustfmt
- cargo fmt --all -- --check
- name: clippy - name: clippy
image: rust:1.96 image: rust:1.99
commands: commands:
- rustup component add clippy - rustup component add clippy
- cargo clippy - cargo clippy --workspace --all-targets --all-features -- -D warnings
- name: deny
image: rust:1.99
commands:
- cargo install cargo-deny --locked --version 0.18.9
- cargo deny check
- name: wait-for-postgres
image: postgres:18
commands:
- until pg_isready -h postgres -U postgres; do sleep 1; done
- name: test - name: test
image: rust:1.96 image: rust:1.99
environment:
DATABASE_URL: postgresql://postgres:postgres@postgres:5432/steep
commands: commands:
- cargo test - bash scripts/migrate-apply.sh
- cargo test --workspace --all-targets -- --include-ignored
- name: container-build
image: quay.io/buildah/stable
privileged: true
volumes:
- /data/woodpecker-builds:/data
commands:
- buildah bud -f Containerfile -t steep-ci .
+12 -1
View File
@@ -1,9 +1,20 @@
{ {
"languages": {
"Rust": {
"format_on_save": "on",
"formatter": "language_server"
}
},
"lsp": { "lsp": {
"rust-analyzer": { "rust-analyzer": {
"initialization_options": { "initialization_options": {
"check": { "check": {
"command": "clippy" "command": "clippy",
"extraArgs": [
"--",
"-D",
"warnings"
]
} }
} }
} }
Generated
+3460 -832
View File
File diff suppressed because it is too large. Load diff
+44 -17
View File
@@ -1,29 +1,56 @@
[package] [workspace]
name = "herald" members = [
version = "1.0.0" "crates/steep-server",
edition = "2024" "crates/devcontainer-rs",
]
resolver = "3"
[dependencies] [workspace.dependencies]
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } # TLS is pure Rust (rustls + ring): no libssl is needed at build or run time.
tokio = { version = "1.52", features = ["full"] } # `rustls-no-provider` leaves the crypto provider to the binary, which installs
# `ring` at startup (see `main`), so that a single provider is ever compiled in.
reqwest = { version = "0.13", default-features = false, features = ["json", "rustls-no-provider", "stream"] }
rustls = { version = "0.23", default-features = false, features = ["ring", "std", "tls12"] }
tokio = { version = "1.53", features = ["full"] }
tokio-stream = "0.1" tokio-stream = "0.1"
tokio-util = "0.7" tokio-util = "0.7"
futures-util = "0.3" futures-util = "0.3"
serde_json = "1.0" serde_json = "1.0"
serde = { version = "1.0", features = ["derive"] } serde = { version = "1.0", features = ["derive"] }
sentry = { version = "0.48", features = ["tower-axum-matched-path"] } # Sentry's default `transport` feature pulls `native-tls`; the same set of features
sentry-anyhow = "0.48" # is spelled out here with rustls instead.
openrouter-rs = "0.10" sentry = { version = "0.49", default-features = false, features = [
"backtrace",
"contexts",
"debug-images",
"logs",
"metrics",
"panic",
"release-health",
"reqwest",
"rustls-no-provider",
"tower-axum-matched-path",
] }
sentry-anyhow = { version = "0.49", features = ["backtrace"] }
dotenvy = "0.15" dotenvy = "0.15"
tower = "0.5" tower = "0.5"
tower-http = {version = "0.6", features = ["trace"] } tower-http = { version = "0.7", features = ["trace", "fs"] }
tracing = "0.1" tracing = "0.1"
tracing-subscriber = { version = "0.3", features=["env-filter"] } tracing-subscriber = { version = "0.3", features = ["env-filter"] }
axum = "0.8" axum = "0.8"
anyhow = "1.0" anyhow = { version = "1", features = ["backtrace"] }
thiserror = "2.0" thiserror = "2.0"
hmac = "0.13" ring = "0.17"
sha2 = "0.11"
hex = "0.4" hex = "0.4"
subtle = "2.6" bytes = "1.1"
bytes = "1.11" metrics = "0.24"
metrics-exporter-prometheus = { version = "0.18", default-features = false, features = ["http-listener"] }
toasty = { version = "0.11", features = ["postgresql", "serde", "jiff"] }
toasty-cli = "0.11"
jiff = { version = "0.2", features = ["serde"] }
tempfile = "3"
bollard = "0.21"
tar = "0.4"
[profile.release]
debug = 1
+24 -5
View File
@@ -1,12 +1,31 @@
FROM rust:1.96 as builder FROM rust:1.99-trixie as builder
WORKDIR /app WORKDIR /app
COPY . .
RUN cargo build --release COPY Cargo.toml Cargo.lock ./
COPY crates/ crates/
RUN cargo build --release --package steep-server
FROM debian:trixie-slim FROM debian:trixie-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
git \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app WORKDIR /app
COPY --from=builder /app/target/release/herald . COPY --from=builder /app/target/release/steep-server ./steep-server
CMD [ "./herald" ]
COPY front ./front
COPY crates/steep-server/toasty ./toasty
ENV MIGRATIONS_DIR=/app/toasty
COPY scripts/migrate-apply.sh ./scripts/migrate-apply.sh
RUN chmod +x ./scripts/migrate-apply.sh
CMD ["sh", "-c", "./scripts/migrate-apply.sh && exec ./steep-server"]
+152 -9
View File
@@ -1,6 +1,10 @@
# Herald ![Steep — Let your code steep](public/banniere_readme.png)
Herald is a Gitea bot that performs automated AI-powered code reviews on pull requests using [OpenRouter](https://openrouter.ai/). # Steep
*Let your code steep.*
Steep is a Gitea bot that performs automated AI-powered code reviews on pull requests using [OpenRouter](https://openrouter.ai/).
## Features ## Features
@@ -9,23 +13,23 @@ Herald is a Gitea bot that performs automated AI-powered code reviews on pull re
- Concurrent review processing with configurable parallelism - Concurrent review processing with configurable parallelism
- Graceful shutdown — in-progress reviews finish before the process exits - Graceful shutdown — in-progress reviews finish before the process exits
- Error tracking via Sentry - Error tracking via Sentry
- Tiny memory footprint (~4MB) thanks to Rust - Prometheus metrics endpoint for monitoring
- Lightweight, performant Rust service with Prometheus metrics and Sentry error tracking
## Installation ## Installation
**Requirements:** Rust toolchain ([rustup.rs](https://rustup.rs)) **Requirements:** Rust toolchain ([rustup.rs](https://rustup.rs))
```sh ```sh
cargo build --release cargo build --release -p steep-server
./target/release/herald ./target/release/steep-server
``` ```
Herald reads its configuration from environment variables (a `.env` file is supported): Steep reads its configuration from environment variables (a `.env` file is supported):
| Variable | Description | | Variable | Description |
|---|---| |---|---|
| `HTTP_PORT` | Port to listen on | | `HTTP_PORT` | Port to listen on |
| `BOT_NAME` | The bot's Gitea username (used to detect mentions) |
| `WEBHOOK_SIG_HEADER_SECRET` | Gitea webhook secret for signature verification | | `WEBHOOK_SIG_HEADER_SECRET` | Gitea webhook secret for signature verification |
| `OPEN_ROUTER_API_KEY` | OpenRouter API key | | `OPEN_ROUTER_API_KEY` | OpenRouter API key |
| `OPEN_ROUTER_MODEL` | Model to use (e.g. `deepseek/deepseek-v4-flash`) | | `OPEN_ROUTER_MODEL` | Model to use (e.g. `deepseek/deepseek-v4-flash`) |
@@ -34,20 +38,159 @@ Herald reads its configuration from environment variables (a `.env` file is supp
| `GITEA_URL` | Base URL of your Gitea instance | | `GITEA_URL` | Base URL of your Gitea instance |
| `GITEA_TOKEN` | Gitea API token | | `GITEA_TOKEN` | Gitea API token |
| `GITEA_TIMEOUT` | Gitea API request timeout in seconds | | `GITEA_TIMEOUT` | Gitea API request timeout in seconds |
| `DATABASE_URL` | PostgreSQL connection URL used for the durable review queue (e.g. `postgresql://user:pass@localhost:5432/steep`) |
| `METRICS_BIND_ADDR` | *(optional)* Bind address for the Prometheus metrics endpoint (e.g. `0.0.0.0:9100`). If unset, the metrics exporter is disabled. |
| `SENTRY_DSN` | *(optional)* Sentry DSN for error tracking | | `SENTRY_DSN` | *(optional)* Sentry DSN for error tracking |
| `RUST_LOG` | *(optional)* Log level, defaults to `info` | | `RUST_LOG` | *(optional)* Log level, defaults to `info` |
| `SANDBOX_MAX_ITERATIONS` | *(optional)* Maximum number of tool-calling iterations per sandboxed review. Defaults to `8`; a value that is set but is not a number stops the startup |
| `DOCKER_HOST` | *(optional)* Container daemon socket Steep drives, e.g. `unix:///run/user/1000/podman/podman.sock` for a rootless podman. Defaults to `unix:///var/run/docker.sock` |
## Sandboxed reviews
Steep reviews pull requests inside an ephemeral
[Dev Container](https://containers.dev/). For each review it:
1. clones the pull request head into a temporary directory,
2. builds and starts the repository's devcontainer (`devcontainer-rs`) — or, when
the repository has none, a default one based on `debian:stable-slim`,
3. reads the pull request diff and file list from the Gitea API with
`GITEA_TOKEN` (so private repositories work), tells the model which files and
lines changed — additions and deletions, with the line numbers of the new and
old versions of the file respectively — then lets it explore the repository
with read-only tools (`ls`, `file_size`, `read_file`, `grep`, `find`) run inside
the container: the code itself is not sent, so the model reads it at those lines,
4. posts the review, anchoring each comment on the added or removed line it
refers to, and removes the container and the temporary clone.
Generated files are left out of the changes handed to the model: lockfiles
(`Cargo.lock`, `package-lock.json`, `yarn.lock`, `go.sum`…) are machine-written
dependency churn whose thousands of lines would drown the code under review, and
they are never a place where a comment belongs.
Each comment is tagged with a severity — `bug`, `security`, `performance` or
`maintainability` — shown at the start of the comment, and the summary also lists
what the pull request does well.
Steep drives the container daemon through its socket: `DOCKER_HOST` (default
`unix:///var/run/docker.sock`), which covers both docker and podman's
Docker-compatible socket. When the repository contains a
`.devcontainer/devcontainer.json` (or `.devcontainer.json`), Steep uses it;
otherwise it falls back to a default devcontainer that pulls `debian:stable-slim`
and runs the review in `/workspace`.
Steep can therefore run inside a container with only that socket mounted (no
shared workspace directory is required): the clone is streamed to the daemon over
the socket, like the build context, instead of being bind-mounted from a host
path the daemon would have to see. This is the setup the `Containerfile`
produces, e.g.:
```sh
podman run --env-file=.env -p 3001:3001 \
-v /run/user/$(id -u)/podman/podman.sock:/var/run/docker.sock \
steep:latest
```
The `runArgs` of that file are ignored: they come from an untrusted pull request,
and one of them (`--network host`) would attach the container to another network
and quietly defeat the network cut described below. A devcontainer that relies on
them (`--gpus all`, `--cap-add`, `--shm-size`…) will not get them.
Each sandbox is isolated: it gets its own image tag, container and network. The
container starts with network access so the `postCreateCommand` /
`postStartCommand` hooks can install dependencies (e.g. `npm install`); once the
hooks have run, the container is disconnected from the network for the rest of
the review. Every container command is bounded by a timeout, and the container,
network and image are removed when the review ends (including on failure).
## Migrations
The database schema is versioned with [Toasty](https://github.com/tokio-rs/toasty) migrations, stored in `crates/steep-server/toasty/` (commit them).
After changing a model in `crates/steep-server/src/schemas/`, generate a migration:
```sh
./scripts/migrate-generate.sh describe_change
```
Apply the pending migrations **before starting the service** — the server does not migrate at boot. The tests need the same schema: run this before `cargo test` too (CI does it as well):
```sh
./scripts/migrate-apply.sh
```
Both scripts load the `.env` file at the project root (so `DATABASE_URL` is picked up). Generation runs through `cargo run -p steep-server --`; applying uses the compiled `steep-server` binary when present (as in the image), falling back to `cargo run` otherwise.
The `Containerfile` does this at startup: its entrypoint runs `scripts/migrate-apply.sh` before exec'ing the server. The CLI reads the migration files from `MIGRATIONS_DIR` (defaults to the crate's `toasty/` directory; the image sets it to `/app/toasty`).
Other subcommands: `migration snapshot`, `migration drop`, `migration reset`. They read `DATABASE_URL`.
## Development ## Development
The easiest way to get started is with the provided [Dev Container](https://containers.dev/) (VS Code or Zed with the dev container extension). The easiest way to get started is with the provided [Dev Container](https://containers.dev/) (VS Code or Zed with the dev container extension).
Open the project and reopen it in the container — the Rust toolchain is pre-installed. Open the project and reopen it in the container — the Rust toolchain is pre-installed, along with rootless podman, so sandboxed reviews can be exercised locally: start its API socket with `podman system service --time=0 &` and set `DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock`.
**Without Dev Container**, you just need a Rust toolchain: **Without Dev Container**, you just need a Rust toolchain:
```sh ```sh
rustup toolchain install stable rustup toolchain install stable
cargo run cargo run -p steep-server
``` ```
Copy `.env.example` to `.env` and fill in your values before running. Copy `.env.example` to `.env` and fill in your values before running.
## Job queue
Steep is meant to run as a **single instance**. Webhooks are stored in the `jobs` table
of PostgreSQL, and the worker is woken up as soon as a job is pushed (no polling).
A job still `processing` at startup was interrupted by the previous process: it is put
back in the queue and reviewed again. Do not run two instances on the same database,
they would both take over each other's running jobs.
## Dependencies
TLS is handled by [rustls](https://github.com/rustls/rustls) (with `ring`), so neither
the build nor the image needs OpenSSL. The `Containerfile` runtime image only
installs `ca-certificates` and `git`.
`cargo deny check` audits the dependencies (security advisories, licenses, banned
crates, sources) with the policy of `deny.toml`. It runs in the CI, and it also fails
if `openssl-sys` or `native-tls` comes back into the dependency tree.
## Metrics
Steep optionally exposes a Prometheus metrics endpoint, useful for scraping with an [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) or any Prometheus-compatible scraper.
Set `METRICS_BIND_ADDR` (e.g. `0.0.0.0:9100`) to enable it. The metrics are then available at `http://<host>:9100/metrics`. The metric namespace changed from `herald_*` to `steep_*`; update any dashboards or alerts that use the old names.
### Exposed metrics
| Metric | Type | Description |
|---|---|---|
| `steep_webhooks_received_total` | counter | Total webhooks received (label: `event_type`) |
| `steep_webhooks_duplicate_total` | counter | Webhooks rejected as duplicates (label: `event_type`) |
| `steep_webhooks_enqueue_failed_total` | counter | Webhooks dropped because enqueueing to PostgreSQL failed (label: `event_type`) |
| `steep_bot_tasks_active` | gauge | Bot tasks currently in progress |
| `steep_bot_tasks_skipped_total` | counter | Reviews skipped because they were already done (label: `event_type`) |
| `steep_bot_tasks_completed_total` | counter | Bot tasks completed successfully (label: `event_type`) |
| `steep_bot_tasks_failed_total` | counter | Bot tasks that failed (label: `event_type`) |
| `steep_openrouter_cost_cents_total` | counter | Total OpenRouter cost in cents (divide by 100 for USD) |
### OTel collector example
```yaml
receivers:
prometheus:
config:
scrape_configs:
- job_name: steep
scrape_interval: 15s
static_configs:
- targets: ["steep:9100"]
service:
pipelines:
metrics:
receivers: [prometheus]
exporters: [otlp]
```
+49
View File
@@ -0,0 +1,49 @@
# Roadmap
## 1.2 — Sandboxed reviews — delivered
- Run each pull request review in an ephemeral Dev Container.
- Let the model explore the checked-out repository through read-only tools instead of sending the source code in the prompt.
- Anchor review comments to changed lines and classify findings by severity.
## 1.2.1 — Default Dev Container — delivered
- If a repository has no Dev Container configuration, run its review in a default container based on `debian:stable-slim`.
- Use `/workspace` as the workspace path for the default container and as the fallback when `workspaceFolder` is not specified.
## 1.3 — Durable jobs & resumption — planned
- Persist review jobs and their progress in PostgreSQL so in-flight work survives a restart and can be resumed instead of being lost.
- Expose an API to retrieve the list of jobs by state: running, pending, and review done.
- Stream job events over Server-Sent Events (SSE) so clients see status changes in real time.
## 1.4 — YAML configuration — planned
- Add a YAML configuration file as the primary way to configure Steep, replacing part of the current environment variables.
- Move non-secret settings such as the model, timeouts, concurrency limit, and sandbox iterations into the file, and keep secrets (API keys, tokens, webhook secret, database URL) in environment variables.
## 2.0 — Typed action routing and implementation requests — planned
- Introduce a type-safe model output for the action Steep should take, such as `Review` or `Implementation`.
- Route review requests (for example, “@steep, review this PR”) to the existing review flow.
- Recognize implementation requests (for example, “@steep, rename method `XXX` to `YYY`”) and route them to the implementation workflow.
- For a request on an issue, clone the repository from its default branch and implement the requested features on a new branch.
- For a request on an existing pull request, clone its branch and implement the requested modifications there.
- When the agent's iteration loop ends, the orchestration layer automatically pushes the resulting changes: it opens a pull request for issue-based work, or updates the existing pull request branch. The agent does not receive or invoke a push tool.
- Keep implementation work isolated in a sandbox; do not modify the default branch directly.
## 2.1 — `AGENTS.md` support and multi models — planned
- Read applicable `AGENTS.md` files from the repository and use their project-specific instructions when carrying out implementation requests.
- Allow configuring a dedicated model for pull request reviews and another for implementation requests.
- Route each action type to its configured model, with a clear fallback when only one model is configured.
## 2.2 — Sub-agents — planned
- Let the agent delegate well-scoped parts of a task to sub-agents that run their own iteration loop.
- Run independent sub-agents in parallel and have the main agent consolidate their results.
## 3.0 — Autonomous Sentry fixes — planned
- Automatically discover project errors reported by Sentry, without waiting for a user comment.
- Investigate the errors in an isolated sandbox, implement candidate fixes, and validate them before reporting the results.
+21
View File
@@ -0,0 +1,21 @@
[package]
name = "devcontainer-rs"
version = "0.1.0"
edition = "2024"
publish = false
[dependencies]
bollard = { workspace = true }
bytes = { workspace = true }
futures-util = { workspace = true }
tar = { workspace = true }
tokio = { workspace = true }
tokio-stream = { workspace = true }
tracing = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
anyhow = { workspace = true }
thiserror = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }
+25
View File
@@ -0,0 +1,25 @@
//! Crate constants: daemon endpoint, timeouts and build context chunking.
use std::time::Duration;
/// Endpoint shown in the logs when `DOCKER_HOST` is not set.
pub(crate) const DEFAULT_ENDPOINT: &str = "unix:///var/run/docker.sock";
/// Timeout applied to build/run/stop/remove operations.
pub(crate) const DEFAULT_COMMAND_TIMEOUT: Duration = Duration::from_secs(600);
/// Timeout applied to commands run inside a running container.
pub(crate) const DEFAULT_EXEC_TIMEOUT: Duration = Duration::from_secs(60);
/// Size of the build context chunks sent to the daemon.
pub(crate) const CONTEXT_CHUNK_SIZE: usize = 64 * 1024;
/// Number of chunks that can wait in the channel: this is what bounds the
/// memory held by the context, regardless of its size.
pub(crate) const CONTEXT_CHUNKS: usize = 4;
/// Successive reads of an exec's exit code, and the delay between them.
pub(crate) const EXIT_CODE_ATTEMPTS: usize = 10;
pub(crate) const EXIT_CODE_DELAY: Duration = Duration::from_millis(20);
pub(crate) const DEFAULT_IMAGE_NAME: &str = "debian:stable-slim";
+94
View File
@@ -0,0 +1,94 @@
//! A running devcontainer: inspection and command execution.
use std::time::Duration;
use crate::{
consts::DEFAULT_EXEC_TIMEOUT, errors::ContainerError, exec::ExecOutput,
runtime::ContainerRuntime,
};
/// A running devcontainer.
#[derive(Debug, Clone)]
pub struct Container {
runtime: ContainerRuntime,
name: String,
workspace_folder: String,
remote_user: Option<String>,
network: Option<String>,
owned_image: Option<String>,
}
impl Container {
/// Assembles a started container, with its network and, if any, the image
/// built for it that must be cleaned up.
pub(crate) fn new(
runtime: ContainerRuntime,
name: String,
workspace_folder: String,
remote_user: Option<String>,
network: Option<String>,
owned_image: Option<String>,
) -> Self {
Self {
runtime,
name,
workspace_folder,
remote_user,
network,
owned_image,
}
}
pub fn name(&self) -> &str {
&self.name
}
pub fn workspace_folder(&self) -> &str {
&self.workspace_folder
}
/// Runs a command in the container and returns its output.
///
/// The command is passed as an argument vector (argv, no shell), so no
/// escaping or interpolation is performed.
pub async fn exec(&self, cmd: &[&str]) -> Result<ExecOutput, ContainerError> {
self.exec_with_timeout(cmd, DEFAULT_EXEC_TIMEOUT).await
}
pub(crate) async fn exec_with_timeout(
&self,
cmd: &[&str],
timeout: Duration,
) -> Result<ExecOutput, ContainerError> {
self.runtime
.exec(&self.name, cmd, self.remote_user.as_deref(), timeout)
.await
}
/// Runs a shell script in the container via `sh -c`.
pub async fn exec_shell(&self, script: &str) -> Result<ExecOutput, ContainerError> {
self.exec(&["sh", "-c", script]).await
}
/// Stops the container.
pub async fn stop(&self) -> Result<(), ContainerError> {
self.runtime.stop_container(&self.name).await
}
/// Removes the container, its network and its image.
///
/// Removing the network and the image is best-effort: they may already be gone.
pub async fn remove(&self) -> Result<(), ContainerError> {
self.runtime.remove_container(&self.name).await?;
if let Some(network) = &self.network {
let _ = self.runtime.remove_network(network).await;
}
if let Some(image) = &self.owned_image {
let _ = self.runtime.remove_image(image).await;
}
Ok(())
}
}
+166
View File
@@ -0,0 +1,166 @@
//! Streaming packaging of the build context sent to the daemon.
use std::{
io::{BufWriter, Write},
path::Path,
};
use bytes::Bytes;
use futures_util::Stream;
use tokio::sync::mpsc;
use tokio_stream::wrappers::ReceiverStream;
use crate::consts::{CONTEXT_CHUNK_SIZE, CONTEXT_CHUNKS};
/// Packs the build context into a tar, **as a stream**.
///
/// The `docker build` CLI packs the context and then sends it; doing that in one go
/// would load the whole folder into memory, which becomes untenable as soon as the
/// `.devcontainer` ships binaries. Here the archive is written by a blocking task
/// and pushed chunk by chunk, with the channel's backpressure: if the daemon reads
/// slowly, packing slows down.
pub(crate) fn tar_directory_stream(
dir: &Path,
) -> impl Stream<Item = Result<Bytes, std::io::Error>> + Send + 'static {
let (sender, receiver) = mpsc::channel(CONTEXT_CHUNKS);
let failures = sender.clone();
let dir = dir.to_path_buf();
tokio::task::spawn_blocking(move || {
let writer = BufWriter::with_capacity(CONTEXT_CHUNK_SIZE, ChannelWriter { sender });
let mut builder = tar::Builder::new(writer);
let result = builder
.append_dir_all(".", &dir)
.and_then(|()| builder.finish())
// `finish` writes the end-of-archive blocks; the buffer still has to be
// flushed so the daemon receives everything, even an empty archive.
.and_then(|()| builder.get_mut().flush());
if let Err(error) = result {
// The failure is reported to the daemon through the stream: otherwise it
// would only see a truncated tar and report a confusing error.
let _ = failures.blocking_send(Err(error));
}
});
ReceiverStream::new(receiver)
}
/// Writes the tar chunks into a channel, waiting for it to drain.
///
/// Used from a blocking task, the only context where `blocking_send` is allowed.
struct ChannelWriter {
sender: mpsc::Sender<Result<Bytes, std::io::Error>>,
}
impl Write for ChannelWriter {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.sender
.blocking_send(Ok(Bytes::copy_from_slice(buf)))
.map_err(|_| {
std::io::Error::new(
std::io::ErrorKind::BrokenPipe,
"the daemon dropped the build context",
)
})?;
Ok(buf.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::fs;
use futures_util::StreamExt;
/// Collects the context stream into a complete tar, for inspection.
async fn context_tar(dir: &Path) -> Vec<u8> {
let mut tar = Vec::new();
let mut chunks = tar_directory_stream(dir);
while let Some(chunk) = chunks.next().await {
tar.extend_from_slice(&chunk.unwrap());
}
tar
}
#[tokio::test]
async fn tar_context_holds_the_devcontainer_files() {
let dir = tempfile::tempdir().unwrap();
let devcontainer_dir = dir.path().join(".devcontainer");
fs::create_dir(&devcontainer_dir).unwrap();
fs::write(devcontainer_dir.join("Dockerfile"), "FROM alpine\n").unwrap();
fs::write(devcontainer_dir.join("devcontainer.json"), "{}").unwrap();
let tar = context_tar(&devcontainer_dir).await;
let mut archive = tar::Archive::new(tar.as_slice());
let names = archive
.entries()
.unwrap()
.map(|entry| entry.unwrap().path().unwrap().display().to_string())
.collect::<Vec<_>>();
assert!(names.iter().any(|name| name.ends_with("Dockerfile")));
assert!(names.iter().any(|name| name.ends_with("devcontainer.json")));
}
#[tokio::test]
async fn tar_context_keeps_the_executable_bit() {
let dir = tempfile::tempdir().unwrap();
let script = dir.path().join("setup.sh");
fs::write(&script, "#!/bin/sh\n").unwrap();
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
fs::set_permissions(&script, fs::Permissions::from_mode(0o755)).unwrap();
}
let tar = context_tar(dir.path()).await;
let mut archive = tar::Archive::new(tar.as_slice());
let mode = archive
.entries()
.unwrap()
.map(|entry| entry.unwrap())
.find(|entry| entry.path().unwrap().ends_with("setup.sh"))
.unwrap()
.header()
.mode()
.unwrap();
assert_eq!(mode & 0o111, 0o111);
}
#[tokio::test]
async fn tar_context_is_sent_in_chunks() {
let dir = tempfile::tempdir().unwrap();
fs::write(
dir.path().join("big.bin"),
vec![0_u8; CONTEXT_CHUNK_SIZE * 3],
)
.unwrap();
let mut chunks = tar_directory_stream(dir.path());
let mut count = 0;
while let Some(chunk) = chunks.next().await {
assert!(chunk.unwrap().len() <= CONTEXT_CHUNK_SIZE);
count += 1;
}
assert!(
count > 1,
"the context should be streamed, not buffered in one piece"
);
}
}
+329
View File
@@ -0,0 +1,329 @@
//! Parsed representation of a `devcontainer.json` and its lifecycle: naming,
//! build, startup and hooks.
use std::{
collections::HashMap,
path::{Path, PathBuf},
time::{Duration, SystemTime, UNIX_EPOCH},
};
use bollard::models::{ContainerCreateBody, HostConfig};
use tracing::info;
use crate::{
consts::DEFAULT_IMAGE_NAME, container::Container, errors::ContainerError,
runtime::ContainerRuntime,
};
#[derive(Debug)]
pub struct DevContainer {
pub image: Option<String>,
pub container_file_path: Option<PathBuf>,
pub name: Option<String>,
pub build_args: HashMap<String, String>,
pub container_env: HashMap<String, String>,
pub workspace_folder: Option<String>,
pub post_create_command: Option<String>,
pub post_start_command: Option<String>,
pub remote_user: Option<String>,
}
impl DevContainer {
/// Workspace folder inside the container, `/workspace` by default.
pub fn workspace_folder(&self) -> String {
self.workspace_folder
.clone()
.unwrap_or_else(|| "/workspace".to_string())
}
/// Tag of the image built locally from the Dockerfile.
fn local_image_tag(&self, id: &str) -> String {
format!("devcontainer-rs/{}:{}", self.sanitized_name(), id)
}
/// Name of this sandbox's container.
fn container_name(&self, id: &str) -> String {
format!("devcontainer-rs-{}-{}", self.sanitized_name(), id)
}
/// Devcontainer name, sanitized to be usable as an image/container name.
fn sanitized_name(&self) -> String {
sanitize(self.name.as_deref().unwrap_or("devcontainer"))
}
/// Builds the devcontainer image as `image_tag`.
async fn build(
&self,
runtime: &ContainerRuntime,
container_file_path: &Path,
image_tag: &str,
) -> Result<(), ContainerError> {
let context = container_file_path.parent().ok_or_else(|| {
ContainerError::Unexpected(String::from(
"the devcontainer file path has no parent directory",
))
})?;
let dockerfile = container_file_path
.file_name()
.and_then(|name| name.to_str())
.ok_or_else(|| {
ContainerError::Unexpected(String::from(
"the devcontainer file name is not valid UTF-8",
))
})?;
runtime
.build_image(context, dockerfile, image_tag, &self.build_args)
.await
}
pub async fn up(
&self,
runtime: &ContainerRuntime,
workspace_dir: &Path,
) -> Result<Container, ContainerError> {
let id = unique_id();
let image = self
.image
.clone()
.unwrap_or_else(|| self.local_image_tag(&id));
let owns_image = self.image.is_none();
let container_name = self.container_name(&id);
let network = format!("{container_name}-net");
let workspace_folder = self.workspace_folder();
info!(image = %image, "Starting sandbox container");
let setup = async {
if owns_image {
let container_file_path = self.container_file_path.as_ref().ok_or_else(|| {
ContainerError::Unexpected(String::from(
"the devcontainer has neither an image nor a container file",
))
})?;
self.build(runtime, container_file_path, &image).await?;
} else {
runtime.pull_image(&image).await?;
}
runtime.create_network(&network).await?;
let body = ContainerCreateBody {
image: Some(image.clone()),
cmd: Some(vec![String::from("sleep"), String::from("infinity")]),
user: self.remote_user.clone(),
env: Some(
self.container_env
.iter()
.map(|(key, value)| format!("{key}={value}"))
.collect(),
),
working_dir: Some(workspace_folder.clone()),
host_config: Some(HostConfig {
network_mode: Some(network.clone()),
..Default::default()
}),
..Default::default()
};
runtime.create_container(&container_name, body).await?;
runtime.start_container(&container_name).await?;
runtime
.upload_directory(&container_name, &workspace_folder, workspace_dir)
.await?;
Ok::<(), ContainerError>(())
}
.await;
if let Err(err) = setup {
cleanup_failed_startup(
runtime,
&container_name,
&network,
owns_image.then_some(image.as_str()),
)
.await;
return Err(err);
}
let container = Container::new(
runtime.clone(),
container_name.clone(),
workspace_folder,
self.remote_user.clone(),
Some(network.clone()),
owns_image.then_some(image.clone()),
);
let post_setup = async {
self.run_hooks(&container, runtime.timeout()).await?;
runtime
.disconnect_network(&network, &container_name)
.await?;
Ok::<(), ContainerError>(())
}
.await;
if let Err(err) = post_setup {
cleanup_failed_startup(
runtime,
&container_name,
&network,
owns_image.then_some(image.as_str()),
)
.await;
return Err(err);
}
Ok(container)
}
async fn run_hooks(
&self,
container: &Container,
timeout: Duration,
) -> Result<(), ContainerError> {
for command in [&self.post_create_command, &self.post_start_command]
.into_iter()
.flatten()
{
let output = container
.exec_with_timeout(&["sh", "-c", command], timeout)
.await?;
if !output.success() {
return Err(ContainerError::Unexpected(format!(
"hook `{command}` failed with status {}: {}",
output.status,
output.stderr.trim()
)));
}
}
Ok(())
}
}
async fn cleanup_failed_startup(
runtime: &ContainerRuntime,
container: &str,
network: &str,
owned_image: Option<&str>,
) {
let _ = runtime.remove_container(container).await;
let _ = runtime.remove_network(network).await;
if let Some(image) = owned_image {
let _ = runtime.remove_image(image).await;
}
}
impl Default for DevContainer {
fn default() -> Self {
Self {
name: Some("generic-devcontainer".to_string()),
image: Some(DEFAULT_IMAGE_NAME.to_string()),
post_create_command: None,
post_start_command: None,
build_args: HashMap::new(),
container_env: HashMap::new(),
container_file_path: None,
remote_user: None,
workspace_folder: None,
}
}
}
/// Sanitizes a string so it can be used as a docker image/container name.
fn sanitize(input: &str) -> String {
let sanitized: String = input
.chars()
.map(|c| {
if c.is_ascii_alphanumeric() || c == '-' || c == '_' || c == '.' {
c.to_ascii_lowercase()
} else {
'-'
}
})
.collect();
let trimmed = sanitized.trim_matches(|c| c == '-' || c == '.' || c == '_');
if trimmed.is_empty() {
"devcontainer".to_string()
} else {
trimmed.to_string()
}
}
/// Identifier unique to a sandbox run, combining the process id and a timestamp.
fn unique_id() -> String {
let nanos = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|duration| duration.as_nanos())
.unwrap_or(0);
format!("{}-{}", std::process::id(), nanos)
}
#[cfg(test)]
mod tests {
use super::*;
use std::fs;
fn devcontainer(dir: &Path) -> DevContainer {
let devcontainer_path = dir.join("devcontainer.json");
let dockerfile_path = dir.join("Dockerfile");
fs::write(&dockerfile_path, "FROM alpine\n").unwrap();
fs::write(
&devcontainer_path,
r#"{
"name": "My Project",
"build": {
"dockerfile": "Dockerfile",
"args": { "VERSION": "1" }
},
"workspaceFolder": "/workspaces/my-project",
"containerEnv": { "RUST_LOG": "debug" },
"remoteUser": "dev"
}"#,
)
.unwrap();
let runtime = tokio::runtime::Runtime::new().unwrap();
runtime.block_on(crate::parse(&devcontainer_path)).unwrap()
}
#[test]
fn local_image_tag_is_sanitized() {
let dir = tempfile::tempdir().unwrap();
let dc = devcontainer(dir.path());
assert_eq!(dc.local_image_tag("abc"), "devcontainer-rs/my-project:abc");
}
#[test]
fn container_name_is_sanitized() {
let dir = tempfile::tempdir().unwrap();
let dc = devcontainer(dir.path());
assert_eq!(dc.container_name("abc"), "devcontainer-rs-my-project-abc");
}
#[test]
fn unique_ids_differ() {
assert_ne!(unique_id(), unique_id());
}
#[test]
fn default_devcontainer_pulls_a_base_image() {
let dc = DevContainer::default();
assert_eq!(dc.image.as_deref(), Some("debian:stable-slim"));
assert_eq!(dc.container_file_path, None);
assert_eq!(dc.workspace_folder(), "/workspace");
}
}
+54
View File
@@ -0,0 +1,54 @@
//! Crate errors: container daemon failures and `devcontainer.json` parsing failures.
use std::{path::PathBuf, time::Duration};
use bollard::errors::Error as BollardError;
#[derive(Debug, thiserror::Error)]
pub enum ContainerError {
/// The daemon is unreachable, or rejected a request.
#[error("container daemon request failed: {source}")]
Request {
#[from]
source: BollardError,
},
/// An operation exceeded its timeout.
#[error("`{operation}` timed out after {timeout:?}")]
Timeout {
operation: String,
timeout: Duration,
},
/// The image build failed.
#[error("image build failed: {message}")]
Build { message: String },
/// The daemon replied with something other than what was expected.
#[error("{0}")]
Unexpected(String),
}
#[derive(Debug, thiserror::Error)]
pub enum ParseError {
#[error("failed to read devcontainer file `{path}`: {source}")]
Read {
path: PathBuf,
source: std::io::Error,
},
#[error("invalid devcontainer JSON in `{path}`: {source}")]
Json {
path: PathBuf,
source: serde_json::Error,
},
#[error("container file `{0}` does not exist or is not a regular file")]
ContainerFileNotFound(PathBuf),
#[error("devcontainer file `{0}` declares neither `image` nor `build`")]
MissingImageOrBuild(PathBuf),
#[error("the devcontainer file path has no parent directory: `{0}`")]
InvalidDevContainerPath(PathBuf),
}
+16
View File
@@ -0,0 +1,16 @@
//! Result of a command run in a container.
/// Result of a command run in a container.
#[derive(Debug, Clone)]
pub struct ExecOutput {
/// Exit code, or `-1` if the process was terminated by a signal.
pub status: i32,
pub stdout: String,
pub stderr: String,
}
impl ExecOutput {
pub fn success(&self) -> bool {
self.status == 0
}
}
+42
View File
@@ -0,0 +1,42 @@
//! Container lifecycle primitives for a parsed [`DevContainer`].
//!
//! This crate drives the container daemon API to prepare the devcontainer image
//! (built from a Dockerfile, or pulled from a registry), start a container and
//! copy the workspace into it, run the `postCreateCommand` / `postStartCommand`
//! hooks, and run commands inside the running container.
//!
//! The daemon is reached through its **socket** (the host's, mounted into the
//! Steep container): [`ContainerRuntime::connect`] follows `DOCKER_HOST` like the
//! `docker` CLI does, and falls back to the local socket. No runtime binary is
//! therefore required in the image, and `podman` works the same way as long as it
//! exposes its Docker-compatible socket.
//!
//! # Isolation
//!
//! Each sandbox has its own image tag, its own container and its own network. The
//! container starts attached to that network so that the `postCreateCommand` /
//! `postStartCommand` hooks can fetch dependencies (e.g. `npm install`); once the
//! hooks have run, the container is disconnected from the network for the rest of
//! its lifetime. Every command is bounded by a timeout.
//!
//! The `devcontainer.json` `runArgs` are ignored: they come from the repository,
//! hence from an untrusted pull request, and could attach the container to another
//! network or grant it privileges that would defeat this isolation.
mod consts;
mod container;
mod context;
mod devcontainer;
mod errors;
mod exec;
mod path;
mod runtime;
mod schema;
pub use container::Container;
pub use devcontainer::DevContainer;
pub use errors::{ContainerError, ParseError};
pub use exec::ExecOutput;
pub use path::normalize;
pub use runtime::ContainerRuntime;
pub use schema::parse;
+40
View File
@@ -0,0 +1,40 @@
//! Lexical path normalization, without accessing the filesystem.
use std::path::{Path, PathBuf};
/// Lexically normalizes a path, resolving `.` and `..` without touching the
/// filesystem. Returns `None` if the path escapes its root.
pub fn normalize(path: &Path) -> Option<PathBuf> {
use std::path::Component;
let mut out = PathBuf::new();
for component in path.components() {
match component {
Component::RootDir => out.push("/"),
Component::CurDir => {}
Component::ParentDir => {
if !out.pop() {
return None;
}
}
Component::Normal(part) => out.push(part),
Component::Prefix(_) => return None,
}
}
Some(out)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn normalize_rejects_escaping_paths() {
assert_eq!(
normalize(Path::new("/workspaces/project/src/../main.rs")),
Some(PathBuf::from("/workspaces/project/main.rs"))
);
assert_eq!(normalize(Path::new("/workspaces/../../etc/passwd")), None);
}
}
+405
View File
@@ -0,0 +1,405 @@
//! Container daemon API client and lifecycle operations.
use std::{collections::HashMap, path::Path, time::Duration};
use bollard::{
Docker, body_try_stream,
container::LogOutput,
errors::Error as BollardError,
exec::{CreateExecOptions, StartExecOptions, StartExecResults},
models::{BuildInfo, ContainerCreateBody, NetworkCreateRequest, NetworkDisconnectRequest},
query_parameters::{
BuildImageOptions, CreateContainerOptions, CreateImageOptionsBuilder,
RemoveContainerOptions, RemoveImageOptions, StartContainerOptions, StopContainerOptions,
UploadToContainerOptions,
},
};
use futures_util::StreamExt;
use tracing::info;
use crate::{
consts::{DEFAULT_COMMAND_TIMEOUT, DEFAULT_ENDPOINT, EXIT_CODE_ATTEMPTS, EXIT_CODE_DELAY},
context::tar_directory_stream,
errors::ContainerError,
exec::ExecOutput,
};
/// A container daemon API client.
#[derive(Debug, Clone)]
pub struct ContainerRuntime {
docker: Docker,
endpoint: String,
timeout: Duration,
}
impl ContainerRuntime {
/// Connects to the daemon designated by `DOCKER_HOST`, or to the default local socket.
pub fn connect() -> Result<Self, ContainerError> {
let endpoint =
std::env::var("DOCKER_HOST").unwrap_or_else(|_| String::from(DEFAULT_ENDPOINT));
Ok(Self {
docker: Docker::connect_with_defaults()?,
endpoint,
timeout: DEFAULT_COMMAND_TIMEOUT,
})
}
/// Replaces the timeout applied to build/run/stop/remove operations.
pub fn with_timeout(mut self, timeout: Duration) -> Self {
self.timeout = timeout;
self
}
/// Daemon endpoint, for logging.
pub fn endpoint(&self) -> &str {
&self.endpoint
}
pub fn timeout(&self) -> Duration {
self.timeout
}
/// Checks that the daemon is reachable and responding.
pub async fn available(&self) -> bool {
self.docker.ping().await.is_ok()
}
/// Runs a daemon request, applying the command timeout.
async fn request<T>(
&self,
operation: &str,
request: impl Future<Output = Result<T, BollardError>>,
) -> Result<T, ContainerError> {
match tokio::time::timeout(self.timeout, request).await {
Ok(Ok(value)) => Ok(value),
Ok(Err(source)) => Err(ContainerError::Request { source }),
Err(_) => Err(ContainerError::Timeout {
operation: String::from(operation),
timeout: self.timeout,
}),
}
}
/// Builds the image `tag` from the `context_dir` context.
pub(crate) async fn build_image(
&self,
context_dir: &Path,
dockerfile: &str,
tag: &str,
build_args: &HashMap<String, String>,
) -> Result<(), ContainerError> {
let options = BuildImageOptions {
dockerfile: String::from(dockerfile),
t: Some(String::from(tag)),
buildargs: Some(build_args.clone()),
rm: true,
..Default::default()
};
let context = tar_directory_stream(context_dir);
let mut stream = self
.docker
.build_image(options, None, Some(body_try_stream(context)));
// The stream carries the build progress: the last error reported by the
// daemon makes the operation fail.
let consume = async {
let mut failure = None;
while let Some(info) = stream.next().await {
let info: BuildInfo = info?;
if let Some(message) = info.error_detail.and_then(|detail| detail.message) {
failure = Some(message);
}
}
Ok::<_, BollardError>(failure)
};
let failure = match tokio::time::timeout(self.timeout, consume).await {
Ok(Ok(failure)) => failure,
Ok(Err(source)) => return Err(ContainerError::Request { source }),
Err(_) => {
return Err(ContainerError::Timeout {
operation: format!("build image `{tag}`"),
timeout: self.timeout,
});
}
};
match failure {
Some(message) => Err(ContainerError::Build { message }),
None => Ok(()),
}
}
pub(crate) async fn pull_image(&self, image: &str) -> Result<(), ContainerError> {
info!("Pulling image `{image}`");
let options = CreateImageOptionsBuilder::default()
.from_image(image)
.build();
let mut stream = self.docker.create_image(Some(options), None, None);
let consume = async {
while let Some(info) = stream.next().await {
info?;
}
Ok::<_, BollardError>(())
};
match tokio::time::timeout(self.timeout, consume).await {
Ok(Ok(())) => Ok(()),
Ok(Err(source)) => Err(ContainerError::Request { source }),
Err(_) => Err(ContainerError::Timeout {
operation: format!("pull image `{image}`"),
timeout: self.timeout,
}),
}
}
/// Creates a container without starting it.
pub(crate) async fn create_container(
&self,
name: &str,
body: ContainerCreateBody,
) -> Result<(), ContainerError> {
let options = CreateContainerOptions {
name: Some(String::from(name)),
..Default::default()
};
self.request(
"create container",
self.docker.create_container(Some(options), body),
)
.await?;
Ok(())
}
pub(crate) async fn upload_directory(
&self,
container: &str,
destination: &str,
directory: &Path,
) -> Result<(), ContainerError> {
let options = UploadToContainerOptions {
path: String::from(destination),
..Default::default()
};
self.request(
"upload workspace",
self.docker.upload_to_container(
container,
Some(options),
body_try_stream(tar_directory_stream(directory)),
),
)
.await?;
Ok(())
}
pub(crate) async fn start_container(&self, name: &str) -> Result<(), ContainerError> {
self.request(
"start container",
self.docker
.start_container(name, None::<StartContainerOptions>),
)
.await?;
Ok(())
}
/// Runs a command in a container and returns its output.
pub(crate) async fn exec(
&self,
container: &str,
cmd: &[&str],
user: Option<&str>,
timeout: Duration,
) -> Result<ExecOutput, ContainerError> {
let config = CreateExecOptions {
cmd: Some(cmd.iter().map(|arg| String::from(*arg)).collect()),
user: user.map(String::from),
attach_stdout: Some(true),
attach_stderr: Some(true),
..Default::default()
};
let created = self
.request("create exec", self.docker.create_exec(container, config))
.await?;
let started = self
.request(
"start exec",
self.docker
.start_exec(&created.id, None::<StartExecOptions>),
)
.await?;
let StartExecResults::Attached { mut output, .. } = started else {
return Err(ContainerError::Unexpected(String::from(
"the daemon detached an exec that was not requested as detached",
)));
};
let mut stdout = String::new();
let mut stderr = String::new();
// The timeout covers the execution of the command itself, not just its
// setup.
let collect = async {
while let Some(message) = output.next().await {
match message? {
LogOutput::StdOut { message } | LogOutput::Console { message } => {
stdout.push_str(&String::from_utf8_lossy(&message));
}
LogOutput::StdErr { message } => {
stderr.push_str(&String::from_utf8_lossy(&message));
}
LogOutput::StdIn { .. } => {}
}
}
Ok::<_, BollardError>(())
};
match tokio::time::timeout(timeout, collect).await {
Ok(Ok(())) => {}
Ok(Err(source)) => return Err(ContainerError::Request { source }),
Err(_) => {
// The process keeps running in the container: left alone, it would
// consume CPU and memory and could still modify the workspace for the
// whole lifetime of the sandbox. The API offers no way to kill an exec,
// so we stop the container that hosts it.
let _ = self.stop_container(container).await;
return Err(ContainerError::Timeout {
operation: format!("exec {}", cmd.join(" ")),
timeout,
});
}
}
// The exit code is not necessarily published by the time the stream closes:
// we give the daemon time to fill it in, without blocking indefinitely.
// Without this, a successful command can be reported as failed (code -1) and
// the tool returns an error to the model instead of the content.
let mut inspected = self
.request("inspect exec", self.docker.inspect_exec(&created.id))
.await?;
for _ in 0..EXIT_CODE_ATTEMPTS {
if !inspected.running.unwrap_or(false) {
break;
}
tokio::time::sleep(EXIT_CODE_DELAY).await;
inspected = self
.request("inspect exec", self.docker.inspect_exec(&created.id))
.await?;
}
Ok(ExecOutput {
status: inspected.exit_code.unwrap_or(-1) as i32,
stdout,
stderr,
})
}
pub(crate) async fn stop_container(&self, name: &str) -> Result<(), ContainerError> {
let options = StopContainerOptions {
t: Some(1),
..Default::default()
};
self.request(
"stop container",
self.docker.stop_container(name, Some(options)),
)
.await?;
Ok(())
}
/// Removes a container, its anonymous volumes and the process running in it.
pub(crate) async fn remove_container(&self, name: &str) -> Result<(), ContainerError> {
let options = RemoveContainerOptions {
force: true,
v: true,
..Default::default()
};
self.request(
"remove container",
self.docker.remove_container(name, Some(options)),
)
.await?;
Ok(())
}
pub(crate) async fn remove_image(&self, tag: &str) -> Result<(), ContainerError> {
let options = RemoveImageOptions {
force: true,
..Default::default()
};
self.request(
"remove image",
self.docker.remove_image(tag, Some(options), None),
)
.await?;
Ok(())
}
/// Creates a sandbox's dedicated network.
pub(crate) async fn create_network(&self, name: &str) -> Result<(), ContainerError> {
let request = NetworkCreateRequest {
name: String::from(name),
..Default::default()
};
self.request("create network", self.docker.create_network(request))
.await?;
Ok(())
}
/// Detaches a container from a network, which cuts off its connectivity.
pub(crate) async fn disconnect_network(
&self,
network: &str,
container: &str,
) -> Result<(), ContainerError> {
let request = NetworkDisconnectRequest {
container: String::from(container),
..Default::default()
};
self.request(
"disconnect network",
self.docker.disconnect_network(network, request),
)
.await?;
Ok(())
}
pub(crate) async fn remove_network(&self, name: &str) -> Result<(), ContainerError> {
self.request("remove network", self.docker.remove_network(name))
.await?;
Ok(())
}
}
+302
View File
@@ -0,0 +1,302 @@
//! `devcontainer.json` schema, file parsing and resolution of `${localEnv:…}`
//! references.
use std::{
collections::HashMap,
path::{Path, PathBuf},
};
use serde::Deserialize;
use crate::{devcontainer::DevContainer, errors::ParseError};
#[derive(Debug, Deserialize)]
struct DevContainerBuildSchema {
#[serde(default)]
dockerfile: String,
#[serde(default)]
args: HashMap<String, String>,
}
#[derive(Debug, Deserialize)]
struct DevContainerSchema {
#[serde(default)]
name: Option<String>,
/// Base image pulled from the registry, alternative to `build`.
#[serde(default)]
image: Option<String>,
#[serde(default)]
build: Option<DevContainerBuildSchema>,
#[serde(rename = "workspaceFolder", default)]
workspace_folder: Option<String>,
#[serde(rename = "containerEnv", default)]
container_env: HashMap<String, String>,
#[serde(rename = "postCreateCommand", default)]
post_create_command: Option<String>,
#[serde(rename = "postStartCommand", default)]
post_start_command: Option<String>,
#[serde(rename = "remoteUser", default)]
remote_user: Option<String>,
}
impl TryFrom<(DevContainerSchema, PathBuf)> for DevContainer {
type Error = ParseError;
fn try_from(
(schema, devcontainer_path): (DevContainerSchema, PathBuf),
) -> Result<Self, Self::Error> {
let base_dir = devcontainer_path
.parent()
.ok_or_else(|| ParseError::InvalidDevContainerPath(devcontainer_path.clone()))?;
let DevContainerSchema {
name,
image,
build,
workspace_folder,
container_env,
post_create_command,
post_start_command,
remote_user,
} = schema;
let (image, container_file_path, build_args) = match (image, build) {
(Some(image), _) => (Some(substitute_local_env(&image)), None, HashMap::new()),
(None, Some(build)) => {
let container_file_path = base_dir.join(&build.dockerfile);
if !container_file_path.is_file() {
return Err(ParseError::ContainerFileNotFound(container_file_path));
}
let build_args = build
.args
.into_iter()
.map(|(k, v)| (k, substitute_local_env(&v)))
.collect();
(None, Some(container_file_path), build_args)
}
(None, None) => return Err(ParseError::MissingImageOrBuild(devcontainer_path)),
};
let container_env = container_env
.into_iter()
.map(|(k, v)| (k, substitute_local_env(&v)))
.collect();
Ok(Self {
image,
container_file_path,
name,
build_args,
container_env,
workspace_folder,
post_create_command,
post_start_command,
remote_user,
})
}
}
/// Resolves the `${localEnv:VAR}` and `${localEnv:VAR:default}` references of a
/// `devcontainer.json`.
///
/// The specification says `${localEnv:VAR}` is read from the client's environment;
/// here the `devcontainer.json` comes from a **pull request**, hence from untrusted
/// code: reading the process environment would let it grab `GITEA_TOKEN`,
/// `OPEN_ROUTER_API_KEY` or any other Steep secret and exfiltrate it through a
/// `build.args`, a `containerEnv`, its Dockerfile or a hook. The environment is
/// therefore **never** consulted: only the default value is used, and a reference
/// without a default becomes an empty string.
fn substitute_local_env(input: &str) -> String {
const PREFIX: &str = "${localEnv:";
let mut out = String::with_capacity(input.len());
let mut rest = input;
while let Some(start) = rest.find(PREFIX) {
out.push_str(&rest[..start]);
let after = &rest[start + PREFIX.len()..];
match after.find('}') {
Some(end) => {
let inner = &after[..end];
// The key is read to delimit the reference, not to resolve it.
let (_key, default) = match inner.split_once(':') {
Some((key, default)) => (key, Some(default)),
None => (inner, None),
};
out.push_str(default.unwrap_or(""));
rest = &after[end + 1..];
}
None => {
out.push_str(PREFIX);
rest = after;
}
}
}
out.push_str(rest);
out
}
pub async fn parse(path: impl AsRef<Path>) -> Result<DevContainer, ParseError> {
let path = path.as_ref().to_path_buf();
let contents = tokio::fs::read_to_string(&path)
.await
.map_err(|source| ParseError::Read {
path: path.clone(),
source,
})?;
let schema = serde_json::from_str::<DevContainerSchema>(&contents).map_err(|source| {
ParseError::Json {
path: path.clone(),
source,
}
})?;
DevContainer::try_from((schema, path))
}
#[cfg(test)]
mod tests {
use super::*;
use std::fs;
#[tokio::test]
async fn parses_devcontainer_file() {
let dir = tempfile::tempdir().unwrap();
let devcontainer_path = dir.path().join("devcontainer.json");
let dockerfile_path = dir.path().join("Dockerfile");
fs::write(&dockerfile_path, "FROM alpine\n").unwrap();
fs::write(
&devcontainer_path,
r#"{
"name": "test",
"build": {
"dockerfile": "Dockerfile",
"args": {
"VERSION": "1"
}
},
"workspaceFolder": "/workspace",
"containerEnv": {
"RUST_LOG": "debug"
},
"remoteUser": "dev"
}"#,
)
.unwrap();
let config = parse(&devcontainer_path).await.unwrap();
assert_eq!(config.name.as_deref(), Some("test"));
assert_eq!(config.image, None);
assert_eq!(config.container_file_path, Some(dockerfile_path));
assert_eq!(config.build_args.get("VERSION").unwrap(), "1");
assert_eq!(config.container_env.get("RUST_LOG").unwrap(), "debug");
assert_eq!(config.workspace_folder.as_deref(), Some("/workspace"));
assert_eq!(config.remote_user.as_deref(), Some("dev"));
}
#[tokio::test]
async fn parses_image_based_devcontainer() {
let dir = tempfile::tempdir().unwrap();
let devcontainer_path = dir.path().join("devcontainer.json");
fs::write(
&devcontainer_path,
r#"{
"name": "image-based",
"image": "mcr.microsoft.com/devcontainers/base:debian"
}"#,
)
.unwrap();
let config = parse(&devcontainer_path).await.unwrap();
assert_eq!(
config.image.as_deref(),
Some("mcr.microsoft.com/devcontainers/base:debian")
);
assert_eq!(config.container_file_path, None);
}
#[tokio::test]
async fn ignores_run_args() {
let dir = tempfile::tempdir().unwrap();
let devcontainer_path = dir.path().join("devcontainer.json");
// `runArgs` comes from an untrusted pull request: it must be accepted by
// parsing but never carried over into the devcontainer.
fs::write(
&devcontainer_path,
r#"{
"image": "alpine",
"runArgs": ["--network", "host"]
}"#,
)
.unwrap();
let config = parse(&devcontainer_path).await.unwrap();
assert_eq!(config.image.as_deref(), Some("alpine"));
}
#[tokio::test]
async fn rejects_devcontainer_without_image_or_build() {
let dir = tempfile::tempdir().unwrap();
let devcontainer_path = dir.path().join("devcontainer.json");
fs::write(&devcontainer_path, r#"{ "name": "empty" }"#).unwrap();
let err = parse(&devcontainer_path).await.unwrap_err();
assert!(matches!(err, ParseError::MissingImageOrBuild(_)));
}
#[test]
fn local_env_references_are_never_read_from_the_environment() {
unsafe { std::env::set_var("DEVCONTAINER_TEST_SECRET", "s3cret") };
// The `devcontainer.json` comes from a pull request: resolving the reference
// from Steep's environment would expose its tokens to it.
assert_eq!(
substitute_local_env("${localEnv:DEVCONTAINER_TEST_SECRET}"),
""
);
assert_eq!(
substitute_local_env("token=${localEnv:DEVCONTAINER_TEST_SECRET}"),
"token="
);
}
#[test]
fn local_env_defaults_are_used() {
assert_eq!(
substitute_local_env("${localEnv:DEVCONTAINER_TEST_MISSING:fallback}"),
"fallback"
);
assert_eq!(
substitute_local_env("${localEnv:DEVCONTAINER_TEST_MISSING}"),
""
);
assert_eq!(substitute_local_env("uid=${localEnv:UID:1000}"), "uid=1000");
assert_eq!(
substitute_local_env("no variables here"),
"no variables here"
);
}
}
+34
View File
@@ -0,0 +1,34 @@
[package]
name = "steep-server"
version = "1.3.0"
edition = "2024"
publish = false
[dependencies]
reqwest = { workspace = true }
tokio = { workspace = true }
tokio-util = { workspace = true }
tokio-stream = { workspace = true, features = ["sync"] }
futures-util = { workspace = true }
serde_json = { workspace = true }
serde = { workspace = true }
sentry = { workspace = true }
sentry-anyhow = { workspace = true }
dotenvy = { workspace = true }
tower = { workspace = true }
tower-http = { workspace = true }
tracing = { workspace = true }
tracing-subscriber = { workspace = true }
axum = { workspace = true }
anyhow = { workspace = true }
thiserror = { workspace = true }
ring = { workspace = true }
hex = { workspace = true }
metrics = { workspace = true }
metrics-exporter-prometheus = { workspace = true }
devcontainer-rs = { path = "../devcontainer-rs" }
tempfile = { workspace = true }
rustls = { workspace = true }
toasty = { workspace = true }
toasty-cli = { workspace = true }
jiff = { workspace = true }
+250
View File
@@ -0,0 +1,250 @@
use axum::body::{Bytes, to_bytes};
use axum::extract::{FromRef, FromRequest, Path, State};
use axum::http::Request;
use axum::response::IntoResponse;
use axum::response::sse::{Event, KeepAlive, Sse};
use axum::routing::{get, post};
use axum::{Json, Router};
use futures_util::stream::{self, Stream, StreamExt};
use reqwest::StatusCode;
use ring::hmac;
use sentry::integrations::tower::{NewSentryLayer, SentryHttpLayer};
use serde_json::{Value, json};
use tokio_stream::wrappers::{BroadcastStream, errors::BroadcastStreamRecvError};
use tower::ServiceBuilder;
use tower_http::services::ServeDir;
use tower_http::trace::TraceLayer;
use tracing::{info, instrument, warn};
use crate::consts::{GITEA_EVENT_TYPE_HEADER_NAME, GITEA_SIG_HEADER_NAME, MAX_WEBHOOK_BODY_SIZE};
use crate::errors::AppError;
use crate::event_bus::JobEvent;
use crate::gitea::WebhookType;
use crate::metrics;
use crate::schemas::JobStatus;
use crate::state::AppState;
pub async fn start(app_state: AppState) -> anyhow::Result<()> {
let http_port = app_state.config.http_port;
let shutdown = app_state.shutdown.clone();
// The dashboard is served from a static directory; the API routes below take
// precedence (the directory is only the fallback).
let front_dir = std::env::var("FRONT_DIR").unwrap_or_else(|_| String::from("front"));
if !std::path::Path::new(&front_dir).is_dir() {
warn!(dir = %front_dir, "front directory not found, the dashboard will 404");
}
let dashboard = ServeDir::new(&front_dir);
let app = Router::new()
.route("/webhook", post(webhook))
.route("/reviews", get(reviews_list))
.route("/jobs", get(jobs_list))
.route("/events", get(lifecycle_events))
.route("/jobs/{id}/events", get(job_events))
.fallback_service(dashboard)
.layer(
ServiceBuilder::new()
.layer(NewSentryLayer::<Request<_>>::new_from_top())
.layer(SentryHttpLayer::new())
.layer(TraceLayer::new_for_http()),
)
.with_state(app_state);
let listener = tokio::net::TcpListener::bind(format!("0.0.0.0:{}", http_port)).await?;
info!("Listening API on port {}", http_port);
axum::serve(listener, app)
.with_graceful_shutdown(async move { shutdown.cancelled().await })
.await
.map_err(anyhow::Error::from)
.inspect(|_| info!("API shutting down complete"))?;
Ok(())
}
async fn reviews_list(State(app_state): State<AppState>) -> Result<impl IntoResponse, AppError> {
let reviews = app_state.reviews.list().await?;
Ok(Json(json!({
"reviews": reviews
})))
}
async fn jobs_list(State(app_state): State<AppState>) -> Result<impl IntoResponse, AppError> {
let pending = app_state.queue.list(JobStatus::Pending).await?;
let processing = app_state.queue.list(JobStatus::Processing).await?;
Ok(Json(json!({
"pending": pending,
"processing": processing,
})))
}
/// SSE stream of the global lifecycle events (the job list).
///
/// The stream is closed as soon as the graceful shutdown starts: a live SSE
/// connection would otherwise keep the server from stopping.
async fn lifecycle_events(
State(app_state): State<AppState>,
) -> Sse<impl Stream<Item = Result<Event, axum::Error>>> {
let rx = app_state.event_bus.subscribe_lifecycle();
let shutdown = app_state.shutdown.cancelled_owned();
let stream = BroadcastStream::new(rx).map(|result| match result {
Ok(event) => sse_json("lifecycle", &event),
Err(BroadcastStreamRecvError::Lagged(skipped)) => sse_gap(skipped),
});
Sse::new(stream.take_until(shutdown)).keep_alive(KeepAlive::default())
}
async fn job_events(
State(app_state): State<AppState>,
Path(id): Path<u64>,
) -> Result<Sse<impl Stream<Item = Result<Event, axum::Error>>>, AppError> {
let subscription = app_state
.event_bus
.subscribe_job(id)
.ok_or(AppError::JobNotFoundErr(id))?;
let shutdown = app_state.shutdown.cancelled_owned();
let history = subscription
.history
.into_iter()
.map(|event| sse_json("progress", &event));
let live = BroadcastStream::new(subscription.live).map(|result| match result {
Ok(event) => sse_json("progress", &event),
Err(BroadcastStreamRecvError::Lagged(skipped)) => sse_gap(skipped),
});
let stream = stream::iter(history).chain(live).take_until(shutdown);
Ok(Sse::new(stream).keep_alive(KeepAlive::default()))
}
fn sse_json<T: serde::Serialize>(name: &'static str, payload: &T) -> Result<Event, axum::Error> {
Event::default().event(name).json_data(payload)
}
fn sse_gap(skipped: u64) -> Result<Event, axum::Error> {
Ok(Event::default().event("gap").data(skipped.to_string()))
}
#[instrument(skip(app_state), fields(webhook_type), err)]
async fn webhook(
State(app_state): State<AppState>,
WebhookExtract(wb): WebhookExtract,
) -> Result<impl IntoResponse, AppError> {
tracing::Span::current().record("webhook_type", tracing::field::debug(&wb));
let event_type = wb.event_type_str();
metrics::webhook_received(event_type);
let event_id = wb.event_id();
match is_known(&app_state, event_id).await {
Ok(true) => {
metrics::webhook_duplicate(event_type);
return Err(AppError::AlreadyProcessedErr);
}
Ok(false) => {}
Err(err) => {
metrics::webhook_enqueue_failed(event_type);
return Err(AppError::EnqueueErr(anyhow::Error::from(err)));
}
}
let job = match app_state.queue.push(wb).await {
Ok(job) => job,
Err(err) => {
metrics::webhook_enqueue_failed(event_type);
return Err(AppError::EnqueueErr(anyhow::Error::from(err)));
}
};
app_state.event_bus.publish(JobEvent::Queued {
id: job.id,
repo: job.repo,
pr: job.pr,
});
Ok((StatusCode::CREATED, "Task started"))
}
/// Whether an event is already queued, or already reviewed.
async fn is_known(app_state: &AppState, event_id: u64) -> toasty::Result<bool> {
Ok(app_state.queue.is_queued(event_id).await?
|| app_state.reviews.is_reviewed(event_id).await?)
}
pub struct WebhookExtract(pub WebhookType);
impl<S> FromRequest<S> for WebhookExtract
where
AppState: FromRef<S>,
S: Send + Sync,
{
type Rejection = AppError;
#[instrument(skip(req, state), err)]
async fn from_request(req: axum::extract::Request, state: &S) -> Result<Self, Self::Rejection> {
let app_state = AppState::from_ref(state);
let headers = req.headers();
let sig_header = extract_header(GITEA_SIG_HEADER_NAME, headers)?;
let type_header = extract_header(GITEA_EVENT_TYPE_HEADER_NAME, headers)?;
let body_bytes = read_body(req.into_body()).await?;
verify_signature(
app_state.config.webhook_secret.as_bytes(),
&sig_header,
&body_bytes,
)?;
let body_str = String::from_utf8_lossy(&body_bytes).into_owned();
sentry::configure_scope(|scope| {
scope.add_event_processor(move |mut event| {
let mut request = event.request.take().unwrap_or_default();
request.data = Some(body_str.clone());
event.request = Some(request);
Some(event)
});
});
let webhook = parse_webhook(&type_header, &app_state.bot_name, &body_bytes)?;
Ok(WebhookExtract(webhook))
}
}
fn extract_header(key: &str, headers: &axum::http::HeaderMap) -> Result<String, AppError> {
let value = headers
.get(key)
.ok_or(AppError::WebHookMissingHeaderErr(key.into()))?
.to_str()
.map_err(anyhow::Error::from)?;
Ok(value.to_owned())
}
async fn read_body(body: axum::body::Body) -> Result<Bytes, AppError> {
to_bytes(body, MAX_WEBHOOK_BODY_SIZE)
.await
.map_err(anyhow::Error::from)
.map_err(AppError::from)
}
fn parse_webhook(header: &str, bot_name: &str, body_bytes: &[u8]) -> Result<WebhookType, AppError> {
let Json(value) =
Json::<Value>::from_bytes(body_bytes).map_err(|_| AppError::MalformedJsonErr)?;
WebhookType::from_event(header, bot_name, value)
}
fn verify_signature(secret_key: &[u8], sig_header: &str, body: &[u8]) -> Result<(), AppError> {
let sig_header_decoded =
hex::decode(sig_header).map_err(|_| AppError::WebHookSigHeaderInvalidErr)?;
let key = hmac::Key::new(hmac::HMAC_SHA256, secret_key);
hmac::verify(&key, body, &sig_header_decoded).map_err(|_| AppError::WebHookSigHeaderInvalidErr)
}
+197
View File
@@ -0,0 +1,197 @@
use std::{sync::Arc, time::Duration};
use tokio::{sync::Semaphore, task::JoinSet};
use tokio_util::sync::CancellationToken;
use tracing::{error, info, instrument};
use crate::{
bot_actions::review::{self, ReviewContext, ReviewOutcome},
event_bus::{EventBus, Reporter},
gitea::{GiteaAPI, WebhookType},
job_queue::JobQueue,
metrics,
open_router::OpenRouterClient,
review_store::{ReviewRecord, ReviewStore},
sandbox::{SandboxConfig, tools},
schemas::Job,
};
/// Delay before claiming again after the queue could not be reached.
const CLAIM_RETRY_DELAY: Duration = Duration::from_secs(1);
/// Everything the bot works with: the external services and the stores.
#[derive(Clone)]
pub struct BotDeps {
pub gitea: GiteaAPI,
pub open_router: OpenRouterClient,
pub sandbox: SandboxConfig,
pub reviews: ReviewStore,
pub queue: JobQueue,
pub event_bus: EventBus,
}
#[derive(Clone)]
pub struct Bot {
deps: BotDeps,
max_concurrent: usize,
}
impl Bot {
pub fn new(deps: BotDeps, max_concurrent: usize) -> Self {
Self {
deps,
max_concurrent,
}
}
/// Consumes the queue until `shutdown` is cancelled.
///
/// A semaphore bounds the concurrency, and a job is only claimed once a slot is
/// free, so the jobs waiting for one stay `pending`. Running jobs are left to
/// finish; the unfinished ones stay `processing` and are requeued at the next
/// startup.
pub async fn start(&self, shutdown: CancellationToken) -> anyhow::Result<()> {
info!("Bot started");
let semaphore = Arc::new(Semaphore::new(self.max_concurrent));
let mut tasks = JoinSet::new();
'consume: loop {
let permit = tokio::select! {
biased;
() = shutdown.cancelled() => break,
permit = semaphore.clone().acquire_owned() => permit?,
};
let job = loop {
while let Some(res) = tasks.try_join_next() {
if let Err(err) = res {
error!("Task panicked: {err}");
}
}
match self.deps.queue.claim_next().await {
Ok(Some(job)) => break job,
Ok(None) => {
tokio::select! {
biased;
() = shutdown.cancelled() => break 'consume,
() = self.deps.queue.wait_for_work() => {}
}
}
Err(err) => {
error!(%err, "failed to claim a job");
tokio::select! {
biased;
() = shutdown.cancelled() => break 'consume,
() = tokio::time::sleep(CLAIM_RETRY_DELAY) => {}
}
}
}
};
let bot = self.clone();
metrics::increment_task_active();
tasks.spawn(async move {
bot.process(job).await;
drop(permit);
metrics::decrement_task_active();
});
}
tasks.join_all().await;
info!("Bot shutting down complete");
Ok(())
}
/// Runs a claimed job and records its outcome.
async fn process(&self, job: Job) {
self.run_job(&job).await;
if let Err(err) = self.deps.queue.remove(job.id).await {
error!(%err, "failed to remove job");
}
}
async fn run_job(&self, job: &Job) {
let deps = &self.deps;
let event_type = job.payload.event_type_str();
match deps.reviews.is_reviewed(job.event_id).await {
Ok(true) => {
metrics::task_skipped(event_type);
info!("Review already done, skipping");
deps.event_bus.skip(job.id, &job.repo, job.pr);
return;
}
Ok(false) => {}
Err(err) => error!(%err, "Review lookup failed, reviewing anyway"),
}
let reporter = deps.event_bus.start(job.id, job.repo.clone(), job.pr);
let started = std::time::Instant::now();
let result = self.exec(&job.payload, &reporter).await;
let duration_ms = u64::try_from(started.elapsed().as_millis()).unwrap_or(u64::MAX);
let (ok, outcome) = match result {
Ok(outcome) => {
metrics::task_completed(event_type);
info!("Task completed");
(true, outcome)
}
Err(err) => {
metrics::task_failed(event_type);
error!(%err, "Task error");
sentry_anyhow::capture_anyhow(&err);
(false, ReviewOutcome::empty())
}
};
let record = ReviewRecord {
event_id: job.event_id,
repo: job.repo.clone(),
pr: job.pr,
ok,
model: deps.open_router.model().to_string(),
markdown: outcome.markdown,
cost: outcome.cost,
comments: outcome.comments,
duration_ms,
};
if let Err(err) = deps.reviews.record(record).await {
error!(%err, "failed to record review");
}
deps.event_bus
.finish(job.id, &job.repo, job.pr, ok, outcome.comments, duration_ms);
}
/// Dispatches a webhook to the action it asks for.
#[instrument(
skip_all,
fields(repo = %webhook.repository_full_name(), pr = webhook.pull_request_number())
)]
async fn exec(
&self,
webhook: &WebhookType,
reporter: &Reporter,
) -> anyhow::Result<ReviewOutcome> {
let deps = &self.deps;
let ctx = ReviewContext {
gitea: &deps.gitea,
open_router: &deps.open_router,
sandbox: &deps.sandbox,
reporter,
};
let tools = tools::for_webhook(webhook);
match webhook {
WebhookType::Review(payload) => review::exec_review(&ctx, tools, payload).await,
}
}
}
File renamed without changes.
@@ -0,0 +1,177 @@
//! Anchoring of the model's reviews on the lines a pull request changes.
use tracing::warn;
use super::diff::ChangedLines;
use crate::domain::review::ReviewResult;
/// Resolves the side each review is anchored on and drops the reviews that do
/// not match a line the pull request changes.
///
/// The model is asked to pick a line and a side from the provided lists, but
/// nothing forces it to, and Gitea accepts any position: a wrong one places the
/// comment on an unrelated line of the file instead of failing. A review that
/// omits its side is resolved from the lists, and one that matches no changed
/// line is dropped.
pub(super) fn resolve_review_sides(review_result: &mut ReviewResult, changed_lines: &ChangedLines) {
let mut dropped = Vec::new();
review_result.reviews.retain_mut(|review| {
let side = review.line.and_then(|line| {
let file = changed_lines
.iter()
.find(|file| file.filename == review.filename)?;
let side = review.side.or_else(|| file.side_of(line))?;
file.lines(side).contains(&line).then_some(side)
});
match side {
Some(side) => {
review.side = Some(side);
true
}
None => {
dropped.push(match review.line {
Some(line) => format!("{}:{line}", review.filename),
None => format!("{}:no line", review.filename),
});
false
}
}
});
if !dropped.is_empty() {
warn!(
dropped = dropped.len(),
reviews = %dropped.join(", "),
"Dropped reviews that are not anchored on a changed line"
);
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::{
bot_actions::review::{diff::parse_changed_lines, fixtures::*},
domain::review::ReviewSide,
};
#[test]
fn reviews_keep_their_changed_line_and_resolve_their_side() {
let changed_lines = parse_changed_lines(DIFF);
let mut review_result = review_result(vec![
review("src/foo.rs", Some(4), Some(ReviewSide::Added)),
review("src/foo.rs", Some(2), Some(ReviewSide::Removed)),
review("src/bar.rs", Some(13), None),
]);
resolve_review_sides(&mut review_result, &changed_lines);
let sides = review_result
.reviews
.iter()
.map(|review| review.side)
.collect::<Vec<_>>();
assert_eq!(
sides,
vec![
Some(ReviewSide::Added),
Some(ReviewSide::Removed),
Some(ReviewSide::Added)
]
);
}
#[test]
fn a_line_changed_on_both_sides_defaults_to_added() {
let changed_lines = parse_changed_lines(DIFF);
let mut review_result = review_result(vec![review("src/foo.rs", Some(2), None)]);
resolve_review_sides(&mut review_result, &changed_lines);
assert_eq!(review_result.reviews[0].side, Some(ReviewSide::Added));
}
#[test]
fn reviews_outside_changed_lines_are_dropped() {
let changed_lines = parse_changed_lines(DIFF);
let mut review_result = review_result(vec![
// valid
review("src/foo.rs", Some(2), Some(ReviewSide::Added)),
// a line that exists but is not part of the change
review("src/foo.rs", Some(3), None),
// a line beyond the change
review("src/foo.rs", Some(999), None),
// a changed line, but on the wrong side
review("src/foo.rs", Some(4), Some(ReviewSide::Removed)),
// a line changed in another file
review("src/foo.rs", Some(11), None),
// an unknown file
review("src/baz.rs", Some(1), None),
// no line at all
review("src/bar.rs", None, None),
]);
resolve_review_sides(&mut review_result, &changed_lines);
assert_eq!(review_result.reviews.len(), 1);
assert_eq!(review_result.reviews[0].filename, "src/foo.rs");
assert_eq!(review_result.reviews[0].line, Some(2));
assert_eq!(review_result.reviews[0].side, Some(ReviewSide::Added));
}
#[test]
fn a_deleted_line_is_kept_as_a_removed_anchor() {
let changed_lines = parse_changed_lines(DELETION_ONLY);
let mut review_result = review_result(vec![
review("src/old.rs", Some(2), Some(ReviewSide::Removed)),
review("src/old.rs", Some(2), Some(ReviewSide::Added)),
]);
resolve_review_sides(&mut review_result, &changed_lines);
assert_eq!(review_result.reviews.len(), 1);
assert_eq!(review_result.reviews[0].side, Some(ReviewSide::Removed));
}
#[test]
fn odd_sides_from_the_model_are_tolerated() {
let changed_lines = parse_changed_lines(DIFF);
let mut review_result: ReviewResult = serde_json::from_str(
r#"{
"reviews": [
{ "filename": "src/foo.rs", "line": 4, "side": "Added", "message": "a" },
{ "filename": "src/foo.rs", "line": 2, "side": "new", "message": "b" },
{ "filename": "src/foo.rs", "line": 2, "side": "REMOVED", "message": "c" },
{ "filename": "src/foo.rs", "line": 2, "side": "banana", "message": "d" },
{ "filename": "src/foo.rs", "line": 5, "message": "e" }
],
"comment": ""
}"#,
)
.unwrap();
resolve_review_sides(&mut review_result, &changed_lines);
let sides = review_result
.reviews
.iter()
.map(|review| review.side)
.collect::<Vec<_>>();
assert_eq!(
sides,
vec![
Some(ReviewSide::Added),
Some(ReviewSide::Added),
Some(ReviewSide::Removed),
Some(ReviewSide::Added),
Some(ReviewSide::Added)
]
);
}
}
@@ -0,0 +1,389 @@
//! Parsing of a pull request unified diff into the lines it changes.
use tracing::warn;
use crate::{domain::review::ReviewSide, gitea::PullRequestFile};
/// The lines a pull request changed, per file.
///
/// Line numbers are the ones Gitea expects to place a review comment:
/// [`ReviewSide::Added`] numbers refer to the new version of the file (sent as
/// `new_position`), [`ReviewSide::Removed`] numbers to the old version (sent as
/// `old_position`).
pub(super) type ChangedLines = Vec<ChangedFile>;
pub(super) struct ChangedFile {
pub(super) filename: String,
pub(super) added: Vec<u64>,
pub(super) removed: Vec<u64>,
}
impl ChangedFile {
fn new(filename: &str) -> Self {
Self {
filename: String::from(filename),
added: Vec::new(),
removed: Vec::new(),
}
}
/// Lines that can be commented on for the given side.
pub(super) fn lines(&self, side: ReviewSide) -> &[u64] {
match side {
ReviewSide::Added => &self.added,
ReviewSide::Removed => &self.removed,
}
}
/// Side a line belongs to, used when the model did not state one.
pub(super) fn side_of(&self, line: u64) -> Option<ReviewSide> {
if self.added.contains(&line) {
Some(ReviewSide::Added)
} else if self.removed.contains(&line) {
Some(ReviewSide::Removed)
} else {
None
}
}
}
/// Lists the lines changed by the diff, per file, on both sides.
///
/// Only the line numbers are kept: the model reads the code itself through the
/// sandbox tools.
pub(super) fn parse_changed_lines(git_diff: &str) -> ChangedLines {
let mut files = Vec::new();
let mut current_file: Option<String> = None;
let mut in_hunk = false;
let mut old_line: u64 = 0;
let mut new_line: u64 = 0;
for line in git_diff.lines() {
if line.starts_with("diff --git ") {
current_file = None;
in_hunk = false;
continue;
}
// `--- a/x` and `+++ b/x` only appear before the first hunk of a file.
// Inside a hunk, a line may legitimately start with them: removing
// `--x` gives `---x`, adding `++i;` gives `+++i;`.
if !in_hunk && (line.starts_with("--- ") || line.starts_with("+++ ")) {
// A header is never a content line: `+++ /dev/null` on a deleted
// file keeps the path of the other side in place.
if let Some(path) = header_file_path(line) {
current_file = Some(path);
}
continue;
}
if line.starts_with("@@") {
if let Some((old_start, new_start)) = parse_hunk_starts(line) {
old_line = old_start;
new_line = new_start;
in_hunk = true;
}
continue;
}
let Some(filename) = current_file.as_deref() else {
continue;
};
match line.as_bytes().first() {
Some(b' ') => {
old_line += 1;
new_line += 1;
}
Some(b'-') => {
changed_file(&mut files, filename).removed.push(old_line);
old_line += 1;
}
Some(b'+') => {
changed_file(&mut files, filename).added.push(new_line);
new_line += 1;
}
// `\ No newline at end of file`, and anything unexpected: a line
// that advances neither side.
_ => {}
}
}
files
}
/// Path of the file on one side of the diff, from a `--- a/<path>` or
/// `+++ b/<path>` header line.
///
/// These lines are the only unambiguous source for the path: the `diff --git`
/// line is cut at the first space, and it names the old path of a renamed file.
/// `None` for `/dev/null`, which leaves the path of the other side in place.
fn header_file_path(line: &str) -> Option<String> {
let (prefix, raw) = match line.strip_prefix("--- ") {
Some(raw) => ("a/", raw),
None => ("b/", line.strip_prefix("+++ ")?),
};
let path = decode_git_path(raw);
Some(String::from(path.strip_prefix(prefix)?))
}
/// Decodes a path as git writes it in a diff header: git wraps it in quotes and
/// escapes the bytes that need it (`\303\251` for `é`) when the path contains
/// non-printable or non-ASCII characters.
fn decode_git_path(raw: &str) -> String {
let Some(quoted) = raw.strip_prefix('"').and_then(|raw| raw.strip_suffix('"')) else {
return String::from(raw);
};
let bytes = quoted.as_bytes();
let mut decoded = Vec::with_capacity(bytes.len());
let mut index = 0;
while let Some(byte) = bytes.get(index) {
index += 1;
if *byte != b'\\' {
decoded.push(*byte);
continue;
}
match bytes.get(index) {
// Octal escapes are the ones that matter for a path: git uses them
// for every non-ASCII byte.
Some(digit @ b'0'..=b'7') => {
let mut value = u32::from(digit - b'0');
let mut digits = 1;
while digits < 3 {
match bytes.get(index + digits) {
Some(next @ b'0'..=b'7') => {
value = value * 8 + u32::from(next - b'0');
digits += 1;
}
_ => break,
}
}
decoded.push(u8::try_from(value).unwrap_or(b'?'));
index += digits;
}
Some(escaped) => {
decoded.push(match escaped {
b't' => b'\t',
b'n' => b'\n',
b'r' => b'\r',
other => *other,
});
index += 1;
}
None => decoded.push(b'\\'),
}
}
String::from_utf8_lossy(&decoded).into_owned()
}
/// Replaces the paths parsed from the diff with the exact paths reported by the
/// API, which are the ones the model sees in the sandbox.
pub(super) fn resolve_filenames(changed_lines: &mut ChangedLines, files: &[PullRequestFile]) {
for changed in changed_lines.iter_mut() {
let parsed = changed.filename.as_str();
let Some(file) = files.iter().find(|file| file.filename == parsed) else {
if !files.is_empty() {
warn!(path = %parsed, "Changed file is not in the pull request file list");
}
continue;
};
changed.filename = file.filename.clone();
}
}
/// Basenames of generated files that are never reviewed.
///
/// Their diffs are machine-written dependency churn: they run to thousands of
/// lines, flood the prompt with line numbers and drown the code the model should
/// look at. A lockfile is also never a place where a review comment belongs.
const IGNORED_FILE_NAMES: [&str; 16] = [
"Cargo.lock",
"package-lock.json",
"npm-shrinkwrap.json",
"yarn.lock",
"pnpm-lock.yaml",
"bun.lock",
"bun.lockb",
"composer.lock",
"Gemfile.lock",
"poetry.lock",
"uv.lock",
"Pipfile.lock",
"go.sum",
"packages.lock.json",
"flake.lock",
"pubspec.lock",
];
/// Whether a changed path is a generated file that is never reviewed.
///
/// The comparison is on the basename, so a lockfile nested in a workspace (for
/// example `crates/foo/Cargo.lock`) is matched too.
pub(super) fn is_ignored(filename: &str) -> bool {
let basename = filename.rsplit('/').next().unwrap_or(filename);
IGNORED_FILE_NAMES.contains(&basename)
}
/// Drops the generated files from the changed lines.
///
/// Removing them here keeps the prompt free of their line numbers, and also makes
/// [`resolve_review_sides`] reject any review the model anchors on one of them.
pub(super) fn drop_generated_files(changed_lines: &mut ChangedLines) {
changed_lines.retain(|file| !is_ignored(&file.filename));
}
/// Borrows the entry of `files` for a file, creating it on its first change.
fn changed_file<'a>(files: &'a mut ChangedLines, filename: &str) -> &'a mut ChangedFile {
let index = match files.iter().position(|file| file.filename == filename) {
Some(index) => index,
None => {
files.push(ChangedFile::new(filename));
files.len() - 1
}
};
&mut files[index]
}
/// Extracts the old and new starting line numbers of a hunk header such as
/// `@@ -12,3 +12,5 @@`. The counts are optional and git may append a section
/// heading after the closing `@@`.
fn parse_hunk_starts(hunk_header: &str) -> Option<(u64, u64)> {
let body = hunk_header.strip_prefix("@@ ")?;
let body = body.split(" @@").next()?;
let (old, new) = body.split_once(" +")?;
let old = old.strip_prefix('-')?.split(',').next()?;
let new = new.split(',').next()?;
Some((old.parse().ok()?, new.parse().ok()?))
}
#[cfg(test)]
mod tests {
use super::*;
use crate::bot_actions::review::{fixtures::*, prompt::format_changed_lines};
#[test]
fn a_deletion_only_pull_request_lists_removed_lines() {
let expected = "src/old.rs: removed 1, 2, 3";
assert_eq!(
format_changed_lines(&parse_changed_lines(DELETION_ONLY)),
expected
);
}
#[test]
fn hunk_content_starting_with_plus_or_minus_is_counted() {
let expected = "src/tricky.js: added 2 / removed 2";
assert_eq!(
format_changed_lines(&parse_changed_lines(TRICKY_CONTENT)),
expected
);
}
#[test]
fn a_renamed_file_uses_its_new_path() {
let expected = "src/new.rs: added 1 / removed 1";
assert_eq!(
format_changed_lines(&parse_changed_lines(RENAMED)),
expected
);
}
#[test]
fn a_path_with_a_space_is_read_from_the_headers() {
let expected = "src/my file.rs: added 2";
assert_eq!(
format_changed_lines(&parse_changed_lines(PATH_WITH_SPACE)),
expected
);
}
#[test]
fn a_quoted_path_is_decoded() {
let expected = "docs/café.md: added 2";
assert_eq!(
format_changed_lines(&parse_changed_lines(QUOTED_PATH)),
expected
);
}
#[test]
fn filenames_are_resolved_against_the_api_list() {
let mut changed_lines = parse_changed_lines(DIFF);
let files = vec![
pull_request_file("src/bar.rs", None, "modified"),
pull_request_file("src/foo.rs", Some("src/renamed.rs"), "renamed"),
];
resolve_filenames(&mut changed_lines, &files);
assert_eq!(changed_lines[0].filename, "src/foo.rs");
assert_eq!(changed_lines[1].filename, "src/bar.rs");
}
#[test]
fn a_file_absent_from_the_api_list_keeps_the_diff_path() {
let mut changed_lines = parse_changed_lines(DIFF);
let files = vec![pull_request_file("src/bar.rs", None, "modified")];
resolve_filenames(&mut changed_lines, &files);
assert_eq!(changed_lines[0].filename, "src/foo.rs");
assert_eq!(changed_lines[1].filename, "src/bar.rs");
}
#[test]
fn generated_lockfiles_are_ignored() {
assert!(is_ignored("Cargo.lock"));
assert!(is_ignored("crates/steep-server/Cargo.lock"));
assert!(is_ignored("package-lock.json"));
assert!(is_ignored("web/yarn.lock"));
assert!(is_ignored("go.sum"));
assert!(!is_ignored("src/main.rs"));
assert!(!is_ignored("Cargo.toml"));
assert!(!is_ignored("docs/lockfiles.md"));
}
#[test]
fn generated_files_are_dropped_from_the_changed_lines() {
const DIFF_WITH_LOCKFILE: &str = concat!(
"diff --git a/Cargo.lock b/Cargo.lock\n",
"--- a/Cargo.lock\n",
"+++ b/Cargo.lock\n",
"@@ -1,1 +1,2 @@\n",
" [[package]]\n",
"+name = \"x\"\n",
"diff --git a/src/foo.rs b/src/foo.rs\n",
"--- a/src/foo.rs\n",
"+++ b/src/foo.rs\n",
"@@ -1,1 +1,2 @@\n",
" fn a() {}\n",
"+fn b() {}\n",
);
let mut changed_lines = parse_changed_lines(DIFF_WITH_LOCKFILE);
drop_generated_files(&mut changed_lines);
assert_eq!(format_changed_lines(&changed_lines), "src/foo.rs: added 2");
}
}
@@ -0,0 +1,119 @@
//! Shared fixtures of the review tests.
use crate::{
domain::review::{ReviewItem, ReviewResult, ReviewSide},
gitea::PullRequestFile,
};
/// Additions, a removal and a line changed on both sides.
pub(super) const DIFF: &str = concat!(
"diff --git a/src/foo.rs b/src/foo.rs\n",
"--- a/src/foo.rs\n",
"+++ b/src/foo.rs\n",
"@@ -1,4 +1,6 @@\n",
" fn main() {\n",
"+ let x = 1;\n",
"- let removed = 0;\n",
" println!(\"hello\");\n",
"+ let y = 2;\n",
"+ let z = 3;\n",
" }\n",
"diff --git a/src/bar.rs b/src/bar.rs\n",
"--- a/src/bar.rs\n",
"+++ b/src/bar.rs\n",
"@@ -10,4 +10,6 @@\n",
" old context\n",
"+ let a = 10;\n",
" more context\n",
"+ let b = 20;\n",
);
/// A pull request that only deletes a file.
pub(super) const DELETION_ONLY: &str = concat!(
"diff --git a/src/old.rs b/src/old.rs\n",
"deleted file mode 100644\n",
"--- a/src/old.rs\n",
"+++ /dev/null\n",
"@@ -1,3 +0,0 @@\n",
"-fn a() {}\n",
"-fn b() {}\n",
"-fn c() {}\n",
);
/// A hunk whose content starts with `+++` / `---`, which must not be taken
/// for the file headers.
pub(super) const TRICKY_CONTENT: &str = concat!(
"diff --git a/src/tricky.js b/src/tricky.js\n",
"--- a/src/tricky.js\n",
"+++ b/src/tricky.js\n",
"@@ -1,4 +1,4 @@\n",
" let i = 0;\n",
"+++i;\n",
"---x;\n",
" console.log(i);\n",
);
/// A pull request that renames a file.
pub(super) const RENAMED: &str = concat!(
"diff --git a/src/old.rs b/src/new.rs\n",
"similarity index 50%\n",
"rename from src/old.rs\n",
"rename to src/new.rs\n",
"--- a/src/old.rs\n",
"+++ b/src/new.rs\n",
"@@ -1,1 +1,1 @@\n",
"-fn old() {}\n",
"+fn new() {}\n",
);
/// A file whose path contains a space, which the `diff --git` line cannot
/// express without ambiguity.
pub(super) const PATH_WITH_SPACE: &str = concat!(
"diff --git a/src/my file.rs b/src/my file.rs\n",
"--- a/src/my file.rs\n",
"+++ b/src/my file.rs\n",
"@@ -1,1 +1,2 @@\n",
" fn a() {}\n",
"+fn b() {}\n",
);
/// A file whose path git quotes and escapes (`caf\\303\\251.md` is
/// `caf\u{e9}.md`).
pub(super) const QUOTED_PATH: &str = concat!(
"diff --git \"a/docs/caf\\303\\251.md\" \"b/docs/caf\\303\\251.md\"\n",
"--- \"a/docs/caf\\303\\251.md\"\n",
"+++ \"b/docs/caf\\303\\251.md\"\n",
"@@ -1,1 +1,2 @@\n",
" intro\n",
"+ajout\n",
);
pub(super) fn review(filename: &str, line: Option<u64>, side: Option<ReviewSide>) -> ReviewItem {
ReviewItem {
filename: String::from(filename),
line,
side,
severity: None,
message: String::from("message"),
}
}
pub(super) fn review_result(reviews: Vec<ReviewItem>) -> ReviewResult {
ReviewResult {
reviews,
comment: String::new(),
cost: None,
}
}
pub(super) fn pull_request_file(
filename: &str,
previous_filename: Option<&str>,
status: &str,
) -> PullRequestFile {
PullRequestFile {
filename: String::from(filename),
previous_filename: previous_filename.map(String::from),
status: String::from(status),
}
}
@@ -0,0 +1,210 @@
//! The pull request review action.
//!
//! [`exec_review`] drives one review end to end. The work is split by concern:
//! - [`diff`] parses the pull request diff into the lines it changes,
//! - [`prompt`] renders those changes for the model,
//! - [`anchor`] checks that the model's reviews land on changed lines,
//! - [`render`] builds the markdown summary posted on the pull request.
mod anchor;
mod diff;
mod prompt;
mod render;
#[cfg(test)]
mod fixtures;
use tracing::{info, instrument, warn};
use crate::{
consts::{BOT_PROCESS_MSG, REVIEW_PROMPT, SANDBOX_SYSTEM_PROMPT},
domain::review::ReviewResult,
event_bus::Reporter,
gitea::{GiteaAPI, ReviewPayload},
metrics,
open_router::{OpenRouterClient, Tool},
sandbox::{Sandbox, SandboxConfig, agent},
};
use self::{
anchor::resolve_review_sides,
diff::{drop_generated_files, parse_changed_lines, resolve_filenames},
prompt::format_changes,
render::review_result_to_markdown,
};
/// The services a review needs, borrowed from the bot for the duration of one job.
pub struct ReviewContext<'a> {
pub gitea: &'a GiteaAPI,
pub open_router: &'a OpenRouterClient,
pub sandbox: &'a SandboxConfig,
pub reporter: &'a Reporter,
}
pub struct ReviewOutcome {
pub markdown: String,
pub cost: Option<f64>,
pub comments: u64,
}
impl ReviewOutcome {
pub fn empty() -> Self {
Self {
markdown: String::new(),
cost: None,
comments: 0,
}
}
}
#[instrument(
skip_all,
fields(repo = %payload.repository.full_name, pr = payload.pull_request.number)
)]
pub async fn exec_review(
ctx: &ReviewContext<'_>,
tools: Vec<Tool>,
payload: &ReviewPayload,
) -> anyhow::Result<ReviewOutcome> {
info!(action = %payload.action, "Starting review");
let full_name = &payload.repository.full_name;
let index = payload.pull_request.number;
let progress_comment = ctx
.gitea
.comment(
&BOT_PROCESS_MSG.replace("{model}", ctx.open_router.model()),
full_name,
index,
)
.await?;
let review = review_pull_request(ctx, tools, payload).await;
// The progress comment is housekeeping: failing to update it must neither turn a
// posted review into a failure nor hide the error that made the review fail.
match review {
Ok((review_result, markdown)) => {
if let Err(err) = ctx
.gitea
.delete_comment(full_name, progress_comment.id)
.await
{
warn!(%err, "Failed to delete the progress comment");
}
Ok(ReviewOutcome {
markdown,
cost: review_result.cost,
comments: review_result.reviews.len() as u64,
})
}
Err(err) => {
if let Err(edit_err) = ctx
.gitea
.edit_comment(
&format!("Error while reviewing: {err}"),
full_name,
progress_comment.id,
)
.await
{
warn!(err = %edit_err, "Failed to report the error on the progress comment");
}
Err(err)
}
}
}
/// Runs the review and posts it, returning the result and the markdown that was
/// posted.
async fn review_pull_request(
ctx: &ReviewContext<'_>,
tools: Vec<Tool>,
payload: &ReviewPayload,
) -> anyhow::Result<(ReviewResult, String)> {
let full_name = &payload.repository.full_name;
let index = payload.pull_request.number;
let git_diff = ctx.gitea.pull_request_diff(full_name, index).await?;
// The file list only refines the paths and describes the changes: a failure is
// not fatal, the diff is the source of truth for the lines.
let files = match ctx.gitea.pull_request_files(full_name, index).await {
Ok(files) => files,
Err(err) => {
warn!(%err, "Failed to list the pull request files");
Vec::new()
}
};
let mut changed_lines = parse_changed_lines(&git_diff);
drop_generated_files(&mut changed_lines);
resolve_filenames(&mut changed_lines, &files);
let changes = format_changes(&files, &changed_lines);
let bot_request = REVIEW_PROMPT
.replace("{subject}", &payload.pull_request.title)
.replace("{comment}", &payload.comment.body)
.replace("{changes}", &changes);
let (mut review_result, cost) = run_sandboxed_review(ctx, tools, payload, &bot_request).await?;
resolve_review_sides(&mut review_result, &changed_lines);
review_result.cost = cost;
if let Some(cost) = review_result.cost {
metrics::openrouter_cost_usd(cost);
}
let markdown = review_result_to_markdown(&review_result);
ctx.gitea
.post_pull_request_review(&review_result, &markdown, full_name, index)
.await?;
Ok((review_result, markdown))
}
async fn run_sandboxed_review(
ctx: &ReviewContext<'_>,
tools: Vec<Tool>,
payload: &ReviewPayload,
bot_request: &str,
) -> anyhow::Result<(ReviewResult, Option<f64>)> {
let sandbox = Sandbox::create(
&ctx.sandbox.runtime,
&payload.repository.clone_url,
ctx.gitea.token(),
payload.pull_request.number,
ctx.reporter,
)
.await?;
let result = agent::run(
ctx.open_router,
&sandbox,
tools,
SANDBOX_SYSTEM_PROMPT,
bot_request,
ctx.sandbox.max_iterations,
ctx.reporter,
)
.await;
if let Err(err) = sandbox.cleanup().await {
warn!(%err, "Failed to clean up sandbox container");
}
let result = result?;
info!(
iterations = result.iterations,
cost = ?result.cost,
"Sandboxed review finished"
);
Ok((result.answer, result.cost))
}
@@ -0,0 +1,125 @@
//! Rendering of a pull request's changes for the model.
use super::diff::{ChangedLines, is_ignored};
use crate::gitea::PullRequestFile;
/// Renders the changes for the model: the files the pull request touches, then
/// the lines to review per file.
pub(super) fn format_changes(files: &[PullRequestFile], changed_lines: &ChangedLines) -> String {
let mut sections = Vec::new();
let described = files
.iter()
.filter(|file| !is_ignored(&file.filename))
.map(describe_file)
.collect::<Vec<_>>()
.join(", ");
if !described.is_empty() {
sections.push(format!("Files changed by the pull request: {described}"));
}
sections.push(format!(
"Lines to review, per file:\n{}",
format_changed_lines(changed_lines)
));
sections.join("\n\n")
}
/// Describes a changed file for the model, including how it changed.
fn describe_file(file: &PullRequestFile) -> String {
match &file.previous_filename {
Some(previous) => format!("{} ({} from {})", file.filename, file.status, previous),
None => format!("{} ({})", file.filename, file.status),
}
}
/// Renders the changed lines as `filename: added 1, 2 / removed 3`, one file per
/// line, keeping only the sides the file actually has.
pub(super) fn format_changed_lines(changed_lines: &ChangedLines) -> String {
changed_lines
.iter()
.map(|file| {
let mut sides = Vec::new();
for (label, lines) in [("added", &file.added), ("removed", &file.removed)] {
if !lines.is_empty() {
sides.push(format!("{label} {}", format_line_numbers(lines)));
}
}
format!("{}: {}", file.filename, sides.join(" / "))
})
.collect::<Vec<_>>()
.join("\n")
}
fn format_line_numbers(lines: &[u64]) -> String {
lines
.iter()
.map(u64::to_string)
.collect::<Vec<_>>()
.join(", ")
}
#[cfg(test)]
mod tests {
use super::*;
use crate::bot_actions::review::{diff::parse_changed_lines, fixtures::*};
#[test]
fn changed_lines_are_listed_per_file_and_side() {
let expected = concat!(
"src/foo.rs: added 2, 4, 5 / removed 2\n",
"src/bar.rs: added 11, 13"
);
assert_eq!(format_changed_lines(&parse_changed_lines(DIFF)), expected);
}
#[test]
fn changes_describe_the_files_then_the_lines() {
let changed_lines = parse_changed_lines(DELETION_ONLY);
let files = vec![pull_request_file("src/old.rs", None, "deleted")];
let expected = concat!(
"Files changed by the pull request: src/old.rs (deleted)\n",
"\n",
"Lines to review, per file:\n",
"src/old.rs: removed 1, 2, 3"
);
assert_eq!(format_changes(&files, &changed_lines), expected);
}
#[test]
fn changes_without_the_api_list_only_hold_the_lines() {
let changed_lines = parse_changed_lines(DELETION_ONLY);
let expected = "Lines to review, per file:\nsrc/old.rs: removed 1, 2, 3";
assert_eq!(format_changes(&[], &changed_lines), expected);
}
#[test]
fn a_generated_file_is_not_described_to_the_model() {
let files = vec![
pull_request_file("Cargo.lock", None, "modified"),
pull_request_file("src/foo.rs", None, "modified"),
];
let changed_lines = parse_changed_lines(DIFF);
let changes = format_changes(&files, &changed_lines);
assert!(!changes.contains("Cargo.lock"));
assert!(changes.contains("src/foo.rs (modified)"));
}
#[test]
fn a_renamed_file_is_described_with_its_previous_path() {
let file = pull_request_file("src/new.rs", Some("src/old.rs"), "renamed");
assert_eq!(describe_file(&file), "src/new.rs (renamed from src/old.rs)");
}
}
@@ -0,0 +1,110 @@
//! Markdown summary of a review, as posted on the pull request.
use crate::domain::review::{ReviewItem, ReviewResult, ReviewSeverity};
pub(super) fn review_result_to_markdown(review_result: &ReviewResult) -> String {
if review_result.reviews.is_empty() {
return String::from("No issues found. ✅");
}
let mut md = String::from("## Review Feedback\n\n");
md.push_str(&format!(
"### {} issues found.\n\n",
review_result.reviews.len()
));
let breakdown = severity_breakdown(&review_result.reviews);
if !breakdown.is_empty() {
md.push_str(&breakdown);
md.push('\n');
}
if !review_result.comment.is_empty() {
md.push_str("\n---\n\n");
md.push_str("### Summary\n\n");
md.push_str(&review_result.comment);
md.push('\n');
}
if let Some(cost) = review_result.cost {
md.push_str("\n---\n\n");
md.push_str(&format!("### Cost: ${}", cost));
md.push('\n');
}
md
}
/// Counts the reviews per severity, most severe first.
///
/// A review whose severity the model omitted or wrote unreadably is counted as
/// [`ReviewSeverity::Maintainability`], the same fallback used when posting it.
fn severity_breakdown(reviews: &[ReviewItem]) -> String {
[
ReviewSeverity::Bug,
ReviewSeverity::Security,
ReviewSeverity::Performance,
ReviewSeverity::Maintainability,
]
.into_iter()
.filter_map(|severity| {
let count = reviews
.iter()
.filter(|review| review.severity.unwrap_or(ReviewSeverity::Maintainability) == severity)
.count();
(count > 0).then(|| format!("- {} {}", count, severity.label()))
})
.collect::<Vec<_>>()
.join("\n")
}
#[cfg(test)]
mod tests {
use super::*;
use crate::bot_actions::review::fixtures::*;
#[test]
fn the_breakdown_counts_severities_most_severe_first() {
let reviews = vec![
ReviewItem {
severity: Some(ReviewSeverity::Maintainability),
..review("a", Some(1), None)
},
ReviewItem {
severity: Some(ReviewSeverity::Bug),
..review("b", Some(2), None)
},
ReviewItem {
severity: Some(ReviewSeverity::Bug),
..review("c", Some(3), None)
},
// A severity the model left out counts as maintainability.
review("d", Some(4), None),
];
assert_eq!(severity_breakdown(&reviews), "- 2 bug\n- 2 maintainability");
}
#[test]
fn the_markdown_breaks_the_issues_down_by_severity() {
let mut result = review_result(vec![
ReviewItem {
severity: Some(ReviewSeverity::Bug),
..review("a", Some(1), None)
},
ReviewItem {
severity: Some(ReviewSeverity::Performance),
..review("b", Some(2), None)
},
]);
result.comment = String::from("Le découpage en crates est propre.");
let markdown = review_result_to_markdown(&result);
assert!(markdown.contains("- 1 bug"));
assert!(markdown.contains("- 1 performance"));
assert!(markdown.contains("Le découpage en crates est propre."));
}
}
+82
View File
@@ -0,0 +1,82 @@
pub const GITEA_SIG_HEADER_NAME: &str = "x-gitea-signature";
pub const GITEA_EVENT_TYPE_HEADER_NAME: &str = "x-gitea-event-type";
pub const MAX_WEBHOOK_BODY_SIZE: usize = 1024 * 1024; // 1 MiB
pub const MAX_DIFF_SIZE: usize = 1024 * 1024; // 1 MiB
pub const BOT_PROCESS_MSG: &str = "
Review in progress with the model \"{model}\"...
";
pub const SANDBOX_SYSTEM_PROMPT: &str = "
You are a senior software engineer reviewing a pull request inside an
isolated sandbox.
The repository is checked out in your working directory. Explore it with the
provided read-only tools (ls, file_size, read_file, grep, find); paths are
relative to the repository root. Every line you read is prefixed with its
absolute line number: that is the number you must cite to anchor a comment.
Read before you assert: a review anchored on a line you did not read is
worthless, so never comment on code you have not seen. Gather enough context
to be confident, but do not re-read what you already have.
When you answer, send the requested JSON only: a raw JSON object, no markdown
code fence, nothing before or after it.
";
pub const REVIEW_PROMPT: &str = "
You are a senior software engineer reviewing a pull request.
Judge the changes for correctness, security, resource use and
maintainability. Report real problems; do not invent issues, and do not
report pure formatting a formatter would fix.
Be exhaustive: one review per distinct issue, and cover every changed file
that has something to report. Do not stop at the first few findings, and do
not merge several issues into a single review.
This is the pull request subject: \"{subject}\"
This is the user comment: \"{comment}\"
If the user asks about something precise, address that first.
The pull request changes these files and lines:
{changes}
The code is not provided. Read the files you need with the available tools
before answering.
`added` line numbers refer to the new version of the file, which is what your
working directory contains now. `removed` line numbers refer to the old
version, which is not in your working directory: you cannot read a removed
line, only the code around where it used to be.
Every review must anchor on one of the listed line numbers:
- filename: the full path exactly as listed above,
- line: one of the numbers listed for that file,
- side: \"added\" for a number from the `added` list, \"removed\" for one from
the `removed` list,
- severity: exactly one of \"bug\", \"security\", \"performance\" or
\"maintainability\" — a bug is wrong behaviour, security a vulnerability,
performance a resource problem, maintainability everything else.
Answer in french, with the raw json object only: no markdown code fence, no
text before or after it. All fields are mandatory. The `comment` field must
hold a short summary that lists the issues, and also what the pull request
does well: the author should get the compliments too.
{
\"reviews\": [
{
\"filename\": \"\",
\"line\": 0,
\"side\": \"\",
\"severity\": \"\",
\"message\": \"\"
}
],
\"comment\": \"\"
}
";
+20
View File
@@ -0,0 +1,20 @@
use toasty::Db;
pub async fn open(url: &str) -> toasty::Result<Db> {
Db::builder()
.models(toasty::models!(crate::*))
.connect(url)
.await
}
/// A connection for a test, on the database `DATABASE_URL` points to.
///
/// The schema must already be migrated: `scripts/migrate-apply.sh` does it, the same
/// way it does for the server. Each test opens its own connection, because the
/// connection tasks live on the runtime of the test that opened them.
#[cfg(test)]
pub(crate) async fn test_connect() -> Db {
let url = std::env::var("DATABASE_URL").expect("DATABASE_URL must be set");
open(&url).await.expect("connect to postgres")
}
+1
View File
@@ -0,0 +1 @@
pub mod review;
+247
View File
@@ -0,0 +1,247 @@
//! Domain types of a review: what the model answers with, and how it is read.
use std::str::FromStr;
use serde::{Deserialize, Deserializer};
use tracing::warn;
use crate::text::excerpt;
#[derive(Deserialize, Debug)]
pub struct ReviewResult {
pub reviews: Vec<ReviewItem>,
pub comment: String,
pub cost: Option<f64>,
}
#[derive(Deserialize, Debug)]
pub struct ReviewItem {
pub filename: String,
pub line: Option<u64>,
#[serde(default, deserialize_with = "deserialize_side")]
pub side: Option<ReviewSide>,
#[serde(default, deserialize_with = "deserialize_severity")]
pub severity: Option<ReviewSeverity>,
pub message: String,
}
/// Which version of the file a review comment is anchored on.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ReviewSide {
/// The line was added by the pull request: `line` is a line number of the
/// new version of the file.
Added,
/// The line was removed by the pull request: `line` is a line number of the
/// old version of the file.
Removed,
}
impl ReviewSide {
/// Reads the side the model asked for, tolerating casing and synonyms.
fn parse(raw: &str) -> Option<Self> {
match raw.trim().to_ascii_lowercase().as_str() {
"added" | "add" | "new" | "right" => Some(Self::Added),
"removed" | "remove" | "deleted" | "delete" | "old" | "left" => Some(Self::Removed),
_ => None,
}
}
}
/// What kind of problem a review reports.
///
/// The categories are the ones the prompt asks for; a review whose severity is
/// missing or unreadable falls back to [`ReviewSeverity::Maintainability`], the
/// least alarming bucket, instead of failing the whole review.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ReviewSeverity {
/// Wrong behaviour.
Bug,
/// A vulnerability.
Security,
/// A resource problem.
Performance,
/// Everything else: readability, structure, naming, tests.
Maintainability,
}
impl ReviewSeverity {
/// Reads the severity the model asked for, tolerating casing and synonyms.
fn parse(raw: &str) -> Option<Self> {
match raw.trim().to_ascii_lowercase().as_str() {
"bug" | "bugs" | "correctness" | "error" => Some(Self::Bug),
"security" | "vulnerability" => Some(Self::Security),
"performance" | "perf" => Some(Self::Performance),
"maintainability" | "quality" | "style" => Some(Self::Maintainability),
_ => None,
}
}
/// Lower-case label used in the review markdown.
pub fn label(self) -> &'static str {
match self {
Self::Bug => "bug",
Self::Security => "security",
Self::Performance => "performance",
Self::Maintainability => "maintainability",
}
}
}
/// Reads the side the model asked for. An unreadable value is ignored instead of
/// failing the whole review: the side is then resolved from the changed lines.
fn deserialize_side<'de, D>(deserializer: D) -> Result<Option<ReviewSide>, D::Error>
where
D: Deserializer<'de>,
{
let raw = Option::<serde_json::Value>::deserialize(deserializer)?;
Ok(raw
.as_ref()
.and_then(serde_json::Value::as_str)
.and_then(ReviewSide::parse))
}
/// Reads the severity the model asked for, ignoring an unreadable value: the
/// review is then reported as [`ReviewSeverity::Maintainability`].
fn deserialize_severity<'de, D>(deserializer: D) -> Result<Option<ReviewSeverity>, D::Error>
where
D: Deserializer<'de>,
{
let raw = Option::<serde_json::Value>::deserialize(deserializer)?;
Ok(raw
.as_ref()
.and_then(serde_json::Value::as_str)
.and_then(ReviewSeverity::parse))
}
/// Number of characters of a model answer kept in the logs when it cannot be
/// parsed.
const MAX_LOGGED_ANSWER: usize = 500;
impl FromStr for ReviewResult {
type Err = anyhow::Error;
/// Parses the review the model answered with.
///
/// This is the contract the agent loop enforces: a rejected answer is sent
/// back to the model, with the reason, so that it can correct itself.
///
/// The contract is a raw JSON object, but models sometimes wrap it in a
/// markdown code fence or surround it with a sentence: the object is then
/// extracted from the answer before failing, and the answer is logged so a
/// breach of the contract can be diagnosed.
fn from_str(message: &str) -> Result<Self, Self::Err> {
let error = match serde_json::from_str::<Self>(message) {
Ok(review_result) => return Ok(review_result),
Err(error) => error,
};
// A markdown code fence or a sentence around the object is tolerated, with
// a warning: the contract asks for a raw JSON object.
if let Some(json) = json_object(message)
&& let Ok(review_result) = serde_json::from_str::<Self>(json)
{
warn!(
"Model answer is not a raw JSON object, it was extracted from the surrounding text"
);
return Ok(review_result);
}
// The reason of the rejection is logged along with the answer: without it,
// a broken answer is impossible to diagnose.
warn!(
answer = %excerpt(message, MAX_LOGGED_ANSWER),
reason = %error,
"Model answer is not the expected JSON"
);
anyhow::bail!("the answer is not valid JSON: {error}")
}
}
/// Returns the outermost `{...}` of an answer, which ignores a markdown code
/// fence or any text around it.
fn json_object(message: &str) -> Option<&str> {
let start = message.find('{')?;
let end = message.rfind('}')?;
(start < end).then(|| &message[start..=end])
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_raw_answer_is_parsed() {
let answer = r#"{"reviews": [], "comment": "ok"}"#;
assert_eq!(answer.parse::<ReviewResult>().unwrap().comment, "ok");
}
#[test]
fn a_fenced_answer_is_extracted() {
let answer = concat!(
"Voici ma review :\n",
"```json\n",
"{\"reviews\": [], \"comment\": \"rien à signaler\"}\n",
"```\n"
);
assert_eq!(
answer.parse::<ReviewResult>().unwrap().comment,
"rien à signaler"
);
}
#[test]
fn a_sentence_around_the_object_is_ignored() {
let answer = r#"Rien à signaler. {"reviews": [], "comment": "ok"} Bonne journée !"#;
assert_eq!(answer.parse::<ReviewResult>().unwrap().comment, "ok");
}
#[test]
fn an_answer_without_json_is_rejected() {
assert!("Je n'ai rien relevé.".parse::<ReviewResult>().is_err());
}
#[test]
fn an_answer_that_is_an_object_but_not_a_review_is_rejected() {
assert!(r#"{"message": "LGTM"}"#.parse::<ReviewResult>().is_err());
}
#[test]
fn severities_are_read_from_the_model() {
let result: ReviewResult = serde_json::from_str(
r#"{
"reviews": [
{ "filename": "a", "line": 1, "severity": "Bug", "message": "x" },
{ "filename": "b", "line": 2, "severity": "SECURITY", "message": "x" },
{ "filename": "c", "line": 3, "severity": "perf", "message": "x" },
{ "filename": "d", "line": 4, "severity": "banana", "message": "x" }
],
"comment": ""
}"#,
)
.unwrap();
let severities = result
.reviews
.iter()
.map(|review| review.severity)
.collect::<Vec<_>>();
assert_eq!(
severities,
vec![
Some(ReviewSeverity::Bug),
Some(ReviewSeverity::Security),
Some(ReviewSeverity::Performance),
None,
]
);
}
}
+156
View File
@@ -0,0 +1,156 @@
use std::{str::FromStr, time::Duration};
use anyhow::Context;
/// Number of tool-calling iterations a sandboxed review may take, unless configured.
const DEFAULT_SANDBOX_MAX_ITERATIONS: usize = 8;
#[derive(Clone)]
pub struct EnvConfig {
pub http_port: u16,
pub webhook_secret: String,
pub open_router_api_key: String,
pub open_router_model: String,
pub open_router_timeout: Duration,
pub bot_max_concurrent: usize,
pub gitea_url: String,
pub gitea_token: String,
pub gitea_timeout: Duration,
pub database_url: String,
pub metrics_bind_addr: Option<String>,
pub sandbox_max_iterations: usize,
}
pub fn load_config() -> anyhow::Result<EnvConfig> {
Ok(EnvConfig {
http_port: parse_env("HTTP_PORT")?,
webhook_secret: try_get_env("WEBHOOK_SIG_HEADER_SECRET")?,
open_router_api_key: try_get_env("OPEN_ROUTER_API_KEY")?,
open_router_model: try_get_env("OPEN_ROUTER_MODEL")?,
open_router_timeout: parse_seconds_env("OPEN_ROUTER_TIMEOUT")?,
bot_max_concurrent: parse_env("BOT_MAX_CONCURRENT")?,
gitea_url: try_get_env("GITEA_URL")?,
gitea_token: try_get_env("GITEA_TOKEN")?,
gitea_timeout: parse_seconds_env("GITEA_TIMEOUT")?,
database_url: try_get_env("DATABASE_URL")?,
metrics_bind_addr: std::env::var("METRICS_BIND_ADDR").ok(),
sandbox_max_iterations: parse_optional_env("SANDBOX_MAX_ITERATIONS")?
.unwrap_or(DEFAULT_SANDBOX_MAX_ITERATIONS),
})
}
pub fn try_get_env(key: &str) -> anyhow::Result<String> {
let env_value = std::env::var(key).map_err(|e| anyhow::anyhow!("{}: {}", key, e))?;
if env_value.trim().is_empty() {
anyhow::bail!("env var {key} is empty");
}
Ok(env_value)
}
/// Reads a required variable and parses it, naming the variable when it is invalid.
fn parse_env<T>(key: &str) -> anyhow::Result<T>
where
T: FromStr,
T::Err: std::error::Error + Send + Sync + 'static,
{
parse_value(key, &try_get_env(key)?)
}
/// Reads an optional variable and parses it. A missing or empty variable is `None`;
/// one that is set but invalid is an error, like it is for the required ones.
fn parse_optional_env<T>(key: &str) -> anyhow::Result<Option<T>>
where
T: FromStr,
T::Err: std::error::Error + Send + Sync + 'static,
{
match std::env::var(key) {
Ok(value) if !value.trim().is_empty() => parse_value(key, &value).map(Some),
_ => Ok(None),
}
}
/// Reads a required variable holding a number of seconds.
fn parse_seconds_env(key: &str) -> anyhow::Result<Duration> {
parse_env::<u64>(key).map(Duration::from_secs)
}
fn parse_value<T>(key: &str, value: &str) -> anyhow::Result<T>
where
T: FromStr,
T::Err: std::error::Error + Send + Sync + 'static,
{
value
.trim()
.parse()
.with_context(|| format!("env var {key} is invalid: `{value}`"))
}
#[cfg(test)]
mod tests {
use super::*;
// Every test uses its own variable: the environment is shared by the tests,
// which run in parallel.
#[test]
fn test_try_get_env_returns_value() {
unsafe { std::env::set_var("TEST_ENV_PRESENT", "hello") };
assert_eq!(try_get_env("TEST_ENV_PRESENT").unwrap(), "hello");
}
#[test]
fn test_try_get_env_missing_var_returns_error() {
unsafe { std::env::remove_var("TEST_ENV_MISSING") };
assert!(try_get_env("TEST_ENV_MISSING").is_err());
}
#[test]
fn test_try_get_env_empty_value_returns_error() {
unsafe { std::env::set_var("TEST_ENV_EMPTY", "") };
let err = try_get_env("TEST_ENV_EMPTY").unwrap_err();
assert!(err.to_string().contains("TEST_ENV_EMPTY"));
}
#[test]
fn test_try_get_env_whitespace_only_returns_error() {
unsafe { std::env::set_var("TEST_ENV_WHITESPACE", " ") };
let err = try_get_env("TEST_ENV_WHITESPACE").unwrap_err();
assert!(err.to_string().contains("TEST_ENV_WHITESPACE"));
}
#[test]
fn a_timeout_is_read_as_seconds() {
unsafe { std::env::set_var("TEST_ENV_SECONDS", "30") };
assert_eq!(
parse_seconds_env("TEST_ENV_SECONDS").unwrap(),
Duration::from_secs(30)
);
}
#[test]
fn an_invalid_number_names_its_variable() {
unsafe { std::env::set_var("TEST_ENV_NOT_A_NUMBER", "soon") };
let err = parse_env::<u16>("TEST_ENV_NOT_A_NUMBER").unwrap_err();
assert!(format!("{err:#}").contains("TEST_ENV_NOT_A_NUMBER"));
assert!(format!("{err:#}").contains("soon"));
}
#[test]
fn an_absent_optional_variable_is_none() {
unsafe { std::env::remove_var("TEST_ENV_OPTIONAL_ABSENT") };
assert_eq!(
parse_optional_env::<usize>("TEST_ENV_OPTIONAL_ABSENT").unwrap(),
None
);
}
#[test]
fn an_invalid_optional_variable_is_an_error_instead_of_the_default() {
unsafe { std::env::set_var("TEST_ENV_OPTIONAL_INVALID", "many") };
assert!(parse_optional_env::<usize>("TEST_ENV_OPTIONAL_INVALID").is_err());
}
}
@@ -1,6 +1,6 @@
use anyhow::anyhow;
use axum::response::IntoResponse; use axum::response::IntoResponse;
use reqwest::StatusCode; use reqwest::StatusCode;
use sentry::capture_error;
#[derive(thiserror::Error, Debug)] #[derive(thiserror::Error, Debug)]
pub enum AppError { pub enum AppError {
@@ -22,15 +22,21 @@ pub enum AppError {
#[error("WebHook have bad action")] #[error("WebHook have bad action")]
InvalidActionErr, InvalidActionErr,
#[error("Channel full")] #[error("Could not enqueue task")]
ChannelFullErr, EnqueueErr(#[source] anyhow::Error),
#[error("Already processed")] #[error("Already processed")]
AlreadyProcessedErr, AlreadyProcessedErr,
#[error("Job {0} is not running")]
JobNotFoundErr(u64),
#[error(transparent)] #[error(transparent)]
BadJsonStructErr(#[from] serde_json::Error), BadJsonStructErr(#[from] serde_json::Error),
#[error("Other toasty error")]
OtherToastyErr(#[from] toasty::Error),
#[error(transparent)] #[error(transparent)]
Other(#[from] anyhow::Error), Other(#[from] anyhow::Error),
} }
@@ -62,11 +68,14 @@ impl IntoResponse for AppError {
"WebHook sig header is invalid".to_string(), "WebHook sig header is invalid".to_string(),
), ),
AppError::AlreadyProcessedErr => (StatusCode::OK, "Already processed".to_string()), AppError::AlreadyProcessedErr => (StatusCode::OK, "Already processed".to_string()),
AppError::ChannelFullErr => { AppError::JobNotFoundErr(id) => {
sentry_anyhow::capture_anyhow(&anyhow!("Max concurrent tasks reached")); (StatusCode::NOT_FOUND, format!("job {id} is not running"))
}
AppError::EnqueueErr(err) => {
sentry_anyhow::capture_anyhow(&err);
( (
StatusCode::SERVICE_UNAVAILABLE, StatusCode::SERVICE_UNAVAILABLE,
"Max concurrent tasks reached".to_string(), "Could not enqueue task".to_string(),
) )
} }
AppError::Other(err) => { AppError::Other(err) => {
@@ -76,6 +85,13 @@ impl IntoResponse for AppError {
"Internal server error".to_string(), "Internal server error".to_string(),
) )
} }
AppError::OtherToastyErr(err) => {
capture_error(&err);
(
StatusCode::INTERNAL_SERVER_ERROR,
"Internal server error".to_string(),
)
}
} }
.into_response() .into_response()
} }
+335
View File
@@ -0,0 +1,335 @@
use std::collections::{HashMap, VecDeque};
use std::sync::{Arc, Mutex};
use serde::Serialize;
use tokio::sync::broadcast;
use crate::text::excerpt;
const LIFECYCLE_CAPACITY: usize = 256;
const PROGRESS_CAPACITY: usize = 256;
const HISTORY_CAPACITY: usize = 256;
const MAX_EVENT_ARGUMENTS: usize = 200;
/// Lifecycle event of a job, broadcast globally.
#[derive(Clone, Debug, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum JobEvent {
Queued {
id: u64,
repo: String,
pr: u64,
},
Started {
id: u64,
repo: String,
pr: u64,
},
ReviewDone {
id: u64,
repo: String,
pr: u64,
ok: bool,
comments: u64,
duration_ms: u64,
},
Failed {
id: u64,
repo: String,
pr: u64,
},
Skipped {
id: u64,
repo: String,
pr: u64,
},
}
#[derive(Clone, Debug, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum Progress {
ToolCall {
iteration: usize,
name: String,
arguments: String,
},
ToolResult {
iteration: usize,
name: String,
truncated: bool,
output: String,
},
Finished {
ok: bool,
},
}
#[derive(Debug)]
struct JobChannel {
tx: broadcast::Sender<Progress>,
history: Mutex<VecDeque<Progress>>,
}
impl JobChannel {
fn subscribe(&self) -> (Vec<Progress>, broadcast::Receiver<Progress>) {
let history = self.history.lock().expect("job history lock poisoned");
let rx = self.tx.subscribe();
(history.iter().cloned().collect(), rx)
}
fn emit(&self, event: Progress) {
let mut history = self.history.lock().expect("job history lock poisoned");
if history.len() >= HISTORY_CAPACITY {
history.pop_front();
}
let _ = self.tx.send(event.clone());
history.push_back(event);
}
}
pub struct JobSubscription {
pub history: Vec<Progress>,
pub live: broadcast::Receiver<Progress>,
}
#[derive(Clone)]
pub struct EventBus {
lifecycle: broadcast::Sender<JobEvent>,
jobs: Arc<Mutex<HashMap<u64, Arc<JobChannel>>>>,
}
impl EventBus {
pub fn new() -> Self {
let (lifecycle, _rx) = broadcast::channel(LIFECYCLE_CAPACITY);
Self {
lifecycle,
jobs: Arc::new(Mutex::new(HashMap::new())),
}
}
pub fn publish(&self, event: JobEvent) {
let _ = self.lifecycle.send(event);
}
pub fn subscribe_lifecycle(&self) -> broadcast::Receiver<JobEvent> {
self.lifecycle.subscribe()
}
pub fn start(&self, id: u64, repo: impl Into<String>, pr: u64) -> Reporter {
let (tx, _rx) = broadcast::channel(PROGRESS_CAPACITY);
let channel = Arc::new(JobChannel {
tx,
history: Mutex::new(VecDeque::new()),
});
self.jobs
.lock()
.expect("job registry lock poisoned")
.insert(id, channel.clone());
self.publish(JobEvent::Started {
id,
repo: repo.into(),
pr,
});
Reporter { channel }
}
pub fn finish(&self, id: u64, repo: &str, pr: u64, ok: bool, comments: u64, duration_ms: u64) {
let event = if ok {
JobEvent::ReviewDone {
id,
repo: repo.to_string(),
pr,
ok,
comments,
duration_ms,
}
} else {
JobEvent::Failed {
id,
repo: repo.to_string(),
pr,
}
};
self.publish(event);
if let Some(channel) = self
.jobs
.lock()
.expect("job registry lock poisoned")
.remove(&id)
{
let _ = channel.tx.send(Progress::Finished { ok });
}
}
pub fn skip(&self, id: u64, repo: &str, pr: u64) {
self.publish(JobEvent::Skipped {
id,
repo: repo.to_string(),
pr,
});
}
pub fn subscribe_job(&self, id: u64) -> Option<JobSubscription> {
let channel = self
.jobs
.lock()
.expect("job registry lock poisoned")
.get(&id)
.cloned()?;
let (history, live) = channel.subscribe();
Some(JobSubscription { history, live })
}
}
impl Default for EventBus {
fn default() -> Self {
Self::new()
}
}
#[derive(Clone, Debug)]
pub struct Reporter {
channel: Arc<JobChannel>,
}
impl Reporter {
pub fn tool_call(&self, iteration: usize, name: &str, arguments: &str) {
self.channel.emit(Progress::ToolCall {
iteration,
name: name.to_string(),
arguments: excerpt(arguments, MAX_EVENT_ARGUMENTS),
});
}
/// Reports the end of a tool call.
pub fn tool_result(&self, iteration: usize, name: &str, truncated: bool, output: &str) {
self.channel.emit(Progress::ToolResult {
iteration,
name: name.to_string(),
truncated,
output: output.to_string(),
});
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_started_job_is_announced_and_subscribable() {
let bus = EventBus::new();
let mut lifecycle = bus.subscribe_lifecycle();
let _reporter = bus.start(1, "owner/repo", 7);
assert!(matches!(
lifecycle.try_recv().unwrap(),
JobEvent::Started { id: 1, pr: 7, .. }
));
assert!(bus.subscribe_job(1).is_some());
}
#[test]
fn progress_emitted_before_subscribing_is_replayed() {
let bus = EventBus::new();
let reporter = bus.start(1, "owner/repo", 7);
reporter.tool_call(1, "ls", "{}");
reporter.tool_result(1, "ls", false, "src\nmain.rs");
let subscription = bus.subscribe_job(1).unwrap();
assert_eq!(subscription.history.len(), 2);
assert!(matches!(
subscription.history[0],
Progress::ToolCall { iteration: 1, .. }
));
assert!(matches!(
&subscription.history[1],
Progress::ToolResult { name, .. } if name == "ls"
));
}
#[test]
fn progress_emitted_after_subscribing_arrives_live() {
let bus = EventBus::new();
let reporter = bus.start(1, "owner/repo", 7);
let mut subscription = bus.subscribe_job(1).unwrap();
reporter.tool_call(2, "grep", r#"{"pattern":"fn main"}"#);
assert!(matches!(
subscription.live.try_recv().unwrap(),
Progress::ToolCall { iteration: 2, name, .. } if name == "grep"
));
}
#[test]
fn finish_closes_the_channel_and_announces_the_review() {
let bus = EventBus::new();
let mut lifecycle = bus.subscribe_lifecycle();
let _reporter = bus.start(1, "owner/repo", 7);
let mut subscription = bus.subscribe_job(1).unwrap();
bus.finish(1, "owner/repo", 7, true, 3, 120);
assert!(bus.subscribe_job(1).is_none());
assert!(matches!(
subscription.live.try_recv().unwrap(),
Progress::Finished { ok: true }
));
let _started = lifecycle.try_recv().unwrap();
assert!(matches!(
lifecycle.try_recv().unwrap(),
JobEvent::ReviewDone {
id: 1,
ok: true,
comments: 3,
duration_ms: 120,
..
}
));
}
#[test]
fn a_failed_job_is_announced_as_failed() {
let bus = EventBus::new();
let mut lifecycle = bus.subscribe_lifecycle();
let _reporter = bus.start(1, "owner/repo", 7);
bus.finish(1, "owner/repo", 7, false, 0, 42);
let _started = lifecycle.try_recv().unwrap();
assert!(matches!(
lifecycle.try_recv().unwrap(),
JobEvent::Failed { id: 1, .. }
));
}
#[test]
fn a_skipped_job_is_announced_without_a_channel() {
let bus = EventBus::new();
let mut lifecycle = bus.subscribe_lifecycle();
bus.skip(1, "owner/repo", 7);
assert!(matches!(
lifecycle.try_recv().unwrap(),
JobEvent::Skipped { id: 1, pr: 7, .. }
));
assert!(bus.subscribe_job(1).is_none());
}
}
+227 -59
View File
@@ -1,19 +1,33 @@
use std::time::Duration; use std::time::Duration;
use serde::Deserialize; use futures_util::stream::TryStreamExt;
use serde::{Deserialize, Serialize};
use serde_json::{Value, json}; use serde_json::{Value, json};
use tracing::instrument; use tokio::io::AsyncReadExt;
use tokio_util::io::StreamReader;
use tracing::{instrument, warn};
use crate::{bot::ReviewResult, errors::AppError}; use crate::{
consts::MAX_DIFF_SIZE,
domain::review::{ReviewResult, ReviewSeverity, ReviewSide},
errors::AppError,
};
/// Page size requested when listing the files of a pull request.
const FILE_PAGE_SIZE: u64 = 50;
/// Maximum number of pages fetched for a pull request file list.
const MAX_FILE_PAGES: u64 = 10;
#[derive(Clone)] #[derive(Clone)]
pub struct GiteaAPI { pub struct GiteaAPI {
base_url: String, base_url: String,
token: String,
client: reqwest::Client, client: reqwest::Client,
} }
impl GiteaAPI { impl GiteaAPI {
pub fn new(base_url: &str, token: &str, timeout: u64) -> anyhow::Result<Self> { pub fn new(base_url: &str, token: &str, timeout: Duration) -> anyhow::Result<Self> {
let mut default_headers = reqwest::header::HeaderMap::new(); let mut default_headers = reqwest::header::HeaderMap::new();
default_headers.insert( default_headers.insert(
reqwest::header::HeaderName::from_static("authorization"), reqwest::header::HeaderName::from_static("authorization"),
@@ -22,14 +36,35 @@ impl GiteaAPI {
Ok(Self { Ok(Self {
base_url: String::from(base_url), base_url: String::from(base_url),
token: String::from(token),
client: reqwest::Client::builder() client: reqwest::Client::builder()
.timeout(Duration::from_secs(timeout)) .timeout(timeout)
.default_headers(default_headers) .default_headers(default_headers)
.build()?, .build()?,
}) })
} }
#[instrument(skip(self), err)] /// API token used to authenticate against Gitea.
pub fn token(&self) -> &str {
&self.token
}
#[instrument(skip(self))]
pub async fn get_authorized_user(&self) -> anyhow::Result<User> {
let url = format!("{}/api/v1/user", self.base_url);
let res = self.client.get(url).send().await?;
if !res.status().is_success() {
return Err(anyhow::anyhow!(
"Failed to get authorized user: {}",
res.status()
));
}
res.json::<User>().await.map_err(anyhow::Error::from)
}
#[instrument(skip(self))]
pub async fn comment( pub async fn comment(
&self, &self,
body: &str, body: &str,
@@ -57,7 +92,7 @@ impl GiteaAPI {
res.json::<Comment>().await.map_err(anyhow::Error::from) res.json::<Comment>().await.map_err(anyhow::Error::from)
} }
#[instrument(skip(self), err)] #[instrument(skip(self))]
pub async fn edit_comment( pub async fn edit_comment(
&self, &self,
body: &str, body: &str,
@@ -85,7 +120,7 @@ impl GiteaAPI {
Ok(()) Ok(())
} }
#[instrument(skip(self), err)] #[instrument(skip(self))]
pub async fn delete_comment(&self, full_name: &str, comment_id: u64) -> anyhow::Result<()> { pub async fn delete_comment(&self, full_name: &str, comment_id: u64) -> anyhow::Result<()> {
let url = format!( let url = format!(
"{}/api/v1/repos/{}/issues/comments/{}", "{}/api/v1/repos/{}/issues/comments/{}",
@@ -104,7 +139,71 @@ impl GiteaAPI {
Ok(()) Ok(())
} }
#[instrument(skip(self, review_result), err)] /// Raw unified diff of a pull request.
///
/// The API endpoint is used rather than the `diff_url` carried by the
/// webhook: that one points at a web route, which is session authenticated
/// and therefore does not serve private repositories to an API token.
#[instrument(skip(self))]
pub async fn pull_request_diff(&self, full_name: &str, index: u64) -> anyhow::Result<String> {
let url = format!(
"{}/api/v1/repos/{}/pulls/{}.diff",
self.base_url, full_name, index
);
let res = self.client.get(url).send().await?;
if !res.status().is_success() {
return Err(anyhow::anyhow!(
"Failed to download pull request diff: {}",
res.status()
));
}
read_capped(res).await
}
/// Files changed by a pull request, with their exact path and status.
#[instrument(skip(self))]
pub async fn pull_request_files(
&self,
full_name: &str,
index: u64,
) -> anyhow::Result<Vec<PullRequestFile>> {
let mut files = Vec::new();
for page in 1..=MAX_FILE_PAGES {
let url = format!(
"{}/api/v1/repos/{}/pulls/{}/files?limit={FILE_PAGE_SIZE}&page={page}",
self.base_url, full_name, index
);
let res = self.client.get(url).send().await?;
if !res.status().is_success() {
return Err(anyhow::anyhow!(
"Failed to list pull request files: {}",
res.status()
));
}
let page_files = res.json::<Vec<PullRequestFile>>().await?;
// The instance may cap the page size below the requested one, so a
// short page is not the end of the list: an empty one is.
if page_files.is_empty() {
return Ok(files);
}
files.extend(page_files);
}
warn!(files = files.len(), "Pull request file list was truncated");
Ok(files)
}
#[instrument(skip(self, review_result))]
pub async fn post_pull_request_review( pub async fn post_pull_request_review(
&self, &self,
review_result: &ReviewResult, review_result: &ReviewResult,
@@ -117,19 +216,31 @@ impl GiteaAPI {
self.base_url, full_name, index self.base_url, full_name, index
); );
let comments = &review_result let comments = review_result
.reviews .reviews
.iter() .iter()
.filter(|r| r.line.is_some()) .filter_map(|review| {
.map(|r| { let line = review.line?;
let path = r.filename.clone(); let path = review.filename.clone();
let line = r.line.unwrap_or(0); let severity = review
let body = r.message.clone(); .severity
.unwrap_or(ReviewSeverity::Maintainability)
.label();
let body = format!("**[{severity}]** {}", review.message);
json!({ // A line removed by the pull request only exists in the old
// version of the file, so it is anchored with `old_position`.
Some(match review.side {
Some(ReviewSide::Removed) => json!({
"path": path,
"old_position": line,
"body": body
}),
_ => json!({
"path": path, "path": path,
"new_position": line, "new_position": line,
"body": body "body": body
}),
}) })
}) })
.collect::<Vec<_>>(); .collect::<Vec<_>>();
@@ -153,7 +264,24 @@ impl GiteaAPI {
} }
} }
#[derive(Debug)] /// Reads a response body, refusing to buffer more than [`MAX_DIFF_SIZE`].
async fn read_capped(response: reqwest::Response) -> anyhow::Result<String> {
let stream = response.bytes_stream().map_err(std::io::Error::other);
let mut buf = Vec::with_capacity(MAX_DIFF_SIZE);
StreamReader::new(stream)
.take((MAX_DIFF_SIZE + 1) as u64)
.read_to_end(&mut buf)
.await?;
if buf.len() > MAX_DIFF_SIZE {
anyhow::bail!("Pull request diff exceeds the maximum allowed size of 1 MiB");
}
Ok(String::from_utf8_lossy(&buf).into_owned())
}
#[derive(Debug, Serialize, Deserialize, Clone)]
pub enum WebhookType { pub enum WebhookType {
Review(ReviewPayload), Review(ReviewPayload),
} }
@@ -164,9 +292,27 @@ impl WebhookType {
WebhookType::Review(payload) => payload.comment.id, WebhookType::Review(payload) => payload.comment.id,
} }
} }
pub fn event_type_str(&self) -> &'static str {
match self {
WebhookType::Review(_) => "review",
}
}
pub fn repository_full_name(&self) -> &str {
match self {
WebhookType::Review(payload) => &payload.repository.full_name,
}
}
pub fn pull_request_number(&self) -> u64 {
match self {
WebhookType::Review(payload) => payload.pull_request.number,
}
}
} }
#[derive(Deserialize, Debug)] #[derive(Deserialize, Serialize, Debug, Clone)]
pub struct ReviewPayload { pub struct ReviewPayload {
pub action: String, pub action: String,
pub pull_request: PullRequest, pub pull_request: PullRequest,
@@ -174,55 +320,67 @@ pub struct ReviewPayload {
pub comment: Comment, pub comment: Comment,
} }
#[derive(Deserialize, Debug)] impl ReviewPayload {
/// Checks that the comment is a new one that mentions the bot.
fn check_addressed_to(&self, bot_name: &str) -> Result<(), AppError> {
if self.action != "created" {
return Err(AppError::InvalidActionErr);
}
if !self.comment.body.starts_with(&format!("@{bot_name}")) {
return Err(AppError::UnauthorizedUserErr);
}
Ok(())
}
}
#[derive(Deserialize, Serialize, Debug, Clone)]
pub struct PullRequest { pub struct PullRequest {
pub id: u64,
pub diff_url: String,
pub number: u64, pub number: u64,
pub title: String, pub title: String,
} }
#[derive(Deserialize, Debug)] #[derive(Deserialize, Serialize, Debug, Clone)]
pub struct Comment { pub struct Comment {
pub id: u64, pub id: u64,
pub body: String, pub body: String,
pub user: User,
} }
#[derive(Deserialize, Debug)] #[derive(Deserialize, Debug)]
pub struct User { pub struct User {
pub id: u64, pub login: String,
} }
#[derive(Deserialize, Debug)] #[derive(Deserialize, Serialize, Debug, Clone)]
pub struct Repository { pub struct Repository {
pub full_name: String, pub full_name: String,
pub clone_url: String,
}
/// A file changed by a pull request, as reported by the API.
#[derive(Deserialize, Debug)]
pub struct PullRequestFile {
/// Path of the file in the new version of the repository.
pub filename: String,
/// Previous path, for a renamed file.
#[serde(default)]
pub previous_filename: Option<String>,
/// `added`, `modified`, `deleted`, `renamed`…
pub status: String,
} }
impl WebhookType { impl WebhookType {
pub fn from_event(event: &str, bot_name: &str, json: Value) -> Result<Self, AppError> { pub fn from_event(event: &str, bot_name: &str, json: Value) -> Result<Self, AppError> {
let wb = match event { match event {
"pull_request_comment" => Ok(WebhookType::Review(serde_json::from_value(json)?)), "pull_request_comment" => {
let payload: ReviewPayload = serde_json::from_value(json)?;
payload.check_addressed_to(bot_name)?;
Ok(WebhookType::Review(payload))
}
_ => Err(AppError::UnknownEventErr), _ => Err(AppError::UnknownEventErr),
}?;
let pr_body = match &wb {
WebhookType::Review(review_payload) => &review_payload.comment.body,
};
if !pr_body.starts_with(&format!("@{}", bot_name)) {
return Err(AppError::UnauthorizedUserErr);
} }
let action = match &wb {
WebhookType::Review(review_payload) => &review_payload.action,
};
if action != "created" {
return Err(AppError::InvalidActionErr);
}
Ok(wb)
} }
} }
@@ -242,13 +400,15 @@ mod tests {
"title": "My PR" "title": "My PR"
}, },
"repository": { "repository": {
"full_name": "owner/repo" "full_name": "owner/repo",
"clone_url": "https://github.com/owner/repo.git"
}, },
"comment": { "comment": {
"id": 7, "id": 7,
"body": "@test_bot LGTM", "body": "@test_bot LGTM",
"user": { "user": {
"id": 100 "id": 100,
"login": "test_user"
} }
} }
}); });
@@ -259,10 +419,8 @@ mod tests {
match result.unwrap() { match result.unwrap() {
WebhookType::Review(payload) => { WebhookType::Review(payload) => {
assert_eq!(payload.action, "created"); assert_eq!(payload.action, "created");
assert_eq!(payload.pull_request.id, 42);
assert_eq!(payload.comment.id, 7); assert_eq!(payload.comment.id, 7);
assert_eq!(payload.comment.body, "@test_bot LGTM"); assert_eq!(payload.comment.body, "@test_bot LGTM");
assert_eq!(payload.comment.user.id, 100);
} }
} }
} }
@@ -306,13 +464,15 @@ mod tests {
"title": "My PR" "title": "My PR"
}, },
"repository": { "repository": {
"full_name": "owner/repo" "full_name": "owner/repo",
"clone_url": "https://github.com/owner/repo.git"
}, },
"comment": { "comment": {
"id": 1, "id": 1,
"body": "@test_bot body", "body": "@test_bot body",
"user": { "user": {
"id": 1 "id": 1,
"login": "test_user"
} }
} }
}); });
@@ -337,23 +497,27 @@ mod tests {
"title": "My PR" "title": "My PR"
}, },
"repository": { "repository": {
"full_name": "owner/repo" "full_name": "owner/repo",
"clone_url": "https://github.com/owner/repo.git"
}, },
"comment": { "comment": {
"id": 12, "id": 12,
"body": "Needs work", "body": "Needs work",
"user": { "user": {
"id": 200 "id": 200,
"login": "test_user"
} }
} }
}); });
let payload: ReviewPayload = serde_json::from_value(json).unwrap(); let payload: ReviewPayload = serde_json::from_value(json).unwrap();
assert_eq!(payload.action, "created"); assert_eq!(payload.action, "created");
assert_eq!(payload.pull_request.id, 99);
assert_eq!(payload.comment.id, 12); assert_eq!(payload.comment.id, 12);
assert_eq!(payload.comment.body, "Needs work"); assert_eq!(payload.comment.body, "Needs work");
assert_eq!(payload.comment.user.id, 200); assert_eq!(
payload.repository.clone_url,
"https://github.com/owner/repo.git"
);
} }
#[test] #[test]
@@ -374,13 +538,15 @@ mod tests {
"title": "My PR" "title": "My PR"
}, },
"repository": { "repository": {
"full_name": "owner/repo" "full_name": "owner/repo",
"clone_url": "https://github.com/owner/repo.git"
}, },
"comment": { "comment": {
"id": 1, "id": 1,
"body": "@other_bot do something", "body": "@other_bot do something",
"user": { "user": {
"id": 1 "id": 1,
"login": "test_user"
} }
} }
}); });
@@ -400,13 +566,15 @@ mod tests {
"title": "My PR" "title": "My PR"
}, },
"repository": { "repository": {
"full_name": "owner/repo" "full_name": "owner/repo",
"clone_url": "https://github.com/owner/repo.git"
}, },
"comment": { "comment": {
"id": 1, "id": 1,
"body": "just a comment without bot mention", "body": "just a comment without bot mention",
"user": { "user": {
"id": 1 "id": 1,
"login": "test_user"
} }
} }
}); });
+273
View File
@@ -0,0 +1,273 @@
use std::sync::Arc;
use toasty::{Db, stmt::Value};
use tokio::sync::Notify;
use crate::{
gitea::WebhookType,
schemas::{Job, JobStatus},
};
/// The reviews to do, stored in PostgreSQL.
///
/// Steep runs as a single instance, so every job is pushed by this process: the
/// worker is woken up by [`JobQueue::push`] instead of polling the table. The table
/// only makes the queue survive a restart.
#[derive(Clone)]
pub struct JobQueue {
db: Db,
work: Arc<Notify>,
}
impl JobQueue {
pub fn new(db: Db) -> Self {
Self {
db,
work: Arc::new(Notify::new()),
}
}
pub async fn push(&self, webhook: WebhookType) -> toasty::Result<Job> {
let event_id = webhook.event_id();
let repo = webhook.repository_full_name().to_string();
let pr = webhook.pull_request_number();
let mut db = self.db.clone();
let job = toasty::create!(Job {
event_id,
repo,
pr,
payload: webhook,
status: JobStatus::Pending,
})
.exec(&mut db)
.await?;
// `notify_one` keeps a permit when nobody is waiting yet, so a job pushed
// while the worker is busy is not missed.
self.work.notify_one();
Ok(job)
}
/// Whether a job is already queued for this event.
pub async fn is_queued(&self, event_id: u64) -> toasty::Result<bool> {
let mut db = self.db.clone();
Ok(Job::filter_by_event_id(event_id)
.first()
.exec(&mut db)
.await?
.is_some())
}
/// Puts back in the queue the jobs a previous process left `processing`.
///
/// To call once at startup, before the worker starts: it is only correct because
/// no other instance can be working on those jobs.
pub async fn requeue_interrupted(&self) -> toasty::Result<()> {
let mut db = self.db.clone();
Job::update_by_status(JobStatus::Processing)
.status(JobStatus::Pending)
.exec(&mut db)
.await?;
Ok(())
}
/// Takes the oldest pending job and marks it `processing`, or returns `None` when
/// the queue is empty.
///
/// The selection and the update are a single statement, so a job cannot be taken
/// twice.
pub async fn claim_next(&self) -> toasty::Result<Option<Job>> {
let mut db = self.db.clone();
let rows = toasty::sql::query(
"UPDATE jobs
SET status = 'processing'
WHERE id = (
SELECT id FROM jobs
WHERE status = 'pending'
ORDER BY id
LIMIT 1
)
RETURNING id",
)
.exec(&mut db)
.await?;
let Some(Value::Record(row)) = rows.into_iter().next() else {
return Ok(None);
};
let id = u64::try_from(row.as_slice()[0].clone())?;
Ok(Some(Job::get_by_id(&mut db, id).await?))
}
/// Waits until a job may have been pushed.
pub async fn wait_for_work(&self) {
self.work.notified().await;
}
/// Removes a job from the queue, once its review is recorded.
pub async fn remove(&self, id: u64) -> toasty::Result<()> {
let mut db = self.db.clone();
Job::delete_by_id(&mut db, id).await?;
Ok(())
}
pub async fn list(&self, status: JobStatus) -> toasty::Result<Vec<Job>> {
let mut db = self.db.clone();
let jobs = Job::filter_by_status(status).exec(&mut db).await?;
Ok(jobs)
}
}
#[cfg(test)]
mod tests {
use std::time::Duration;
use tokio::sync::{Mutex, MutexGuard};
use super::*;
use crate::gitea::{Comment, PullRequest, Repository, ReviewPayload};
const FIRST_EVENT: u64 = 999_000_200;
fn webhook(event_id: u64) -> WebhookType {
WebhookType::Review(ReviewPayload {
action: "created".to_string(),
pull_request: PullRequest {
number: 1,
title: "Test".to_string(),
},
repository: Repository {
full_name: "owner/repo".to_string(),
clone_url: "https://example.com/owner/repo.git".to_string(),
},
comment: Comment {
id: event_id,
body: "@bot review".to_string(),
},
})
}
/// The tests claim whichever job is next, so they must not overlap: one would
/// claim the jobs of another.
async fn exclusive_queue() -> (MutexGuard<'static, ()>, JobQueue, Db) {
static LOCK: Mutex<()> = Mutex::const_new(());
let guard = LOCK.lock().await;
let db = crate::db::test_connect().await;
toasty::sql::statement(
"DELETE FROM jobs WHERE event_id >= 999000200 AND event_id < 999000300",
)
.exec(&mut db.clone())
.await
.unwrap();
(guard, JobQueue::new(db.clone()), db)
}
#[tokio::test]
#[ignore = "requires DATABASE_URL"]
async fn a_pushed_job_is_claimed_then_removed() {
let (_guard, queue, db) = exclusive_queue().await;
queue.push(webhook(FIRST_EVENT)).await.unwrap();
assert!(queue.is_queued(FIRST_EVENT).await.unwrap());
let job = queue.claim_next().await.unwrap().expect("a job to claim");
assert_eq!(job.event_id, FIRST_EVENT);
assert_eq!(job.status, JobStatus::Processing);
queue.remove(job.id).await.unwrap();
assert!(
Job::filter_by_event_id(FIRST_EVENT)
.first()
.exec(&mut db.clone())
.await
.unwrap()
.is_none()
);
}
#[tokio::test]
#[ignore = "requires DATABASE_URL"]
async fn a_duplicate_event_id_is_rejected() {
let (_guard, queue, _db) = exclusive_queue().await;
queue.push(webhook(FIRST_EVENT + 1)).await.unwrap();
assert!(queue.push(webhook(FIRST_EVENT + 1)).await.is_err());
}
#[tokio::test]
#[ignore = "requires DATABASE_URL"]
async fn a_claimed_job_is_not_claimed_again() {
let (_guard, queue, _db) = exclusive_queue().await;
queue.push(webhook(FIRST_EVENT + 2)).await.unwrap();
assert!(queue.claim_next().await.unwrap().is_some());
assert!(queue.claim_next().await.unwrap().is_none());
}
#[tokio::test]
#[ignore = "requires DATABASE_URL"]
async fn jobs_are_claimed_oldest_first() {
let (_guard, queue, _db) = exclusive_queue().await;
queue.push(webhook(FIRST_EVENT + 3)).await.unwrap();
queue.push(webhook(FIRST_EVENT + 4)).await.unwrap();
let first = queue.claim_next().await.unwrap().unwrap();
let second = queue.claim_next().await.unwrap().unwrap();
assert_eq!(first.event_id, FIRST_EVENT + 3);
assert_eq!(second.event_id, FIRST_EVENT + 4);
}
#[tokio::test]
#[ignore = "requires DATABASE_URL"]
async fn an_interrupted_job_is_requeued() {
let (_guard, queue, _db) = exclusive_queue().await;
queue.push(webhook(FIRST_EVENT + 5)).await.unwrap();
let job = queue.claim_next().await.unwrap().unwrap();
assert!(queue.claim_next().await.unwrap().is_none());
// The process died here: on restart, the job goes back to the queue.
queue.requeue_interrupted().await.unwrap();
let again = queue.claim_next().await.unwrap().expect("the job back");
assert_eq!(again.id, job.id);
}
#[tokio::test]
#[ignore = "requires DATABASE_URL"]
async fn waiting_for_work_returns_as_soon_as_a_job_is_pushed() {
let (_guard, queue, _db) = exclusive_queue().await;
let waiter = {
let queue = queue.clone();
tokio::spawn(async move { queue.wait_for_work().await })
};
tokio::time::sleep(Duration::from_millis(100)).await;
queue.push(webhook(FIRST_EVENT + 6)).await.unwrap();
tokio::time::timeout(Duration::from_secs(1), waiter)
.await
.expect("the worker was not woken up")
.unwrap();
}
}
+209
View File
@@ -0,0 +1,209 @@
use std::time::Duration;
use crate::{
bot::{Bot, BotDeps},
event_bus::EventBus,
gitea::GiteaAPI,
job_queue::JobQueue,
open_router::OpenRouterClient,
review_store::ReviewStore,
sandbox::SandboxConfig,
state::AppState,
};
use dotenvy::dotenv;
use tokio::signal::unix::{SignalKind, signal};
use tokio_util::sync::CancellationToken;
use tracing::{info, warn};
use tracing_subscriber::{EnvFilter, fmt, layer::SubscriberExt, util::SubscriberInitExt};
mod api;
mod bot;
mod bot_actions;
mod consts;
mod db;
mod domain;
mod env;
mod errors;
mod event_bus;
mod gitea;
mod job_queue;
mod metrics;
mod open_router;
mod review_store;
mod sandbox;
mod schemas;
mod state;
mod text;
/// Time left to the running reviews and the servers to stop cleanly.
///
/// Without it, a review stuck in an unresponsive sandbox would keep the process
/// alive until the supervisor kills it.
const SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(60);
fn main() -> anyhow::Result<()> {
dotenv().ok();
// TLS goes through rustls, which has no default crypto provider here: the
// workspace compiles `ring` only, and it is installed once, for every client
// (reqwest, sentry).
rustls::crypto::ring::default_provider()
.install_default()
.map_err(|_| anyhow::anyhow!("a rustls crypto provider is already installed"))?;
// Migration subcommand: `steep-server migration <generate|apply|...>`.
if std::env::args().nth(1).as_deref() == Some("migration") {
return tokio::runtime::Runtime::new()?.block_on(run_migrations());
}
tracing_subscriber::registry()
.with(fmt::layer())
.with(
EnvFilter::try_from_default_env() // reads RUST_LOG from the environment
.unwrap_or_else(|_| EnvFilter::new("info")),
)
.init();
let _sentry_guard = if let Ok(sentry_dsn) = env::try_get_env("SENTRY_DSN") {
info!("Initialize sentry");
Some(sentry::init(
sentry::ClientOptions::new()
.dsn(&sentry_dsn)
.maybe_release(sentry::release_name!())
.send_default_pii(true),
))
} else {
warn!("SENTRY_DSN not set, sentry will not be initialized");
None
};
tokio::runtime::Runtime::new()?.block_on(run())
}
async fn run_migrations() -> anyhow::Result<()> {
let database_url = env::try_get_env("DATABASE_URL")?;
let db = db::open(&database_url).await?;
let migrations = match std::env::var("MIGRATIONS_DIR") {
Ok(dir) => std::path::PathBuf::from(dir),
Err(_) => std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("toasty"),
};
let config =
toasty_cli::Config::new().migration(toasty_cli::MigrationConfig::new().path(migrations));
toasty_cli::ToastyCli::with_config(db, config)
.parse_and_run()
.await?;
Ok(())
}
async fn run() -> anyhow::Result<()> {
let config = env::load_config()?;
if let Some(metric_bind_addr) = &config.metrics_bind_addr {
metrics::install(metric_bind_addr)?;
}
let gitea_api = GiteaAPI::new(&config.gitea_url, &config.gitea_token, config.gitea_timeout)?;
let gitea_user = gitea_api.get_authorized_user().await?;
info!(
port = config.http_port,
model = %config.open_router_model,
gitea_url = %config.gitea_url,
bot_name = %gitea_user.login,
"Starting Steep"
);
let open_router_client = OpenRouterClient::new(
&config.open_router_api_key,
&config.open_router_model,
config.open_router_timeout,
)?;
let shutdown = CancellationToken::new();
let sandbox = SandboxConfig {
runtime: devcontainer_rs::ContainerRuntime::connect()?,
max_iterations: config.sandbox_max_iterations,
};
if !sandbox.runtime.available().await {
warn!(
endpoint = sandbox.runtime.endpoint(),
"Container daemon is not reachable, every review will fail"
);
}
let db = db::open(&config.database_url).await?;
let reviews = ReviewStore::new(db.clone());
let queue = JobQueue::new(db);
queue.requeue_interrupted().await?;
let event_bus = EventBus::new();
let bot = Bot::new(
BotDeps {
gitea: gitea_api,
open_router: open_router_client,
sandbox,
reviews: reviews.clone(),
queue: queue.clone(),
event_bus: event_bus.clone(),
},
config.bot_max_concurrent,
);
let app_state = AppState {
queue,
reviews,
event_bus,
bot_name: gitea_user.login.clone(),
config,
shutdown: shutdown.clone(),
};
let signal = async {
let mut sigterm = signal(SignalKind::terminate())?;
let mut sigint = signal(SignalKind::interrupt())?;
tokio::select! {
_ = sigterm.recv() => info!("Received SIGTERM"),
_ = sigint.recv() => info!("Received SIGINT"),
}
info!("Shutting down...");
shutdown.cancel();
anyhow::Ok(())
};
let shutdown_deadline = async {
shutdown.cancelled().await;
tokio::time::sleep(SHUTDOWN_TIMEOUT).await;
};
tokio::select! {
result = async {
tokio::try_join!(
bot.start(shutdown.clone()),
api::start(app_state),
signal
)
} => {
result?;
}
() = shutdown_deadline => {
warn!(
timeout = ?SHUTDOWN_TIMEOUT,
"Shutdown did not finish in time, exiting anyway"
);
}
}
info!("Shutdown complete");
Ok(())
}
+96
View File
@@ -0,0 +1,96 @@
use std::{net::SocketAddr, str::FromStr};
use metrics::{Unit, counter, describe_counter, describe_gauge, gauge};
pub fn webhook_received(event_type: &str) {
counter!("steep_webhooks_received_total", "event_type" => event_type.to_string()).increment(1);
}
pub fn webhook_duplicate(event_type: &str) {
counter!("steep_webhooks_duplicate_total", "event_type" => event_type.to_string()).increment(1);
}
pub fn webhook_enqueue_failed(event_type: &str) {
counter!("steep_webhooks_enqueue_failed_total", "event_type" => event_type.to_string())
.increment(1);
}
pub fn increment_task_active() {
gauge!("steep_bot_tasks_active").increment(1.0);
}
pub fn decrement_task_active() {
gauge!("steep_bot_tasks_active").decrement(1.0);
}
pub fn task_skipped(event_type: &str) {
counter!("steep_bot_tasks_skipped_total", "event_type" => event_type.to_string()).increment(1);
}
pub fn task_completed(event_type: &str) {
counter!("steep_bot_tasks_completed_total", "event_type" => event_type.to_string())
.increment(1);
}
pub fn task_failed(event_type: &str) {
counter!("steep_bot_tasks_failed_total", "event_type" => event_type.to_string()).increment(1);
}
pub fn openrouter_cost_usd(cost: f64) {
counter!("steep_openrouter_cost_cents_total").increment((cost * 100.0).round() as u64);
}
pub fn describe() {
describe_counter!(
"steep_webhooks_received_total",
Unit::Count,
"Total webhooks received"
);
describe_counter!(
"steep_webhooks_duplicate_total",
Unit::Count,
"Webhooks rejected as duplicates"
);
describe_counter!(
"steep_webhooks_enqueue_failed_total",
Unit::Count,
"Webhooks dropped because enqueueing to PostgreSQL failed"
);
describe_gauge!(
"steep_bot_tasks_active",
Unit::Count,
"Bot tasks currently in progress"
);
describe_counter!(
"steep_bot_tasks_skipped_total",
Unit::Count,
"Reviews skipped because they were already done"
);
describe_counter!(
"steep_bot_tasks_completed_total",
Unit::Count,
"Bot tasks completed successfully"
);
describe_counter!(
"steep_bot_tasks_failed_total",
Unit::Count,
"Bot tasks that failed"
);
describe_counter!(
"steep_openrouter_cost_cents_total",
Unit::Count,
"Total OpenRouter cost in cents (divide by 100 for USD)"
);
}
pub fn install(bind_addr: &str) -> anyhow::Result<()> {
describe();
let builder = metrics_exporter_prometheus::PrometheusBuilder::new();
builder
.with_http_listener(SocketAddr::from_str(bind_addr)?)
.install()?;
tracing::info!(bind_addr, "Prometheus metrics exporter installed");
Ok(())
}
+476
View File
@@ -0,0 +1,476 @@
//! Minimal OpenRouter chat-completions client.
//!
//! Steep only needs a non-streaming `POST /chat/completions` with optional
//! tool calling, so the wire types are implemented in-tree instead of pulling a
//! third-party SDK (and its own `reqwest` version) into the workspace.
//!
//! Only the response fields Steep consumes are modelled: `content`,
//! `tool_calls` and `usage.cost`. Unknown fields are ignored.
use std::time::Duration;
use anyhow::Context;
use serde::{Deserialize, Serialize};
use serde_json::Value;
use tracing::instrument;
/// OpenRouter API root, version prefix included.
const BASE_URL: &str = "https://openrouter.ai/api/v1";
/// What the model is allowed to do on a given turn.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ToolChoice {
/// The model decides whether to call a tool.
Auto,
/// The model must answer, without calling any tool. The tools stay declared,
/// so the conversation remains the same shape as on the other turns.
None,
}
impl ToolChoice {
fn as_str(self) -> &'static str {
match self {
Self::Auto => "auto",
Self::None => "none",
}
}
}
/// Result of a completion that may contain tool calls.
pub struct ToolChatResult {
pub message: Option<String>,
pub tool_calls: Vec<ToolCall>,
pub cost: Option<f64>,
}
#[derive(Serialize, Deserialize, Debug, Clone, Copy, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum Role {
System,
User,
Assistant,
Tool,
}
/// A single turn of the conversation.
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct Message {
pub role: Role,
#[serde(skip_serializing_if = "Option::is_none")]
pub content: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tool_calls: Option<Vec<ToolCall>>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tool_call_id: Option<String>,
}
impl Message {
pub fn new(role: Role, content: impl Into<String>) -> Self {
Self {
role,
content: Some(content.into()),
tool_calls: None,
tool_call_id: None,
}
}
/// Assistant turn requesting tool calls. Models often answer with tool
/// calls but no text, in which case the content is sent as `null`.
pub fn assistant_with_tool_calls(
content: impl Into<String>,
tool_calls: Vec<ToolCall>,
) -> Self {
let content = content.into();
Self {
role: Role::Assistant,
content: (!content.is_empty()).then_some(content),
tool_calls: Some(tool_calls),
tool_call_id: None,
}
}
/// Result of a tool call, linked to the request by `tool_call_id`.
pub fn tool_response(tool_call_id: impl Into<String>, content: impl Into<String>) -> Self {
Self {
role: Role::Tool,
content: Some(content.into()),
tool_calls: None,
tool_call_id: Some(tool_call_id.into()),
}
}
}
/// A tool the model may call.
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct Tool {
#[serde(rename = "type", default = "function_type")]
pub kind: String,
pub function: FunctionDefinition,
}
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct FunctionDefinition {
pub name: String,
pub description: String,
/// JSON schema describing the accepted arguments.
pub parameters: Value,
}
impl Tool {
pub fn new(name: impl Into<String>, description: impl Into<String>, parameters: Value) -> Self {
Self {
kind: function_type(),
function: FunctionDefinition {
name: name.into(),
description: description.into(),
parameters,
},
}
}
}
/// A tool call requested by the model.
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct ToolCall {
pub id: String,
#[serde(rename = "type", default = "function_type")]
pub kind: String,
pub function: FunctionCall,
}
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct FunctionCall {
pub name: String,
/// Arguments as a JSON-encoded string. Kept verbatim so that echoing the
/// call back into the conversation does not re-encode or corrupt it.
pub arguments: String,
}
impl ToolCall {
pub fn id(&self) -> &str {
&self.id
}
pub fn name(&self) -> &str {
&self.function.name
}
pub fn arguments_json(&self) -> &str {
&self.function.arguments
}
}
fn function_type() -> String {
String::from("function")
}
#[derive(Serialize)]
struct ChatRequest<'a> {
model: &'a str,
messages: &'a [Message],
reasoning: Reasoning,
tools: &'a [Tool],
tool_choice: &'a str,
}
/// OpenRouter reasoning config; Steep always asks for reasoning.
#[derive(Serialize)]
struct Reasoning {
enabled: bool,
}
#[derive(Deserialize)]
struct ChatResponse {
#[serde(default)]
choices: Vec<Choice>,
#[serde(default)]
usage: Option<Usage>,
}
#[derive(Deserialize)]
struct Choice {
message: ResponseMessage,
}
#[derive(Deserialize)]
struct ResponseMessage {
#[serde(default)]
content: Option<Content>,
#[serde(default)]
tool_calls: Option<Vec<ToolCall>>,
}
/// Message content is a plain string for most models and a list of typed parts
/// for others; both are collapsed to text.
#[derive(Deserialize)]
#[serde(untagged)]
enum Content {
Text(String),
Parts(Vec<ContentPart>),
}
#[derive(Deserialize)]
struct ContentPart {
#[serde(default)]
text: Option<String>,
}
impl Content {
fn into_text(self) -> String {
match self {
Self::Text(text) => text,
Self::Parts(parts) => parts
.into_iter()
.filter_map(|part| part.text)
.collect::<Vec<_>>()
.join(""),
}
}
}
#[derive(Deserialize)]
struct Usage {
#[serde(default)]
cost: Option<f64>,
}
/// Error payload returned by OpenRouter for a failed request.
#[derive(Deserialize)]
struct ErrorResponse {
error: ErrorDetail,
}
#[derive(Deserialize)]
struct ErrorDetail {
message: String,
}
#[derive(Clone)]
pub struct OpenRouterClient {
client: reqwest::Client,
api_key: String,
model: String,
}
impl OpenRouterClient {
pub fn new(token: &str, model: &str, timeout: Duration) -> anyhow::Result<Self> {
Ok(Self {
client: reqwest::Client::builder().timeout(timeout).build()?,
api_key: String::from(token),
model: String::from(model),
})
}
/// The model every completion is requested from.
pub fn model(&self) -> &str {
&self.model
}
/// Sends a conversation with tool definitions and returns either a final
/// message or the tool calls requested by the model.
///
/// The conversation is borrowed rather than taken by value: the caller keeps
/// appending to it between iterations, and a copy per iteration would be pure
/// waste.
#[instrument(skip(self, messages, tools), err)]
pub async fn chat_with_tools(
&self,
messages: &[Message],
tools: &[Tool],
tool_choice: ToolChoice,
) -> anyhow::Result<ToolChatResult> {
let response = self.complete(messages, tools, tool_choice).await?;
let cost = response.usage.and_then(|usage| usage.cost);
let message = response
.choices
.into_iter()
.next()
.ok_or_else(|| anyhow::anyhow!("No choices in response"))?
.message;
Ok(ToolChatResult {
message: message.content.map(Content::into_text),
tool_calls: message.tool_calls.unwrap_or_default(),
cost,
})
}
async fn complete(
&self,
messages: &[Message],
tools: &[Tool],
tool_choice: ToolChoice,
) -> anyhow::Result<ChatResponse> {
let request = ChatRequest {
model: &self.model,
messages,
reasoning: Reasoning { enabled: true },
tools,
tool_choice: tool_choice.as_str(),
};
let response = self
.client
.post(format!("{BASE_URL}/chat/completions"))
.bearer_auth(&self.api_key)
.json(&request)
.send()
.await
.context("failed to reach OpenRouter")?;
let status = response.status();
if !status.is_success() {
let body = response.text().await.unwrap_or_default();
anyhow::bail!("OpenRouter returned {status}: {}", error_message(&body));
}
response
.json::<ChatResponse>()
.await
.context("invalid OpenRouter response")
}
}
/// Extracts the message from an OpenRouter error body, falling back to the raw
/// body when it is not the expected JSON shape.
fn error_message(body: &str) -> String {
serde_json::from_str::<ErrorResponse>(body)
.map(|response| response.error.message)
.unwrap_or_else(|_| body.trim().to_string())
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
/// Builds a tool call the way the API returns one, so the fixture also
/// covers deserialization.
fn tool_call(id: &str, name: &str, arguments: &str) -> ToolCall {
serde_json::from_value(json!({
"id": id,
"type": "function",
"function": { "name": name, "arguments": arguments }
}))
.unwrap()
}
#[test]
fn tool_call_arguments_are_kept_verbatim() {
let call = tool_call("call_1", "grep", r#"{"pattern":"fn main"}"#);
let message = Message::assistant_with_tool_calls("", vec![call]);
let serialized = serde_json::to_value(&message).unwrap();
assert_eq!(serialized["content"], Value::Null);
assert_eq!(serialized["tool_calls"][0]["type"], "function");
assert_eq!(serialized["tool_calls"][0]["function"]["name"], "grep");
assert_eq!(
serialized["tool_calls"][0]["function"]["arguments"],
r#"{"pattern":"fn main"}"#
);
}
#[test]
fn tool_response_carries_role_and_tool_call_id() {
let message = Message::tool_response("call_1", "src/main.rs");
let serialized = serde_json::to_value(&message).unwrap();
assert_eq!(serialized["role"], "tool");
assert_eq!(serialized["tool_call_id"], "call_1");
assert_eq!(serialized["content"], "src/main.rs");
}
#[test]
fn request_serializes_tool_choice_and_reasoning() {
let messages = [Message::new(Role::User, "hi")];
let tools = [Tool::new("ls", "List files", json!({"type": "object"}))];
let request = serde_json::to_value(ChatRequest {
model: "some/model",
messages: &messages,
reasoning: Reasoning { enabled: true },
tools: &tools,
tool_choice: ToolChoice::Auto.as_str(),
})
.unwrap();
assert_eq!(request["model"], "some/model");
assert_eq!(request["reasoning"]["enabled"], true);
assert_eq!(request["tool_choice"], "auto");
assert_eq!(request["tools"][0]["function"]["name"], "ls");
assert_eq!(request["messages"][0]["role"], "user");
}
#[test]
fn a_forced_answer_turn_disables_the_tools() {
assert_eq!(ToolChoice::None.as_str(), "none");
assert_eq!(ToolChoice::Auto.as_str(), "auto");
}
#[test]
fn response_parses_tool_calls_and_cost() {
let response: ChatResponse = serde_json::from_value(json!({
"choices": [{
"message": {
"role": "assistant",
"content": null,
"tool_calls": [{
"id": "call_1",
"type": "function",
"function": { "name": "ls", "arguments": "{\"path\":\"src\"}" }
}]
}
}],
"usage": { "cost": 0.0021 }
}))
.unwrap();
assert_eq!(response.usage.and_then(|usage| usage.cost), Some(0.0021));
let message = response.choices.into_iter().next().unwrap().message;
assert!(message.content.is_none());
let calls = message.tool_calls.unwrap();
assert_eq!(calls[0].name(), "ls");
assert_eq!(calls[0].id(), "call_1");
assert_eq!(calls[0].arguments_json(), r#"{"path":"src"}"#);
}
#[test]
fn response_parses_parts_content() {
let response: ChatResponse = serde_json::from_value(json!({
"choices": [{
"message": {
"role": "assistant",
"content": [
{ "type": "text", "text": "hello " },
{ "type": "text", "text": "world" }
]
}
}]
}))
.unwrap();
assert!(response.usage.is_none());
let message = response.choices.into_iter().next().unwrap().message;
assert_eq!(message.content.unwrap().into_text(), "hello world");
}
#[test]
fn error_message_prefers_api_message() {
let body = r#"{"error":{"message":"No auth credentials found","code":401}}"#;
assert_eq!(error_message(body), "No auth credentials found");
}
#[test]
fn error_message_falls_back_to_raw_body() {
assert_eq!(
error_message(" <html>bad gateway</html> "),
"<html>bad gateway</html>"
);
}
}
+114
View File
@@ -0,0 +1,114 @@
use toasty::Db;
use crate::schemas::Review;
/// What is kept of a processed review.
pub struct ReviewRecord {
pub event_id: u64,
pub repo: String,
pub pr: u64,
pub ok: bool,
pub model: String,
pub markdown: String,
pub cost: Option<f64>,
pub comments: u64,
pub duration_ms: u64,
}
/// The history of the reviews Steep has processed.
///
/// It also is what keeps an event from being reviewed twice.
#[derive(Clone)]
pub struct ReviewStore {
db: Db,
}
impl ReviewStore {
pub fn new(db: Db) -> Self {
Self { db }
}
pub async fn is_reviewed(&self, event_id: u64) -> toasty::Result<bool> {
let mut db = self.db.clone();
Ok(Review::filter_by_event_id(event_id)
.first()
.exec(&mut db)
.await?
.is_some())
}
/// Records a processed review, replacing the one of the same event if any.
pub async fn record(&self, record: ReviewRecord) -> toasty::Result<()> {
let mut db = self.db.clone();
Review::upsert_by_event_id(record.event_id)
.repo(record.repo)
.pr(record.pr)
.ok(record.ok)
.model(record.model)
.markdown(record.markdown)
.cost(record.cost)
.comments(record.comments)
.duration_ms(record.duration_ms)
.exec(&mut db)
.await?;
Ok(())
}
pub async fn list(&self) -> toasty::Result<Vec<Review>> {
let mut db = self.db.clone();
let reviews = Review::all().exec(&mut db).await?;
Ok(reviews)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
#[ignore = "requires DATABASE_URL"]
async fn a_recorded_review_is_reviewed() {
let db = crate::db::test_connect().await;
let store = ReviewStore::new(db.clone());
let event_id = 999_000_101;
Review::delete_by_event_id(&mut db.clone(), event_id)
.await
.ok();
assert!(!store.is_reviewed(event_id).await.unwrap());
store
.record(ReviewRecord {
event_id,
repo: "owner/repo".to_string(),
pr: 1,
ok: true,
model: "test/model".to_string(),
markdown: "looks good".to_string(),
cost: Some(0.01),
comments: 2,
duration_ms: 5,
})
.await
.unwrap();
assert!(store.is_reviewed(event_id).await.unwrap());
assert!(
store
.list()
.await
.unwrap()
.iter()
.any(|r| r.event_id == event_id)
);
Review::delete_by_event_id(&mut db.clone(), event_id)
.await
.unwrap();
}
}
+311
View File
@@ -0,0 +1,311 @@
//! Tool-calling loop driving the model against a [`Sandbox`].
//!
//! The loop sends the conversation and the available tools to OpenRouter,
//! executes any requested tool call inside the sandbox and feeds the results
//! back, until the model produces a final message or the iteration budget is
//! exhausted.
use std::str::FromStr;
use std::time::Instant;
use anyhow::Context;
use serde_json::Value;
use tracing::{info, warn};
use crate::{
event_bus::Reporter,
open_router::{Message, OpenRouterClient, Role, Tool, ToolCall, ToolChoice},
sandbox::{Sandbox, tools},
text::{excerpt, truncate_bytes},
};
/// Number of characters of the arguments of a tool call kept in the logs.
const MAX_LOGGED_ARGUMENTS: usize = 200;
/// Number of characters of a tool result kept in the logs.
///
/// Sizes alone make a broken tool invisible: a command that fails returns a short
/// `error: …` instead of the expected content, which is exactly what this excerpt
/// makes obvious.
const MAX_LOGGED_OUTPUT: usize = 120;
/// Maximum number of bytes of a tool result handed to the model.
///
/// A result stays in the conversation and is re-sent on every following
/// iteration, so an unbounded one (a whole file, a match on every line) inflates
/// the context and the cost of the whole rest of the run — and can overflow the
/// model context window outright.
const MAX_TOOL_OUTPUT_BYTES: usize = 32 * 1024;
/// Asked of the model when its answer does not satisfy the caller's contract.
const RETRY_PROMPT: &str = "
Your answer is not valid: {error}
Answer again with the requested format only: a raw json object, without any
markdown code fence and without any text before or after it.
";
/// Asked of the model on the last turn, in place of one more exploration turn.
///
/// Without it, a model still exploring on its last turn would ask for another
/// tool, and the run would end on the iteration budget with no answer at all.
const FINAL_PROMPT: &str = "
You are out of turns: this is your last one. Answer now, with the requested
format, from what you have already gathered, without calling any tool.
";
/// Final output of an agent run.
pub struct AgentResult<A> {
/// Answer of the model, parsed into the type the caller asked for.
pub answer: A,
pub cost: Option<f64>,
pub iterations: usize,
}
pub async fn run<A>(
open_router: &OpenRouterClient,
sandbox: &Sandbox,
tool_definitions: Vec<Tool>,
system_prompt: &str,
user_prompt: &str,
max_iterations: usize,
reporter: &Reporter,
) -> anyhow::Result<AgentResult<A>>
where
A: FromStr,
A::Err: std::fmt::Display,
{
let mut messages = vec![
Message::new(Role::System, system_prompt),
Message::new(Role::User, user_prompt),
];
info!(
tools = %tool_definitions
.iter()
.map(|tool| tool.function.name.as_str())
.collect::<Vec<_>>()
.join(", "),
max_iterations,
"Starting tool-calling loop"
);
let mut total_cost = 0.0_f64;
let mut has_cost = false;
let mut last_rejection: Option<String> = None;
for iteration in 1..=max_iterations {
// The last turn is no longer an exploration turn: the model must give its
// answer, with what it has already seen.
let last = iteration == max_iterations;
if last {
info!(iteration, "Last turn: asking for the final answer");
messages.push(Message::new(Role::User, FINAL_PROMPT));
}
let started = Instant::now();
let response = open_router
.chat_with_tools(
&messages,
&tool_definitions,
if last {
ToolChoice::None
} else {
ToolChoice::Auto
},
)
.await?;
if let Some(cost) = response.cost {
total_cost += cost;
has_cost = true;
}
info!(
iteration,
max_iterations,
last_turn = last,
tool_calls = response.tool_calls.len(),
answer_bytes = response.message.as_deref().map_or(0, str::len),
elapsed_ms = started.elapsed().as_millis() as u64,
"Model answered"
);
// On the last turn, tools are refused: even if the provider returns them
// anyway, we do not run them and try the answer instead.
if last || response.tool_calls.is_empty() {
let answer = response.message.unwrap_or_default();
match answer.parse::<A>() {
Ok(parsed) => {
return Ok(AgentResult {
answer: parsed,
cost: has_cost.then_some(total_cost),
iterations: iteration,
});
}
Err(err) => {
warn!(iteration, %err, "Model answer was rejected, asking for another one");
// Keep the rejected answer in the history, so that the model
// sees what it has to fix.
messages.push(Message::new(Role::Assistant, answer));
messages.push(Message::new(
Role::User,
RETRY_PROMPT.replace("{error}", &err.to_string()),
));
last_rejection = Some(err.to_string());
continue;
}
}
}
messages.push(Message::assistant_with_tool_calls(
response.message.unwrap_or_default(),
response.tool_calls.clone(),
));
for call in &response.tool_calls {
reporter.tool_call(iteration, call.name(), call.arguments_json());
let started = Instant::now();
let (content, truncated) = execute(sandbox, call).await;
reporter.tool_result(iteration, call.name(), truncated, &content);
info!(
iteration,
tool = call.name(),
arguments = %excerpt(call.arguments_json(), MAX_LOGGED_ARGUMENTS),
output_bytes = content.len(),
truncated,
// Line breaks would hurt the readability of a log line.
output = %excerpt(&content, MAX_LOGGED_OUTPUT).replace('\n', " "),
elapsed_ms = started.elapsed().as_millis() as u64,
"Tool call finished"
);
messages.push(Message::tool_response(call.id(), content));
}
}
warn!(
max_iterations,
"Agent used its last turn without a valid answer"
);
match last_rejection {
Some(error) => anyhow::bail!(
"agent exceeded the maximum number of iterations ({max_iterations}), \
last answer rejected: {error}"
),
None => anyhow::bail!("agent exceeded the maximum number of iterations ({max_iterations})"),
}
}
/// Runs a tool call and returns its result for the model, along with whether the
/// result had to be truncated.
async fn execute(sandbox: &Sandbox, call: &ToolCall) -> (String, bool) {
let args = match parse_args(call) {
Ok(args) => args,
Err(err) => return cap_output(format!("error: {err}")),
};
match tools::dispatch(sandbox, call.name(), &args).await {
Ok(output) => cap_output(output),
Err(err) => cap_output(format!("error: {err}")),
}
}
/// Bounds [`MAX_TOOL_OUTPUT_BYTES`] of a tool result, telling the model what was
/// dropped so that it can narrow its request.
fn cap_output(mut output: String) -> (String, bool) {
let total = output.len();
if !truncate_bytes(&mut output, MAX_TOOL_OUTPUT_BYTES) {
return (output, false);
}
let kept = output.len();
output.push_str(&format!(
"\n… output truncated: {total} bytes in total, the first {kept} are shown. \
Narrow the request (path, pattern or line range) to see the rest."
));
(output, true)
}
fn parse_args(call: &ToolCall) -> anyhow::Result<Value> {
let raw = call.arguments_json().trim();
if raw.is_empty() {
return Ok(Value::Object(serde_json::Map::new()));
}
serde_json::from_str(raw)
.with_context(|| format!("invalid arguments for tool `{}`", call.name()))
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
/// Builds a tool call the way the API returns one, so the fixture also
/// covers deserialization.
fn tool_call(name: &str, arguments: &str) -> ToolCall {
serde_json::from_value(json!({
"id": "call_1",
"type": "function",
"function": { "name": name, "arguments": arguments }
}))
.unwrap()
}
#[test]
fn parse_args_accepts_empty_arguments() {
let call = tool_call("ls", "");
assert_eq!(
parse_args(&call).unwrap(),
Value::Object(serde_json::Map::new())
);
}
#[test]
fn parse_args_parses_json_object() {
let call = tool_call("ls", r#"{"path":"src"}"#);
assert_eq!(parse_args(&call).unwrap()["path"], "src");
}
#[test]
fn parse_args_rejects_invalid_json() {
let call = tool_call("ls", "not json");
assert!(parse_args(&call).is_err());
}
#[test]
fn a_small_tool_output_is_kept_as_is() {
let (content, truncated) = cap_output(String::from("src/main.rs"));
assert_eq!(content, "src/main.rs");
assert!(!truncated);
}
#[test]
fn a_large_tool_output_is_truncated_and_told_to_the_model() {
let (content, truncated) = cap_output("a".repeat(MAX_TOOL_OUTPUT_BYTES + 1));
assert!(truncated);
assert!(content.starts_with(&"a".repeat(MAX_TOOL_OUTPUT_BYTES)));
assert!(content.contains("output truncated"));
}
#[test]
fn a_truncated_tool_output_stays_valid_utf8() {
let (content, truncated) = cap_output("é".repeat(MAX_TOOL_OUTPUT_BYTES));
assert!(truncated);
assert!(content.starts_with('é'));
}
}
+295
View File
@@ -0,0 +1,295 @@
//! The [`Sandbox`]: a pull request clone running inside an ephemeral
//! devcontainer, and its configuration.
use std::{
path::{Path, PathBuf},
process::Stdio,
};
use anyhow::Context;
use devcontainer_rs::{Container, ContainerRuntime, ExecOutput};
use tempfile::TempDir;
use tracing::{info, instrument};
use crate::event_bus::Reporter;
/// Devcontainer locations recognized within a repository, in priority order.
const DEVCONTAINER_PATHS: [&str; 2] = [".devcontainer/devcontainer.json", ".devcontainer.json"];
/// Sandbox-related runtime configuration.
#[derive(Clone)]
pub struct SandboxConfig {
/// Client of the container daemon that runs the sandbox.
pub runtime: ContainerRuntime,
/// Maximum number of tool-calling iterations per agent run.
pub max_iterations: usize,
}
/// A cloned repository running inside an ephemeral devcontainer.
pub struct Sandbox {
// Owns the temporary directory; dropping it cleans up the clone.
_workspace: TempDir,
/// Path of the clone, as copied into the container.
repo_dir: PathBuf,
container: Container,
}
impl Sandbox {
/// Clones the pull request head, builds the devcontainer and starts it.
///
/// The clone is PR-aware: it fetches `refs/pull/<number>/head`, which works
/// for both same-repository and forked pull requests.
#[instrument(skip(runtime, token), fields(pr = pull_request_number))]
pub async fn create(
runtime: &ContainerRuntime,
clone_url: &str,
token: &str,
pull_request_number: u64,
reporter: &Reporter,
) -> anyhow::Result<Self> {
let workspace = tempfile::tempdir().context("failed to create sandbox workspace")?;
let repo_dir = workspace.path().join("repo");
clone_pull_request(clone_url, token, pull_request_number, &repo_dir).await?;
make_readable(&repo_dir).await?;
let devcontainer = match find_devcontainer(&repo_dir) {
Some(path) => devcontainer_rs::parse(&path).await?,
None => {
info!(
"no devcontainer found in the pull request, falling back to the default image"
);
devcontainer_rs::DevContainer::default()
}
};
let container = devcontainer.up(runtime, &repo_dir).await?;
let sandbox = Self {
_workspace: workspace,
repo_dir,
container,
};
if let Err(err) = sandbox.check_workspace().await {
let _ = sandbox.container.remove().await;
return Err(err);
}
Ok(sandbox)
}
/// Checks that the clone is actually visible inside the container.
///
/// Without this check, an empty workspace (for example a daemon that could not
/// receive the clone) makes every tool fail; the model then chains failed calls
/// until the iteration budget is spent, without ever being able to review
/// anything.
async fn check_workspace(&self) -> anyhow::Result<()> {
let workspace_folder = self.workspace_folder();
let probe = self
.container
.exec(&["ls", "-A", "--", workspace_folder])
.await
.with_context(|| format!("failed to list `{workspace_folder}` in the sandbox"))?;
if !probe.success() {
let details = self.diagnose_workspace().await;
anyhow::bail!(
"the sandbox cannot list `{workspace_folder}`: {} ({details})",
probe.stderr.trim()
);
}
if probe.stdout.trim().is_empty() {
anyhow::bail!(
"the sandbox workspace `{workspace_folder}` is empty: the container daemon does not see the clone"
);
}
info!(
clone = %self.repo_dir.display(),
workspace = %workspace_folder,
entries = probe.stdout.lines().count(),
"Sandbox workspace is readable"
);
Ok(())
}
/// Gathers enough information to explain a refused access to the workspace.
///
/// An `EACCES` has two possible causes, indistinguishable in the `ls` message:
/// the clone's permissions, or kernel confinement blocking the access. The
/// identity of the exec user and the permissions of the mount point tell them
/// apart.
async fn diagnose_workspace(&self) -> String {
let mut details = Vec::new();
if let Ok(output) = self.container.exec(&["id"]).await {
details.push(output.stdout.trim().to_string());
}
if let Ok(output) = self
.container
.exec(&["ls", "-ld", "--", self.workspace_folder()])
.await
&& output.success()
{
details.push(output.stdout.trim().to_string());
}
details.join(", ")
}
/// Executes a command in the container as an argv vector (no shell).
pub async fn exec(&self, cmd: &[&str]) -> anyhow::Result<ExecOutput> {
Ok(self.container.exec(cmd).await?)
}
/// Path of the repository inside the container.
pub fn workspace_folder(&self) -> &str {
self.container.workspace_folder()
}
/// Stops and removes the container. The temporary clone is removed on drop.
pub async fn cleanup(self) -> anyhow::Result<()> {
self.container.remove().await?;
Ok(())
}
}
fn find_devcontainer(repo_dir: &Path) -> Option<PathBuf> {
DEVCONTAINER_PATHS
.iter()
.map(|relative| repo_dir.join(relative))
.find(|candidate| candidate.is_file())
}
async fn clone_pull_request(
repo_url: &str,
token: &str,
pull_request_number: u64,
dest: &Path,
) -> anyhow::Result<()> {
let dest = dest.display().to_string();
run_git(
token,
&[
"clone".to_string(),
"--depth".to_string(),
"1".to_string(),
repo_url.to_string(),
dest.clone(),
],
)
.await?;
run_git(
token,
&[
"-C".to_string(),
dest.clone(),
"fetch".to_string(),
"--depth".to_string(),
"1".to_string(),
"origin".to_string(),
format!("refs/pull/{pull_request_number}/head"),
],
)
.await?;
run_git(
token,
&[
"-C".to_string(),
dest,
"checkout".to_string(),
"FETCH_HEAD".to_string(),
],
)
.await?;
Ok(())
}
/// Makes the clone readable by the users of the sandbox container.
///
/// The container may not share Steep's uids (rootless podman maps uids through
/// subuid ranges), and the operator's umask may be restrictive: without this, the
/// sandbox tools fail with `Permission denied` on files that Steep itself just
/// cloned.
async fn make_readable(repo_dir: &Path) -> anyhow::Result<()> {
let output = tokio::process::Command::new("chmod")
.args(["-R", "a+rX"])
.arg(repo_dir)
.stdin(Stdio::null())
.output()
.await
.context("failed to spawn chmod")?;
if !output.status.success() {
anyhow::bail!(
"chmod failed on `{}`: {}",
repo_dir.display(),
String::from_utf8_lossy(&output.stderr).trim()
);
}
Ok(())
}
/// Runs git with the token injected through `http.extraHeader`, keeping the
/// secret out of the process arguments.
async fn run_git(token: &str, args: &[String]) -> anyhow::Result<()> {
let output = tokio::process::Command::new("git")
.args(args)
.env("GIT_CONFIG_COUNT", "1")
.env("GIT_CONFIG_KEY_0", "http.extraHeader")
.env(
"GIT_CONFIG_VALUE_0",
format!("Authorization: token {token}"),
)
.env("GIT_TERMINAL_PROMPT", "0")
.stdin(Stdio::null())
.output()
.await
.context("failed to spawn git")?;
if !output.status.success() {
anyhow::bail!(
"git {} failed: {}",
args.join(" "),
String::from_utf8_lossy(&output.stderr).trim()
);
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn find_devcontainer_prefers_dot_devcontainer_dir() {
let dir = tempfile::tempdir().unwrap();
let nested = dir.path().join(".devcontainer");
std::fs::create_dir(&nested).unwrap();
std::fs::write(nested.join("devcontainer.json"), "{}").unwrap();
std::fs::write(dir.path().join(".devcontainer.json"), "{}").unwrap();
assert_eq!(
find_devcontainer(dir.path()),
Some(nested.join("devcontainer.json"))
);
}
#[test]
fn find_devcontainer_returns_none_when_absent() {
let dir = tempfile::tempdir().unwrap();
assert_eq!(find_devcontainer(dir.path()), None);
}
}
+13
View File
@@ -0,0 +1,13 @@
//! Sandboxed tool execution for the AI bot.
//!
//! A [`Sandbox`] clones a pull request into a temporary directory, builds and
//! starts its devcontainer (via `devcontainer-rs`) and exposes command
//! execution inside the resulting container. The [`tools`] module maps model
//! tool calls to commands run in that container, and [`agent`] drives the
//! tool-calling loop against OpenRouter.
pub mod agent;
mod instance;
pub mod tools;
pub use instance::{Sandbox, SandboxConfig};
+303
View File
@@ -0,0 +1,303 @@
//! Tool definitions exposed to the model and their execution inside a
//! [`Sandbox`].
//!
//! Every tool is read-only and confined to the repository workspace: paths are
//! resolved relative to the container workspace folder and rejected if they
//! escape it. Commands are executed as argv vectors (never through a shell),
//! so tool arguments cannot be used for shell injection.
use std::path::Path;
use anyhow::{Context, bail};
use devcontainer_rs::{ExecOutput, normalize};
use serde_json::{Value, json};
use super::Sandbox;
use crate::{gitea::WebhookType, open_router::Tool};
/// Tools available to the model for a given webhook action.
///
/// The match is exhaustive on [`WebhookType`], so adding a new action forces a
/// decision here about which tools that action may use.
pub fn for_webhook(webhook: &WebhookType) -> Vec<Tool> {
match webhook {
WebhookType::Review(_) => review_tools(),
}
}
/// Read-only tools used to explore a repository during a review.
fn review_tools() -> Vec<Tool> {
vec![
Tool::new(
"ls",
"List the entries of a directory inside the repository.",
json!({
"type": "object",
"properties": {
"path": {
"type": "string",
"description": "Directory path relative to the repository root. Defaults to the repository root."
}
}
}),
),
Tool::new(
"file_size",
"Get the size of a file inside the repository in bytes.",
json!({
"type": "object",
"properties": {
"path": {
"type": "string",
"description": "File path relative to the repository root."
}
}
}),
),
Tool::new(
"read_file",
"Read the content of a text file inside the repository. Every line is \
prefixed with its absolute line number, even when only a range is read.",
json!({
"type": "object",
"properties": {
"path": {
"type": "string",
"description": "File path relative to the repository root."
},
"start_line": {
"type": "integer",
"description": "First line to read (1-based, inclusive). Defaults to the first line."
},
"end_line": {
"type": "integer",
"description": "Last line to read (1-based, inclusive). Defaults to the last line."
}
},
"required": ["path"]
}),
),
Tool::new(
"grep",
"Search for a regular expression across the repository files.",
json!({
"type": "object",
"properties": {
"pattern": {
"type": "string",
"description": "Extended regular expression to search for."
},
"path": {
"type": "string",
"description": "File or directory to search in, relative to the repository root. Defaults to the repository root."
}
},
"required": ["pattern"]
}),
),
Tool::new(
"find",
"Find files by name pattern inside the repository.",
json!({
"type": "object",
"properties": {
"pattern": {
"type": "string",
"description": "Glob pattern matched against file names, e.g. `*.rs`."
},
"path": {
"type": "string",
"description": "Directory to search in, relative to the repository root. Defaults to the repository root."
}
},
"required": ["pattern"]
}),
),
]
}
/// Executes a tool call and returns its textual result.
///
/// Errors are returned as `Err` so the caller can decide whether to surface
/// them to the model or abort.
pub async fn dispatch(sandbox: &Sandbox, name: &str, args: &Value) -> anyhow::Result<String> {
match name {
"ls" => ls(sandbox, args).await,
"read_file" => read_file(sandbox, args).await,
"file_size" => file_size(sandbox, args).await,
"grep" => grep(sandbox, args).await,
"find" => find(sandbox, args).await,
other => bail!("unknown tool `{other}`"),
}
}
async fn file_size(sandbox: &Sandbox, args: &Value) -> anyhow::Result<String> {
let path = resolve(sandbox, required_str(args, "path")?)?;
let size = sandbox.exec(&["du", "-b", "--", &path]).await?;
into_stdout(size)
}
async fn ls(sandbox: &Sandbox, args: &Value) -> anyhow::Result<String> {
let path = resolve(sandbox, optional_str(args, "path").unwrap_or("."))?;
let output = sandbox.exec(&["ls", "-la", "--", &path]).await?;
into_stdout(output)
}
async fn read_file(sandbox: &Sandbox, args: &Value) -> anyhow::Result<String> {
let path = resolve(sandbox, required_str(args, "path")?)?;
let start = args.get("start_line").and_then(Value::as_u64);
let end = args.get("end_line").and_then(Value::as_u64);
let (first_line, output) = if start.is_none() && end.is_none() {
(1, sandbox.exec(&["cat", "--", &path]).await?)
} else {
let start = start.unwrap_or(1);
let end = end
.map(|line| line.to_string())
.unwrap_or_else(|| "$".to_string());
let range = format!("{start},{end}p");
(
start,
sandbox.exec(&["sed", "-n", &range, "--", &path]).await?,
)
};
Ok(number_lines(&into_stdout(output)?, first_line))
}
/// Prefixes each line with its number.
///
/// The model must cite a precise line to anchor its comment: without numbers, it
/// counts them itself and drifts by a few lines, which puts the comment next to
/// the targeted code.
fn number_lines(content: &str, first_line: u64) -> String {
content
.lines()
.enumerate()
.map(|(offset, line)| format!("{}:{line}", first_line + offset as u64))
.collect::<Vec<_>>()
.join("\n")
}
async fn grep(sandbox: &Sandbox, args: &Value) -> anyhow::Result<String> {
let pattern = required_str(args, "pattern")?;
let path = resolve(sandbox, optional_str(args, "path").unwrap_or("."))?;
let output = sandbox
.exec(&["grep", "-rn", "-E", "--", pattern, &path])
.await?;
// grep exits with 1 when there is no match, which is not an error.
match output.status {
0 | 1 => Ok(output.stdout),
_ => bail!("grep failed: {}", output.stderr.trim()),
}
}
async fn find(sandbox: &Sandbox, args: &Value) -> anyhow::Result<String> {
let pattern = required_str(args, "pattern")?;
let path = resolve(sandbox, optional_str(args, "path").unwrap_or("."))?;
let output = sandbox
.exec(&["find", &path, "-type", "f", "-name", pattern])
.await?;
into_stdout(output)
}
/// Resolves a tool path against the container workspace folder, rejecting any
/// path that escapes it.
fn resolve(sandbox: &Sandbox, path: &str) -> anyhow::Result<String> {
let workspace = sandbox.workspace_folder();
let candidate = Path::new(workspace).join(path);
let normalized =
normalize(&candidate).with_context(|| format!("path `{path}` escapes the workspace"))?;
if !normalized.starts_with(workspace) {
bail!("path `{path}` escapes the workspace");
}
Ok(normalized.display().to_string())
}
fn required_str<'a>(args: &'a Value, key: &str) -> anyhow::Result<&'a str> {
args.get(key)
.and_then(Value::as_str)
.with_context(|| format!("`{key}` is required"))
}
fn optional_str<'a>(args: &'a Value, key: &str) -> Option<&'a str> {
args.get(key).and_then(Value::as_str)
}
fn into_stdout(output: ExecOutput) -> anyhow::Result<String> {
if output.success() {
Ok(output.stdout)
} else {
bail!(
"command failed (status {}): {}",
output.status,
output.stderr.trim()
)
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::gitea::{Comment, PullRequest, Repository, ReviewPayload};
fn review_webhook() -> WebhookType {
WebhookType::Review(ReviewPayload {
action: "created".to_string(),
pull_request: PullRequest {
number: 1,
title: "My PR".to_string(),
},
repository: Repository {
full_name: "owner/repo".to_string(),
clone_url: "https://github.com/owner/repo.git".to_string(),
},
comment: Comment {
id: 1,
body: "@bot review".to_string(),
},
})
}
#[test]
fn review_webhook_exposes_read_only_tools() {
let names: Vec<String> = for_webhook(&review_webhook())
.into_iter()
.map(|tool| tool.function.name)
.collect();
assert_eq!(names, vec!["ls", "file_size", "read_file", "grep", "find"]);
}
#[test]
fn required_str_reports_missing_key() {
let err = required_str(&json!({}), "path").unwrap_err();
assert!(err.to_string().contains("path"));
}
#[test]
fn lines_are_numbered_from_the_first_one() {
assert_eq!(number_lines("a\nb\n", 1), "1:a\n2:b");
}
#[test]
fn a_range_keeps_the_absolute_line_numbers() {
// An excerpt read from line 12 must keep the file's numbering: otherwise the
// model would cite shifted lines.
assert_eq!(number_lines("x\ny", 12), "12:x\n13:y");
}
#[test]
fn an_empty_read_stays_empty() {
assert_eq!(number_lines("", 1), "");
}
}
+33
View File
@@ -0,0 +1,33 @@
use jiff::Timestamp;
use serde::Serialize;
use crate::gitea::WebhookType;
#[derive(Debug, Clone, Copy, PartialEq, Eq, toasty::Embed, Serialize)]
#[serde(tag = "type", rename_all = "snake_case")]
pub enum JobStatus {
Pending,
Processing,
}
#[derive(Debug, toasty::Model, Clone, Serialize)]
pub struct Job {
#[key]
#[auto]
pub id: u64,
#[unique]
pub event_id: u64,
pub repo: String,
pub pr: u64,
#[column(type = jsonb)]
pub payload: toasty::Json<WebhookType>,
#[index]
pub status: JobStatus,
#[auto]
pub created_at: Timestamp,
}
+5
View File
@@ -0,0 +1,5 @@
pub mod job;
pub mod review;
pub use job::{Job, JobStatus};
pub use review::Review;
+36
View File
@@ -0,0 +1,36 @@
use jiff::Timestamp;
use serde::Serialize;
#[derive(Debug, toasty::Model, Serialize, Clone)]
pub struct Review {
#[key]
#[auto]
pub id: u64,
#[unique]
pub event_id: u64,
pub repo: String,
pub pr: u64,
/// True if the review succeeded.
pub ok: bool,
/// Model used.
pub model: String,
/// Markdown posted on the pull request (empty on failure).
pub markdown: String,
/// OpenRouter cost in USD, if known.
pub cost: Option<f64>,
/// Number of comments produced.
pub comments: u64,
/// Processing duration in milliseconds.
pub duration_ms: u64,
#[default(jiff::Timestamp::now())]
pub completed_at: Timestamp,
}
+14
View File
@@ -0,0 +1,14 @@
use tokio_util::sync::CancellationToken;
use crate::{env::EnvConfig, event_bus::EventBus, job_queue::JobQueue, review_store::ReviewStore};
#[derive(Clone)]
pub struct AppState {
pub queue: JobQueue,
pub reviews: ReviewStore,
pub event_bus: EventBus,
pub bot_name: String,
pub config: EnvConfig,
/// Cancelled when the graceful shutdown starts: it closes the SSE streams.
pub shutdown: CancellationToken,
}
+81
View File
@@ -0,0 +1,81 @@
//! Small text helpers shared by the modules that log what the model answered and
//! bound what the tools return to it.
/// First `limit` characters of `text`, for logs.
///
/// Cuts on character boundaries, so the excerpt stays valid UTF-8, and marks a
/// truncation with an ellipsis.
pub fn excerpt(text: &str, limit: usize) -> String {
let mut chars = text.chars();
let excerpt = chars.by_ref().take(limit).collect::<String>();
if chars.next().is_some() {
return format!("{excerpt}…");
}
excerpt
}
/// Truncates `text` in place to at most `limit` bytes, cutting on a character
/// boundary so the result stays valid UTF-8.
///
/// Returns `true` when something was dropped.
pub fn truncate_bytes(text: &mut String, limit: usize) -> bool {
if text.len() <= limit {
return false;
}
let mut end = limit;
while !text.is_char_boundary(end) {
end -= 1;
}
text.truncate(end);
true
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn a_short_text_is_kept_as_is() {
assert_eq!(excerpt("abc", 3), "abc");
}
#[test]
fn a_long_text_is_truncated_and_marked() {
assert_eq!(excerpt("abcdef", 3), "abc…");
}
#[test]
fn truncation_cuts_on_character_boundaries() {
assert_eq!(excerpt("ééé", 2), "éé…");
}
#[test]
fn a_short_text_is_not_truncated() {
let mut text = String::from("abc");
assert!(!truncate_bytes(&mut text, 3));
assert_eq!(text, "abc");
}
#[test]
fn a_long_text_is_truncated_within_the_limit() {
let mut text = String::from("abcdef");
assert!(truncate_bytes(&mut text, 4));
assert_eq!(text, "abcd");
}
#[test]
fn byte_truncation_never_splits_a_character() {
let mut text = String::from("ééé");
// The limit falls in the middle of the second `é`: it is dropped.
assert!(truncate_bytes(&mut text, 3));
assert_eq!(text, "é");
}
}
+6
View File
@@ -0,0 +1,6 @@
version = 1
[[migrations]]
id = 3569579561409902287
name = "0000_init.sql"
snapshot_name = "0000_snapshot.toml"
@@ -0,0 +1,28 @@
CREATE TYPE "job_status" AS ENUM ('pending', 'processing');
CREATE TABLE "jobs" (
"id" BIGINT NOT NULL GENERATED BY DEFAULT AS IDENTITY,
"event_id" BIGINT NOT NULL,
"repo" TEXT NOT NULL,
"pr" BIGINT NOT NULL,
"payload" JSONB NOT NULL,
"status" "job_status" NOT NULL,
"created_at" TIMESTAMPTZ(6) NOT NULL,
PRIMARY KEY ("id")
);
CREATE UNIQUE INDEX "index_jobs_by_event_id" ON "jobs" ("event_id");
CREATE INDEX "index_jobs_by_status" ON "jobs" ("status");
CREATE TABLE "reviews" (
"id" BIGINT NOT NULL GENERATED BY DEFAULT AS IDENTITY,
"event_id" BIGINT NOT NULL,
"repo" TEXT NOT NULL,
"pr" BIGINT NOT NULL,
"ok" BOOLEAN NOT NULL,
"model" TEXT NOT NULL,
"markdown" TEXT NOT NULL,
"cost" DOUBLE PRECISION,
"comments" BIGINT NOT NULL,
"duration_ms" BIGINT NOT NULL,
"completed_at" TIMESTAMPTZ(6) NOT NULL,
PRIMARY KEY ("id")
);
CREATE UNIQUE INDEX "index_reviews_by_event_id" ON "reviews" ("event_id");
@@ -0,0 +1,160 @@
version = 1
[schema]
[[schema.tables]]
id = 0
name = "jobs"
primary_key = { columns = [{ table = 0, index = 0 }], index = { table = 0, index = 0 } }
[[schema.tables.columns]]
id = { table = 0, index = 0 }
name = "id"
ty = "U64"
storage_ty = { UnsignedInteger = 8 }
auto_increment = true
[[schema.tables.columns]]
id = { table = 0, index = 1 }
name = "event_id"
ty = "U64"
storage_ty = { UnsignedInteger = 8 }
[[schema.tables.columns]]
id = { table = 0, index = 2 }
name = "repo"
ty = "String"
storage_ty = "Text"
[[schema.tables.columns]]
id = { table = 0, index = 3 }
name = "pr"
ty = "U64"
storage_ty = { UnsignedInteger = 8 }
[[schema.tables.columns]]
id = { table = 0, index = 4 }
name = "payload"
ty = "String"
storage_ty = "Jsonb"
[[schema.tables.columns]]
id = { table = 0, index = 5 }
name = "status"
ty = "String"
storage_ty = { Enum = { name = "job_status", variants = [{ name = "pending" }, { name = "processing" }] } }
[[schema.tables.columns]]
id = { table = 0, index = 6 }
name = "created_at"
ty = "Timestamp"
storage_ty = { Timestamp = 6 }
[[schema.tables.indices]]
id = { table = 0, index = 0 }
name = "index_jobs_by_id"
on = 0
columns = [{ column = { table = 0, index = 0 }, op = "Eq", scope = "Partition" }]
unique = true
primary_key = true
[[schema.tables.indices]]
id = { table = 0, index = 1 }
name = "index_jobs_by_event_id"
on = 0
columns = [{ column = { table = 0, index = 1 }, op = "Eq", scope = "Partition" }]
unique = true
[[schema.tables.indices]]
id = { table = 0, index = 2 }
name = "index_jobs_by_status"
on = 0
columns = [{ column = { table = 0, index = 5 }, op = "Eq", scope = "Partition" }]
[[schema.tables]]
id = 1
name = "reviews"
primary_key = { columns = [{ table = 1, index = 0 }], index = { table = 1, index = 0 } }
[[schema.tables.columns]]
id = { table = 1, index = 0 }
name = "id"
ty = "U64"
storage_ty = { UnsignedInteger = 8 }
auto_increment = true
[[schema.tables.columns]]
id = { table = 1, index = 1 }
name = "event_id"
ty = "U64"
storage_ty = { UnsignedInteger = 8 }
[[schema.tables.columns]]
id = { table = 1, index = 2 }
name = "repo"
ty = "String"
storage_ty = "Text"
[[schema.tables.columns]]
id = { table = 1, index = 3 }
name = "pr"
ty = "U64"
storage_ty = { UnsignedInteger = 8 }
[[schema.tables.columns]]
id = { table = 1, index = 4 }
name = "ok"
ty = "Bool"
storage_ty = "Boolean"
[[schema.tables.columns]]
id = { table = 1, index = 5 }
name = "model"
ty = "String"
storage_ty = "Text"
[[schema.tables.columns]]
id = { table = 1, index = 6 }
name = "markdown"
ty = "String"
storage_ty = "Text"
[[schema.tables.columns]]
id = { table = 1, index = 7 }
name = "cost"
ty = "F64"
storage_ty = { Float = 8 }
nullable = true
[[schema.tables.columns]]
id = { table = 1, index = 8 }
name = "comments"
ty = "U64"
storage_ty = { UnsignedInteger = 8 }
[[schema.tables.columns]]
id = { table = 1, index = 9 }
name = "duration_ms"
ty = "U64"
storage_ty = { UnsignedInteger = 8 }
[[schema.tables.columns]]
id = { table = 1, index = 10 }
name = "completed_at"
ty = "Timestamp"
storage_ty = { Timestamp = 6 }
[[schema.tables.indices]]
id = { table = 1, index = 0 }
name = "index_reviews_by_id"
on = 1
columns = [{ column = { table = 1, index = 0 }, op = "Eq", scope = "Partition" }]
unique = true
primary_key = true
[[schema.tables.indices]]
id = { table = 1, index = 1 }
name = "index_reviews_by_event_id"
on = 1
columns = [{ column = { table = 1, index = 1 }, op = "Eq", scope = "Partition" }]
unique = true
+53
View File
@@ -0,0 +1,53 @@
# cargo-deny configuration: https://embarkstudios.github.io/cargo-deny/
# Run locally with `cargo deny check`.
[graph]
all-features = true
[advisories]
# A vulnerable, unmaintained or yanked dependency fails the build.
version = 2
yanked = "deny"
ignore = [
# Unmaintained, but only a thin wrapper around `rustls-pki-types`, and pulled by
# `toasty-driver-postgresql`: it can only go away with a toasty release.
{ id = "RUSTSEC-2025-0134", reason = "transitive dependency of toasty, no vulnerability" },
]
[licenses]
version = 2
# The workspace crates are not published, and have no license of their own.
private = { ignore = true }
allow = [
"Apache-2.0",
"Apache-2.0 WITH LLVM-exception",
"BSD-2-Clause",
"BSD-3-Clause",
"BSL-1.0",
"CC0-1.0",
"CDLA-Permissive-2.0",
"ISC",
"MIT",
"MIT-0",
"MPL-2.0",
"OpenSSL",
"Unicode-3.0",
"Zlib",
]
[bans]
multiple-versions = "warn"
wildcards = "deny"
# `path` dependencies between the (unpublished) workspace crates carry no version.
allow-wildcard-paths = true
# TLS goes through rustls: linking libssl would be a regression of the image
# (it would need `libssl` installed again).
deny = [
{ crate = "openssl-sys", reason = "TLS is rustls only, so that the image needs no libssl" },
{ crate = "native-tls", reason = "TLS is rustls only, so that the image needs no libssl" },
]
[sources]
unknown-registry = "deny"
unknown-git = "deny"
allow-registry = ["https://github.com/rust-lang/crates.io-index"]
+366
View File
@@ -0,0 +1,366 @@
/* Dashboard Steep — reprend le design system du site public
(mêmes jetons de couleur, mêmes composants d'en-tête/pied) et ajoute les
styles propres au tableau de bord. */
:root {
color-scheme: dark;
--paper: #0b1520;
--surface: #111f2d;
--ink: #edf4f7;
--ink-soft: #c2cfd9;
--muted: #94a8b8;
--line: #263b4d;
--green: #087b5b;
--green-dark: #5bd4a7;
--green-pale: #153a32;
--blue-pale: #1a3044;
--shadow: 0 18px 50px rgba(0, 0, 0, 0.24);
--button-hover: #066449;
--brand-mark-bg: #149b72;
--brand-mark-fg: #07111b;
--step-line: #23604c;
--setup-bg: #07111b;
--radius: 20px;
--content: 1160px;
--red: #d1667f;
--amber: #d7a35b;
}
:root[data-theme="light"] {
color-scheme: light;
--paper: #f7f8f5;
--surface: #ffffff;
--ink: #102f50;
--ink-soft: #38536c;
--muted: #718092;
--line: #e1e8e5;
--green: #149b72;
--green-dark: #087b5b;
--green-pale: #e8f5ef;
--blue-pale: #edf3f8;
--shadow: 0 18px 50px rgba(16, 47, 80, 0.08);
--button-hover: #087b5b;
--brand-mark-bg: #102f50;
--brand-mark-fg: #ffffff;
--step-line: #cce8da;
--setup-bg: #102f50;
--red: #b04a63;
--amber: #a9741f;
}
*, *::before, *::after { box-sizing: border-box; }
html { scroll-behavior: smooth; scroll-padding-top: 90px; }
body {
margin: 0;
background: var(--paper);
color: var(--ink);
font-family: Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
font-size: 16px;
line-height: 1.65;
-webkit-font-smoothing: antialiased;
text-rendering: optimizeLegibility;
}
a { color: inherit; }
img { display: block; max-width: 100%; }
.container { width: min(var(--content), calc(100% - 48px)); margin-inline: auto; }
.mono { font-family: "SFMono-Regular", Consolas, "Liberation Mono", monospace; }
.muted { color: var(--muted); }
.site-header {
position: sticky;
z-index: 5;
top: 0;
border-bottom: 1px solid var(--line);
background: var(--paper);
backdrop-filter: blur(16px);
}
.nav {
min-height: 76px;
display: flex;
align-items: center;
justify-content: space-between;
gap: 28px;
}
.wordmark {
display: inline-flex;
align-items: center;
gap: 11px;
color: var(--ink);
text-decoration: none;
font-size: 18px;
font-weight: 850;
letter-spacing: 0.13em;
}
.wordmark-mark {
width: 34px;
height: 34px;
display: grid;
place-items: center;
border-radius: 11px;
background: var(--brand-mark-bg);
color: var(--brand-mark-fg);
font-size: 17px;
letter-spacing: 0;
}
.nav-links { display: flex; align-items: center; gap: 22px; }
.nav-links a {
color: var(--ink-soft);
font-size: 14px;
font-weight: 650;
text-decoration: none;
transition: color 160ms ease;
}
.nav-links a:hover { color: var(--green-dark); }
.nav-links .nav-cta {
padding: 10px 16px;
border: 1px solid var(--line);
border-radius: 999px;
background: var(--surface);
color: var(--ink);
}
.nav-links .nav-cta:hover { border-color: var(--green); color: var(--green-dark); }
.theme-toggle {
display: inline-flex;
align-items: center;
gap: 8px;
padding: 9px 13px;
border: 1px solid var(--line);
border-radius: 999px;
background: var(--surface);
color: var(--ink-soft);
font: inherit;
font-size: 13px;
font-weight: 700;
cursor: pointer;
transition: border-color 160ms ease, color 160ms ease, background 160ms ease;
}
.theme-toggle:hover { border-color: var(--green-dark); color: var(--green-dark); }
.theme-icon { color: var(--green-dark); font-size: 16px; line-height: 1; }
.section { padding: 46px 0 82px; }
.section-heading { max-width: 720px; margin-bottom: 26px; }
.section-heading h1 {
margin-bottom: 12px;
font-size: clamp(30px, 4vw, 44px);
line-height: 1.12;
letter-spacing: -0.045em;
}
.section-heading p { margin-bottom: 0; color: var(--muted); font-size: 17px; }
.eyebrow {
display: inline-flex;
align-items: center;
gap: 9px;
margin: 0 0 16px;
color: var(--green-dark);
font-size: 12px;
font-weight: 800;
letter-spacing: 0.12em;
text-transform: uppercase;
}
.eyebrow::before {
width: 8px;
height: 8px;
border-radius: 50%;
background: var(--green);
content: "";
}
.button {
display: inline-flex;
min-height: 42px;
align-items: center;
justify-content: center;
gap: 9px;
padding: 9px 16px;
border: 1px solid transparent;
border-radius: 999px;
background: var(--green);
color: white;
font: inherit;
font-size: 14px;
font-weight: 750;
text-decoration: none;
cursor: pointer;
transition: transform 160ms ease, background 160ms ease, box-shadow 160ms ease;
}
.button:hover { transform: translateY(-2px); background: var(--button-hover); box-shadow: 0 8px 20px rgba(20, 155, 114, 0.2); }
.button-secondary { border-color: var(--line); background: var(--surface); color: var(--ink); }
.button-secondary:hover { border-color: #b8d8ca; background: var(--green-pale); color: var(--green-dark); box-shadow: none; }
.arrow { font-size: 17px; line-height: 1; }
/* ---- Tableau de bord ---- */
.toolbar {
display: flex;
align-items: center;
justify-content: flex-end;
margin-bottom: 22px;
}
.board {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 18px;
margin-bottom: 18px;
}
.panel {
padding: 22px;
border: 1px solid var(--line);
border-radius: var(--radius);
background: var(--surface);
box-shadow: 0 5px 20px rgba(16, 47, 80, 0.025);
}
.panel-head {
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
margin-bottom: 16px;
}
.panel-head h2 { margin: 0; font-size: 17px; letter-spacing: -0.02em; }
.count {
min-width: 26px;
padding: 3px 9px;
border-radius: 999px;
background: var(--blue-pale);
color: var(--ink);
font-size: 12px;
font-weight: 850;
text-align: center;
}
.detail-panel { margin-bottom: 18px; }
steep-job-list { display: grid; gap: 10px; }
.job {
border: 1px solid var(--line);
border-radius: 14px;
background: var(--paper);
transition: border-color 140ms ease, background 140ms ease;
}
.job:hover { border-color: var(--green); }
.job[data-selected="true"] { border-color: var(--green); background: var(--green-pale); }
.job-head {
display: grid;
grid-template-columns: auto 1fr auto;
align-items: baseline;
gap: 10px;
padding: 13px 15px;
cursor: pointer;
list-style: none;
}
.job-head::-webkit-details-marker { display: none; }
.job-repo { font-weight: 700; font-size: 14px; overflow-wrap: anywhere; }
.job-pr { color: var(--green-dark); font-weight: 800; font-size: 13px; white-space: nowrap; }
.job-info { display: grid; gap: 12px; padding: 0 15px 15px; }
.job-info dl { display: grid; gap: 6px; margin: 0; }
.job-info dl > div { display: grid; grid-template-columns: 74px 1fr; gap: 10px; }
.job-info dt { color: var(--muted); font-size: 12px; }
.job-info dd { margin: 0; font-size: 12px; overflow-wrap: anywhere; }
.job-watch { justify-self: start; }
/* Chevron partagé par tous les <details> repliables. */
.chev::before { color: var(--muted); content: "▸"; font-size: 11px; }
details[open] .chev::before { content: "▾"; }
.empty { margin: 0; padding: 14px 2px; color: var(--muted); font-size: 14px; }
.pill {
display: inline-flex;
align-items: center;
padding: 3px 10px;
border-radius: 999px;
font-size: 12px;
font-weight: 800;
text-transform: lowercase;
}
.pill.done { background: var(--green-pale); color: var(--green-dark); }
.pill.failed { background: rgba(209, 102, 127, 0.16); color: var(--red); }
.reviews { overflow-x: auto; }
.reviews-table { width: 100%; border-collapse: collapse; font-size: 13px; }
.reviews-table th {
padding: 0 12px 10px 0;
border-bottom: 1px solid var(--line);
color: var(--muted);
font-size: 11px;
font-weight: 800;
letter-spacing: 0.08em;
text-transform: uppercase;
text-align: left;
}
.reviews-table td { padding: 11px 12px 11px 0; border-bottom: 1px solid var(--line); vertical-align: top; }
.reviews-table tr:last-child td { border-bottom: 0; }
/* Journal d'activité d'un job */
.log {
max-height: 360px;
overflow-y: auto;
padding: 4px 2px;
font-family: "SFMono-Regular", Consolas, "Liberation Mono", monospace;
font-size: 12.5px;
line-height: 1.6;
}
.log-entry { display: grid; grid-template-columns: auto 1fr; gap: 10px; padding: 6px 0; border-bottom: 1px solid var(--line); }
.log-entry:last-child { border-bottom: 0; }
/* Résultat d'outil replié par défaut : le résumé tient sur une ligne, le corps
complet s'affiche au dépliage. */
.log-result { padding: 6px 0; border-bottom: 1px solid var(--line); }
.log-result:last-child { border-bottom: 0; }
.log-result > summary {
display: grid;
grid-template-columns: auto auto 1fr;
gap: 10px;
align-items: baseline;
cursor: pointer;
list-style: none;
}
.log-result > summary::-webkit-details-marker { display: none; }
.log-result > .log-body { padding-top: 6px; }
.log-preview { color: var(--muted); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.log-tag {
align-self: start;
padding: 2px 8px;
border-radius: 7px;
background: var(--blue-pale);
color: var(--ink);
font-size: 11px;
font-weight: 800;
white-space: nowrap;
}
.log-entry[data-kind="tool_call"] .log-tag { background: var(--green-pale); color: var(--green-dark); }
.log-entry[data-kind="finished"] .log-tag { background: var(--green-pale); color: var(--green-dark); }
.log-entry[data-kind="gap"] .log-tag { background: rgba(215, 163, 91, 0.18); color: var(--amber); }
.log-body { overflow-wrap: anywhere; white-space: pre-wrap; }
.log-args { color: var(--muted); }
.log-truncated { color: var(--amber); font-style: italic; }
footer { padding: 26px 0; background: #0b233b; color: #b6c5d3; font-size: 13px; }
.footer-inner { display: flex; align-items: center; justify-content: space-between; gap: 20px; }
footer a { color: white; text-decoration: none; }
footer a:hover { color: #8be0bc; }
.footer-brand { color: white; font-weight: 800; letter-spacing: 0.1em; }
:focus-visible { outline: 3px solid #35bb8e; outline-offset: 4px; }
@media (max-width: 820px) {
.board { grid-template-columns: 1fr; }
}
@media (max-width: 600px) {
.container { width: min(var(--content), calc(100% - 32px)); }
.nav { min-height: 66px; gap: 12px; }
.nav-links { gap: 12px; }
.nav-links .nav-cta { display: none; }
.section { padding: 30px 0 62px; }
.toolbar { flex-direction: column; align-items: stretch; }
.footer-inner { align-items: flex-start; flex-direction: column; gap: 6px; }
}
@media (prefers-reduced-motion: reduce) {
*, *::before, *::after { scroll-behavior: auto !important; transition-duration: 0.01ms !important; animation-duration: 0.01ms !important; }
}
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 206 KiB

+46
View File
@@ -0,0 +1,46 @@
<!DOCTYPE html>
<html lang="en" data-theme="dark">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#0b1520">
<link rel="icon" type="image/png" sizes="512x512" href="/favicon.png">
<link rel="apple-touch-icon" href="/favicon.png">
<meta name="description" content="Steep dashboard — pending jobs, work in progress, and completed reviews, live over SSE.">
<title>Steep — Dashboard</title>
<link rel="stylesheet" href="/css/dashboard.css">
<script>
// Applique le thème enregistré avant le premier rendu, pour éviter un flash.
try {
const savedTheme = localStorage.getItem("steep-theme");
if (savedTheme === "light" || savedTheme === "dark") {
document.documentElement.dataset.theme = savedTheme;
}
} catch (_) {}
</script>
</head>
<body>
<header class="site-header">
<nav class="nav container" aria-label="Main navigation">
<a class="wordmark" href="/" aria-label="Steep dashboard"><span class="wordmark-mark">S</span> STEEP</a>
<div class="nav-links">
<a class="nav-cta" href="https://gitea.qpismont.fr/qpismont/herald">View source <span class="arrow" aria-hidden="true">↗</span></a>
<steep-theme-toggle></steep-theme-toggle>
</div>
</nav>
</header>
<main id="top" class="container">
<steep-dashboard></steep-dashboard>
</main>
<footer>
<div class="container footer-inner">
<span><span class="footer-brand">STEEP</span> · Let your code steep.</span>
<span>Built with Rust · <a href="https://gitea.qpismont.fr/qpismont/herald">Source on Gitea</a></span>
</div>
</footer>
<script type="module" src="/js/dashboard.js"></script>
</body>
</html>
+78
View File
@@ -0,0 +1,78 @@
// Accès aux routes exposées par le serveur axum et petits utilitaires partagés
// par les web components du dashboard.
export const endpoints = {
jobs: "/jobs",
reviews: "/reviews",
lifecycle: "/events",
jobEvents: (id) => `/jobs/${id}/events`,
};
async function getJson(url) {
const response = await fetch(url, { headers: { accept: "application/json" } });
if (!response.ok) {
throw new Error(`${url} → ${response.status}`);
}
return response.json();
}
export const fetchJobs = () => getJson(endpoints.jobs);
export const fetchReviews = () => getJson(endpoints.reviews);
// Ouvre un flux SSE et route les évènements nommés vers les callbacks fournis.
// Les évènements du serveur sont nommés (`lifecycle`, `progress`, `gap`), donc
// `addEventListener("message")` ne suffit pas : il faut écouter par nom.
export function subscribe(url, handlers = {}) {
const source = new EventSource(url);
const onData = (name, handler) => {
if (handler) {
source.addEventListener(name, (event) => handler(JSON.parse(event.data)));
}
};
if (handlers.onOpen) source.addEventListener("open", () => handlers.onOpen());
if (handlers.onError) source.addEventListener("error", () => handlers.onError());
onData("lifecycle", handlers.onLifecycle);
onData("progress", handlers.onProgress);
if (handlers.onGap) {
source.addEventListener("gap", (event) => handlers.onGap(Number(event.data)));
}
return source;
}
export function escapeHtml(value) {
return String(value ?? "").replace(/[&<>"']/g, (char) => ({
"&": "&amp;",
"<": "&lt;",
">": "&gt;",
'"': "&quot;",
"'": "&#39;",
})[char]);
}
// Horodatage relatif (« il y a 3 min »), tolérant sur les valeurs absentes.
export function relativeTime(value) {
if (!value) return "—";
const then = Date.parse(value);
if (Number.isNaN(then)) return "—";
const seconds = Math.round((then - Date.now()) / 1000);
const steps = [
["year", 60 * 60 * 24 * 365],
["month", 60 * 60 * 24 * 30],
["day", 60 * 60 * 24],
["hour", 60 * 60],
["minute", 60],
["second", 1],
];
const formatter = new Intl.RelativeTimeFormat("en", { numeric: "auto" });
for (const [unit, size] of steps) {
if (Math.abs(seconds) >= size || unit === "second") {
return formatter.format(Math.round(seconds / size), unit);
}
}
return "—";
}
+212
View File
@@ -0,0 +1,212 @@
// <steep-dashboard> : assemble le tableau de bord.
//
// Au montage, il charge l'état initial via REST (`/jobs`, `/reviews`), puis
// s'abonne au flux global `/events` pour tenir les listes à jour en direct.
// La sélection d'un job ouvre son flux fin via <steep-job-detail>.
import {
endpoints,
escapeHtml,
fetchJobs,
fetchReviews,
relativeTime,
subscribe,
} from "./api.js";
import "./theme-toggle.js";
import "./job-list.js";
import "./job-detail.js";
class Dashboard extends HTMLElement {
connectedCallback() {
this._jobs = { pending: [], processing: [] };
this._reviews = [];
this._selectedId = null;
this.render();
this.refresh();
this.connect();
}
disconnectedCallback() {
this._source?.close();
if (this._refreshTimer) clearTimeout(this._refreshTimer);
}
// ---- Chargement et flux -------------------------------------------------
async refresh() {
try {
const [jobs, reviews] = await Promise.all([fetchJobs(), fetchReviews()]);
this._jobs = {
pending: jobs.pending ?? [],
processing: jobs.processing ?? [],
};
this._reviews = reviews.reviews ?? [];
this.update();
} catch (error) {
console.error("failed to refresh the dashboard", error);
}
}
connect() {
this._source = subscribe(endpoints.lifecycle, {
onLifecycle: (event) => this.applyLifecycle(event),
// En cas de gap (évènements ratés), on recharge l'état depuis le REST.
onGap: () => this.refresh(),
});
}
// Recharge la liste des reviews peu après une fin de job, pour récupérer les
// champs que l'évènement ne porte pas (markdown, coût, modèle).
scheduleRefresh() {
if (this._refreshTimer) clearTimeout(this._refreshTimer);
this._refreshTimer = setTimeout(() => this.refresh(), 400);
}
applyLifecycle(event) {
switch (event.type) {
case "queued":
if (!this._jobs.pending.some((job) => job.id === event.id)) {
this._jobs.pending.push(this.jobFromEvent(event));
}
break;
case "started": {
// On garde le job déjà connu s'il existe, pour conserver les champs que
// l'évènement `started` ne porte pas (event_id, created_at).
const known = this._jobs.pending.find((job) => job.id === event.id);
this.removeJob(event.id);
this._jobs.processing.push(known ?? this.jobFromEvent(event));
break;
}
case "review_done":
case "failed":
this.removeJob(event.id);
this.scheduleRefresh();
break;
case "skipped":
this.removeJob(event.id);
break;
default:
return;
}
this.update();
}
removeJob(id) {
this._jobs.pending = this._jobs.pending.filter((job) => job.id !== id);
this._jobs.processing = this._jobs.processing.filter((job) => job.id !== id);
}
// L'évènement de cycle de vie porte assez de champs pour l'affichage ; le
// rafraîchissement REST fournira le reste (created_at, payload…).
jobFromEvent(event) {
return {
id: event.id,
event_id: event.event_id,
repo: event.repo,
pr: event.pr,
created_at: event.created_at,
};
}
// ---- Rendu --------------------------------------------------------------
render() {
this.innerHTML = `
<section class="section">
<div class="section-heading">
<p class="eyebrow">Live operations</p>
<h1>Review dashboard</h1>
<p>Pending jobs, work in progress, and completed reviews — kept in sync over SSE.</p>
</div>
<div class="toolbar">
<button class="button button-secondary" id="refresh" type="button">Refresh</button>
</div>
<div class="board">
<section class="panel">
<header class="panel-head"><h2>Pending</h2><span class="count" id="pending-count">0</span></header>
<steep-job-list id="pending" empty="No pending job."></steep-job-list>
</section>
<section class="panel">
<header class="panel-head"><h2>In progress</h2><span class="count" id="processing-count">0</span></header>
<steep-job-list id="processing" empty="No job is running."></steep-job-list>
</section>
</div>
<section class="panel detail-panel">
<header class="panel-head"><h2>Activity</h2></header>
<steep-job-detail id="detail"></steep-job-detail>
</section>
<section class="panel">
<header class="panel-head"><h2>Completed reviews</h2><span class="count" id="reviews-count">0</span></header>
<div class="reviews" id="reviews"></div>
</section>
</section>`;
this._pending = this.querySelector("#pending");
this._processing = this.querySelector("#processing");
this._detail = this.querySelector("#detail");
this._reviewsEl = this.querySelector("#reviews");
this.querySelector("#refresh").addEventListener("click", () => this.refresh());
this.addEventListener("job-select", (event) => this.select(event.detail.id));
this.update();
}
update() {
this._pending.jobs = this._jobs.pending;
this._pending.selected = this._selectedId;
this._processing.jobs = this._jobs.processing;
this._processing.selected = this._selectedId;
this.querySelector("#pending-count").textContent = String(this._jobs.pending.length);
this.querySelector("#processing-count").textContent = String(this._jobs.processing.length);
this.querySelector("#reviews-count").textContent = String(this._reviews.length);
this._reviewsEl.innerHTML = this.reviewsHtml();
}
select(id) {
this._selectedId = id;
this._detail.jobId = id;
this.update();
this._detail.scrollIntoView({ behavior: "smooth", block: "nearest" });
}
reviewsHtml() {
if (this._reviews.length === 0) {
return `<p class="empty">No review completed yet.</p>`;
}
const rows = [...this._reviews]
.sort((a, b) => String(b.completed_at ?? "").localeCompare(String(a.completed_at ?? "")))
.map(
(review) => `
<tr>
<td><span class="pill ${review.ok ? "done" : "failed"}">${review.ok ? "done" : "failed"}</span></td>
<td class="mono">${escapeHtml(review.repo)}#${review.pr}</td>
<td>${review.comments}</td>
<td>${review.duration_ms} ms</td>
<td class="mono muted">${escapeHtml(review.model ?? "—")}</td>
<td class="muted">${relativeTime(review.completed_at)}</td>
</tr>`,
)
.join("");
return `
<table class="reviews-table">
<thead>
<tr>
<th>Status</th><th>Pull request</th><th>Comments</th><th>Duration</th><th>Model</th><th>Completed</th>
</tr>
</thead>
<tbody>${rows}</tbody>
</table>`;
}
}
customElements.define("steep-dashboard", Dashboard);
+61
View File
@@ -0,0 +1,61 @@
// Carte d'un job. Le détail (id, event, date) est replié par défaut ; le
// bouton « Watch activity » remonte un évènement `job-select` que le dashboard
// écoute pour ouvrir le flux d'activité du job.
import { escapeHtml, relativeTime } from "./api.js";
class JobCard extends HTMLElement {
set job(value) {
this._job = value;
this.render();
}
get job() {
return this._job;
}
set selected(value) {
this._selected = Boolean(value);
this.render();
}
connectedCallback() {
this.render();
}
render() {
if (!this._job) return;
const job = this._job;
// Préserve l'état replié/déplié à travers les re-rendus (mises à jour SSE).
const open = this._details?.open ? " open" : "";
this.innerHTML = `
<details class="job" data-selected="${this._selected ? "true" : "false"}"${open}>
<summary class="job-head">
<span class="chev" aria-hidden="true"></span>
<span class="job-repo">${escapeHtml(job.repo)}</span>
<span class="job-pr">#${job.pr}</span>
</summary>
<div class="job-info">
<dl>
<div><dt>Job</dt><dd>${job.id}</dd></div>
${job.event_id != null ? `<div><dt>Event</dt><dd>${job.event_id}</dd></div>` : ""}
<div><dt>Created</dt><dd>${escapeHtml(relativeTime(job.created_at))}</dd></div>
</dl>
<button class="button button-secondary job-watch" type="button">Watch activity</button>
</div>
</details>`;
this._details = this.querySelector("details");
this.querySelector(".job-watch").addEventListener("click", () => this.select());
}
select() {
this.dispatchEvent(
new CustomEvent("job-select", { detail: { id: this._job.id }, bubbles: true }),
);
}
}
customElements.define("steep-job-card", JobCard);
+150
View File
@@ -0,0 +1,150 @@
// Panneau d'activité d'un job : ouvre un flux SSE vers `/jobs/{id}/events`,
// rejoue l'historique renvoyé par le serveur puis affiche le direct.
import { endpoints, escapeHtml, subscribe } from "./api.js";
class JobDetail extends HTMLElement {
set jobId(value) {
const id = value ?? null;
if (this._jobId === id) return;
this._jobId = id;
this.open();
}
connectedCallback() {
if (!this._jobId) this.renderPlaceholder();
}
disconnectedCallback() {
this.close();
}
close() {
if (this._source) {
this._source.close();
this._source = null;
}
}
open() {
this.close();
this._finished = false;
if (!this._jobId) {
this.renderPlaceholder();
return;
}
this.innerHTML = `
<div class="log" id="log" role="log" aria-live="polite"></div>`;
this._log = this.querySelector("#log");
this._source = subscribe(endpoints.jobEvents(this._jobId), {
onProgress: (event) => {
this.append(event);
if (event.type === "finished") {
this._finished = true;
this.close();
}
},
onGap: (skipped) =>
this.append({ type: "gap", note: `missed ${skipped} events, refresh to resync` }),
// Un `error` survient aussi quand le serveur ferme le flux (job terminé).
// On ferme alors la source pour éviter la boucle de reconnexion (le job
// n'est plus en cours, la route renverrait 404).
onError: () => {
if (this._finished) return;
this.append({ type: "gap", note: "stream closed" });
this.close();
},
});
}
renderPlaceholder() {
this.innerHTML = `<p class="empty">Select a job in progress to watch the agent work.</p>`;
}
append(event) {
if (!this._log) return;
this._log.append(this.entry(event));
this._log.scrollTop = this._log.scrollHeight;
}
entry(event) {
return event.type === "tool_result" ? this.resultEntry(event) : this.lineEntry(event);
}
// Ligne simple : appel d'outil, fin de review, gap.
lineEntry(event) {
const row = document.createElement("div");
row.className = "log-entry";
row.dataset.kind = event.type;
const { tag, body } = this.describe(event);
const tagEl = document.createElement("span");
tagEl.className = "log-tag";
tagEl.textContent = tag;
const bodyEl = document.createElement("span");
bodyEl.className = "log-body";
bodyEl.innerHTML = body;
row.append(tagEl, bodyEl);
return row;
}
// Résultat d'outil : replié par défaut, avec un aperçu sur une ligne.
resultEntry(event) {
const details = document.createElement("details");
details.className = "log-result";
const summary = document.createElement("summary");
const chev = document.createElement("span");
chev.className = "chev";
chev.setAttribute("aria-hidden", "true");
const tag = document.createElement("span");
tag.className = "log-tag";
tag.textContent = "result";
const preview = document.createElement("span");
preview.className = "log-preview";
preview.textContent = this.preview(event.output);
summary.append(chev, tag, preview);
const body = document.createElement("div");
body.className = "log-body";
const truncated = event.truncated
? ' <span class="log-truncated">(truncated)</span>'
: "";
body.innerHTML = `${escapeHtml(event.output)}${truncated}`;
details.append(summary, body);
return details;
}
preview(text) {
const line = String(text ?? "").replace(/\s+/g, " ").trim();
return line.length > 80 ? `${line.slice(0, 80)}…` : line;
}
describe(event) {
switch (event.type) {
case "tool_call":
return {
tag: "call",
body: `${escapeHtml(event.name)} <span class="log-args">${escapeHtml(event.arguments)}</span>`,
};
case "finished":
return {
tag: "done",
body: event.ok ? "review finished" : "review failed",
};
case "gap":
return { tag: "gap", body: escapeHtml(event.note ?? "gap") };
default:
return { tag: escapeHtml(event.type ?? "event"), body: "" };
}
}
}
customElements.define("steep-job-detail", JobDetail);
+58
View File
@@ -0,0 +1,58 @@
// Liste de jobs (une colonne du tableau). Reçoit un tableau de jobs via la
// propriété `jobs` et l'id sélectionné via `selected`.
import "./job-card.js";
class JobList extends HTMLElement {
set jobs(value) {
this._jobs = Array.isArray(value) ? value : [];
this.render();
}
get jobs() {
return this._jobs ?? [];
}
set selected(value) {
this._selected = value;
this.render();
}
connectedCallback() {
if (this._mounted) return;
this._mounted = true;
this.render();
}
render() {
const jobs = this.jobs;
if (jobs.length === 0) {
this.replaceChildren();
const empty = document.createElement("p");
empty.className = "empty";
empty.textContent = this.getAttribute("empty") || "Nothing here.";
this.append(empty);
return;
}
// Réconciliation par id : on réutilise les cartes existantes pour préserver
// l'état replié/déplié de chacune à travers les mises à jour.
const known = new Map();
for (const card of this.querySelectorAll("steep-job-card")) {
if (card.job) known.set(card.job.id, card);
}
for (const job of jobs) {
const card = known.get(job.id) ?? document.createElement("steep-job-card");
known.delete(job.id);
card.job = job;
card.selected = job.id === this._selected;
this.append(card);
}
for (const card of known.values()) card.remove();
}
}
customElements.define("steep-job-list", JobList);
+59
View File
@@ -0,0 +1,59 @@
// Bascule de thème, identique à celle du site public (même clé localStorage,
// même rendu), empaquetée en web component.
const STORAGE_KEY = "steep-theme";
export function currentTheme() {
return document.documentElement.dataset.theme === "light" ? "light" : "dark";
}
export function applyTheme(theme, persist = true) {
document.documentElement.dataset.theme = theme;
const meta = document.querySelector('meta[name="theme-color"]');
if (meta) meta.content = theme === "dark" ? "#0b1520" : "#f7f8f5";
if (persist) {
try {
localStorage.setItem(STORAGE_KEY, theme);
} catch (_) {
// stockage indisponible : on garde juste le thème en mémoire.
}
}
window.dispatchEvent(new CustomEvent("steep:theme", { detail: theme }));
}
class ThemeToggle extends HTMLElement {
connectedCallback() {
this.innerHTML = `
<button class="theme-toggle" type="button" role="switch" aria-label="Toggle color theme">
<span class="theme-icon" aria-hidden="true"></span><span class="theme-label"></span>
</button>`;
this._button = this.querySelector("button");
this._icon = this.querySelector(".theme-icon");
this._label = this.querySelector(".theme-label");
this._onTheme = () => this.render();
this._button.addEventListener("click", () => {
applyTheme(currentTheme() === "dark" ? "light" : "dark");
});
window.addEventListener("steep:theme", this._onTheme);
this.render();
}
disconnectedCallback() {
window.removeEventListener("steep:theme", this._onTheme);
}
render() {
const theme = currentTheme();
this._icon.textContent = theme === "dark" ? "☾" : "☀";
this._label.textContent = theme === "dark" ? "Dark theme" : "Light theme";
this._button.setAttribute("aria-checked", String(theme === "dark"));
}
}
customElements.define("steep-theme-toggle", ThemeToggle);
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 149 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 843 KiB

+5
View File
@@ -0,0 +1,5 @@
:80 {
root * /usr/share/caddy
encode zstd gzip
file_server
}
+4
View File
@@ -0,0 +1,4 @@
FROM caddy:2-alpine
COPY Caddyfile /etc/caddy/Caddyfile
COPY index.html favicon.png banniere_readme.png /usr/share/caddy/
Binary file not shown.

After

Width:  |  Height:  |  Size: 1009 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 206 KiB

+646
View File
@@ -0,0 +1,646 @@
<!DOCTYPE html>
<html lang="en" data-theme="dark">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#0b1520">
<link rel="icon" type="image/png" sizes="512x512" href="favicon.png">
<link rel="apple-touch-icon" href="favicon.png">
<meta name="description" content="Steep is an open-source Gitea bot that reviews pull requests in isolated Dev Containers.">
<title>Steep — Let your code steep</title>
<style>
:root {
color-scheme: dark;
--paper: #0b1520;
--surface: #111f2d;
--ink: #edf4f7;
--ink-soft: #c2cfd9;
--muted: #94a8b8;
--line: #263b4d;
--green: #087b5b;
--green-dark: #5bd4a7;
--green-pale: #153a32;
--blue-pale: #1a3044;
--shadow: 0 18px 50px rgba(0, 0, 0, 0.24);
--button-hover: #066449;
--brand-mark-bg: #149b72;
--brand-mark-fg: #07111b;
--step-line: #23604c;
--setup-bg: #07111b;
--radius: 20px;
--content: 1160px;
}
:root[data-theme="light"] {
color-scheme: light;
--paper: #f7f8f5;
--surface: #ffffff;
--ink: #102f50;
--ink-soft: #38536c;
--muted: #718092;
--line: #e1e8e5;
--green: #149b72;
--green-dark: #087b5b;
--green-pale: #e8f5ef;
--blue-pale: #edf3f8;
--shadow: 0 18px 50px rgba(16, 47, 80, 0.08);
--button-hover: #087b5b;
--brand-mark-bg: #102f50;
--brand-mark-fg: #ffffff;
--step-line: #cce8da;
--setup-bg: #102f50;
}
*, *::before, *::after { box-sizing: border-box; }
html { scroll-behavior: smooth; scroll-padding-top: 90px; }
body {
margin: 0;
background: var(--paper);
color: var(--ink);
font-family: Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
font-size: 16px;
line-height: 1.65;
-webkit-font-smoothing: antialiased;
text-rendering: optimizeLegibility;
}
a { color: inherit; }
img { display: block; max-width: 100%; }
.container { width: min(var(--content), calc(100% - 48px)); margin-inline: auto; }
.site-header {
position: sticky;
z-index: 5;
top: 0;
border-bottom: 1px solid var(--line);
background: var(--paper);
backdrop-filter: blur(16px);
}
.nav {
min-height: 76px;
display: flex;
align-items: center;
justify-content: space-between;
gap: 28px;
}
.wordmark {
display: inline-flex;
align-items: center;
gap: 11px;
color: var(--ink);
text-decoration: none;
font-size: 18px;
font-weight: 850;
letter-spacing: 0.13em;
}
.wordmark-mark {
width: 34px;
height: 34px;
display: grid;
place-items: center;
border-radius: 11px;
background: var(--brand-mark-bg);
color: var(--brand-mark-fg);
font-size: 17px;
letter-spacing: 0;
}
.nav-links { display: flex; align-items: center; gap: 30px; }
.nav-links a {
color: var(--ink-soft);
font-size: 14px;
font-weight: 650;
text-decoration: none;
transition: color 160ms ease;
}
.nav-links a:hover { color: var(--green-dark); }
.nav-links .nav-cta {
padding: 10px 16px;
border: 1px solid var(--line);
border-radius: 999px;
background: var(--surface);
color: var(--ink);
}
.nav-links .nav-cta:hover { border-color: var(--green); color: var(--green-dark); }
.theme-toggle {
display: inline-flex;
align-items: center;
gap: 8px;
padding: 9px 13px;
border: 1px solid var(--line);
border-radius: 999px;
background: var(--surface);
color: var(--ink-soft);
font: inherit;
font-size: 13px;
font-weight: 700;
cursor: pointer;
transition: border-color 160ms ease, color 160ms ease, background 160ms ease;
}
.theme-toggle:hover { border-color: var(--green-dark); color: var(--green-dark); }
.theme-icon { color: var(--green-dark); font-size: 16px; line-height: 1; }
.hero { padding: 46px 0 94px; }
.brand-banner {
position: relative;
overflow: hidden;
border: 1px solid var(--line);
border-radius: 26px;
background: white;
box-shadow: var(--shadow);
}
.brand-banner img { width: 100%; height: auto; }
.hero-copy {
display: grid;
grid-template-columns: 1.05fr 0.75fr;
align-items: end;
gap: 54px;
padding: 50px 10px 0;
}
.eyebrow {
display: inline-flex;
align-items: center;
gap: 9px;
margin: 0 0 16px;
color: var(--green-dark);
font-size: 12px;
font-weight: 800;
letter-spacing: 0.12em;
text-transform: uppercase;
}
.eyebrow::before {
width: 8px;
height: 8px;
border-radius: 50%;
background: var(--green);
content: "";
}
h1, h2, h3, p { margin-top: 0; }
h1 {
max-width: 690px;
margin-bottom: 0;
font-size: clamp(38px, 6vw, 66px);
line-height: 1.04;
letter-spacing: -0.055em;
}
.hero-aside p {
margin-bottom: 22px;
color: var(--ink-soft);
font-size: 17px;
line-height: 1.7;
}
.button-row { display: flex; flex-wrap: wrap; gap: 11px; }
.button {
display: inline-flex;
min-height: 46px;
align-items: center;
justify-content: center;
gap: 9px;
padding: 11px 18px;
border: 1px solid transparent;
border-radius: 999px;
background: var(--green);
color: white;
font-size: 14px;
font-weight: 750;
text-decoration: none;
transition: transform 160ms ease, background 160ms ease, box-shadow 160ms ease;
}
.button:hover { transform: translateY(-2px); background: var(--button-hover); box-shadow: 0 8px 20px rgba(20, 155, 114, 0.2); }
.button-secondary { border-color: var(--line); background: var(--surface); color: var(--ink); }
.button-secondary:hover { border-color: #b8d8ca; background: var(--green-pale); color: var(--green-dark); box-shadow: none; }
.arrow { font-size: 17px; line-height: 1; }
.section { padding: 82px 0; }
.section-white { border-block: 1px solid var(--line); background: var(--surface); }
.section-heading { max-width: 700px; margin-bottom: 36px; }
.section-heading h2 {
margin-bottom: 12px;
font-size: clamp(30px, 4vw, 44px);
line-height: 1.12;
letter-spacing: -0.045em;
}
.section-heading p { margin-bottom: 0; color: var(--muted); font-size: 17px; }
.features { display: grid; grid-template-columns: repeat(3, 1fr); gap: 18px; }
.feature-card {
min-height: 226px;
padding: 27px;
border: 1px solid var(--line);
border-radius: var(--radius);
background: var(--surface);
box-shadow: 0 5px 20px rgba(16, 47, 80, 0.025);
}
.feature-icon {
width: 44px;
height: 44px;
display: grid;
place-items: center;
margin-bottom: 24px;
border-radius: 14px;
background: var(--green-pale);
color: var(--green-dark);
font-size: 20px;
font-weight: 800;
}
.feature-card:nth-child(2) .feature-icon { background: var(--blue-pale); color: var(--ink); }
.feature-card h3 { margin-bottom: 8px; font-size: 18px; letter-spacing: -0.02em; }
.feature-card p { margin-bottom: 0; color: var(--muted); font-size: 14px; }
.how-grid { display: grid; grid-template-columns: 0.85fr 1.15fr; gap: 70px; align-items: start; }
.how-intro { position: sticky; top: 118px; }
.how-intro p { color: var(--ink-soft); }
.steps-list { display: grid; gap: 0; }
.step {
display: grid;
grid-template-columns: 48px 1fr;
gap: 19px;
padding: 0 0 30px;
}
.step + .step { padding-top: 26px; border-top: 1px solid var(--line); }
.step-number {
width: 42px;
height: 42px;
display: grid;
place-items: center;
border: 1px solid var(--step-line);
border-radius: 50%;
background: var(--green-pale);
color: var(--green-dark);
font-size: 14px;
font-weight: 800;
}
.step h3 { margin: 3px 0 6px; font-size: 17px; }
.step p { margin-bottom: 0; color: var(--muted); font-size: 14px; }
.roadmap-list { display: grid; grid-template-columns: repeat(2, 1fr); gap: 12px; padding: 0; list-style: none; }
.roadmap-list li {
display: flex;
align-items: flex-start;
gap: 15px;
padding: 19px 20px;
border: 1px solid var(--line);
border-radius: 15px;
background: var(--surface);
color: var(--ink-soft);
font-size: 14px;
}
.version {
flex: 0 0 auto;
min-width: 48px;
padding: 4px 8px;
border-radius: 8px;
background: var(--blue-pale);
color: var(--ink);
font-size: 12px;
font-weight: 850;
text-align: center;
}
.setup-section { background: var(--setup-bg); color: white; }
.setup-section .eyebrow { color: #8be0bc; }
.setup-section .section-heading p { color: #c0cedb; }
.setup-section .section-heading a { color: #8be0bc; }
.setup-layout { display: grid; grid-template-columns: 1fr 0.9fr; gap: 22px; align-items: start; }
.setup-card {
padding: 27px;
border: 1px solid rgba(255,255,255,0.14);
border-radius: var(--radius);
background: rgba(255,255,255,0.06);
}
.setup-card h3 { margin-bottom: 19px; font-size: 18px; }
.setup-steps { display: grid; gap: 20px; margin: 0; padding: 0; list-style: none; counter-reset: setup; }
.setup-steps li { position: relative; padding-left: 40px; counter-increment: setup; }
.setup-steps li::before {
position: absolute;
top: 1px;
left: 0;
width: 26px;
height: 26px;
display: grid;
place-items: center;
border-radius: 50%;
background: rgba(139, 224, 188, 0.15);
color: #8be0bc;
content: counter(setup);
font-size: 12px;
font-weight: 800;
}
.setup-steps strong { display: block; margin-bottom: 7px; font-size: 14px; }
.setup-steps p, .setup-card > p { margin-bottom: 9px; color: #c0cedb; font-size: 13px; }
.required-vars { margin: 8px 0 12px; padding-left: 18px; color: #c0cedb; font-size: 12px; }
.required-vars li { position: static; padding-left: 0; counter-increment: none; }
.required-vars li::before { content: none; }
.required-vars li + li { margin-top: 5px; }
.required-vars code { overflow-wrap: anywhere; }
code, pre { font-family: "SFMono-Regular", Consolas, "Liberation Mono", monospace; }
code { color: inherit; }
.setup-card code:not(pre code) {
padding: 2px 5px;
border-radius: 5px;
background: rgba(255,255,255,0.1);
color: #d5f5e6;
overflow-wrap: anywhere;
}
pre {
overflow-x: auto;
margin: 9px 0 0;
padding: 13px 15px;
border: 1px solid rgba(255,255,255,0.12);
border-radius: 11px;
background: rgba(4, 18, 31, 0.7);
color: #e9f3fb;
font-size: 12px;
line-height: 1.7;
white-space: pre;
}
.webhook-card { border-color: rgba(139, 224, 188, 0.3); background: rgba(20, 155, 114, 0.11); }
.webhook-card h3 { display: flex; align-items: center; gap: 10px; }
.webhook-card h3::before { color: #8be0bc; content: "↗"; }
.webhook-card .callout {
margin: 15px 0;
padding: 12px 14px;
border-left: 3px solid #8be0bc;
border-radius: 0 9px 9px 0;
background: rgba(255,255,255,0.07);
color: #e4f4ed;
font-size: 13px;
}
.setup-card .subtle { margin-top: 13px; color: #aabaca; font-size: 12px; }
.source-link { display: inline-flex; align-items: center; gap: 8px; margin-top: 24px; color: #8be0bc; font-size: 14px; font-weight: 750; text-decoration: none; }
.source-link:hover { color: white; }
footer { padding: 26px 0; background: #0b233b; color: #b6c5d3; font-size: 13px; }
.footer-inner { display: flex; align-items: center; justify-content: space-between; gap: 20px; }
footer a { color: white; text-decoration: none; }
footer a:hover { color: #8be0bc; }
.footer-brand { color: white; font-weight: 800; letter-spacing: 0.1em; }
:focus-visible { outline: 3px solid #35bb8e; outline-offset: 4px; }
@media (max-width: 820px) {
.nav-links { gap: 18px; }
.hero-copy, .how-grid, .setup-layout { grid-template-columns: 1fr; gap: 26px; }
.hero-copy { padding: 36px 4px 0; }
.hero-aside p { max-width: 620px; }
.how-intro { position: static; }
.features { grid-template-columns: 1fr; }
.feature-card { min-height: auto; }
}
@media (max-width: 600px) {
.container { width: min(var(--content), calc(100% - 32px)); }
.nav { min-height: 66px; gap: 12px; }
.nav-links { gap: 12px; }
.nav-links a { font-size: 12px; }
.nav-links a:nth-child(1), .nav-links a:nth-child(2), .nav-links .nav-cta { display: none; }
.theme-toggle { padding: 8px 10px; font-size: 12px; }
.wordmark { font-size: 15px; }
.wordmark-mark { width: 30px; height: 30px; }
.hero { padding: 22px 0 64px; }
.brand-banner { border-radius: 16px; }
.hero-copy { padding-top: 29px; }
h1 { font-size: clamp(38px, 11vw, 54px); }
.section { padding: 62px 0; }
.roadmap-list { grid-template-columns: 1fr; }
.setup-card { padding: 21px; }
.footer-inner { align-items: flex-start; flex-direction: column; gap: 6px; }
}
@media (prefers-reduced-motion: reduce) {
*, *::before, *::after { scroll-behavior: auto !important; transition-duration: 0.01ms !important; }
}
</style>
<script>
try {
const savedTheme = localStorage.getItem("steep-theme");
if (savedTheme === "light" || savedTheme === "dark") {
document.documentElement.dataset.theme = savedTheme;
}
} catch (_) {}
</script>
</head>
<body>
<header class="site-header">
<nav class="nav container" aria-label="Main navigation">
<a class="wordmark" href="#top" aria-label="Steep home"><span class="wordmark-mark">S</span> STEEP</a>
<div class="nav-links">
<a href="#features">Features</a>
<a href="#roadmap">Roadmap</a>
<a href="#setup">Get started</a>
<a class="nav-cta" href="https://gitea.qpismont.fr/qpismont/herald">View source <span class="arrow" aria-hidden="true">↗</span></a>
<button class="theme-toggle" id="theme-toggle" type="button" role="switch" aria-checked="true" aria-label="Dark theme">
<span class="theme-icon" id="theme-icon" aria-hidden="true">☾</span><span>Dark theme</span>
</button>
</div>
</nav>
</header>
<main id="top">
<section class="hero container" aria-labelledby="hero-title">
<div class="brand-banner">
<img src="banniere_readme.png" alt="Steep — Let your code steep">
</div>
<div class="hero-copy">
<div>
<p class="eyebrow">AI code review · built for Gitea</p>
<h1 id="hero-title">Code reviews with the full picture.</h1>
</div>
<div class="hero-aside">
<p>Steep explores pull requests inside an isolated Dev Container, then leaves focused, line-by-line feedback right where your team works.</p>
<div class="button-row">
<a class="button" href="#setup">Get started <span class="arrow" aria-hidden="true">↓</span></a>
<a class="button button-secondary" href="#how-it-works">How it works</a>
</div>
</div>
</div>
</section>
<section class="section section-white" id="features" aria-labelledby="features-title">
<div class="container">
<div class="section-heading">
<p class="eyebrow">Thoughtful by design</p>
<h2 id="features-title">Useful reviews. No noisy pipeline.</h2>
<p>Bring an AI reviewer into your Gitea workflow with a single comment.</p>
</div>
<div class="features">
<article class="feature-card">
<div class="feature-icon" aria-hidden="true">↳</div>
<h3>Reviews on demand</h3>
<p>Mention Steep in a pull-request comment and get an AI review streamed back to the PR. No CI hook to wire, no pipeline to learn.</p>
</article>
<article class="feature-card">
<div class="feature-icon" aria-hidden="true">◇</div>
<h3>Sandboxed by design</h3>
<p>Every review runs in an ephemeral Dev Container with read-only tools. The network is cut after setup, and the container is removed when the review ends.</p>
</article>
<article class="feature-card">
<div class="feature-icon" aria-hidden="true">⌁</div>
<h3>Lightweight & observable</h3>
<p>A lean Rust service designed for fast reviews, with Prometheus metrics and Sentry error tracking.</p>
</article>
</div>
</div>
</section>
<section class="section" id="how-it-works" aria-labelledby="how-title">
<div class="container how-grid">
<div class="how-intro">
<p class="eyebrow">From comment to context</p>
<div class="section-heading">
<h2 id="how-title">Your code stays in its environment.</h2>
<p>Steep gives the model tools to inspect the repository instead of sending the source code in the prompt.</p>
</div>
<a class="button button-secondary" href="#setup">Set up Steep <span class="arrow" aria-hidden="true">↓</span></a>
</div>
<div class="steps-list">
<article class="step">
<span class="step-number">01</span>
<div><h3>Steep hears your request</h3><p>A Gitea pull-request comment triggers a review when you mention the bot.</p></div>
</article>
<article class="step">
<span class="step-number">02</span>
<div><h3>A disposable workspace spins up</h3><p>Steep clones the pull-request head and starts the repository's Dev Container—or a safe default when none is configured.</p></div>
</article>
<article class="step">
<span class="step-number">03</span>
<div><h3>The model explores, read-only</h3><p>It can inspect files with tools such as <code>ls</code>, <code>read_file</code>, and <code>grep</code>. Setup hooks run first; then network access is removed.</p></div>
</article>
<article class="step">
<span class="step-number">04</span>
<div><h3>Feedback lands on the changed lines</h3><p>Steep posts actionable findings with severity labels, plus a concise summary of what the PR does well.</p></div>
</article>
</div>
</div>
</section>
<section class="section section-white" id="roadmap" aria-labelledby="roadmap-title">
<div class="container">
<div class="section-heading">
<p class="eyebrow">What’s brewing</p>
<h2 id="roadmap-title">The roadmap</h2>
<p>From safer reviews to an assistant that can help move work forward.</p>
</div>
<ul class="roadmap-list">
<li><span class="version">1.2</span><span>Sandboxed pull-request reviews.</span></li>
<li><span class="version">1.2.1</span><span>A default Dev Container when a repository has no configuration.</span></li>
<li><span class="version">1.3</span><span>Durable reviews: persist jobs in PostgreSQL, list them by state (running, pending, review done), and stream live updates over SSE.</span></li>
<li><span class="version">1.4</span><span>Configuration from a YAML file, replacing part of the environment variables.</span></li>
<li><span class="version">2.0</span><span>Type-safe action routing between reviews and implementation requests, implement requested changes from issues or existing PRs, then create or update a PR automatically.</span></li>
<li><span class="version">2.1</span><span>Project instructions from <code>AGENTS.md</code> and separate review/implementation models.</span></li>
<li><span class="version">2.2</span><span>Sub-agents that take on well-scoped parts of a task and run in parallel.</span></li>
<li><span class="version">3.0</span><span>Automatically investigate and fix project errors reported by Sentry.</span></li>
</ul>
</div>
</section>
<section class="section setup-section" id="setup" aria-labelledby="setup-title">
<div class="container">
<div class="section-heading">
<p class="eyebrow">A few minutes to get steeping</p>
<h2 id="setup-title">Bring Steep to your Gitea.</h2>
<p>Follow these steps in order. You will need a <a href="https://rustup.rs/">Rust toolchain</a>, a reachable Docker or Podman daemon, an OpenRouter API key, and a Gitea account for Steep.</p>
</div>
<div class="setup-layout">
<article class="setup-card">
<h3>1. Install and configure</h3>
<ol class="setup-steps">
<li>
<strong>Clone the source</strong>
<pre>git clone https://gitea.qpismont.fr/qpismont/herald.git
cd herald</pre>
</li>
<li>
<strong>Create a Gitea token</strong>
<p>Create a token for the account Steep should act as. Give it access to the repositories it will review, including permission to read pull requests and publish comments/reviews. Steep uses this token to identify the account; you will use that account's username when asking for a review.</p>
</li>
<li>
<strong>Configure every required setting</strong>
<pre>cp .env.example .env
$EDITOR .env</pre>
<p>Set all nine required variables:</p>
<ul class="required-vars">
<li><code>GITEA_URL</code> — base URL, e.g. <code>https://gitea.example.com</code> (no <code>/api/v1</code>)</li>
<li><code>GITEA_TOKEN</code> — the token created above</li>
<li><code>GITEA_TIMEOUT</code> — API timeout in seconds</li>
<li><code>OPEN_ROUTER_API_KEY</code> — your OpenRouter key</li>
<li><code>OPEN_ROUTER_MODEL</code> — an available OpenRouter model ID</li>
<li><code>OPEN_ROUTER_TIMEOUT</code> — request timeout in seconds</li>
<li><code>BOT_MAX_CONCURRENT</code> — maximum simultaneous reviews</li>
<li><code>HTTP_PORT</code> — web server port; use <code>3001</code> for the commands below</li>
<li><code>WEBHOOK_SIG_HEADER_SECRET</code> — generate this in the next step</li>
</ul>
<p class="subtle">Keep the example's numeric values for the timeouts and concurrency, or replace them with positive integers. Do not commit your completed <code>.env</code>.</p>
</li>
<li>
<strong>Generate and configure the webhook secret</strong>
<pre>openssl rand -hex 32</pre>
<p>Copy the output into <code>WEBHOOK_SIG_HEADER_SECRET</code> in <code>.env</code>. You will enter the exact same value in Gitea in the next step.</p>
</li>
<li>
<strong>Start the daemon, build, and launch Steep</strong>
<pre>podman system service --time=0 &amp;
export DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock
cargo build --release -p steep-server
./target/release/steep-server</pre>
<p class="subtle">Keep this process running. With Docker, omit the Podman command and use its socket; the default is <code>unix:///var/run/docker.sock</code>. Optional settings such as metrics and Sentry can be added later.</p>
</li>
</ol>
<a class="source-link" href="https://gitea.qpismont.fr/qpismont/herald">Explore the source on Gitea <span aria-hidden="true">↗</span></a>
</article>
<article class="setup-card webhook-card">
<h3>2. Add the Gitea webhook</h3>
<ol class="setup-steps">
<li>
<strong>Choose an address Gitea can reach</strong>
<p>In the repository's <strong>Settings → Webhooks</strong>, add a Gitea webhook. Set the target to Steep's public/reachable address and port:</p>
<div class="callout"><code>http://steep.example.com:3001/webhook</code><br>Replace the host and scheme with yours. Do not use <code>localhost</code> unless Gitea and Steep share that network.</div>
</li>
<li>
<strong>Select the event and set the secret</strong>
<p>Enable at least <strong>Pull Request Comment</strong>. Paste the generated value into Gitea's <strong>Secret</strong> field; it must exactly match <code>WEBHOOK_SIG_HEADER_SECRET</code>.</p>
</li>
<li>
<strong>Allow the connection</strong>
<p>Make sure Gitea can connect to Steep on port <code>3001</code> (or your configured <code>HTTP_PORT</code>), including through any firewall or reverse proxy. Steep must also be able to reach Gitea's API, OpenRouter, and the container daemon.</p>
</li>
<li>
<strong>Trigger a review</strong>
<p>Open a pull request and add a comment that starts with <code>@username</code>, using the username of the Gitea account behind <code>GITEA_TOKEN</code>. For example:</p>
<pre>@steep review this PR</pre>
<p class="subtle">The mention must be at the very start of the comment. If nothing happens, check the webhook's delivery history in Gitea and Steep's logs.</p>
</li>
</ol>
</article>
</div>
</div>
</section>
</main>
<footer>
<div class="container footer-inner">
<span><span class="footer-brand">STEEP</span> · Let your code steep.</span>
<span>Built with Rust · <a href="https://gitea.qpismont.fr/qpismont/herald">Source on Gitea</a></span>
</div>
</footer>
<script>
const themeToggle = document.getElementById("theme-toggle");
const themeIcon = document.getElementById("theme-icon");
const themeColor = document.querySelector('meta[name="theme-color"]');
function setTheme(theme, persist = true) {
document.documentElement.dataset.theme = theme;
themeToggle.setAttribute("aria-checked", String(theme === "dark"));
themeIcon.textContent = theme === "dark" ? "☾" : "☀";
themeColor.content = theme === "dark" ? "#0b1520" : "#f7f8f5";
if (persist) {
try {
localStorage.setItem("steep-theme", theme);
} catch (_) {}
}
}
setTheme(document.documentElement.dataset.theme || "dark", false);
themeToggle.addEventListener("click", () => {
setTheme(document.documentElement.dataset.theme === "dark" ? "light" : "dark");
});
</script>
</body>
</html>
+23
View File
@@ -0,0 +1,23 @@
#!/usr/bin/env bash
set -euo pipefail
IMAGE="docker.io/tintounn/steep-website"
if [ -z "${CI_COMMIT_TAG:-}" ]; then
echo "Error: CI_COMMIT_TAG is not set" >&2
exit 1
fi
TAG="${CI_COMMIT_TAG}"
IMAGE_REF="${IMAGE}:${TAG}"
echo "Building ${IMAGE_REF}..."
buildah build \
--file public/Containerfile \
--tag "${IMAGE_REF}" \
public
echo "Pushing ${IMAGE_REF}..."
buildah push "${IMAGE_REF}"
echo "Done: ${IMAGE_REF}"
+1 -1
View File
@@ -1,7 +1,7 @@
#!/usr/bin/env bash #!/usr/bin/env bash
set -euo pipefail set -euo pipefail
IMAGE="tintounn/herald" IMAGE="tintounn/steep"
if [ -z "${CI_COMMIT_TAG:-}" ]; then if [ -z "${CI_COMMIT_TAG:-}" ]; then
echo "Error: CI_COMMIT_TAG is not set" >&2 echo "Error: CI_COMMIT_TAG is not set" >&2
+19
View File
@@ -0,0 +1,19 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
if [ -f "${ROOT_DIR}/.env" ]; then
set -a
# shellcheck disable=SC1091
. "${ROOT_DIR}/.env"
set +a
fi
cd "${ROOT_DIR}"
if [ -x "${ROOT_DIR}/steep-server" ]; then
exec "${ROOT_DIR}/steep-server" migration apply "$@"
else
exec cargo run -p steep-server -- migration apply "$@"
fi
+22
View File
@@ -0,0 +1,22 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
if [ -f "${ROOT_DIR}/.env" ]; then
set -a
. "${ROOT_DIR}/.env"
set +a
fi
cd "${ROOT_DIR}"
if [ "$#" -eq 0 ]; then
echo "Usage: $0 <nom_de_la_migration>" >&2
echo "Exemple: $0 add_users_table" >&2
exit 1
fi
NAME="$1"
exec cargo run -p steep-server -- migration generate --name "${NAME}"
-148
View File
@@ -1,148 +0,0 @@
use axum::body::{Bytes, to_bytes};
use axum::extract::{FromRef, FromRequest, State};
use axum::http::Request;
use axum::response::IntoResponse;
use axum::routing::{get, post};
use axum::{Json, Router};
use hmac::{Hmac, KeyInit, Mac};
use reqwest::StatusCode;
use sentry::integrations::tower::{NewSentryLayer, SentryHttpLayer};
use serde_json::Value;
use sha2::Sha256;
use subtle::ConstantTimeEq;
use tower::ServiceBuilder;
use tower_http::trace::TraceLayer;
use tracing::{info, instrument};
use tokio_util::sync::CancellationToken;
use crate::consts::{GITEA_EVENT_TYPE_HEADER_NAME, GITEA_SIG_HEADER_NAME, MAX_WEBHOOK_BODY_SIZE};
use crate::errors::AppError;
use crate::gitea::WebhookType;
use crate::state::AppState;
pub async fn start(app_state: AppState, shutdown: CancellationToken) -> anyhow::Result<()> {
let http_port = app_state.config.http_port;
let app = Router::new()
.route("/", get(root))
.route("/webhook", post(webhook))
.layer(
ServiceBuilder::new()
.layer(NewSentryLayer::<Request<_>>::new_from_top())
.layer(SentryHttpLayer::new())
.layer(TraceLayer::new_for_http()),
)
.with_state(app_state);
let listener = tokio::net::TcpListener::bind(format!("0.0.0.0:{}", http_port)).await?;
info!("Listening API on port {}", http_port);
axum::serve(listener, app)
.with_graceful_shutdown(async move { shutdown.cancelled().await })
.await
.map_err(anyhow::Error::from)
.inspect(|_| info!("API shutting down complete"))?;
Ok(())
}
async fn root() -> &'static str {
"Hi, i'm Herald :)"
}
#[instrument(skip(app_state), fields(webhook_type), err)]
async fn webhook(
State(app_state): State<AppState>,
WebhookExtract(wb): WebhookExtract,
) -> Result<impl IntoResponse, AppError> {
tracing::Span::current().record("webhook_type", tracing::field::debug(&wb));
let event_id = wb.event_id();
if app_state.bot.check_and_mark(event_id).await {
return Err(AppError::AlreadyProcessedErr);
}
if app_state.bot_tx.try_send(wb).is_err() {
app_state.bot.unmark(event_id).await;
return Err(AppError::ChannelFullErr);
}
Ok((StatusCode::CREATED, "Task started"))
}
pub struct WebhookExtract(pub WebhookType);
impl<S> FromRequest<S> for WebhookExtract
where
AppState: FromRef<S>,
S: Send + Sync,
{
type Rejection = AppError;
#[instrument(skip(req, state), err)]
async fn from_request(req: axum::extract::Request, state: &S) -> Result<Self, Self::Rejection> {
let app_state = AppState::from_ref(state);
let headers = req.headers();
let sig_header = extract_header(GITEA_SIG_HEADER_NAME, headers)?;
let type_header = extract_header(GITEA_EVENT_TYPE_HEADER_NAME, headers)?;
let body_bytes = read_body(req.into_body()).await?;
verify_signature(
app_state.config.webhook_secret.as_bytes(),
&sig_header,
&body_bytes,
)?;
let body_str = String::from_utf8_lossy(&body_bytes).into_owned();
sentry::configure_scope(|scope| {
scope.add_event_processor(move |mut event| {
let mut request = event.request.take().unwrap_or_default();
request.data = Some(body_str.clone());
event.request = Some(request);
Some(event)
});
});
let webhook = parse_webhook(&type_header, &app_state.config.bot_name, &body_bytes)?;
Ok(WebhookExtract(webhook))
}
}
fn extract_header(key: &str, headers: &axum::http::HeaderMap) -> Result<String, AppError> {
let value = headers
.get(key)
.ok_or(AppError::WebHookMissingHeaderErr(key.into()))?
.to_str()
.map_err(anyhow::Error::from)?;
Ok(value.to_owned())
}
async fn read_body(body: axum::body::Body) -> Result<Bytes, AppError> {
to_bytes(body, MAX_WEBHOOK_BODY_SIZE)
.await
.map_err(anyhow::Error::from)
.map_err(AppError::from)
}
fn parse_webhook(header: &str, bot_name: &str, body_bytes: &[u8]) -> Result<WebhookType, AppError> {
let Json(value) =
Json::<Value>::from_bytes(body_bytes).map_err(|_| AppError::MalformedJsonErr)?;
WebhookType::from_event(header, bot_name, value)
}
fn verify_signature(secret_key: &[u8], sig_header: &str, body: &[u8]) -> Result<(), AppError> {
let sig_header_decoded =
hex::decode(sig_header).map_err(|_| AppError::WebHookSigHeaderInvalidErr)?;
let mut mac = Hmac::<Sha256>::new_from_slice(secret_key).map_err(anyhow::Error::from)?;
mac.update(body);
let generated_hmac = mac.finalize().into_bytes();
bool::from(generated_hmac.ct_eq(&sig_header_decoded))
.then_some(())
.ok_or(AppError::WebHookSigHeaderInvalidErr)
}
-205
View File
@@ -1,205 +0,0 @@
use crate::{
env::EnvConfig,
gitea::{GiteaAPI, WebhookType},
open_router::OpenRouterClient,
};
use serde::Deserialize;
use std::{collections::HashSet, sync::Arc, time::Duration};
use tokio::sync::Mutex;
use tokio_util::sync::CancellationToken;
use tracing::{error, info, instrument};
#[derive(Deserialize, Debug)]
pub struct ReviewResult {
pub reviews: Vec<ReviewItem>,
pub comment: String,
pub cost: Option<f64>,
}
#[derive(Deserialize, Debug)]
pub struct ReviewItem {
pub filename: String,
pub line: Option<u64>,
pub code: String,
pub message: String,
}
#[derive(Clone)]
pub struct Bot {
config: EnvConfig,
gitea_api: GiteaAPI,
open_router_client: OpenRouterClient,
http_client: reqwest::Client,
max_concurrent: usize,
actions_handled: Arc<Mutex<HashSet<u64>>>,
}
impl Bot {
pub fn new(config: EnvConfig) -> anyhow::Result<Self> {
let gitea_timeout = config.gitea_timeout;
let open_router_timeout = config.open_router_timeout;
Ok(Self {
gitea_api: GiteaAPI::new(&config.gitea_url, &config.gitea_token, gitea_timeout)?,
open_router_client: OpenRouterClient::new(
&config.open_router_api_key,
&config.open_router_model,
open_router_timeout,
)?,
max_concurrent: config.bot_max_concurrent,
config,
actions_handled: Arc::new(Mutex::new(HashSet::new())),
http_client: reqwest::Client::builder()
.timeout(Duration::from_secs(gitea_timeout))
.build()?,
})
}
pub async fn start(
&self,
mut rx: tokio::sync::mpsc::Receiver<WebhookType>,
shutdown: CancellationToken,
) -> anyhow::Result<()> {
info!("Bot started");
let sem = Arc::new(tokio::sync::Semaphore::new(self.max_concurrent));
let mut tasks = tokio::task::JoinSet::new();
loop {
let wb = tokio::select! {
biased;
_ = shutdown.cancelled() => break,
msg = rx.recv() => match msg {
Some(wb) => wb,
None => break,
},
};
while let Some(res) = tasks.try_join_next() {
if let Err(e) = res {
error!("Task panicked: {e}");
}
}
info!(queued = rx.len(), active = tasks.len(), "Webhook received");
let permit = sem.clone().acquire_owned().await?;
let self_clone = self.clone();
tasks.spawn(async move {
self_clone.exec(wb).await;
drop(permit);
});
}
tasks.join_all().await;
info!("Bot shutting down complete");
Ok(())
}
#[instrument(skip(self))]
pub async fn exec(&self, webhook: WebhookType) {
tracing::Span::current().record("webhook_type", tracing::field::debug(&webhook));
let exec_result = match webhook {
WebhookType::Review(review_payload) => crate::bot_actions::review::exec_review(
&self.gitea_api,
&self.open_router_client,
&self.http_client,
&self.config.open_router_model,
review_payload,
),
}
.await;
match exec_result {
Ok(_) => info!("Task completed"),
Err(err) => {
error!(%err, "Task error");
sentry_anyhow::capture_anyhow(&err);
}
}
}
pub async fn check_and_mark(&self, event_id: u64) -> bool {
let mut action_handled_lock = self.actions_handled.lock().await;
!action_handled_lock.insert(event_id)
}
pub async fn unmark(&self, event_id: u64) {
let mut action_handled_lock = self.actions_handled.lock().await;
action_handled_lock.remove(&event_id);
}
}
#[cfg(test)]
mod tests {
use super::*;
fn make_actions_handled() -> Arc<Mutex<HashSet<u64>>> {
Arc::new(Mutex::new(HashSet::new()))
}
async fn check_and_mark(actions_handled: &Arc<Mutex<HashSet<u64>>>, event_id: u64) -> bool {
let mut lock = actions_handled.lock().await;
!lock.insert(event_id)
}
async fn unmark(actions_handled: &Arc<Mutex<HashSet<u64>>>, event_id: u64) {
let mut lock = actions_handled.lock().await;
lock.remove(&event_id);
}
#[tokio::test]
async fn test_check_and_mark_first_call_returns_false() {
let actions_handled = make_actions_handled();
assert!(!check_and_mark(&actions_handled, 1).await);
}
#[tokio::test]
async fn test_check_and_mark_second_call_returns_true() {
let actions_handled = make_actions_handled();
check_and_mark(&actions_handled, 1).await;
assert!(check_and_mark(&actions_handled, 1).await);
}
#[tokio::test]
async fn test_check_and_mark_different_ids_are_independent() {
let actions_handled = make_actions_handled();
check_and_mark(&actions_handled, 1).await;
assert!(!check_and_mark(&actions_handled, 2).await);
}
#[tokio::test]
async fn test_unmark_allows_reprocessing() {
let actions_handled = make_actions_handled();
check_and_mark(&actions_handled, 1).await;
unmark(&actions_handled, 1).await;
assert!(!check_and_mark(&actions_handled, 1).await);
}
#[tokio::test]
async fn test_unmark_nonexistent_id_is_noop() {
let actions_handled = make_actions_handled();
unmark(&actions_handled, 99).await;
assert!(!check_and_mark(&actions_handled, 99).await);
}
#[tokio::test]
async fn test_check_and_mark_concurrent_same_id() {
let actions_handled = make_actions_handled();
let actions_handled2 = Arc::clone(&actions_handled);
let t1 = tokio::spawn(async move { check_and_mark(&actions_handled, 42).await });
let t2 = tokio::spawn(async move { check_and_mark(&actions_handled2, 42).await });
let (r1, r2) = tokio::join!(t1, t2);
let results = [r1.unwrap(), r2.unwrap()];
// exactement un seul des deux doit retourner false (non traité)
assert_eq!(results.iter().filter(|&&r| !r).count(), 1);
// l'autre doit retourner true (déjà traité)
assert_eq!(results.iter().filter(|&&r| r).count(), 1);
}
}
-214
View File
@@ -1,214 +0,0 @@
use futures_util::stream::TryStreamExt;
use tokio::io::AsyncReadExt;
use tokio_util::io::StreamReader;
use tracing::instrument;
use crate::{
bot::ReviewResult,
consts::{BOT_PROCESS_MSG, MAX_DIFF_SIZE, REVIEW_PROMPT},
gitea::{GiteaAPI, ReviewPayload},
open_router::OpenRouterClient,
};
#[instrument(skip(gitea_api, open_router_client, http_client, review_payload), err)]
pub async fn exec_review(
gitea_api: &GiteaAPI,
open_router_client: &OpenRouterClient,
http_client: &reqwest::Client,
model: &str,
review_payload: ReviewPayload,
) -> anyhow::Result<()> {
tracing::info!(
repo = %review_payload.repository.full_name,
pr = review_payload.pull_request.number,
action = %review_payload.action,
"Starting review"
);
let new_comment = gitea_api
.comment(
&BOT_PROCESS_MSG.replace("{model}", model),
&review_payload.repository.full_name,
review_payload.pull_request.number,
)
.await?;
let bot_result: Result<ReviewResult, anyhow::Error> = async {
let git_diff =
download_git_diff(http_client, &review_payload.pull_request.diff_url).await?;
let diff_for_llm = format_diff_for_review(&git_diff);
let bot_request = REVIEW_PROMPT
.replace("{subject}", &review_payload.pull_request.title)
.replace("{comment}", &review_payload.comment.body)
.replace("{diff}", &diff_for_llm);
let chat_result = open_router_client.chat(&bot_request).await?;
let mut review_result = serde_json::from_str::<ReviewResult>(&chat_result.message)?;
review_result.cost = chat_result.cost;
let final_review_markdown = review_result_to_markdown(&review_result);
gitea_api
.post_pull_request_review(
&review_result,
&final_review_markdown,
&review_payload.repository.full_name,
review_payload.pull_request.number,
)
.await?;
Ok(review_result)
}
.await;
match bot_result {
Ok(_) => {
gitea_api
.delete_comment(&review_payload.repository.full_name, new_comment.id)
.await
}
Err(e) => {
gitea_api
.edit_comment(
&format!("Error while reviewing: {}", e),
&review_payload.repository.full_name,
new_comment.id,
)
.await
}
}
}
fn review_result_to_markdown(review_result: &ReviewResult) -> String {
if review_result.reviews.is_empty() {
return String::from("No issues found. ✅");
}
let mut md = String::from("## Review Feedback\n\n");
md.push_str(&format!(
"### {} issues found.\n\n",
review_result.reviews.len()
));
if !review_result.comment.is_empty() {
md.push_str("\n---\n\n");
md.push_str("### Summary\n\n");
md.push_str(&review_result.comment);
md.push('\n');
}
if let Some(cost) = review_result.cost {
md.push_str("\n---\n\n");
md.push_str(&format!("### Cost: ${}", cost));
md.push('\n');
}
md
}
async fn download_git_diff(http_client: &reqwest::Client, url: &str) -> anyhow::Result<String> {
let response = http_client.get(url).send().await?;
let stream = response.bytes_stream().map_err(std::io::Error::other);
let mut buf = Vec::with_capacity(MAX_DIFF_SIZE);
StreamReader::new(stream)
.take((MAX_DIFF_SIZE + 1) as u64)
.read_to_end(&mut buf)
.await?;
if buf.len() > MAX_DIFF_SIZE {
anyhow::bail!("Git diff exceeds the maximum allowed size of 1 Mo");
}
Ok(String::from_utf8_lossy(&buf).into_owned())
}
fn format_diff_for_review(git_diff: &str) -> String {
let mut output = String::new();
let mut current_file: Option<&str> = None;
let mut new_line: u64 = 0;
for line in git_diff.lines() {
if let Some(rest) = line.strip_prefix("diff --git a/") {
if let Some(end) = rest.find(' ') {
current_file = Some(&rest[..end]);
}
new_line = 0;
continue;
}
if line.starts_with("---") || line.starts_with("+++") {
continue;
}
if line.starts_with("@@") && line.contains('+') {
if let Some(start) = parse_hunk_new_start(line) {
new_line = start;
}
continue;
}
let Some(filename) = current_file else {
continue;
};
if line.starts_with(' ') {
new_line += 1;
continue;
}
if let Some(code) = line.strip_prefix('+') {
use std::fmt::Write;
let _ = writeln!(output, "{filename}:{new_line}:{code}");
new_line += 1;
}
}
output
}
fn parse_hunk_new_start(hunk_header: &str) -> Option<u64> {
let plus_part = hunk_header.split('+').nth(1)?;
let num_str = plus_part.split(|c: char| !c.is_ascii_digit()).next()?;
num_str.parse::<u64>().ok()
}
#[cfg(test)]
#[test]
fn test_format_diff_for_review() {
let diff = concat!(
"diff --git a/src/foo.rs b/src/foo.rs\n",
"--- a/src/foo.rs\n",
"+++ b/src/foo.rs\n",
"@@ -1,3 +1,6 @@\n",
" fn main() {\n",
"+ let x = 1;\n",
" println!(\"hello\");\n",
"+ let y = 2;\n",
"+ let z = 3;\n",
" }\n",
"diff --git a/src/bar.rs b/src/bar.rs\n",
"--- a/src/bar.rs\n",
"+++ b/src/bar.rs\n",
"@@ -10,4 +10,6 @@\n",
" old context\n",
"+ let a = 10;\n",
" more context\n",
"+ let b = 20;\n",
);
let result = format_diff_for_review(diff);
let expected = concat!(
"src/foo.rs:2: let x = 1;\n",
"src/foo.rs:4: let y = 2;\n",
"src/foo.rs:5: let z = 3;\n",
"src/bar.rs:11: let a = 10;\n",
"src/bar.rs:13: let b = 20;\n",
);
assert_eq!(result, expected);
}
-44
View File
@@ -1,44 +0,0 @@
pub const GITEA_SIG_HEADER_NAME: &str = "x-gitea-signature";
pub const GITEA_EVENT_TYPE_HEADER_NAME: &str = "x-gitea-event-type";
pub const MAX_WEBHOOK_BODY_SIZE: usize = 1024 * 1024; // 1 MiB
pub const MAX_DIFF_SIZE: usize = 1024 * 1024; // 1 MiB
pub const BOT_PROCESS_MSG: &str = "
Review in progress with the model \"{model}\"...
";
pub const REVIEW_PROMPT: &str = "
You are a senior software engineer reviewing code changes.
Check good practices and code quality.
This is the pull request subject: \"{subject}\"
This is the user comment: \"{comment}\"
The code changes (only added lines, with line numbers):
{diff}
Please review the code changes and provide feedback.
IMPORTANT: the `line` field must be the line number shown before each line.
The provided code has the format: `filename:line:code`
Return your feedback, in french, with only this json format, reviews must contain each review
All fields are mandatory.
(filename field must contain the full path with extension) and comment must contain a final summary:
{
\"reviews\": [
{
\"filename\": \"\",
\"line\": ,
\"code\": \"\",
\"message\": \"\"
}
],
\"comment\": \"\"
}
";
-82
View File
@@ -1,82 +0,0 @@
use anyhow::anyhow;
#[derive(Clone)]
pub struct EnvConfig {
pub http_port: u16,
pub webhook_secret: String,
pub open_router_api_key: String,
pub open_router_model: String,
pub open_router_timeout: u64,
pub bot_name: String,
pub bot_max_concurrent: usize,
pub gitea_url: String,
pub gitea_token: String,
pub gitea_timeout: u64,
}
pub fn load_config() -> anyhow::Result<EnvConfig> {
let http_port = try_get_env("HTTP_PORT")?.parse()?;
let bot_name = try_get_env("BOT_NAME")?;
let webhook_secret = try_get_env("WEBHOOK_SIG_HEADER_SECRET")?;
let open_router_api_key = try_get_env("OPEN_ROUTER_API_KEY")?;
let open_router_model = try_get_env("OPEN_ROUTER_MODEL")?;
let open_router_timeout = try_get_env("OPEN_ROUTER_TIMEOUT")?.parse()?;
let bot_max_concurrent = try_get_env("BOT_MAX_CONCURRENT")?.parse()?;
let gitea_url = try_get_env("GITEA_URL")?;
let gitea_token = try_get_env("GITEA_TOKEN")?;
let gitea_timeout = try_get_env("GITEA_TIMEOUT")?.parse()?;
Ok(EnvConfig {
http_port,
webhook_secret,
bot_name,
open_router_api_key,
open_router_model,
open_router_timeout,
bot_max_concurrent,
gitea_url,
gitea_token,
gitea_timeout,
})
}
pub fn try_get_env(key: &str) -> anyhow::Result<String> {
let env = std::env::var(key)?;
if env.trim().is_empty() {
return Err(anyhow!(format!("env var {} is empty", key)));
}
Ok(env)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_try_get_env_returns_value() {
unsafe { std::env::set_var("TEST_ENV_PRESENT", "hello") };
assert_eq!(try_get_env("TEST_ENV_PRESENT").unwrap(), "hello");
}
#[test]
fn test_try_get_env_missing_var_returns_error() {
unsafe { std::env::remove_var("TEST_ENV_MISSING") };
assert!(try_get_env("TEST_ENV_MISSING").is_err());
}
#[test]
fn test_try_get_env_empty_value_returns_error() {
unsafe { std::env::set_var("TEST_ENV_EMPTY", "") };
let err = try_get_env("TEST_ENV_EMPTY").unwrap_err();
assert!(err.to_string().contains("TEST_ENV_EMPTY"));
}
#[test]
fn test_try_get_env_whitespace_only_returns_error() {
unsafe { std::env::set_var("TEST_ENV_WHITESPACE", " ") };
let err = try_get_env("TEST_ENV_WHITESPACE").unwrap_err();
assert!(err.to_string().contains("TEST_ENV_WHITESPACE"));
}
}
-94
View File
@@ -1,94 +0,0 @@
use std::sync::Arc;
use crate::{bot::Bot, gitea::WebhookType, state::AppState};
use dotenvy::dotenv;
use tokio::signal::unix::{SignalKind, signal};
use tokio_util::sync::CancellationToken;
use tracing::{info, warn};
use tracing_subscriber::{EnvFilter, fmt, layer::SubscriberExt, util::SubscriberInitExt};
mod api;
mod bot;
mod bot_actions;
mod consts;
mod env;
mod errors;
mod gitea;
mod open_router;
mod state;
fn main() -> anyhow::Result<()> {
dotenv().ok();
tracing_subscriber::registry()
.with(fmt::layer())
.with(
EnvFilter::try_from_default_env() // lit RUST_LOG depuis l'env
.unwrap_or_else(|_| EnvFilter::new("info")),
)
.init();
let _sentry_guard = if let Ok(sentry_dsn) = env::try_get_env("SENTRY_DSN") {
info!("Initialize sentry");
Some(sentry::init((
sentry_dsn,
sentry::ClientOptions {
release: sentry::release_name!(),
send_default_pii: true,
..Default::default()
},
)))
} else {
warn!("SENTRY_DSN not set, sentry will not be initialized");
None
};
tokio::runtime::Runtime::new()?.block_on(run())
}
async fn run() -> anyhow::Result<()> {
let config = env::load_config()?;
info!(
port = config.http_port,
model = %config.open_router_model,
gitea_url = %config.gitea_url,
bot_name = %config.bot_name,
"Starting Herald"
);
let shutdown = CancellationToken::new();
let bot = Bot::new(config.clone())?;
let (tx, rx) = tokio::sync::mpsc::channel::<WebhookType>(config.bot_max_concurrent * 2);
let app_state = AppState {
bot_tx: tx,
bot: bot.clone(),
config,
};
let signal = async {
let mut sigterm = signal(SignalKind::terminate())?;
let mut sigint = signal(SignalKind::interrupt())?;
tokio::select! {
_ = sigterm.recv() => info!("Received SIGTERM"),
_ = sigint.recv() => info!("Received SIGINT"),
}
info!("Shutting down...");
shutdown.cancel();
anyhow::Ok(())
};
tokio::try_join!(
bot.start(rx, shutdown.clone()),
api::start(app_state, shutdown.clone()),
signal
)?;
info!("Shutdown complete");
Ok(())
}
-50
View File
@@ -1,50 +0,0 @@
use std::time::Duration;
use openrouter_rs::{Message, api::chat::ChatCompletionRequest};
use tracing::instrument;
pub struct ChatResult {
pub message: String,
pub cost: Option<f64>,
}
#[derive(Clone)]
pub struct OpenRouterClient {
client: openrouter_rs::OpenRouterClient,
model: String,
}
impl OpenRouterClient {
pub fn new(token: &str, model: &str, timeout: u64) -> anyhow::Result<Self> {
Ok(Self {
client: openrouter_rs::OpenRouterClient::builder()
.api_key(token)
.http_client(
reqwest::Client::builder()
.timeout(Duration::from_secs(timeout))
.build()?,
)
.build()?,
model: String::from(model),
})
}
#[instrument(skip(self), err)]
pub async fn chat(&self, msg: &str) -> anyhow::Result<ChatResult> {
let request = ChatCompletionRequest::builder()
.model(&self.model)
.enable_reasoning()
.messages(vec![Message::new(openrouter_rs::types::Role::User, msg)])
.build()?;
let response = self.client.chat().create(&request).await?;
Ok(ChatResult {
message: response.choices[0]
.content()
.map(String::from)
.ok_or(anyhow::anyhow!("No content"))?,
cost: response.usage.and_then(|u| u.cost),
})
}
}
-8
View File
@@ -1,8 +0,0 @@
use crate::{bot::Bot, env::EnvConfig, gitea::WebhookType};
#[derive(Clone)]
pub struct AppState {
pub bot_tx: tokio::sync::mpsc::Sender<WebhookType>,
pub bot: Bot,
pub config: EnvConfig,
}