Monitor API reference

HTTP routes the CosmicAC monitor serves for job metrics and job logs, with parameters, responses, and status codes.

These routes return the job metrics and job logs that cosmicac-wrk-monitor collects. CosmicAC services send that telemetry over a shared topic. Each route lists its request, response, and errors.

In every path, <monitor-url> is the base URL of the monitor, which listens on port 9110 by default.

The monitor requires no authentication

By default, every route accepts any caller that can reach the port. Job logs and job metrics hold operational detail for every tenant of the deployment. See Restrict access to cosmicac-wrk-monitor.

Endpoints

Method	Path	Route
`GET`	`/metrics`	Scrape metrics
`GET`	`/metrics/<endpoint-name>`	Scrape metrics for one endpoint
`GET`	`/job-metrics/<job-id>`	Get current job metrics
`GET`	`/endpoint-metrics/<endpoint-name>`	Get current endpoint metrics
`GET`	`/job-metrics/<job-id>/history`	Get job metrics history
`GET`	`/endpoint-metrics/<endpoint-name>/history`	Get endpoint metrics history
`GET`	`/job-metrics/<job-id>/stream`	Stream job metrics
`GET`	`/endpoint-metrics/<endpoint-name>/stream`	Stream endpoint metrics
`GET`	`/logs`	Stream logs
`GET`	`/logs/history`	Get log history

Only Scrape metrics accepts a credential, and only when your deployment sets metricsScrapeToken in the cosmicac-wrk-monitor configuration. That value is empty by default, which leaves /metrics open to any caller.

Authorization: Bearer <scrape-token>

Scrape metrics

Returns the metrics registry in Prometheus text exposition format.

HTTP request

GET <monitor-url>/metrics

Query parameters

Parameter	Type	Required	Description
`endpoint`	string	No	Returns only the series for this inference endpoint.
`job_id`	string	No	Returns only the series for this job.
`component`	string	No	Returns only the series for this CosmicAC component.

Request headers

Header	Type	Required	Description
`Authorization`	string	No	Scrape token as a Bearer token, `Bearer <scrape-token>`. Required only when the deployment sets `metricsScrapeToken`.

Response

The response body is Prometheus exposition text.

Content-Type: text/plain; version=0.0.4; charset=utf-8

A filter that matches no series returns 200 with an empty body.

Errors

Failed requests return an error identifier.

{
  "error": "ERR_UNAUTHORIZED"
}

Field	Type	Description
`error`	string	Identifies the error.

The route returns these status codes.

Status	Meaning
`401`	The deployment sets `metricsScrapeToken`, and the request carries no matching Bearer token. The response carries a `WWW-Authenticate: Bearer` header.

Scrape metrics for one endpoint

Returns the registry filtered to one inference endpoint, in Prometheus text exposition format.

HTTP request

GET <monitor-url>/metrics/<endpoint-name>

Path parameters

Parameter	Type	Required	Description
`endpoint-name`	string	Yes	Inference endpoint to return series for.

Query parameters

Parameter	Type	Required	Description
`job_id`	string	No	Returns only the series for this job.
`component`	string	No	Returns only the series for this CosmicAC component.

An endpoint query parameter has no effect on this route, because the path value takes precedence.

Request headers

Header	Type	Required	Description
`Authorization`	string	No	Scrape token as a Bearer token, `Bearer <scrape-token>`. Required only when the deployment sets `metricsScrapeToken`.

Response

The response body is Prometheus exposition text, carrying only the series labelled with the named endpoint.

Content-Type: text/plain; version=0.0.4; charset=utf-8

An endpoint with no series returns 200 with an empty body.

Errors

Failed requests return an error identifier.

{
  "error": "ERR_UNAUTHORIZED"
}

Field	Type	Description
`error`	string	Identifies the error.

The route returns these status codes.

Status	Meaning
`401`	The deployment sets `metricsScrapeToken`, and the request carries no matching Bearer token. The response carries a `WWW-Authenticate: Bearer` header.

Get current job metrics

Returns one sample of the CPU and GPU values for a job. CosmicAC reads the sample from your Prometheus.

HTTP request

GET <monitor-url>/job-metrics/<job-id>

Path parameters

Parameter	Type	Required	Description
`job-id`	string	Yes	Job to sample.

Response

{
  "job_id": "job-4f2a",
  "timestamp": 1783079106559,
  "cpu": {
    "cpu_percent": 42.5,
    "memory_bytes": 8589934592,
    "memory_limit_bytes": 34359738368,
    "source": "container"
  },
  "gpu": [
    {
      "gpu": "0",
      "uuid": "GPU-1a2b3c",
      "utilization_pct": 96,
      "memory_used_bytes": 74490249216,
      "memory_total_bytes": 85520809984,
      "temp_c": 61,
      "power_w": 412
    }
  ]
}

Field	Type	Description
`job_id`	string	Job the sample covers.
`timestamp`	integer	Unix timestamp in milliseconds, taken when the monitor built the sample.
`cpu.cpu_percent`	number	Processor use as a percentage, or `null` when no series matched.
`cpu.memory_bytes`	number	Resident memory in bytes, or `null` when no series matched.
`cpu.memory_limit_bytes`	number	Memory limit in bytes, or `null` when no series matched.
`cpu.source`	string	Where the processor figure came from, either `container` or `agent`.
`gpu`	array	One entry per device, sorted by `gpu`.
`gpu[].gpu`	string	Device index.
`gpu[].uuid`	string	Device identifier reported by the driver.
`gpu[].utilization_pct`	number	Device use as a percentage.
`gpu[].memory_used_bytes`	number	Device memory in use, in bytes.
`gpu[].memory_total_bytes`	number	Total device memory, in bytes.
`gpu[].temp_c`	number	Device temperature in degrees Celsius.
`gpu[].power_w`	number	Device power draw, in watts.

Errors

Failed requests return an error identifier.

{
  "error": "ERR_JOB_ID_INVALID"
}

Field	Type	Description
`error`	string	Identifies the error.

The route returns these status codes.

Status	Meaning
`400`	The job ID isn't a valid label value. The `error` is `ERR_JOB_ID_INVALID`.
`503`	The deployment stores no Prometheus URL. The `error` is `ERR_PROMETHEUS_NOT_CONFIGURED`.

Get current endpoint metrics

Returns one sample of the CPU and GPU values for an inference endpoint. CosmicAC resolves the endpoint to its jobs, then reads the sample from your Prometheus.

HTTP request

GET <monitor-url>/endpoint-metrics/<endpoint-name>

Path parameters

Parameter	Type	Required	Description
`endpoint-name`	string	Yes	Inference endpoint to sample.

Response

{
  "endpoint": "qwen-2-prod",
  "timestamp": 1783079106559,
  "cpu": {
    "cpu_percent": 42.5,
    "memory_bytes": 8589934592,
    "memory_limit_bytes": 34359738368,
    "source": "container"
  },
  "gpu": [
    {
      "gpu": "0",
      "uuid": "GPU-1a2b3c",
      "utilization_pct": 96,
      "memory_used_bytes": 74490249216,
      "memory_total_bytes": 85520809984,
      "temp_c": 61,
      "power_w": 412
    }
  ]
}

Field	Type	Description
`endpoint`	string	Inference endpoint the sample covers.
`timestamp`	integer	Unix timestamp in milliseconds, taken when the monitor built the sample.
`cpu.cpu_percent`	number	Processor use as a percentage, or `null` when no series matched.
`cpu.memory_bytes`	number	Resident memory in bytes, or `null` when no series matched.
`cpu.memory_limit_bytes`	number	Memory limit in bytes, or `null` when no series matched.
`cpu.source`	string	Where the processor figure came from, either `container` or `agent`.
`gpu`	array	One entry per device, sorted by `gpu`.
`gpu[].gpu`	string	Device index.
`gpu[].uuid`	string	Device identifier reported by the driver.
`gpu[].utilization_pct`	number	Device use as a percentage.
`gpu[].memory_used_bytes`	number	Device memory in use, in bytes.
`gpu[].memory_total_bytes`	number	Total device memory, in bytes.
`gpu[].temp_c`	number	Device temperature in degrees Celsius.
`gpu[].power_w`	number	Device power draw, in watts.

Errors

Failed requests return an error identifier.

{
  "error": "ERR_ENDPOINT_INVALID"
}

Field	Type	Description
`error`	string	Identifies the error.

The route returns these status codes.

Status	Meaning
`400`	The endpoint name isn't a valid label value. The `error` is `ERR_ENDPOINT_INVALID`.
`503`	The deployment stores no Prometheus URL. The `error` is `ERR_PROMETHEUS_NOT_CONFIGURED`.

Get job metrics history

Returns metric series for a job over a time range. CosmicAC reads the series from your Prometheus.

HTTP request

GET <monitor-url>/job-metrics/<job-id>/history

Path parameters

Parameter	Type	Required	Description
`job-id`	string	Yes	Job to read.

Query parameters

Parameter	Type	Required	Description
`start`	string	No	Start of the range, as an RFC 3339 timestamp or as a Unix timestamp in nanoseconds. Defaults to one hour before `end`.
`end`	string	No	End of the range, in the same forms. Defaults to now.
`step`	string	No	Resolution, as a duration such as `15s` or `1m`, or as seconds. Defaults to a step the monitor derives from the range.

Response

{
  "job_id": "job-4f2a",
  "start": 1783075506559,
  "end": 1783079106559,
  "step_ms": 15000,
  "cpu": {
    "source": "container",
    "cpu_percent": [{ "timestamp": 1783075506559, "value": 42.5 }],
    "memory_bytes": [{ "timestamp": 1783075506559, "value": 8589934592 }],
    "memory_limit_bytes": [{ "timestamp": 1783075506559, "value": 34359738368 }]
  },
  "gpu": [
    {
      "gpu": "0",
      "uuid": "GPU-1a2b3c",
      "utilization_pct": [{ "timestamp": 1783075506559, "value": 96 }],
      "memory_used_bytes": [{ "timestamp": 1783075506559, "value": 74490249216 }],
      "memory_total_bytes": [{ "timestamp": 1783075506559, "value": 85520809984 }],
      "temp_c": [{ "timestamp": 1783075506559, "value": 61 }],
      "power_w": [{ "timestamp": 1783075506559, "value": 412 }]
    }
  ],
  "inference": {
    "traffic": [{ "timestamp": 1783075506559, "value": 120 }],
    "failures": [{ "timestamp": 1783075506559, "value": 2 }],
    "rps": [{ "timestamp": 1783075506559, "value": 8 }],
    "latency_p50_ms": [{ "timestamp": 1783075506559, "value": 240 }],
    "latency_p95_ms": [{ "timestamp": 1783075506559, "value": 910 }]
  }
}

Field	Type	Description
`job_id`	string	Job the series cover.
`start`	integer	Start of the range, as a Unix timestamp in milliseconds.
`end`	integer	End of the range, as a Unix timestamp in milliseconds.
`step_ms`	integer	Resolution the monitor used, in milliseconds.
`cpu.source`	string	Origin of the processor figures, either `container` or `agent`. `null` when no series matched.
`cpu.cpu_percent`	array	Processor use as a percentage.
`cpu.cpu_percent[].timestamp`	integer	Unix timestamp in milliseconds.
`cpu.cpu_percent[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`cpu.memory_bytes`	array	Resident memory in bytes.
`cpu.memory_bytes[].timestamp`	integer	Unix timestamp in milliseconds.
`cpu.memory_bytes[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`cpu.memory_limit_bytes`	array	Memory limit in bytes.
`cpu.memory_limit_bytes[].timestamp`	integer	Unix timestamp in milliseconds.
`cpu.memory_limit_bytes[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`gpu`	array	One entry per device, sorted by `gpu`.
`gpu[].gpu`	string	Device index.
`gpu[].uuid`	string	Device identifier reported by the driver.
`gpu[].utilization_pct`	array	Device use as a percentage.
`gpu[].utilization_pct[].timestamp`	integer	Unix timestamp in milliseconds.
`gpu[].utilization_pct[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`gpu[].memory_used_bytes`	array	Device memory in use, in bytes.
`gpu[].memory_used_bytes[].timestamp`	integer	Unix timestamp in milliseconds.
`gpu[].memory_used_bytes[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`gpu[].memory_total_bytes`	array	Total device memory, in bytes.
`gpu[].memory_total_bytes[].timestamp`	integer	Unix timestamp in milliseconds.
`gpu[].memory_total_bytes[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`gpu[].temp_c`	array	Device temperature in degrees Celsius.
`gpu[].temp_c[].timestamp`	integer	Unix timestamp in milliseconds.
`gpu[].temp_c[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`gpu[].power_w`	array	Device power draw, in watts.
`gpu[].power_w[].timestamp`	integer	Unix timestamp in milliseconds.
`gpu[].power_w[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`inference`	object	Request and latency series, or `null` when the job served no inference traffic.
`inference.traffic`	array	Requests counted at each point.
`inference.traffic[].timestamp`	integer	Unix timestamp in milliseconds.
`inference.traffic[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`inference.failures`	array	Failed requests counted at each point.
`inference.failures[].timestamp`	integer	Unix timestamp in milliseconds.
`inference.failures[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`inference.rps`	array	Requests per second.
`inference.rps[].timestamp`	integer	Unix timestamp in milliseconds.
`inference.rps[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`inference.latency_p50_ms`	array	Request latency at the median, in milliseconds.
`inference.latency_p50_ms[].timestamp`	integer	Unix timestamp in milliseconds.
`inference.latency_p50_ms[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`inference.latency_p95_ms`	array	Request latency at the ninety-fifth percentile, in milliseconds.
`inference.latency_p95_ms[].timestamp`	integer	Unix timestamp in milliseconds.
`inference.latency_p95_ms[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.

Errors

Failed requests return an error identifier and a message.

{
  "error": "ERR_PROMETHEUS_NOT_CONFIGURED",
  "message": "No Prometheus URL configured; history unavailable"
}

Field	Type	Description
`error`	string	Identifies the error.
`message`	string	Explains what went wrong.

The route returns these status codes.

Status	Meaning
`400`	The `error` is `ERR_JOB_ID_INVALID` for a job ID that isn't a valid label value, `ERR_TIME_RANGE_INVALID` for a range that doesn't parse or runs backward, or `ERR_STEP_INVALID` for an unsupported step.
`503`	The deployment stores no Prometheus URL. The `error` is `ERR_PROMETHEUS_NOT_CONFIGURED`.

Get endpoint metrics history

Returns metric series for an inference endpoint over a time range. CosmicAC reads the series from your Prometheus.

HTTP request

GET <monitor-url>/endpoint-metrics/<endpoint-name>/history

Path parameters

Parameter	Type	Required	Description
`endpoint-name`	string	Yes	Inference endpoint to read.

Query parameters

Parameter	Type	Required	Description
`start`	string	No	Start of the range, as an RFC 3339 timestamp or as a Unix timestamp in nanoseconds. Defaults to one hour before `end`.
`end`	string	No	End of the range, in the same forms. Defaults to now.
`step`	string	No	Resolution, as a duration such as `15s` or `1m`, or as seconds. Defaults to a step the monitor derives from the range.

Response

{
  "endpoint": "qwen-2-prod",
  "start": 1783075506559,
  "end": 1783079106559,
  "step_ms": 15000,
  "cpu": {
    "source": "container",
    "cpu_percent": [{ "timestamp": 1783075506559, "value": 42.5 }],
    "memory_bytes": [{ "timestamp": 1783075506559, "value": 8589934592 }],
    "memory_limit_bytes": [{ "timestamp": 1783075506559, "value": 34359738368 }]
  },
  "gpu": [
    {
      "gpu": "0",
      "uuid": "GPU-1a2b3c",
      "utilization_pct": [{ "timestamp": 1783075506559, "value": 96 }],
      "memory_used_bytes": [{ "timestamp": 1783075506559, "value": 74490249216 }],
      "memory_total_bytes": [{ "timestamp": 1783075506559, "value": 85520809984 }],
      "temp_c": [{ "timestamp": 1783075506559, "value": 61 }],
      "power_w": [{ "timestamp": 1783075506559, "value": 412 }]
    }
  ],
  "inference": {
    "traffic": [{ "timestamp": 1783075506559, "value": 120 }],
    "failures": [{ "timestamp": 1783075506559, "value": 2 }],
    "rps": [{ "timestamp": 1783075506559, "value": 8 }],
    "latency_p50_ms": [{ "timestamp": 1783075506559, "value": 240 }],
    "latency_p95_ms": [{ "timestamp": 1783075506559, "value": 910 }]
  }
}

Field	Type	Description
`endpoint`	string	Inference endpoint the series cover.
`start`	integer	Start of the range, as a Unix timestamp in milliseconds.
`end`	integer	End of the range, as a Unix timestamp in milliseconds.
`step_ms`	integer	Resolution the monitor used, in milliseconds.
`cpu.source`	string	Origin of the processor figures, either `container` or `agent`. `null` when no series matched.
`cpu.cpu_percent`	array	Processor use as a percentage.
`cpu.cpu_percent[].timestamp`	integer	Unix timestamp in milliseconds.
`cpu.cpu_percent[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`cpu.memory_bytes`	array	Resident memory in bytes.
`cpu.memory_bytes[].timestamp`	integer	Unix timestamp in milliseconds.
`cpu.memory_bytes[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`cpu.memory_limit_bytes`	array	Memory limit in bytes.
`cpu.memory_limit_bytes[].timestamp`	integer	Unix timestamp in milliseconds.
`cpu.memory_limit_bytes[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`gpu`	array	One entry per device, sorted by `gpu`.
`gpu[].gpu`	string	Device index.
`gpu[].uuid`	string	Device identifier reported by the driver.
`gpu[].utilization_pct`	array	Device use as a percentage.
`gpu[].utilization_pct[].timestamp`	integer	Unix timestamp in milliseconds.
`gpu[].utilization_pct[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`gpu[].memory_used_bytes`	array	Device memory in use, in bytes.
`gpu[].memory_used_bytes[].timestamp`	integer	Unix timestamp in milliseconds.
`gpu[].memory_used_bytes[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`gpu[].memory_total_bytes`	array	Total device memory, in bytes.
`gpu[].memory_total_bytes[].timestamp`	integer	Unix timestamp in milliseconds.
`gpu[].memory_total_bytes[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`gpu[].temp_c`	array	Device temperature in degrees Celsius.
`gpu[].temp_c[].timestamp`	integer	Unix timestamp in milliseconds.
`gpu[].temp_c[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`gpu[].power_w`	array	Device power draw, in watts.
`gpu[].power_w[].timestamp`	integer	Unix timestamp in milliseconds.
`gpu[].power_w[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`inference`	object	Request and latency series, or `null` when the endpoint served no inference traffic.
`inference.traffic`	array	Requests counted at each point.
`inference.traffic[].timestamp`	integer	Unix timestamp in milliseconds.
`inference.traffic[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`inference.failures`	array	Failed requests counted at each point.
`inference.failures[].timestamp`	integer	Unix timestamp in milliseconds.
`inference.failures[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`inference.rps`	array	Requests per second.
`inference.rps[].timestamp`	integer	Unix timestamp in milliseconds.
`inference.rps[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`inference.latency_p50_ms`	array	Request latency at the median, in milliseconds.
`inference.latency_p50_ms[].timestamp`	integer	Unix timestamp in milliseconds.
`inference.latency_p50_ms[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.
`inference.latency_p95_ms`	array	Request latency at the ninety-fifth percentile, in milliseconds.
`inference.latency_p95_ms[].timestamp`	integer	Unix timestamp in milliseconds.
`inference.latency_p95_ms[].value`	number	Measurement at that timestamp, or `null` when the step held no sample.

Errors

Failed requests return an error identifier and a message.

{
  "error": "ERR_PROMETHEUS_NOT_CONFIGURED",
  "message": "No Prometheus URL configured; history unavailable"
}

Field	Type	Description
`error`	string	Identifies the error.
`message`	string	Explains what went wrong.

The route returns these status codes.

Status	Meaning
`400`	The `error` is `ERR_ENDPOINT_INVALID` for an endpoint name that isn't a valid label value, `ERR_TIME_RANGE_INVALID` for a range that doesn't parse or runs backward, or `ERR_STEP_INVALID` for an unsupported step.
`503`	The deployment stores no Prometheus URL. The `error` is `ERR_PROMETHEUS_NOT_CONFIGURED`.

Stream job metrics

Pushes a sample of a job's CPU and GPU values over a WebSocket connection at a fixed interval.

HTTP request

GET <monitor-url>/job-metrics/<job-id>/stream

The route serves WebSocket connections only. A request that isn't a WebSocket handshake returns 426.

Path parameters

Parameter	Type	Required	Description
`job-id`	string	Yes	Job to sample.

Query parameters

Parameter	Type	Required	Description
`interval_ms`	integer	No	Milliseconds between samples.

WebSocket frames

{
  "type": "metrics",
  "job_id": "job-4f2a",
  "timestamp": 1783079106559,
  "cpu": {
    "cpu_percent": 42.5,
    "memory_bytes": 8589934592,
    "memory_limit_bytes": 34359738368,
    "source": "container"
  },
  "gpu": [
    {
      "gpu": "0",
      "uuid": "GPU-1a2b3c",
      "utilization_pct": 96,
      "memory_used_bytes": 74490249216,
      "memory_total_bytes": 85520809984,
      "temp_c": 61,
      "power_w": 412
    }
  ]
}

Field	Type	Description
`type`	string	Either `metrics` for a sample, or `error` for a failure.
`job_id`	string	Job the sample covers.
`timestamp`	integer	Unix timestamp in milliseconds, taken when the monitor built the sample.
`cpu.cpu_percent`	number	Processor use as a percentage, or `null` when no series matched.
`cpu.memory_bytes`	number	Resident memory in bytes, or `null` when no series matched.
`cpu.memory_limit_bytes`	number	Memory limit in bytes, or `null` when no series matched.
`cpu.source`	string	Where the processor figure came from, either `container` or `agent`.
`gpu`	array	One entry per device, sorted by `gpu`.
`gpu[].gpu`	string	Device index.
`gpu[].uuid`	string	Device identifier reported by the driver.
`gpu[].utilization_pct`	number	Device use as a percentage.
`gpu[].memory_used_bytes`	number	Device memory in use, in bytes.
`gpu[].memory_total_bytes`	number	Total device memory, in bytes.
`gpu[].temp_c`	number	Device temperature in degrees Celsius.
`gpu[].power_w`	number	Device power draw, in watts.

The monitor sends the first sample when the connection opens, then one every interval.

Errors

Failed requests return an error identifier and a message.

{
  "error": "ERR_UPGRADE_REQUIRED",
  "message": "Connect to this route over WebSocket"
}

Field	Type	Description
`error`	string	Identifies the error.
`message`	string	Explains what went wrong.

The route returns these status codes.

Status	Meaning
`426`	The request reached the route without a WebSocket upgrade. The `error` is `ERR_UPGRADE_REQUIRED`.

When a job ID isn't a valid label value, the monitor sends an ERR_JOB_ID_INVALID frame and closes the connection.

Stream endpoint metrics

Pushes a sample of an inference endpoint's CPU and GPU values over a WebSocket connection at a fixed interval.

HTTP request

GET <monitor-url>/endpoint-metrics/<endpoint-name>/stream

The route serves WebSocket connections only. A request that isn't a WebSocket handshake returns 426.

Path parameters

Parameter	Type	Required	Description
`endpoint-name`	string	Yes	Inference endpoint to sample.

Query parameters

Parameter	Type	Required	Description
`interval_ms`	integer	No	Milliseconds between samples.

WebSocket frames

{
  "type": "metrics",
  "endpoint": "qwen-2-prod",
  "timestamp": 1783079106559,
  "cpu": {
    "cpu_percent": 42.5,
    "memory_bytes": 8589934592,
    "memory_limit_bytes": 34359738368,
    "source": "container"
  },
  "gpu": [
    {
      "gpu": "0",
      "uuid": "GPU-1a2b3c",
      "utilization_pct": 96,
      "memory_used_bytes": 74490249216,
      "memory_total_bytes": 85520809984,
      "temp_c": 61,
      "power_w": 412
    }
  ]
}

Field	Type	Description
`type`	string	Either `metrics` for a sample, or `error` for a failure.
`endpoint`	string	Inference endpoint the sample covers.
`timestamp`	integer	Unix timestamp in milliseconds, taken when the monitor built the sample.
`cpu.cpu_percent`	number	Processor use as a percentage, or `null` when no series matched.
`cpu.memory_bytes`	number	Resident memory in bytes, or `null` when no series matched.
`cpu.memory_limit_bytes`	number	Memory limit in bytes, or `null` when no series matched.
`cpu.source`	string	Where the processor figure came from, either `container` or `agent`.
`gpu`	array	One entry per device, sorted by `gpu`.
`gpu[].gpu`	string	Device index.
`gpu[].uuid`	string	Device identifier reported by the driver.
`gpu[].utilization_pct`	number	Device use as a percentage.
`gpu[].memory_used_bytes`	number	Device memory in use, in bytes.
`gpu[].memory_total_bytes`	number	Total device memory, in bytes.
`gpu[].temp_c`	number	Device temperature in degrees Celsius.
`gpu[].power_w`	number	Device power draw, in watts.

The monitor sends the first sample when the connection opens, then one every interval.

Errors

Failed requests return an error identifier and a message.

{
  "error": "ERR_UPGRADE_REQUIRED",
  "message": "Connect to this route over WebSocket"
}

Field	Type	Description
`error`	string	Identifies the error.
`message`	string	Explains what went wrong.

The route returns these status codes.

Status	Meaning
`426`	The request reached the route without a WebSocket upgrade. The `error` is `ERR_UPGRADE_REQUIRED`.

When an endpoint name isn't a valid label value, the monitor sends an ERR_ENDPOINT_INVALID frame and closes the connection.

Stream logs

Pushes job log lines over a WebSocket connection as they arrive.

HTTP request

GET <monitor-url>/logs

The route serves WebSocket connections only. A request that isn't a WebSocket handshake returns 426.

Query parameters

Parameter	Type	Required	Description
`job_id`	string	Yes	Job to follow. Required unless the request carries `all`.
`replica_id`	string	No	Sends only lines from this replica.
`type`	string	No	Sends only lines of this type, either `application` or `system`.
`all`	boolean	No	A value of `true` follows every component instead of one job.

WebSocket frames

{
  "type": "log",
  "labels": {
    "job_id": "job-4f2a",
    "replica_id": "0",
    "type": "application"
  },
  "message": "Model loaded",
  "level": "info",
  "timestamp": 1783079106559,
  "data": {}
}

Field	Type	Description
`type`	string	Either `log` for a line, or `error` for a failure.
`labels`	object	Labels the producer attached to the line.
`message`	string	The log line.
`level`	string	Severity the producer recorded.
`timestamp`	integer	Unix timestamp in milliseconds.
`data`	object	Structured fields the producer attached, when it attached any.

Errors

Failed requests return an error identifier and a message.

{
  "error": "ERR_UPGRADE_REQUIRED",
  "message": "Connect to /logs over WebSocket"
}

Field	Type	Description
`error`	string	Identifies the error.
`message`	string	Explains what went wrong.

The route returns these status codes.

Status	Meaning
`426`	The request reached the route without a WebSocket upgrade. The `error` is `ERR_UPGRADE_REQUIRED`.

When a request names no job and omits all, the monitor sends an ERR_JOB_ID_REQUIRED frame and closes the connection.

Get log history

Returns stored log lines for a job. CosmicAC reads them from your Loki.

HTTP request

GET <monitor-url>/logs/history

Query parameters

Parameter	Type	Required	Description
`job_id`	string	Yes	Job to read. Required unless the request carries `all`.
`all`	boolean	No	A value of `true` reads across every component instead of one job.
`replica_id`	string	No	Returns only lines from this replica.
`type`	string	No	Returns only lines of this type, either `application` or `system`.
`component`	string	No	Returns only lines from this CosmicAC component.
`level`	string	No	Returns only lines at this severity.
`search`	string	No	Returns only lines containing this text. Up to 512 characters.
`start`	string	No	Start of the range, as an RFC 3339 timestamp or as a Unix timestamp in nanoseconds.
`end`	string	No	End of the range, in the same forms.
`limit`	integer	No	Lines per page. Defaults to `200`, up to `1000`.
`direction`	string	No	Either `backward` for newest first, or `forward`. Defaults to `backward`.
`cursor`	string	No	Continues a previous page. The value comes from `next_cursor` or `newer_cursor` in that page.

Response

{
  "total": 200,
  "entries": [
    {
      "labels": { "job_id": "job-4f2a" },
      "timestamp": 1783079106559,
      "timestamp_ns": "1783079106559000000",
      "message": "Model loaded",
      "level": "info"
    }
  ],
  "limit": 200,
  "direction": "backward",
  "has_more": true,
  "next_cursor": "eyJ0cyI6...",
  "newer_cursor": null
}

Field	Type	Description
`total`	integer	Lines in this page.
`entries`	array	The lines, ordered by `direction`.
`entries[].labels`	object	Labels the producer attached to the line.
`entries[].timestamp`	integer	Unix timestamp in milliseconds.
`entries[].timestamp_ns`	string	Unix timestamp in nanoseconds. Cursors build on this value rather than on `timestamp`.
`entries[].message`	string	The log line.
`entries[].level`	string	Severity the producer recorded.
`limit`	integer	Page size the monitor applied.
`direction`	string	Order the monitor applied.
`has_more`	boolean	Whether more lines follow in the same direction.
`next_cursor`	string	Cursor for the next page, or `null`.
`newer_cursor`	string	Cursor for the page in the opposite direction, or `null`.

Errors

Failed requests return an error identifier and a message.

{
  "error": "ERR_LOKI_NOT_CONFIGURED",
  "message": "No Loki URL configured; history unavailable"
}

Field	Type	Description
`error`	string	Identifies the error.
`message`	string	Explains what went wrong.

The route returns these status codes.

Status	Meaning
`400`	The request names no job and omits `all`, the search text is too long, a bound doesn't parse, the range runs backward, or the cursor doesn't decode. The `error` is `ERR_JOB_ID_REQUIRED`, `ERR_SEARCH_TOO_LONG`, `ERR_START_INVALID`, `ERR_END_INVALID`, `ERR_TIME_RANGE_INVALID`, or `ERR_CURSOR_INVALID`.
`400`	Loki rejected the query, often because the range exceeded its `max_query_length`. The `error` is `ERR_LOKI_QUERY_REJECTED`.
`502`	Loki didn't answer. The `error` is `ERR_LOKI_UNAVAILABLE`.
`503`	The deployment stores no Loki URL. The `error` is `ERR_LOKI_NOT_CONFIGURED`.

Monitor API reference

On this page