Skip to content
TAIP

Products / For AI builders

TrainX

Available

A template is a shape. Your project fills it. Kubernetes runs the job.

TrainX is the training engine of TAIP. It turns ad-hoc kubectl job submission into a curated product, and it now meets projects two ways. An admin can build a TrainXJobTemplate with typed parameters and an opinionated script, and a user fills in a form rendered straight from it — the appliance path. Or a project carries a trainx.yaml beside its code that declares its own entrypoints, parameters, outputs, metrics file and checkpoint directory, and runs on a platform shape: a plain image family with no program baked in. Either way TrainX produces the underlying Kubernetes Job — or a JobSet with Kueue gang admission for multi-node runs — checks the user's live ResourceQuota before submission, streams logs over SSE, parses progress into a real progress bar, records parameters and metrics in MLflow before the script even starts, and launches TensorBoard on demand.

All products

Specification

Version
v1.7.0 — generally available
Backed by
TrainXJob · TrainXJobTemplate CRDs → Kubernetes Job, or JobSet + Kueue gang admission
Two paths
Appliances (a shape with a program) · shapes + your project's trainx.yaml
Catalog
One PyTorch image family, build-time manifests · pytorch-cpu/gpu · transformers-gpu · unsloth-gpu
Appliances
download-model · download-dataset · publish-model · sft-lora · unsloth-lora · merge-adapter · eval-benchmark · image-preflight
Observability
Live logs · parsed progress · K8s events · TensorBoard · MLflow run record
Reliability
Declared checkpointing → automatic resume with a restart budget and attempt history
Access
Web UI · taip trainx CLI plugin · taip_ tokens with trainx scopes
Languages
English · 简体中文 (UI and docs)

Proof, not promises

See it in one block.

No proprietary SDKs, no rewrites — TrainX meets your tools where they already are.

your project declares itself 
# trainx.yaml, beside your code — the platform reads it at submission
entrypoints:
  train:
    script: train.py
    outputs: [{ name: adapter, type: adapter, path: out/ }]
    checkpointing: { supported: true, dir: ckpt/ }

$ taip trainx run --entrypoint train    # submits the chain, needs first
sft-lora  ████████████░░░░░░░░  62/100  loss=0.8214   resumable▌

No SDK and no callback hooks: a print of TRAINX_PROGRESS: i/N still draws the bar, and a metrics file the job already writes is read at archive time. Logs, events, artifacts, and TensorBoard ride along.

Capabilities

What TrainX gives you

01

Two ways in: appliances and shapes

An appliance is a shape with a program — download a model, merge an adapter, run lm-eval — and its TrainXJobTemplate carries typed parameter metadata the web UI renders straight into a form. A shape is just a curated image: your project supplies the program through a trainx.yaml that declares its entrypoints, parameters, outputs, metrics and checkpoint contract. Tenants can author their own templates in their own namespace.

02

Run, watch, and keep the record

Live streaming logs with a viewer that handles 10k-line runs. Progress parsed from TRAINX_PROGRESS: i/N lines, or read from a metrics file the job already writes. An Events tab with the run's own lifecycle events. One-click TensorBoard, auto-reaped. Declared outputs — adapters, checkpoints, reports — are resolved to real paths and browsable long after the CRD is gone.

03

A run that survives its first OOM

A template that promises a checkpoint contract gets automatic resume: a restart budget, one pod per attempt so the failed attempt's logs survive, and an attempt history that makes 'it OOM'd twice and is now on attempt 3' legible without reading pod history. A run that cannot resume gets zero restarts, because spending the same GPU-hours to reach the same failure helps nobody.

04

Every run has a record

TrainX opens the MLflow run and logs the parameters itself, at job creation, before the script starts — so a run whose script never touches the SDK still carries a full record. Tracking is per-user opt-out, and it is tracking only: ModelSphere stays TAIP's model registry.

05

Multi-node without the YAML

Set nodes and processes per node and TrainX reconciles a JobSet with Kueue gang admission, so a distributed run is admitted whole or not at all instead of half-scheduling and deadlocking. Elastic runs, launcher selection, queue choice, and accelerator pick are all on the form — and Clone carries every one of them.

06

Tenant-aware by construction

Every run is a TrainXJob CRD in the user's namespace. ConsoleX provides the namespace, live quota, and the taip_ token a script authenticates with — the form rejects over-quota submissions before they ever hit the cluster. Platform-defined environment presets inject proxy, tracking, and Hub settings without a user pasting credentials into a form, and saved job secrets stay out of the job spec. Every job is labeled to its user for audit.

07

Air-gap friendly

No required outbound dependencies at runtime. Bundling scripts load every image into a cluster-local registry; the same chart deploys connected or disconnected — and runs in production on an internetless cluster today.

How it works

From a project to a tracked, resumable run.

  1. Step 01

    Pick an appliance, or a shape

    An appliance brings its own program and renders a form from typed parameter metadata. A shape brings only the image — your project's trainx.yaml declares the rest. Both are TrainXJobTemplate CRDs: auditable, reusable, tenant-authorable.

  2. Step 02

    Submit from the form or the CLI

    The web UI renders from the template; taip trainx run submits from your project directory and follows declared needs to submit a chain in order. Quota-checked before anything reaches the cluster.

  3. Step 03

    Watch, resume, iterate

    Streaming logs, parsed progress, lifecycle events, one-click TensorBoard, an MLflow run record. An interrupted run resumes from its checkpoint; Clone re-runs with different params in two clicks.

  4. Step 04

    Hand the output on

    A finished run declares what it produced — adapter, checkpoint, model, report. Merge the adapter, evaluate it, publish it to ModelSphere, or serve it on InferX, without hunting through a PVC for the path.

Who it's for

Built for these teams

  • Research teams running fine-tunes, RLHF, and evals
  • ML engineers tired of editing Job YAML by hand
  • Projects that already have their own training code and want a cluster, not a framework
  • Platform teams curating an opinionated training surface

Pairs well with

Other builder products

ConsoleX

 Available

Log in, get a governed Kubernetes workspace. No kubectl, no tickets.

On first SSO login every user gets an isolated namespace with quotas, default-deny networking, storage, and a web terminal — provisioned automatically, reconciled continuously. It is also the platform's front door: the token authority every TAIP app trusts, the SSH gateway into every workspace, and the taip CLI the cluster serves itself.

Learn more

DevSpace

 Available

Jupyter or VS Code on a GPU in seconds. Idle environments shut themselves down.

Single-click Jupyter, Marimo, Streamlit, Gradio, and VS Code environments — GPU-ready, isolated per user behind a per-pod auth proxy, shareable with a colleague, and idle-shutdown by default.

Learn more

ModelSphere

 Available

Your own Hugging Face Hub. Change one env var — every client just works.

A self-hosted, HF-compatible model and dataset registry: transformers, datasets, huggingface-cli, git-lfs, and the Xet protocol all work unchanged — plus a browsable Hub UI with model cards, a paging dataset viewer, file history, OIDC, audit, quotas, and pull-through caching of the public Hub.

Learn more