Intro

Flymodel is a model version management system. It centers around the following premise:

• Tests should be recorded, reproducible, and deterministic
• Models must be immutable
• Artifacts must be validated
• All server & client side effects must be controlled and safe

Installation

Source

cargo install --git https://github.com/flymodel/flymodel flymodel-cli

Concepts

• Namespaces
• Models
• Model Versions
• Experiments
• Artifacts

Namespaces

Namespaces are subgroupings in which each delegated item is distinct from its counterparty namespaces. For example, models may be replicated between namespaces, but may not be replicated within a unique namespace.

Models

Models are units within a namespace which are the highest level of order for versioned content. This is the 'repo' for a model & each of its versions.

Model Versions

Models are segregated by each version tag. Standard syntax is encouraged, however will not be enforced as there are many semver standards in effect:

Experiments

Experiments are performed against a specific model version. An experiment may not be repeated multiple times across a model version. As such, experiments are deterministic sources of truth for the state of the monitored parameters in a model.

Artifacts

Artifacts may be stored in two methods:

1. Against a specific model version
2. Against a specific experiment run (of a model version)

It is advised to utilize the file encoding & compression parameters when available, as this will allow for analysis of experiment artifacts between runs.

Common Metadata Stored

• sha256 sum
• Content-type
• Compression parameters
• File name
• Artifact name

Model Version Artifacts

Model version artifacts provide an artifact name, and additional optional JSON encoded metadata.

Experiment Artifacts

Experiment artifacts provide an artifact name.

Cli

Usage: flymodel [OPTIONS] <COMMAND>

Commands:
  serve
  migrate
  setup-storage
  upsert
  help           Print this message or the help of the given subcommand(s)

Options:
  -c, --config <CONFIG>  [default: ./flymodel.toml]
      --dry
  -h, --help             Print help
  -V, --version          Print version

Serve

Usage: flymodel serve [OPTIONS] --database-url <DATABASE_URL>

Options:
  -p, --port <PORT>                  [default: 9009]
  -b, --bind <BIND>                  [default: localhost]
  -d, --database-url <DATABASE_URL>  [env: DB_URL=postgresql://postgres:postgres@localhost:5432/mlops]
  -c, --config <CONFIG>              [default: ./flymodel.toml]
  -h, --help                         Print help

Migrate

Usage: flymodel migrate [OPTIONS] <COMMAND>

Commands:
  up
  down
  help  Print this message or the help of the given subcommand(s)

Options:
  -c, --config <CONFIG>  [default: ./flymodel.toml]
  -h, --help             Print help

Up

Usage: flymodel migrate up [OPTIONS] --database-url <DATABASE_URL>

Options:
  -d, --database-url <DATABASE_URL>  [env: DB_URL=postgresql://postgres:postgres@localhost:5432/mlops]
      --test-data <TEST_DATA>        [possible values: basic, multi_region]
      --steps <STEPS>
  -c, --config <CONFIG>              [default: ./flymodel.toml]
  -h, --help                         Print help

Down

Usage: flymodel migrate down [OPTIONS] --database-url <DATABASE_URL>

Options:
  -d, --database-url <DATABASE_URL>  [env: DB_URL=postgresql://postgres:postgres@localhost:5432/mlops]
      --test-data <TEST_DATA>        [possible values: basic, multi_region]
      --steps <STEPS>
  -c, --config <CONFIG>              [default: ./flymodel.toml]
  -h, --help                         Print help

Setup-Storage

Usage: flymodel setup-storage [OPTIONS]

Options:
  -c, --config <CONFIG>  [default: ./flymodel.toml]
  -h, --help             Print help

Upsert

Usage: flymodel upsert [OPTIONS]

Options:
  -c, --config <CONFIG>  [default: ./flymodel.toml]
  -h, --help             Print help

Configuration

Formats

Static

Configuration files are searched in config/flymodel (with extension added). The following file formats are supported:

• toml
• yaml
• json
• json5

Environment

Environment variables are searched by the following methodology:

1. Replace . in a key with _.

   storage.s3 -> storage_s3

2. Convert to screaming snake case.

   storage_s3 -> STORAGE_S3

3. Prefix with FLYMODEL_.

   FLYMODEL_STORAGE_S3

Options

• Logs
• Retention
• Storage
• Server
• Tracing

Logs

Level

Values

• error
• info
• debug
• trace

Sample

log.level = "info"

Retention

Configuration keys follow per-lifecycle groupings. E.g. retention.prod.models.

retention.<lifecycle>.models

Days to retain models in the specified lifecycle. None is forever.

retention.<lifecycle>.runs

Days to retain run details in the specified lifecycle. None is forever.

Sample

[[retention.prod]]
models = 365
runs = 28

Storage

S3

S3-compatible object storage.

Setup

The below command is required to pre-initialize the storage database with the known storage hosts:

flymodel setup-storage

Keys

s3.bucket

The bucket name.

s3.endpoint

The target endpoint to connect to.

s3.region

The region to connect to. Optional if s3.endpoint is provided and region is not required.

s3.role

The role (assigned lifecycle) of the bucket to store artifacts. Required.

s3.public

Whether assets should be public. Defaults to false.

s3.path_style

Force use path style object access in the bucket. Default true.

Sample

[[storage.s3]]
bucket = "ml-prod"
endpoint = "https://my-minio-endpoint:9000"
region = "ca"
role = "prod"

Server

Keys

server.temp_dir

The temporary directory used for out of memory file staging. Required.

server.tls.ca_source

The ca source used to validate mTLS requests. (future-state).

server.tls.certs

If a string, it is assumed that the certificate and key are clubbed together in the respective order. * Else provide the below.

server.tls.certs.cert_file

The cert file used for https. Optional if tls not specified. Required if specified.

server.tls.certs.key_file

The pkcs8 key used for https. Optional if tls not specified. Required if specified.

Example

[server.tls.certs]
cert_file = "./certs/my-domain.pem"
key_file = "./certs/my-domain.key"

Notes

• mTLS is not implemented
• Client Certificate validation is not implemented
• Certificate clubbing is not implemented.

Tracing

Keys

tracing.otlp.target

The target URL to send spans to.

tracing.otlp.tls.ca_source

The ca chain used to validate requests to a TLS bound exporter. Optional.

tracing.otlp.tls.certs.cert_file

The certificate to send (client) to the otlp exporter. Optional.

tracing.otlp.tls.certs.key_file

The certificate key to send (client) to the otlp exporter. Optional.

Example

[tracing]
otlp = { target = "localhost:4317" }