Euler View
No jobs

Run Dossier

Training run for SD 2.1. Mari

54 metrics · 61,071 train · 782 val

Chart Overlays

Compare this run against its peers

0/105
Add peer runs here to layer their train and val traces onto the current metric charts.
crashed Jul 24, 15:15 Duration: 33h 9m Python 3.11.7 train_joint.py --config /cluster/home/drothenpiele/models/stable_diffusion_mari/mari/.euler-launches/euler_launch_9048b465-63ee-43de-8d8f-eb9db29653d7/joint_vkitti.yaml
lr 4e-5batch 24epochs 200precision fp16model stable-diffusion-2-1 seed 42wd 0grad accum 1warmup 0.1000ema nograd clip 1
ID 8319862
mari/runs/2026-07-24_15-15-25_3900
Error
BrokenPipeError: [Errno 32] Broken pipe
Metrics 54/54
camera.train 1
camera_valid_fraction
depth.train 31
diag 25
depth_abs_rel
depth_delta1
depth_edge_smoothness
depth_head_abs_rel
depth_head_delta1
depth_head_mae
depth_hf_energy
depth_mae
depth_road_hsmooth
depth_tv
vae_decode_abs_rel
vae_decode_delta1
vae_decode_mae
valset_muses_absrel
valset_muses_delta1
valset_muses_delta2
valset_muses_delta3
valset_muses_mae
valset_muses_num_samples
valset_muses_projected_pixels
valset_muses_rmse
valset_muses_rmse_log
valset_muses_silog
valset_muses_sqrel
valset_muses_valid_pixels
loss 6
depth
depth_head
depth_tv
edge_smoothness
multiscale_smoothness
smoothness_aux
rgb.train 11
diag 2
image_psnr
image_ssim
loss 7
dehaze
preserve
total
visibility
visibility_aux
visibility_rank
visibility_tv
stat 2
visibility_mean
visibility_std
sys.train 6
gpu_mem_total_gb
gpu_mem_used_gb
gpu_mem_util_pct
gpu_util_pct
lr
visibility_aux_scale
Legacy 5
data/camera 3
enabled/any/source
fields/emitted
rays/emitted
train/camera/valid/fraction
val/source/rds/camera/valid/fraction
54 selected
X-Axis
Y-Scale
Series
Group
Namespace

Camera Valid Fraction

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Depth Abs Rel

↓ kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Depth Delta1

↑ kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Depth Edge Smoothness

↓ kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Depth Head Abs Rel

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Depth Head Delta1

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Depth Head Mae

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Depth Hf Energy

↓ kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Depth Mae

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Depth Road Hsmooth

↓ kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Depth Tv

↓ kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Vae Decode Abs Rel

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Vae Decode Delta1

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Vae Decode Mae

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Valset Muses Absrel

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Valset Muses Delta1

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Valset Muses Delta2

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Valset Muses Delta3

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Valset Muses Mae

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Valset Muses Num Samples

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Valset Muses Projected Pixels

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Valset Muses Rmse

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Valset Muses Rmse Log

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Valset Muses Silog

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Valset Muses Sqrel

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Valset Muses Valid Pixels

kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Depth

↓ kind=loss
T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Depth Head

kind=loss
T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Depth Tv

↓ kind=loss
T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Edge Smoothness

↓ kind=loss
T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Multiscale Smoothness

↓ kind=loss
T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Smoothness Aux

↓ kind=loss
T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Image Psnr

↑ dB kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Image Ssim

↑ kind=diag
V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Dehaze

↓ kind=loss
T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Preserve

↓ kind=loss
T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Total

↓ kind=loss
T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Visibility

↓ kind=loss
T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Visibility Aux

↓ kind=loss
T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Visibility Rank

↓ kind=loss
T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Visibility Tv

↓ kind=loss
T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Visibility Mean

kind=stat
T V
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —
2026-07-24_15-15-25_3900 val (raw) —

Visibility Std

kind=stat
T V
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —
2026-07-24_15-15-25_3900 val (raw) —

Gpu Mem Total Gb

T V
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —
2026-07-24_15-15-25_3900 val (raw) —

Gpu Mem Used Gb

T V
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —
2026-07-24_15-15-25_3900 val (raw) —

Gpu Mem Util Pct

T V
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —
2026-07-24_15-15-25_3900 val (raw) —

Gpu Util Pct

T V
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —
2026-07-24_15-15-25_3900 val (raw) —

Lr

T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Visibility Aux Scale

T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Data/Camera Enabled Any Source

V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Data/Camera Fields Emitted

V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Data/Camera Rays Emitted

V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

Train/Camera Valid Fraction

T
2026-07-24_15-15-25_3900 train (smoothed) —
2026-07-24_15-15-25_3900 train (raw) —

Val/Source/Rds/Camera Valid Fraction

V
2026-07-24_15-15-25_3900 val (smoothed) —
2026-07-24_15-15-25_3900 val (raw) —

No output snapshots found for this run.

Outputs are generated during training and saved to outputs/epoch_N_step_M/ directories.

epoch 1 / step 2644 Checkpoint #4059
7/24/2026, 7:05:28 PM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-24-15-15-25-3900/checkpoint-1
epoch 3 / step 5288 Checkpoint #4060
7/24/2026, 7:05:28 PM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-24-15-15-25-3900/checkpoint-3
epoch 5 / step 7932 Checkpoint #4065
7/25/2026, 9:55:59 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-24-15-15-25-3900/checkpoint-5
epoch 7 / step 10576 Checkpoint #4066
7/25/2026, 9:55:59 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-24-15-15-25-3900/checkpoint-7
epoch 9 / step 13220 Checkpoint #4067
7/25/2026, 9:55:59 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-24-15-15-25-3900/checkpoint-9
epoch 11 / step 15864 Checkpoint #4068
7/25/2026, 9:55:59 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-24-15-15-25-3900/checkpoint-11
epoch 13 / step 18508 Checkpoint #4069
7/25/2026, 9:55:59 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-24-15-15-25-3900/checkpoint-13
epoch 15 / step 21152 Checkpoint #4077
7/25/2026, 11:06:53 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-24-15-15-25-3900/checkpoint-15
epoch 17 / step 23796 Checkpoint #4102
7/25/2026, 1:21:10 PM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-24-15-15-25-3900/checkpoint-17
epoch 19 / step 26440 Checkpoint #4130
7/25/2026, 3:21:36 PM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-24-15-15-25-3900/checkpoint-19
epoch 21 / step 29084 Checkpoint #4171
7/25/2026, 6:48:43 PM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-24-15-15-25-3900/checkpoint-21
epoch 23 / step 31728 Checkpoint #4222
7/25/2026, 9:10:02 PM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-24-15-15-25-3900/checkpoint-23
Producer launch exports are available. Manage launch-owned exports here for quick reference.
Open Launch

Inherited Launch Exports

These exports are published by the run's producer launch.

Published

The producer launch does not publish any exports yet.

Run-Owned Exports

Publish direct filesystem paths here.

Published

No run-owned exports are published yet.

Raw Artifacts

Run-owned exports are typically direct paths, so there are no captured artifacts to publish from here.

Euler View - ML Experiment Monitor