Euler View
No jobs

Run Dossier

Training run for SD 2.1. Mari

42 metrics · 53,378 train · 182 val

Chart Overlays

Compare this run against its peers

0/105
Add peer runs here to layer their train and val traces onto the current metric charts.
crashed Jul 9, 16:18 Duration: 21h 9m Python 3.11.7 train_joint.py --config /cluster/home/drothenpiele/models/stable_diffusion_mari/mari/.euler-launches/euler_launch_878eb639-eaf5-4cb5-9eba-77f48e66a947/joint_vkitti.yaml --resume /cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-08-16-53-47-9bad/checkpoint-17
lr 4e-5batch 16epochs 200precision fp16model stable-diffusion-2-1 seed 42wd 0grad accum 1warmup 0.05000ema nograd clip 1
ID 6422047
mari/runs/2026-07-09_16-18-21_29b9
Error
BrokenPipeError: [Errno 32] Broken pipe
Metrics 42/42
camera.train 1
camera_valid_fraction
depth.train 19
diag 13
depth_abs_rel
depth_delta1
depth_edge_smoothness
depth_head_abs_rel
depth_head_delta1
depth_head_mae
depth_hf_energy
depth_mae
depth_road_hsmooth
depth_tv
vae_decode_abs_rel
vae_decode_delta1
vae_decode_mae
loss 6
depth
depth_head
depth_tv
edge_smoothness
multiscale_smoothness
smoothness_aux
rgb.train 11
diag 2
image_psnr
image_ssim
loss 7
dehaze
preserve
total
visibility
visibility_aux
visibility_rank
visibility_tv
stat 2
visibility_mean
visibility_std
sys.train 6
gpu_mem_total_gb
gpu_mem_used_gb
gpu_mem_util_pct
gpu_util_pct
lr
visibility_aux_scale
Legacy 5
data/camera 3
enabled/any/source
fields/emitted
rays/emitted
train/camera/valid/fraction
val/source/rds/camera/valid/fraction
42 selected
X-Axis
Y-Scale
Series
Group
Namespace

Camera Valid Fraction

kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Depth Abs Rel

↓ kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Depth Delta1

↑ kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Depth Edge Smoothness

↓ kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Depth Head Abs Rel

kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Depth Head Delta1

kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Depth Head Mae

kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Depth Hf Energy

↓ kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Depth Mae

kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Depth Road Hsmooth

↓ kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Depth Tv

↓ kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Vae Decode Abs Rel

kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Vae Decode Delta1

kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Vae Decode Mae

kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Depth

↓ kind=loss
T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Depth Head

kind=loss
T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Depth Tv

↓ kind=loss
T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Edge Smoothness

↓ kind=loss
T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Multiscale Smoothness

↓ kind=loss
T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Smoothness Aux

↓ kind=loss
T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Image Psnr

↑ dB kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Image Ssim

↑ kind=diag
V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Dehaze

↓ kind=loss
T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Preserve

↓ kind=loss
T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Total

↓ kind=loss
T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Visibility

↓ kind=loss
T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Visibility Aux

↓ kind=loss
T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Visibility Rank

↓ kind=loss
T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Visibility Tv

↓ kind=loss
T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Visibility Mean

kind=stat
T V
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —
2026-07-09_16-18-21_29b9 val (raw) —

Visibility Std

kind=stat
T V
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —
2026-07-09_16-18-21_29b9 val (raw) —

Gpu Mem Total Gb

T V
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —
2026-07-09_16-18-21_29b9 val (raw) —

Gpu Mem Used Gb

T V
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —
2026-07-09_16-18-21_29b9 val (raw) —

Gpu Mem Util Pct

T V
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —
2026-07-09_16-18-21_29b9 val (raw) —

Gpu Util Pct

T V
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —
2026-07-09_16-18-21_29b9 val (raw) —

Lr

T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Visibility Aux Scale

T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Data/Camera Enabled Any Source

V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Data/Camera Fields Emitted

V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Data/Camera Rays Emitted

V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

Train/Camera Valid Fraction

T
2026-07-09_16-18-21_29b9 train (smoothed) —
2026-07-09_16-18-21_29b9 train (raw) —

Val/Source/Rds/Camera Valid Fraction

V
2026-07-09_16-18-21_29b9 val (smoothed) —
2026-07-09_16-18-21_29b9 val (raw) —

No output snapshots found for this run.

Outputs are generated during training and saved to outputs/epoch_N_step_M/ directories.

epoch 19 / step 31700 Checkpoint #3889
7/10/2026, 9:15:23 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-09-16-18-21-29b9/checkpoint-19
epoch 21 / step 34870 Checkpoint #3890
7/10/2026, 9:15:23 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-09-16-18-21-29b9/checkpoint-21
epoch 23 / step 38040 Checkpoint #3891
7/10/2026, 9:15:23 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-09-16-18-21-29b9/checkpoint-23
epoch 25 / step 41210 Checkpoint #3892
7/10/2026, 9:15:23 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-09-16-18-21-29b9/checkpoint-25
epoch 27 / step 44380 Checkpoint #3893
7/10/2026, 9:15:23 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-09-16-18-21-29b9/checkpoint-27
epoch 29 / step 47550 Checkpoint #3894
7/10/2026, 9:15:23 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-09-16-18-21-29b9/checkpoint-29
epoch 31 / step 50720 Checkpoint #3895
7/10/2026, 9:15:23 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-09-16-18-21-29b9/checkpoint-31
epoch 33 / step 53890 Checkpoint #3896
7/10/2026, 9:15:23 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-09-16-18-21-29b9/checkpoint-33
epoch 35 / step 57060 Checkpoint #3921
7/10/2026, 11:14:37 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-09-16-18-21-29b9/checkpoint-35
Producer launch exports are available. Manage launch-owned exports here for quick reference.
Open Launch

Inherited Launch Exports

These exports are published by the run's producer launch.

Published

The producer launch does not publish any exports yet.

Run-Owned Exports

Publish direct filesystem paths here.

Published

No run-owned exports are published yet.

Raw Artifacts

Run-owned exports are typically direct paths, so there are no captured artifacts to publish from here.

Euler View - ML Experiment Monitor