Euler View
No jobs

Run Dossier

Training run for SD 2.1. Mari

85 metrics · 27,721 train · 163 val

Chart Overlays

Compare this run against its peers

0/63
Add peer runs here to layer their train and val traces onto the current metric charts.
crashed Jul 1, 10:50 Duration: 8h 8m Python 3.11.7 train_joint.py --config /cluster/home/drothenpiele/models/stable_diffusion_mari/mari/.euler-launches/euler_launch_51d8f6ee-a130-4390-b782-be32a097e6c6/joint_vkitti.yaml
lr 4e-5batch 16epochs 200precision fp16model stable-diffusion-2-1 seed 42wd 0grad accum 1warmup 0.05000ema nograd clip 1
ID 5244736
mari/runs/2026-07-01_10-50-20_bcf0
Error
BrokenPipeError: [Errno 32] Broken pipe
Metrics 85/85
camera.train 9
camera_cx_abs_error_px
camera_cy_abs_error_px
camera_focal_rel_error
camera_fx_rel_error
camera_fy_rel_error
camera_intrinsics_rel_error
camera_principal_point_saturation_fraction
camera_principal_rel_error
camera_valid_fraction
depth.train 29
diag 13
depth_abs_rel
depth_delta1
depth_edge_smoothness
depth_head_abs_rel
depth_head_delta1
depth_head_mae
depth_hf_energy
depth_mae
depth_road_hsmooth
depth_tv
vae_decode_abs_rel
vae_decode_delta1
vae_decode_mae
loss 14
camera_ray
depth
depth_head
depth_tv
edge_smoothness
eg_ssi
geometry
geometry_consistency
geometry_log_depth
lambda_mse
multiscale_smoothness
point_aux
smoothness_aux
uncertainty
stat 2
geometry_depth_mean
geometry_uncertainty_mean
geometry.train 25
geometry_abs_rel
geometry_delta0_5
geometry_delta1
geometry_delta2
geometry_delta3
geometry_edge_f1
geometry_edge_grad_mae
geometry_edge_precision
geometry_edge_recall
geometry_enabled
geometry_log_depth_error_visibility_0p33_0p66
geometry_log_depth_error_visibility_0p66_1p00
geometry_log10
geometry_mae
geometry_ray_angle_deg
geometry_ray_angle_median_deg
geometry_ray_rho_auc_15
geometry_rmse
geometry_rmse_log
geometry_silog
geometry_sq_rel
geometry_texture_edge_false_positive_rate
geometry_uncertainty_ause
geometry_uncertainty_nause
geometry_uncertainty_spearman
rgb.train 11
diag 2
image_psnr
image_ssim
loss 7
dehaze
preserve
total
visibility
visibility_aux
visibility_rank
visibility_tv
stat 2
visibility_mean
visibility_std
sys.train 6
gpu_mem_total_gb
gpu_mem_used_gb
gpu_mem_util_pct
gpu_util_pct
lr
visibility_aux_scale
Legacy 5
data/camera 3
enabled/any/source
fields/emitted
rays/emitted
train/camera/valid/fraction
val/source/rds/camera/valid/fraction
85 selected
X-Axis
Y-Scale
Series
Group
Namespace

Camera Cx Abs Error Px

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Camera Cy Abs Error Px

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Camera Focal Rel Error

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Camera Fx Rel Error

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Camera Fy Rel Error

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Camera Intrinsics Rel Error

↓ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Camera Principal Point Saturation Fraction

↓ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Camera Principal Rel Error

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Camera Valid Fraction

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Depth Abs Rel

↓ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Depth Delta1

↑ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Depth Edge Smoothness

↓ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Depth Head Abs Rel

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Depth Head Delta1

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Depth Head Mae

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Depth Hf Energy

↓ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Depth Mae

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Depth Road Hsmooth

↓ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Depth Tv

↓ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Vae Decode Abs Rel

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Vae Decode Delta1

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Vae Decode Mae

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Camera Ray

kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Depth

↓ kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Depth Head

kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Depth Tv

↓ kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Edge Smoothness

↓ kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Eg Ssi

kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Geometry

kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Geometry Consistency

kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Geometry Log Depth

kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Lambda Mse

kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Multiscale Smoothness

↓ kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Point Aux

kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Smoothness Aux

↓ kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Uncertainty

kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Geometry Depth Mean

kind=stat
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Geometry Uncertainty Mean

kind=stat
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Geometry Abs Rel

↓ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Delta0 5

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Delta1

↑ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Delta2

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Delta3

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Edge F1

↑ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Edge Grad Mae

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Edge Precision

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Edge Recall

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Enabled

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Log Depth Error Visibility 0p33 0p66

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Log Depth Error Visibility 0p66 1p00

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Log10

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Mae

↓ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Ray Angle Deg

↓ deg kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Ray Angle Median Deg

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Ray Rho Auc 15

↑ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Rmse

↓ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Rmse Log

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Silog

↓ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Sq Rel

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Texture Edge False Positive Rate

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Uncertainty Ause

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Uncertainty Nause

↓ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Geometry Uncertainty Spearman

kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Image Psnr

↑ dB kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Image Ssim

↑ kind=diag
V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Dehaze

↓ kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Preserve

↓ kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Total

↓ kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Visibility

↓ kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Visibility Aux

↓ kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Visibility Rank

↓ kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Visibility Tv

↓ kind=loss
T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Visibility Mean

kind=stat
T V
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —
2026-07-01_10-50-20_bcf0 val (raw) —

Visibility Std

kind=stat
T V
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —
2026-07-01_10-50-20_bcf0 val (raw) —

Gpu Mem Total Gb

T V
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —
2026-07-01_10-50-20_bcf0 val (raw) —

Gpu Mem Used Gb

T V
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —
2026-07-01_10-50-20_bcf0 val (raw) —

Gpu Mem Util Pct

T V
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —
2026-07-01_10-50-20_bcf0 val (raw) —

Gpu Util Pct

T V
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —
2026-07-01_10-50-20_bcf0 val (raw) —

Lr

T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Visibility Aux Scale

T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Data/Camera Enabled Any Source

V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Data/Camera Fields Emitted

V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Data/Camera Rays Emitted

V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

Train/Camera Valid Fraction

T
2026-07-01_10-50-20_bcf0 train (smoothed) —
2026-07-01_10-50-20_bcf0 train (raw) —

Val/Source/Rds/Camera Valid Fraction

V
2026-07-01_10-50-20_bcf0 val (smoothed) —
2026-07-01_10-50-20_bcf0 val (raw) —

No output snapshots found for this run.

Outputs are generated during training and saved to outputs/epoch_N_step_M/ directories.

epoch 1 / step 2442 Checkpoint #3782
7/1/2026, 11:12:04 AM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-01-10-50-20-bcf0/checkpoint-1
epoch 3 / step 4884 Checkpoint #3786
7/1/2026, 1:03:39 PM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-01-10-50-20-bcf0/checkpoint-3
epoch 5 / step 7326 Checkpoint #3795
7/1/2026, 3:38:09 PM
/cluster/scratch/drothenpiele/euler_train/mari/checkpoints/joint-dehaze-metric-2026-07-01-10-50-20-bcf0/checkpoint-5
Producer launch exports are available. Manage launch-owned exports here for quick reference.
Open Launch

Inherited Launch Exports

These exports are published by the run's producer launch.

Published

The producer launch does not publish any exports yet.

Run-Owned Exports

Publish direct filesystem paths here.

Published

No run-owned exports are published yet.

Raw Artifacts

Run-owned exports are typically direct paths, so there are no captured artifacts to publish from here.

Euler View - ML Experiment Monitor