Chart Overlays
Compare this run against its peers
Add peer runs here to layer their train and val traces onto the current metric charts.
crashed Apr 12, 14:12 Duration: 2h 46m Python 3.11.7
src/train.py --config configs/model/inverted_joint_metric.yaml --config configs/model/backbone_dinov2_vitl14.yaml --config configs/data/real_drive_sim_euler_loading.yaml --config configs/training/phase_b_joint_structure.yaml --init-from /cluster/scratch/drothenpiele/euler_train/unifuse/checkpoints/geometry-prompt/checkpoint-epoch-2.ptlr 2e-4batch 4epochs 12precision automodel weather_metric_system wd 1e-4
ID 63055282
unifuse/runs/2026-04-12_14-12-55_d068 Error
BrokenPipeError: [Errno 32] Broken pipe
Metrics 71/71
depth.train 49
diag 26
consolidation 1
delta_vs_prior
final 5
depth_abs_rel
depth_mae
depth_rmse
log_radius_l1
xyz_l1
pred 5
depth_abs_rel
depth_mae
depth_rmse
log_radius_l1
xyz_l1
prior 5
depth_abs_rel
depth_mae
depth_rmse
log_radius_l1
xyz_l1
structure 10
depth_l1
edge_l1
eval_ensemble_size
first_nonfinite_step_index
first_nonfinite_timestep
had_nonfinite
mode_is_sampled
mode_is_teacher_forced
sampled_debug_enabled
used_fp32_sampler
loss 18
consolidation 3
low_frequency
structure_align
structure_align_edge
final 4
confidence
depth
log_radius
xyz
pred 1
log_radius
prior 4
aux_metric_probe
aux_metric_probe_confidence
confidence
log_radius
structure 5
confidence
depth
depth_denoise_mse
edge
rgb_denoise_mse
no stage 1
total
stat 5
final 1
confidence
prior 1
confidence
structure 3
confidence
pred_finite_fraction
pred_has_nonfinite
rgb.train 6
diag 2
image_mae
image_psnr
loss 3
clear
confidence
rgb_denoise_mse
stat 1
visibility
sys.train 16
data_mean_depth
data_median_depth
data_valid_fraction
flag_refiner_enabled
flag_structure_enabled
gpu_mem_total_gb
gpu_mem_used_gb
gpu_mem_util_pct
gpu_util_pct
lr
lr.default
lr.geometry_encoder_adapter
lr.joint_structure_expert_heads
lr.joint_structure_expert_model
lr.prompt_encoder
ray_source_is_predicted
71 selected
X-Axis
Y-Scale
Series
Group
Namespace
Delta Vs Prior
↓ kind=diagstage=consolidation T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Depth Abs Rel
↓ kind=diagstage=final V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Depth Mae
↓ meters kind=diagstage=final T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Depth Rmse
↓ meters kind=diagstage=final V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Log Radius L1
↓ kind=diagstage=final T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Xyz L1
↓ meters kind=diagstage=final T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Depth Abs Rel
↓ kind=diagstage=pred V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Depth Mae
↓ meters kind=diagstage=pred T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Depth Rmse
↓ meters kind=diagstage=pred V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Log Radius L1
↓ kind=diagstage=pred T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Xyz L1
↓ meters kind=diagstage=pred T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Depth Abs Rel
↓ kind=diagstage=prior V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Depth Mae
↓ meters kind=diagstage=prior T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Depth Rmse
↓ meters kind=diagstage=prior V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Log Radius L1
↓ kind=diagstage=prior T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Xyz L1
↓ meters kind=diagstage=prior T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Depth L1
↓ kind=diagstage=structure T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Edge L1
↓ kind=diagstage=structure T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Eval Ensemble Size
kind=diagstage=structure V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
First Nonfinite Step Index
kind=diagstage=structure V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
First Nonfinite Timestep
kind=diagstage=structure V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Had Nonfinite
kind=diagstage=structure V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Mode Is Sampled
kind=diagstage=structure V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Mode Is Teacher Forced
kind=diagstage=structure V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Sampled Debug Enabled
kind=diagstage=structure V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Used Fp32 Sampler
kind=diagstage=structure V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Low Frequency
↓ kind=lossstage=consolidation T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Structure Align
↓ kind=lossstage=consolidation T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Structure Align Edge
↓ kind=lossstage=consolidation T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Confidence
↑ kind=lossstage=final T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Depth
↓ meters kind=lossstage=final T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Log Radius
↓ kind=lossstage=final T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Xyz
↓ meters kind=lossstage=final T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Log Radius
↓ kind=lossstage=pred T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Aux Metric Probe
kind=lossstage=prior T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Aux Metric Probe Confidence
kind=lossstage=prior T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Confidence
↑ kind=lossstage=prior T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Log Radius
↓ kind=lossstage=prior T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Confidence
↑ kind=lossstage=structure T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Depth
↓ meters kind=lossstage=structure T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Depth Denoise Mse
kind=lossstage=structure T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Edge
↓ kind=lossstage=structure T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Rgb Denoise Mse
kind=lossstage=structure T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Total
↓ kind=loss T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Confidence
↑ kind=statstage=final V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Confidence
↑ kind=statstage=prior V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Confidence
↑ kind=statstage=structure T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Pred Finite Fraction
kind=statstage=structure V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Pred Has Nonfinite
kind=statstage=structure V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Image Mae
↓ kind=diagstage=structure V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Image Psnr
↑ dB kind=diagstage=structure V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Clear
kind=lossstage=structure T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Confidence
↑ kind=lossstage=structure T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Rgb Denoise Mse
kind=lossstage=structure T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Visibility
↑ kind=statstage=weather V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Data Mean Depth
V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Data Median Depth
V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Data Valid Fraction
V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Flag Refiner Enabled
V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Flag Structure Enabled
V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
Gpu Mem Total Gb
T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Gpu Mem Used Gb
T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Gpu Mem Util Pct
T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Gpu Util Pct
T V
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
2026-04-12_14-12-55_d068 val (raw) —
Lr
T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Lr.Default
T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Lr.Geometry Encoder Adapter
T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Lr.Joint Structure Expert Heads
T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Lr.Joint Structure Expert Model
T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Lr.Prompt Encoder
T
2026-04-12_14-12-55_d068 train (smoothed) —
2026-04-12_14-12-55_d068 train (raw) —
Ray Source Is Predicted
V
2026-04-12_14-12-55_d068 val (smoothed) —
2026-04-12_14-12-55_d068 val (raw) —
No output snapshots found for this run.
Outputs are generated during training and saved to outputs/epoch_N_step_M/ directories.
epoch 1 / step 800 Checkpoint #2714
4/12/2026, 2:23:43 PM
/cluster/scratch/drothenpiele/euler_train/unifuse/checkpoints/joint-structure-expert-2026-04-12-14-12-55-d068/checkpoint-epoch-1.ptepoch 2 / step 1600 Checkpoint #2716
4/12/2026, 2:23:43 PM
/cluster/scratch/drothenpiele/euler_train/unifuse/checkpoints/joint-structure-expert-2026-04-12-14-12-55-d068/checkpoint-epoch-2.ptRun-Owned Exports
Publish direct filesystem paths here.
Published
No run-owned exports are published yet.
Raw Artifacts
Run-owned exports are typically direct paths, so there are no captured artifacts to publish from here.