Euler View
1 pending
Back to SD 2.1. Mari

mari-uncert 48986b3d

training failed

Output

pending 48986b3d
Launch Output
Waiting for SLURM job to start...

Failure Metadata

failed
slurm.monitor SBATCH_RUNTIME_FAILED

SLURM job failed

Context JSON 
{
  "slurmJobId": 61669783,
  "monitorMode": "slurm",
  "interactiveAllocationJobId": null
}

Datasets

4
Dataset clear_root
Dataset real-drive-sim
Type rgb
Split full
Path /cluster/work/igp_psr/drothenpiele/data/rds/rgb.zip/
Dataset depth_root
Dataset real-drive-sim
Type depth
Split full
Path /cluster/work/igp_psr/drothenpiele/data/rds/depth_sky_2k.zip/
Dataset train_hazy_root
Dataset real-drive-sim (euler-fog-dcp-heuristic)
Type rgb
Split train
Path /cluster/work/igp_psr/drothenpiele/data/rds/foggy_rgb.zip/
Dataset val_hazy_root
Dataset real-drive-sim (euler-fog-dcp-heuristic)
Type rgb
Split val
Path /cluster/work/igp_psr/drothenpiele/data/rds/foggy_rgb.zip/

Execution Artifacts

2
run.sh

Published Exports

Launch-Owned Exports

These semantic handles are what downstream pipelines should resolve against. Auto-published exports come from config-template metadata captured on this launch.

Published

This launch does not publish any exports yet.

Raw Artifacts

Parameters

100

Typed Parameters

Dataset clear_root
real-drive-sim / full (4)
Dataset depth_root
real-drive-sim / full (32)
Dataset train_hazy_root
real-drive-sim / train (34)
Dataset val_hazy_root
real-drive-sim / val (35)

Simple Parameters

clip
true
seed
42
n_gpus
1
tracker
wandb
use_ema
false
gpu_type
rtx_4090
job_name
sd_train_concat
norm_max
1
norm_min
-1
run_name
joint-dehaze-depth-vkitti
tmp_size
10G
adam_8bit
true
data_kind
vkitti2
ema_decay
0.9999
ema_dtype
float16
log_every
50
max_depth
2000
min_depth
0.00001
adam_beta1
0.9
adam_beta2
0.999
batch_size
1
image_size
[512, 512]
log_images
true
num_epochs
200
output_dir
/cluster/scratch/drothenpiele/SD21/exp_1
time_limit
2-00:00:00
mem_per_cpu
8G
num_workers
4
adam_epsilon
1e-8
conditioning
concat
env_activate
source ~/.cache/venv/mari/bin/activate
lambda_depth
1
min_lr_ratio
0.01
project_name
joint-dehazing
warmup_ratio
0.05
weight_decay
0.01
cpus_per_task
8
lambda_dehaze
1
learning_rate
0.00003
max_grad_norm
1
freeze_encoder
false
num_log_images
4
val_batch_size
1
enable_xformers
true
euler_train_dir
/cluster/work/igp_psr/drothenpiele/data/out/train/sd21-joint
mixed_precision
fp16
prediction_type
v_prediction
depth_noise_type
annealed_multires
lr_schedule_type
constant_with_warmup
min_max_quantile
0.02
no_decay_enabled
false
pretrained_model
sd2-community/stable-diffusion-2-1
conv_in_init_mode
marigold
joint_weight_psnr
1
joint_weight_ssim
10
lambda_visibility
0.05
depth_ensemble_tol
0.001
val_every_n_epochs
10
depth_ensemble_size
4
num_inference_steps
25
save_every_n_epochs
10
use_visibility_head
true
warmup_start_factor
0.001
lambda_visibility_tv
0.005
depth_multires_levels
4
encoder_learning_rate
0.000015
visibility_rank_pairs
128
weight_decay_backbone
0.01
weight_decay_no_decay
0
depth_ensemble_max_res
1024
gradient_checkpointing
true
lambda_visibility_rank
0.02
visibility_rank_margin
0.05
depth_ensemble_max_iter
2
depth_multires_strength
0.9
joint_weight_delta1_pct
0.5
keep_last_n_checkpoints
3
visibility_target_gamma
4
visibility_warmup_steps
1000
depth_ensemble_reduction
median
depth_normalization_type
scale_shift_depth
joint_weight_abs_rel_pct
0.5
num_inference_steps_depth
5
enable_efficient_attention
true
lambda_visibility_preserve
0.05
visibility_hidden_channels
32
visibility_target_quantile
0.9
weight_decay_depth_decoder
0.01
checkpoint_selection_metric
"joint" # "joint" or single metric: psnr/ssim/delta1/abs_rel/val/...
depth_decoder_learning_rate
0.000036
gradient_accumulation_steps
16
weight_decay_dehaze_decoder
0.01
dehaze_decoder_learning_rate
0.00003
checkpoint_selection_direction
"auto" # used for single-metric selection: auto|max|min
depth_multires_downscale_factor
2
depth_ensemble_regularizer_strength
0.02
Raw JSON 
{
  "clip": "true",
  "seed": 42,
  "n_gpus": "1",
  "tracker": "wandb",
  "use_ema": "false",
  "gpu_type": "rtx_4090",
  "job_name": "sd_train_concat",
  "norm_max": 1,
  "norm_min": -1,
  "run_name": "joint-dehaze-depth-vkitti",
  "tmp_s...

Events

Launch Events

0
No launch events recorded.
Euler View - ML Experiment Monitor