Package evaluation of Schedulers on Julia 1.13.0-DEV.790 (77b90b9e62*) started at 2025-07-01T01:13:16.756 ################################################################################ # Set-up # Installing PkgEval dependencies (TestEnv)... Set-up completed after 8.24s ################################################################################ # Installation # Installing Schedulers... Resolving package versions... Updating `~/.julia/environments/v1.13/Project.toml` [fcb5363a] + Schedulers v0.11.3 Updating `~/.julia/environments/v1.13/Manifest.toml` [682c06a0] + JSON v0.21.4 [69de0a69] + Parsers v2.8.3 [aea7be01] + PrecompileTools v1.3.2 [21216c6a] + Preferences v1.4.3 [fcb5363a] + Schedulers v0.11.3 [10745b16] + Statistics v1.11.1 [56f22d72] + Artifacts v1.11.0 [ade2ca70] + Dates v1.11.0 [8ba89e20] + Distributed v1.11.0 [8f399da3] + Libdl v1.11.0 [37e2e46d] + LinearAlgebra v1.12.0 [56ddb016] + Logging v1.11.0 [a63ad114] + Mmap v1.11.0 [de0858da] + Printf v1.11.0 [9a3f8284] + Random v1.11.0 [ea8e919c] + SHA v0.7.0 [9e88b42a] + Serialization v1.11.0 [6462fe0b] + Sockets v1.11.0 [fa267f1f] + TOML v1.0.3 [cf7118a7] + UUIDs v1.11.0 [4ec0a83e] + Unicode v1.11.0 [e66e0078] + CompilerSupportLibraries_jll v1.3.0+1 [4536629a] + OpenBLAS_jll v0.3.29+0 [8e850b90] + libblastrampoline_jll v5.13.1+0 Installation completed after 1.04s ################################################################################ # Precompilation # Precompiling PkgEval dependencies... Precompiling package dependencies... Precompilation completed after 16.2s ################################################################################ # Testing # Testing Schedulers Status `/tmp/jl_MK3KLR/Project.toml` [fcb5363a] Schedulers v0.11.3 [8ba89e20] Distributed v1.11.0 [56ddb016] Logging v1.11.0 [9a3f8284] Random v1.11.0 [9e88b42a] Serialization v1.11.0 [8dfed614] Test v1.11.0 Status `/tmp/jl_MK3KLR/Manifest.toml` [682c06a0] JSON v0.21.4 [69de0a69] Parsers v2.8.3 [aea7be01] PrecompileTools v1.3.2 [21216c6a] Preferences v1.4.3 [fcb5363a] Schedulers v0.11.3 [10745b16] Statistics v1.11.1 [56f22d72] Artifacts v1.11.0 [2a0f44e3] Base64 v1.11.0 [ade2ca70] Dates v1.11.0 [8ba89e20] Distributed v1.11.0 [b77e0a4c] InteractiveUtils v1.11.0 [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0 [8f399da3] Libdl v1.11.0 [37e2e46d] LinearAlgebra v1.12.0 [56ddb016] Logging v1.11.0 [d6f4376e] Markdown v1.11.0 [a63ad114] Mmap v1.11.0 [de0858da] Printf v1.11.0 [9a3f8284] Random v1.11.0 [ea8e919c] SHA v0.7.0 [9e88b42a] Serialization v1.11.0 [6462fe0b] Sockets v1.11.0 [f489334b] StyledStrings v1.11.0 [fa267f1f] TOML v1.0.3 [8dfed614] Test v1.11.0 [cf7118a7] UUIDs v1.11.0 [4ec0a83e] Unicode v1.11.0 [e66e0078] CompilerSupportLibraries_jll v1.3.0+1 [4536629a] OpenBLAS_jll v0.3.29+0 [8e850b90] libblastrampoline_jll v5.13.1+0 Testing Running tests... [ Info: running task 1 on process 2 (Schedulers-primary-S6HJ4WCf); 5 workers total; 9 tasks left in task-pool. [ Info: running task 2 on process 3 (Schedulers-primary-S6HJ4WCf); 5 workers total; 5 tasks left in task-pool. [ Info: running task 3 on process 4 (Schedulers-primary-S6HJ4WCf); 5 workers total; 5 tasks left in task-pool. [ Info: running task 4 on process 5 (Schedulers-primary-S6HJ4WCf); 5 workers total; 5 tasks left in task-pool. [ Info: running task 5 on process 6 (Schedulers-primary-S6HJ4WCf); 5 workers total; 5 tasks left in task-pool. [ Info: running task 6 on process 2 (Schedulers-primary-S6HJ4WCf); 5 workers total; 4 tasks left in task-pool. [ Info: running task 7 on process 3 (Schedulers-primary-S6HJ4WCf); 5 workers total; 3 tasks left in task-pool. [ Info: running task 8 on process 4 (Schedulers-primary-S6HJ4WCf); 5 workers total; 0 tasks left in task-pool. [ Info: running task 9 on process 5 (Schedulers-primary-S6HJ4WCf); 5 workers total; 0 tasks left in task-pool. [ Info: running task 10 on process 6 (Schedulers-primary-S6HJ4WCf); 5 workers total; 0 tasks left in task-pool. Test Summary: | Pass Total Time pmap, stable cluster test | 16 16 2m32.5s [ Info: running task 1 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 99 tasks left in task-pool. [ Info: running task 2 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 98 tasks left in task-pool. [ Info: running task 3 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 97 tasks left in task-pool. [ Info: running task 4 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 96 tasks left in task-pool. ┌ Warning: addprocs/rmprocs taking longer than expected, cancelling. └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:776 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 │ [7] top-level scope │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:46 │ [8] macro expansion │ @ /opt/julia/share/julia/stdlib/v1.13/Test/src/Test.jl:1855 [inlined] │ [9] macro expansion │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:55 [inlined] │ [10] include(mapexpr::Function, mod::Module, _path::String) │ @ Base ./Base.jl:310 │ [11] top-level scope │ @ none:6 │ [12] eval(m::Module, e::Any) │ @ Core ./boot.jl:489 │ [13] exec_options(opts::Base.JLOptions) │ @ Base ./client.jl:290 │ [14] _start() │ @ Base ./client.jl:557 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 5 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 95 tasks left in task-pool. [ Info: running task 6 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 94 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 │ [7] top-level scope │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:46 │ [8] macro expansion │ @ /opt/julia/share/julia/stdlib/v1.13/Test/src/Test.jl:1855 [inlined] │ [9] macro expansion │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:55 [inlined] │ [10] include(mapexpr::Function, mod::Module, _path::String) │ @ Base ./Base.jl:310 │ [11] top-level scope │ @ none:6 │ [12] eval(m::Module, e::Any) │ @ Core ./boot.jl:489 │ [13] exec_options(opts::Base.JLOptions) │ @ Base ./client.jl:290 │ [14] _start() │ @ Base ./client.jl:557 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 │ [7] top-level scope │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:46 │ [8] macro expansion │ @ /opt/julia/share/julia/stdlib/v1.13/Test/src/Test.jl:1855 [inlined] │ [9] macro expansion │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:55 [inlined] │ [10] include(mapexpr::Function, mod::Module, _path::String) │ @ Base ./Base.jl:310 │ [11] top-level scope │ @ none:6 │ [12] eval(m::Module, e::Any) │ @ Core ./boot.jl:489 │ [13] exec_options(opts::Base.JLOptions) │ @ Base ./client.jl:290 │ [14] _start() │ @ Base ./client.jl:557 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 7 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 93 tasks left in task-pool. [ Info: running task 8 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 92 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 │ [7] top-level scope │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:46 │ [8] macro expansion │ @ /opt/julia/share/julia/stdlib/v1.13/Test/src/Test.jl:1855 [inlined] │ [9] macro expansion │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:55 [inlined] │ [10] include(mapexpr::Function, mod::Module, _path::String) │ @ Base ./Base.jl:310 │ [11] top-level scope │ @ none:6 │ [12] eval(m::Module, e::Any) │ @ Core ./boot.jl:489 │ [13] exec_options(opts::Base.JLOptions) │ @ Base ./client.jl:290 │ [14] _start() │ @ Base ./client.jl:557 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 │ [7] top-level scope │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:46 │ [8] macro expansion │ @ /opt/julia/share/julia/stdlib/v1.13/Test/src/Test.jl:1855 [inlined] │ [9] macro expansion │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:55 [inlined] │ [10] include(mapexpr::Function, mod::Module, _path::String) │ @ Base ./Base.jl:310 │ [11] top-level scope │ @ none:6 │ [12] eval(m::Module, e::Any) │ @ Core ./boot.jl:489 │ [13] exec_options(opts::Base.JLOptions) │ @ Base ./client.jl:290 │ [14] _start() │ @ Base ./client.jl:557 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 9 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 91 tasks left in task-pool. [ Info: running task 10 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 90 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 │ [7] top-level scope │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:46 │ [8] macro expansion │ @ /opt/julia/share/julia/stdlib/v1.13/Test/src/Test.jl:1855 [inlined] │ [9] macro expansion │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:55 [inlined] │ [10] include(mapexpr::Function, mod::Module, _path::String) │ @ Base ./Base.jl:310 │ [11] top-level scope │ @ none:6 │ [12] eval(m::Module, e::Any) │ @ Core ./boot.jl:489 │ [13] exec_options(opts::Base.JLOptions) │ @ Base ./client.jl:290 │ [14] _start() │ @ Base ./client.jl:557 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 │ [7] top-level scope │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:46 │ [8] macro expansion │ @ /opt/julia/share/julia/stdlib/v1.13/Test/src/Test.jl:1855 [inlined] │ [9] macro expansion │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:55 [inlined] │ [10] include(mapexpr::Function, mod::Module, _path::String) │ @ Base ./Base.jl:310 │ [11] top-level scope │ @ none:6 │ [12] eval(m::Module, e::Any) │ @ Core ./boot.jl:489 │ [13] exec_options(opts::Base.JLOptions) │ @ Base ./client.jl:290 │ [14] _start() │ @ Base ./client.jl:557 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 11 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 89 tasks left in task-pool. [ Info: running task 12 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 88 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 │ [7] top-level scope │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:46 │ [8] macro expansion │ @ /opt/julia/share/julia/stdlib/v1.13/Test/src/Test.jl:1855 [inlined] │ [9] macro expansion │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:55 [inlined] │ [10] include(mapexpr::Function, mod::Module, _path::String) │ @ Base ./Base.jl:310 │ [11] top-level scope │ @ none:6 │ [12] eval(m::Module, e::Any) │ @ Core ./boot.jl:489 │ [13] exec_options(opts::Base.JLOptions) │ @ Base ./client.jl:290 │ [14] _start() │ @ Base ./client.jl:557 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 │ [7] top-level scope │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:46 │ [8] macro expansion │ @ /opt/julia/share/julia/stdlib/v1.13/Test/src/Test.jl:1855 [inlined] │ [9] macro expansion │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:55 [inlined] │ [10] include(mapexpr::Function, mod::Module, _path::String) │ @ Base ./Base.jl:310 │ [11] top-level scope │ @ none:6 │ [12] eval(m::Module, e::Any) │ @ Core ./boot.jl:489 │ [13] exec_options(opts::Base.JLOptions) │ @ Base ./client.jl:290 │ [14] _start() │ @ Base ./client.jl:557 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 From worker 16: ErrorException("Process(1) - Invalid connection credentials sent by remote.")CapturedException(ErrorException("Process(1) - Invalid connection credentials sent by remote."), Any[(error(s::String) at Master process (id 1) could not connect within 120.0 seconds. From worker 16: exiting. From worker 11: ErrorException("Process(1) - Invalid connection credentials sent by remote.")CapturedException(ErrorException("Process(1) - Invalid connection credentials sent by remote."), Any[(error(s::String) at Master process (id 1) could not connect within 120.0 seconds. From worker 11: exiting. From worker 15: ErrorException("Process(1) - Invalid connection credentials sent by remote.")CapturedException(ErrorException("Process(1) - Invalid connection credentials sent by remote."), Any[(error(s::String) at Master process (id 1) could not connect within 120.0 seconds. From worker 15: exiting. From worker 13: ErrorException("Process(1) - Invalid connection credentials sent by remote.")CapturedException(ErrorException("Process(1) - Invalid connection credentials sent by remote."), Any[(error(s::String) at Master process (id 1) could not connect within 120.0 seconds. From worker 13: exiting. From worker 12: ErrorException("Process(1) - Invalid connection credentials sent by remote.")CapturedException(ErrorException("Process(1) - Invalid connection credentials sent by remote."), Any[(error(s::String) at Master process (id 1) could not connect within 120.0 seconds. From worker 12: exiting. From worker 14: ErrorException("Process(1) - Invalid connection credentials sent by remote.")CapturedException(ErrorException("Process(1) - Invalid connection credentials sent by remote."), Any[(error(s::String) at Master process (id 1) could not connect within 120.0 seconds. From worker 14: exiting. [ Info: running task 13 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 87 tasks left in task-pool. [ Info: running task 14 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 86 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 │ [7] top-level scope │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:46 │ [8] macro expansion │ @ /opt/julia/share/julia/stdlib/v1.13/Test/src/Test.jl:1855 [inlined] │ [9] macro expansion │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:55 [inlined] │ [10] include(mapexpr::Function, mod::Module, _path::String) │ @ Base ./Base.jl:310 │ [11] top-level scope │ @ none:6 │ [12] eval(m::Module, e::Any) │ @ Core ./boot.jl:489 │ [13] exec_options(opts::Base.JLOptions) │ @ Base ./client.jl:290 │ [14] _start() │ @ Base ./client.jl:557 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 From worker 16: error.jl: From worker 15: error.jl: From worker 11: error.jl:Worker 16 terminated. From worker 12: error.jl:Worker 15 terminated. UNHANDLED TASK ERROR: Version read failed. Connection closed by peer. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] process_hdr(s::Sockets.TCPSocket, validate_cookie::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:276 [3] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:158 [4] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:133 [5] (::Distributed.var"#process_messages##0#process_messages##1"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})() @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:121 [ Info: running task 15 on process 9 (Schedulers-primary-S6HJ4WCf); 4 workers total; 85 tasks left in task-pool. From worker 14: error.jl:[ Info: running task 16 on process 10 (Schedulers-primary-S6HJ4WCf); 4 workers total; 84 tasks left in task-pool. Worker 11 terminated. UNHANDLED TASK ERROR: Version read failed. Connection closed by peer. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] process_hdr(s::Sockets.TCPSocket, validate_cookie::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:276 [3] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:158 [4] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:133 [5] (::Distributed.var"#process_messages##0#process_messages##1"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})() @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:121 From worker 13: error.jl:Worker 12 terminated. UNHANDLED TASK ERROR: Version read failed. Connection closed by peer. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] process_hdr(s::Sockets.TCPSocket, validate_cookie::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:276 [3] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:158 [4] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:133 [5] (::Distributed.var"#process_messages##0#process_messages##1"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})() @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:121 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 Worker 14 terminated.┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 │ [7] top-level scope │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:46 │ [8] macro expansion │ @ /opt/julia/share/julia/stdlib/v1.13/Test/src/Test.jl:1855 [inlined] │ [9] macro expansion │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:55 [inlined] │ [10] include(mapexpr::Function, mod::Module, _path::String) │ @ Base ./Base.jl:310 │ [11] top-level scope │ @ none:6 │ [12] eval(m::Module, e::Any) │ @ Core ./boot.jl:489 │ [13] exec_options(opts::Base.JLOptions) │ @ Base ./client.jl:290 │ [14] _start() │ @ Base ./client.jl:557 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 UNHANDLED TASK ERROR: Version read failed. Connection closed by peer. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] process_hdr(s::Sockets.TCPSocket, validate_cookie::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:276 [3] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:158 [4] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:133 [5] (::Distributed.var"#process_messages##0#process_messages##1"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})() @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:121 Worker 13 terminated.UNHANDLED TASK ERROR: Version read failed. Connection closed by peer. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] process_hdr(s::Sockets.TCPSocket, validate_cookie::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:276 [3] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:158 [4] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:133 [5] (::Distributed.var"#process_messages##0#process_messages##1"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})() @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:121 UNHANDLED TASK ERROR: Version read failed. Connection closed by peer. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] process_hdr(s::Sockets.TCPSocket, validate_cookie::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:276 [3] message_handler_loop(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:158 [4] process_tcp_streams(r_stream::Sockets.TCPSocket, w_stream::Sockets.TCPSocket, incoming::Bool) @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:133 [5] (::Distributed.var"#process_messages##0#process_messages##1"{Sockets.TCPSocket, Sockets.TCPSocket, Bool})() @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:121 [ Info: running task 17 on process 7 (Schedulers-primary-S6HJ4WCf); 4 workers total; 83 tasks left in task-pool. [ Info: running task 18 on process 8 (Schedulers-primary-S6HJ4WCf); 4 workers total; 82 tasks left in task-pool. [ Info: running task 19 on process 9 (Schedulers-primary-S6HJ4WCf); 4 workers total; 81 tasks left in task-pool. [ Info: running task 20 on process 10 (Schedulers-primary-S6HJ4WCf); 4 workers total; 80 tasks left in task-pool. [ Info: running task 21 on process 7 (Schedulers-primary-S6HJ4WCf); 4 workers total; 79 tasks left in task-pool. [ Info: running task 22 on process 8 (Schedulers-primary-S6HJ4WCf); 4 workers total; 78 tasks left in task-pool. [ Info: running task 23 on process 9 (Schedulers-primary-S6HJ4WCf); 4 workers total; 77 tasks left in task-pool. [ Info: running task 24 on process 10 (Schedulers-primary-S6HJ4WCf); 4 workers total; 76 tasks left in task-pool. [ Info: running task 25 on process 7 (Schedulers-primary-S6HJ4WCf); 4 workers total; 75 tasks left in task-pool. [ Info: running task 26 on process 8 (Schedulers-primary-S6HJ4WCf); 4 workers total; 74 tasks left in task-pool. [ Info: running task 27 on process 9 (Schedulers-primary-S6HJ4WCf); 10 workers total; 73 tasks left in task-pool. [ Info: running task 28 on process 10 (Schedulers-primary-S6HJ4WCf); 10 workers total; 72 tasks left in task-pool. [ Info: running task 29 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 71 tasks left in task-pool. [ Info: running task 30 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 70 tasks left in task-pool. [ Info: running task 31 on process 9 (Schedulers-primary-S6HJ4WCf); 10 workers total; 69 tasks left in task-pool. [ Info: running task 32 on process 10 (Schedulers-primary-S6HJ4WCf); 10 workers total; 68 tasks left in task-pool. [ Info: running task 33 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 67 tasks left in task-pool. [ Info: running task 34 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 66 tasks left in task-pool. [ Info: running task 35 on process 9 (Schedulers-primary-S6HJ4WCf); 10 workers total; 65 tasks left in task-pool. [ Info: running task 36 on process 10 (Schedulers-primary-S6HJ4WCf); 10 workers total; 64 tasks left in task-pool. [ Info: running task 37 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 63 tasks left in task-pool. [ Info: running task 38 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 62 tasks left in task-pool. [ Info: running task 39 on process 9 (Schedulers-primary-S6HJ4WCf); 10 workers total; 61 tasks left in task-pool. [ Info: running task 40 on process 10 (Schedulers-primary-S6HJ4WCf); 10 workers total; 60 tasks left in task-pool. [ Info: running task 41 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 59 tasks left in task-pool. [ Info: running task 42 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 58 tasks left in task-pool. [ Info: running task 43 on process 9 (Schedulers-primary-S6HJ4WCf); 10 workers total; 57 tasks left in task-pool. [ Info: running task 44 on process 10 (Schedulers-primary-S6HJ4WCf); 10 workers total; 56 tasks left in task-pool. [ Info: running task 45 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 55 tasks left in task-pool. [ Info: running task 46 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 54 tasks left in task-pool. [ Info: running task 47 on process 10 (Schedulers-primary-S6HJ4WCf); 10 workers total; 53 tasks left in task-pool. [ Info: running task 48 on process 9 (Schedulers-primary-S6HJ4WCf); 10 workers total; 52 tasks left in task-pool. [ Info: running task 49 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 51 tasks left in task-pool. [ Info: running task 50 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 50 tasks left in task-pool. [ Info: running task 51 on process 10 (Schedulers-primary-S6HJ4WCf); 10 workers total; 49 tasks left in task-pool. [ Info: running task 52 on process 9 (Schedulers-primary-S6HJ4WCf); 10 workers total; 48 tasks left in task-pool. [ Info: running task 53 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 47 tasks left in task-pool. [ Info: running task 54 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 46 tasks left in task-pool. [ Info: running task 55 on process 10 (Schedulers-primary-S6HJ4WCf); 10 workers total; 45 tasks left in task-pool. [ Info: running task 56 on process 9 (Schedulers-primary-S6HJ4WCf); 10 workers total; 44 tasks left in task-pool. [ Info: running task 57 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 43 tasks left in task-pool. [ Info: running task 58 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 42 tasks left in task-pool. [ Info: running task 59 on process 17 (Schedulers-primary-S6HJ4WCf); 10 workers total; 41 tasks left in task-pool. [ Info: running task 60 on process 18 (Schedulers-primary-S6HJ4WCf); 10 workers total; 36 tasks left in task-pool. [ Info: running task 61 on process 19 (Schedulers-primary-S6HJ4WCf); 10 workers total; 36 tasks left in task-pool. [ Info: running task 62 on process 20 (Schedulers-primary-S6HJ4WCf); 10 workers total; 36 tasks left in task-pool. [ Info: running task 63 on process 21 (Schedulers-primary-S6HJ4WCf); 10 workers total; 36 tasks left in task-pool. [ Info: running task 64 on process 22 (Schedulers-primary-S6HJ4WCf); 10 workers total; 36 tasks left in task-pool. [ Info: running task 65 on process 10 (Schedulers-primary-S6HJ4WCf); 10 workers total; 35 tasks left in task-pool. [ Info: running task 66 on process 9 (Schedulers-primary-S6HJ4WCf); 10 workers total; 34 tasks left in task-pool. [ Info: running task 67 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 33 tasks left in task-pool. [ Info: running task 68 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 32 tasks left in task-pool. [ Info: running task 69 on process 17 (Schedulers-primary-S6HJ4WCf); 10 workers total; 31 tasks left in task-pool. [ Info: running task 70 on process 18 (Schedulers-primary-S6HJ4WCf); 10 workers total; 26 tasks left in task-pool. [ Info: running task 71 on process 19 (Schedulers-primary-S6HJ4WCf); 10 workers total; 26 tasks left in task-pool. [ Info: running task 72 on process 20 (Schedulers-primary-S6HJ4WCf); 10 workers total; 26 tasks left in task-pool. [ Info: running task 73 on process 21 (Schedulers-primary-S6HJ4WCf); 10 workers total; 26 tasks left in task-pool. [ Info: running task 74 on process 22 (Schedulers-primary-S6HJ4WCf); 10 workers total; 26 tasks left in task-pool. [ Info: running task 75 on process 10 (Schedulers-primary-S6HJ4WCf); 10 workers total; 25 tasks left in task-pool. [ Info: running task 76 on process 9 (Schedulers-primary-S6HJ4WCf); 10 workers total; 24 tasks left in task-pool. [ Info: running task 77 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 23 tasks left in task-pool. [ Info: running task 78 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 22 tasks left in task-pool. [ Info: running task 79 on process 17 (Schedulers-primary-S6HJ4WCf); 10 workers total; 21 tasks left in task-pool. [ Info: running task 80 on process 18 (Schedulers-primary-S6HJ4WCf); 10 workers total; 16 tasks left in task-pool. [ Info: running task 81 on process 19 (Schedulers-primary-S6HJ4WCf); 10 workers total; 16 tasks left in task-pool. [ Info: running task 82 on process 20 (Schedulers-primary-S6HJ4WCf); 10 workers total; 16 tasks left in task-pool. [ Info: running task 83 on process 21 (Schedulers-primary-S6HJ4WCf); 10 workers total; 16 tasks left in task-pool. [ Info: running task 84 on process 22 (Schedulers-primary-S6HJ4WCf); 10 workers total; 16 tasks left in task-pool. [ Info: running task 85 on process 10 (Schedulers-primary-S6HJ4WCf); 10 workers total; 15 tasks left in task-pool. [ Info: running task 86 on process 9 (Schedulers-primary-S6HJ4WCf); 10 workers total; 14 tasks left in task-pool. [ Info: running task 87 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 13 tasks left in task-pool. [ Info: running task 88 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 12 tasks left in task-pool. [ Info: running task 89 on process 17 (Schedulers-primary-S6HJ4WCf); 10 workers total; 11 tasks left in task-pool. [ Info: running task 90 on process 18 (Schedulers-primary-S6HJ4WCf); 10 workers total; 6 tasks left in task-pool. [ Info: running task 91 on process 19 (Schedulers-primary-S6HJ4WCf); 10 workers total; 6 tasks left in task-pool. [ Info: running task 92 on process 20 (Schedulers-primary-S6HJ4WCf); 10 workers total; 6 tasks left in task-pool. [ Info: running task 93 on process 21 (Schedulers-primary-S6HJ4WCf); 10 workers total; 6 tasks left in task-pool. [ Info: running task 94 on process 22 (Schedulers-primary-S6HJ4WCf); 10 workers total; 6 tasks left in task-pool. [ Info: running task 95 on process 10 (Schedulers-primary-S6HJ4WCf); 10 workers total; 5 tasks left in task-pool. [ Info: running task 96 on process 9 (Schedulers-primary-S6HJ4WCf); 10 workers total; 4 tasks left in task-pool. [ Info: running task 97 on process 8 (Schedulers-primary-S6HJ4WCf); 10 workers total; 3 tasks left in task-pool. [ Info: running task 98 on process 7 (Schedulers-primary-S6HJ4WCf); 10 workers total; 2 tasks left in task-pool. [ Info: running task 99 on process 17 (Schedulers-primary-S6HJ4WCf); 10 workers total; 1 tasks left in task-pool. [ Info: running task 100 on process 18 (Schedulers-primary-S6HJ4WCf); 10 workers total; 0 tasks left in task-pool. Test Summary: | Pass Total Time pmap, growing cluster test | 111 111 6m35.2s [ Info: running task 1 on process 23 (Schedulers-primary-S6HJ4WCf); 8 workers total; 99 tasks left in task-pool. [ Info: running task 2 on process 24 (Schedulers-primary-S6HJ4WCf); 8 workers total; 92 tasks left in task-pool. [ Info: running task 3 on process 25 (Schedulers-primary-S6HJ4WCf); 8 workers total; 92 tasks left in task-pool. [ Info: running task 4 on process 26 (Schedulers-primary-S6HJ4WCf); 8 workers total; 92 tasks left in task-pool. [ Info: running task 5 on process 27 (Schedulers-primary-S6HJ4WCf); 8 workers total; 92 tasks left in task-pool. [ Info: running task 6 on process 29 (Schedulers-primary-S6HJ4WCf); 8 workers total; 92 tasks left in task-pool. [ Info: running task 7 on process 30 (Schedulers-primary-S6HJ4WCf); 8 workers total; 92 tasks left in task-pool. [ Info: running task 8 on process 32 (Schedulers-primary-S6HJ4WCf); 8 workers total; 92 tasks left in task-pool. [ Info: running task 9 on process 23 (Schedulers-primary-S6HJ4WCf); 10 workers total; 91 tasks left in task-pool. [ Info: running task 10 on process 24 (Schedulers-primary-S6HJ4WCf); 10 workers total; 90 tasks left in task-pool. [ Info: running task 11 on process 25 (Schedulers-primary-S6HJ4WCf); 10 workers total; 89 tasks left in task-pool. [ Info: running task 12 on process 26 (Schedulers-primary-S6HJ4WCf); 10 workers total; 87 tasks left in task-pool. [ Info: running task 13 on process 27 (Schedulers-primary-S6HJ4WCf); 10 workers total; 84 tasks left in task-pool. [ Info: running task 14 on process 29 (Schedulers-primary-S6HJ4WCf); 10 workers total; 84 tasks left in task-pool. [ Info: running task 15 on process 30 (Schedulers-primary-S6HJ4WCf); 10 workers total; 84 tasks left in task-pool. [ Info: running task 16 on process 32 (Schedulers-primary-S6HJ4WCf); 10 workers total; 84 tasks left in task-pool. [ Info: running task 17 on process 23 (Schedulers-primary-S6HJ4WCf); 10 workers total; 83 tasks left in task-pool. [ Info: running task 18 on process 24 (Schedulers-primary-S6HJ4WCf); 10 workers total; 82 tasks left in task-pool. [ Info: running task 19 on process 25 (Schedulers-primary-S6HJ4WCf); 10 workers total; 79 tasks left in task-pool. [ Info: running task 20 on process 26 (Schedulers-primary-S6HJ4WCf); 10 workers total; 79 tasks left in task-pool. [ Info: running task 21 on process 29 (Schedulers-primary-S6HJ4WCf); 10 workers total; 79 tasks left in task-pool. [ Info: running task 22 on process 30 (Schedulers-primary-S6HJ4WCf); 10 workers total; 78 tasks left in task-pool. [ Info: running task 23 on process 32 (Schedulers-primary-S6HJ4WCf); 10 workers total; 77 tasks left in task-pool. [ Info: running task 24 on process 27 (Schedulers-primary-S6HJ4WCf); 10 workers total; 76 tasks left in task-pool. [ Info: running task 25 on process 33 (Schedulers-primary-S6HJ4WCf); 10 workers total; 75 tasks left in task-pool. [ Info: running task 26 on process 34 (Schedulers-primary-S6HJ4WCf); 10 workers total; 74 tasks left in task-pool. [ Info: running task 27 on process 23 (Schedulers-primary-S6HJ4WCf); 10 workers total; 73 tasks left in task-pool. [ Info: running task 28 on process 24 (Schedulers-primary-S6HJ4WCf); 10 workers total; 72 tasks left in task-pool. [ Info: running task 29 on process 25 (Schedulers-primary-S6HJ4WCf); 10 workers total; 67 tasks left in task-pool. [ Info: running task 30 on process 26 (Schedulers-primary-S6HJ4WCf); 10 workers total; 67 tasks left in task-pool. [ Info: running task 31 on process 29 (Schedulers-primary-S6HJ4WCf); 10 workers total; 67 tasks left in task-pool. [ Info: running task 32 on process 30 (Schedulers-primary-S6HJ4WCf); 10 workers total; 67 tasks left in task-pool. [ Info: running task 33 on process 32 (Schedulers-primary-S6HJ4WCf); 10 workers total; 67 tasks left in task-pool. [ Info: running task 34 on process 27 (Schedulers-primary-S6HJ4WCf); 10 workers total; 66 tasks left in task-pool. [ Info: running task 35 on process 23 (Schedulers-primary-S6HJ4WCf); 10 workers total; 65 tasks left in task-pool. [ Info: running task 36 on process 24 (Schedulers-primary-S6HJ4WCf); 10 workers total; 64 tasks left in task-pool. [ Info: running task 37 on process 25 (Schedulers-primary-S6HJ4WCf); 10 workers total; 56 tasks left in task-pool. [ Info: running task 38 on process 27 (Schedulers-primary-S6HJ4WCf); 10 workers total; 56 tasks left in task-pool. [ Info: running task 39 on process 26 (Schedulers-primary-S6HJ4WCf); 10 workers total; 56 tasks left in task-pool. [ Info: running task 40 on process 33 (Schedulers-primary-S6HJ4WCf); 10 workers total; 56 tasks left in task-pool. [ Info: running task 41 on process 29 (Schedulers-primary-S6HJ4WCf); 10 workers total; 56 tasks left in task-pool. [ Info: running task 42 on process 30 (Schedulers-primary-S6HJ4WCf); 10 workers total; 56 tasks left in task-pool. [ Info: running task 43 on process 32 (Schedulers-primary-S6HJ4WCf); 10 workers total; 56 tasks left in task-pool. [ Info: running task 44 on process 34 (Schedulers-primary-S6HJ4WCf); 10 workers total; 56 tasks left in task-pool. [ Info: running task 45 on process 23 (Schedulers-primary-S6HJ4WCf); 10 workers total; 55 tasks left in task-pool. [ Info: running task 46 on process 34 (Schedulers-primary-S6HJ4WCf); 10 workers total; 54 tasks left in task-pool. [ Info: running task 47 on process 24 (Schedulers-primary-S6HJ4WCf); 10 workers total; 53 tasks left in task-pool. [ Info: running task 48 on process 25 (Schedulers-primary-S6HJ4WCf); 10 workers total; 47 tasks left in task-pool. [ Info: running task 49 on process 27 (Schedulers-primary-S6HJ4WCf); 10 workers total; 47 tasks left in task-pool. [ Info: running task 50 on process 26 (Schedulers-primary-S6HJ4WCf); 10 workers total; 47 tasks left in task-pool. [ Info: running task 51 on process 33 (Schedulers-primary-S6HJ4WCf); 10 workers total; 47 tasks left in task-pool. [ Info: running task 52 on process 29 (Schedulers-primary-S6HJ4WCf); 10 workers total; 47 tasks left in task-pool. [ Info: running task 53 on process 30 (Schedulers-primary-S6HJ4WCf); 10 workers total; 46 tasks left in task-pool. [ Info: running task 54 on process 32 (Schedulers-primary-S6HJ4WCf); 10 workers total; 46 tasks left in task-pool. [ Info: running task 55 on process 23 (Schedulers-primary-S6HJ4WCf); 10 workers total; 45 tasks left in task-pool. [ Info: running task 56 on process 34 (Schedulers-primary-S6HJ4WCf); 10 workers total; 44 tasks left in task-pool. [ Info: running task 57 on process 24 (Schedulers-primary-S6HJ4WCf); 10 workers total; 43 tasks left in task-pool. [ Info: running task 58 on process 25 (Schedulers-primary-S6HJ4WCf); 10 workers total; 36 tasks left in task-pool. [ Info: running task 59 on process 27 (Schedulers-primary-S6HJ4WCf); 10 workers total; 36 tasks left in task-pool. [ Info: running task 60 on process 26 (Schedulers-primary-S6HJ4WCf); 10 workers total; 36 tasks left in task-pool. [ Info: running task 61 on process 33 (Schedulers-primary-S6HJ4WCf); 10 workers total; 36 tasks left in task-pool. [ Info: running task 62 on process 29 (Schedulers-primary-S6HJ4WCf); 10 workers total; 36 tasks left in task-pool. [ Info: running task 63 on process 30 (Schedulers-primary-S6HJ4WCf); 10 workers total; 36 tasks left in task-pool. [ Info: running task 64 on process 32 (Schedulers-primary-S6HJ4WCf); 10 workers total; 36 tasks left in task-pool. [ Info: running task 65 on process 23 (Schedulers-primary-S6HJ4WCf); 10 workers total; 35 tasks left in task-pool. [ Info: running task 66 on process 34 (Schedulers-primary-S6HJ4WCf); 10 workers total; 34 tasks left in task-pool. [ Info: running task 67 on process 24 (Schedulers-primary-S6HJ4WCf); 10 workers total; 33 tasks left in task-pool. [ Info: running task 68 on process 25 (Schedulers-primary-S6HJ4WCf); 10 workers total; 31 tasks left in task-pool. [ Info: running task 69 on process 27 (Schedulers-primary-S6HJ4WCf); 10 workers total; 31 tasks left in task-pool. [ Info: running task 70 on process 26 (Schedulers-primary-S6HJ4WCf); 10 workers total; 30 tasks left in task-pool. [ Info: running task 71 on process 33 (Schedulers-primary-S6HJ4WCf); 10 workers total; 26 tasks left in task-pool. [ Info: running task 72 on process 29 (Schedulers-primary-S6HJ4WCf); 10 workers total; 26 tasks left in task-pool. [ Info: running task 73 on process 30 (Schedulers-primary-S6HJ4WCf); 10 workers total; 26 tasks left in task-pool. [ Info: running task 74 on process 32 (Schedulers-primary-S6HJ4WCf); 10 workers total; 26 tasks left in task-pool. [ Info: running task 75 on process 23 (Schedulers-primary-S6HJ4WCf); 10 workers total; 25 tasks left in task-pool. [ Info: running task 76 on process 34 (Schedulers-primary-S6HJ4WCf); 10 workers total; 24 tasks left in task-pool. [ Info: running task 77 on process 24 (Schedulers-primary-S6HJ4WCf); 10 workers total; 23 tasks left in task-pool. [ Info: running task 78 on process 25 (Schedulers-primary-S6HJ4WCf); 10 workers total; 16 tasks left in task-pool. [ Info: running task 79 on process 27 (Schedulers-primary-S6HJ4WCf); 10 workers total; 16 tasks left in task-pool. [ Info: running task 80 on process 26 (Schedulers-primary-S6HJ4WCf); 10 workers total; 16 tasks left in task-pool. [ Info: running task 81 on process 33 (Schedulers-primary-S6HJ4WCf); 10 workers total; 16 tasks left in task-pool. [ Info: running task 82 on process 29 (Schedulers-primary-S6HJ4WCf); 10 workers total; 16 tasks left in task-pool. [ Info: running task 83 on process 30 (Schedulers-primary-S6HJ4WCf); 10 workers total; 16 tasks left in task-pool. [ Info: running task 84 on process 32 (Schedulers-primary-S6HJ4WCf); 10 workers total; 16 tasks left in task-pool. [ Info: running task 85 on process 23 (Schedulers-primary-S6HJ4WCf); 10 workers total; 15 tasks left in task-pool. [ Info: running task 86 on process 34 (Schedulers-primary-S6HJ4WCf); 10 workers total; 14 tasks left in task-pool. [ Info: running task 87 on process 24 (Schedulers-primary-S6HJ4WCf); 10 workers total; 6 tasks left in task-pool. [ Info: running task 88 on process 25 (Schedulers-primary-S6HJ4WCf); 10 workers total; 6 tasks left in task-pool. [ Info: running task 89 on process 27 (Schedulers-primary-S6HJ4WCf); 10 workers total; 6 tasks left in task-pool. [ Info: running task 90 on process 26 (Schedulers-primary-S6HJ4WCf); 10 workers total; 6 tasks left in task-pool. [ Info: running task 91 on process 33 (Schedulers-primary-S6HJ4WCf); 10 workers total; 6 tasks left in task-pool. [ Info: running task 92 on process 29 (Schedulers-primary-S6HJ4WCf); 10 workers total; 6 tasks left in task-pool. [ Info: running task 93 on process 30 (Schedulers-primary-S6HJ4WCf); 10 workers total; 6 tasks left in task-pool. [ Info: running task 94 on process 32 (Schedulers-primary-S6HJ4WCf); 10 workers total; 6 tasks left in task-pool. [ Info: running task 95 on process 23 (Schedulers-primary-S6HJ4WCf); 10 workers total; 5 tasks left in task-pool. [ Info: running task 96 on process 34 (Schedulers-primary-S6HJ4WCf); 10 workers total; 4 tasks left in task-pool. [ Info: running task 97 on process 24 (Schedulers-primary-S6HJ4WCf); 10 workers total; 2 tasks left in task-pool. [ Info: running task 98 on process 25 (Schedulers-primary-S6HJ4WCf); 10 workers total; 2 tasks left in task-pool. [ Info: running task 99 on process 27 (Schedulers-primary-S6HJ4WCf); 10 workers total; 1 tasks left in task-pool. [ Info: running task 100 on process 26 (Schedulers-primary-S6HJ4WCf); 10 workers total; 0 tasks left in task-pool. Test Summary: | Pass Total Time pmap, elastic cluster with faults | 111 111 5m12.1s [ Info: running task 1 on process 35 (Schedulers-primary-S6HJ4WCf); 10 workers total; 104 tasks left in task-pool. [ Info: running task 2 on process 36 (Schedulers-primary-S6HJ4WCf); 10 workers total; 95 tasks left in task-pool. [ Info: running task 3 on process 37 (Schedulers-primary-S6HJ4WCf); 10 workers total; 95 tasks left in task-pool. [ Info: running task 4 on process 38 (Schedulers-primary-S6HJ4WCf); 10 workers total; 95 tasks left in task-pool. [ Info: running task 5 on process 40 (Schedulers-primary-S6HJ4WCf); 10 workers total; 95 tasks left in task-pool. [ Info: running task 6 on process 41 (Schedulers-primary-S6HJ4WCf); 10 workers total; 95 tasks left in task-pool. [ Info: running task 7 on process 43 (Schedulers-primary-S6HJ4WCf); 10 workers total; 95 tasks left in task-pool. [ Info: running task 8 on process 39 (Schedulers-primary-S6HJ4WCf); 10 workers total; 95 tasks left in task-pool. [ Info: running task 9 on process 42 (Schedulers-primary-S6HJ4WCf); 10 workers total; 95 tasks left in task-pool. [ Info: running task 10 on process 44 (Schedulers-primary-S6HJ4WCf); 10 workers total; 95 tasks left in task-pool. [ Info: running task 11 on process 35 (Schedulers-primary-S6HJ4WCf); 10 workers total; 94 tasks left in task-pool. [ Info: running task 12 on process 36 (Schedulers-primary-S6HJ4WCf); 10 workers total; 93 tasks left in task-pool. [ Info: running task 13 on process 37 (Schedulers-primary-S6HJ4WCf); 10 workers total; 92 tasks left in task-pool. [ Info: running task 14 on process 38 (Schedulers-primary-S6HJ4WCf); 10 workers total; 91 tasks left in task-pool. [ Info: running task 15 on process 40 (Schedulers-primary-S6HJ4WCf); 10 workers total; 90 tasks left in task-pool. [ Info: running task 16 on process 41 (Schedulers-primary-S6HJ4WCf); 10 workers total; 88 tasks left in task-pool. [ Info: running task 17 on process 43 (Schedulers-primary-S6HJ4WCf); 10 workers total; 88 tasks left in task-pool. [ Info: running task 18 on process 39 (Schedulers-primary-S6HJ4WCf); 10 workers total; 87 tasks left in task-pool. [ Info: running task 19 on process 42 (Schedulers-primary-S6HJ4WCf); 10 workers total; 85 tasks left in task-pool. [ Info: running task 20 on process 44 (Schedulers-primary-S6HJ4WCf); 10 workers total; 85 tasks left in task-pool. [ Info: running task 21 on process 35 (Schedulers-primary-S6HJ4WCf); 10 workers total; 84 tasks left in task-pool. [ Info: running task 22 on process 42 (Schedulers-primary-S6HJ4WCf); 10 workers total; 83 tasks left in task-pool. [ Info: running task 23 on process 44 (Schedulers-primary-S6HJ4WCf); 10 workers total; 82 tasks left in task-pool. [ Info: running task 24 on process 36 (Schedulers-primary-S6HJ4WCf); 10 workers total; 81 tasks left in task-pool. [ Info: running task 25 on process 37 (Schedulers-primary-S6HJ4WCf); 10 workers total; 80 tasks left in task-pool. [ Info: running task 26 on process 38 (Schedulers-primary-S6HJ4WCf); 10 workers total; 79 tasks left in task-pool. [ Info: running task 27 on process 40 (Schedulers-primary-S6HJ4WCf); 10 workers total; 77 tasks left in task-pool. [ Info: running task 28 on process 41 (Schedulers-primary-S6HJ4WCf); 10 workers total; 77 tasks left in task-pool. [ Info: running task 29 on process 43 (Schedulers-primary-S6HJ4WCf); 10 workers total; 76 tasks left in task-pool. [ Info: running task 30 on process 39 (Schedulers-primary-S6HJ4WCf); 10 workers total; 75 tasks left in task-pool. [ Info: running task 31 on process 35 (Schedulers-primary-S6HJ4WCf); 10 workers total; 74 tasks left in task-pool. [ Info: running task 32 on process 44 (Schedulers-primary-S6HJ4WCf); 10 workers total; 73 tasks left in task-pool. [ Info: running task 33 on process 37 (Schedulers-primary-S6HJ4WCf); 10 workers total; 72 tasks left in task-pool. [ Info: running task 34 on process 38 (Schedulers-primary-S6HJ4WCf); 10 workers total; 71 tasks left in task-pool. [ Info: running task 35 on process 40 (Schedulers-primary-S6HJ4WCf); 10 workers total; 68 tasks left in task-pool. [ Info: running task 36 on process 41 (Schedulers-primary-S6HJ4WCf); 10 workers total; 68 tasks left in task-pool. [ Info: running task 37 on process 43 (Schedulers-primary-S6HJ4WCf); 10 workers total; 68 tasks left in task-pool. [ Info: running task 38 on process 39 (Schedulers-primary-S6HJ4WCf); 10 workers total; 67 tasks left in task-pool. [ Info: running task 39 on process 42 (Schedulers-primary-S6HJ4WCf); 10 workers total; 66 tasks left in task-pool. [ Info: running task 40 on process 36 (Schedulers-primary-S6HJ4WCf); 10 workers total; 65 tasks left in task-pool. [ Info: running task 41 on process 35 (Schedulers-primary-S6HJ4WCf); 10 workers total; 64 tasks left in task-pool. [ Info: running task 42 on process 44 (Schedulers-primary-S6HJ4WCf); 10 workers total; 63 tasks left in task-pool. [ Info: running task 43 on process 38 (Schedulers-primary-S6HJ4WCf); 10 workers total; 62 tasks left in task-pool. [ Info: running task 44 on process 40 (Schedulers-primary-S6HJ4WCf); 10 workers total; 61 tasks left in task-pool. [ Info: running task 45 on process 41 (Schedulers-primary-S6HJ4WCf); 10 workers total; 60 tasks left in task-pool. [ Info: running task 46 on process 43 (Schedulers-primary-S6HJ4WCf); 10 workers total; 58 tasks left in task-pool. [ Info: running task 47 on process 39 (Schedulers-primary-S6HJ4WCf); 10 workers total; 58 tasks left in task-pool. [ Info: running task 48 on process 42 (Schedulers-primary-S6HJ4WCf); 10 workers total; 57 tasks left in task-pool. [ Info: running task 49 on process 36 (Schedulers-primary-S6HJ4WCf); 10 workers total; 56 tasks left in task-pool. [ Info: running task 50 on process 37 (Schedulers-primary-S6HJ4WCf); 10 workers total; 55 tasks left in task-pool. [ Info: running task 51 on process 35 (Schedulers-primary-S6HJ4WCf); 10 workers total; 54 tasks left in task-pool. [ Info: running task 52 on process 38 (Schedulers-primary-S6HJ4WCf); 10 workers total; 53 tasks left in task-pool. [ Info: running task 53 on process 40 (Schedulers-primary-S6HJ4WCf); 10 workers total; 52 tasks left in task-pool. [ Info: running task 54 on process 42 (Schedulers-primary-S6HJ4WCf); 10 workers total; 51 tasks left in task-pool. [ Info: running task 55 on process 44 (Schedulers-primary-S6HJ4WCf); 10 workers total; 50 tasks left in task-pool. [ Info: running task 56 on process 41 (Schedulers-primary-S6HJ4WCf); 10 workers total; 49 tasks left in task-pool. [ Info: running task 57 on process 43 (Schedulers-primary-S6HJ4WCf); 10 workers total; 48 tasks left in task-pool. [ Info: running task 58 on process 39 (Schedulers-primary-S6HJ4WCf); 10 workers total; 47 tasks left in task-pool. [ Info: running task 59 on process 36 (Schedulers-primary-S6HJ4WCf); 10 workers total; 46 tasks left in task-pool. [ Info: running task 60 on process 37 (Schedulers-primary-S6HJ4WCf); 10 workers total; 45 tasks left in task-pool. [ Info: running task 61 on process 35 (Schedulers-primary-S6HJ4WCf); 10 workers total; 44 tasks left in task-pool. [ Info: running task 62 on process 42 (Schedulers-primary-S6HJ4WCf); 10 workers total; 43 tasks left in task-pool. [ Info: running task 63 on process 44 (Schedulers-primary-S6HJ4WCf); 10 workers total; 42 tasks left in task-pool. [ Info: running task 64 on process 38 (Schedulers-primary-S6HJ4WCf); 10 workers total; 41 tasks left in task-pool. [ Info: running task 65 on process 40 (Schedulers-primary-S6HJ4WCf); 10 workers total; 40 tasks left in task-pool. [ Info: running task 66 on process 36 (Schedulers-primary-S6HJ4WCf); 10 workers total; 39 tasks left in task-pool. [ Info: running task 67 on process 37 (Schedulers-primary-S6HJ4WCf); 10 workers total; 35 tasks left in task-pool. [ Info: running task 68 on process 41 (Schedulers-primary-S6HJ4WCf); 10 workers total; 35 tasks left in task-pool. [ Info: running task 69 on process 43 (Schedulers-primary-S6HJ4WCf); 10 workers total; 35 tasks left in task-pool. [ Info: running task 70 on process 39 (Schedulers-primary-S6HJ4WCf); 10 workers total; 35 tasks left in task-pool. [ Info: running task 71 on process 35 (Schedulers-primary-S6HJ4WCf); 10 workers total; 34 tasks left in task-pool. [ Info: running task 72 on process 42 (Schedulers-primary-S6HJ4WCf); 10 workers total; 33 tasks left in task-pool. [ Info: running task 73 on process 44 (Schedulers-primary-S6HJ4WCf); 10 workers total; 32 tasks left in task-pool. [ Info: running task 74 on process 38 (Schedulers-primary-S6HJ4WCf); 10 workers total; 31 tasks left in task-pool. [ Info: running task 75 on process 40 (Schedulers-primary-S6HJ4WCf); 10 workers total; 30 tasks left in task-pool. [ Info: running task 76 on process 36 (Schedulers-primary-S6HJ4WCf); 10 workers total; 29 tasks left in task-pool. [ Info: running task 77 on process 37 (Schedulers-primary-S6HJ4WCf); 10 workers total; 25 tasks left in task-pool. [ Info: running task 78 on process 41 (Schedulers-primary-S6HJ4WCf); 10 workers total; 25 tasks left in task-pool. [ Info: running task 79 on process 43 (Schedulers-primary-S6HJ4WCf); 10 workers total; 25 tasks left in task-pool. [ Info: running task 80 on process 39 (Schedulers-primary-S6HJ4WCf); 10 workers total; 25 tasks left in task-pool. [ Info: running task 81 on process 35 (Schedulers-primary-S6HJ4WCf); 10 workers total; 24 tasks left in task-pool. [ Info: running task 82 on process 42 (Schedulers-primary-S6HJ4WCf); 10 workers total; 23 tasks left in task-pool. [ Info: running task 83 on process 44 (Schedulers-primary-S6HJ4WCf); 10 workers total; 22 tasks left in task-pool. [ Info: running task 84 on process 38 (Schedulers-primary-S6HJ4WCf); 10 workers total; 21 tasks left in task-pool. [ Info: running task 85 on process 40 (Schedulers-primary-S6HJ4WCf); 10 workers total; 20 tasks left in task-pool. [ Info: running task 86 on process 36 (Schedulers-primary-S6HJ4WCf); 10 workers total; 19 tasks left in task-pool. [ Info: running task 87 on process 37 (Schedulers-primary-S6HJ4WCf); 10 workers total; 15 tasks left in task-pool. [ Info: running task 88 on process 41 (Schedulers-primary-S6HJ4WCf); 10 workers total; 15 tasks left in task-pool. [ Info: running task 89 on process 43 (Schedulers-primary-S6HJ4WCf); 10 workers total; 15 tasks left in task-pool. [ Info: running task 90 on process 39 (Schedulers-primary-S6HJ4WCf); 10 workers total; 15 tasks left in task-pool. [ Info: running task 91 on process 35 (Schedulers-primary-S6HJ4WCf); 10 workers total; 14 tasks left in task-pool. [ Info: running task 92 on process 42 (Schedulers-primary-S6HJ4WCf); 10 workers total; 13 tasks left in task-pool. [ Info: running task 93 on process 44 (Schedulers-primary-S6HJ4WCf); 10 workers total; 12 tasks left in task-pool. [ Info: running task 94 on process 38 (Schedulers-primary-S6HJ4WCf); 10 workers total; 11 tasks left in task-pool. [ Info: running task 95 on process 40 (Schedulers-primary-S6HJ4WCf); 10 workers total; 10 tasks left in task-pool. [ Info: running task 96 on process 36 (Schedulers-primary-S6HJ4WCf); 10 workers total; 9 tasks left in task-pool. [ Info: running task 97 on process 37 (Schedulers-primary-S6HJ4WCf); 10 workers total; 5 tasks left in task-pool. [ Info: running task 98 on process 41 (Schedulers-primary-S6HJ4WCf); 10 workers total; 5 tasks left in task-pool. [ Info: running task 99 on process 43 (Schedulers-primary-S6HJ4WCf); 10 workers total; 5 tasks left in task-pool. [ Info: running task 100 on process 39 (Schedulers-primary-S6HJ4WCf); 10 workers total; 5 tasks left in task-pool. [ Info: running task 101 on process 35 (Schedulers-primary-S6HJ4WCf); 10 workers total; 4 tasks left in task-pool. [ Info: running task 102 on process 42 (Schedulers-primary-S6HJ4WCf); 10 workers total; 3 tasks left in task-pool. [ Info: running task 103 on process 44 (Schedulers-primary-S6HJ4WCf); 10 workers total; 2 tasks left in task-pool. [ Info: running task 104 on process 38 (Schedulers-primary-S6HJ4WCf); 10 workers total; 1 tasks left in task-pool. [ Info: running task 105 on process 40 (Schedulers-primary-S6HJ4WCf); 10 workers total; 0 tasks left in task-pool. Test Summary: | Pass Total Time pmap with shrinking cluster | 116 116 5m09.2s [ Info: running task 1 on process 45 (Schedulers-primary-S6HJ4WCf); 5 workers total; 99 tasks left in task-pool. [ Info: running task 2 on process 46 (Schedulers-primary-S6HJ4WCf); 5 workers total; 98 tasks left in task-pool. [ Info: running task 3 on process 46 (Schedulers-primary-S6HJ4WCf); 5 workers total; 97 tasks left in task-pool. [ Info: running task 4 on process 45 (Schedulers-primary-S6HJ4WCf); 5 workers total; 96 tasks left in task-pool. [ Info: running task 5 on process 46 (Schedulers-primary-S6HJ4WCf); 5 workers total; 95 tasks left in task-pool. [ Info: running task 6 on process 45 (Schedulers-primary-S6HJ4WCf); 5 workers total; 94 tasks left in task-pool. [ Info: running task 7 on process 46 (Schedulers-primary-S6HJ4WCf); 5 workers total; 93 tasks left in task-pool. [ Info: running task 8 on process 45 (Schedulers-primary-S6HJ4WCf); 5 workers total; 92 tasks left in task-pool. [ Info: running task 9 on process 45 (Schedulers-primary-S6HJ4WCf); 5 workers total; 91 tasks left in task-pool. [ Info: running task 10 on process 46 (Schedulers-primary-S6HJ4WCf); 5 workers total; 90 tasks left in task-pool. [ Info: running task 11 on process 45 (Schedulers-primary-S6HJ4WCf); 10 workers total; 89 tasks left in task-pool. [ Info: running task 12 on process 46 (Schedulers-primary-S6HJ4WCf); 10 workers total; 88 tasks left in task-pool. [ Info: running task 13 on process 46 (Schedulers-primary-S6HJ4WCf); 10 workers total; 87 tasks left in task-pool. [ Info: running task 14 on process 45 (Schedulers-primary-S6HJ4WCf); 10 workers total; 86 tasks left in task-pool. [ Info: running task 15 on process 47 (Schedulers-primary-S6HJ4WCf); 10 workers total; 85 tasks left in task-pool. [ Info: running task 16 on process 48 (Schedulers-primary-S6HJ4WCf); 10 workers total; 83 tasks left in task-pool. [ Info: running task 17 on process 49 (Schedulers-primary-S6HJ4WCf); 10 workers total; 83 tasks left in task-pool. [ Info: running task 18 on process 46 (Schedulers-primary-S6HJ4WCf); 10 workers total; 82 tasks left in task-pool. [ Info: running task 19 on process 45 (Schedulers-primary-S6HJ4WCf); 10 workers total; 81 tasks left in task-pool. [ Info: running task 20 on process 46 (Schedulers-primary-S6HJ4WCf); 10 workers total; 80 tasks left in task-pool. [ Info: running task 21 on process 45 (Schedulers-primary-S6HJ4WCf); 10 workers total; 79 tasks left in task-pool. [ Info: running task 22 on process 47 (Schedulers-primary-S6HJ4WCf); 10 workers total; 78 tasks left in task-pool. [ Info: running task 23 on process 48 (Schedulers-primary-S6HJ4WCf); 10 workers total; 76 tasks left in task-pool. [ Info: running task 24 on process 49 (Schedulers-primary-S6HJ4WCf); 10 workers total; 76 tasks left in task-pool. ┌ Warning: addprocs/rmprocs taking longer than expected, cancelling. └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:776 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 25 on process 45 (Schedulers-primary-S6HJ4WCf); 10 workers total; 75 tasks left in task-pool. [ Info: running task 26 on process 46 (Schedulers-primary-S6HJ4WCf); 10 workers total; 71 tasks left in task-pool. [ Info: running task 27 on process 47 (Schedulers-primary-S6HJ4WCf); 10 workers total; 71 tasks left in task-pool. [ Info: running task 28 on process 48 (Schedulers-primary-S6HJ4WCf); 10 workers total; 71 tasks left in task-pool. [ Info: running task 29 on process 49 (Schedulers-primary-S6HJ4WCf); 10 workers total; 71 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 30 on process 45 (Schedulers-primary-S6HJ4WCf); 10 workers total; 70 tasks left in task-pool. [ Info: running task 31 on process 46 (Schedulers-primary-S6HJ4WCf); 10 workers total; 67 tasks left in task-pool. [ Info: running task 32 on process 47 (Schedulers-primary-S6HJ4WCf); 10 workers total; 67 tasks left in task-pool. [ Info: running task 33 on process 48 (Schedulers-primary-S6HJ4WCf); 10 workers total; 67 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 34 on process 49 (Schedulers-primary-S6HJ4WCf); 10 workers total; 66 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 35 on process 45 (Schedulers-primary-S6HJ4WCf); 10 workers total; 65 tasks left in task-pool. [ Info: running task 36 on process 46 (Schedulers-primary-S6HJ4WCf); 10 workers total; 63 tasks left in task-pool. [ Info: running task 37 on process 47 (Schedulers-primary-S6HJ4WCf); 10 workers total; 63 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 38 on process 48 (Schedulers-primary-S6HJ4WCf); 10 workers total; 62 tasks left in task-pool. [ Info: running task 39 on process 49 (Schedulers-primary-S6HJ4WCf); 10 workers total; 61 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 40 on process 45 (Schedulers-primary-S6HJ4WCf); 10 workers total; 60 tasks left in task-pool. [ Info: running task 41 on process 46 (Schedulers-primary-S6HJ4WCf); 10 workers total; 58 tasks left in task-pool. [ Info: running task 42 on process 47 (Schedulers-primary-S6HJ4WCf); 10 workers total; 58 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 43 on process 50 (Schedulers-primary-S6HJ4WCf); 10 workers total; 57 tasks left in task-pool. [ Info: running task 44 on process 51 (Schedulers-primary-S6HJ4WCf); 10 workers total; 53 tasks left in task-pool. [ Info: running task 45 on process 52 (Schedulers-primary-S6HJ4WCf); 10 workers total; 53 tasks left in task-pool. [ Info: running task 46 on process 53 (Schedulers-primary-S6HJ4WCf); 10 workers total; 53 tasks left in task-pool. [ Info: running task 47 on process 54 (Schedulers-primary-S6HJ4WCf); 10 workers total; 53 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 48 on process 48 (Schedulers-primary-S6HJ4WCf); 10 workers total; 52 tasks left in task-pool. [ Info: running task 49 on process 49 (Schedulers-primary-S6HJ4WCf); 10 workers total; 51 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 50 on process 45 (Schedulers-primary-S6HJ4WCf); 10 workers total; 50 tasks left in task-pool. [ Info: running task 51 on process 46 (Schedulers-primary-S6HJ4WCf); 10 workers total; 48 tasks left in task-pool. [ Info: running task 52 on process 47 (Schedulers-primary-S6HJ4WCf); 10 workers total; 48 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 53 on process 50 (Schedulers-primary-S6HJ4WCf); 10 workers total; 47 tasks left in task-pool. [ Info: running task 54 on process 51 (Schedulers-primary-S6HJ4WCf); 10 workers total; 41 tasks left in task-pool. [ Info: running task 55 on process 52 (Schedulers-primary-S6HJ4WCf); 10 workers total; 41 tasks left in task-pool. [ Info: running task 56 on process 53 (Schedulers-primary-S6HJ4WCf); 10 workers total; 41 tasks left in task-pool. [ Info: running task 57 on process 54 (Schedulers-primary-S6HJ4WCf); 10 workers total; 41 tasks left in task-pool. [ Info: running task 58 on process 48 (Schedulers-primary-S6HJ4WCf); 10 workers total; 41 tasks left in task-pool. [ Info: running task 59 on process 49 (Schedulers-primary-S6HJ4WCf); 10 workers total; 41 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 60 on process 45 (Schedulers-primary-S6HJ4WCf); 10 workers total; 40 tasks left in task-pool. [ Info: running task 61 on process 46 (Schedulers-primary-S6HJ4WCf); 10 workers total; 38 tasks left in task-pool. [ Info: running task 62 on process 47 (Schedulers-primary-S6HJ4WCf); 10 workers total; 38 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 63 on process 52 (Schedulers-primary-S6HJ4WCf); 10 workers total; 37 tasks left in task-pool. [ Info: running task 64 on process 53 (Schedulers-primary-S6HJ4WCf); 10 workers total; 34 tasks left in task-pool. [ Info: running task 65 on process 48 (Schedulers-primary-S6HJ4WCf); 10 workers total; 34 tasks left in task-pool. [ Info: running task 66 on process 49 (Schedulers-primary-S6HJ4WCf); 10 workers total; 34 tasks left in task-pool. [ Info: running task 67 on process 50 (Schedulers-primary-S6HJ4WCf); 10 workers total; 33 tasks left in task-pool. [ Info: running task 68 on process 51 (Schedulers-primary-S6HJ4WCf); 10 workers total; 31 tasks left in task-pool. [ Info: running task 69 on process 54 (Schedulers-primary-S6HJ4WCf); 10 workers total; 31 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 70 on process 45 (Schedulers-primary-S6HJ4WCf); 10 workers total; 30 tasks left in task-pool. [ Info: running task 71 on process 47 (Schedulers-primary-S6HJ4WCf); 10 workers total; 29 tasks left in task-pool. [ Info: running task 72 on process 46 (Schedulers-primary-S6HJ4WCf); 10 workers total; 28 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 73 on process 52 (Schedulers-primary-S6HJ4WCf); 10 workers total; 27 tasks left in task-pool. [ Info: running task 74 on process 53 (Schedulers-primary-S6HJ4WCf); 10 workers total; 24 tasks left in task-pool. [ Info: running task 75 on process 48 (Schedulers-primary-S6HJ4WCf); 10 workers total; 24 tasks left in task-pool. [ Info: running task 76 on process 49 (Schedulers-primary-S6HJ4WCf); 10 workers total; 24 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 77 on process 50 (Schedulers-primary-S6HJ4WCf); 10 workers total; 23 tasks left in task-pool. [ Info: running task 78 on process 51 (Schedulers-primary-S6HJ4WCf); 10 workers total; 21 tasks left in task-pool. [ Info: running task 79 on process 54 (Schedulers-primary-S6HJ4WCf); 10 workers total; 21 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 80 on process 45 (Schedulers-primary-S6HJ4WCf); 10 workers total; 20 tasks left in task-pool. [ Info: running task 81 on process 47 (Schedulers-primary-S6HJ4WCf); 10 workers total; 19 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 82 on process 46 (Schedulers-primary-S6HJ4WCf); 10 workers total; 18 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 83 on process 52 (Schedulers-primary-S6HJ4WCf); 10 workers total; 17 tasks left in task-pool. [ Info: running task 84 on process 53 (Schedulers-primary-S6HJ4WCf); 10 workers total; 15 tasks left in task-pool. [ Info: running task 85 on process 48 (Schedulers-primary-S6HJ4WCf); 10 workers total; 15 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 86 on process 49 (Schedulers-primary-S6HJ4WCf); 10 workers total; 14 tasks left in task-pool. [ Info: running task 87 on process 50 (Schedulers-primary-S6HJ4WCf); 10 workers total; 13 tasks left in task-pool. [ Info: running task 88 on process 51 (Schedulers-primary-S6HJ4WCf); 10 workers total; 11 tasks left in task-pool. [ Info: running task 89 on process 54 (Schedulers-primary-S6HJ4WCf); 10 workers total; 11 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 90 on process 45 (Schedulers-primary-S6HJ4WCf); 10 workers total; 10 tasks left in task-pool. [ Info: running task 91 on process 47 (Schedulers-primary-S6HJ4WCf); 10 workers total; 9 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 92 on process 46 (Schedulers-primary-S6HJ4WCf); 10 workers total; 8 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 93 on process 52 (Schedulers-primary-S6HJ4WCf); 10 workers total; 7 tasks left in task-pool. [ Info: running task 94 on process 53 (Schedulers-primary-S6HJ4WCf); 10 workers total; 5 tasks left in task-pool. [ Info: running task 95 on process 48 (Schedulers-primary-S6HJ4WCf); 10 workers total; 5 tasks left in task-pool. [ Info: running task 96 on process 49 (Schedulers-primary-S6HJ4WCf); 10 workers total; 4 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 97 on process 50 (Schedulers-primary-S6HJ4WCf); 10 workers total; 3 tasks left in task-pool. [ Info: running task 98 on process 51 (Schedulers-primary-S6HJ4WCf); 10 workers total; 1 tasks left in task-pool. [ Info: running task 99 on process 54 (Schedulers-primary-S6HJ4WCf); 10 workers total; 1 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 100 on process 45 (Schedulers-primary-S6HJ4WCf); 10 workers total; 0 tasks left in task-pool. ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 [inlined] │ [7] (::var"#11#12"{SchedulerOptions{String}, String})() │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:166 │ │ nested task error: InterruptException: │ Stacktrace: │ [1] try_yieldto(undo::typeof(identity)) │ @ Base ./task.jl:1128 │ [2] yieldto(t::Task, x::Any) │ @ Base ./task.jl:1109 │ [3] yieldto │ @ ./task.jl:1095 [inlined] │ [4] wait() │ @ Base ./task.jl:1211 │ [5] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool) │ @ Base ./condition.jl:141 │ [6] wait │ @ ./condition.jl:136 [inlined] │ [7] _wait(t::Task) │ @ Base ./task.jl:312 │ [8] sync_end(c::Channel{Any}) │ @ Base ./task.jl:560 │ [9] macro expansion │ @ ./task.jl:634 [inlined] │ [10] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [11] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [12] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [13] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [14] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [15] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [16] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", var"#9#10", var"#7#8", Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 Test Summary: | Pass Total Time pmap with interactive growing cluster | 111 111 4m40.0s [ Info: nworkers()=1, i=1 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: MethodError: no method matching addprocs(::Int64, ::String) │ The function `addprocs` exists, but no method is defined for this combination of argument types. │ │ Closest candidates are: │ addprocs(::Integer; restrict, kwargs...) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ addprocs(; ...) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ addprocs(!Matched::ClusterManager; kwargs...) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 │ ... │ │ Stacktrace: │ [1] (::var"#myaddprocs#myaddprocs##0")(n::Int64) │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:208 │ [2] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{var"#myaddprocs#myaddprocs##0", Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#94#95"{Int64}, Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{var"#myaddprocs#myaddprocs##0", Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#94#95"{Int64}, Int64, Nothing}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1046 │ [6] epmap(options::SchedulerOptions{String}, f::Function, tasks::UnitRange{Int64}, args::String) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1041 │ [7] top-level scope │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:195 │ [8] macro expansion │ @ /opt/julia/share/julia/stdlib/v1.13/Test/src/Test.jl:1855 [inlined] │ [9] macro expansion │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:218 [inlined] │ [10] include(mapexpr::Function, mod::Module, _path::String) │ @ Base ./Base.jl:310 │ [11] top-level scope │ @ none:6 │ [12] eval(m::Module, e::Any) │ @ Core ./boot.jl:489 │ [13] exec_options(opts::Base.JLOptions) │ @ Base ./client.jl:290 │ [14] _start() │ @ Base ./client.jl:557 │ │ nested task error: MethodError: no method matching addprocs(::Int64, ::String) │ The function `addprocs` exists, but no method is defined for this combination of argument types. │ │ Closest candidates are: │ addprocs(::Integer; restrict, kwargs...) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ addprocs(; ...) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ addprocs(!Matched::ClusterManager; kwargs...) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 │ ... │ │ Stacktrace: │ [1] (::var"#myaddprocs#myaddprocs##0")(n::Int64) │ @ Main ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:208 │ [2] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{var"#myaddprocs#myaddprocs##0", Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#94#95"{Int64}, Int64, Nothing}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: nworkers()=1, i=2 [ Info: nworkers()=1, i=3 [ Info: running task 1 on process 55 (Schedulers-primary-S6HJ4WCf); 2 workers total; 99 tasks left in task-pool. [ Info: running task 2 on process 55 (Schedulers-primary-S6HJ4WCf); 2 workers total; 98 tasks left in task-pool. [ Info: running task 3 on process 55 (Schedulers-primary-S6HJ4WCf); 2 workers total; 97 tasks left in task-pool. [ Info: running task 4 on process 55 (Schedulers-primary-S6HJ4WCf); 2 workers total; 96 tasks left in task-pool. [ Info: nworkers()=2, i=4 [ Info: running task 5 on process 55 (Schedulers-primary-S6HJ4WCf); 2 workers total; 95 tasks left in task-pool. [ Info: running task 6 on process 55 (Schedulers-primary-S6HJ4WCf); 2 workers total; 94 tasks left in task-pool. [ Info: running task 7 on process 55 (Schedulers-primary-S6HJ4WCf); 2 workers total; 93 tasks left in task-pool. [ Info: running task 8 on process 55 (Schedulers-primary-S6HJ4WCf); 2 workers total; 92 tasks left in task-pool. [ Info: running task 9 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 91 tasks left in task-pool. [ Info: running task 10 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 90 tasks left in task-pool. [ Info: running task 11 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 89 tasks left in task-pool. [ Info: running task 12 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 88 tasks left in task-pool. [ Info: running task 13 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 87 tasks left in task-pool. [ Info: running task 14 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 86 tasks left in task-pool. [ Info: running task 15 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 85 tasks left in task-pool. [ Info: running task 16 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 84 tasks left in task-pool. [ Info: running task 17 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 83 tasks left in task-pool. [ Info: nworkers()=3, i=5 [ Info: running task 18 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 82 tasks left in task-pool. [ Info: running task 19 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 81 tasks left in task-pool. [ Info: running task 20 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 80 tasks left in task-pool. [ Info: running task 21 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 79 tasks left in task-pool. [ Info: running task 22 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 78 tasks left in task-pool. [ Info: running task 23 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 77 tasks left in task-pool. [ Info: running task 24 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 76 tasks left in task-pool. [ Info: running task 25 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 75 tasks left in task-pool. [ Info: running task 26 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 74 tasks left in task-pool. [ Info: running task 27 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 73 tasks left in task-pool. [ Info: running task 28 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 72 tasks left in task-pool. [ Info: running task 29 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 71 tasks left in task-pool. [ Info: running task 30 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 70 tasks left in task-pool. [ Info: running task 31 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 69 tasks left in task-pool. [ Info: running task 32 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 68 tasks left in task-pool. [ Info: running task 33 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 67 tasks left in task-pool. [ Info: running task 34 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 66 tasks left in task-pool. [ Info: running task 35 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 65 tasks left in task-pool. [ Info: running task 36 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 64 tasks left in task-pool. [ Info: running task 37 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 63 tasks left in task-pool. [ Info: running task 38 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 62 tasks left in task-pool. [ Info: running task 39 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 61 tasks left in task-pool. [ Info: running task 40 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 60 tasks left in task-pool. [ Info: running task 41 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 59 tasks left in task-pool. [ Info: running task 42 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 58 tasks left in task-pool. [ Info: running task 43 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 57 tasks left in task-pool. [ Info: running task 44 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 56 tasks left in task-pool. [ Info: running task 45 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 55 tasks left in task-pool. [ Info: running task 46 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 54 tasks left in task-pool. [ Info: running task 47 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 53 tasks left in task-pool. [ Info: running task 48 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 52 tasks left in task-pool. [ Info: running task 49 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 51 tasks left in task-pool. [ Info: running task 50 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 50 tasks left in task-pool. [ Info: running task 51 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 49 tasks left in task-pool. [ Info: running task 52 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 48 tasks left in task-pool. [ Info: running task 53 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 47 tasks left in task-pool. [ Info: running task 54 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 46 tasks left in task-pool. [ Info: running task 55 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 45 tasks left in task-pool. [ Info: running task 56 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 44 tasks left in task-pool. [ Info: running task 57 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 43 tasks left in task-pool. [ Info: running task 58 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 42 tasks left in task-pool. [ Info: running task 59 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 41 tasks left in task-pool. [ Info: running task 60 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 40 tasks left in task-pool. [ Info: running task 61 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 39 tasks left in task-pool. [ Info: running task 62 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 38 tasks left in task-pool. [ Info: running task 63 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 37 tasks left in task-pool. [ Info: running task 64 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 36 tasks left in task-pool. [ Info: running task 65 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 35 tasks left in task-pool. [ Info: running task 66 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 34 tasks left in task-pool. [ Info: running task 67 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 33 tasks left in task-pool. [ Info: running task 68 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 32 tasks left in task-pool. [ Info: running task 69 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 31 tasks left in task-pool. [ Info: running task 70 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 30 tasks left in task-pool. [ Info: running task 71 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 29 tasks left in task-pool. [ Info: running task 72 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 28 tasks left in task-pool. [ Info: running task 73 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 27 tasks left in task-pool. [ Info: running task 74 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 26 tasks left in task-pool. [ Info: running task 75 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 25 tasks left in task-pool. [ Info: running task 76 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 24 tasks left in task-pool. [ Info: running task 77 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 23 tasks left in task-pool. [ Info: running task 78 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 22 tasks left in task-pool. [ Info: running task 79 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 21 tasks left in task-pool. [ Info: running task 80 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 20 tasks left in task-pool. [ Info: running task 81 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 19 tasks left in task-pool. [ Info: running task 82 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 18 tasks left in task-pool. [ Info: running task 83 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 17 tasks left in task-pool. [ Info: running task 84 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 16 tasks left in task-pool. [ Info: running task 85 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 15 tasks left in task-pool. [ Info: running task 86 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 14 tasks left in task-pool. [ Info: running task 87 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 13 tasks left in task-pool. [ Info: running task 88 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 12 tasks left in task-pool. [ Info: running task 89 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 11 tasks left in task-pool. [ Info: running task 90 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 10 tasks left in task-pool. [ Info: running task 91 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 9 tasks left in task-pool. [ Info: running task 92 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 8 tasks left in task-pool. [ Info: running task 93 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 7 tasks left in task-pool. [ Info: running task 94 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 6 tasks left in task-pool. [ Info: running task 95 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 5 tasks left in task-pool. [ Info: running task 96 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 4 tasks left in task-pool. [ Info: running task 97 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 3 tasks left in task-pool. [ Info: running task 98 on process 55 (Schedulers-primary-S6HJ4WCf); 3 workers total; 2 tasks left in task-pool. [ Info: running task 99 on process 56 (Schedulers-primary-S6HJ4WCf); 3 workers total; 1 tasks left in task-pool. [ Info: running task 100 on process 57 (Schedulers-primary-S6HJ4WCf); 3 workers total; 0 tasks left in task-pool. ┌ Warning: addprocs/rmprocs taking longer than expected, cancelling. └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:823 Test Summary: | Pass Total Time pmap with blocking addprocs | 103 103 2m13.3s [ Info: running task 1 on process 58 (Schedulers-primary-S6HJ4WCf); 5 workers total; 19 tasks left in task-pool. [ Info: running task 2 on process 59 (Schedulers-primary-S6HJ4WCf); 5 workers total; 18 tasks left in task-pool. [ Info: running task 3 on process 60 (Schedulers-primary-S6HJ4WCf); 5 workers total; 17 tasks left in task-pool. [ Info: running task 4 on process 61 (Schedulers-primary-S6HJ4WCf); 5 workers total; 15 tasks left in task-pool. [ Info: running task 5 on process 62 (Schedulers-primary-S6HJ4WCf); 5 workers total; 15 tasks left in task-pool. [ Info: running task 6 on process 58 (Schedulers-primary-S6HJ4WCf); 5 workers total; 14 tasks left in task-pool. [ Info: running task 7 on process 59 (Schedulers-primary-S6HJ4WCf); 5 workers total; 13 tasks left in task-pool. [ Info: running task 8 on process 60 (Schedulers-primary-S6HJ4WCf); 5 workers total; 11 tasks left in task-pool. [ Info: running task 9 on process 61 (Schedulers-primary-S6HJ4WCf); 5 workers total; 11 tasks left in task-pool. [ Info: running task 10 on process 62 (Schedulers-primary-S6HJ4WCf); 5 workers total; 10 tasks left in task-pool. [ Info: running task 11 on process 58 (Schedulers-primary-S6HJ4WCf); 5 workers total; 9 tasks left in task-pool. [ Info: running task 12 on process 59 (Schedulers-primary-S6HJ4WCf); 5 workers total; 8 tasks left in task-pool. [ Info: running task 13 on process 60 (Schedulers-primary-S6HJ4WCf); 5 workers total; 7 tasks left in task-pool. [ Info: running task 14 on process 61 (Schedulers-primary-S6HJ4WCf); 5 workers total; 6 tasks left in task-pool. [ Info: running task 15 on process 62 (Schedulers-primary-S6HJ4WCf); 5 workers total; 5 tasks left in task-pool. [ Info: running task 16 on process 58 (Schedulers-primary-S6HJ4WCf); 5 workers total; 4 tasks left in task-pool. [ Info: running task 17 on process 59 (Schedulers-primary-S6HJ4WCf); 5 workers total; 3 tasks left in task-pool. [ Info: running task 18 on process 60 (Schedulers-primary-S6HJ4WCf); 5 workers total; 2 tasks left in task-pool. [ Info: running task 19 on process 61 (Schedulers-primary-S6HJ4WCf); 5 workers total; 1 tasks left in task-pool. [ Info: running task 20 on process 62 (Schedulers-primary-S6HJ4WCf); 5 workers total; 0 tasks left in task-pool. ┌ Warning: caught an exception, there have been 0 failure(s) on process 62 (Schedulers-primary-S6HJ4WCf)... └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1111 ┌ Warning: Schedulers.TimeoutException(62, 17.03245997428894) │ error type: Schedulers.TimeoutException │ 1-element ExceptionStack: │ Schedulers.TimeoutException(62, 17.03245997428894) │ Stacktrace: │ [1] remotecall_func_wait_timeout(::Vector{Float64}, ::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, ::SchedulerOptions{String}, ::Nothing, ::Function, ::Function, ::Int64, ::var"#foo5c#foo5c##0", ::Int64, ::Int64, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:343 │ [2] remotecall_func_wait_timeout(::Vector{Float64}, ::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, ::SchedulerOptions{String}, ::Nothing, ::Function, ::Function, ::Int64, ::Function, ::Int64, ::Int64, ::Vararg{Any}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:338 │ [3] (::Schedulers.var"#107#108"{@Kwargs{}, SchedulerOptions{String}, var"#foo5c#foo5c##0", Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, Dict{String, Any}, Tuple{String}, Vector{Float64}, Int64})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1105 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 20 on process 58 (Schedulers-primary-S6HJ4WCf); 5 workers total; 0 tasks left in task-pool. Test Summary: | Pass Total Time pmap with timeout | 21 21 2m04.3s [ Info: running task 1 on process 58 (Schedulers-primary-S6HJ4WCf); 5 workers total; 19 tasks left in task-pool. [ Info: running task 2 on process 59 (Schedulers-primary-S6HJ4WCf); 5 workers total; 16 tasks left in task-pool. [ Info: running task 3 on process 60 (Schedulers-primary-S6HJ4WCf); 5 workers total; 16 tasks left in task-pool. [ Info: running task 4 on process 61 (Schedulers-primary-S6HJ4WCf); 5 workers total; 16 tasks left in task-pool. [ Info: running task 5 on process 58 (Schedulers-primary-S6HJ4WCf); 5 workers total; 15 tasks left in task-pool. [ Info: running task 6 on process 59 (Schedulers-primary-S6HJ4WCf); 5 workers total; 14 tasks left in task-pool. [ Info: running task 7 on process 60 (Schedulers-primary-S6HJ4WCf); 5 workers total; 13 tasks left in task-pool. [ Info: running task 8 on process 61 (Schedulers-primary-S6HJ4WCf); 5 workers total; 12 tasks left in task-pool. [ Info: running task 9 on process 58 (Schedulers-primary-S6HJ4WCf); 5 workers total; 11 tasks left in task-pool. [ Info: running task 10 on process 59 (Schedulers-primary-S6HJ4WCf); 5 workers total; 10 tasks left in task-pool. [ Info: running task 11 on process 60 (Schedulers-primary-S6HJ4WCf); 5 workers total; 9 tasks left in task-pool. [ Info: running task 12 on process 61 (Schedulers-primary-S6HJ4WCf); 5 workers total; 8 tasks left in task-pool. [ Info: running task 13 on process 58 (Schedulers-primary-S6HJ4WCf); 5 workers total; 7 tasks left in task-pool. [ Info: running task 14 on process 59 (Schedulers-primary-S6HJ4WCf); 5 workers total; 6 tasks left in task-pool. [ Info: running task 15 on process 60 (Schedulers-primary-S6HJ4WCf); 5 workers total; 5 tasks left in task-pool. [ Info: running task 16 on process 61 (Schedulers-primary-S6HJ4WCf); 5 workers total; 4 tasks left in task-pool. [ Info: running task 17 on process 58 (Schedulers-primary-S6HJ4WCf); 5 workers total; 3 tasks left in task-pool. [ Info: running task 18 on process 59 (Schedulers-primary-S6HJ4WCf); 5 workers total; 2 tasks left in task-pool. [ Info: running task 19 on process 60 (Schedulers-primary-S6HJ4WCf); 5 workers total; 1 tasks left in task-pool. [ Info: running task 20 on process 61 (Schedulers-primary-S6HJ4WCf); 5 workers total; 0 tasks left in task-pool. ┌ Warning: caught an exception, there have been 0 failure(s) on process 61 (Schedulers-primary-S6HJ4WCf)... └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1111 ┌ Warning: skipping task '20' that timed out └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1114 ┌ Warning: Schedulers.TimeoutException(61, 10.021085977554321) │ error type: Schedulers.TimeoutException │ 1-element ExceptionStack: │ Schedulers.TimeoutException(61, 10.021085977554321) │ Stacktrace: │ [1] remotecall_func_wait_timeout(::Vector{Float64}, ::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, ::SchedulerOptions{String}, ::Nothing, ::Function, ::Function, ::Int64, ::var"#foo5c#foo5c##1", ::Int64, ::Int64, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:343 │ [2] remotecall_func_wait_timeout(::Vector{Float64}, ::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, ::SchedulerOptions{String}, ::Nothing, ::Function, ::Function, ::Int64, ::Function, ::Int64, ::Int64, ::Vararg{Any}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:338 │ [3] (::Schedulers.var"#107#108"{@Kwargs{}, SchedulerOptions{String}, var"#foo5c#foo5c##1", Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, Dict{String, Any}, Tuple{String}, Vector{Float64}, Int64})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1105 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 Test Summary: | Pass Total Time pmap with timeout and skip_tasks_that_timeout=true | 21 21 25.2s ┌ Warning: problem initializing 58, removing 58 from cluster. └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:675 ┌ Warning: no process with id 58 exists │ error type: ErrorException │ 1-element ExceptionStack: │ no process with id 58 exists │ Stacktrace: │ [1] error(s::String) │ @ Base ./error.jl:44 │ [2] worker_from_id(pg::Distributed.ProcessGroup, i::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:1114 │ [3] worker_from_id │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:1106 [inlined] │ [4] remotecall(::Function, ::Int64, ::Type, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:447 │ [5] remotecall │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:447 [inlined] │ [6] init │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:299 [inlined] │ [7] #20 │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:340 [inlined] │ [8] (::Schedulers.var"#loop##12#loop##13"{Schedulers.ElasticLoop{typeof(addprocs), var"#20#21"{var"#init#init##0"{Dict{Int64, Future}}, Dict{Int64, Future}}, Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, Set{Int64}, Set{Int64}, Int64})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:668 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 [ Info: running task 1 on process 61 (Schedulers-primary-S6HJ4WCf); 2 workers total; 3 tasks left in task-pool. [ Info: running task 2 on process 63 (Schedulers-primary-S6HJ4WCf); 2 workers total; 2 tasks left in task-pool. ┌ Warning: problem initializing 60, removing 60 from cluster. └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:675 ┌ Warning: ProcessExitedException(60) │ error type: ProcessExitedException │ 1-element ExceptionStack: │ ProcessExitedException(60) │ Stacktrace: │ [1] worker_from_id(pg::Distributed.ProcessGroup, i::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:1109 │ [2] worker_from_id │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:1106 [inlined] │ [3] remotecall(::Function, ::Int64, ::Type, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:447 │ [4] remotecall │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:447 [inlined] │ [5] init │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:299 [inlined] │ [6] #20 │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:340 [inlined] │ [7] (::Schedulers.var"#loop##12#loop##13"{Schedulers.ElasticLoop{typeof(addprocs), var"#20#21"{var"#init#init##0"{Dict{Int64, Future}}, Dict{Int64, Future}}, Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, Set{Int64}, Set{Int64}, Int64})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:668 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: problem initializing 59, removing 59 from cluster. └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:675 ┌ Warning: ProcessExitedException(59) │ error type: ProcessExitedException │ 1-element ExceptionStack: │ ProcessExitedException(59) │ Stacktrace: │ [1] worker_from_id(pg::Distributed.ProcessGroup, i::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:1109 │ [2] worker_from_id │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:1106 [inlined] │ [3] remotecall(::Function, ::Int64, ::Type, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:447 │ [4] remotecall │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:447 [inlined] │ [5] init │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:299 [inlined] │ [6] #20 │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:340 [inlined] │ [7] (::Schedulers.var"#loop##12#loop##13"{Schedulers.ElasticLoop{typeof(addprocs), var"#20#21"{var"#init#init##0"{Dict{Int64, Future}}, Dict{Int64, Future}}, Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, Set{Int64}, Set{Int64}, Int64})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:668 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 From worker 61: [ Info: restart task... From worker 61: ┌ Info: ...restart task, From worker 61: │ _s = From worker 61: │ 1-element Vector{Int64}: From worker 61: └ 1 From worker 61: [ Info: _s on pid=61 is 1 From worker 63: [ Info: restart task... From worker 63: ┌ Info: ...restart task, From worker 63: │ _s = From worker 63: │ 1-element Vector{Int64}: From worker 63: └ 1 From worker 63: [ Info: _s on pid=63 is 1 From worker 61: [ Info: _s on pid=61 is 2 From worker 63: [ Info: _s on pid=63 is 2 From worker 61: [ Info: checkpoint task... From worker 61: [ Info: ...checkpoint task with state=2. ┌ Warning: caught an exception, there have been 0 failure(s) on process 61 (Schedulers-primary-S6HJ4WCf)... └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1111 ┌ Warning: TaskFailedException │ │ nested task error: On worker 61: │ Schedulers.PreemptException() │ Stacktrace: │ [1] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:278 │ [2] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [3] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [4] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [5] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [6] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Future, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#43#44"{@Kwargs{}, Future, var"#22#23"{var"#checkpoint_task#checkpoint_task##0", Dict{Int64, Future}}, var"#24#25"{var"#restart_task!#restart_task!##0", Dict{Int64, Future}}, Int64, var"#30#31"{var"#foo5d#foo5d##0", String, Dict{Int64, Future}}, Int64, Tuple{Int64}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:339 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] remotecall_func_wait_timeout(tsk_times::Vector{Float64}, eloop::Schedulers.ElasticLoop{typeof(addprocs), var"#20#21"{var"#init#init##0"{Dict{Int64, Future}}, Dict{Int64, Future}}, Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, options::SchedulerOptions{String}, preempt_channel_future::Future, checkpoint_task::Function, restart_task::Function, tsk::Int64, f::var"#30#31"{var"#foo5d#foo5d##0", String, Dict{Int64, Future}}, pid::Int64, args::Int64; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:349 │ [5] remotecall_func_wait_timeout(tsk_times::Vector{Float64}, eloop::Schedulers.ElasticLoop{typeof(addprocs), var"#20#21"{var"#init#init##0"{Dict{Int64, Future}}, Dict{Int64, Future}}, Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, options::SchedulerOptions{String}, preempt_channel_future::Future, checkpoint_task::Function, restart_task::Function, tsk::Int64, f::Function, pid::Int64, args::Int64) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:338 │ [6] (::Schedulers.var"#107#108"{@Kwargs{}, SchedulerOptions{String}, var"#30#31"{var"#foo5d#foo5d##0", String, Dict{Int64, Future}}, Schedulers.ElasticLoop{typeof(addprocs), var"#20#21"{var"#init#init##0"{Dict{Int64, Future}}, Dict{Int64, Future}}, Schedulers.var"#90#91"{Int64}, Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, Nothing}, Dict{String, Any}, Tuple{}, Vector{Float64}, Int64})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1105 │ │ nested task error: On worker 61: │ Schedulers.PreemptException() │ Stacktrace: │ [1] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:278 │ [2] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [3] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [4] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [5] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [6] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Future, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#43#44"{@Kwargs{}, Future, var"#22#23"{var"#checkpoint_task#checkpoint_task##0", Dict{Int64, Future}}, var"#24#25"{var"#restart_task!#restart_task!##0", Dict{Int64, Future}}, Int64, var"#30#31"{var"#foo5d#foo5d##0", String, Dict{Int64, Future}}, Int64, Tuple{Int64}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:339 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: too many failures on process with id=61 (Schedulers-primary-S6HJ4WCf), removing from process list └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:393 From worker 63: [ Info: _s on pid=63 is 3 From worker 63: [ Info: _s on pid=63 is 4 From worker 63: [ Info: _s on pid=63 is 5 From worker 63: [ Info: _s on pid=63 is 6 [ Info: running task 3 on process 64 (Schedulers-primary-S6HJ4WCf); 2 workers total; 2 tasks left in task-pool. From worker 63: [ Info: _s on pid=63 is 7 From worker 64: [ Info: restart task... From worker 64: ┌ Info: ...restart task, From worker 64: │ _s = From worker 64: │ 1-element Vector{Int64}: From worker 64: └ 1 From worker 64: [ Info: _s on pid=64 is 1 From worker 63: [ Info: _s on pid=63 is 8 From worker 64: [ Info: _s on pid=64 is 2 From worker 63: [ Info: _s on pid=63 is 9 From worker 64: [ Info: _s on pid=64 is 3 From worker 63: [ Info: _s on pid=63 is 10 From worker 64: [ Info: _s on pid=64 is 4 [ Info: running task 4 on process 63 (Schedulers-primary-S6HJ4WCf); 2 workers total; 1 tasks left in task-pool. From worker 63: [ Info: restart task... From worker 63: ┌ Info: ...restart task, From worker 63: │ _s = From worker 63: │ 1-element Vector{Int64}: From worker 63: └ 1 From worker 63: [ Info: _s on pid=63 is 1 From worker 64: [ Info: _s on pid=64 is 5 From worker 63: [ Info: _s on pid=63 is 2 From worker 64: [ Info: _s on pid=64 is 6 From worker 63: [ Info: _s on pid=63 is 3 From worker 64: [ Info: _s on pid=64 is 7 From worker 63: [ Info: _s on pid=63 is 4 From worker 64: [ Info: _s on pid=64 is 8 From worker 63: [ Info: _s on pid=63 is 5 From worker 64: [ Info: _s on pid=64 is 9 From worker 63: [ Info: _s on pid=63 is 6 From worker 64: [ Info: _s on pid=64 is 10 From worker 63: [ Info: _s on pid=63 is 7 [ Info: running task 1 on process 64 (Schedulers-primary-S6HJ4WCf); 2 workers total; 0 tasks left in task-pool. From worker 64: [ Info: restart task... From worker 64: ┌ Info: ...restart task, From worker 64: │ _s = From worker 64: │ 1-element Vector{Int64}: From worker 64: └ 2 From worker 64: [ Info: _s on pid=64 is 2 From worker 63: [ Info: _s on pid=63 is 8 From worker 64: [ Info: _s on pid=64 is 3 From worker 63: [ Info: _s on pid=63 is 9 From worker 64: [ Info: _s on pid=64 is 4 From worker 63: [ Info: _s on pid=63 is 10 From worker 64: [ Info: _s on pid=64 is 5 From worker 64: [ Info: _s on pid=64 is 6 From worker 64: [ Info: _s on pid=64 is 7 From worker 64: [ Info: _s on pid=64 is 8 From worker 64: [ Info: _s on pid=64 is 9 From worker 64: [ Info: _s on pid=64 is 10 Test Summary: | Pass Total Time pmap with task checkpoint and restart | 1 1 2m27.9s [ Info: running task 1 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 99 tasks left in task-pool. [ Info: running task 2 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 98 tasks left in task-pool. [ Info: running task 3 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 97 tasks left in task-pool. [ Info: running task 4 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 95 tasks left in task-pool. [ Info: running task 5 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 95 tasks left in task-pool. [ Info: running task 6 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 94 tasks left in task-pool. [ Info: running task 7 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 93 tasks left in task-pool. [ Info: running task 8 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 92 tasks left in task-pool. [ Info: running task 9 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 90 tasks left in task-pool. [ Info: running task 10 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 90 tasks left in task-pool. [ Info: running task 11 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 89 tasks left in task-pool. [ Info: running task 12 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 88 tasks left in task-pool. [ Info: running task 13 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 87 tasks left in task-pool. [ Info: running task 14 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 86 tasks left in task-pool. [ Info: running task 15 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 85 tasks left in task-pool. [ Info: running task 16 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 84 tasks left in task-pool. [ Info: running task 17 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 83 tasks left in task-pool. [ Info: running task 18 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 82 tasks left in task-pool. [ Info: running task 19 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 81 tasks left in task-pool. [ Info: running task 20 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 80 tasks left in task-pool. [ Info: running task 21 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 79 tasks left in task-pool. [ Info: running task 22 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 78 tasks left in task-pool. [ Info: running task 23 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 77 tasks left in task-pool. [ Info: running task 24 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 76 tasks left in task-pool. [ Info: running task 25 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 75 tasks left in task-pool. [ Info: running task 26 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 74 tasks left in task-pool. [ Info: running task 27 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 73 tasks left in task-pool. [ Info: running task 28 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 72 tasks left in task-pool. [ Info: running task 29 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 70 tasks left in task-pool. [ Info: running task 30 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 70 tasks left in task-pool. [ Info: running task 31 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 69 tasks left in task-pool. [ Info: running task 32 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 68 tasks left in task-pool. [ Info: running task 33 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 67 tasks left in task-pool. [ Info: running task 34 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 65 tasks left in task-pool. [ Info: running task 35 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 65 tasks left in task-pool. [ Info: running task 36 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 64 tasks left in task-pool. [ Info: running task 37 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 63 tasks left in task-pool. [ Info: running task 38 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 62 tasks left in task-pool. [ Info: running task 39 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 61 tasks left in task-pool. [ Info: running task 40 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 60 tasks left in task-pool. [ Info: running task 41 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 59 tasks left in task-pool. [ Info: running task 42 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 58 tasks left in task-pool. [ Info: running task 43 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 57 tasks left in task-pool. [ Info: running task 44 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 56 tasks left in task-pool. [ Info: running task 45 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 55 tasks left in task-pool. [ Info: running task 46 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 54 tasks left in task-pool. [ Info: running task 47 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 53 tasks left in task-pool. [ Info: running task 48 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 52 tasks left in task-pool. [ Info: running task 49 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 50 tasks left in task-pool. [ Info: running task 50 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 50 tasks left in task-pool. [ Info: running task 51 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 49 tasks left in task-pool. [ Info: running task 52 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 48 tasks left in task-pool. [ Info: running task 53 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 47 tasks left in task-pool. [ Info: running task 54 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 45 tasks left in task-pool. [ Info: running task 55 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 45 tasks left in task-pool. [ Info: running task 56 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 44 tasks left in task-pool. [ Info: running task 57 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 43 tasks left in task-pool. [ Info: running task 58 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 42 tasks left in task-pool. [ Info: running task 59 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 41 tasks left in task-pool. [ Info: running task 60 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 40 tasks left in task-pool. [ Info: running task 61 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 39 tasks left in task-pool. [ Info: running task 62 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 38 tasks left in task-pool. [ Info: running task 63 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 37 tasks left in task-pool. [ Info: running task 64 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 36 tasks left in task-pool. [ Info: running task 65 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 35 tasks left in task-pool. [ Info: running task 66 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 34 tasks left in task-pool. [ Info: running task 67 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 33 tasks left in task-pool. [ Info: running task 68 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 32 tasks left in task-pool. [ Info: running task 69 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 31 tasks left in task-pool. [ Info: running task 70 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 30 tasks left in task-pool. [ Info: running task 71 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 29 tasks left in task-pool. [ Info: running task 72 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 28 tasks left in task-pool. [ Info: running task 73 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 27 tasks left in task-pool. [ Info: running task 74 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 25 tasks left in task-pool. [ Info: running task 75 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 25 tasks left in task-pool. [ Info: running task 76 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 24 tasks left in task-pool. [ Info: running task 77 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 23 tasks left in task-pool. [ Info: running task 78 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 22 tasks left in task-pool. [ Info: running task 79 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 21 tasks left in task-pool. [ Info: running task 80 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 20 tasks left in task-pool. [ Info: running task 81 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 19 tasks left in task-pool. [ Info: running task 82 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 18 tasks left in task-pool. [ Info: running task 83 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 17 tasks left in task-pool. [ Info: running task 84 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 16 tasks left in task-pool. [ Info: running task 85 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 15 tasks left in task-pool. [ Info: running task 86 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 14 tasks left in task-pool. [ Info: running task 87 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 13 tasks left in task-pool. [ Info: running task 88 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 12 tasks left in task-pool. [ Info: running task 89 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 11 tasks left in task-pool. [ Info: running task 90 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 10 tasks left in task-pool. [ Info: running task 91 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 9 tasks left in task-pool. [ Info: running task 92 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 8 tasks left in task-pool. [ Info: running task 93 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 7 tasks left in task-pool. [ Info: running task 94 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 6 tasks left in task-pool. [ Info: running task 95 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 5 tasks left in task-pool. [ Info: running task 96 on process 65 (Schedulers-primary-S6HJ4WCf); 5 workers total; 4 tasks left in task-pool. [ Info: running task 97 on process 66 (Schedulers-primary-S6HJ4WCf); 5 workers total; 3 tasks left in task-pool. [ Info: running task 98 on process 67 (Schedulers-primary-S6HJ4WCf); 5 workers total; 2 tasks left in task-pool. [ Info: running task 99 on process 68 (Schedulers-primary-S6HJ4WCf); 5 workers total; 1 tasks left in task-pool. [ Info: running task 100 on process 69 (Schedulers-primary-S6HJ4WCf); 5 workers total; 0 tasks left in task-pool. [ Info: reducing from 5 checkpoints using process 65 (5 workers, 1 reduce workers). [ Info: reducing from 4 checkpoints using process 66 (5 workers, 2 reduce workers). [ Info: reducing from 2 checkpoints using process 65 (5 workers, 2 reduce workers). [ Info: reducing from 2 checkpoints using process 65 (5 workers, 2 reduce workers). Test Summary: | Pass Total Time pmapreduce, stable cluster test, backwards compatability | 36 36 3m27.1s [ Info: running task 1 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 99 tasks left in task-pool. [ Info: running task 2 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 98 tasks left in task-pool. [ Info: running task 3 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 97 tasks left in task-pool. [ Info: running task 4 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 95 tasks left in task-pool. [ Info: running task 5 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 95 tasks left in task-pool. [ Info: running task 6 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 94 tasks left in task-pool. [ Info: running task 7 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 93 tasks left in task-pool. [ Info: running task 8 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 92 tasks left in task-pool. [ Info: running task 9 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 89 tasks left in task-pool. [ Info: running task 10 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 89 tasks left in task-pool. [ Info: running task 11 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 89 tasks left in task-pool. [ Info: running task 12 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 88 tasks left in task-pool. [ Info: running task 13 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 87 tasks left in task-pool. [ Info: running task 14 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 84 tasks left in task-pool. [ Info: running task 15 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 84 tasks left in task-pool. [ Info: running task 16 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 84 tasks left in task-pool. [ Info: running task 17 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 83 tasks left in task-pool. [ Info: running task 18 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 82 tasks left in task-pool. [ Info: running task 19 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 79 tasks left in task-pool. [ Info: running task 20 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 79 tasks left in task-pool. [ Info: running task 21 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 79 tasks left in task-pool. [ Info: running task 22 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 78 tasks left in task-pool. [ Info: running task 23 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 77 tasks left in task-pool. [ Info: running task 24 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 74 tasks left in task-pool. [ Info: running task 25 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 74 tasks left in task-pool. [ Info: running task 26 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 74 tasks left in task-pool. [ Info: running task 27 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 73 tasks left in task-pool. [ Info: running task 28 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 72 tasks left in task-pool. [ Info: running task 29 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 69 tasks left in task-pool. [ Info: running task 30 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 69 tasks left in task-pool. [ Info: running task 31 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 69 tasks left in task-pool. [ Info: running task 32 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 68 tasks left in task-pool. [ Info: running task 33 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 67 tasks left in task-pool. [ Info: running task 35 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 65 tasks left in task-pool. [ Info: running task 36 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 64 tasks left in task-pool. [ Info: running task 34 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 64 tasks left in task-pool. [ Info: running task 37 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 63 tasks left in task-pool. [ Info: running task 38 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 62 tasks left in task-pool. [ Info: running task 39 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 59 tasks left in task-pool. [ Info: running task 40 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 59 tasks left in task-pool. [ Info: running task 41 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 59 tasks left in task-pool. [ Info: running task 42 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 58 tasks left in task-pool. [ Info: running task 43 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 57 tasks left in task-pool. [ Info: running task 44 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 54 tasks left in task-pool. [ Info: running task 45 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 54 tasks left in task-pool. [ Info: running task 46 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 54 tasks left in task-pool. [ Info: running task 47 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 53 tasks left in task-pool. [ Info: running task 48 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 52 tasks left in task-pool. [ Info: running task 49 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 51 tasks left in task-pool. [ Info: running task 50 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 50 tasks left in task-pool. [ Info: running task 51 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 49 tasks left in task-pool. [ Info: running task 52 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 48 tasks left in task-pool. [ Info: running task 53 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 47 tasks left in task-pool. [ Info: running task 54 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 44 tasks left in task-pool. [ Info: running task 55 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 44 tasks left in task-pool. [ Info: running task 56 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 44 tasks left in task-pool. [ Info: running task 57 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 43 tasks left in task-pool. [ Info: running task 58 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 42 tasks left in task-pool. [ Info: running task 59 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 39 tasks left in task-pool. [ Info: running task 60 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 39 tasks left in task-pool. [ Info: running task 61 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 39 tasks left in task-pool. [ Info: running task 62 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 38 tasks left in task-pool. [ Info: running task 63 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 37 tasks left in task-pool. [ Info: running task 64 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 34 tasks left in task-pool. [ Info: running task 65 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 34 tasks left in task-pool. [ Info: running task 66 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 34 tasks left in task-pool. [ Info: running task 67 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 33 tasks left in task-pool. [ Info: running task 68 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 32 tasks left in task-pool. [ Info: running task 69 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 29 tasks left in task-pool. [ Info: running task 70 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 29 tasks left in task-pool. [ Info: running task 71 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 29 tasks left in task-pool. [ Info: running task 72 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 28 tasks left in task-pool. [ Info: running task 73 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 27 tasks left in task-pool. [ Info: running task 74 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 26 tasks left in task-pool. [ Info: running task 75 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 25 tasks left in task-pool. [ Info: running task 76 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 24 tasks left in task-pool. [ Info: running task 77 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 23 tasks left in task-pool. [ Info: running task 78 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 22 tasks left in task-pool. [ Info: running task 79 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 21 tasks left in task-pool. [ Info: running task 80 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 20 tasks left in task-pool. [ Info: running task 81 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 19 tasks left in task-pool. [ Info: running task 82 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 18 tasks left in task-pool. [ Info: running task 83 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 17 tasks left in task-pool. [ Info: running task 84 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 14 tasks left in task-pool. [ Info: running task 85 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 14 tasks left in task-pool. [ Info: running task 86 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 14 tasks left in task-pool. [ Info: running task 87 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 13 tasks left in task-pool. [ Info: running task 88 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 12 tasks left in task-pool. [ Info: running task 89 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 11 tasks left in task-pool. [ Info: running task 90 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 9 tasks left in task-pool. [ Info: running task 91 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 9 tasks left in task-pool. [ Info: running task 92 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 8 tasks left in task-pool. [ Info: running task 93 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 7 tasks left in task-pool. [ Info: running task 94 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 4 tasks left in task-pool. [ Info: running task 95 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 4 tasks left in task-pool. [ Info: running task 96 on process 73 (Schedulers-primary-S6HJ4WCf); 5 workers total; 4 tasks left in task-pool. [ Info: running task 97 on process 70 (Schedulers-primary-S6HJ4WCf); 5 workers total; 3 tasks left in task-pool. [ Info: running task 98 on process 72 (Schedulers-primary-S6HJ4WCf); 5 workers total; 2 tasks left in task-pool. [ Info: running task 99 on process 74 (Schedulers-primary-S6HJ4WCf); 5 workers total; 1 tasks left in task-pool. [ Info: running task 100 on process 71 (Schedulers-primary-S6HJ4WCf); 5 workers total; 0 tasks left in task-pool. [ Info: reducing from 5 checkpoints using process 73 (5 workers, 1 reduce workers). [ Info: reducing from 4 checkpoints using process 70 (5 workers, 2 reduce workers). [ Info: reducing from 2 checkpoints using process 73 (5 workers, 2 reduce workers). [ Info: reducing from 2 checkpoints using process 70 (5 workers, 2 reduce workers). Test Summary: | Pass Total Time pmapreduce, stable cluster test | 2 2 3m02.7s [ Info: running task 1 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 99 tasks left in task-pool. [ Info: running task 2 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 98 tasks left in task-pool. [ Info: running task 3 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 97 tasks left in task-pool. [ Info: running task 4 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 96 tasks left in task-pool. [ Info: running task 5 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 95 tasks left in task-pool. [ Info: running task 6 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 94 tasks left in task-pool. [ Info: running task 7 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 93 tasks left in task-pool. [ Info: running task 8 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 92 tasks left in task-pool. [ Info: running task 9 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 91 tasks left in task-pool. [ Info: running task 10 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 90 tasks left in task-pool. [ Info: running task 11 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 88 tasks left in task-pool. [ Info: running task 12 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 88 tasks left in task-pool. [ Info: running task 13 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 87 tasks left in task-pool. [ Info: running task 14 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 86 tasks left in task-pool. [ Info: running task 15 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 85 tasks left in task-pool. [ Info: running task 16 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 84 tasks left in task-pool. [ Info: running task 17 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 83 tasks left in task-pool. [ Info: running task 18 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 82 tasks left in task-pool. [ Info: running task 19 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 81 tasks left in task-pool. [ Info: running task 20 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 80 tasks left in task-pool. [ Info: running task 21 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 79 tasks left in task-pool. [ Info: running task 22 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 78 tasks left in task-pool. [ Info: running task 23 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 77 tasks left in task-pool. [ Info: running task 24 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 76 tasks left in task-pool. [ Info: running task 25 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 75 tasks left in task-pool. [ Info: running task 26 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 74 tasks left in task-pool. [ Info: running task 27 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 73 tasks left in task-pool. [ Info: running task 28 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 72 tasks left in task-pool. [ Info: running task 29 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 70 tasks left in task-pool. [ Info: running task 30 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 70 tasks left in task-pool. [ Info: running task 31 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 69 tasks left in task-pool. [ Info: running task 32 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 68 tasks left in task-pool. [ Info: running task 33 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 67 tasks left in task-pool. [ Info: running task 34 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 65 tasks left in task-pool. [ Info: running task 35 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 65 tasks left in task-pool. [ Info: running task 36 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 64 tasks left in task-pool. [ Info: running task 37 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 63 tasks left in task-pool. [ Info: running task 38 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 62 tasks left in task-pool. [ Info: running task 39 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 59 tasks left in task-pool. [ Info: running task 40 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 59 tasks left in task-pool. [ Info: running task 41 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 59 tasks left in task-pool. [ Info: running task 42 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 58 tasks left in task-pool. [ Info: running task 43 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 57 tasks left in task-pool. [ Info: running task 45 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 55 tasks left in task-pool. [ Info: running task 46 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 54 tasks left in task-pool. [ Info: running task 44 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 54 tasks left in task-pool. [ Info: running task 47 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 53 tasks left in task-pool. [ Info: running task 48 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 52 tasks left in task-pool. [ Info: running task 49 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 49 tasks left in task-pool. [ Info: running task 50 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 49 tasks left in task-pool. [ Info: running task 51 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 49 tasks left in task-pool. [ Info: running task 52 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 48 tasks left in task-pool. [ Info: running task 53 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 47 tasks left in task-pool. [ Info: running task 54 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 44 tasks left in task-pool. [ Info: running task 55 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 44 tasks left in task-pool. [ Info: running task 56 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 44 tasks left in task-pool. [ Info: running task 57 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 43 tasks left in task-pool. [ Info: running task 58 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 42 tasks left in task-pool. [ Info: running task 59 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 39 tasks left in task-pool. [ Info: running task 60 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 39 tasks left in task-pool. [ Info: running task 61 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 39 tasks left in task-pool. [ Info: running task 62 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 38 tasks left in task-pool. [ Info: running task 63 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 37 tasks left in task-pool. [ Info: running task 64 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 34 tasks left in task-pool. [ Info: running task 65 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 34 tasks left in task-pool. [ Info: running task 66 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 34 tasks left in task-pool. [ Info: running task 67 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 33 tasks left in task-pool. [ Info: running task 68 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 32 tasks left in task-pool. [ Info: running task 69 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 29 tasks left in task-pool. [ Info: running task 70 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 29 tasks left in task-pool. [ Info: running task 71 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 29 tasks left in task-pool. [ Info: running task 72 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 28 tasks left in task-pool. [ Info: running task 73 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 27 tasks left in task-pool. [ Info: running task 74 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 26 tasks left in task-pool. [ Info: running task 75 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 24 tasks left in task-pool. [ Info: running task 76 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 24 tasks left in task-pool. [ Info: running task 77 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 23 tasks left in task-pool. [ Info: running task 78 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 22 tasks left in task-pool. [ Info: running task 79 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 19 tasks left in task-pool. [ Info: running task 80 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 19 tasks left in task-pool. [ Info: running task 81 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 19 tasks left in task-pool. [ Info: running task 82 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 18 tasks left in task-pool. [ Info: running task 83 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 17 tasks left in task-pool. [ Info: running task 84 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 14 tasks left in task-pool. [ Info: running task 85 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 14 tasks left in task-pool. [ Info: running task 86 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 14 tasks left in task-pool. [ Info: running task 87 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 13 tasks left in task-pool. [ Info: running task 88 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 12 tasks left in task-pool. [ Info: running task 89 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 9 tasks left in task-pool. [ Info: running task 90 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 9 tasks left in task-pool. [ Info: running task 91 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 9 tasks left in task-pool. [ Info: running task 92 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 8 tasks left in task-pool. [ Info: running task 93 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 7 tasks left in task-pool. [ Info: running task 94 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 6 tasks left in task-pool. [ Info: running task 95 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 5 tasks left in task-pool. [ Info: running task 96 on process 78 (Schedulers-primary-S6HJ4WCf); 5 workers total; 4 tasks left in task-pool. [ Info: running task 97 on process 75 (Schedulers-primary-S6HJ4WCf); 5 workers total; 3 tasks left in task-pool. [ Info: running task 98 on process 77 (Schedulers-primary-S6HJ4WCf); 5 workers total; 2 tasks left in task-pool. [ Info: running task 99 on process 76 (Schedulers-primary-S6HJ4WCf); 5 workers total; 0 tasks left in task-pool. [ Info: running task 100 on process 80 (Schedulers-primary-S6HJ4WCf); 5 workers total; 0 tasks left in task-pool. [ Info: reducing from 5 checkpoints using process 78 (5 workers, 1 reduce workers). [ Info: reducing from 4 checkpoints using process 75 (5 workers, 2 reduce workers). [ Info: reducing from 2 checkpoints using process 78 (5 workers, 2 reduce workers). [ Info: reducing from 2 checkpoints using process 78 (5 workers, 2 reduce workers). Test Summary: | Pass Total Time pmapreduce, cluster with ProcessExitedException during tasks | 4 4 3m37.8s [ Info: running task 1 on process 82 (Schedulers-primary-S6HJ4WCf); 5 workers total; 99 tasks left in task-pool. [ Info: running task 2 on process 85 (Schedulers-primary-S6HJ4WCf); 5 workers total; 98 tasks left in task-pool. [ Info: running task 3 on process 81 (Schedulers-primary-S6HJ4WCf); 5 workers total; 97 tasks left in task-pool. [ Info: running task 4 on process 83 (Schedulers-primary-S6HJ4WCf); 5 workers total; 95 tasks left in task-pool. [ Info: running task 5 on process 84 (Schedulers-primary-S6HJ4WCf); 5 workers total; 95 tasks left in task-pool. ┌ Warning: pid=82 (Schedulers-primary-S6HJ4WCf), checkpoint=nothing, task loop, caught exception during save_checkpoint └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1410 ┌ Warning: TaskFailedException │ │ nested task error: On worker 82: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.9620610079373348 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:448 │ [3] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [4] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::typeof(Schedulers.save_checkpoint), ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:298 │ [5] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::Function, ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:287 │ [6] (::Schedulers.var"#134#135"{Vector{Float32}, @Kwargs{}, typeof(foo7b), Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, String}, Dict{String, Any}, SchedulerOptions{String}, Tuple{Int64, Int64}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{Any}, Dict{Int64, Future}, Int64})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1405 │ │ nested task error: On worker 82: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.9620610079373348 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:448 │ [3] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [4] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: too many failures on process with id=82 (Schedulers-primary-S6HJ4WCf), removing from process list └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:393 ┌ Warning: pid=85 (Schedulers-primary-S6HJ4WCf), checkpoint=nothing, task loop, caught exception during save_checkpoint └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1410 ┌ Warning: TaskFailedException │ │ nested task error: On worker 85: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.8668249597307215 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:448 │ [3] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [4] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::typeof(Schedulers.save_checkpoint), ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:298 │ [5] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::Function, ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:287 │ [6] (::Schedulers.var"#134#135"{Vector{Float32}, @Kwargs{}, typeof(foo7b), Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, String}, Dict{String, Any}, SchedulerOptions{String}, Tuple{Int64, Int64}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{Any}, Dict{Int64, Future}, Int64})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1405 │ │ nested task error: On worker 85: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.8668249597307215 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:448 │ [3] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [4] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: too many failures on process with id=85 (Schedulers-primary-S6HJ4WCf), removing from process list └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:393 [ Info: running task 6 on process 81 (Schedulers-primary-S6HJ4WCf); 5 workers total; 95 tasks left in task-pool. [ Info: running task 7 on process 83 (Schedulers-primary-S6HJ4WCf); 5 workers total; 95 tasks left in task-pool. ┌ Warning: pid=84 (Schedulers-primary-S6HJ4WCf), checkpoint=nothing, task loop, caught exception during save_checkpoint └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1410 ┌ Warning: TaskFailedException │ │ nested task error: On worker 84: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.8185069514979048 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:448 │ [3] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [4] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::typeof(Schedulers.save_checkpoint), ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:298 │ [5] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::Function, ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:287 │ [6] (::Schedulers.var"#134#135"{Vector{Float32}, @Kwargs{}, typeof(foo7b), Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, String}, Dict{String, Any}, SchedulerOptions{String}, Tuple{Int64, Int64}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{Any}, Dict{Int64, Future}, Int64})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1405 │ │ nested task error: On worker 84: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.8185069514979048 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:448 │ [3] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [4] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: too many failures on process with id=84 (Schedulers-primary-S6HJ4WCf), removing from process list └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:393 [ Info: running task 8 on process 83 (Schedulers-primary-S6HJ4WCf); 2 workers total; 95 tasks left in task-pool. ┌ Warning: pid=81 (Schedulers-primary-S6HJ4WCf), checkpoint=/tmp/jl_mHp9Iv/checkpoint-oO1X2X-319, task loop, caught exception during save_checkpoint └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1410 ┌ Warning: TaskFailedException │ │ nested task error: On worker 81: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.8065322705137038 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:448 │ [3] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [4] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::typeof(Schedulers.save_checkpoint), ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:298 │ [5] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::Function, ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:287 │ [6] (::Schedulers.var"#134#135"{Vector{Float32}, @Kwargs{}, typeof(foo7b), Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, String}, Dict{String, Any}, SchedulerOptions{String}, Tuple{Int64, Int64}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{Any}, Dict{Int64, Future}, Int64})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1405 │ │ nested task error: On worker 81: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.8065322705137038 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:448 │ [3] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [4] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: too many failures on process with id=81 (Schedulers-primary-S6HJ4WCf), removing from process list └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:393 [ Info: running task 9 on process 83 (Schedulers-primary-S6HJ4WCf); 2 workers total; 95 tasks left in task-pool. ┌ Warning: pid=83 (Schedulers-primary-S6HJ4WCf), checkpoint=/tmp/jl_mHp9Iv/checkpoint-oO1X2X-324, task loop, caught exception during save_checkpoint └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1410 ┌ Warning: TaskFailedException │ │ nested task error: On worker 83: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.8587406824271978 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:448 │ [3] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [4] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::typeof(Schedulers.save_checkpoint), ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:298 │ [5] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::Function, ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:287 │ [6] (::Schedulers.var"#134#135"{Vector{Float32}, @Kwargs{}, typeof(foo7b), Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, String}, Dict{String, Any}, SchedulerOptions{String}, Tuple{Int64, Int64}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{Any}, Dict{Int64, Future}, Int64})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1405 │ │ nested task error: On worker 83: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.8587406824271978 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:448 │ [3] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [4] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: too many failures on process with id=83 (Schedulers-primary-S6HJ4WCf), removing from process list └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:393 ====================================================================================== Information request received. A stacktrace will print followed by a 1.0 second profile ====================================================================================== cmd: /opt/julia/bin/julia 368 running 1 of 1 signal (10): User defined signal 1 _ZN4llvm22MCDwarfLineTableHeader10tryGetFileERNS_9StringRefES2_St8optionalINS_3MD59MD5ResultEES3_IS1_Etj at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16MCDwarfLineTable10tryGetFileERNS_9StringRefES2_St8optionalINS_3MD59MD5ResultEES3_IS1_Etj at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm9MCContext12getDwarfFileENS_9StringRefES1_jSt8optionalINS_3MD59MD5ResultEES2_IS1_Ej at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm10MCStreamer25tryEmitDwarfFileDirectiveEjNS_9StringRefES1_St8optionalINS_3MD59MD5ResultEES2_IS1_Ej at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16DwarfCompileUnit19getOrCreateSourceIDEPKNS_6DIFileE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16DwarfCompileUnit24constructInlinedScopeDIEEPNS_12LexicalScopeERNS_3DIEE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16DwarfCompileUnit17constructScopeDIEEPNS_12LexicalScopeERNS_3DIEE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16DwarfCompileUnit25createAndAddScopeChildrenEPNS_12LexicalScopeERNS_3DIEE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16DwarfCompileUnit25createAndAddScopeChildrenEPNS_12LexicalScopeERNS_3DIEE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16DwarfCompileUnit25createAndAddScopeChildrenEPNS_12LexicalScopeERNS_3DIEE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16DwarfCompileUnit27constructSubprogramScopeDIEEPKNS_12DISubprogramEPNS_12LexicalScopeEPNS_8MCSymbolE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm10DwarfDebug15endFunctionImplEPKNS_15MachineFunctionE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16DebugHandlerBase11endFunctionEPKNS_15MachineFunctionE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm10AsmPrinter16emitFunctionBodyEv at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm13X86AsmPrinter20runOnMachineFunctionERNS_15MachineFunctionE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE.part.0 at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm3orc14SimpleCompilerclERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) operator() at /source/src/jitlayers.cpp:1620 addModule at /source/src/jitlayers.cpp:2101 jl_compile_codeinst_now at /source/src/jitlayers.cpp:683 jl_compile_codeinst_impl at /source/src/jitlayers.cpp:883 jl_compile_method_internal at /source/src/gf.c:3213 _jl_invoke at /source/src/gf.c:3685 [inlined] ijl_apply_generic at /source/src/gf.c:3893 jl_apply at /source/src/julia.h:2344 [inlined] jl_f_invokelatest at /source/src/builtins.c:877 exec_options at ./client.jl:266 _start at ./client.jl:557 jfptr__start_72357.1 at /opt/julia/lib/julia/sys.so (unknown line) _jl_invoke at /source/src/gf.c:3693 [inlined] ijl_apply_generic at /source/src/gf.c:3893 jl_apply at /source/src/julia.h:2344 [inlined] true_main at /source/src/jlapi.c:964 jl_repl_entrypoint at /source/src/jlapi.c:1132 main at /source/cli/loader_exe.c:58 unknown function (ip: 0x71b5fbe10249) at /lib/x86_64-linux-gnu/libc.so.6 __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) unknown function (ip: 0x4010b8) at /workspace/srcdir/glibc-2.17/csu/../sysdeps/x86_64/start.S unknown function (ip: (nil)) at (unknown file) ============================================================== Profile collected. A report will print at the next yield point ============================================================== ====================================================================================== Information request received. A stacktrace will print followed by a 1.0 second profile ====================================================================================== cmd: /opt/julia/bin/julia 367 running 1 of 1 signal (10): User defined signal 1 _ZNK4llvm12DenseMapBaseINS_8DenseMapIPKNS_5ValueENS_13MDAttachmentsENS_12DenseMapInfoIS4_vEENS_6detail12DenseMapPairIS4_S5_EEEES4_S5_S7_SA_E2atES4_ at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZNK4llvm5Value15getMetadataImplEj at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZNK4llvm18TargetLoweringBase23getStoreMemOperandFlagsERKNS_9StoreInstERKNS_10DataLayoutE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm19SelectionDAGBuilder10visitStoreERKNS_9StoreInstE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm19SelectionDAGBuilder5visitERKNS_11InstructionE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16SelectionDAGISel16SelectBasicBlockENS_21ilist_iterator_w_bitsINS_12ilist_detail12node_optionsINS_11InstructionELb1ELb0EvLb1ENS_10BasicBlockEEELb0ELb1EEES7_Rb at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16SelectionDAGISel20SelectAllBasicBlocksERKNS_8FunctionE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16SelectionDAGISel20runOnMachineFunctionERNS_15MachineFunctionE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm22SelectionDAGISelLegacy20runOnMachineFunctionERNS_15MachineFunctionE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE.part.0 at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm3orc14SimpleCompilerclERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) operator() at /source/src/jitlayers.cpp:1620 addModule at /source/src/jitlayers.cpp:2101 jl_compile_codeinst_now at /source/src/jitlayers.cpp:683 jl_compile_codeinst_impl at /source/src/jitlayers.cpp:883 jl_compile_method_internal at /source/src/gf.c:3213 _jl_invoke at /source/src/gf.c:3685 [inlined] ijl_apply_generic at /source/src/gf.c:3893 #start_worker#23 at /source/usr/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:236 start_worker at /source/usr/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:236 start_worker at /source/usr/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:236 process_opts at /source/usr/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:1387 unknown function (ip: 0x7d7ea6125172) at (unknown file) _jl_invoke at /source/src/gf.c:3693 [inlined] ijl_apply_generic at /source/src/gf.c:3893 jl_apply at /source/src/julia.h:2344 [inlined] jl_f_invokelatest at /source/src/builtins.c:877 exec_options at ./client.jl:266 _start at ./client.jl:557 jfptr__start_72357.1 at /opt/julia/lib/julia/sys.so (unknown line) _jl_invoke at /source/src/gf.c:3693 [inlined] ijl_apply_generic at /source/src/gf.c:3893 jl_apply at /source/src/julia.h:2344 [inlined] true_main at /source/src/jlapi.c:964 jl_repl_entrypoint at /source/src/jlapi.c:1132 main at /source/cli/loader_exe.c:58 unknown function (ip: 0x7d7ea76da249) at /lib/x86_64-linux-gnu/libc.so.6 __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) unknown function (ip: 0x4010b8) at /workspace/srcdir/glibc-2.17/csu/../sysdeps/x86_64/start.S unknown function (ip: (nil)) at (unknown file) ============================================================== Profile collected. A report will print at the next yield point ============================================================== From worker 88: From worker 88: ====================================================================================== From worker 88: Information request received. A stacktrace will print followed by a 1.0 second profile From worker 88: ====================================================================================== From worker 88: From worker 88: cmd: /opt/julia/bin/julia 358 running 1 of 1 From worker 88: From worker 88: signal (10): User defined signal 1 From worker 88: _ZN12_GLOBAL__N_17Formula12canonicalizeERKN4llvm4LoopE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 88: _ZN12_GLOBAL__N_111LSRInstance26GenerateReassociationsImplERNS_6LSRUseEjRKNS_7FormulaEjmb at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 88: _ZN12_GLOBAL__N_111LSRInstance26GenerateReassociationsImplERNS_6LSRUseEjRKNS_7FormulaEjmb at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 88: _ZN12_GLOBAL__N_111LSRInstance24GenerateAllReuseFormulaeEv at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 88: _ZN12_GLOBAL__N_111LSRInstanceC1EPN4llvm4LoopERNS1_7IVUsersERNS1_15ScalarEvolutionERNS1_13DominatorTreeERNS1_8LoopInfoERKNS1_19TargetTransformInfoERNS1_15AssumptionCacheERNS1_17TargetLibraryInfoEPNS1_16MemorySSAUpdaterE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 88: _ZL18ReduceLoopStrengthPN4llvm4LoopERNS_7IVUsersERNS_15ScalarEvolutionERNS_13DominatorTreeERNS_8LoopInfoERKNS_19TargetTransformInfoERNS_15AssumptionCacheERNS_17TargetLibraryInfoEPNS_9MemorySSAE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 88: _ZN12_GLOBAL__N_118LoopStrengthReduce9runOnLoopEPN4llvm4LoopERNS1_13LPPassManagerE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 88: _ZN4llvm13LPPassManager13runOnFunctionERNS_8FunctionE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 88: _ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 88: _ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 88: _ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 88: _ZN4llvm3orc14SimpleCompilerclERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 88: operator() at /source/src/jitlayers.cpp:1620 From worker 88: addModule at /source/src/jitlayers.cpp:2101 From worker 88: jl_compile_codeinst_now at /source/src/jitlayers.cpp:683 From worker 88: jl_compile_codeinst_impl at /source/src/jitlayers.cpp:883 From worker 88: jl_compile_method_internal at /source/src/gf.c:3213 From worker 88: _jl_invoke at /source/src/gf.c:3685 [inlined] From worker 88: ijl_apply_generic at /source/src/gf.c:3893 From worker 88: #118 at /home/pkgeval/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1253 From worker 88: #28 at /home/pkgeval/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 From worker 88: unknown function (ip: 0x7a3dbe4f00da) at (unknown file) From worker 88: _jl_invoke at /source/src/gf.c:3693 [inlined] From worker 88: ijl_apply_generic at /source/src/gf.c:3893 From worker 88: jl_apply at /source/src/julia.h:2344 [inlined] From worker 88: start_task at /source/src/task.c:1249 From worker 88: unknown function (ip: (nil)) at (unknown file) From worker 88: From worker 88: ============================================================== From worker 88: Profile collected. A report will print at the next yield point From worker 88: ============================================================== From worker 88: [ Info: running task 10 on process 86 (Schedulers-primary-S6HJ4WCf); 4 workers total; 95 tasks left in task-pool. [ Info: running task 11 on process 87 (Schedulers-primary-S6HJ4WCf); 4 workers total; 93 tasks left in task-pool. [ Info: running task 12 on process 88 (Schedulers-primary-S6HJ4WCf); 4 workers total; 93 tasks left in task-pool. From worker 89: ┌ Warning: There were no samples collected in one or more groups. From worker 89: │ This may be due to idle threads, or you may need to run your From worker 89: │ program longer (perhaps by running it multiple times), From worker 89: │ or adjust the delay between samples with `Profile.init()`. From worker 89: └ @ Profile /opt/julia/share/julia/stdlib/v1.13/Profile/src/Profile.jl:1362 From worker 89: Overhead ╎ [+additional indent] Count File:Line Function From worker 89: ========================================================= From worker 89: Thread 1 (default) Task 0x00007d7e8d600010 Total snapshots: 1. Utilization: 100% From worker 89: ╎1 @Base/client.jl:557 _start() From worker 89: ╎ 1 @Base/client.jl:266 exec_options(opts::Base.JLOptions) From worker 89: ╎ 1 @Distributed/…cluster.jl:1387 process_opts(opts::Base.JLOptions) From worker 89: ╎ 1 @Distributed/…cluster.jl:236 start_worker() From worker 89: ╎ 1 @Distributed/…luster.jl:236 start_worker(cookie::String) From worker 89: ╎ 1 @Distributed/…luster.jl:236 start_worker(cookie::String; kwargs::Base… From worker 89: From worker 88: ┌ Warning: There were no samples collected in one or more groups. From worker 88: │ This may be due to idle threads, or you may need to run your From worker 88: │ program longer (perhaps by running it multiple times), From worker 88: │ or adjust the delay between samples with `Profile.init()`. From worker 88: └ @ Profile /opt/julia/share/julia/stdlib/v1.13/Profile/src/Profile.jl:1362 From worker 88: Overhead ╎ [+additional indent] Count File:Line Function From worker 88: ========================================================= From worker 88: Thread 1 (default) Task 0x00007a3d9fe20c40 Total snapshots: 1. Utilization: 0% From worker 88: ╎1 @Base/task.jl:1150 wait_forever() From worker 88: ╎ 1 @Base/task.jl:1213 wait() From worker 88: [ Info: running task 13 on process 86 (Schedulers-primary-S6HJ4WCf); 4 workers total; 92 tasks left in task-pool. [ Info: running task 14 on process 87 (Schedulers-primary-S6HJ4WCf); 4 workers total; 91 tasks left in task-pool. [ Info: running task 15 on process 86 (Schedulers-primary-S6HJ4WCf); 4 workers total; 90 tasks left in task-pool. [ Info: running task 16 on process 87 (Schedulers-primary-S6HJ4WCf); 4 workers total; 89 tasks left in task-pool. ┌ Warning: pid=88 (Schedulers-primary-S6HJ4WCf), checkpoint=nothing, task loop, caught exception during save_checkpoint └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1410 ┌ Warning: TaskFailedException │ │ nested task error: On worker 88: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.8817393338542696 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ./none:-1 [inlined] │ [3] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:0 │ [4] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [5] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::typeof(Schedulers.save_checkpoint), ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:298 │ [5] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::Function, ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:287 │ [6] (::Schedulers.var"#134#135"{Vector{Float32}, @Kwargs{}, typeof(foo7b), Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, String}, Dict{String, Any}, SchedulerOptions{String}, Tuple{Int64, Int64}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{Any}, Dict{Int64, Future}, Int64})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1405 │ │ nested task error: On worker 88: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.8817393338542696 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ./none:-1 [inlined] │ [3] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:0 │ [4] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [5] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: too many failures on process with id=88 (Schedulers-primary-S6HJ4WCf), removing from process list └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:393 ┌ Warning: problem adding or removing processes └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:737 ┌ Warning: TaskFailedException │ │ nested task error: TaskFailedException │ │ nested task error: DNSError: ┌ Warning: There were no samples collected in one or more groups., unknown node or service (EAI_NONAME) │ Stacktrace: │ [1] worker_from_id(pg::Distributed.ProcessGroup, i::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:1109 │ [2] worker_from_id │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:1106 [inlined] │ [3] remote_do(::Function, ::Int64; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:557 │ [4] remote_do │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:557 [inlined] │ [5] kill(manager::Distributed.LocalManager, pid::Int64, config::WorkerConfig; profile_wait::Int64, exit_timeout::Int64, term_timeout::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:746 │ [6] kill │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:743 [inlined] │ [7] create_worker(manager::Distributed.LocalManager, wconfig::WorkerConfig) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:614 │ [8] setup_launched_worker(manager::Distributed.LocalManager, wconfig::WorkerConfig, launched_q::Vector{Int64}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:555 │ [9] (::Distributed.var"#39#40"{Distributed.LocalManager, Vector{Int64}, WorkerConfig})() │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:511 │ │ caused by: DNSError: ┌ Warning: There were no samples collected in one or more groups., unknown node or service (EAI_NONAME) │ Stacktrace: │ [1] getalladdrinfo(host::String) │ @ Sockets /opt/julia/share/julia/stdlib/v1.13/Sockets/src/addrinfo.jl:113 │ [2] getaddrinfo │ @ /opt/julia/share/julia/stdlib/v1.13/Sockets/src/addrinfo.jl:159 [inlined] │ [3] connect_to_worker(host::String, port::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:692 │ [4] connect(manager::Distributed.LocalManager, pid::Int64, config::WorkerConfig) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:624 │ [5] create_worker(manager::Distributed.LocalManager, wconfig::WorkerConfig) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:610 │ [6] setup_launched_worker(manager::Distributed.LocalManager, wconfig::WorkerConfig, launched_q::Vector{Int64}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:555 │ [7] (::Distributed.var"#39#40"{Distributed.LocalManager, Vector{Int64}, WorkerConfig})() │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:511 │ │ caused by: ArgumentError: invalid base 16 digit '┌' in "┌ Warning" │ Stacktrace: │ [1] tryparse_internal(::Type{Int64}, s::SubString{String}, startpos::Int64, endpos::Int64, base_::Int64, raise::Bool) │ @ Base ./parse.jl:143 │ [2] parse(::Type{Int64}, s::SubString{String}; base::Int64) │ @ Base ./parse.jl:254 │ [3] parse │ @ ./parse.jl:253 [inlined] │ [4] parseipv6fields │ @ /opt/julia/share/julia/stdlib/v1.13/Sockets/src/IPAddr.jl:240 [inlined] │ [5] parse(::Type{Sockets.IPv6}, str::String) │ @ Sockets /opt/julia/share/julia/stdlib/v1.13/Sockets/src/IPAddr.jl:254 │ [6] parse │ @ /opt/julia/share/julia/stdlib/v1.13/Sockets/src/IPAddr.jl:269 [inlined] │ [7] connect_to_worker(host::String, port::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:690 │ [8] connect(manager::Distributed.LocalManager, pid::Int64, config::WorkerConfig) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:624 │ [9] create_worker(manager::Distributed.LocalManager, wconfig::WorkerConfig) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:610 │ [10] setup_launched_worker(manager::Distributed.LocalManager, wconfig::WorkerConfig, launched_q::Vector{Int64}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:555 │ [11] (::Distributed.var"#39#40"{Distributed.LocalManager, Vector{Int64}, WorkerConfig})() │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:511 │ Stacktrace: │ [1] sync_end(c::Channel{Any}) │ @ Base ./task.jl:601 │ [2] macro expansion │ @ ./task.jl:634 [inlined] │ [3] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [4] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [5] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [6] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [7] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [8] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [9] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, String}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch(t::Task) │ @ Base ./task.jl:525 │ [4] loop(eloop::Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, String}, journal::Dict{String, Any}, journal_task_callback::Function, tsk_map::Task, tsk_reduce::Task) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:735 │ [5] epmapreduce!(::Vector{Float32}, ::SchedulerOptions{String}, ::Function, ::UnitRange{Int64}, ::Int64, ::Vararg{Int64}; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1265 │ [6] epmapreduce!(::Vector{Float32}, ::SchedulerOptions{String}, ::Function, ::UnitRange{Int64}, ::Int64, ::Vararg{Int64}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1243 │ [7] top-level scope │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:434 │ [8] macro expansion │ @ /opt/julia/share/julia/stdlib/v1.13/Test/src/Test.jl:1855 [inlined] │ [9] macro expansion │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:466 [inlined] │ [10] include(mapexpr::Function, mod::Module, _path::String) │ @ Base ./Base.jl:310 │ [11] top-level scope │ @ none:6 │ [12] eval(m::Module, e::Any) │ @ Core ./boot.jl:489 │ [13] exec_options(opts::Base.JLOptions) │ @ Base ./client.jl:290 │ [14] _start() │ @ Base ./client.jl:557 │ │ nested task error: TaskFailedException │ │ nested task error: DNSError: ┌ Warning: There were no samples collected in one or more groups., unknown node or service (EAI_NONAME) │ Stacktrace: │ [1] worker_from_id(pg::Distributed.ProcessGroup, i::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:1109 │ [2] worker_from_id │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:1106 [inlined] │ [3] remote_do(::Function, ::Int64; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:557 │ [4] remote_do │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:557 [inlined] │ [5] kill(manager::Distributed.LocalManager, pid::Int64, config::WorkerConfig; profile_wait::Int64, exit_timeout::Int64, term_timeout::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:746 │ [6] kill │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:743 [inlined] │ [7] create_worker(manager::Distributed.LocalManager, wconfig::WorkerConfig) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:614 │ [8] setup_launched_worker(manager::Distributed.LocalManager, wconfig::WorkerConfig, launched_q::Vector{Int64}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:555 │ [9] (::Distributed.var"#39#40"{Distributed.LocalManager, Vector{Int64}, WorkerConfig})() │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:511 │ │ caused by: DNSError: ┌ Warning: There were no samples collected in one or more groups., unknown node or service (EAI_NONAME) │ Stacktrace: │ [1] getalladdrinfo(host::String) │ @ Sockets /opt/julia/share/julia/stdlib/v1.13/Sockets/src/addrinfo.jl:113 │ [2] getaddrinfo │ @ /opt/julia/share/julia/stdlib/v1.13/Sockets/src/addrinfo.jl:159 [inlined] │ [3] connect_to_worker(host::String, port::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:692 │ [4] connect(manager::Distributed.LocalManager, pid::Int64, config::WorkerConfig) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:624 │ [5] create_worker(manager::Distributed.LocalManager, wconfig::WorkerConfig) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:610 │ [6] setup_launched_worker(manager::Distributed.LocalManager, wconfig::WorkerConfig, launched_q::Vector{Int64}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:555 │ [7] (::Distributed.var"#39#40"{Distributed.LocalManager, Vector{Int64}, WorkerConfig})() │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:511 │ │ caused by: ArgumentError: invalid base 16 digit '┌' in "┌ Warning" │ Stacktrace: │ [1] tryparse_internal(::Type{Int64}, s::SubString{String}, startpos::Int64, endpos::Int64, base_::Int64, raise::Bool) │ @ Base ./parse.jl:143 │ [2] parse(::Type{Int64}, s::SubString{String}; base::Int64) │ @ Base ./parse.jl:254 │ [3] parse │ @ ./parse.jl:253 [inlined] │ [4] parseipv6fields │ @ /opt/julia/share/julia/stdlib/v1.13/Sockets/src/IPAddr.jl:240 [inlined] │ [5] parse(::Type{Sockets.IPv6}, str::String) │ @ Sockets /opt/julia/share/julia/stdlib/v1.13/Sockets/src/IPAddr.jl:254 │ [6] parse │ @ /opt/julia/share/julia/stdlib/v1.13/Sockets/src/IPAddr.jl:269 [inlined] │ [7] connect_to_worker(host::String, port::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:690 │ [8] connect(manager::Distributed.LocalManager, pid::Int64, config::WorkerConfig) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:624 │ [9] create_worker(manager::Distributed.LocalManager, wconfig::WorkerConfig) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:610 │ [10] setup_launched_worker(manager::Distributed.LocalManager, wconfig::WorkerConfig, launched_q::Vector{Int64}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:555 │ [11] (::Distributed.var"#39#40"{Distributed.LocalManager, Vector{Int64}, WorkerConfig})() │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:511 │ Stacktrace: │ [1] sync_end(c::Channel{Any}) │ @ Base ./task.jl:601 │ [2] macro expansion │ @ ./task.jl:634 [inlined] │ [3] addprocs_locked(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:496 │ [4] addprocs_locked │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:461 [inlined] │ [5] addprocs(manager::Distributed.LocalManager; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:455 │ [6] addprocs │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/cluster.jl:448 [inlined] │ [7] #addprocs#197 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:472 [inlined] │ [8] addprocs(np::Int64) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:469 │ [9] (::Schedulers.var"#loop##20#loop##21"{Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, String}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:768 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 From worker 87: From worker 87: ====================================================================================== From worker 87: Information request received. A stacktrace will print followed by a 1.0 second profile From worker 87: ====================================================================================== From worker 87: From worker 87: cmd: /opt/julia/bin/julia 357 running 1 of 1 From worker 87: From worker 87: signal (10): User defined signal 1 From worker 87: unknown function (ip: 0x78d1553804e3) at /lib/x86_64-linux-gnu/libc.so.6 From worker 87: _ZN4llvm13StringMapImpl15LookupBucketForENS_9StringRefEj at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 87: _ZN4llvm8MDString3getERNS_11LLVMContextENS_9StringRefE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 87: _ZN4llvm9MDBuilder24createTBAAScalarTypeNodeENS_9StringRefEPNS_6MDNodeEm at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 87: tbaa_make_child_with_context at /source/src/llvm-codegen-shared.h:146 [inlined] From worker 87: initFunctions at /source/src/llvm-pass-helpers.cpp:44 From worker 87: initAll at /source/src/llvm-pass-helpers.cpp:69 From worker 87: doInitialization at /source/src/llvm-alloc-opt.cpp:1307 [inlined] From worker 87: run at /source/src/llvm-alloc-opt.cpp:1347 From worker 87: run at /source/usr/include/llvm/IR/PassManagerInternal.h:91 From worker 87: _ZN4llvm11PassManagerINS_8FunctionENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 87: run at /source/usr/include/llvm/IR/PassManagerInternal.h:91 From worker 87: _ZN4llvm27ModuleToFunctionPassAdaptor3runERNS_6ModuleERNS_15AnalysisManagerIS1_JEEE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 87: run at /source/usr/include/llvm/IR/PassManagerInternal.h:91 From worker 87: _ZN4llvm11PassManagerINS_6ModuleENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) From worker 87: run at /source/src/pipeline.cpp:791 From worker 87: operator() at /source/src/jitlayers.cpp:1518 From worker 87: withModuleDo<(anonymous namespace)::sizedOptimizerT::operator()(llvm::orc::ThreadSafeModule) [with long unsigned int N = 4]:: > at /source/usr/include/llvm/ExecutionEngine/Orc/ThreadSafeModule.h:136 [inlined] From worker 87: operator() at /source/src/jitlayers.cpp:1479 [inlined] From worker 87: operator() at /source/src/jitlayers.cpp:1631 [inlined] From worker 87: addModule at /source/src/jitlayers.cpp:2088 From worker 87: jl_compile_codeinst_now at /source/src/jitlayers.cpp:683 From worker 87: jl_compile_codeinst_impl at /source/src/jitlayers.cpp:883 From worker 87: jl_compile_method_internal at /source/src/gf.c:3213 From worker 87: _jl_invoke at /source/src/gf.c:3685 [inlined] From worker 87: ijl_apply_generic at /source/src/gf.c:3893 From worker 87: put! at /source/usr/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:703 From worker 87: unknown function (ip: 0x78d15392972c) at (unknown file) From worker 87: _jl_invoke at /source/src/gf.c:3693 [inlined] From worker 87: ijl_apply_generic at /source/src/gf.c:3893 From worker 87: run_work_thunk at /source/usr/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 From worker 87: #schedule_call##2 at /source/usr/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 From worker 87: unknown function (ip: 0x78d153959a1f) at (unknown file) From worker 87: _jl_invoke at /source/src/gf.c:3693 [inlined] From worker 87: ijl_apply_generic at /source/src/gf.c:3893 From worker 87: jl_apply at /source/src/julia.h:2344 [inlined] From worker 87: start_task at /source/src/task.c:1249 From worker 87: unknown function (ip: (nil)) at (unknown file) From worker 87: From worker 87: ============================================================== From worker 87: Profile collected. A report will print at the next yield point From worker 87: ============================================================== From worker 87: [ Info: running task 17 on process 86 (Schedulers-primary-S6HJ4WCf); 3 workers total; 89 tasks left in task-pool. ┌ Warning: pid=87 (Schedulers-primary-S6HJ4WCf), checkpoint=/tmp/jl_mHp9Iv/checkpoint-oO1X2X-330, task loop, caught exception during save_checkpoint └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1410 ┌ Warning: TaskFailedException │ │ nested task error: On worker 87: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.9547635921523003 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ./none:-1 [inlined] │ [3] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:0 │ [4] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [5] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::typeof(Schedulers.save_checkpoint), ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:298 │ [5] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::Function, ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:287 │ [6] (::Schedulers.var"#134#135"{Vector{Float32}, @Kwargs{}, typeof(foo7b), Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, String}, Dict{String, Any}, SchedulerOptions{String}, Tuple{Int64, Int64}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{Any}, Dict{Int64, Future}, Int64})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1405 │ │ nested task error: On worker 87: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.9547635921523003 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ./none:-1 [inlined] │ [3] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:0 │ [4] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [5] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: too many failures on process with id=87 (Schedulers-primary-S6HJ4WCf), removing from process list └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:393 [ Info: running task 18 on process 86 (Schedulers-primary-S6HJ4WCf); 2 workers total; 89 tasks left in task-pool. From worker 86: From worker 86: ====================================================================================== From worker 86: Information request received. A stacktrace will print followed by a 1.0 second profile From worker 86: ====================================================================================== From worker 86: From worker 86: cmd: /opt/julia/bin/julia 355 running 0 of 1 From worker 86: From worker 86: signal (10): User defined signal 1 From worker 86: epoll_pwait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) From worker 86: uv__io_poll at /workspace/srcdir/libuv/src/unix/linux.c:1404 From worker 86: uv_run at /workspace/srcdir/libuv/src/unix/core.c:430 From worker 86: ijl_task_get_next at /source/src/scheduler.c:457 From worker 86: wait at ./task.jl:1213 From worker 86: wait_forever at ./task.jl:1150 From worker 86: jfptr_wait_forever_42615.1 at /opt/julia/lib/julia/sys.so (unknown line) From worker 86: _jl_invoke at /source/src/gf.c:3693 [inlined] From worker 86: ijl_apply_generic at /source/src/gf.c:3893 From worker 86: jl_apply at /source/src/julia.h:2344 [inlined] From worker 86: start_task at /source/src/task.c:1249 From worker 86: unknown function (ip: (nil)) at (unknown file) From worker 86: From worker 86: ============================================================== From worker 86: Profile collected. A report will print at the next yield point From worker 86: ============================================================== From worker 86: [ Info: running task 19 on process 89 (Schedulers-primary-S6HJ4WCf); 2 workers total; 88 tasks left in task-pool. From worker 86: ┌ Warning: There were no samples collected in one or more groups. From worker 86: │ This may be due to idle threads, or you may need to run your From worker 86: │ program longer (perhaps by running it multiple times), From worker 86: │ or adjust the delay between samples with `Profile.init()`. From worker 86: └ @ Profile /opt/julia/share/julia/stdlib/v1.13/Profile/src/Profile.jl:1362 From worker 86: Overhead ╎ [+additional indent] Count File:Line Function From worker 86: ========================================================= From worker 86: Thread 1 (default) Task 0x000075c225966e00 Total snapshots: 212. Utilization: 0% From worker 86: ╎212 @Base/task.jl:1150 wait_forever() From worker 86: 211╎ 212 @Base/task.jl:1213 wait() From worker 86: [ Info: running task 20 on process 86 (Schedulers-primary-S6HJ4WCf); 2 workers total; 87 tasks left in task-pool. ====================================================================================== Information request received. A stacktrace will print followed by a 1.0 second profile ====================================================================================== cmd: /opt/julia/bin/julia 15 running 0 of 1 signal (10): User defined signal 1 epoll_pwait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) uv__io_poll at /workspace/srcdir/libuv/src/unix/linux.c:1404 uv_run at /workspace/srcdir/libuv/src/unix/core.c:430 ijl_task_get_next at /source/src/scheduler.c:457 wait at ./task.jl:1213 wait_forever at ./task.jl:1150 jfptr_wait_forever_42615.1 at /opt/julia/lib/julia/sys.so (unknown line) _jl_invoke at /source/src/gf.c:3693 [inlined] ijl_apply_generic at /source/src/gf.c:3893 jl_apply at /source/src/julia.h:2344 [inlined] start_task at /source/src/task.c:1249 unknown function (ip: (nil)) at (unknown file) [ Info: running task 21 on process 86 (Schedulers-primary-S6HJ4WCf); 2 workers total; 86 tasks left in task-pool. ============================================================== Profile collected. A report will print at the next yield point ============================================================== ┌ Warning: Failed to gracefully kill worker 87 └ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:753 ┌ Warning: Sending profile SIGUSR1 to worker 87 └ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:756 From worker 87: From worker 87: ====================================================================================== From worker 87: Information request received. A stacktrace will print followed by a 1.0 second profile From worker 87: ====================================================================================== From worker 87: From worker 87: cmd: /opt/julia/bin/julia 357 running 1 of 1 From worker 87: From worker 87: signal (10): User defined signal 1 From worker 87: has_free_typevars at /source/src/jltypes.c:115 From worker 87: subtype_tuple_tail at /source/src/subtype.c:1292 [inlined] From worker 87: subtype_tuple at /source/src/subtype.c:1376 [inlined] From worker 87: subtype at /source/src/subtype.c:1550 From worker 87: subtype_unionall at /source/src/subtype.c:965 From worker 87: subtype at /source/src/subtype.c:1513 From worker 87: subtype_unionall at /source/src/subtype.c:965┌ Warning: There were no samples collected in one or more groups. │ This may be due to idle threads, or you may need to run your │ program longer (perhaps by running it multiple times), │ or adjust the delay between samples with `Profile.init()`. └ @ Profile /opt/julia/share/julia/stdlib/v1.13/Profile/src/Profile.jl:1362 Overhead ╎ [+additional indent] Count File:Line Function ========================================================= Thread 1 (default) Task 0x00007993a63ca4a0 Total snapshots: 315. Utilization: 0% ╎315 @Base/task.jl:1150 wait_forever() 314╎ 315 @Base/task.jl:1213 wait() [ Info: running task 22 on process 89 (Schedulers-primary-S6HJ4WCf); 2 workers total; 85 tasks left in task-pool. From worker 87: subtype at /source/src/subtype.c:1513 From worker 87: exists_subtype at /source/src/subtype.c:1793 [inlined] From worker 87: forall_exists_subtype at /source/src/subtype.c:1822 From worker 87: ijl_subtype_env at /source/src/subtype.c:2281 From worker 87: jl_type_intersection_env_s at /source/src/subtype.c:4582 From worker 87: jl_f__compute_sparams at /source/src/builtins.c:2128 From worker 87: NamedTuple at ./boot.jl:843 [inlined] From worker 87: setindex at ./namedtuple.jl:484 [inlined] From worker 87: #annotations##0 at ./none (unknown line) [inlined] From worker 87: iterate at ./generator.jl:48 [inlined] From worker 87: _collect at ./array.jl:653 From worker 87: collect at ./array.jl:647 [inlined] From worker 87: annotations at ./strings/annotated.jl:399 [inlined] From worker 87: annotations at ./strings/annotated.jl:404 [inlined] From worker 87: getindex at ./strings/annotated.jl:164 From worker 87: string_truncate_boundaries at ./strings/util.jl:673 From worker 87: string_truncate_boundaries at ./strings/util.jl:653 [inlined] From worker 87: rtruncate at ./strings/util.jl:560 [inlined] From worker 87: rtruncate at ./strings/util.jl:560 [inlined] From worker 87: tree_format at /source/usr/share/julia/stdlib/v1.13/Profile/src/Profile.jl:1075 From worker 87: print_tree at /source/usr/share/julia/stdlib/v1.13/Profile/src/Profile.jl:1255 From worker 87: tree at /source/usr/share/julia/stdlib/v1.13/Profile/src/Profile.jl:1309 From worker 87: print_group at /source/usr/share/julia/stdlib/v1.13/Profile/src/Profile.jl:389 From worker 87: #print#5 at /source/usr/share/julia/stdlib/v1.13/Profile/src/Profile.jl:331 From worker 87: print at /source/usr/share/julia/stdlib/v1.13/Profile/src/Profile.jl:271 [inlined] From worker 87: print at /source/usr/share/julia/stdlib/v1.13/Profile/src/Profile.jl:271 [inlined] From worker 87: print at /source/usr/share/julia/stdlib/v1.13/Profile/src/Profile.jl:271 From worker 87: _peek_report at /source/usr/share/julia/stdlib/v1.13/Profile/src/Profile.jl:94 From worker 87: jfptr__peek_report_4553.1 at /opt/julia/share/julia/compiled/v1.13/Profile/nGhxz_9HAIE.so (unknown line) From worker 87: _jl_invoke at /source/src/gf.c:3693 [inlined] From worker 87: ijl_apply_generic at /source/src/gf.c:3893 From worker 87: jl_apply at /source/src/julia.h:2344 [inlined] From worker 87: jl_f_invokelatest at /source/src/builtins.c:877 From worker 87: profile_printing_listener at ./Base.jl:337 From worker 87: #start_profile_listener##0 at ./Base.jl:355 From worker 87: jfptr_YY.start_profile_listenerYY.YY.0_7494.1 at /opt/julia/lib/julia/sys.so (unknown line) From worker 87: _jl_invoke at /source/src/gf.c:3693 [inlined] From worker 87: ijl_apply_generic at /source/src/gf.c:3893 From worker 87: jl_apply at /source/src/julia.h:2344 [inlined] From worker 87: start_task at /source/src/task.c:1249 From worker 87: unknown function (ip: (nil)) at (unknown file) From worker 87: From worker 87: ============================================================== From worker 87: Profile collected. A report will print at the next yield point From worker 87: ============================================================== From worker 87: From worker 87: ┌ Warning: There were no samples collected in one or more groups. From worker 87: │ This may be due to idle threads, or you may need to run your From worker 87: │ program longer (perhaps by running it multiple times), From worker 87: │ or adjust the delay between samples with `Profile.init()`. From worker 87: └ @ Profile /opt/julia/share/julia/stdlib/v1.13/Profile/src/Profile.jl:1362 From worker 87: Overhead ╎ [+additional indent] Count File:Line Function From worker 87: ========================================================= From worker 87: Thread 1 (default) Task 0x000078d134308e20 Total snapshots: 43. Utilization: 100% From worker 87: ╎40 @Distributed/…essages.jl:304 (::Distributed.var"#handle_msg##6#handle_ms… From worker 87: ╎ 40 @Distributed/…essages.jl:102 deliver_result(sock::Sockets.TCPSocket, ms… From worker 87: ╎ 40 @Distributed/…ssages.jl:113 send_msg_now From worker 87: ╎ 40 @Distributed/…ssages.jl:118 send_msg_now From worker 87: ╎ 40 @Distributed/…sages.jl:181 send_msg_(w::Distributed.Worker, header::… From worker 87: ╎ 40 @Distributed/…sages.jl:78 serialize_msg(s::Distributed.ClusterSeria… From worker 87: ╎ ╎ 40 @Serialization/…ion.jl:684 serialize(s::Distributed.ClusterSeriali… From worker 87: ╎ ╎ 40 @Serialization/…on.jl:710 serialize_any(s::Distributed.ClusterSer… From worker 87: ╎ ╎ 1 @Distributed/…lize.jl:196 serialize(s::Distributed.ClusterSerial… From worker 87: ╎ ╎ 1 @Base/strings/io.jl:189 string From worker 87: ╎ ╎ 1 @Base/strings/io.jl:147 print_to_string From worker 87: ╎ ╎ ╎ 1 @Base/strings/io.jl:35 print(io::IOBuffer, x::Type) From worker 87: ╎ ╎ ╎ 1 @Base/show.jl:975 show From worker 87: ╎ ╎ ╎ 1 @Base/show.jl:978 _show_type(io::IOBuffer, x::Type) From worker 87: ╎ ╎ ╎ 1 @Base/show.jl:1085 show_type_name(io::IOBuffer, tn::Core.T… From worker 87: ╎ ╎ ╎ 1 @Base/show.jl:1051 is_global_function(tn::Core.TypeName, … From worker 87: ╎ ╎ ╎ ╎ 1 @Base/…nternals.jl:418 isconst From worker 87: ╎ ╎ 30 @Distributed/…lize.jl:197 serialize(s::Distributed.ClusterSerial… From worker 87: 24╎ ╎ 30 @Serialization/…n.jl:276 serialize(s::Distributed.ClusterSerial… From worker 87: ╎ ╎ 6 @Compiler/…einfer.jl:1570 typeinf_ext_toplevel(mi::Core.Method… From worker 87: ╎ ╎ ╎ 5 @Compiler/…infer.jl:1561 typeinf_ext_toplevel(interp::Compile… From worker 87: ╎ ╎ ╎ 5 @Compiler/…infer.jl:1378 typeinf_ext(interp::Compiler.Native… From worker 87: ╎ ╎ ╎ 1 @Compiler/…tion.jl:4460 typeinf(interp::Compiler.NativeInte… From worker 87: ╎ ╎ ╎ 1 @Compiler/…tate.jl:1205 doworkloop(interp::Compiler.Native… From worker 87: ╎ ╎ ╎ 1 @Compiler/…tion.jl:178 (::Compiler.var"#handle1#abstract_… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…tion.jl:868 abstract_call_method_with_const_a… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…ion.jl:898 abstract_call_method_with_const_a… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…on.jl:1319 const_prop_call(interp::Compiler… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…on.jl:1414 matching_cache_argtypes(𝕃::Comp… From worker 87: 1╎ ╎ ╎ ╎ 1 @Compiler/…lt.jl:30 pick_const_arg(𝕃::Compiler.Infer… From worker 87: ╎ ╎ ╎ 3 @Compiler/…tion.jl:4469 typeinf(interp::Compiler.NativeInte… From worker 87: ╎ ╎ ╎ 3 @Compiler/…tion.jl:4311 typeinf_local(interp::Compiler.Nat… From worker 87: ╎ ╎ ╎ 3 @Compiler/…tion.jl:3762 abstract_eval_basic_statement From worker 87: ╎ ╎ ╎ ╎ 3 @Compiler/…ion.jl:3805 abstract_eval_basic_statement From worker 87: ╎ ╎ ╎ ╎ 3 @Compiler/…ion.jl:3430 abstract_eval_statement_expr(int… From worker 87: ╎ ╎ ╎ ╎ 3 @Compiler/…on.jl:3074 abstract_eval_call(interp::Compi… From worker 87: ╎ ╎ ╎ ╎ 3 @Compiler/…on.jl:3056 abstract_call(interp::Compiler.… From worker 87: ╎ ╎ ╎ ╎ 3 @Compiler/…on.jl:2896 abstract_call(interp::Compiler… From worker 87: ╎ ╎ ╎ ╎ ╎ 3 @Compiler/…n.jl:2903 abstract_call(interp::Compiler… From worker 87: ╎ ╎ ╎ ╎ ╎ 3 @Compiler/…n.jl:2796 abstract_call_known(interp::C… From worker 87: ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…n.jl:120 abstract_call_gf_by_type(inte… From worker 87: ╎ ╎ ╎ ╎ ╎ 1 @Compiler/….jl:342 kwcall(::@NamedTuple{max_meth… From worker 87: ╎ ╎ ╎ ╎ ╎ 1 @Compiler/….jl:348 find_method_matches(interp::… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/….jl:386 find_simple_method_matches(… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…jl:102 findall From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…jl:111 findall(sig::Type, table::… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…jl:70 findall From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:70 #findall#5 From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:97 _findall From worker 87: 1╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Base/…ls.jl:1562 _methods_by_ftype From worker 87: ╎ ╎ ╎ ╎ ╎ 2 @Compiler/…n.jl:338 abstract_call_gf_by_type(inte… From worker 87: ╎ ╎ ╎ ╎ ╎ 2 @Compiler/….jl:252 (::Compiler.var"#infercalls#a… From worker 87: ╎ ╎ ╎ ╎ ╎ 2 @Compiler/….jl:178 (::Compiler.var"#handle1#abs… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 2 @Compiler/….jl:868 abstract_call_method_with_c… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…jl:892 abstract_call_method_with_c… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:1262 semi_concrete_eval_call(in… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:877 Compiler.IRInterpretationS… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:59 inflate_ir(ci::Core.CodeIn… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:14 inflate_ir!(ci::Core.Code… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:5 matching_cache_argtypes From worker 87: 1╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:128 most_general_argtypes(… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…jl:898 abstract_call_method_with_c… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:1348 const_prop_call(interp::Co… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:4469 typeinf(interp::Compiler.… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:4311 typeinf_local(interp::Co… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:3762 abstract_eval_basic_sta… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:3805 abstract_eval_basic_st… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:3430 abstract_eval_stateme… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:3074 abstract_eval_call(i… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:3056 abstract_call(inter… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:2896 abstract_call(inte… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:2903 abstract_call(int… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:2796 abstract_call_kn… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:338 abstract_call_gf… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:252 (::Compiler.var… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +1 1 @Compiler/…l:178 (::Compiler.var… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +2 1 @Compiler/…l:868 abstract_call_m… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +3 1 @Compiler/…l:898 abstract_call_m… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +4 1 @Compiler/…l:1348 const_prop_cal… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +5 1 @Compiler/…l:4476 typeinf(interp… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +6 1 @Compiler/…l:232 finish_nocycle(… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +7 1 @Compiler/…l:1019 optimize(inter… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +8 1 @Compiler/…l:1046 run_passes_ipo… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +9 1 @Compiler/…l:1033 run_passes_ipo… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +10 1 @Compiler/…l:2146 compact!(code:… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +11 1 @Compiler/…l:2148 compact!(code:… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +12 1 @Compiler/…l:1871 iterate(compac… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +13 1 @Compiler/…l:1871 iterate(compac… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +14 1 @Compiler/…l:1949 iterate_compac… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +15 1 @Compiler/…l:1478 process_node!(… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +16 1 @Compiler/…l:318 getindex From worker 87: 1╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +17 1 @Base/int.jl:522 <= From worker 87: ╎ ╎ ╎ 1 @Compiler/…tion.jl:4476 typeinf(interp::Compiler.NativeInte… From worker 87: ╎ ╎ ╎ 1 @Compiler/…nfer.jl:232 finish_nocycle(::Compiler.NativeInt… From worker 87: ╎ ╎ ╎ 1 @Compiler/…mize.jl:1019 optimize(interp::Compiler.NativeI… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…ize.jl:1046 run_passes_ipo_safe(ci::Core.Code… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…ize.jl:1033 run_passes_ipo_safe(ci::Core.Cod… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…ing.jl:79 ssa_inlining_pass!(ir::Compiler.I… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…ng.jl:663 batch_inline!(ir::Compiler.IRCod… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…ng.jl:445 ir_inline_item!(compact::Compil… From worker 87: ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…r.jl:1051 insert_node_here!(compact::Com… From worker 87: ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…r.jl:1060 insert_node_here!(compact::Co… From worker 87: ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…r.jl:972 recompute_newinst_flag(newins… From worker 87: 1╎ ╎ ╎ ╎ ╎ 1 @Compiler/….jl:373 stmt_effect_flags(𝕃ₒ::Compile… From worker 87: ╎ ╎ ╎ 1 @Compiler/…infer.jl:1562 typeinf_ext_toplevel(interp::Compile… From worker 87: 1╎ ╎ ╎ 1 @Compiler/…infer.jl:1555 add_codeinsts_to_jit!(interp::Compi… From worker 87: ╎ ╎ 9 @Distributed/…lize.jl:198 serialize(s::Distributed.ClusterSerial… From worker 87: ╎ ╎ 9 @Serialization/…n.jl:684 serialize(s::Distributed.ClusterSerial… From worker 87: ╎ ╎ 9 @Serialization/…n.jl:710 serialize_any(s::Distributed.ClusterS… From worker 87: ╎ ╎ ╎ 9 @Compiler/…infer.jl:1570 typeinf_ext_toplevel(mi::Core.Method… From worker 87: ╎ ╎ ╎ 9 @Compiler/…infer.jl:1561 typeinf_ext_toplevel(interp::Compil… From worker 87: ╎ ╎ ╎ 9 @Compiler/…nfer.jl:1378 typeinf_ext(interp::Compiler.Native… From worker 87: ╎ ╎ ╎ 1 @Compiler/…tion.jl:4460 typeinf(interp::Compiler.NativeInt… From worker 87: 1╎ ╎ ╎ 1 @Compiler/…tate.jl:1202 doworkloop(interp::Compiler.Nativ… From worker 87: ╎ ╎ ╎ 6 @Compiler/…tion.jl:4469 typeinf(interp::Compiler.NativeInt… From worker 87: ╎ ╎ ╎ 1 @Compiler/…tion.jl:4277 typeinf_local(interp::Compiler.Na… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…ion.jl:4043 update_bestguess!(interp::Compile… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…ion.jl:3860 widenreturn From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…on.jl:3864 widenreturn From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…on.jl:3917 widenreturn(𝕃ᵢ::Compiler.Condit… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…on.jl:3960 widenreturn From worker 87: 1╎ ╎ ╎ ╎ ╎ 1 @Compiler/…n.jl:3994 widenreturn_consts(rt::Any) From worker 87: ╎ ╎ ╎ 5 @Compiler/…tion.jl:4311 typeinf_local(interp::Compiler.Na… From worker 87: ╎ ╎ ╎ ╎ 5 @Compiler/…ion.jl:3762 abstract_eval_basic_statement From worker 87: ╎ ╎ ╎ ╎ 5 @Compiler/…ion.jl:3805 abstract_eval_basic_statement From worker 87: ╎ ╎ ╎ ╎ 5 @Compiler/…on.jl:3430 abstract_eval_statement_expr(int… From worker 87: ╎ ╎ ╎ ╎ 5 @Compiler/…on.jl:3074 abstract_eval_call(interp::Comp… From worker 87: ╎ ╎ ╎ ╎ 5 @Compiler/…on.jl:3056 abstract_call(interp::Compiler… From worker 87: ╎ ╎ ╎ ╎ ╎ 5 @Compiler/…n.jl:2896 abstract_call(interp::Compiler… From worker 87: ╎ ╎ ╎ ╎ ╎ 5 @Compiler/…n.jl:2903 abstract_call(interp::Compile… From worker 87: ╎ ╎ ╎ ╎ ╎ 1 @Compiler/….jl:2795 abstract_call_known(interp::C… From worker 87: 1╎ ╎ ╎ ╎ ╎ 1 @Compiler/…s.jl:55 argtypes_to_type(argtypes::Ve… From worker 87: ╎ ╎ ╎ ╎ ╎ 4 @Compiler/….jl:2796 abstract_call_known(interp::C… From worker 87: ╎ ╎ ╎ ╎ ╎ 4 @Compiler/….jl:338 abstract_call_gf_by_type(inte… From worker 87: ╎ ╎ ╎ ╎ ╎ 4 @Compiler/….jl:252 (::Compiler.var"#infercalls#… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 4 @Compiler/….jl:178 (::Compiler.var"#handle1#ab… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 4 @Compiler/…jl:868 abstract_call_method_with_c… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 4 @Compiler/…jl:898 abstract_call_method_with_… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 3 @Compiler/…l:1348 const_prop_call(interp::C… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:4469 typeinf(interp::Compiler… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:4311 typeinf_local(interp::C… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:3762 abstract_eval_basic_st… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:3805 abstract_eval_basic_s… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:3430 abstract_eval_statem… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:3074 abstract_eval_call(… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:3056 abstract_call(inte… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:2896 abstract_call(int… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:2903 abstract_call(in… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:2796 abstract_call_k… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:338 abstract_call_g… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +1 1 @Compiler/…l:252 (::Compiler.var… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +2 1 @Compiler/…l:178 (::Compiler.var… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +3 1 @Compiler/…l:868 abstract_call_m… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +4 1 @Compiler/…l:898 abstract_call_m… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +5 1 @Compiler/…l:1348 const_prop_cal… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +6 1 @Compiler/…l:4469 typeinf(interp… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +7 1 @Compiler/…l:4311 typeinf_local(… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +8 1 @Compiler/…l:3762 abstract_eval_… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +9 1 @Compiler/…l:3805 abstract_eval_… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +10 1 @Compiler/…l:3430 abstract_eval_… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +11 1 @Compiler/…l:3074 abstract_eval_… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +12 1 @Compiler/…l:3056 abstract_call(… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +13 1 @Compiler/…l:2896 abstract_call(… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +14 1 @Compiler/…l:2903 abstract_call(… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +15 1 @Compiler/…l:2683 abstract_call_… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +16 1 @Compiler/…l:1983 abstract_call_… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +17 1 @Compiler/…l:2851 builtin_tfunct… From worker 87: 1╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +18 1 @Compiler/…l:1068 getfield_tfunc… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ 2 @Compiler/…l:4476 typeinf(interp::Compiler… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:232 finish_nocycle(::Compile… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:1019 optimize(interp::Compi… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:1046 run_passes_ipo_safe(c… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:1033 run_passes_ipo_safe(… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:76 ssa_inlining_pass!(ir… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:1630 assemble_inline_to… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:1262 process_simple!(t… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:1072 call_sig(ir::Com… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:482 argextype(x::Any… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:482 argextype(x::An… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +1 1 @Compiler/…l:521 argextype(x::An… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +2 1 @Compiler/…l:3559 abstract_eval_… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +3 1 @Compiler/…l:3689 abstract_load_… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +4 1 @Compiler/…l:3674 scan_leaf_part… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +5 1 @Compiler/…l:3651 scan_specified… From worker 87: 1╎ ╎ ╎ ╎ ╎ ╎ ╎ ╎ +6 1 @Compiler/…l:3580 walk_binding_p… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:236 finish_nocycle(::Compile… From worker 87: ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:109 finish!(interp::Compile… From worker 87: 1╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:411 transform_result_for_l… From worker 87: 1╎ ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…l:1359 const_prop_call(interp::C… From worker 87: ╎ ╎ ╎ 2 @Compiler/…tion.jl:4476 typeinf(interp::Compiler.NativeInt… From worker 87: ╎ ╎ ╎ 2 @Compiler/…nfer.jl:232 finish_nocycle(::Compiler.NativeIn… From worker 87: ╎ ╎ ╎ ╎ 2 @Compiler/…ize.jl:1019 optimize(interp::Compiler.NativeI… From worker 87: ╎ ╎ ╎ ╎ 2 @Compiler/…ize.jl:1046 run_passes_ipo_safe(ci::Core.Cod… From worker 87: ╎ ╎ ╎ ╎ 2 @Compiler/…ze.jl:1033 run_passes_ipo_safe(ci::Core.Cod… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…ze.jl:1331 slot2reg(ir::Compiler.IRCode, c… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…sa.jl:894 construct_ssa!(ci::Core.CodeInf… From worker 87: ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…sa.jl:437 domsort_ssa!(ir::Compiler.IRCo… From worker 87: ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…r.jl:328 setindex! From worker 87: 1╎ ╎ ╎ ╎ ╎ 1 @Base/…tials.jl:973 setindex! From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…ng.jl:79 ssa_inlining_pass!(ir::Compiler.I… From worker 87: ╎ ╎ ╎ ╎ 1 @Compiler/…ng.jl:663 batch_inline!(ir::Compiler.IRCo… From worker 87: ╎ ╎ ╎ ╎ ╎ 1 @Compiler/…ng.jl:427 ir_inline_item!(compact::Compi… From worker 87: 1╎ ╎ ╎ ╎ ╎ 1 @Base/int.jl:87 + From worker 87: [ Info: running task 23 on process 86 (Schedulers-primary-S6HJ4WCf); 2 workers total; 84 tasks left in task-pool. ┌ Warning: Sending SIGQUIT to worker 87 └ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/managers.jl:760 From worker 87: From worker 87: [357] signal 3: Quit From worker 87: in expression starting at none:0 From worker 87: getAddress at /source/usr/include/llvm/Object/ObjectFile.h:465 [inlined] From worker 87: get_function_name_and_base at /source/src/debuginfo.cpp:751 [inlined] From worker 87: jl_dylib_DI_for_fptr at /source/src/debuginfo.cpp:1150 From worker 87: jl_getDylibFunctionInfo at /source/src/debuginfo.cpp:1182 [inlined] From worker 87: jl_getFunctionInfo_impl at /source/src/debuginfo.cpp:1278 From worker 87: ijl_lookup_code_address at /source/src/stackwalk.c:616 From worker 87: lookup at ./stacktraces.jl:110 From worker 87: _lookup_corrected at /source/usr/share/julia/stdlib/v1.13/Profile/src/Profile.jl:489 From worker 87: #getdict!##0 at /source/usr/share/julia/stdlib/v1.13/Profile/src/Profile.jl:478 From worker 87: unknown function (ip: 0x78d1539660f7) at (unknown file) From worker 87: _jl_invoke at /source/src/gf.c:3693 [inlined] From worker 87: ijl_apply_generic at /source/src/gf.c:3893 From worker 87: jl_apply at /source/src/julia.h:2344 [inlined] From worker 87: start_task at /source/src/task.c:1249 From worker 87: unknown function (ip: (nil)) at (unknown file) From worker 87: Allocations: 12786607 (Pool: 12786529; Big: 78); GC: 7 ====================================================================================== Information request received. A stacktrace will print followed by a 1.0 second profile ====================================================================================== cmd: /opt/julia/bin/julia 1 running 0 of 1 signal (10): User defined signal 1 epoll_pwait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) uv__io_poll at /workspace/srcdir/libuv/src/unix/linux.c:1404 uv_run at /workspace/srcdir/libuv/src/unix/core.c:430 ijl_task_get_next at /source/src/scheduler.c:457 [ Info: running task 24 on process 89 (Schedulers-primary-S6HJ4WCf); 2 workers total; 83 tasks left in task-pool. wait at ./task.jl:1213 wait_forever at ./task.jl:1150 jfptr_wait_forever_42615.1 at /opt/julia/lib/julia/sys.so (unknown line) _jl_invoke at /source/src/gf.c:3693 [inlined] ijl_apply_generic at /source/src/gf.c:3893 jl_apply at /source/src/julia.h:2344 [inlined] start_task at /source/src/task.c:1249 unknown function (ip: (nil)) at (unknown file) ============================================================== Profile collected. A report will print at the next yield point ============================================================== [ Info: running task 25 on process 86 (Schedulers-primary-S6HJ4WCf); 2 workers total; 82 tasks left in task-pool. [ Info: running task 26 on process 89 (Schedulers-primary-S6HJ4WCf); 2 workers total; 81 tasks left in task-pool. ┌ Warning: There were no samples collected in one or more groups. │ This may be due to idle threads, or you may need to run your │ program longer (perhaps by running it multiple times), │ or adjust the delay between samples with `Profile.init()`. └ @ Profile /opt/julia/share/julia/stdlib/v1.13/Profile/src/Profile.jl:1362 Overhead ╎ [+additional indent] Count File:Line Function ========================================================= Thread 1 (default) Task 0x0000720b8a91e950 Total snapshots: 54. Utilization: 0% ╎54 @Base/task.jl:1150 wait_forever() 53╎ 54 @Base/task.jl:1213 wait() ┌ Warning: pid=86 (Schedulers-primary-S6HJ4WCf), checkpoint=/tmp/jl_mHp9Iv/checkpoint-oO1X2X-339, task loop, caught exception during save_checkpoint └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1410 ┌ Warning: TaskFailedException │ │ nested task error: On worker 86: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.8461689424218354 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ./none:-1 [inlined] │ [3] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:0 │ [4] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [5] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 │ error type: TaskFailedException │ 1-element ExceptionStack: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::typeof(Schedulers.save_checkpoint), ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:298 │ [5] remotecall_wait_timeout(::Vector{Float64}, ::Int64, ::Float64, ::Nothing, ::Function, ::Function, ::Int64, ::Function, ::Int64, ::typeof(test_save_checkpoint), ::Vararg{Any}) │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:287 │ [6] (::Schedulers.var"#134#135"{Vector{Float32}, @Kwargs{}, typeof(foo7b), Schedulers.ElasticLoop{typeof(addprocs), Schedulers.var"#14#15", typeof(nworkers), Schedulers.var"#92#93"{Int64}, Schedulers.var"#SchedulerOptions##14#SchedulerOptions##15", Schedulers.var"#SchedulerOptions##24#SchedulerOptions##25", Schedulers.var"#SchedulerOptions##26#SchedulerOptions##27", Schedulers.var"#SchedulerOptions##16#SchedulerOptions##17", Int64, String}, Dict{String, Any}, SchedulerOptions{String}, Tuple{Int64, Int64}, Vector{Float64}, Vector{Float64}, Vector{Float64}, Vector{Any}, Dict{Int64, Future}, Int64})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1405 │ │ nested task error: On worker 86: │ TaskFailedException │ Stacktrace: │ [1] #wait#583 │ @ ./task.jl:363 [inlined] │ [2] wait │ @ ./task.jl:360 [inlined] │ [3] fetch │ @ ./task.jl:525 [inlined] │ [4] #default_threadpool_checkpoint_call#25 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:284 │ [5] default_threadpool_checkpoint_call │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:250 │ [6] #handle_msg##8 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:303 │ [7] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:70 │ [8] run_work_thunk │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:79 │ [9] #schedule_call##2 │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/process_messages.jl:88 │ │ nested task error: foo,x=0.8461689424218354 │ Stacktrace: │ [1] error │ @ ./error.jl:44 │ [2] test_save_checkpoint │ @ ./none:-1 [inlined] │ [3] test_save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/test/runtests.jl:0 │ [4] save_checkpoint │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:1785 │ [5] #28 │ @ ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:262 │ Stacktrace: │ [1] remotecall_wait(::Function, ::Distributed.Worker, ::Nothing, ::Vararg{Any}; kwargs::@Kwargs{}) │ @ Distributed /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:507 │ [2] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:497 [inlined] │ [3] remotecall_wait │ @ /opt/julia/share/julia/stdlib/v1.13/Distributed/src/remotecall.jl:519 [inlined] │ [4] (::Schedulers.var"#34#35"{@Kwargs{}, Nothing, Schedulers.var"#140#141", Schedulers.var"#142#143", Int64, typeof(Schedulers.save_checkpoint), Int64, Tuple{typeof(test_save_checkpoint), typeof(fetch), String, Future, DataType}})() │ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:288 └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:23 ┌ Warning: too many failures on process with id=86 (Schedulers-primary-S6HJ4WCf), removing from process list └ @ Schedulers ~/.julia/packages/Schedulers/JBB7J/src/Schedulers.jl:393 [ Info: running task 27 on process 89 (Schedulers-primary-S6HJ4WCf); 2 workers total; 81 tasks left in task-pool. [15] signal 15: Terminated in expression starting at /home/pkgeval/.julia/packages/Schedulers/JBB7J/test/runtests.jl:433 epoll_pwait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) uv__io_poll at /workspace/srcdir/libuv/src/unix/linux.c:1404 uv_run at /workspace/srcdir/libuv/src/unix/core.c:430 ijl_task_get_next at /source/src/scheduler.c:457 [1] signal 15: Terminated in expression starting at /PkgEval.jl/scripts/evaluate.jl:210 epoll_pwait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) uv__io_poll at /workspace/srcdir/libuv/src/unix/linux.c:1404 uv_run at /workspace/srcdir/libuv/src/unix/core.c:430 ijl_task_get_next at /source/src/scheduler.c:457 [380] signal 15: Terminated in expression starting at none:0 [381] signal 15: Terminated in expression starting at none:0 [379] signal 15: Terminated in expression starting at none:0 _ZN4llvm13StringMapImpl15LookupBucketForENS_9StringRefEj at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) wait at ./task.jl:1213 wait at ./task.jl:1213 _ZL18computeForAddCarryRKN4llvm9KnownBitsES2_bb at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm9KnownBits16computeForAddSubEbbbRKS0_S2_ at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm9StringMapIPNS_12MCSectionELFENS_15MallocAllocatorEE21try_emplace_with_hashIJS2_EEESt4pairINS_17StringMapIteratorIS2_EEbENS_9StringRefEjDpOT_ at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZL19foldTwoEntryPHINodePN4llvm7PHINodeERKNS_19TargetTransformInfoEPNS_14DomTreeUpdaterEPNS_15AssumptionCacheERKNS_10DataLayoutEb at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm11simplifyCFGEPNS_10BasicBlockERKNS_19TargetTransformInfoEPNS_14DomTreeUpdaterERKNS_18SimplifyCFGOptionsENS_8ArrayRefINS_6WeakVHEEE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm9MCContext13getELFSectionERKNS_5TwineEjjjPKNS_11MCSymbolELFEbjS6_ at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) wait_forever at ./task.jl:1150 wait_forever at ./task.jl:1150 _ZZL28computeKnownBitsFromOperatorPKN4llvm8OperatorERKNS_5APIntERNS_9KnownBitsEjRKNS_13SimplifyQueryEENKUlS6_E3_clES6_ at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZL22iterativelySimplifyCFGRN4llvm8FunctionERKNS_19TargetTransformInfoEPNS_14DomTreeUpdaterERKNS_18SimplifyCFGOptionsE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZL25selectELFSectionForGlobalRN4llvm9MCContextEPKNS_12GlobalObjectENS_11SectionKindERNS_7ManglerERKNS_13TargetMachineEbjPjPKNS_11MCSymbolELFE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZNK4llvm27TargetLoweringObjectFileELF22SelectSectionForGlobalEPKNS_12GlobalObjectENS_11SectionKindERKNS_13TargetMachineE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm10AsmPrinter18emitFunctionHeaderEv at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm10AsmPrinter16emitFunctionBodyEv at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) jfptr_wait_forever_42615.1 at /opt/julia/lib/julia/sys.so (unknown line) _jl_invoke at /source/src/gf.c:3693 [inlined] ijl_apply_generic at /source/src/gf.c:3893 jfptr_wait_forever_42615.1 at /opt/julia/lib/julia/sys.so (unknown line) _jl_invoke at /source/src/gf.c:3693 [inlined] ijl_apply_generic at /source/src/gf.c:3893 jl_apply at /source/src/julia.h:2344 [inlined] start_task at /source/src/task.c:1249 unknown function (ip: (nil)) at (unknown file) Allocations: 56520220 (Pool: 56518961; Big: 1259); GC: 70 jl_apply at /source/src/julia.h:2344 [inlined] start_task at /source/src/task.c:1249 unknown function (ip: (nil)) at (unknown file) Allocations: 10688834 (Pool: 10688699; Big: 135); GC: 9 _ZL16computeKnownBitsPKN4llvm5ValueERKNS_5APIntERNS_9KnownBitsEjRKNS_13SimplifyQueryE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZL23simplifyFunctionCFGImplRN4llvm8FunctionERKNS_19TargetTransformInfoEPNS_13DominatorTreeERKNS_18SimplifyCFGOptionsE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16computeKnownBitsEPKNS_5ValueERNS_9KnownBitsEjRKNS_13SimplifyQueryE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16computeKnownBitsEPKNS_5ValueERKNS_10DataLayoutEjPNS_15AssumptionCacheEPKNS_11InstructionEPKNS_13DominatorTreeEb at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm26getOrEnforceKnownAlignmentEPNS_5ValueENS_10MaybeAlignERKNS_10DataLayoutEPKNS_11InstructionEPNS_15AssumptionCacheEPKNS_13DominatorTreeE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm13X86AsmPrinter20runOnMachineFunctionERNS_15MachineFunctionE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16InstCombinerImpl22SimplifyAnyMemTransferEPNS_18AnyMemTransferInstE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZL19simplifyFunctionCFGRN4llvm8FunctionERKNS_19TargetTransformInfoEPNS_13DominatorTreeERKNS_18SimplifyCFGOptionsE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm15SimplifyCFGPass3runERNS_8FunctionERNS_15AnalysisManagerIS1_JEEE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm19MachineFunctionPass13runOnFunctionERNS_8FunctionE.part.0 at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm3orc14SimpleCompilerclERNS_6ModuleE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm16InstCombinerImpl13visitCallInstERNS_8CallInstE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) run at /source/usr/include/llvm/IR/PassManagerInternal.h:91 _ZN4llvm11PassManagerINS_8FunctionENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) run at /source/usr/include/llvm/IR/PassManagerInternal.h:91 _ZN4llvm27ModuleToFunctionPassAdaptor3runERNS_6ModuleERNS_15AnalysisManagerIS1_JEEE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) run at /source/usr/include/llvm/IR/PassManagerInternal.h:91 _ZN4llvm11PassManagerINS_6ModuleENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) operator() at /source/src/jitlayers.cpp:1620 run at /source/src/pipeline.cpp:791 addModule at /source/src/jitlayers.cpp:2101 jl_compile_codeinst_now at /source/src/jitlayers.cpp:683 jl_compile_codeinst_impl at /source/src/jitlayers.cpp:883 jl_compile_method_internal at /source/src/gf.c:3213 _jl_invoke at /source/src/gf.c:3685 [inlined] ijl_apply_generic at /source/src/gf.c:3893 jl_apply at /source/src/julia.h:2344 [inlined] jl_f_invokelatest at /source/src/builtins.c:877 _ZN4llvm16InstCombinerImpl3runEv at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) operator() at /source/src/jitlayers.cpp:1518 withModuleDo<(anonymous namespace)::sizedOptimizerT::operator()(llvm::orc::ThreadSafeModule) [with long unsigned int N = 4]:: > at /source/usr/include/llvm/ExecutionEngine/Orc/ThreadSafeModule.h:136 [inlined] operator() at /source/src/jitlayers.cpp:1479 [inlined] operator() at /source/src/jitlayers.cpp:1631 [inlined] addModule at /source/src/jitlayers.cpp:2088 jl_compile_codeinst_now at /source/src/jitlayers.cpp:683 jl_compile_codeinst_impl at /source/src/jitlayers.cpp:883 jl_compile_method_internal at /source/src/gf.c:3213 _jl_invoke at /source/src/gf.c:3685 [inlined] ijl_apply_generic at /source/src/gf.c:3893 jl_apply at /source/src/julia.h:2344 [inlined] jl_f_invokelatest at /source/src/builtins.c:877 _ZL31combineInstructionsOverFunctionRN4llvm8FunctionERNS_19InstructionWorklistEPNS_9AAResultsERNS_15AssumptionCacheERNS_17TargetLibraryInfoERNS_19TargetTransformInfoERNS_13DominatorTreeERNS_25OptimizationRemarkEmitterEPNS_18BlockFrequencyInfoEPNS_21BranchProbabilityInfoEPNS_18ProfileSummaryInfoERKNS_18InstCombineOptionsE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) _ZN4llvm15InstCombinePass3runERNS_8FunctionERNS_15AnalysisManagerIS1_JEEE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) exec_options at ./client.jl:266 run at /source/usr/include/llvm/IR/PassManagerInternal.h:91 _ZN4llvm11PassManagerINS_8FunctionENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) run at /source/usr/include/llvm/IR/PassManagerInternal.h:91 _ZN4llvm27ModuleToFunctionPassAdaptor3runERNS_6ModuleERNS_15AnalysisManagerIS1_JEEE at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) run at /source/usr/include/llvm/IR/PassManagerInternal.h:91 _ZN4llvm11PassManagerINS_6ModuleENS_15AnalysisManagerIS1_JEEEJEE3runERS1_RS3_ at /opt/julia/bin/../lib/julia/libLLVM.so.20.1jl (unknown line) run at /source/src/pipeline.cpp:791 _start at ./client.jl:557 exec_options at ./client.jl:266 jfptr__start_72357.1 at /opt/julia/lib/julia/sys.so (unknown line) _jl_invoke at /source/src/gf.c:3693 [inlined] ijl_apply_generic at /source/src/gf.c:3893 jl_apply at /source/src/julia.h:2344 [inlined] true_main at /source/src/jlapi.c:964 jl_repl_entrypoint at /source/src/jlapi.c:1132 main at /source/cli/loader_exe.c:58 unknown function (ip: 0x7b241a5db249) at /lib/x86_64-linux-gnu/libc.so.6 __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) unknown function (ip: 0x4010b8) at /workspace/srcdir/glibc-2.17/csu/../sysdeps/x86_64/start.S unknown function (ip: (nil)) at (unknown file) Allocations: 1734393 (Pool: 1734390; Big: 3); GC: 2 operator() at /source/src/jitlayers.cpp:1518 withModuleDo<(anonymous namespace)::sizedOptimizerT::operator()(llvm::orc::ThreadSafeModule) [with long unsigned int N = 4]:: > at /source/usr/include/llvm/ExecutionEngine/Orc/ThreadSafeModule.h:136 [inlined] operator() at /source/src/jitlayers.cpp:1479 [inlined] operator() at /source/src/jitlayers.cpp:1631 [inlined] addModule at /source/src/jitlayers.cpp:2088 jl_compile_codeinst_now at /source/src/jitlayers.cpp:683 jl_compile_codeinst_impl at /source/src/jitlayers.cpp:883 jl_compile_method_internal at /source/src/gf.c:3213 _jl_invoke at /source/src/gf.c:3685 [inlined] ijl_apply_generic at /source/src/gf.c:3893 _start at ./client.jl:557 jl_apply at /source/src/julia.h:2344 [inlined] jl_f_invokelatest at /source/src/builtins.c:877 jfptr__start_72357.1 at /opt/julia/lib/julia/sys.so (unknown line) _jl_invoke at /source/src/gf.c:3693 [inlined] ijl_apply_generic at /source/src/gf.c:3893 jl_apply at /source/src/julia.h:2344 [inlined] true_main at /source/src/jlapi.c:964 jl_repl_entrypoint at /source/src/jlapi.c:1132 main at /source/cli/loader_exe.c:58 unknown function (ip: 0x7a284c173249) at /lib/x86_64-linux-gnu/libc.so.6 __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) unknown function (ip: 0x4010b8) at /workspace/srcdir/glibc-2.17/csu/../sysdeps/x86_64/start.S unknown function (ip: (nil)) at (unknown file) Allocations: 1735365 (Pool: 1735362; Big: 3); GC: 2 exec_options at ./client.jl:266 _start at ./client.jl:557 jfptr__start_72357.1 at /opt/julia/lib/julia/sys.so (unknown line) _jl_invoke at /source/src/gf.c:3693 [inlined] ijl_apply_generic at /source/src/gf.c:3893 jl_apply at /source/src/julia.h:2344 [inlined] true_main at /source/src/jlapi.c:964 jl_repl_entrypoint at /source/src/jlapi.c:1132 main at /source/cli/loader_exe.c:58 unknown function (ip: 0x71757e4a2249) at /lib/x86_64-linux-gnu/libc.so.6 __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) unknown function (ip: 0x4010b8) at /workspace/srcdir/glibc-2.17/csu/../sysdeps/x86_64/start.S unknown function (ip: (nil)) at (unknown file) Allocations: 1735140 (Pool: 1735137; Big: 3); GC: 2 PkgEval terminated after 2791.16s: test duration exceeded the time limit