Package evaluation of Dagger on Julia 1.11.5 (32ac370b68*) started at 2025-06-29T12:51:45.410 ################################################################################ # Set-up # Installing PkgEval dependencies (TestEnv)... Set-up completed after 8.7s ################################################################################ # Installation # Installing Dagger... Resolving package versions... Updating `~/.julia/environments/v1.11/Project.toml` [d58978e5] + Dagger v0.18.16 Updating `~/.julia/environments/v1.11/Manifest.toml` [1520ce14] + AbstractTrees v0.4.5 [79e6a3ab] + Adapt v4.3.0 [66dad0bd] + AliasTables v1.1.3 [ec485272] + ArnoldiMethod v0.4.0 [34da2185] + Compat v4.16.0 [5060bff5] + ConcurrentCollections v0.1.1 [d58978e5] + Dagger v0.18.16 [9a962f9c] + DataAPI v1.16.0 [864edb3b] + DataStructures v0.18.22 [fab6aee4] + DistributedNext v1.0.0 [31c24e10] + Distributions v0.25.120 [ffbed154] + DocStringExtensions v0.9.5 [1a297f60] + FillArrays v1.13.0 [86223c79] + Graphs v1.13.0 [076d061b] + HashArrayMappedTries v0.2.0 [34004b35] + HypergeometricFunctions v0.3.28 [d25df0c9] + Inflate v0.1.5 [92d709cd] + IrrationalConstants v0.2.4 [692b3bcd] + JLLWrappers v1.7.0 [2ab3a3ac] + LogExpFunctions v0.3.29 [1914dd2f] + MacroTools v0.5.16 [f9f48841] + MemPool v0.4.13 [e1d29d7a] + Missings v1.2.0 [a15396b6] + OnlineStats v1.7.1 [925886fa] + OnlineStatsBase v1.7.1 [bac558e1] + OrderedCollections v1.8.1 [90014a1f] + PDMats v0.11.35 ⌅ [aea7be01] + PrecompileTools v1.2.1 [21216c6a] + Preferences v1.4.3 [43287f4e] + PtrArrays v1.3.0 [1fd47b50] + QuadGK v2.11.2 [3cdcf5f2] + RecipesBase v1.3.4 [189a3867] + Reexport v1.2.2 [ae029012] + Requires v1.3.1 [79098fc4] + Rmath v0.8.0 [7e506255] + ScopedValues v1.3.0 [699a6c99] + SimpleTraits v0.9.4 [a2af1166] + SortingAlgorithms v1.2.1 [276daf66] + SpecialFunctions v2.5.1 [90137ffa] + StaticArrays v1.9.13 [1e83bf80] + StaticArraysCore v1.4.3 [10745b16] + Statistics v1.11.1 [82ae8749] + StatsAPI v1.7.1 [2913bbd2] + StatsBase v0.34.5 [4c63d2b9] + StatsFuns v1.5.0 [ed4db957] + TaskLocalValues v0.1.2 [a526e669] + TimespanLogging v0.1.1 [efe28fd5] + OpenSpecFun_jll v0.5.6+0 [f50d1b31] + Rmath_jll v0.5.1+0 [56f22d72] + Artifacts v1.11.0 [2a0f44e3] + Base64 v1.11.0 [ade2ca70] + Dates v1.11.0 [8ba89e20] + Distributed v1.11.0 [b77e0a4c] + InteractiveUtils v1.11.0 [8f399da3] + Libdl v1.11.0 [37e2e46d] + LinearAlgebra v1.11.0 [56ddb016] + Logging v1.11.0 [d6f4376e] + Markdown v1.11.0 [a63ad114] + Mmap v1.11.0 [de0858da] + Printf v1.11.0 [9abbd945] + Profile v1.11.0 [9a3f8284] + Random v1.11.0 [ea8e919c] + SHA v0.7.0 [9e88b42a] + Serialization v1.11.0 [1a1011a3] + SharedArrays v1.11.0 [6462fe0b] + Sockets v1.11.0 [2f01184e] + SparseArrays v1.11.0 [4607b0f0] + SuiteSparse [fa267f1f] + TOML v1.0.3 [cf7118a7] + UUIDs v1.11.0 [4ec0a83e] + Unicode v1.11.0 [e66e0078] + CompilerSupportLibraries_jll v1.1.1+0 [4536629a] + OpenBLAS_jll v0.3.27+1 [05823500] + OpenLibm_jll v0.8.5+0 [bea87d4a] + SuiteSparse_jll v7.7.0+0 [8e850b90] + libblastrampoline_jll v5.11.0+0 Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m` Installation completed after 4.33s ################################################################################ # Precompilation # Precompiling PkgEval dependencies... Precompiling package dependencies... Precompilation completed after 573.61s ################################################################################ # Testing # Testing Dagger Status `/tmp/jl_Z9xAQO/Project.toml` [c7e460c6] ArgParse v1.2.0 [5ae59095] Colors v0.13.1 [992eb4ea] CondaPkg v0.2.29 [d58978e5] Dagger v0.18.16 [a93c6f00] DataFrames v1.7.0 [31c24e10] Distributions v0.25.120 [f526b714] GraphViz v0.2.0 [86223c79] Graphs v1.13.0 [0f8b85d8] JSON3 v1.14.3 [f9f48841] MemPool v0.4.13 [a15396b6] OnlineStats v1.7.1 [91a5bcdd] Plots v1.40.14 [21216c6a] Preferences v1.4.3 [6099a3de] PythonCall v0.9.25 [10745b16] Statistics v1.11.1 [a526e669] TimespanLogging v0.1.1 [8ba89e20] Distributed v1.11.0 [37e2e46d] LinearAlgebra v1.11.0 [44cfe95a] Pkg v1.11.0 [9a3f8284] Random v1.11.0 [9e88b42a] Serialization v1.11.0 [1a1011a3] SharedArrays v1.11.0 [2f01184e] SparseArrays v1.11.0 [8dfed614] Test v1.11.0 [cf7118a7] UUIDs v1.11.0 Status `/tmp/jl_Z9xAQO/Manifest.toml` [1520ce14] AbstractTrees v0.4.5 [79e6a3ab] Adapt v4.3.0 [66dad0bd] AliasTables v1.1.3 [c7e460c6] ArgParse v1.2.0 [ec485272] ArnoldiMethod v0.4.0 [d1d4a3ce] BitFlags v0.1.9 [944b1d66] CodecZlib v0.7.8 [35d6a980] ColorSchemes v3.29.0 [3da002f7] ColorTypes v0.12.1 [c3611d14] ColorVectorSpace v0.11.0 [5ae59095] Colors v0.13.1 [34da2185] Compat v4.16.0 [5060bff5] ConcurrentCollections v0.1.1 [f0e56b4a] ConcurrentUtilities v2.5.0 [992eb4ea] CondaPkg v0.2.29 [d38c429a] Contour v0.6.3 [a8cc5b0e] Crayons v4.1.1 [d58978e5] Dagger v0.18.16 [9a962f9c] DataAPI v1.16.0 [a93c6f00] DataFrames v1.7.0 [864edb3b] DataStructures v0.18.22 [e2d170a0] DataValueInterfaces v1.0.0 [8bb1440f] DelimitedFiles v1.9.1 [fab6aee4] DistributedNext v1.0.0 [31c24e10] Distributions v0.25.120 [ffbed154] DocStringExtensions v0.9.5 [460bff9d] ExceptionUnwrapping v0.1.11 [c87230d0] FFMPEG v0.4.2 [5789e2e9] FileIO v1.17.0 [1a297f60] FillArrays v1.13.0 [53c48c17] FixedPointNumbers v0.8.5 [1fa38f19] Format v1.3.7 [28b8d3ca] GR v0.73.17 [f526b714] GraphViz v0.2.0 [86223c79] Graphs v1.13.0 [42e2da0e] Grisu v1.0.2 [cd3eb016] HTTP v1.10.17 [076d061b] HashArrayMappedTries v0.2.0 [34004b35] HypergeometricFunctions v0.3.28 [d25df0c9] Inflate v0.1.5 [842dd82b] InlineStrings v1.4.4 [41ab1584] InvertedIndices v1.3.1 [92d709cd] IrrationalConstants v0.2.4 [82899510] IteratorInterfaceExtensions v1.0.0 [1019f520] JLFzf v0.1.11 [692b3bcd] JLLWrappers v1.7.0 [682c06a0] JSON v0.21.4 [0f8b85d8] JSON3 v1.14.3 [b964fa9f] LaTeXStrings v1.4.0 [23fbe1c1] Latexify v0.16.8 [2ab3a3ac] LogExpFunctions v0.3.29 [e6f89c97] LoggingExtras v1.1.0 [1914dd2f] MacroTools v0.5.16 [739be429] MbedTLS v1.1.9 [442fdcdd] Measures v0.3.2 [f9f48841] MemPool v0.4.13 [0b3b1443] MicroMamba v0.1.14 [e1d29d7a] Missings v1.2.0 [77ba4419] NaNMath v1.1.3 [a15396b6] OnlineStats v1.7.1 [925886fa] OnlineStatsBase v1.7.1 [4d8831e6] OpenSSL v1.5.0 [bac558e1] OrderedCollections v1.8.1 [90014a1f] PDMats v0.11.35 [69de0a69] Parsers v2.8.3 [fa939f87] Pidfile v1.3.0 [ccf2f8ad] PlotThemes v3.3.0 [995b91a9] PlotUtils v1.4.3 [91a5bcdd] Plots v1.40.14 [2dfb63ee] PooledArrays v1.4.3 ⌅ [aea7be01] PrecompileTools v1.2.1 [21216c6a] Preferences v1.4.3 [08abe8d2] PrettyTables v2.4.0 [43287f4e] PtrArrays v1.3.0 [6099a3de] PythonCall v0.9.25 [1fd47b50] QuadGK v2.11.2 [3cdcf5f2] RecipesBase v1.3.4 [01d81517] RecipesPipeline v0.6.12 [189a3867] Reexport v1.2.2 [05181044] RelocatableFolders v1.0.1 [ae029012] Requires v1.3.1 [79098fc4] Rmath v0.8.0 [7e506255] ScopedValues v1.3.0 [6c6a2e73] Scratch v1.3.0 [91c51154] SentinelArrays v1.4.8 [992d4aef] Showoff v1.0.3 [777ac1f9] SimpleBufferStream v1.2.0 [699a6c99] SimpleTraits v0.9.4 [a2af1166] SortingAlgorithms v1.2.1 [276daf66] SpecialFunctions v2.5.1 [860ef19b] StableRNGs v1.0.3 [90137ffa] StaticArrays v1.9.13 [1e83bf80] StaticArraysCore v1.4.3 [10745b16] Statistics v1.11.1 [82ae8749] StatsAPI v1.7.1 [2913bbd2] StatsBase v0.34.5 [4c63d2b9] StatsFuns v1.5.0 [892a3eda] StringManipulation v0.4.1 [856f2bd8] StructTypes v1.11.0 [3783bdb8] TableTraits v1.0.1 [bd369af6] Tables v1.12.1 [ed4db957] TaskLocalValues v0.1.2 [62fd8b95] TensorCore v0.1.1 [b718987f] TextWrap v1.0.2 [a526e669] TimespanLogging v0.1.1 [3bb67fe8] TranscodingStreams v0.11.3 [5c2747f8] URIs v1.6.1 [1cfade01] UnicodeFun v0.4.1 [1986cc42] Unitful v1.23.1 [45397f5d] UnitfulLatexify v1.7.0 [e17b2a0c] UnsafePointers v1.0.0 [41fe7b60] Unzip v0.2.0 [6e34b625] Bzip2_jll v1.0.9+0 [83423d85] Cairo_jll v1.18.5+0 [ee1fde0b] Dbus_jll v1.16.2+0 [2702e6a9] EpollShim_jll v0.0.20230411+1 [2e619515] Expat_jll v2.6.5+0 ⌅ [b22a6f82] FFMPEG_jll v4.4.4+1 [a3f928ae] Fontconfig_jll v2.16.0+0 [d7e528f0] FreeType2_jll v2.13.4+0 [559328eb] FriBidi_jll v1.0.17+0 [0656b61e] GLFW_jll v3.4.0+2 [d2c73de3] GR_jll v0.73.17+0 [b0724c58] GettextRuntime_jll v0.22.4+0 [7746bdde] Glib_jll v2.84.3+0 [3b182d85] Graphite2_jll v1.3.15+0 [3c863552] Graphviz_jll v2.50.0+1 [2e76f6c2] HarfBuzz_jll v8.5.1+0 [aacddb02] JpegTurbo_jll v3.1.1+0 [c1c5ebd0] LAME_jll v3.100.2+0 [88015f11] LERC_jll v4.0.1+0 [1d63c593] LLVMOpenMP_jll v18.1.8+0 [dd4b983a] LZO_jll v2.10.3+0 [e9f186c6] Libffi_jll v3.4.7+0 [7e76a0d4] Libglvnd_jll v1.7.1+1 [94ce4f54] Libiconv_jll v1.18.0+0 [4b2f31a3] Libmount_jll v2.41.0+0 [89763e89] Libtiff_jll v4.7.1+0 [38a345b3] Libuuid_jll v2.41.0+0 [e7412a2a] Ogg_jll v1.3.5+1 [458c3c95] OpenSSL_jll v3.5.0+0 [efe28fd5] OpenSpecFun_jll v0.5.6+0 [91d4177d] Opus_jll v1.3.3+0 [36c8627f] Pango_jll v1.56.3+0 ⌅ [30392449] Pixman_jll v0.44.2+0 [c0090381] Qt6Base_jll v6.8.2+1 [629bc702] Qt6Declarative_jll v6.8.2+1 [ce943373] Qt6ShaderTools_jll v6.8.2+1 [e99dba38] Qt6Wayland_jll v6.8.2+1 [f50d1b31] Rmath_jll v0.5.1+0 [a44049a8] Vulkan_Loader_jll v1.3.243+0 [a2964d1f] Wayland_jll v1.23.1+1 ⌅ [02c8fc9c] XML2_jll v2.13.6+1 [ffd25f8a] XZ_jll v5.8.1+0 [f67eecfb] Xorg_libICE_jll v1.1.2+0 [c834827a] Xorg_libSM_jll v1.2.6+0 [4f6342f7] Xorg_libX11_jll v1.8.12+0 [0c0b7dd1] Xorg_libXau_jll v1.0.13+0 [935fb764] Xorg_libXcursor_jll v1.2.4+0 [a3789734] Xorg_libXdmcp_jll v1.1.6+0 [1082639a] Xorg_libXext_jll v1.3.7+0 [d091e8ba] Xorg_libXfixes_jll v6.0.1+0 [a51aa0fd] Xorg_libXi_jll v1.8.3+0 [d1454406] Xorg_libXinerama_jll v1.1.6+0 [ec84b674] Xorg_libXrandr_jll v1.5.5+0 [ea2f1a96] Xorg_libXrender_jll v0.9.12+0 [c7cfdc94] Xorg_libxcb_jll v1.17.1+0 [cc61e674] Xorg_libxkbfile_jll v1.1.3+0 [e920d4aa] Xorg_xcb_util_cursor_jll v0.1.5+0 [12413925] Xorg_xcb_util_image_jll v0.4.1+0 [2def613f] Xorg_xcb_util_jll v0.4.1+0 [975044d2] Xorg_xcb_util_keysyms_jll v0.4.1+0 [0d47668e] Xorg_xcb_util_renderutil_jll v0.3.10+0 [c22f9ab0] Xorg_xcb_util_wm_jll v0.4.2+0 [35661453] Xorg_xkbcomp_jll v1.4.7+0 [33bec58e] Xorg_xkeyboard_config_jll v2.44.0+0 [c5fb5394] Xorg_xtrans_jll v1.6.0+0 [3161d3a3] Zstd_jll v1.5.7+1 [35ca27e7] eudev_jll v3.2.14+0 [214eeab7] fzf_jll v0.61.1+0 [a4ae2306] libaom_jll v3.11.0+0 [0ac62f75] libass_jll v0.15.2+0 [1183f4f0] libdecor_jll v0.2.2+0 [2db6ffa8] libevdev_jll v1.13.4+0 [f638f0a6] libfdk_aac_jll v2.0.3+0 [36db933b] libinput_jll v1.28.1+0 [b53b4c65] libpng_jll v1.6.49+0 [f27f6e37] libvorbis_jll v1.3.7+2 [f8abcde7] micromamba_jll v1.5.8+0 [009596ad] mtdev_jll v1.1.7+0 [4d7b5844] pixi_jll v0.41.3+0 ⌅ [1270edf5] x264_jll v2021.5.5+0 ⌅ [dfaa095f] x265_jll v3.5.0+0 [d8fb68d0] xkbcommon_jll v1.9.2+0 [0dad84c5] ArgTools v1.1.2 [56f22d72] Artifacts v1.11.0 [2a0f44e3] Base64 v1.11.0 [ade2ca70] Dates v1.11.0 [8ba89e20] Distributed v1.11.0 [f43a241f] Downloads v1.6.0 [7b1f6079] FileWatching v1.11.0 [9fa8497b] Future v1.11.0 [b77e0a4c] InteractiveUtils v1.11.0 [4af54fe1] LazyArtifacts v1.11.0 [b27032c2] LibCURL v0.6.4 [76f85450] LibGit2 v1.11.0 [8f399da3] Libdl v1.11.0 [37e2e46d] LinearAlgebra v1.11.0 [56ddb016] Logging v1.11.0 [d6f4376e] Markdown v1.11.0 [a63ad114] Mmap v1.11.0 [ca575930] NetworkOptions v1.2.0 [44cfe95a] Pkg v1.11.0 [de0858da] Printf v1.11.0 [9abbd945] Profile v1.11.0 [3fa0cd96] REPL v1.11.0 [9a3f8284] Random v1.11.0 [ea8e919c] SHA v0.7.0 [9e88b42a] Serialization v1.11.0 [1a1011a3] SharedArrays v1.11.0 [6462fe0b] Sockets v1.11.0 [2f01184e] SparseArrays v1.11.0 [f489334b] StyledStrings v1.11.0 [4607b0f0] SuiteSparse [fa267f1f] TOML v1.0.3 [a4e569a6] Tar v1.10.0 [8dfed614] Test v1.11.0 [cf7118a7] UUIDs v1.11.0 [4ec0a83e] Unicode v1.11.0 [e66e0078] CompilerSupportLibraries_jll v1.1.1+0 [deac9b47] LibCURL_jll v8.6.0+0 [e37daf67] LibGit2_jll v1.7.2+0 [29816b5a] LibSSH2_jll v1.11.0+1 [c8ffd9c3] MbedTLS_jll v2.28.6+0 [14a3606d] MozillaCACerts_jll v2023.12.12 [4536629a] OpenBLAS_jll v0.3.27+1 [05823500] OpenLibm_jll v0.8.5+0 [efcefdf7] PCRE2_jll v10.42.0+1 [bea87d4a] SuiteSparse_jll v7.7.0+0 [83775a58] Zlib_jll v1.2.13+1 [8e850b90] libblastrampoline_jll v5.11.0+0 [8e850ede] nghttp2_jll v1.59.0+0 [3f19e933] p7zip_jll v17.4.0+2 Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. Testing Running tests... [ Info: Running all tests Precompiling DistributionsExt... 7854.9 ms ✓ Dagger → DistributionsExt 1 dependency successfully precompiled in 10 seconds. 76 already precompiled. [ Info: Testing Thunk (thunk) ┌ Warning: `delayed` is deprecated. Use `Dagger.@spawn` or `Dagger.spawn` instead. └ @ Dagger ~/.julia/packages/Dagger/cPlyf/src/thunk.jl:178 ┌ Warning: The `single` option is deprecated, please use scopes instead │ See https://juliaparallel.org/Dagger.jl/stable/scopes/ for details └ @ Dagger.Sch ~/.julia/packages/Dagger/cPlyf/src/sch/util.jl:357 ┌ Warning: The `proclist` option is deprecated, please use scopes instead │ See https://juliaparallel.org/Dagger.jl/stable/scopes/ for details └ @ Dagger.Sch ~/.julia/packages/Dagger/cPlyf/src/sch/util.jl:332 Test Summary: | Pass Broken Total Time Thunk | 140 1 141 5m07.6s [ Info: Testing Scheduler (scheduler) ┌ Warning: Threading tests running in serial └ @ Main ~/.julia/packages/Dagger/cPlyf/test/scheduler.jl:108 From worker 5: UNHANDLED TASK ERROR: On worker 1: From worker 5: On worker 10: From worker 5: peer 5 has not connected to 10 From worker 5: Stacktrace: From worker 5: [1] error From worker 5: @ ./error.jl:35 From worker 5: [2] wait_for_conn From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:192 From worker 5: [3] #27 From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:180 From worker 5: [4] exec_conn_func From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:181 From worker 5: [5] exec_conn_func From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:175 From worker 5: [6] #invokelatest#2 From worker 5: @ ./essentials.jl:1055 From worker 5: [7] invokelatest From worker 5: @ ./essentials.jl:1052 From worker 5: [8] #110 From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:287 From worker 5: [9] run_work_thunk From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:70 From worker 5: [10] #109 From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:287 From worker 5: Stacktrace: From worker 5: [1] #remotecall_fetch#159 From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:465 From worker 5: [2] remotecall_fetch From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:454 From worker 5: [3] remotecall_fetch From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:492 [inlined] From worker 5: [4] #23 From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:167 From worker 5: [5] #invokelatest#2 From worker 5: @ ./essentials.jl:1055 From worker 5: [6] invokelatest From worker 5: @ ./essentials.jl:1052 From worker 5: [7] #110 From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:287 From worker 5: [8] run_work_thunk From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:70 From worker 5: [9] #109 From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:287 From worker 5: Stacktrace: From worker 5: [1] remotecall_fetch(::Function, ::Distributed.Worker, ::Int64, ::Vararg{Int64}; kwargs::@Kwargs{}) From worker 5: @ Distributed /opt/julia/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:465 From worker 5: [2] remotecall_fetch(::Function, ::Distributed.Worker, ::Int64, ::Vararg{Int64}) From worker 5: @ Distributed /opt/julia/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:454 From worker 5: [3] remotecall_fetch From worker 5: @ /opt/julia/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:492 [inlined] From worker 5: [4] (::Distributed.var"#22#25"{Distributed.Worker})() From worker 5: @ Distributed /opt/julia/share/julia/stdlib/v1.11/Distributed/src/cluster.jl:167 WARNING: Method definition _list_workers(Any, Any, Any, Any, Any) in module Main at /home/pkgeval/.julia/packages/Dagger/cPlyf/test/scheduler.jl:196 overwritten on the same line (check for duplicate calls to `include`). WARNING: Method definition testfun(Any) in module Main at /home/pkgeval/.julia/packages/Dagger/cPlyf/test/scheduler.jl:202 overwritten on the same line (check for duplicate calls to `include`). Test Summary: | Pass Broken Total Time Scheduler | 125 4 129 6m29.5s [ Info: Testing Processors (processors) Test Summary: | Pass Total Time Processors | 30 30 1m55.1s [ Info: Testing Memory Spaces (memory-spaces) Test Summary: | Pass Total Time Memory Spaces | 19 19 4.4s [ Info: Testing Logging (logging) Precompiling GraphVizSimpleExt... 8473.4 ms ✓ Dagger → GraphVizSimpleExt 1 dependency successfully precompiled in 12 seconds. 80 already precompiled. Precompiling GraphViz... 4215.2 ms ✓ GraphViz 1 dependency successfully precompiled in 5 seconds. 56 already precompiled. Precompiling GraphVizExt... 11396.4 ms ✓ Dagger → GraphVizExt 1 dependency successfully precompiled in 14 seconds. 122 already precompiled. Precompiling DataFrames... 85314.7 ms ✓ DataFrames 1 dependency successfully precompiled in 86 seconds. 32 already precompiled. Precompiling Plots... 127352.8 ms ✓ Plots 17905.9 ms ✓ Plots → UnitfulExt 2 dependencies successfully precompiled in 147 seconds. 179 already precompiled. Precompiling FileIOExt... 18712.2 ms ✓ Plots → FileIOExt 1 dependency successfully precompiled in 21 seconds. 183 already precompiled. Precompiling PlotsExt... 4685.6 ms ✓ Latexify → DataFramesExt 27730.7 ms ✓ Dagger → PlotsExt 2 dependencies successfully precompiled in 36 seconds. 237 already precompiled. Test Summary: | Pass Total Time Logging | 698 698 9m55.5s [ Info: Testing Checkpointing (checkpoint) Test Summary: | Pass Total Time Checkpointing | 34 34 9.6s [ Info: Testing Scopes (scopes) Test Summary: | ┌ Warning: Worker 17 died, rescheduling work └ @ Dagger.Sch ~/.julia/packages/Dagger/cPlyf/src/sch/Sch.jl:568 Pass Total Time ┌ Warning: Worker 18 died, rescheduling work └ @ Dagger.Sch ~/.julia/packages/Dagger/cPlyf/src/sch/Sch.jl:568 Scopes | 113 113 3m06.4s [ Info: Testing Options (options) Test Summary: | Pass Total Time Options | 32 32 59.4s [ Info: Testing Mutation (mutation) Test Summary: | Pass Total Time Mutation | 71 71 39.5s [ Info: Testing Task Queues (task-queues) Test Summary: | Pass Total Time Task Queues | 26 26 18.9s [ Info: Testing Datadeps (datadeps) From worker 4: From worker 4: ====================================================================================== From worker 4: Information request received. A stacktrace will print followed by a 1.0 second profile From worker 4: ====================================================================================== From worker 4: From worker 4: cmd: /opt/julia/bin/julia 202 running 0 of 1 From worker 4: From worker 4: signal (10): User defined signal 1 From worker 4: epoll_pwait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) From worker 4: uv__io_poll at /workspace/srcdir/libuv/src/unix/linux.c:1404 From worker 4: uv_run at /workspace/srcdir/libuv/src/unix/core.c:430 From worker 4: ijl_task_get_next at /source/src/scheduler.c:522 From worker 4: poptask at ./task.jl:1012 From worker 4: wait at ./task.jl:1021 From worker 4: task_done_hook at ./task.jl:694 From worker 4: jfptr_task_done_hook_66694.1 at /opt/julia/lib/julia/sys.so (unknown line) From worker 4: _jl_invoke at /source/src/gf.c:2948 [inlined] From worker 4: ijl_apply_generic at /source/src/gf.c:3125 From worker 4: jl_apply at /source/src/julia.h:2157 [inlined] From worker 4: jl_finish_task at /source/src/task.c:319 From worker 4: start_task at /source/src/task.c:1213 From worker 4: unknown function (ip: (nil)) From worker 4: From worker 4: ============================================================== From worker 4: Profile collected. A report will print at the next yield point From worker 4: ============================================================== From worker 4: From worker 3: From worker 3: ====================================================================================== From worker 3: Information request received. A stacktrace will print followed by a 1.0 second profile From worker 3: ====================================================================================== From worker 3: From worker 3: cmd: /opt/julia/bin/julia 200 running 0 of 1 From worker 3: From worker 3: signal (10): User defined signal 1 From worker 3: epoll_pwait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) From worker 3: uv__io_poll at /workspace/srcdir/libuv/src/unix/linux.c:1404 From worker 3: uv_run at /workspace/srcdir/libuv/src/unix/core.c:430 From worker 3: ijl_task_get_next at /source/src/scheduler.c:522 From worker 3: poptask at ./task.jl:1012 From worker 3: wait at ./task.jl:1021 From worker 3: task_done_hook at ./task.jl:694 From worker 3: jfptr_task_done_hook_66694.1 at /opt/julia/lib/julia/sys.so (unknown line) From worker 3: _jl_invoke at /source/src/gf.c:2948 [inlined] From worker 3: ijl_apply_generic at /source/src/gf.c:3125 From worker 3: jl_apply at /source/src/julia.h:2157 [inlined] From worker 3: jl_finish_task at /source/src/task.c:319 From worker 3: start_task at /source/src/task.c:1213 From worker 3: unknown function (ip: (nil)) From worker 4: ┌ Warning: There were no samples collected in one or more groups. From worker 4: │ This may be due to idle threads, or you may need to run your From worker 4: │ program longer (perhaps by running it multiple times), From worker 4: │ or adjust the delay between samples with `Profile.init()`. From worker 4: └ @ Profile /opt/julia/share/julia/stdlib/v1.11/Profile/src/Profile.jl:1240 From worker 4: Overhead ╎ [+additional indent] Count File:Line; Function From worker 4: ========================================================= From worker 4: Thread 1 Task 0x00007a59977a4010 Total snapshots: 444. Utilization: 0% From worker 4: ╎444 @Base/task.jl:694; task_done_hook(t::Task) From worker 4: ╎ 444 @Base/task.jl:1021; wait() From worker 4: 443╎ 444 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{T… From worker 4: From worker 3: From worker 3: ============================================================== From worker 3: Profile collected. A report will print at the next yield point From worker 3: ============================================================== From worker 3: From worker 2: From worker 2: ====================================================================================== From worker 2: Information request received. A stacktrace will print followed by a 1.0 second profile From worker 2: ====================================================================================== From worker 2: From worker 2: cmd: /opt/julia/bin/julia 196 running 0 of 1 From worker 2: From worker 2: signal (10): User defined signal 1 From worker 2: epoll_pwait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) From worker 2: uv__io_poll at /workspace/srcdir/libuv/src/unix/linux.c:1404 From worker 2: uv_run at /workspace/srcdir/libuv/src/unix/core.c:430 From worker 2: ijl_task_get_next at /source/src/scheduler.c:522 From worker 2: poptask at ./task.jl:1012 From worker 2: wait at ./task.jl:1021 From worker 2: task_done_hook at ./task.jl:694 From worker 2: jfptr_task_done_hook_66694.1 at /opt/julia/lib/julia/sys.so (unknown line) From worker 2: _jl_invoke at /source/src/gf.c:2948 [inlined] From worker 2: ijl_apply_generic at /source/src/gf.c:3125 From worker 2: jl_apply at /source/src/julia.h:2157 [inlined] From worker 2: jl_finish_task at /source/src/task.c:319 From worker 2: start_task at /source/src/task.c:1213 From worker 2: unknown function (ip: (nil)) From worker 3: ┌ Warning: There were no samples collected in one or more groups. From worker 3: │ This may be due to idle threads, or you may need to run your From worker 3: │ program longer (perhaps by running it multiple times), From worker 3: │ or adjust the delay between samples with `Profile.init()`. From worker 3: └ @ Profile /opt/julia/share/julia/stdlib/v1.11/Profile/src/Profile.jl:1240 From worker 3: Overhead ╎ [+additional indent] Count File:Line; Function From worker 3: ========================================================= From worker 3: Thread 1 From worker 3: Task 0x00007b0e74bed910 Total snapshots: 327. Utilization: 0% From worker 3: ╎327 @Base/task.jl:694; task_done_hook(t::Task) From worker 3: ╎ 327 @Base/task.jl:1021; wait() From worker 3: 326╎ 327 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{T… From worker 3: From worker 3: Task 0x00007b0e74bed5f0 Total snapshots: 79. Utilization: 0% From worker 3: ╎79 @Base/task.jl:694; task_done_hook(t::Task) From worker 3: ╎ 79 @Base/task.jl:1021; wait() From worker 3: 78╎ 79 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Tas… From worker 3: From worker 2: From worker 2: ============================================================== From worker 2: Profile collected. A report will print at the next yield point From worker 2: ============================================================== From worker 2: From worker 2: ┌ Warning: There were no samples collected in one or more groups. From worker 2: │ This may be due to idle threads, or you may need to run your From worker 2: │ program longer (perhaps by running it multiple times), From worker 2: │ or adjust the delay between samples with `Profile.init()`. From worker 2: └ @ Profile /opt/julia/share/julia/stdlib/v1.11/Profile/src/Profile.jl:1240 From worker 2: Overhead ╎ [+additional indent] Count File:Line; Function From worker 2: ========================================================= From worker 2: Thread 1 From worker 2: Task 0x000079d2ed33e400 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed33ebd0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed332a40 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d36fedd5f0 Total snapshots: 23. Utilization: 17% From worker 2: ╎23 …emPool/src/datastore.jl:249; (::MemPool.var"#149#150")() From worker 2: ╎ 23 …buted/src/remotecall.jl:519; remotecall_wait(::Function, ::Int64, ::Int… From worker 2: ╎ 23 …uted/src/remotecall.jl:519; remotecall_wait(::Function, ::Int64, ::Int… From worker 2: ╎ 23 …uted/src/remotecall.jl:497; remotecall_wait(::Function, ::Distributed… From worker 2: ╎ 3 …ted/src/remotecall.jl:502; remotecall_wait(::Function, ::Distributed… From worker 2: ╎ 3 …buted/src/messages.jl:122; send_msg From worker 2: ╎ ╎ 1 …buted/src/messages.jl:180; send_msg_(w::Distributed.Worker, header… From worker 2: ╎ ╎ 1 …uted/src/messages.jl:163; serialize_hdr_raw(io::Sockets.TCPSocket… From worker 2: ╎ ╎ 1 @Base/io.jl:794; write From worker 2: ╎ ╎ 1 @Base/io.jl:814; write From worker 2: ╎ ╎ 1 @Base/io.jl:801; unsafe_write(s::Sockets.TCPSocket, p::Base.Ref… From worker 2: ╎ ╎ ╎ 1 @Base/io.jl:803; unsafe_write From worker 2: ╎ ╎ ╎ 1 @Base/stream.jl:1139; unsafe_write(s::Sockets.TCPSocket, p::P… From worker 2: 1╎ ╎ ╎ 1 @Base/libuv.jl:48; iolock_begin From worker 2: ╎ ╎ 1 …buted/src/messages.jl:181; send_msg_(w::Distributed.Worker, header… From worker 2: ╎ ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ ╎ 1 …uted/src/messages.jl:78; serialize_msg(s::Distributed.ClusterSe… From worker 2: ╎ ╎ 1 …rc/Serialization.jl:680; serialize(s::Distributed.ClusterSeria… From worker 2: ╎ ╎ ╎ 1 …rc/Serialization.jl:701; serialize_any(s::Distributed.Cluster… From worker 2: 1╎ ╎ ╎ 1 …c/Serialization.jl:580; serialize_type_data(s::Distributed.C… From worker 2: ╎ ╎ 1 …buted/src/messages.jl:187; send_msg_(w::Distributed.Worker, header… From worker 2: ╎ ╎ 1 @Base/stream.jl:1163; flush(s::Sockets.TCPSocket) From worker 2: ╎ ╎ 1 @Base/stream.jl:1065; uv_write From worker 2: ╎ ╎ 1 @Base/stream.jl:1069; uv_write(s::Sockets.TCPSocket, p::Ptr{UInt… From worker 2: 1╎ ╎ 1 @Base/stream.jl:1114; uv_write_async(s::Sockets.TCPSocket, p::P… From worker 2: ╎ 20 …ted/src/remotecall.jl:503; remotecall_wait(::Function, ::Distributed… From worker 2: ╎ 20 @Base/channels.jl:441; fetch(c::Channel{Any}) From worker 2: ╎ ╎ 20 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 20 @Base/condition.jl:125; wait From worker 2: ╎ ╎ 20 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantLo… From worker 2: ╎ ╎ 20 @Base/task.jl:1021; wait() From worker 2: 19╎ ╎ 20 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchron… From worker 2: From worker 2: Task 0x000079d2ed333080 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed3268b0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed326ef0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed325c30 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d36e7c4650 Total snapshots: 5. Utilization: 0% From worker 2: ╎5 …/src/process_messages.jl:121; (::Distributed.var"#103#104"{Sockets.TCPSock… From worker 2: ╎ 5 …src/process_messages.jl:133; process_tcp_streams(r_stream::Sockets.TCPSoc… From worker 2: ╎ 5 …src/process_messages.jl:176; message_handler_loop(r_stream::Sockets.TCPS… From worker 2: ╎ 5 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 5 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 5 …ibuted/src/messages.jl:87; deserialize_msg(s::Distributed.ClusterSeri… From worker 2: ╎ ╎ 5 …/src/Serialization.jl:844; deserialize From worker 2: ╎ ╎ 5 …/src/Serialization.jl:908; handle_deserialize(s::Distributed.Cluste… From worker 2: ╎ ╎ 5 …src/Serialization.jl:1585; deserialize(s::Distributed.ClusterSeria… From worker 2: ╎ ╎ 5 …src/Serialization.jl:895; handle_deserialize(s::Distributed.Clust… From worker 2: ╎ ╎ 5 …rc/Serialization.jl:1302; deserialize_array(s::Distributed.Clust… From worker 2: ╎ ╎ ╎ 5 @Base/io.jl:932; read!(s::Sockets.TCPSocket, A::Matrix{Float64}) From worker 2: ╎ ╎ ╎ 5 @Base/io.jl:913; read! From worker 2: ╎ ╎ ╎ 5 @Base/io.jl:890; unsafe_read From worker 2: ╎ ╎ ╎ 5 @Base/stream.jl:997; unsafe_read(s::Sockets.TCPSocket, p::Ptr… From worker 2: ╎ ╎ ╎ 5 @Base/stream.jl:981; (::Base.var"#wait_locked#834")(s::Socke… From worker 2: ╎ ╎ ╎ ╎ 5 @Base/stream.jl:416; wait_readnb(x::Sockets.TCPSocket, nb::… From worker 2: ╎ ╎ ╎ ╎ 5 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 5 @Base/condition.jl:130; wait(c::Base.GenericCondition{Bas… From worker 2: ╎ ╎ ╎ ╎ 5 @Base/task.jl:1021; wait() From worker 2: 4╎ ╎ ╎ ╎ 5 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedList… From worker 2: From worker 2: Task 0x000079d355ca9910 Total snapshots: 25. Utilization: 20% From worker 2: ╎21 @Dagger/src/sch/Sch.jl:1233; (::Dagger.Sch.var"#150#159"{Dagger.Sch.Proce… From worker 2: ╎ 21 …buted/src/remotecall.jl:769; isopen From worker 2: ╎ 21 …uted/src/remotecall.jl:565; call_on_owner From worker 2: ╎ 21 …uted/src/remotecall.jl:492; remotecall_fetch From worker 2: ╎ 21 …ted/src/remotecall.jl:492; #remotecall_fetch#162 From worker 2: ╎ 21 …ted/src/remotecall.jl:454; remotecall_fetch(f::Function, w::Distrib… From worker 2: ╎ ╎ 2 …ted/src/remotecall.jl:460; remotecall_fetch(f::Function, w::Distri… From worker 2: ╎ ╎ 2 …uted/src/messages.jl:122; send_msg From worker 2: ╎ ╎ 1 …uted/src/messages.jl:181; send_msg_(w::Distributed.Worker, heade… From worker 2: ╎ ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ ╎ ╎ 1 …ted/src/messages.jl:78; serialize_msg(s::Distributed.ClusterS… From worker 2: ╎ ╎ ╎ 1 …c/Serialization.jl:680; serialize(s::Distributed.ClusterSeri… From worker 2: ╎ ╎ ╎ 1 …c/Serialization.jl:706; serialize_any(s::Distributed.Cluste… From worker 2: ╎ ╎ ╎ 1 …/Serialization.jl:680; serialize(s::Distributed.ClusterSer… From worker 2: ╎ ╎ ╎ 1 …/Serialization.jl:688; serialize_any(s::Distributed.Clust… From worker 2: 1╎ ╎ ╎ ╎ 1 …/Serialization.jl:95; sertag From worker 2: ╎ ╎ 1 …uted/src/messages.jl:187; send_msg_(w::Distributed.Worker, heade… From worker 2: ╎ ╎ 1 @Base/stream.jl:1163; flush(s::Sockets.TCPSocket) From worker 2: ╎ ╎ 1 @Base/stream.jl:1065; uv_write From worker 2: ╎ ╎ ╎ 1 @Base/stream.jl:1069; uv_write(s::Sockets.TCPSocket, p::Ptr{UI… From worker 2: 1╎ ╎ ╎ 1 @Base/stream.jl:1114; uv_write_async(s::Sockets.TCPSocket, p:… From worker 2: ╎ ╎ 19 …ted/src/remotecall.jl:461; remotecall_fetch(f::Function, w::Distri… From worker 2: ╎ ╎ 19 …ed/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 19 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 19 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 19 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 19 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… From worker 2: ╎ ╎ ╎ 19 @Base/task.jl:1021; wait() From worker 2: 18╎ ╎ ╎ 19 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynch… From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1236; (::Dagger.Sch.var"#150#159"{Dagger.Sch.Proce… From worker 2: ╎ 1 …r/src/utils/dagdebug.jl:32; macro expansion From worker 2: ╎ 1 …ase/logging/logging.jl:399; macro expansion From worker 2: ╎ 1 @Base/operators.jl:426; >= From worker 2: 1╎ 1 @Base/int.jl:514; <= From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1238; (::Dagger.Sch.var"#150#159"{Dagger.Sch.Proce… From worker 2: ╎ 1 @Base/lock.jl:520; wait(e::Base.Event) From worker 2: ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantLock}; … From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: 1╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{… From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1277; (::Dagger.Sch.var"#150#159"{Dagger.Sch.Proce… From worker 2: 1╎ 1 @Base/promotion.jl:639; ==(x::UInt32, y::UInt32) From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1337; (::Dagger.Sch.var"#150#159"{Dagger.Sch.Proce… From worker 2: ╎ 1 …r/src/utils/dagdebug.jl:28; macro expansion From worker 2: ╎ 1 …ase/logging/logging.jl:402; macro expansion From worker 2: ╎ 1 …ase/logging/logging.jl:341; current_logger_for_env(std_level::Base.Co… From worker 2: ╎ 1 …se/logging/logging.jl:550; env_override_minlevel(group::Symbol, _mod… From worker 2: ╎ 1 @Base/env.jl:165; get From worker 2: ╎ ╎ 1 @Base/env.jl:59; access_env From worker 2: ╎ ╎ 1 @Base/env.jl:55; _getenv(var::String) From worker 2: ╎ ╎ 1 …e/strings/cstring.jl:85; unsafe_convert From worker 2: 1╎ ╎ 1 …e/strings/cstring.jl:78; containsnul From worker 2: From worker 2: Task 0x000079d2ed31aef0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed31abd0 Total snapshots: 1. Utilization: 100% From worker 2: ╎1 @Dagger/src/threadproc.jl:20; (::Dagger.var"#59#61"{Base.Pairs{Symbol, Unio… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 …er/src/memory-spaces.jl:54; move!(dep_mod::typeof(identity), to_space::… From worker 2: ╎ 1 …er/src/memory-spaces.jl:57; move! From worker 2: ╎ 1 @Base/array.jl:322; copyto! From worker 2: ╎ ╎ 1 @Base/array.jl:299; copyto! From worker 2: ╎ ╎ 1 @Base/array.jl:308; _copyto_impl! From worker 2: ╎ ╎ 1 @Base/genericmemory.jl:121; unsafe_copyto! From worker 2: ╎ ╎ 1 @Base/cmem.jl:28; memmove From worker 2: From worker 2: Task 0x000079d2ed3181a0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1350; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 @Dagger/src/sch/Sch.jl:1632; do_task(to_proc::ThreadProc, task_desc::Vecto… From worker 2: ╎ 1 @Dagger/src/sch/util.jl:271; fetch_report(task::Task) From worker 2: ╎ 1 @Base/task.jl:390; fetch From worker 2: ╎ 1 @Base/task.jl:368; wait(t::Task) From worker 2: ╎ 1 @Base/task.jl:328; _wait(t::Task) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.S… From worker 2: ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronize… From worker 2: From worker 2: Task 0x000079d2ed3192d0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed318fb0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed30e720 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed30e0e0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed30d780 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed30d460 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed2f9f50 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed30c010 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed2fabd0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed2f9460 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed2f8e20 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed2edc30 Total snapshots: 2. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1350; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 @Dagger/src/sch/Sch.jl:1632; do_task(to_proc::ThreadProc, task_desc::Vecto… From worker 2: ╎ 1 @Dagger/src/sch/util.jl:271; fetch_report(task::Task) From worker 2: ╎ 1 @Base/task.jl:390; fetch From worker 2: ╎ 1 @Base/task.jl:368; wait(t::Task) From worker 2: ╎ 1 @Base/task.jl:328; _wait(t::Task) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.S… From worker 2: ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronize… From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed2eeef0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed2eebd0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed2df850 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed2dddc0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed2dcb00 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed2d39e0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed2d1aa0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed2d01a0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed2c3850 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed2c04c0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed2c0b00 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 2 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 2 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 2 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 2 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 2 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 2 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 2 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 2 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ ╎ ╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed2bb850 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed2b9780 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed2ba0e0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed2a3080 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed2a3850 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed2a01a0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed2a0970 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed299c30 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed299140 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed28f3a0 Total snapshots: 1. Utilization: 100% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1350; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 @Dagger/src/sch/Sch.jl:1721; do_task(to_proc::ThreadProc, task_desc::Vecto… From worker 2: ╎ 1 @Base/sysinfo.jl:331; loadavg From worker 2: From worker 2: Task 0x000079d2ed298b00 Total snapshots: 1. Utilization: 100% From worker 2: From worker 2: Task 0x000079d2ed28d460 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed279780 Total snapshots: 2. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1350; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 @Dagger/src/sch/Sch.jl:1632; do_task(to_proc::ThreadProc, task_desc::Vecto… From worker 2: ╎ 1 @Dagger/src/sch/util.jl:271; fetch_report(task::Task) From worker 2: ╎ 1 @Base/task.jl:390; fetch From worker 2: ╎ 1 @Base/task.jl:368; wait(t::Task) From worker 2: ╎ 1 @Base/task.jl:328; _wait(t::Task) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.S… From worker 2: ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronize… From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed27a8b0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed269f50 Total snapshots: 2. Utilization: 50% From worker 2: ╎2 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 2 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 2 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 2 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 2 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 2 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 2 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 2 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 2 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 2 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 2 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 2 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 2 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1010; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1004; trypoptask(W::Base.IntrusiveLinkedListSyn… From worker 2: ╎ ╎ ╎ 1 @Base/partr.jl:131; multiq_deletemin() From worker 2: ╎ ╎ ╎ ╎ 1 …ase/essentials.jl:916; getindex From worker 2: 1╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed26b080 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed269aa0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed257d00 Total snapshots: 2. Utilization: 50% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1350; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 @Dagger/src/sch/Sch.jl:1576; do_task(to_proc::ThreadProc, task_desc::Vecto… From worker 2: ╎ 1 @Base/abstractarray.jl:3399; map(f::Function, A::Base.Iterators.Zip{Tuple… From worker 2: ╎ 1 @Base/array.jl:801; collect(itr::Base.Generator{Base.Iterators.Zip{Tuple… From worker 2: ╎ 1 @Base/array.jl:827; collect_to_with_first! From worker 2: ╎ 1 @Base/array.jl:849; collect_to!(dest::Vector{Task}, itr::Base.Generato… From worker 2: ╎ ╎ 1 @Base/generator.jl:48; iterate From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed268e20 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed255aa0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1350; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 @Dagger/src/sch/Sch.jl:1632; do_task(to_proc::ThreadProc, task_desc::Vecto… From worker 2: ╎ 1 @Dagger/src/sch/util.jl:271; fetch_report(task::Task) From worker 2: ╎ 1 @Base/task.jl:390; fetch From worker 2: ╎ 1 @Base/task.jl:368; wait(t::Task) From worker 2: ╎ 1 @Base/task.jl:328; _wait(t::Task) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.S… From worker 2: ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronize… From worker 2: From worker 2: Task 0x000079d2ed256a40 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed24f9e0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed254970 Total snapshots: 2. Utilization: 50% From worker 2: ╎2 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 2 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 2 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 2 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 2 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 2 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:460; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …ed/src/messages.jl:122; send_msg From worker 2: ╎ ╎ ╎ ╎ 1 …d/src/messages.jl:187; send_msg_(w::Distributed.Worker, he… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/stream.jl:1163; flush(s::Sockets.TCPSocket) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/stream.jl:1065; uv_write From worker 2: ╎ ╎ ╎ ╎ 1 @Base/stream.jl:1069; uv_write(s::Sockets.TCPSocket, p::… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/stream.jl:1114; uv_write_async(s::Sockets.TCPSock… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed2433a0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed24d2d0 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed243b70 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 2 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 2 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 2 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 2 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 2 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 2 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 2 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 2 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ ╎ ╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed2412d0 Total snapshots: 2. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1350; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 @Dagger/src/sch/Sch.jl:1632; do_task(to_proc::ThreadProc, task_desc::Vecto… From worker 2: ╎ 1 @Dagger/src/sch/util.jl:271; fetch_report(task::Task) From worker 2: ╎ 1 @Base/task.jl:390; fetch From worker 2: ╎ 1 @Base/task.jl:368; wait(t::Task) From worker 2: ╎ 1 @Base/task.jl:328; _wait(t::Task) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.S… From worker 2: ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronize… From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed242270 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed22c010 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed22f6c0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed22d460 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed22ce20 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed22cb00 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed21f3a0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed21ebd0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed21e590 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed21dc30 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed21d910 Total snapshots: 1. Utilization: 100% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:460; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …ed/src/messages.jl:122; send_msg From worker 2: ╎ ╎ ╎ ╎ 1 …d/src/messages.jl:187; send_msg_(w::Distributed.Worker, he… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/stream.jl:1163; flush(s::Sockets.TCPSocket) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/stream.jl:1065; uv_write From worker 2: ╎ ╎ ╎ ╎ 1 @Base/stream.jl:1069; uv_write(s::Sockets.TCPSocket, p::… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/stream.jl:1110; uv_write_async(s::Sockets.TCPSock… From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/libc.jl:392; malloc From worker 2: From worker 2: Task 0x000079d2ed2139e0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed211910 Total snapshots: 2. Utilization: 50% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1350; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 @Dagger/src/sch/Sch.jl:1699; do_task(to_proc::ThreadProc, task_desc::Vecto… From worker 2: ╎ 1 @Dagger/src/chunks.jl:267; kwcall(::@NamedTuple{device::MemPool.CPURAMDev… From worker 2: ╎ 1 @Dagger/src/chunks.jl:267; tochunk From worker 2: ╎ 1 @Dagger/src/chunks.jl:275; tochunk(x::Nothing, proc::ThreadProc, scope:… From worker 2: ╎ 1 …mPool/src/datastore.jl:454; poolset From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:454; poolset From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:461; poolset(x::Any, pid::Int64; size::Int64,… From worker 2: ╎ ╎ 1 @Base/lock.jl:273; macro expansion From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:399; ensure_memory_reserved From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:405; ensure_memory_reserved(size::Int64; ma… From worker 2: ╎ ╎ ╎ 1 …Pool/src/storage.jl:177; storage_available From worker 2: ╎ ╎ ╎ 1 …Pool/src/storage.jl:202; _query_mem_periodically(kind::Symbol) From worker 2: ╎ ╎ ╎ 1 …Pool/src/storage.jl:165; free_memory From worker 2: ╎ ╎ ╎ 1 @Base/io.jl:407; open From worker 2: ╎ ╎ ╎ 1 @Base/io.jl:408; open(::MemPool.var"#28#29", ::String, ::Var… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/iostream.jl:357; open(fname::String, mode::String) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/iostream.jl:358; open(fname::String, mode::String; l… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/iostream.jl:277; open From worker 2: ╎ ╎ ╎ ╎ 1 @Base/iostream.jl:291; open(fname::String; lock::Bool, r… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/iostream.jl:32; IOStream From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/iostream.jl:25; IOStream From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/array.jl:585; zeros From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/array.jl:589; zeros From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/boot.jl:591; Array From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/boot.jl:578; Array From worker 2: ╎ ╎ ╎ ╎ ╎ ╎ 1 @Base/boot.jl:516; GenericMemory From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed212720 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed212590 Total snapshots: 1. Utilization: 100% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1624; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 …er/src/utils/dagdebug.jl:38; macro expansion From worker 2: ╎ 1 @Base/task.jl:902; yield() From worker 2: ╎ 1 @Base/task.jl:1023; wait() From worker 2: ╎ 1 @Base/libuv.jl:125; process_events From worker 2: From worker 2: Task 0x000079d2ed210b00 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1ff530 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed1fe8b0 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1fe270 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1fd910 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1eed60 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 2 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 2 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 2 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 2 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 2 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 2 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 2 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 2 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 2 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 2 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 2 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed1fc330 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1fc010 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 2 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 2 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 2 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 2 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 2 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 2 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 2 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 2 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ ╎ ╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed1dd910 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1ece20 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1ecb00 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed1dd2d0 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 2 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 2 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 2 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 2 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 2 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 2 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 2 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 2 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 2 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 2 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 2 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed1dea40 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1dc4c0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1d36c0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1d15f0 Total snapshots: 2. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1350; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 @Dagger/src/sch/Sch.jl:1632; do_task(to_proc::ThreadProc, task_desc::Vecto… From worker 2: ╎ 1 @Dagger/src/sch/util.jl:271; fetch_report(task::Task) From worker 2: ╎ 1 @Base/task.jl:390; fetch From worker 2: ╎ 1 @Base/task.jl:368; wait(t::Task) From worker 2: ╎ 1 @Base/task.jl:328; _wait(t::Task) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.S… From worker 2: ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronize… From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed1d04c0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1c6270 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed1c68b0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed1c4010 Total snapshots: 2. Utilization: 50% From worker 2: ╎2 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 2 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 2 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 2 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 2 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 2 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:460; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …buted/src/messages.jl:122; send_msg From worker 2: ╎ ╎ 1 …buted/src/messages.jl:181; send_msg_(w::Distributed.Worker, header… From worker 2: ╎ ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ ╎ ╎ 1 …uted/src/messages.jl:78; serialize_msg(s::Distributed.ClusterSe… From worker 2: ╎ ╎ ╎ 1 …rc/Serialization.jl:204; serialize(s::Distributed.ClusterSeria… From worker 2: ╎ ╎ ╎ 1 …rc/Serialization.jl:204; serialize(s::Distributed.ClusterSeri… From worker 2: ╎ ╎ ╎ 1 …c/Serialization.jl:680; serialize From worker 2: ╎ ╎ ╎ 1 …c/Serialization.jl:701; serialize_any(s::Distributed.Cluste… From worker 2: ╎ ╎ ╎ ╎ 1 …/Serialization.jl:627; serialize_type(s::Distributed.Clust… From worker 2: ╎ ╎ ╎ ╎ 1 …/Serialization.jl:595; serialize_type_data(s::Distributed… From worker 2: ╎ ╎ ╎ ╎ 1 …/Serialization.jl:402; serialize From worker 2: ╎ ╎ ╎ ╎ 1 …Serialization.jl:393; serialize_mod_names(s::Distribute… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/boot.jl:621; Symbol(s::String) From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/boot.jl:614; _Symbol From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed1b01a0 Total snapshots: 2. Utilization: 50% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1350; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 @Dagger/src/sch/Sch.jl:1511; do_task(to_proc::ThreadProc, task_desc::Vecto… From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed1b1140 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed18f6c0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed1a2a40 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1a1780 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed18cfb0 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1a1140 Total snapshots: 3. Utilization: 0% From worker 2: ╎3 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 3 @Base/task.jl:1021; wait() From worker 2: 2╎ 3 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed18fd00 Total snapshots: 3. Utilization: 0% From worker 2: ╎3 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 3 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 3 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 3 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 3 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 3 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 3 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 3 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 3 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 3 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 3 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 3 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 3 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 3 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 3 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 3 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 3 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 3 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 3 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 3 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 3 @Base/task.jl:1021; wait() From worker 2: 2╎ ╎ ╎ ╎ ╎ 3 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed18eef0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed18e590 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed18d910 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed183210 Total snapshots: 1. Utilization: 100% From worker 2: ╎1 @Base/task.jl:672; task_done_hook(t::Task) From worker 2: From worker 2: Task 0x000079d2ed167b70 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed182270 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed181c30 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1815f0 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed180330 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed1676c0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1673a0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed165140 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed166400 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 2 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 2 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 2 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 2 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 2 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 2 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 2 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 2 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ ╎ ╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed155c30 Total snapshots: 3. Utilization: 0% From worker 2: ╎3 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 3 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 3 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 3 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 3 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 3 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 3 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 3 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 3 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 3 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 3 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 3 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 3 @Base/task.jl:1021; wait() From worker 2: 2╎ ╎ ╎ 3 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed156400 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 2 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 2 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 2 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 2 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 2 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 2 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 2 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 2 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ ╎ ╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed12fd00 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 2 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 2 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 2 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 2 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 2 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 2 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 2 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 2 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ ╎ ╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed12c4c0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed12dc30 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed119dc0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed12cb00 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed12c010 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed1195f0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed11aef0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed119f50 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 2 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 2 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 2 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 2 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 2 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 2 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 2 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 2 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 2 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 2 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 2 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 2 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ ╎ ╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed0ed140 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 2 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 2 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 2 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 2 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 2 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 2 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 2 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 2 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 2 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 2 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 2 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ ╎ ╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed0dad60 Total snapshots: 1. Utilization: 100% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:460; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …buted/src/messages.jl:122; send_msg From worker 2: ╎ ╎ 1 …buted/src/messages.jl:181; send_msg_(w::Distributed.Worker, header… From worker 2: ╎ ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ ╎ ╎ 1 …uted/src/messages.jl:78; serialize_msg(s::Distributed.ClusterSe… From worker 2: ╎ ╎ ╎ 1 …rc/Serialization.jl:204; serialize(s::Distributed.ClusterSeria… From worker 2: ╎ ╎ ╎ 1 …rc/Serialization.jl:204; serialize(s::Distributed.ClusterSeri… From worker 2: ╎ ╎ ╎ 1 …c/Serialization.jl:204; serialize(s::Distributed.ClusterSeri… From worker 2: ╎ ╎ ╎ 1 …c/Serialization.jl:680; serialize From worker 2: ╎ ╎ ╎ ╎ 1 …/Serialization.jl:705; serialize_any(s::Distributed.Cluste… From worker 2: From worker 2: Task 0x000079d2ed0dbd00 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed0d9460 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed0c3850 Total snapshots: 1. Utilization: 100% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1350; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 @Dagger/src/sch/Sch.jl:1672; do_task(to_proc::ThreadProc, task_desc::Vecto… From worker 2: From worker 2: Task 0x000079d2ed0d8b00 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed0c1aa0 Total snapshots: 1. Utilization: 100% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:460; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …ed/src/messages.jl:122; send_msg From worker 2: ╎ ╎ ╎ ╎ 1 …d/src/messages.jl:187; send_msg_(w::Distributed.Worker, he… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/stream.jl:1163; flush(s::Sockets.TCPSocket) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/stream.jl:1065; uv_write From worker 2: ╎ ╎ ╎ ╎ 1 @Base/stream.jl:1069; uv_write(s::Sockets.TCPSocket, p::… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/stream.jl:1114; uv_write_async(s::Sockets.TCPSock… From worker 2: From worker 2: Task 0x000079d2ed082d60 Total snapshots: 1. Utilization: 100% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:460; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …buted/src/messages.jl:122; send_msg From worker 2: ╎ ╎ 1 …buted/src/messages.jl:187; send_msg_(w::Distributed.Worker, header… From worker 2: ╎ ╎ 1 @Base/stream.jl:1163; flush(s::Sockets.TCPSocket) From worker 2: ╎ ╎ 1 @Base/stream.jl:1065; uv_write From worker 2: ╎ ╎ ╎ 1 @Base/stream.jl:1081; uv_write(s::Sockets.TCPSocket, p::Ptr{UInt… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed0c0010 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed080650 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: From worker 2: Task 0x000079d2ed081aa0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed06a270 Total snapshots: 1. Utilization: 100% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:460; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …buted/src/messages.jl:122; send_msg From worker 2: ╎ ╎ 1 …buted/src/messages.jl:181; send_msg_(w::Distributed.Worker, header… From worker 2: ╎ ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ ╎ ╎ 1 …uted/src/messages.jl:78; serialize_msg(s::Distributed.ClusterSe… From worker 2: ╎ ╎ ╎ 1 …rc/Serialization.jl:204; serialize(s::Distributed.ClusterSeria… From worker 2: ╎ ╎ ╎ 1 …rc/Serialization.jl:204; serialize(s::Distributed.ClusterSeri… From worker 2: ╎ ╎ ╎ 1 …c/Serialization.jl:204; serialize(s::Distributed.ClusterSeri… From worker 2: ╎ ╎ ╎ 1 …c/Serialization.jl:680; serialize From worker 2: ╎ ╎ ╎ ╎ 1 …/Serialization.jl:706; serialize_any(s::Distributed.Cluste… From worker 2: ╎ ╎ ╎ ╎ 1 …/src/datastore.jl:33; serialize(io::Distributed.ClusterSe… From worker 2: ╎ ╎ ╎ ╎ 1 …/src/datastore.jl:38; _pooltransfer_send(io::Distributed… From worker 2: ╎ ╎ ╎ ╎ 1 …src/datastore.jl:364; pooltransfer_send_local From worker 2: ╎ ╎ ╎ ╎ 1 …Pool/src/lock.jl:78; with_lock From worker 2: ╎ ╎ ╎ ╎ ╎ 1 …Pool/src/lock.jl:80; with_lock(f::MemPool.var"#161#16… From worker 2: ╎ ╎ ╎ ╎ ╎ 1 …rc/datastore.jl:365; (::MemPool.var"#161#163"{Int64,… From worker 2: From worker 2: Task 0x000079d2ed06b210 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed0692d0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed068b00 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ed0428b0 Total snapshots: 1. Utilization: 100% From worker 2: ╎1 …/src/process_messages.jl:304; (::Distributed.var"#113#115"{Distributed.Cal… From worker 2: From worker 2: Task 0x000079d2ed042270 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ed040fb0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ecffb9e0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ecffa8b0 Total snapshots: 1. Utilization: 100% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::OSProc, to_proc::ThreadProc,… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: From worker 2: Task 0x000079d2ecff9c30 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ecff8970 Total snapshots: 2. Utilization: 0% From worker 2: ╎2 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 2 @Base/task.jl:1021; wait() From worker 2: 1╎ 2 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ecf94970 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Base/task.jl:694; task_done_hook(t::Task) From worker 2: ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) From worker 2: From worker 2: Task 0x000079d2ecf97850 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1621; (::Dagger.Sch.var"#180#188"{Int64, Int64, Dagg… From worker 2: ╎ 1 @Base/essentials.jl:1052; invokelatest From worker 2: ╎ 1 @Base/essentials.jl:1055; #invokelatest#2 From worker 2: ╎ 1 @Dagger/src/chunks.jl:102; move(to_proc::ThreadProc, chunk::Dagger.Chunk… From worker 2: ╎ 1 @Dagger/src/chunks.jl:96; move From worker 2: ╎ 1 @Dagger/src/chunks.jl:98; move(from_proc::ThreadProc, to_proc::ThreadP… From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:528; poolget From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:530; access_ref From worker 2: ╎ ╎ 1 …Pool/src/datastore.jl:543; access_ref(::Function, ::MemPool.DRef; … From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:517; forwardkeyerror(f::MemPool.var"#193#196… From worker 2: ╎ ╎ 1 …ool/src/datastore.jl:544; (::MemPool.var"#193#196"{Bool, typeof(… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, … From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64,… From worker 2: ╎ ╎ ╎ 1 …d/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distr… From worker 2: ╎ ╎ ╎ 1 …/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ ╎ 1 …ase/condition.jl:130; wait(c::Base.GenericCondition{Ree… From worker 2: ╎ ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedLis… From worker 2: From worker 2: Task 0x000079d2ecf944c0 Total snapshots: 1. Utilization: 0% From worker 2: ╎1 @Dagger/src/sch/Sch.jl:1376; (::Dagger.Sch.var"#155#164"{UInt64, UInt32, Da… From worker 2: ╎ 1 …buted/src/remotecall.jl:722; put! From worker 2: ╎ 1 …buted/src/remotecall.jl:565; call_on_owner(::Function, ::RemoteChannel{C… From worker 2: ╎ 1 …buted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:492; remotecall_fetch(::Function, ::Int64, ::Di… From worker 2: ╎ 1 …uted/src/remotecall.jl:454; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:461; remotecall_fetch(::Function, ::Distribute… From worker 2: ╎ ╎ 1 …ted/src/remotecall.jl:726; take!(::Distributed.RemoteValue) From worker 2: ╎ ╎ 1 @Base/channels.jl:487; take!(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/channels.jl:493; take_buffered(c::Channel{Any}) From worker 2: ╎ ╎ 1 @Base/condition.jl:125; wait From worker 2: ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{ReentrantL… From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() From worker 2: ╎ ╎ ╎ 1 @Base/task.jl:1012; poptask(W::Base.IntrusiveLinkedListSynchro… From worker 2: ====================================================================================== Information request received. A stacktrace will print followed by a 1.0 second profile ====================================================================================== cmd: /opt/julia/bin/julia 192 running 1 of 1 signal (10): User defined signal 1 jl_to_typeof at /source/src/julia.h:777 [inlined] sig_match_fast at /source/src/gf.c:2967 [inlined] sig_match_fast at /source/src/gf.c:2958 [inlined] jl_lookup_generic_ at /source/src/gf.c:3049 [inlined] ijl_apply_generic at /source/src/gf.c:3121 run_finalizer at /source/src/gc.c:303 jl_gc_run_finalizers_in_list at /source/src/gc.c:393 run_finalizers at /source/src/gc.c:439 run_finalizers at /source/src/gc.c:420 [inlined] ijl_gc_collect at /source/src/gc.c:3915 maybe_collect at /source/src/gc.c:926 [inlined] jl_gc_pool_alloc_inner at /source/src/gc.c:1319 jl_gc_alloc_ at /source/src/julia_internal.h:523 [inlined] _new_genericmemory_ at /source/src/genericmemory.c:56 GenericMemory at ./boot.jl:516 [inlined] Array at ./boot.jl:578 [inlined] Array at ./boot.jl:591 [inlined] Array at ./boot.jl:599 [inlined] rand at /source/usr/share/julia/stdlib/v1.11/Random/src/Random.jl:290 [inlined] rand at /source/usr/share/julia/stdlib/v1.11/Random/src/Random.jl:291 [inlined] rand at /source/usr/share/julia/stdlib/v1.11/Random/src/Random.jl:279 [inlined] #700 at ./none:0 [inlined] iterate at ./generator.jl:48 [inlined] collect_to! at ./array.jl:849 collect_to_with_first! at ./array.jl:827 [inlined] collect at ./array.jl:801 #test_datadeps#624 at /home/pkgeval/.julia/packages/Dagger/cPlyf/test/datadeps.jl:455 unknown function (ip: 0x7aab5893e9a2) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_invoke at /source/src/gf.c:2955 unknown function (ip: 0x7aab589129f9) unknown function (ip: 0x7aab589129c9) test_datadeps at /home/pkgeval/.julia/packages/Dagger/cPlyf/test/datadeps.jl:112 [inlined] #810 at /home/pkgeval/.julia/packages/Dagger/cPlyf/test/datadeps.jl:523 [inlined] #37 at /home/pkgeval/.julia/packages/Dagger/cPlyf/src/options.jl:18 [inlined] with at ./scopedvalues.jl:269 unknown function (ip: 0x7aab589128e6) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 with_options at /home/pkgeval/.julia/packages/Dagger/cPlyf/src/options.jl:17 unknown function (ip: 0x7aab58911846) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 #with_options#39 at /home/pkgeval/.julia/packages/Dagger/cPlyf/src/options.jl:21 with_options at /home/pkgeval/.julia/packages/Dagger/cPlyf/src/options.jl:21 unknown function (ip: 0x7aab59100206) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 jl_apply at /source/src/julia.h:2157 [inlined] do_call at /source/src/interpreter.c:126 eval_value at /source/src/interpreter.c:223 eval_stmt_value at /source/src/interpreter.c:174 [inlined] eval_body at /source/src/interpreter.c:670 eval_body at /source/src/interpreter.c:539 eval_body at /source/src/interpreter.c:539 eval_body at /source/src/interpreter.c:539 eval_body at /source/src/interpreter.c:539 eval_body at /source/src/interpreter.c:539 eval_body at /source/src/interpreter.c:539 jl_interpret_toplevel_thunk at /source/src/interpreter.c:824 jl_toplevel_eval_flex at /source/src/toplevel.c:943 jl_toplevel_eval_flex at /source/src/toplevel.c:886 ijl_toplevel_eval_in at /source/src/toplevel.c:994 eval at ./boot.jl:430 [inlined] include_string at ./loading.jl:2734 _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 _include at ./loading.jl:2794 include at ./sysimg.jl:38 unknown function (ip: 0x7aab7bc4c082) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 jl_apply at /source/src/julia.h:2157 [inlined] do_call at /source/src/interpreter.c:126 eval_value at /source/src/interpreter.c:223 eval_stmt_value at /source/src/interpreter.c:174 [inlined] eval_body at /source/src/interpreter.c:670 eval_body at /source/src/interpreter.c:539 eval_body at /source/src/interpreter.c:539 eval_body at /source/src/interpreter.c:539 eval_body at /source/src/interpreter.c:539 jl_interpret_toplevel_thunk at /source/src/interpreter.c:824 jl_toplevel_eval_flex at /source/src/toplevel.c:943 jl_toplevel_eval_flex at /source/src/toplevel.c:886 ijl_toplevel_eval_in at /source/src/toplevel.c:994 eval at ./boot.jl:430 [inlined] include_string at ./loading.jl:2734 _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 _include at ./loading.jl:2794 include at ./sysimg.jl:38 unknown function (ip: 0x7aab7bc4c082) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 jl_apply at /source/src/julia.h:2157 [inlined] do_call at /source/src/interpreter.c:126 eval_value at /source/src/interpreter.c:223 eval_stmt_value at /source/src/interpreter.c:174 [inlined] eval_body at /source/src/interpreter.c:670 jl_interpret_toplevel_thunk at /source/src/interpreter.c:824 jl_toplevel_eval_flex at /source/src/toplevel.c:943 jl_toplevel_eval_flex at /source/src/toplevel.c:886 ijl_toplevel_eval_in at /source/src/toplevel.c:994 eval at ./boot.jl:430 [inlined] exec_options at ./client.jl:296 _start at ./client.jl:531 jfptr__start_73694.1 at /opt/julia/lib/julia/sys.so (unknown line) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 jl_apply at /source/src/julia.h:2157 [inlined] true_main at /source/src/jlapi.c:900 jl_repl_entrypoint at /source/src/jlapi.c:1059 main at /source/cli/loader_exe.c:58 unknown function (ip: 0x7aab7ccef249) __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) unknown function (ip: 0x4010b8) unknown function (ip: (nil)) ============================================================== Profile collected. A report will print at the next yield point ============================================================== ┌ Warning: There were no samples collected in one or more groups. │ This may be due to idle threads, or you may need to run your │ program longer (perhaps by running it multiple times), │ or adjust the delay between samples with `Profile.init()`. └ @ Profile /opt/julia/share/julia/stdlib/v1.11/Profile/src/Profile.jl:1240 Overhead ╎ [+additional indent] Count File:Line; Function ========================================================= Thread 1 Task 0x00007aaadfd2f850 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1023; wait() ╎ ╎ 1 @Base/libuv.jl:125; process_events Task 0x00007aaad7716720 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1023; wait() ╎ ╎ 1 @Base/libuv.jl:125; process_events Task 0x00007aaad75a0fb0 Total snapshots: 1. Utilization: 100% Task 0x00007aab729ef6c0 Total snapshots: 2. Utilization: 100% ╎2 …ibuted/src/remotecall.jl:286; (::Distributed.var"#137#139")() ╎ 2 @Base/essentials.jl:1052; invokelatest ╎ 2 @Base/essentials.jl:1055; #invokelatest#2 ╎ 2 …ributed/src/messages.jl:198; flush_gc_msgs() ╎ 2 …ibuted/src/messages.jl:150; flush_gc_msgs(w::Distributed.Worker) ╎ 2 …uted/src/remotecall.jl:531; remote_do ╎ ╎ 2 …ted/src/remotecall.jl:532; #remote_do#169 ╎ ╎ 2 …buted/src/messages.jl:122; send_msg(w::Distributed.Worker, header::… ╎ ╎ 2 …buted/src/messages.jl:181; send_msg_(w::Distributed.Worker, header… ╎ ╎ 2 @Base/essentials.jl:1052; invokelatest ╎ ╎ 2 @Base/essentials.jl:1055; #invokelatest#2 ╎ ╎ ╎ 2 …uted/src/messages.jl:78; serialize_msg(s::Distributed.ClusterSe… ╎ ╎ ╎ 2 …rc/Serialization.jl:204; serialize(s::Distributed.ClusterSeria… ╎ ╎ ╎ 2 …rc/Serialization.jl:276; serialize(s::Distributed.ClusterSeri… ╎ ╎ ╎ 2 …c/Serialization.jl:204; serialize(s::Distributed.ClusterSeri… ╎ ╎ ╎ 2 …c/Serialization.jl:680; serialize ╎ ╎ ╎ ╎ 2 …/Serialization.jl:701; serialize_any(s::Distributed.Cluste… ╎ ╎ ╎ ╎ 1 …/Serialization.jl:624; serialize_type(s::Distributed.Clus… 1╎ ╎ ╎ ╎ 1 …/Serialization.jl:95; sertag ╎ ╎ ╎ ╎ 1 …/Serialization.jl:627; serialize_type(s::Distributed.Clus… ╎ ╎ ╎ ╎ 1 …/Serialization.jl:588; serialize_type_data(s::Distribute… ╎ ╎ ╎ ╎ 1 …Serialization.jl:158; serialize_cycle(s::Distributed.Cl… ╎ ╎ ╎ ╎ 1 …Serialization.jl:145; writetag ╎ ╎ ╎ ╎ ╎ 1 @Base/stream.jl:1187; write(s::Sockets.TCPSocket, b::U… ╎ ╎ ╎ ╎ ╎ 1 @Base/libuv.jl:49; iolock_end Task 0x00007aa6095fc7e0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1023; wait() ╎ ╎ ╎ 1 @Base/libuv.jl:125; process_events Task 0x00007aa608f27d00 Total snapshots: 1. Utilization: 100% Task 0x00007aa6091107e0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1023; wait() ╎ ╎ ╎ 1 @Base/libuv.jl:125; process_events Task 0x00007aaadf9b28b0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_resc… Task 0x00007aa6076392d0 Total snapshots: 1. Utilization: 100% Task 0x00007aaad42c5aa0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1023; wait() ╎ ╎ ╎ 1 @Base/libuv.jl:125; process_events Task 0x00007aaad4cee0e0 Total snapshots: 2. Utilization: 100% ╎1 @Base/task.jl:677; task_done_hook(t::Task) ╎ 1 @Base/condition.jl:148; notify ╎ 1 @Base/condition.jl:148; notify ╎ 1 @Base/condition.jl:148; #notify#734 ╎ 1 @Base/condition.jl:154; notify(c::Base.GenericCondition{Base.Threads.Sp… ╎ 1 @Base/task.jl:876; schedule 1╎ ╎ 1 @Base/task.jl:878; schedule(t::Task, arg::Any; error::Bool) ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:115; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#74#78"{UInt64}, x::D… ╎ 1 @Dagger/src/sch/eager.jl:116; (::Dagger.Sch.var"#74#78"{UInt64})(id_map:… ╎ 1 @Base/dict.jl:548; haskey ╎ 1 @Base/dict.jl:246; ht_keyindex ╎ ╎ 1 @Base/dict.jl:133; isslotempty ╎ ╎ 1 @Base/promotion.jl:639; == Task 0x00007aaad6ecc1a0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1023; wait() ╎ ╎ ╎ 1 @Base/libuv.jl:125; process_events Task 0x00007aaad701d780 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1023; wait() ╎ ╎ ╎ 1 @Base/libuv.jl:125; process_events Task 0x00007aa6076d3210 Total snapshots: 1. Utilization: 100% ╎1 @Base/task.jl:694; task_done_hook(t::Task) ╎ 1 @Base/task.jl:1021; wait() ╎ 1 @Base/task.jl:1010; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) ╎ 1 @Base/task.jl:992; trypoptask(W::Base.IntrusiveLinkedListSynchronized{Ta… ╎ 1 @Base/task.jl:749; popfirst!(W::Base.IntrusiveLinkedListSynchronized{Ta… ╎ 1 @Base/linked_list.jl:86; popfirst! ╎ ╎ 1 @Base/linked_list.jl:98; list_deletefirst! Task 0x00007aa60771a270 Total snapshots: 1. Utilization: 100% ╎1 @Base/task.jl:600; (::Base.var"#761#762"{Task})() Task 0x00007aaae2950010 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1023; wait() ╎ ╎ ╎ 1 @Base/libuv.jl:125; process_events Task 0x00007aaae17b3b70 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 …ibuted/src/remotecall.jl:0; eager_cleanup(state::ComputeState, uid::UInt6… Task 0x00007aaadf279140 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1023; wait() ╎ ╎ ╎ 1 @Base/libuv.jl:125; process_events Task 0x00007aaad771d460 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1023; wait() ╎ ╎ ╎ 1 @Base/libuv.jl:125; process_events Task 0x00007aaad76dfb70 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1023; wait() ╎ ╎ ╎ 1 @Base/libuv.jl:125; process_events Task 0x00007aaad75a4650 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1023; wait() ╎ ╎ ╎ 1 @Base/libuv.jl:125; process_events Task 0x00007aa60798cc90 Total snapshots: 1. Utilization: 100% Task 0x00007aa607d16590 Total snapshots: 1. Utilization: 100% ╎1 @Base/task.jl:680; task_done_hook(t::Task) ╎ 1 @Base/condition.jl:74; unlock ╎ 1 @Base/locks-mt.jl:65; unlock ╎ 1 @Base/promotion.jl:639; == Task 0x00007aa608158c90 Total snapshots: 1. Utilization: 100% ╎1 @Base/task.jl:694; task_done_hook(t::Task) ╎ 1 @Base/task.jl:1022; wait() ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_rescheduled)) Task 0x00007aa6083c6ef0 Total snapshots: 1. Utilization: 100% ╎1 @Base/task.jl:680; task_done_hook(t::Task) ╎ 1 @Base/condition.jl:74; unlock ╎ 1 @Base/locks-mt.jl:65; unlock ╎ 1 @Base/promotion.jl:639; == Task 0x00007aa6095b8010 Total snapshots: 1. Utilization: 100% ╎1 @Base/task.jl:672; task_done_hook(t::Task) Task 0x00007aa607577080 Total snapshots: 1. Utilization: 100% ╎1 @Base/task.jl:673; task_done_hook(t::Task) ╎ 1 @Base/condition.jl:73; lock ╎ 1 @Base/locks-mt.jl:43; lock(l::Base.Threads.SpinLock) ╎ 1 @Base/locks-mt.jl:53; trylock Task 0x00007aaad4f433a0 Total snapshots: 1. Utilization: 100% ╎1 @Base/task.jl:694; task_done_hook(t::Task) ╎ 1 @Base/task.jl:1020; wait() ╎ 1 @Base/task.jl:769; workqueue_for(tid::Int64) Task 0x00007aaad7a9ddc0 Total snapshots: 1. Utilization: 100% ╎1 @Base/task.jl:694; task_done_hook(t::Task) ╎ 1 @Base/task.jl:1021; wait() ╎ 1 @Base/task.jl:1010; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) ╎ 1 @Base/task.jl:992; trypoptask(W::Base.IntrusiveLinkedListSynchronized{Ta… ╎ 1 @Base/task.jl:746; popfirst!(W::Base.IntrusiveLinkedListSynchronized{Ta… Task 0x00007aab106f2d60 Total snapshots: 1. Utilization: 100% Task 0x00007aaae126c1a0 Total snapshots: 1. Utilization: 100% ╎1 @Base/task.jl:694; task_done_hook(t::Task) ╎ 1 @Base/task.jl:1021; wait() ╎ 1 @Base/task.jl:1010; poptask(W::Base.IntrusiveLinkedListSynchronized{Task}) ╎ 1 @Base/task.jl:992; trypoptask(W::Base.IntrusiveLinkedListSynchronized{Ta… ╎ 1 @Base/task.jl:751; popfirst!(W::Base.IntrusiveLinkedListSynchronized{Ta… ╎ 1 @Base/locks-mt.jl:65; unlock ╎ ╎ 1 @Base/promotion.jl:639; == Task 0x00007aa6078d6d60 Total snapshots: 1. Utilization: 100% ╎1 @MemPool/src/datastore.jl:589; (::MemPool.var"#203#204"{Int64, MemPool.RefS… ╎ 1 @MemPool/src/datastore.jl:97; storage_read ╎ 1 @Base/Base.jl:70; getproperty Task 0x00007aa6079041a0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext ╎ 1 @Dagger/src/sch/util.jl:12; #3 Task 0x00007aa60792ddc0 Total snapshots: 1. Utilization: 100% Task 0x00007aa607952d60 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext ╎ 1 @Base/essentials.jl:917; getindex Task 0x00007aa6079776c0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2354; findnext Task 0x00007aa60799ad60 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext Task 0x00007aa6079b8b00 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:15; #2 ╎ 1 @Base/array.jl:1773; deleteat! ╎ 1 @Base/array.jl:1223; _deleteat!(a::Vector{Pair{String, Task}}, i::Int64… ╎ 1 @Base/array.jl:284; unsafe_copyto! ╎ ╎ 1 @Base/genericmemory.jl:125; unsafe_copyto! Task 0x00007aa6079e3080 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext ╎ 1 @Base/essentials.jl:917; getindex Task 0x00007aa607a079e0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext ╎ 1 @Base/essentials.jl:917; getindex Task 0x00007aa607a28b00 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext ╎ 1 @Base/essentials.jl:917; getindex Task 0x00007aa607a536c0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext ╎ 1 @Base/essentials.jl:917; getindex Task 0x00007aa607a7b3a0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext ╎ 1 @Base/essentials.jl:917; getindex Task 0x00007aa607aff3a0 Total snapshots: 1. Utilization: 100% Task 0x00007aa607ce01a0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext Task 0x00007aa607d29140 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext Task 0x00007aa607ddcb00 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:15; #2 ╎ 1 @Base/array.jl:1773; deleteat! ╎ 1 @Base/array.jl:1223; _deleteat!(a::Vector{Pair{String, Task}}, i::Int64… ╎ 1 @Base/array.jl:284; unsafe_copyto! ╎ ╎ 1 @Base/genericmemory.jl:125; unsafe_copyto! Task 0x00007aa607e53080 Total snapshots: 1. Utilization: 100% Task 0x00007aa607e733a0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext ╎ 1 @Dagger/src/sch/util.jl:12; #3 Task 0x00007aa607eb8e20 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:15; #2 ╎ 1 @Base/array.jl:1773; deleteat! ╎ 1 @Base/array.jl:1223; _deleteat!(a::Vector{Pair{String, Task}}, i::Int64… ╎ 1 @Base/array.jl:284; unsafe_copyto! ╎ ╎ 1 @Base/genericmemory.jl:125; unsafe_copyto! Task 0x00007aa607eeb9e0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2354; findnext Task 0x00007aa607f1ddc0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext ╎ 1 @Base/essentials.jl:917; getindex Task 0x00007aa607fdddc0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:15; #2 ╎ 1 @Base/array.jl:1773; deleteat! ╎ 1 @Base/array.jl:1223; _deleteat!(a::Vector{Pair{String, Task}}, i::Int64… ╎ 1 @Base/array.jl:284; unsafe_copyto! ╎ ╎ 1 @Base/genericmemory.jl:125; unsafe_copyto! Task 0x00007aa6080d44c0 Total snapshots: 1. Utilization: 100% Task 0x00007aa608110b00 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2354; findnext Task 0x00007aa60871ea40 Total snapshots: 1. Utilization: 100% ╎1 @MemPool/src/datastore.jl:591; (::MemPool.var"#203#204"{Int64, MemPool.RefS… ╎ 1 @MemPool/src/storage.jl:601; delete_from_device! ╎ 1 …emPool/src/datastore.jl:104; storage_rcu!(f::MemPool.var"#82#83", state:… ╎ 1 @MemPool/src/storage.jl:602; #82 ╎ 1 @MemPool/src/storage.jl:290; StorageState ╎ 1 @MemPool/src/storage.jl:290; #StorageState#36 ╎ ╎ 1 @Base/lock.jl:504; Event ╎ ╎ 1 @Base/lock.jl:504; Event ╎ ╎ 1 @Base/condition.jl:67; GenericCondition ╎ ╎ 1 @Base/lock.jl:49; ReentrantLock ╎ ╎ 1 @Base/condition.jl:67; GenericCondition ╎ ╎ ╎ 1 @Base/locks-mt.jl:34; SpinLock Task 0x00007aa608766400 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext Task 0x00007aa6088ae400 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext ╎ 1 @Base/essentials.jl:917; getindex Task 0x00007aa6089aad60 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext Task 0x00007aa608c860e0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext ╎ 1 @Base/essentials.jl:917; getindex Task 0x00007aa608d887e0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext ╎ 1 @Base/essentials.jl:917; getindex Task 0x00007aa60901d460 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/submission.jl:150; (::Dagger.var"#161#162"{Dagger.UnrefThunkByU… ╎ 1 @Base/lock.jl:273; macro expansion ╎ 1 …agger/src/submission.jl:151; macro expansion ╎ 1 @Dagger/src/sch/Sch.jl:981; delete_unused_task! ╎ 1 @Dagger/src/sch/Sch.jl:989; task_unused(state::ComputeState, thunk::Thu… ╎ 1 @Base/weakkeydict.jl:173; haskey ╎ ╎ 1 @Base/weakkeydict.jl:81; lock Task 0x00007aa60909ce20 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:12; #2 ╎ 1 @Base/array.jl:2404; findfirst ╎ 1 @Base/array.jl:2353; findnext ╎ 1 @Base/essentials.jl:917; getindex Task 0x00007aa609352400 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:15; #2 ╎ 1 @Base/array.jl:1773; deleteat! ╎ 1 @Base/array.jl:1223; _deleteat!(a::Vector{Pair{String, Task}}, i::Int64… ╎ 1 @Base/array.jl:284; unsafe_copyto! ╎ ╎ 1 @Base/genericmemory.jl:125; unsafe_copyto! Task 0x00007aa6094104c0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:11; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 …c/utils/locked-object.jl:12; lock(f::Dagger.Sch.var"#2#5"{Task}, x::Dagge… ╎ 1 @Dagger/src/sch/util.jl:15; #2 ╎ 1 @Base/array.jl:1773; deleteat! ╎ 1 @Base/array.jl:1223; _deleteat!(a::Vector{Pair{String, Task}}, i::Int64… ╎ 1 @Base/array.jl:284; unsafe_copyto! ╎ ╎ 1 @Base/genericmemory.jl:125; unsafe_copyto! Task 0x00007aa6095e4e20 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/submission.jl:150; (::Dagger.var"#161#162"{Dagger.UnrefThunkByU… ╎ 1 @Base/lock.jl:273; macro expansion ╎ 1 …agger/src/submission.jl:151; macro expansion ╎ 1 @Dagger/src/sch/Sch.jl:983; delete_unused_task! ╎ 1 @Dagger/src/sch/Sch.jl:994; task_delete! ╎ 1 @Base/weakkeydict.jl:160; delete! ╎ ╎ 1 @Base/weakkeydict.jl:81; lock ╎ ╎ 1 @Base/lock.jl:232; lock(f::Base.var"#794#795"{WeakKeyDict{Thunk, Not… ╎ ╎ 1 @Base/weakkeydict.jl:161; #794 ╎ ╎ 1 @Base/dict.jl:675; delete!(h::Dict{WeakRef, Nothing}, key::Thunk) ╎ ╎ 1 @Base/dict.jl:249; ht_keyindex(h::Dict{WeakRef, Nothing}, key::Th… ╎ ╎ ╎ 1 @Base/operators.jl:134; isequal ╎ ╎ ╎ 1 @Base/gcutils.jl:36; ==(w::Thunk, v::WeakRef) Task 0x00007aa6095f9aa0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_rescheduled)) Task 0x00007aa609606400 Total snapshots: 1. Utilization: 100% Task 0x00007aa60845d140 Total snapshots: 1. Utilization: 100% ╎1 @MemPool/src/datastore.jl:589; (::MemPool.var"#203#204"{Int64, MemPool.RefS… ╎ 1 @MemPool/src/storage.jl:302; getproperty(sstate::MemPool.StorageState, fie… Task 0x00007aa608c4ce20 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:428; remotecall ╎ ╎ 1 …ted/src/remotecall.jl:430; remotecall(f::Function, w::Distributed.L… ╎ ╎ 1 …c/process_messages.jl:84; schedule_call ╎ ╎ 1 @Base/weakkeydict.jl:81; lock(f::Function, wkh::WeakKeyDict{Distri… ╎ ╎ 1 @Base/lock.jl:232; lock(f::Distributed.var"#99#101"{Distributed.R… ╎ ╎ ╎ 1 …/process_messages.jl:88; (::Distributed.var"#99#101"{Distribute… ╎ ╎ ╎ 1 @Base/task.jl:593; errormonitor ╎ ╎ ╎ 1 @Base/task.jl:5; Task ╎ ╎ ╎ 1 @Base/task.jl:5; Task ╎ ╎ ╎ 1 @Base/boot.jl:502; _Task Task 0x00007aa608c77080 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_rescheduled)) Task 0x00007aa608c9ce20 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_resc… Task 0x00007aa608cbe0e0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_rescheduled)) Task 0x00007aa608d44330 Total snapshots: 1. Utilization: 100% Task 0x00007aa608d9aa40 Total snapshots: 1. Utilization: 100% Task 0x00007aa608db52d0 Total snapshots: 1. Utilization: 100% Task 0x00007aa608de36c0 Total snapshots: 1. Utilization: 100% Task 0x00007aa608e8e8b0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:428; remotecall ╎ ╎ 1 …ted/src/remotecall.jl:430; remotecall(f::Function, w::Distributed.L… ╎ ╎ 1 …c/process_messages.jl:84; schedule_call ╎ ╎ 1 @Base/weakkeydict.jl:81; lock(f::Function, wkh::WeakKeyDict{Distri… ╎ ╎ 1 @Base/lock.jl:232; lock(f::Distributed.var"#99#101"{Distributed.R… ╎ ╎ ╎ 1 …/process_messages.jl:88; (::Distributed.var"#99#101"{Distribute… ╎ ╎ ╎ 1 @Base/task.jl:593; errormonitor ╎ ╎ ╎ 1 @Base/task.jl:5; Task ╎ ╎ ╎ 1 @Base/task.jl:5; Task ╎ ╎ ╎ 1 @Base/boot.jl:502; _Task Task 0x00007aa608f01460 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:428; remotecall ╎ ╎ 1 …ted/src/remotecall.jl:430; remotecall(f::Function, w::Distributed.L… ╎ ╎ 1 …ted/src/remotecall.jl:425; local_remotecall_thunk Task 0x00007aa608f96a40 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1021; wait() ╎ ╎ 1 @Base/task.jl:1010; poptask(W::Base.IntrusiveLinkedListSynchronized{T… ╎ ╎ 1 @Base/task.jl:992; trypoptask(W::Base.IntrusiveLinkedListSynchronize… ╎ ╎ 1 @Base/task.jl:746; popfirst!(W::Base.IntrusiveLinkedListSynchronize… Task 0x00007aa608fc92d0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_rescheduled)) Task 0x00007aa608fd8970 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_resc… Task 0x00007aa608ff76c0 Total snapshots: 1. Utilization: 100% Task 0x00007aa609051460 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1021; wait() ╎ ╎ ╎ 1 @Base/task.jl:1010; poptask(W::Base.IntrusiveLinkedListSynch… ╎ ╎ ╎ ╎ 1 @Base/task.jl:992; trypoptask(W::Base.IntrusiveLinkedListSy… ╎ ╎ ╎ ╎ 1 @Base/task.jl:749; popfirst!(W::Base.IntrusiveLinkedListSy… ╎ ╎ ╎ ╎ 1 …se/linked_list.jl:86; popfirst! ╎ ╎ ╎ ╎ 1 …se/linked_list.jl:98; list_deletefirst! Task 0x00007aa609068b00 Total snapshots: 1. Utilization: 100% Task 0x00007aa6090833a0 Total snapshots: 1. Utilization: 100% Task 0x00007aa6090aea40 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_resc… Task 0x00007aa6090c1460 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_rescheduled)) Task 0x00007aa6090d84c0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_resc… Task 0x00007aa6090f7530 Total snapshots: 1. Utilization: 100% Task 0x00007aa609112270 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_resc… Task 0x00007aa6091b2d60 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:428; remotecall ╎ ╎ 1 …ted/src/remotecall.jl:430; remotecall(f::Function, w::Distributed.L… ╎ ╎ 1 …c/process_messages.jl:84; schedule_call ╎ ╎ 1 @Base/weakkeydict.jl:81; lock(f::Function, wkh::WeakKeyDict{Distri… ╎ ╎ 1 @Base/lock.jl:232; lock(f::Distributed.var"#99#101"{Distributed.R… ╎ ╎ ╎ 1 …/process_messages.jl:88; (::Distributed.var"#99#101"{Distribute… ╎ ╎ ╎ 1 @Base/task.jl:544; macro expansion ╎ ╎ ╎ 1 @Base/task.jl:833; schedule ╎ ╎ ╎ 1 @Base/task.jl:829; enq_work(t::Task) Task 0x00007aa6091c8c90 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_rescheduled)) Task 0x00007aa609206400 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_rescheduled)) Task 0x00007aa6092552d0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:586; wait ╎ ╎ 1 …ted/src/remotecall.jl:563; call_on_owner ╎ ╎ 1 …ted/src/remotecall.jl:570; wait_ref ╎ ╎ 1 …ed/src/remotecall.jl:648; fetch_ref(::Distributed.RRID) ╎ ╎ 1 @Base/channels.jl:441; fetch(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/channels.jl:447; fetch_buffered(c::Channel{Any}) ╎ ╎ ╎ 1 @Base/condition.jl:125; wait ╎ ╎ ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Reentran… ╎ ╎ ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_resc… Task 0x00007aa609287d00 Total snapshots: 1. Utilization: 100% Task 0x00007aa6092a1aa0 Total snapshots: 1. Utilization: 100% Task 0x00007aa6092d3530 Total snapshots: 1. Utilization: 100% Task 0x00007aa6092ecfb0 Total snapshots: 1. Utilization: 100% Task 0x00007aa609307080 Total snapshots: 1. Utilization: 100% Task 0x00007aa609316720 Total snapshots: 1. Utilization: 100% Task 0x00007aaae0aeebd0 Total snapshots: 1. Utilization: 100% Task 0x00007aaae1e084c0 Total snapshots: 1. Utilization: 100% Task 0x00007aa60afd95f0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_rescheduled)) Task 0x00007aaadf4473a0 Total snapshots: 1. Utilization: 100% Task 0x00007aaadf2b8970 Total snapshots: 1. Utilization: 100% Task 0x00007aaad40d39e0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… ╎ 1 @Dagger/src/sch/eager.jl:127; eager_cleanup(state::ComputeState, uid::UInt… ╎ 1 …buted/src/remotecall.jl:519; remotecall_wait ╎ 1 …buted/src/remotecall.jl:519; #remotecall_wait#167 ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait ╎ 1 …uted/src/remotecall.jl:495; remotecall_wait(f::Function, w::Distribut… ╎ ╎ 1 …ted/src/remotecall.jl:428; remotecall Task 0x00007aaad4145460 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_rescheduled)) Task 0x00007aaad43bebd0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/eager.jl:112; (::Dagger.Sch.var"#72#73"{Dagger.DTaskFinaliz… Task 0x00007aaad447cfb0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_rescheduled)) Task 0x00007aaad4596bd0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_rescheduled)) Task 0x00007aaad4602590 Total snapshots: 1. Utilization: 100% Task 0x00007aaad4a476c0 Total snapshots: 1. Utilization: 100% ╎1 @Dagger/src/sch/util.jl:9; (::Dagger.Sch.var"#1#4"{Task})() ╎ 1 @Base/task.jl:368; wait(t::Task) ╎ 1 @Base/task.jl:328; _wait(t::Task) ╎ 1 @Base/condition.jl:125; wait ╎ 1 @Base/condition.jl:130; wait(c::Base.GenericCondition{Base.Threads.Spin… ╎ 1 @Base/task.jl:1022; wait() ╎ ╎ 1 @Base/task.jl:948; try_yieldto(undo::typeof(Base.ensure_rescheduled)) Task 0x00007aaad4c1ddc0 Total snapshots: 1. Utilization: 100% ====================================================================================== Information request received. A stacktrace will print followed by a 1.0 second profile ====================================================================================== cmd: /opt/julia/bin/julia 1 running 0 of 1 signal (10): User defined signal 1 epoll_pwait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) uv__io_poll at /workspace/srcdir/libuv/src/unix/linux.c:1404 uv_run at /workspace/srcdir/libuv/src/unix/core.c:430 ijl_task_get_next at /source/src/scheduler.c:522 poptask at ./task.jl:1012 wait at ./task.jl:1021 #wait#733 at ./condition.jl:130 wait at ./condition.jl:125 [inlined] wait at ./process.jl:694 wait at ./process.jl:687 unknown function (ip: 0x7f9f008bfff2) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 subprocess_handler at /source/usr/share/julia/stdlib/v1.11/Pkg/src/Operations.jl:2146 #131 at /source/usr/share/julia/stdlib/v1.11/Pkg/src/Operations.jl:2086 withenv at ./env.jl:265 #118 at /source/usr/share/julia/stdlib/v1.11/Pkg/src/Operations.jl:1935 with_temp_env at /source/usr/share/julia/stdlib/v1.11/Pkg/src/Operations.jl:1793 #116 at /source/usr/share/julia/stdlib/v1.11/Pkg/src/Operations.jl:1902 #mktempdir#28 at ./file.jl:819 unknown function (ip: 0x7f9f008b645d) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 mktempdir at ./file.jl:815 mktempdir at ./file.jl:815 [inlined] #sandbox#115 at /source/usr/share/julia/stdlib/v1.11/Pkg/src/Operations.jl:1849 [inlined] sandbox at /source/usr/share/julia/stdlib/v1.11/Pkg/src/Operations.jl:1841 unknown function (ip: 0x7f9f008aa9e6) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 #test#128 at /source/usr/share/julia/stdlib/v1.11/Pkg/src/Operations.jl:2067 test at /source/usr/share/julia/stdlib/v1.11/Pkg/src/Operations.jl:2011 [inlined] #test#146 at /source/usr/share/julia/stdlib/v1.11/Pkg/src/API.jl:481 test at /source/usr/share/julia/stdlib/v1.11/Pkg/src/API.jl:460 unknown function (ip: 0x7f9f008aa6ed) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 #test#77 at /source/usr/share/julia/stdlib/v1.11/Pkg/src/API.jl:159 unknown function (ip: 0x7f9f008aa0dd) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 test at /source/usr/share/julia/stdlib/v1.11/Pkg/src/API.jl:148 #test#75 at /source/usr/share/julia/stdlib/v1.11/Pkg/src/API.jl:147 [inlined] test at /source/usr/share/julia/stdlib/v1.11/Pkg/src/API.jl:147 [inlined] #test#74 at /source/usr/share/julia/stdlib/v1.11/Pkg/src/API.jl:146 [inlined] test at /source/usr/share/julia/stdlib/v1.11/Pkg/src/API.jl:146 unknown function (ip: 0x7f9f008a6566) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 jl_apply at /source/src/julia.h:2157 [inlined] do_call at /source/src/interpreter.c:126 eval_value at /source/src/interpreter.c:223 eval_stmt_value at /source/src/interpreter.c:174 [inlined] eval_body at /source/src/interpreter.c:670 eval_body at /source/src/interpreter.c:539 eval_body at /source/src/interpreter.c:539 jl_interpret_toplevel_thunk at /source/src/interpreter.c:824 jl_toplevel_eval_flex at /source/src/toplevel.c:943 jl_toplevel_eval_flex at /source/src/toplevel.c:886 ijl_toplevel_eval_in at /source/src/toplevel.c:994 eval at ./boot.jl:430 [inlined] include_string at ./loading.jl:2734 _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 _include at ./loading.jl:2794 include at ./Base.jl:562 jfptr_include_47034.1 at /opt/julia/lib/julia/sys.so (unknown line) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 exec_options at ./client.jl:323 _start at ./client.jl:531 jfptr__start_73694.1 at /opt/julia/lib/julia/sys.so (unknown line) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 jl_apply at /source/src/julia.h:2157 [inlined] true_main at /source/src/jlapi.c:900 jl_repl_entrypoint at /source/src/jlapi.c:1059 main at /source/cli/loader_exe.c:58 unknown function (ip: 0x7f9f018e2249) __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) unknown function (ip: 0x4010b8) unknown function (ip: (nil)) ============================================================== Profile collected. A report will print at the next yield point ============================================================== ┌ Warning: There were no samples collected in one or more groups. │ This may be due to idle threads, or you may need to run your │ program longer (perhaps by running it multiple times), │ or adjust the delay between samples with `Profile.init()`. └ @ Profile /opt/julia/share/julia/stdlib/v1.11/Profile/src/Profile.jl:1240 Overhead ╎ [+additional indent] Count File:Line; Function ========================================================= Thread 1 Task 0x00007f9ef4200010 Total snapshots: 1. Utilization: 0% ╎1 @Base/client.jl:531; _start() ╎ 1 @Base/client.jl:323; exec_options(opts::Base.JLOptions) ╎ 1 @Base/Base.jl:562; include(mod::Module, _path::String) ╎ 1 @Base/loading.jl:2794; _include(mapexpr::Function, mod::Module, _path::S… ╎ 1 @Base/loading.jl:2734; include_string(mapexpr::typeof(identity), mod::M… ╎ 1 @Base/boot.jl:430; eval ╎ ╎ 1 @Pkg/src/API.jl:146; kwcall(::@NamedTuple{julia_args::Cmd}, ::typeof(… ╎ ╎ 1 @Pkg/src/API.jl:146; #test#74 ╎ ╎ 1 @Pkg/src/API.jl:147; test ╎ ╎ 1 @Pkg/src/API.jl:147; #test#75 ╎ ╎ 1 @Pkg/src/API.jl:148; kwcall(::@NamedTuple{julia_args::Cmd}, ::typ… ╎ ╎ ╎ 1 @Pkg/src/API.jl:159; test(pkgs::Vector{Pkg.Types.PackageSpec}; i… ╎ ╎ ╎ 1 @Pkg/src/API.jl:460; kwcall(::@NamedTuple{julia_args::Cmd, io::… ╎ ╎ ╎ 1 @Pkg/src/API.jl:481; test(ctx::Pkg.Types.Context, pkgs::Vector… ╎ ╎ ╎ 1 …/src/Operations.jl:2011; test ╎ ╎ ╎ 1 …src/Operations.jl:2067; test(ctx::Pkg.Types.Context, pkgs::… ╎ ╎ ╎ ╎ 1 …src/Operations.jl:1841; kwcall(::@NamedTuple{preferences::… ╎ ╎ ╎ ╎ 1 …src/Operations.jl:1849; #sandbox#115 ╎ ╎ ╎ ╎ 1 @Base/file.jl:815; mktempdir ╎ ╎ ╎ ╎ 1 @Base/file.jl:815; mktempdir(fn::Function, parent::Strin… ╎ ╎ ╎ ╎ 1 @Base/file.jl:819; mktempdir(fn::Pkg.Operations.var"#11… ╎ ╎ ╎ ╎ ╎ 1 …c/Operations.jl:1902; (::Pkg.Operations.var"#116#121"… ╎ ╎ ╎ ╎ ╎ 1 …c/Operations.jl:1793; with_temp_env(fn::Pkg.Operatio… ╎ ╎ ╎ ╎ ╎ 1 …/Operations.jl:1935; (::Pkg.Operations.var"#118#123… ╎ ╎ ╎ ╎ ╎ 1 @Base/env.jl:265; withenv(::Pkg.Operations.var"#131… ╎ ╎ ╎ ╎ ╎ 1 …Operations.jl:2086; (::Pkg.Operations.var"#131#13… ╎ ╎ ╎ ╎ ╎ ╎ 1 …Operations.jl:2146; subprocess_handler(cmd::Cmd,… ╎ ╎ ╎ ╎ ╎ ╎ 1 …se/process.jl:687; wait(x::Base.Process) ╎ ╎ ╎ ╎ ╎ ╎ 1 …se/process.jl:694; wait(x::Base.Process, syncd… ╎ ╎ ╎ ╎ ╎ ╎ 1 …condition.jl:125; wait ╎ ╎ ╎ ╎ ╎ ╎ 1 …condition.jl:130; wait(c::Base.GenericCondit… ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 …ase/task.jl:1021; wait() ╎ ╎ ╎ ╎ ╎ ╎ ╎ 1 …ase/task.jl:1012; poptask(W::Base.Intrusiv… From worker 3: From worker 3: [200] signal 15: Terminated From worker 3: in expression starting at none:0 From worker 3: epoll_pwait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) From worker 3: uv__io_poll at /workspace/srcdir/libuv/src/unix/linux.c:1404 From worker 3: uv_run at /workspace/srcdir/libuv/src/unix/core.c:430 From worker 3: ijl_task_get_next at /source/src/scheduler.c:522 From worker 4: From worker 4: [202] signal 15: Terminated From worker 4: in expression starting at none:0 From worker 4: epoll_pwait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) From worker 4: uv__io_poll at /workspace/srcdir/libuv/src/unix/linux.c:1404 From worker 4: uv_run at /workspace/srcdir/libuv/src/unix/core.c:430 From worker 4: ijl_task_get_next at /source/src/scheduler.c:522 From worker 2: From worker 2: [196] signal 15: Terminated From worker 2: in expression starting at none:0 [192] signal 15: Terminated in expression starting at /home/pkgeval/.julia/packages/Dagger/cPlyf/test/datadeps.jl:513 epoll_pwait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) uv__io_poll at /workspace/srcdir/libuv/src/unix/linux.c:1404 uv_run at /workspace/srcdir/libuv/src/unix/core.c:430 ijl_task_get_next at /source/src/scheduler.c:522 poptask at ./task.jl:1012 wait at ./task.jl:1021 #wait#733 at ./condition.jl:130 wait at ./condition.jl:125 [inlined] fetch_buffered at ./channels.jl:447 fetch at ./channels.jl:441 unknown function (ip: 0x7aab7bcadd92) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 #remotecall_wait#164 at /source/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:503 remotecall_wait at /source/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:497 [inlined] #remotecall_wait#167 at /source/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:519 [inlined] remotecall_wait at /source/usr/share/julia/stdlib/v1.11/Distributed/src/remotecall.jl:519 [inlined] #146 at /home/pkgeval/.julia/packages/Dagger/cPlyf/src/sch/Sch.jl:1100 unknown function (ip: 0x7aab7474949f) _jl_invoke at /source/src/gf.c:2948 [inlined] ijl_apply_generic at /source/src/gf.c:3125 jl_apply at /source/src/julia.h:2157 [inlined] start_task at /source/src/task.c:1202 unknown function (ip: (nil)) Allocations: 293647435 (Pool: 293626932; Big: 20503); GC: 172 PkgEval terminated after 2767.22s: test duration exceeded the time limit