Package evaluation of RustyObjectStore on Julia 1.13.0-DEV.745 (a23ce4bcbb*) started at 2025-06-15T15:52:52.763 ################################################################################ # Set-up # Installing PkgEval dependencies (TestEnv)... Set-up completed after 7.49s ################################################################################ # Installation # Installing RustyObjectStore... Resolving package versions... Updating `~/.julia/environments/v1.13/Project.toml` [1b5eed3d] + RustyObjectStore v0.12.3 Updating `~/.julia/environments/v1.13/Manifest.toml` [d1d4a3ce] + BitFlags v0.1.9 [85eb1798] + CloudBase v1.4.8 [944b1d66] + CodecZlib v0.7.8 [f0e56b4a] + ConcurrentUtilities v2.5.0 [ffbed154] + DocStringExtensions v0.9.5 [460bff9d] + ExceptionUnwrapping v0.1.11 [937e9308] + Figgy v1.1.1 [069b7b12] + FunctionWrappers v1.1.3 [cd3eb016] + HTTP v1.10.16 [692b3bcd] + JLLWrappers v1.7.0 [0f8b85d8] + JSON3 v1.14.3 [e6f89c97] + LoggingExtras v1.1.0 [6ac74813] + MD5 v0.2.3 [739be429] + MbedTLS v1.1.9 [4d8831e6] + OpenSSL v1.5.0 [69de0a69] + Parsers v2.8.3 [aea7be01] + PrecompileTools v1.3.2 [21216c6a] + Preferences v1.4.3 [1b5eed3d] + RustyObjectStore v0.12.3 [777ac1f9] + SimpleBufferStream v1.2.0 [856f2bd8] + StructTypes v1.11.0 [3bb67fe8] + TranscodingStreams v0.11.3 [5c2747f8] + URIs v1.5.2 [c8ffd9c3] + MbedTLS_jll v2.28.6+2 [a4b94fbf] + NodeJS_16_jll v16.15.0+0 [d59cea07] + azurite_jll v3.29.0+0 [ad83bbfc] + minio_jll v1.0.0+1 [0e112785] + object_store_ffi_jll v0.12.3+0 [0dad84c5] + ArgTools v1.1.2 [56f22d72] + Artifacts v1.11.0 [2a0f44e3] + Base64 v1.11.0 [ade2ca70] + Dates v1.11.0 [f43a241f] + Downloads v1.7.0 [7b1f6079] + FileWatching v1.11.0 [b77e0a4c] + InteractiveUtils v1.11.0 [ac6e5ff7] + JuliaSyntaxHighlighting v1.12.0 [b27032c2] + LibCURL v0.6.4 [76f85450] + LibGit2 v1.11.0 [8f399da3] + Libdl v1.11.0 [56ddb016] + Logging v1.11.0 [d6f4376e] + Markdown v1.11.0 [a63ad114] + Mmap v1.11.0 [ca575930] + NetworkOptions v1.3.0 [44cfe95a] + Pkg v1.13.0 [de0858da] + Printf v1.11.0 [9a3f8284] + Random v1.11.0 [ea8e919c] + SHA v0.7.0 [9e88b42a] + Serialization v1.11.0 [6462fe0b] + Sockets v1.11.0 [f489334b] + StyledStrings v1.11.0 [fa267f1f] + TOML v1.0.3 [a4e569a6] + Tar v1.10.0 [8dfed614] + Test v1.11.0 [cf7118a7] + UUIDs v1.11.0 [4ec0a83e] + Unicode v1.11.0 [e66e0078] + CompilerSupportLibraries_jll v1.3.0+1 [deac9b47] + LibCURL_jll v8.14.1+1 [e37daf67] + LibGit2_jll v1.9.0+0 [29816b5a] + LibSSH2_jll v1.11.3+1 [14a3606d] + MozillaCACerts_jll v2025.5.20 [458c3c95] + OpenSSL_jll v3.5.0+0 [83775a58] + Zlib_jll v1.3.1+2 [8e850ede] + nghttp2_jll v1.65.0+0 [3f19e933] + p7zip_jll v17.5.0+2 Installation completed after 1.54s ################################################################################ # Precompilation # Precompiling PkgEval dependencies... Precompiling package dependencies... Precompilation completed after 30.61s ################################################################################ # Testing # Testing RustyObjectStore Status `/tmp/jl_7aKpwP/Project.toml` [85eb1798] CloudBase v1.4.8 [944b1d66] CodecZlib v0.7.8 [ffbed154] DocStringExtensions v0.9.5 [cd3eb016] HTTP v1.10.16 [0f8b85d8] JSON3 v1.14.3 [817f1d60] ReTestItems v1.31.0 [1b5eed3d] RustyObjectStore v0.12.3 [0e112785] object_store_ffi_jll v0.12.3+0 [2a0f44e3] Base64 v1.11.0 [6462fe0b] Sockets v1.11.0 [8dfed614] Test v1.11.0 Status `/tmp/jl_7aKpwP/Manifest.toml` [d1d4a3ce] BitFlags v0.1.9 [85eb1798] CloudBase v1.4.8 [944b1d66] CodecZlib v0.7.8 [f0e56b4a] ConcurrentUtilities v2.5.0 [ffbed154] DocStringExtensions v0.9.5 [460bff9d] ExceptionUnwrapping v0.1.11 [937e9308] Figgy v1.1.1 [069b7b12] FunctionWrappers v1.1.3 [cd3eb016] HTTP v1.10.16 [692b3bcd] JLLWrappers v1.7.0 [0f8b85d8] JSON3 v1.14.3 [e6f89c97] LoggingExtras v1.1.0 [6ac74813] MD5 v0.2.3 [739be429] MbedTLS v1.1.9 [4d8831e6] OpenSSL v1.5.0 [69de0a69] Parsers v2.8.3 [aea7be01] PrecompileTools v1.3.2 [21216c6a] Preferences v1.4.3 [817f1d60] ReTestItems v1.31.0 [1b5eed3d] RustyObjectStore v0.12.3 [777ac1f9] SimpleBufferStream v1.2.0 [856f2bd8] StructTypes v1.11.0 [1e6cf692] TestEnv v1.102.1 [3bb67fe8] TranscodingStreams v0.11.3 [5c2747f8] URIs v1.5.2 [c8ffd9c3] MbedTLS_jll v2.28.6+2 [a4b94fbf] NodeJS_16_jll v16.15.0+0 [d59cea07] azurite_jll v3.29.0+0 [ad83bbfc] minio_jll v1.0.0+1 [0e112785] object_store_ffi_jll v0.12.3+0 [0dad84c5] ArgTools v1.1.2 [56f22d72] Artifacts v1.11.0 [2a0f44e3] Base64 v1.11.0 [ade2ca70] Dates v1.11.0 [f43a241f] Downloads v1.7.0 [7b1f6079] FileWatching v1.11.0 [b77e0a4c] InteractiveUtils v1.11.0 [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0 [b27032c2] LibCURL v0.6.4 [76f85450] LibGit2 v1.11.0 [8f399da3] Libdl v1.11.0 [56ddb016] Logging v1.11.0 [d6f4376e] Markdown v1.11.0 [a63ad114] Mmap v1.11.0 [ca575930] NetworkOptions v1.3.0 [44cfe95a] Pkg v1.13.0 [de0858da] Printf v1.11.0 [9a3f8284] Random v1.11.0 [ea8e919c] SHA v0.7.0 [9e88b42a] Serialization v1.11.0 [6462fe0b] Sockets v1.11.0 [f489334b] StyledStrings v1.11.0 [fa267f1f] TOML v1.0.3 [a4e569a6] Tar v1.10.0 [8dfed614] Test v1.11.0 [cf7118a7] UUIDs v1.11.0 [4ec0a83e] Unicode v1.11.0 [e66e0078] CompilerSupportLibraries_jll v1.3.0+1 [deac9b47] LibCURL_jll v8.14.1+1 [e37daf67] LibGit2_jll v1.9.0+0 [29816b5a] LibSSH2_jll v1.11.3+1 [14a3606d] MozillaCACerts_jll v2025.5.20 [458c3c95] OpenSSL_jll v3.5.0+0 [83775a58] Zlib_jll v1.3.1+2 [8e850ede] nghttp2_jll v1.65.0+0 [3f19e933] p7zip_jll v17.5.0+2 Testing Running tests... [ Info: Scanning for test items in project `RustyObjectStore` at paths: /home/pkgeval/.julia/packages/RustyObjectStore/JF6aG [ Info: Finished scanning for test items in 5.93 seconds. Scheduling 20 tests on pid 35 with 1 worker processes and 2 threads per worker. [ Info: Starting test workers Worker 39: [ Info: Starting test worker 1 on pid = 39, with 2 threads [ Info: Starting running test items Worker 39: 15:54:40 | maxrss 2.0% | mem 3.2% | START ( 1/20) test item "AWSConfig" at test/aws_api_tests.jl:1 Worker 39: 15:54:45 | maxrss 2.0% | mem 3.3% | DONE ( 1/20) test item "AWSConfig" 4.6 secs (69.4% compile, 23.6% recompile, 3.4% GC), 2.91 M allocs (170.172 MB) Worker 39: 15:54:49 | maxrss 2.0% | mem 3.4% | START ( 2/20) test item "Basic S3 exceptions" at test/aws_s3_exception_tests.jl:1 Worker 39: 15:55:39 | maxrss 2.4% | mem 4.1% | DONE ( 2/20) test item "Basic S3 exceptions" 49.8 secs (91.8% compile, 2.0% recompile, 2.1% GC), 20.69 M allocs (1.173 GB) ┌ Warning: Test item "Basic S3 exceptions" at test/aws_s3_exception_tests.jl:1 contains test sets without tests: │ "Delete non-existing file" └ @ ReTestItems ~/.julia/packages/ReTestItems/Q6zs8/src/log_capture.jl:328 Worker 39: 15:55:42 | maxrss 2.4% | mem 4.1% | START ( 3/20) test item "AWS S3 retries" at test/aws_s3_exception_tests.jl:159 Worker 39: 15:56:24 | maxrss 2.4% | mem 4.1% | DONE ( 3/20) test item "AWS S3 retries" 41.9 secs (23.7% compile, 0.3% recompile, 0.2% GC), 6.36 M allocs (386.636 MB) Worker 39: 15:56:31 | maxrss 2.4% | mem 4.1% | START ( 4/20) test item "AzureConfig" at test/azure_api_tests.jl:1 Worker 39: 15:56:31 | maxrss 2.4% | mem 4.1% | DONE ( 4/20) test item "AzureConfig" 0.4 secs (95.0% compile), 78.67 K allocs (4.589 MB) Worker 39: 15:56:32 | maxrss 2.4% | mem 4.1% | START ( 5/20) test item "Basic BlobStorage exceptions" at test/azure_blobs_exception_tests.jl:1 Worker 39: 15:57:06 | maxrss 2.8% | mem 4.2% | DONE ( 5/20) test item "Basic BlobStorage exceptions" 34.0 secs (78.2% compile, <0.1% recompile, 2.2% GC), 9.26 M allocs (620.733 MB) Worker 39: 15:57:10 | maxrss 2.8% | mem 4.2% | START ( 6/20) test item "BlobStorage retries" at test/azure_blobs_exception_tests.jl:321 Worker 39: 15:57:47 | maxrss 2.8% | mem 4.3% | DONE ( 6/20) test item "BlobStorage retries" 37.5 secs (14.6% compile, <0.1% recompile, 0.3% GC), 4.90 M allocs (338.846 MB) Worker 39: [ Info: Server on 127.0.0.1:8081 closing Worker 39: 15:57:54 | maxrss 2.8% | mem 4.3% | START ( 7/20) test item "Basic BlobStorage usage" at test/basic_unified_tests.jl:686 Worker 39: 15:59:15 | maxrss 5.2% | mem 7.1% | DONE ( 7/20) test item "Basic BlobStorage usage" 79.8 secs (58.4% compile, <0.1% recompile, 3.0% GC), 12.15 M allocs (2.226 GB) Worker 39: 15:59:26 | maxrss 5.2% | mem 7.1% | START ( 8/20) test item "Basic BlobStorage usage (anonymous read enabled)" at test/basic_unified_tests.jl:719 Worker 39: 15:59:32 | maxrss 5.3% | mem 7.2% | DONE ( 8/20) test item "Basic BlobStorage usage (anonymous read enabled)" 5.4 secs (33.5% compile, <0.1% recompile, 13.7% GC), 1.52 M allocs (375.217 MB) Worker 39: 15:59:36 | maxrss 5.3% | mem 7.2% | START ( 9/20) test item "Basic AWS S3 usage" at test/basic_unified_tests.jl:744 Worker 39: 16:00:41 | maxrss 5.7% | mem 6.8% | DONE ( 9/20) test item "Basic AWS S3 usage" 65.0 secs (66.3% compile, <0.1% recompile, 4.2% GC), 8.54 M allocs (2.024 GB) Worker 39: 16:00:52 | maxrss 5.7% | mem 6.8% | START (10/20) test item "Basic AWS S3 usage (anonymous read enabled)" at test/basic_unified_tests.jl:778 Worker 39: 16:00:56 | maxrss 5.7% | mem 7.5% | DONE (10/20) test item "Basic AWS S3 usage (anonymous read enabled)" 3.8 secs (27.6% compile, 5.0% GC), 1.51 M allocs (374.937 MB) Worker 39: 16:01:00 | maxrss 5.7% | mem 7.5% | START (11/20) test item "Basic Snowflake Stage usage: AWS, non-encrypted" at test/basic_unified_tests.jl:805 Worker 39: 16:02:30 | maxrss 6.0% | mem 8.0% | DONE (11/20) test item "Basic Snowflake Stage usage: AWS, non-encrypted" 90.0 secs (67.4% compile, <0.1% recompile, 2.7% GC), 14.93 M allocs (2.135 GB) Worker 39: 16:02:41 | maxrss 6.0% | mem 7.9% | START (12/20) test item "Basic Snowflake Stage usage: AWS, encrypted" at test/basic_unified_tests.jl:822 Worker 39: 16:04:14 | maxrss 6.4% | mem 6.4% | DONE (12/20) test item "Basic Snowflake Stage usage: AWS, encrypted" 92.4 secs (10.1% compile, <0.1% recompile, 1.8% GC), 6.64 M allocs (1.435 GB) Worker 39: 16:04:25 | maxrss 6.4% | mem 6.3% | START (13/20) test item "Basic Snowflake Stage usage: Azure, non-encrypted" at test/basic_unified_tests.jl:839 Worker 39: 16:05:07 | maxrss 6.4% | mem 7.3% | DONE (13/20) test item "Basic Snowflake Stage usage: Azure, non-encrypted" 42.1 secs (8.9% compile, 4.2% GC), 4.11 M allocs (1.283 GB) Worker 39: 16:05:19 | maxrss 6.4% | mem 7.3% | START (14/20) test item "Basic Snowflake Stage usage: Azure, encrypted" at test/basic_unified_tests.jl:856 Worker 39: 16:06:07 | maxrss 6.4% | mem 7.5% | DONE (14/20) test item "Basic Snowflake Stage usage: Azure, encrypted" 48.2 secs (21.2% compile, 4.1% GC), 6.23 M allocs (1.411 GB) Worker 39: 16:06:19 | maxrss 6.4% | mem 7.5% | START (15/20) test item "destroy_* functions do not panic" at test/destroy_test.jl:1 Worker 39: 16:06:19 | maxrss 6.4% | mem 7.5% | DONE (15/20) test item "destroy_* functions do not panic" <0.1 secs (77.1% compile), 2.08 K allocs (97.488 KB) Worker 39: 16:06:19 | maxrss 6.4% | mem 7.5% | START (16/20) test item "Using object_store_ffi_jll" at test/release_test.jl:4 Worker 39: 16:06:19 | maxrss 6.4% | mem 7.5% | DONE (16/20) test item "Using object_store_ffi_jll" <0.1 secs (<0.1% compile), 1.31 K allocs (72.352 KB) Worker 39: 16:06:20 | maxrss 6.4% | mem 7.5% | START (17/20) test item "Handle object_store_ffi panic" at test/rust_panic_test.jl:1 Worker 39: 16:06:27 | maxrss 6.4% | mem 7.5% | DONE (17/20) test item "Handle object_store_ffi panic" 6.8 secs (6.3% compile), 149.90 K allocs (8.715 MB) Worker 39: 16:06:28 | maxrss 6.4% | mem 7.5% | START (18/20) test item "SnowflakeConfig" at test/snowflake_api_tests.jl:1 Worker 39: 16:06:28 | maxrss 6.4% | mem 7.5% | DONE (18/20) test item "SnowflakeConfig" 0.3 secs (92.8% compile), 110.46 K allocs (6.966 MB) Worker 39: 16:06:29 | maxrss 6.4% | mem 7.5% | START (19/20) test item "Basic Stage exceptions" at test/snowflake_stage_exception_tests.jl:1 Worker 39: 16:06:49 | maxrss 6.4% | mem 7.5% | DONE (19/20) test item "Basic Stage exceptions" 20.3 secs (20.4% compile), 1.47 M allocs (91.289 MB) Worker 39: 2025-06-15T16:06:51.571122Z  WARN object_store_ffi::snowflake::client: Heartbeat failed: error sending request for url (http://127.0.0.1:8081/session/heartbeat) Worker 39: Caused by: Worker 39: 0: error sending request for url (http://127.0.0.1:8081/session/heartbeat) Worker 39: 1: client error (Connect) Worker 39: 2: tcp connect error: Connection refused (os error 111) Worker 39: 3: Connection refused (os error 111) Worker 39: 16:06:54 | maxrss 6.4% | mem 7.5% | START (20/20) test item "Snowflake Stage retries" at test/snowflake_stage_exception_tests.jl:158 Worker 39: 16:08:10 | maxrss 6.4% | mem 7.5% | DONE (20/20) test item "Snowflake Stage retries" 76.1 secs (12.7% compile, <0.1% recompile, 0.2% GC), 5.18 M allocs (350.043 MB) Captured Logs for test item "Snowflake Stage retries" at test/snowflake_stage_exception_tests.jl:158 on worker 39 2025-06-15T16:06:55.940410Z  WARN object_store_ffi: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 161.938239ms 2025-06-15T16:06:56.020172Z  WARN object_store_ffi: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 853.312µs 2025-06-15T16:06:56.807889Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 73.103148ms 2025-06-15T16:06:57.460352Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 762.542µs 2025-06-15T16:06:58.313951Z  WARN object_store_ffi: The operation lacked the necessary privileges to complete for path xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: The operation lacked the necessary privileges to complete for path xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 944.87µs 2025-06-15T16:06:58.985017Z  WARN object_store_ffi: The operation lacked the necessary privileges to complete for path xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: The operation lacked the necessary privileges to complete for path xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 999.17µs 2025-06-15T16:06:59.631740Z  WARN object_store_ffi: The operation lacked the necessary privileges to complete for path blob: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: The operation lacked the necessary privileges to complete for path blob: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 816.062µs 2025-06-15T16:07:00.289482Z  WARN object_store_ffi: The operation lacked the necessary privileges to complete for path blob: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: The operation lacked the necessary privileges to complete for path blob: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 836.441µs 2025-06-15T16:07:01.133849Z  WARN object_store_ffi: Object at location xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob not found: Client error with status 404 Not Found: response body from the dummy server Caused by: 0: Object at location xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob not found: Client error with status 404 Not Found: response body from the dummy server 1: Client error with status 404 Not Found: response body from the dummy server Recent attempts (1 out of 1): reason: Code(404) after 0 retries Total retries: 0 Total Time: 826.832µs 2025-06-15T16:07:01.794596Z  WARN object_store_ffi: Object at location blob not found: Client error with status 404 Not Found: response body from the dummy server Caused by: 0: Object at location blob not found: Client error with status 404 Not Found: response body from the dummy server 1: Client error with status 404 Not Found: response body from the dummy server Recent attempts (1 out of 1): reason: Code(404) after 0 retries Total retries: 0 Total Time: 674.264µs 2025-06-15T16:07:03.049810Z  WARN object_store_ffi: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 154.358925ms 2025-06-15T16:07:03.120830Z  WARN object_store_ffi: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 960.31µs 2025-06-15T16:07:03.773581Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 890.081µs 2025-06-15T16:07:04.418660Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 1.04832ms 2025-06-15T16:07:05.272401Z  WARN object_store_ffi: Object at location xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 881.791µs 2025-06-15T16:07:05.935029Z  WARN object_store_ffi: Object at location xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 1.011559ms 2025-06-15T16:07:06.544658Z  WARN object_store_ffi: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 834.192µs 2025-06-15T16:07:07.205307Z  WARN object_store_ffi: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 914.501µs 2025-06-15T16:07:08.057848Z  WARN object_store_ffi: Request precondition failure for path xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 985.901µs 2025-06-15T16:07:08.734069Z  WARN object_store_ffi: Request precondition failure for path xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 905.51µs 2025-06-15T16:07:09.326260Z  WARN object_store_ffi: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 759.233µs 2025-06-15T16:07:09.984446Z  WARN object_store_ffi: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 788.322µs 2025-06-15T16:07:10.817113Z  WARN object_store_ffi: Generic S3 error: Client error with status 413 Payload Too Large: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 413 Payload Too Large: response body from the dummy server 1: Client error with status 413 Payload Too Large: response body from the dummy server Recent attempts (1 out of 1): reason: Code(413) after 0 retries Total retries: 0 Total Time: 810.622µs 2025-06-15T16:07:11.469311Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 413 Payload Too Large: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 413 Payload Too Large: response body from the dummy server 1: Client error with status 413 Payload Too Large: response body from the dummy server Recent attempts (1 out of 1): reason: Code(413) after 0 retries Total retries: 0 Total Time: 874.681µs 2025-06-15T16:07:12.601936Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 966.441µs 2025-06-15T16:07:13.242192Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 845.362µs 2025-06-15T16:07:14.009599Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 145.138306ms 2025-06-15T16:07:14.683425Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 21.9332ms 2025-06-15T16:07:15.309002Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 670.493µs 2025-06-15T16:07:15.958172Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 733.303µs 2025-06-15T16:07:16.555966Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 773.952µs 2025-06-15T16:07:17.208114Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 806.172µs 2025-06-15T16:07:18.392306Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 210.999107ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 210.999107ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 1: Error after 2 retries in 210.999107ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 211.201516ms 2025-06-15T16:07:19.327357Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 277.070557ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 277.070557ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob) 1: Error after 2 retries in 277.070557ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 277.365433ms 2025-06-15T16:07:20.182910Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 252.058647ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 252.058647ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/dummy/blob) 1: Error after 2 retries in 252.058647ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/dummy/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/dummy/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 252.221055ms 2025-06-15T16:07:21.094576Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 262.884207ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 262.884207ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/dummy/blob) 1: Error after 2 retries in 262.884207ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/dummy/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/dummy/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 263.130925ms 2025-06-15T16:07:22.266150Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 300.088296ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 300.088296ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 1: Error after 2 retries in 300.088296ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 300.284674ms 2025-06-15T16:07:23.163887Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 235.908248ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 235.908248ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob) 1: Error after 2 retries in 235.908248ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 236.171876ms 2025-06-15T16:07:24.023015Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 218.898398ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 218.898398ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) 1: Error after 2 retries in 218.898398ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 219.054027ms 2025-06-15T16:07:24.936679Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 253.802679ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 253.802679ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) 1: Error after 2 retries in 253.802679ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 258.110026ms 2025-06-15T16:07:26.106780Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 288.964667ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 288.964667ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 1: Error after 2 retries in 288.964667ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 289.256374ms 2025-06-15T16:07:27.034616Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 271.810559ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 271.810559ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob) 1: Error after 2 retries in 271.810559ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/b4b23aa5-50d6-b3d0-b4b2-3aa550d6b3d0/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 272.164635ms 2025-06-15T16:07:27.843060Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 208.202116ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 208.202116ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/dummy/blob) 1: Error after 2 retries in 208.202116ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/dummy/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/dummy/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 208.360003ms 2025-06-15T16:07:28.813404Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 293.315704ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 293.315704ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/dummy/blob) 1: Error after 2 retries in 293.315704ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/dummy/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/dummy/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 293.571771ms 2025-06-15T16:07:33.348929Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 3.218213449s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 3.218213449s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 1: Error after 2 retries in 3.218213449s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 2: error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.218432037s 2025-06-15T16:07:37.835947Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 3.252882472s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/c427d321-2b2c-53d1-c427-d3212b2c53d1/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 3.252882472s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/c427d321-2b2c-53d1-c427-d3212b2c53d1/blob) 1: Error after 2 retries in 3.252882472s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/c427d321-2b2c-53d1-c427-d3212b2c53d1/blob) 2: error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/c427d321-2b2c-53d1-c427-d3212b2c53d1/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.253173269s 2025-06-15T16:07:42.281070Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 3.243472507s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 3.243472507s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) 1: Error after 2 retries in 3.243472507s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) 2: error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.243638734s [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/MIUdD/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:242 [inlined] │ [5] write │ @ ./io.jl:803 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/MIUdD/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:242 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/MIUdD/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob HTTP/1.1 │ host: 127.0.0.1:8081 │ x-amz-date: 20250615T160730Z │ x-amz-content-sha256: UNSIGNED-PAYLOAD │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/MIUdD/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:242 [inlined] │ [5] write │ @ ./io.jl:803 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/MIUdD/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:242 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/MIUdD/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob HTTP/1.1 │ host: 127.0.0.1:8081 │ x-amz-date: 20250615T160730Z │ x-amz-content-sha256: UNSIGNED-PAYLOAD │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/MIUdD/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:242 [inlined] │ [5] write │ @ ./io.jl:803 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/MIUdD/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:242 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/MIUdD/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob HTTP/1.1 │ host: 127.0.0.1:8081 │ x-amz-date: 20250615T160730Z │ x-amz-content-sha256: UNSIGNED-PAYLOAD │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/MIUdD/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:242 [inlined] │ [5] write │ @ ./io.jl:803 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/MIUdD/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:242 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/MIUdD/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /dummy/xox50000-s/stages/c427d321-2b2c-53d1-c427-d3212b2c53d1/blob HTTP/1.1 │ content-length: 16 │ x-amz-meta-x-amz-cek-alg: AES/CBC/PKCS5Padding │ x-amz-meta-x-amz-unencrypted-content-length: 5 │ x-amz-meta-x-amz-iv: bZc5Fgna/aApK+a68w5U2A== │ x-amz-meta-x-amz-matdesc: {"smkId":"1","queryId":"1","keySize":"128"} │ x-amz-meta-x-amz-key: CtSPcOdGk48gNPJEBfAvCvTTIZR/1TLYeoL4P32NwwM= │ host: 127.0.0.1:8081 │ x-amz-date: 20250615T160734Z │ x-amz-content-sha256: UNSIGNED-PAYLOAD │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ │ │ ⋮ │ 16-byte body │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/MIUdD/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:242 [inlined] │ [5] write │ @ ./io.jl:803 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/MIUdD/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:242 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/MIUdD/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /dummy/xox50000-s/stages/c427d321-2b2c-53d1-c427-d3212b2c53d1/blob HTTP/1.1 │ content-length: 16 │ x-amz-meta-x-amz-cek-alg: AES/CBC/PKCS5Padding │ x-amz-meta-x-amz-unencrypted-content-length: 5 │ x-amz-meta-x-amz-iv: bZc5Fgna/aApK+a68w5U2A== │ x-amz-meta-x-amz-matdesc: {"smkId":"1","queryId":"1","keySize":"128"} │ x-amz-meta-x-amz-key: CtSPcOdGk48gNPJEBfAvCvTTIZR/1TLYeoL4P32NwwM= │ host: 127.0.0.1:8081 │ x-amz-date: 20250615T160734Z │ x-amz-content-sha256: UNSIGNED-PAYLOAD │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ │ │ ⋮ │ 16-byte body │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/MIUdD/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:242 [inlined] │ [5] write │ @ ./io.jl:803 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/MIUdD/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:242 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/MIUdD/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /dummy/xox50000-s/stages/c427d321-2b2c-53d1-c427-d3212b2c53d1/blob HTTP/1.1 │ content-length: 16 │ x-amz-meta-x-amz-cek-alg: AES/CBC/PKCS5Padding │ x-amz-meta-x-amz-unencrypted-content-length: 5 │ x-amz-meta-x-amz-iv: bZc5Fgna/aApK+a68w5U2A== │ x-amz-meta-x-amz-matdesc: {"smkId":"1","queryId":"1","keySize":"128"} │ x-amz-meta-x-amz-key: CtSPcOdGk48gNPJEBfAvCvTTIZR/1TLYeoL4P32NwwM= │ host: 127.0.0.1:8081 │ x-amz-date: 20250615T160734Z │ x-amz-content-sha256: UNSIGNED-PAYLOAD │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ │ │ ⋮ │ 16-byte body │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/MIUdD/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:242 [inlined] │ [5] write │ @ ./io.jl:803 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/MIUdD/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:242 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/MIUdD/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /dummy/dummy/blob HTTP/1.1 │ content-length: 0 │ date: Sun, 15 Jun 2025 16:07:39 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/MIUdD/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:242 [inlined] │ [5] write │ @ ./io.jl:803 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/MIUdD/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:242 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/MIUdD/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /dummy/dummy/blob HTTP/1.1 │ content-length: 0 │ date: Sun, 15 Jun 2025 16:07:39 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing 2025-06-15T16:07:46.734911Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 3.23903852s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 3.23903852s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) 1: Error after 2 retries in 3.23903852s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) 2: error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.239281788s 2025-06-15T16:07:48.887172Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 283.65573ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 283.65573ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 1: Error after 2 retries in 283.65573ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 283.869688ms 2025-06-15T16:07:49.256162Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 320.516751ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/69ae7e2f-ff23-268b-69ae-7e2fff23268b/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 320.516751ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/69ae7e2f-ff23-268b-69ae-7e2fff23268b/blob) 1: Error after 2 retries in 320.516751ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/69ae7e2f-ff23-268b-69ae-7e2fff23268b/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/69ae7e2f-ff23-268b-69ae-7e2fff23268b/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 320.801788ms 2025-06-15T16:07:49.594142Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 282.239144ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 282.239144ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 1: Error after 2 retries in 282.239144ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 282.554371ms 2025-06-15T16:07:49.923141Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 281.58723ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 281.58723ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 1: Error after 2 retries in 281.58723ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 281.830468ms 2025-06-15T16:07:50.833160Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 307.442682ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 307.442682ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 1: Error after 2 retries in 307.442682ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/168371ce-87dd-71cc-1683-71ce87dd71cc/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 307.66572ms 2025-06-15T16:07:51.223152Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 347.592359ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/69ae7e2f-ff23-268b-69ae-7e2fff23268b/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 347.592359ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/69ae7e2f-ff23-268b-69ae-7e2fff23268b/blob) 1: Error after 2 retries in 347.592359ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/69ae7e2f-ff23-268b-69ae-7e2fff23268b/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/69ae7e2f-ff23-268b-69ae-7e2fff23268b/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 347.943166ms 2025-06-15T16:07:51.565136Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 293.62244ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 293.62244ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 1: Error after 2 retries in 293.62244ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 293.801688ms 2025-06-15T16:07:51.938137Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 314.540841ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 314.540841ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 1: Error after 2 retries in 314.540841ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 314.795478ms 2025-06-15T16:07:52.704414Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-06-15T16:07:53.394161Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-06-15T16:07:53.920134Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-06-15T16:07:53.920257Z  WARN object_store_ffi: io error while streaming body: Generic S3 error: error decoding response body Caused by: 0: Generic S3 error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: end of file before message length reached Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 1.316848555s 2025-06-15T16:07:53.979789Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-06-15T16:07:54.704973Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-06-15T16:07:55.143876Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-06-15T16:07:55.143991Z  WARN object_store_ffi: io error while streaming body: Generic MicrosoftAzure error: error decoding response body Caused by: 0: Generic MicrosoftAzure error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: end of file before message length reached Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 1.164808988s 2025-06-15T16:07:55.947888Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-06-15T16:07:56.487945Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-06-15T16:07:57.508018Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-06-15T16:07:57.508134Z  WARN object_store_ffi: io error while streaming body: Generic S3 error: error decoding response body Caused by: 0: Generic S3 error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: Connection reset by peer (os error 104) Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 1.671091628s 2025-06-15T16:07:57.558389Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-06-15T16:07:57.958325Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-06-15T16:07:58.878747Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-06-15T16:07:58.878863Z  WARN object_store_ffi: io error while streaming body: Generic MicrosoftAzure error: error decoding response body Caused by: 0: Generic MicrosoftAzure error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: Connection reset by peer (os error 104) Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 1.321181692s 2025-06-15T16:08:00.554485Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-06-15T16:08:01.986871Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-06-15T16:08:03.549046Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-06-15T16:08:03.549163Z  WARN object_store_ffi: io error while streaming body: Generic S3 error: error decoding response body Caused by: 0: Generic S3 error: error decoding response body 1: error decoding response body 2: request or response body error 3: operation timed out Recent attempts (3 out of 3): reason: Timeout after 0 retries reason: Timeout after 0 retries reason: Timeout after 0 retries Total retries: 2 Total Time: 3.995705094s 2025-06-15T16:08:04.615896Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-06-15T16:08:06.155312Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-06-15T16:08:07.898871Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-06-15T16:08:07.898999Z  WARN object_store_ffi: io error while streaming body: Generic MicrosoftAzure error: error decoding response body Caused by: 0: Generic MicrosoftAzure error: error decoding response body 1: error decoding response body 2: request or response body error 3: operation timed out Recent attempts (3 out of 3): reason: Timeout after 0 retries reason: Timeout after 0 retries reason: Timeout after 0 retries Total retries: 2 Total Time: 4.284200935s 2025-06-15T16:08:09.544771Z  WARN object_store_ffi: operation was cancelled 2025-06-15T16:08:10.629128Z  WARN object_store_ffi: operation was cancelled ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/MIUdD/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:242 [inlined] │ [5] write │ @ ./io.jl:803 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/MIUdD/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:242 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/MIUdD/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /dummy/dummy/blob HTTP/1.1 │ content-length: 0 │ date: Sun, 15 Jun 2025 16:07:39 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/MIUdD/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:242 [inlined] │ [5] write │ @ ./io.jl:803 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/MIUdD/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:242 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/MIUdD/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /dummy/dummy/blob HTTP/1.1 │ x-ms-meta-matdesc: {"smkId":"1","queryId":"1","keySize":"128"} │ x-ms-meta-encryptiondata: {"EncryptionMode":"FullBlob","WrappedContentKey":{"KeyId":"symmKey1","EncryptedKey":"C84qAwY+f7YKn5Nh8ekaBPTTIZR/1TLYeoL4P32NwwM=","Algorithm":"AES_CBC_256"},"ContentEncryptionIV":"fABM3trLe5m/ksex6LPAfw==","EncryptionAgent":{"Protocol":"1.0","EncryptionAlgorithm":"AES_CBC_128"},"KeyWrappingMetadata":{"EncryptionLibrary":"Java 5.3.0"}} │ x-ms-blob-type: BlockBlob │ content-length: 16 │ date: Sun, 15 Jun 2025 16:07:43 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ host: 127.0.0.1:8081 │ │ │ ⋮ │ 16-byte body │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/MIUdD/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:242 [inlined] │ [5] write │ @ ./io.jl:803 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/MIUdD/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:242 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/MIUdD/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /dummy/dummy/blob HTTP/1.1 │ x-ms-meta-matdesc: {"smkId":"1","queryId":"1","keySize":"128"} │ x-ms-meta-encryptiondata: {"EncryptionMode":"FullBlob","WrappedContentKey":{"KeyId":"symmKey1","EncryptedKey":"C84qAwY+f7YKn5Nh8ekaBPTTIZR/1TLYeoL4P32NwwM=","Algorithm":"AES_CBC_256"},"ContentEncryptionIV":"fABM3trLe5m/ksex6LPAfw==","EncryptionAgent":{"Protocol":"1.0","EncryptionAlgorithm":"AES_CBC_128"},"KeyWrappingMetadata":{"EncryptionLibrary":"Java 5.3.0"}} │ x-ms-blob-type: BlockBlob │ content-length: 16 │ date: Sun, 15 Jun 2025 16:07:43 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ host: 127.0.0.1:8081 │ │ │ ⋮ │ 16-byte body │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/MIUdD/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:242 [inlined] │ [5] write │ @ ./io.jl:803 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/MIUdD/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:242 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/MIUdD/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##Snowflake Stage retries#319".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /dummy/dummy/blob HTTP/1.1 │ x-ms-meta-matdesc: {"smkId":"1","queryId":"1","keySize":"128"} │ x-ms-meta-encryptiondata: {"EncryptionMode":"FullBlob","WrappedContentKey":{"KeyId":"symmKey1","EncryptedKey":"C84qAwY+f7YKn5Nh8ekaBPTTIZR/1TLYeoL4P32NwwM=","Algorithm":"AES_CBC_256"},"ContentEncryptionIV":"fABM3trLe5m/ksex6LPAfw==","EncryptionAgent":{"Protocol":"1.0","EncryptionAlgorithm":"AES_CBC_128"},"KeyWrappingMetadata":{"EncryptionLibrary":"Java 5.3.0"}} │ x-ms-blob-type: BlockBlob │ content-length: 16 │ date: Sun, 15 Jun 2025 16:07:43 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ host: 127.0.0.1:8081 │ │ │ ⋮ │ 16-byte body │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/MIUdD/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8083 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8083 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8080 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8083 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8080 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8080 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing Error in testset "azure" on worker 39: Test Failed at /home/pkgeval/.julia/packages/RustyObjectStore/JF6aG/test/snowflake_stage_exception_tests.jl:686 Expression: nrequests == 1 + max_retries Evaluated: 4 == 3 [ Info: All tests on worker 1 completed. Closing Worker(pid=39). Worker 39: [39] signal 15: Terminated [ Tests Completed: 20/20 test items were run. Test Summary: | Pass Fail Broken Total Time RustyObjectStore | 15233 1 8 15242 14m26.2s test | 15233 1 8 15242 test/aws_api_tests.jl | 3 3 test/aws_s3_exception_tests.jl | 163 2 165 test/azure_api_tests.jl | 5 5 test/azure_blobs_exception_tests.jl | 183 2 185 test/basic_unified_tests.jl | 14560 14560 test/destroy_test.jl | 2 2 test/release_test.jl | 1 1 test/rust_panic_test.jl | 1 1 test/snowflake_api_tests.jl | 3 3 test/snowflake_stage_exception_tests.jl | 312 1 4 317 Basic Stage exceptions | 33 33 20.3s Snowflake Stage retries | 279 1 4 284 1m16.1s 400: Bad Request | 20 20 3.1s 403: Forbidden | 20 20 2.8s 404: Not Found | 10 10 1.5s 405: Method Not Supported | 20 20 2.6s 409: Conflict | 20 20 2.8s 412: Precondition Failed | 20 20 2.8s 413: Content Too Large | 10 10 1.5s 429: Too Many Requests | 36 4 40 5.7s 502: Bad Gateway | 16 16 3.8s 503: Service Unavailable | 16 16 3.8s 504: Gateway Timeout | 16 16 3.9s Timeout | 15 1 16 18.6s aws | 8 8 9.0s azure | 7 1 8 8.9s TCP Closed | 16 16 2.0s TCP reset | 16 16 2.0s Incomplete GET body | 8 8 3.2s Incomplete GET body reset | 8 8 3.7s Incomplete GET body timeout | 8 8 9.0s Cancellation | 4 4 2.7s ERROR: LoadError: Some tests did not pass: 15233 passed, 1 failed, 0 errored, 8 broken. in expression starting at /home/pkgeval/.julia/packages/RustyObjectStore/JF6aG/test/runtests.jl:4 Testing failed after 666.4s ERROR: LoadError: Package RustyObjectStore errored during testing Stacktrace: [1] pkgerror(msg::String) @ Pkg.Types /opt/julia/share/julia/stdlib/v1.13/Pkg/src/Types.jl:68 [2] test(ctx::Pkg.Types.Context, pkgs::Vector{PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool) @ Pkg.Operations /opt/julia/share/julia/stdlib/v1.13/Pkg/src/Operations.jl:2421 [3] test @ /opt/julia/share/julia/stdlib/v1.13/Pkg/src/Operations.jl:2276 [inlined] [4] test(ctx::Pkg.Types.Context, pkgs::Vector{PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{io::IOContext{IO}}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.13/Pkg/src/API.jl:498 [5] test(pkgs::Vector{PackageSpec}; io::IOContext{IO}, kwargs::@Kwargs{julia_args::Cmd}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.13/Pkg/src/API.jl:164 [6] test(pkgs::Vector{String}; kwargs::@Kwargs{julia_args::Cmd}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.13/Pkg/src/API.jl:152 [7] test @ /opt/julia/share/julia/stdlib/v1.13/Pkg/src/API.jl:152 [inlined] [8] #test#81 @ /opt/julia/share/julia/stdlib/v1.13/Pkg/src/API.jl:151 [inlined] [9] top-level scope @ /PkgEval.jl/scripts/evaluate.jl:219 [10] include(mod::Module, _path::String) @ Base ./Base.jl:309 [11] exec_options(opts::Base.JLOptions) @ Base ./client.jl:324 [12] _start() @ Base ./client.jl:557 in expression starting at /PkgEval.jl/scripts/evaluate.jl:210 PkgEval failed after 965.77s: package has test failures