Package evaluation of RustyObjectStore on Julia 1.10.8 (92f03a4775*) started at 2025-02-25T07:42:31.329 ################################################################################ # Set-up # Installing PkgEval dependencies (TestEnv)... Set-up completed after 4.92s ################################################################################ # Installation # Installing RustyObjectStore... Resolving package versions... Updating `~/.julia/environments/v1.10/Project.toml` [1b5eed3d] + RustyObjectStore v0.11.1 Updating `~/.julia/environments/v1.10/Manifest.toml` [d1d4a3ce] + BitFlags v0.1.9 [85eb1798] + CloudBase v1.4.7 [944b1d66] + CodecZlib v0.7.8 [f0e56b4a] + ConcurrentUtilities v2.5.0 [ffbed154] + DocStringExtensions v0.9.3 [460bff9d] + ExceptionUnwrapping v0.1.11 [937e9308] + Figgy v1.1.1 [069b7b12] + FunctionWrappers v1.1.3 [cd3eb016] + HTTP v1.10.15 [692b3bcd] + JLLWrappers v1.7.0 [0f8b85d8] + JSON3 v1.14.1 [e6f89c97] + LoggingExtras v1.1.0 [6ac74813] + MD5 v0.2.3 [739be429] + MbedTLS v1.1.9 [4d8831e6] + OpenSSL v1.4.3 [69de0a69] + Parsers v2.8.1 [aea7be01] + PrecompileTools v1.2.1 [21216c6a] + Preferences v1.4.3 [1b5eed3d] + RustyObjectStore v0.11.1 [777ac1f9] + SimpleBufferStream v1.2.0 [856f2bd8] + StructTypes v1.11.0 [3bb67fe8] + TranscodingStreams v0.11.3 [5c2747f8] + URIs v1.5.1 [a4b94fbf] + NodeJS_16_jll v16.15.0+0 [458c3c95] + OpenSSL_jll v3.0.16+0 [d59cea07] + azurite_jll v3.29.0+0 [ad83bbfc] + minio_jll v1.0.0+1 ⌅ [0e112785] + object_store_ffi_jll v0.11.1+0 [0dad84c5] + ArgTools v1.1.1 [56f22d72] + Artifacts [2a0f44e3] + Base64 [ade2ca70] + Dates [f43a241f] + Downloads v1.6.0 [7b1f6079] + FileWatching [b77e0a4c] + InteractiveUtils [b27032c2] + LibCURL v0.6.4 [76f85450] + LibGit2 [8f399da3] + Libdl [56ddb016] + Logging [d6f4376e] + Markdown [a63ad114] + Mmap [ca575930] + NetworkOptions v1.2.0 [44cfe95a] + Pkg v1.10.0 [de0858da] + Printf [3fa0cd96] + REPL [9a3f8284] + Random [ea8e919c] + SHA v0.7.0 [9e88b42a] + Serialization [6462fe0b] + Sockets [fa267f1f] + TOML v1.0.3 [a4e569a6] + Tar v1.10.0 [8dfed614] + Test [cf7118a7] + UUIDs [4ec0a83e] + Unicode [deac9b47] + LibCURL_jll v8.4.0+0 [e37daf67] + LibGit2_jll v1.6.4+0 [29816b5a] + LibSSH2_jll v1.11.0+1 [c8ffd9c3] + MbedTLS_jll v2.28.2+1 [14a3606d] + MozillaCACerts_jll v2023.1.10 [83775a58] + Zlib_jll v1.2.13+1 [8e850ede] + nghttp2_jll v1.52.0+1 [3f19e933] + p7zip_jll v17.4.0+2 Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m` Installation completed after 5.3s ################################################################################ # Precompilation # Precompiling PkgEval dependencies... Precompiling package dependencies... Precompilation completed after 20.27s ################################################################################ # Testing # Testing RustyObjectStore Status `/tmp/jl_7A2ER2/Project.toml` [85eb1798] CloudBase v1.4.7 [944b1d66] CodecZlib v0.7.8 [ffbed154] DocStringExtensions v0.9.3 [cd3eb016] HTTP v1.10.15 [0f8b85d8] JSON3 v1.14.1 [817f1d60] ReTestItems v1.29.0 [1b5eed3d] RustyObjectStore v0.11.1 ⌅ [0e112785] object_store_ffi_jll v0.11.1+0 [2a0f44e3] Base64 [6462fe0b] Sockets [8dfed614] Test Status `/tmp/jl_7A2ER2/Manifest.toml` [d1d4a3ce] BitFlags v0.1.9 [85eb1798] CloudBase v1.4.7 [944b1d66] CodecZlib v0.7.8 [f0e56b4a] ConcurrentUtilities v2.5.0 [ffbed154] DocStringExtensions v0.9.3 [460bff9d] ExceptionUnwrapping v0.1.11 [937e9308] Figgy v1.1.1 [069b7b12] FunctionWrappers v1.1.3 [cd3eb016] HTTP v1.10.15 [692b3bcd] JLLWrappers v1.7.0 [0f8b85d8] JSON3 v1.14.1 [e6f89c97] LoggingExtras v1.1.0 [6ac74813] MD5 v0.2.3 [739be429] MbedTLS v1.1.9 [4d8831e6] OpenSSL v1.4.3 [69de0a69] Parsers v2.8.1 [aea7be01] PrecompileTools v1.2.1 [21216c6a] Preferences v1.4.3 [817f1d60] ReTestItems v1.29.0 [1b5eed3d] RustyObjectStore v0.11.1 [777ac1f9] SimpleBufferStream v1.2.0 [856f2bd8] StructTypes v1.11.0 [1e6cf692] TestEnv v1.102.0 [3bb67fe8] TranscodingStreams v0.11.3 [5c2747f8] URIs v1.5.1 [a4b94fbf] NodeJS_16_jll v16.15.0+0 [458c3c95] OpenSSL_jll v3.0.16+0 [d59cea07] azurite_jll v3.29.0+0 [ad83bbfc] minio_jll v1.0.0+1 ⌅ [0e112785] object_store_ffi_jll v0.11.1+0 [0dad84c5] ArgTools v1.1.1 [56f22d72] Artifacts [2a0f44e3] Base64 [ade2ca70] Dates [f43a241f] Downloads v1.6.0 [7b1f6079] FileWatching [b77e0a4c] InteractiveUtils [b27032c2] LibCURL v0.6.4 [76f85450] LibGit2 [8f399da3] Libdl [56ddb016] Logging [d6f4376e] Markdown [a63ad114] Mmap [ca575930] NetworkOptions v1.2.0 [44cfe95a] Pkg v1.10.0 [de0858da] Printf [3fa0cd96] REPL [9a3f8284] Random [ea8e919c] SHA v0.7.0 [9e88b42a] Serialization [6462fe0b] Sockets [fa267f1f] TOML v1.0.3 [a4e569a6] Tar v1.10.0 [8dfed614] Test [cf7118a7] UUIDs [4ec0a83e] Unicode [deac9b47] LibCURL_jll v8.4.0+0 [e37daf67] LibGit2_jll v1.6.4+0 [29816b5a] LibSSH2_jll v1.11.0+1 [c8ffd9c3] MbedTLS_jll v2.28.2+1 [14a3606d] MozillaCACerts_jll v2023.1.10 [83775a58] Zlib_jll v1.2.13+1 [8e850ede] nghttp2_jll v1.52.0+1 [3f19e933] p7zip_jll v17.4.0+2 Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. Testing Running tests... ┌ Warning: Verbosity logging macros are deprecated as they are not compatible with juliac-compiled programs │ caller = ip:0x0 └ @ Core :-1 [ Info: Scanning for test items in project `RustyObjectStore` at paths: /home/pkgeval/.julia/packages/RustyObjectStore/7Ie5z [ Info: Finished scanning for test items in 8.58 seconds. Scheduling 20 tests on pid 27 with 1 worker processes and 2 threads per worker. [ Info: Starting test workers Worker 36: [ Info: Starting test worker 1 on pid = 36, with 2 threads [ Info: Starting running test items Worker 36: 07:43:58 | maxrss 1.7% | mem 2.7% | START ( 1/20) test item "AWSConfig" at test/aws_api_tests.jl:1 Worker 36: ┌ Warning: Verbosity logging macros are deprecated as they are not compatible with juliac-compiled programs Worker 36: │ caller = ip:0x0 Worker 36: └ @ Core :-1 Worker 36: 07:44:04 | maxrss 1.7% | mem 2.9% | DONE ( 1/20) test item "AWSConfig" 3.9 secs (75.0% compile, 3.3% GC), 813.67 K allocs (58.825 MB) Worker 36: 07:44:06 | maxrss 1.7% | mem 2.9% | START ( 2/20) test item "Basic S3 exceptions" at test/aws_s3_exception_tests.jl:1 Worker 36: 07:45:00 | maxrss 1.7% | mem 3.1% | DONE ( 2/20) test item "Basic S3 exceptions" 53.8 secs (122.5% compile, 1.8% GC), 8.01 M allocs (577.278 MB) ┌ Warning: Test item "Basic S3 exceptions" at test/aws_s3_exception_tests.jl:1 contains test sets without tests: │ "Delete non-existing file" └ @ ReTestItems ~/.julia/packages/ReTestItems/ecp6j/src/log_capture.jl:324 Worker 36: 07:45:01 | maxrss 1.7% | mem 3.1% | START ( 3/20) test item "AWS S3 retries" at test/aws_s3_exception_tests.jl:159 Worker 36: 07:45:55 | maxrss 1.8% | mem 3.2% | DONE ( 3/20) test item "AWS S3 retries" 54.2 secs (14.8% compile, <0.1% recompile, 0.3% GC), 3.59 M allocs (283.693 MB) Worker 36: 07:45:55 | maxrss 1.8% | mem 3.2% | START ( 4/20) test item "AzureConfig" at test/azure_api_tests.jl:1 Worker 36: 07:45:56 | maxrss 1.8% | mem 3.2% | DONE ( 4/20) test item "AzureConfig" 0.4 secs (84.0% compile), 42.17 K allocs (3.128 MB) Worker 36: 07:45:57 | maxrss 1.8% | mem 3.2% | START ( 5/20) test item "Basic BlobStorage exceptions" at test/azure_blobs_exception_tests.jl:1 Worker 36: 07:46:26 | maxrss 2.1% | mem 3.4% | DONE ( 5/20) test item "Basic BlobStorage exceptions" 28.7 secs (61.7% compile, 1.8% GC), 4.98 M allocs (459.761 MB) Worker 36: 07:46:29 | maxrss 2.1% | mem 3.4% | START ( 6/20) test item "BlobStorage retries" at test/azure_blobs_exception_tests.jl:213 Worker 36: 07:47:24 | maxrss 2.1% | mem 3.4% | DONE ( 6/20) test item "BlobStorage retries" 55.0 secs (9.3% compile, 0.6% GC), 3.34 M allocs (281.964 MB) Worker 36: [ Info: Server on 127.0.0.1:8081 closing Captured Logs for test item "BlobStorage retries" at test/azure_blobs_exception_tests.jl:213 on worker 36 2025-02-25T07:46:30.428103Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 541.828922ms 2025-02-25T07:46:31.611178Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 637.24167ms 2025-02-25T07:46:32.614160Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 423.513921ms 2025-02-25T07:46:33.750135Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 589.443271ms 2025-02-25T07:46:34.681115Z  WARN object_store_ffi: Object at location blob not found: Client error with status 404 Not Found: response body from the dummy server Caused by: 0: Object at location blob not found: Client error with status 404 Not Found: response body from the dummy server 1: Client error with status 404 Not Found: response body from the dummy server Recent attempts (1 out of 1): reason: Code(404) after 0 retries Total retries: 0 Total Time: 351.781129ms 2025-02-25T07:46:36.226156Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 681.079522ms 2025-02-25T07:46:37.118175Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 346.16002ms 2025-02-25T07:46:38.192150Z  WARN object_store_ffi: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 494.844865ms 2025-02-25T07:46:39.371160Z  WARN object_store_ffi: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 633.133594ms 2025-02-25T07:46:40.341149Z  WARN object_store_ffi: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 389.497717ms 2025-02-25T07:46:41.199134Z  WARN object_store_ffi: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 312.122347ms 2025-02-25T07:46:42.043105Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 413 Payload Too Large: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 413 Payload Too Large: response body from the dummy server 1: Client error with status 413 Payload Too Large: response body from the dummy server Recent attempts (1 out of 1): reason: Code(413) after 0 retries Total retries: 0 Total Time: 250.92687ms 2025-02-25T07:46:42.839148Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 239.825311ms 2025-02-25T07:46:44.009131Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 624.142454ms 2025-02-25T07:46:45.546163Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 628.099268ms 2025-02-25T07:46:46.712155Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 653.941224ms 2025-02-25T07:46:48.275402Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 1.031682326s, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 1.031682326s, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 1.031682326s, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 1.031854784s 2025-02-25T07:46:50.040403Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 1.077257543s, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 1.077257543s, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 1.077257543s, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 1.077432961s 2025-02-25T07:46:52.515536Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 1.326137399s, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 1.326137399s, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 1.326137399s, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 1.326406767s 2025-02-25T07:46:53.969458Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 830.314575ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 830.314575ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 830.314575ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 830.490664ms 2025-02-25T07:46:55.495403Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 923.824948ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 923.824948ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 923.824948ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 923.994357ms 2025-02-25T07:46:56.936279Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 833.992001ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 833.992001ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 833.992001ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 834.16661ms 2025-02-25T07:47:01.038062Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 3.230386736s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 3.230386736s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 3.230386736s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.230525785s [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:691 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /myaccount/mycontainer/blob HTTP/1.1 │ content-length: 0 │ date: Tue, 25 Feb 2025 07:46:47 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:691 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /myaccount/mycontainer/blob HTTP/1.1 │ x-ms-blob-type: BlockBlob │ content-length: 5 │ date: Tue, 25 Feb 2025 07:46:48 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:691 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /myaccount/mycontainer/blob HTTP/1.1 │ content-length: 0 │ date: Tue, 25 Feb 2025 07:46:51 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:691 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /myaccount/mycontainer/blob HTTP/1.1 │ x-ms-blob-type: BlockBlob │ content-length: 5 │ date: Tue, 25 Feb 2025 07:46:53 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:691 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /myaccount/mycontainer/blob HTTP/1.1 │ content-length: 0 │ date: Tue, 25 Feb 2025 07:46:54 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Listening on: 127.0.0.1:8081, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [5] write │ @ ./strings/io.jl:248 [inlined] │ [6] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [7] (::HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [8] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [9] invokelatest │ @ ./essentials.jl:889 [inlined] │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /myaccount/mycontainer/blob HTTP/1.1 │ x-ms-blob-type: BlockBlob │ content-length: 5 │ date: Tue, 25 Feb 2025 07:46:56 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [5] write │ @ ./strings/io.jl:248 [inlined] │ [6] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [7] (::HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [8] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [9] invokelatest │ @ ./essentials.jl:889 [inlined] │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /myaccount/mycontainer/blob HTTP/1.1 │ x-ms-blob-type: BlockBlob │ content-length: 5 │ date: Tue, 25 Feb 2025 07:46:56 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:691 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#1#2"{Int64, Nothing, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /myaccount/mycontainer/blob HTTP/1.1 │ x-ms-blob-type: BlockBlob │ content-length: 5 │ date: Tue, 25 Feb 2025 07:46:56 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Listening on: 127.0.0.1:8081, thread id: 2 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#17#18"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#17#18"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /myaccount/mycontainer/blob HTTP/1.1 │ content-length: 0 │ date: Tue, 25 Feb 2025 07:46:57 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#17#18"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#17#18"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /myaccount/mycontainer/blob HTTP/1.1 │ content-length: 0 │ date: Tue, 25 Feb 2025 07:46:57 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#17#18"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#17#18"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /myaccount/mycontainer/blob HTTP/1.1 │ content-length: 0 │ date: Tue, 25 Feb 2025 07:46:57 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Listening on: 127.0.0.1:8081, thread id: 1 2025-02-25T07:47:06.572186Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 3.29976766s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 3.29976766s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 3.29976766s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.299929179s 2025-02-25T07:47:09.001455Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 265.05789ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 265.05789ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 1: Error after 2 retries in 265.05789ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 2: error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 265.192568ms 2025-02-25T07:47:09.297496Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 295.332126ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 295.332126ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 1: Error after 2 retries in 295.332126ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 2: error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 295.471404ms 2025-02-25T07:47:09.685453Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 246.091001ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 246.091001ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 1: Error after 2 retries in 246.091001ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 2: error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 246.24884ms 2025-02-25T07:47:09.952463Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 266.179779ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 266.179779ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 1: Error after 2 retries in 266.179779ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 2: error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 266.335877ms 2025-02-25T07:47:10.666164Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:47:11.710172Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:47:12.606137Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:47:12.606238Z  WARN object_store_ffi: io error while streaming body: Generic MicrosoftAzure error: error decoding response body Caused by: 0: Generic MicrosoftAzure error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: end of file before message length reached Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 2.39996541s 2025-02-25T07:47:14.130105Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:47:15.362118Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:47:16.331099Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:47:16.331224Z  WARN object_store_ffi: io error while streaming body: Generic MicrosoftAzure error: error decoding response body Caused by: 0: Generic MicrosoftAzure error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: Connection reset by peer (os error 104) Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 2.938138202s 2025-02-25T07:47:17.589371Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:47:19.111699Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:47:20.597821Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:47:20.598099Z  WARN object_store_ffi: io error while streaming body: Generic MicrosoftAzure error: error decoding response body Caused by: 0: Generic MicrosoftAzure error: error decoding response body 1: error decoding response body 2: request or response body error 3: operation timed out Recent attempts (3 out of 3): reason: Timeout after 0 retries reason: Timeout after 0 retries reason: Timeout after 0 retries Total retries: 2 Total Time: 4.009652778s 2025-02-25T07:47:21.821483Z  WARN object_store_ffi: operation was cancelled ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#17#18"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#17#18"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /myaccount/mycontainer/blob HTTP/1.1 │ x-ms-blob-type: BlockBlob │ content-length: 5 │ date: Tue, 25 Feb 2025 07:47:03 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#17#18"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#17#18"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /myaccount/mycontainer/blob HTTP/1.1 │ x-ms-blob-type: BlockBlob │ content-length: 5 │ date: Tue, 25 Feb 2025 07:47:03 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#17#18"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##BlobStorage retries#236".var"#17#18"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /myaccount/mycontainer/blob HTTP/1.1 │ x-ms-blob-type: BlockBlob │ content-length: 5 │ date: Tue, 25 Feb 2025 07:47:03 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Listening on: 127.0.0.1:8083, thread id: 2 [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 2 [ Info: Listening on: 127.0.0.1:8083, thread id: 2 [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Server on 127.0.0.1:8083 closing [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 Error in testset "Cancellation" on worker 36: Test Failed at /home/pkgeval/.julia/packages/RustyObjectStore/7Ie5z/test/azure_blobs_exception_tests.jl:820 Expression: nrequests == 1 Evaluated: 0 == 1 Worker 36: 07:47:25 | maxrss 2.1% | mem 3.4% | START ( 7/20) test item "Basic BlobStorage usage" at test/basic_unified_tests.jl:659 Worker 36: 07:48:39 | maxrss 4.7% | mem 4.9% | DONE ( 7/20) test item "Basic BlobStorage usage" 73.0 secs (40.3% compile, 1.8% GC), 6.75 M allocs (1.847 GB) Worker 36: 07:48:39 | maxrss 4.7% | mem 4.9% | START ( 8/20) test item "Basic BlobStorage usage (anonymous read enabled)" at test/basic_unified_tests.jl:693 Worker 36: 07:48:47 | maxrss 4.7% | mem 5.7% | DONE ( 8/20) test item "Basic BlobStorage usage (anonymous read enabled)" 7.8 secs (11.5% compile), 716.41 K allocs (330.003 MB) Worker 36: 07:48:48 | maxrss 4.7% | mem 5.7% | START ( 9/20) test item "Basic AWS S3 usage" at test/basic_unified_tests.jl:718 Worker 36: 07:49:35 | maxrss 5.2% | mem 6.8% | DONE ( 9/20) test item "Basic AWS S3 usage" 47.0 secs (48.8% compile, 1.7% GC), 4.66 M allocs (1.701 GB) Worker 36: 07:49:35 | maxrss 5.2% | mem 6.8% | START (10/20) test item "Basic AWS S3 usage (anonymous read enabled)" at test/basic_unified_tests.jl:752 Worker 36: 07:49:39 | maxrss 5.2% | mem 5.6% | DONE (10/20) test item "Basic AWS S3 usage (anonymous read enabled)" 3.7 secs (31.0% compile, 6.8% GC), 700.48 K allocs (328.938 MB) Worker 36: 07:49:39 | maxrss 5.2% | mem 5.6% | START (11/20) test item "Basic Snowflake Stage usage: AWS, non-encrypted" at test/basic_unified_tests.jl:779 Worker 36: 07:51:24 | maxrss 5.2% | mem 6.5% | DONE (11/20) test item "Basic Snowflake Stage usage: AWS, non-encrypted" 104.0 secs (42.2% compile, 0.9% GC), 9.22 M allocs (1.795 GB) Worker 36: 07:51:24 | maxrss 5.2% | mem 6.5% | START (12/20) test item "Basic Snowflake Stage usage: AWS, encrypted" at test/basic_unified_tests.jl:796 Worker 36: 07:52:18 | maxrss 5.7% | mem 7.7% | DONE (12/20) test item "Basic Snowflake Stage usage: AWS, encrypted" 53.5 secs (12.9% compile, 0.9% GC), 3.99 M allocs (1.216 GB) Worker 36: 07:52:19 | maxrss 5.7% | mem 7.7% | START (13/20) test item "Basic Snowflake Stage usage: Azure, non-encrypted" at test/basic_unified_tests.jl:813 Worker 36: 07:53:58 | maxrss 5.7% | mem 7.5% | DONE (13/20) test item "Basic Snowflake Stage usage: Azure, non-encrypted" 99.0 secs (2.1% compile, 0.5% GC), 2.14 M allocs (1.075 GB) Worker 36: 07:53:58 | maxrss 5.7% | mem 7.5% | START (14/20) test item "Basic Snowflake Stage usage: Azure, encrypted" at test/basic_unified_tests.jl:830 Worker 36: 07:54:50 | maxrss 5.9% | mem 8.0% | DONE (14/20) test item "Basic Snowflake Stage usage: Azure, encrypted" 50.9 secs (13.0% compile, 1.4% GC), 3.44 M allocs (1.174 GB) Worker 36: 07:54:56 | maxrss 5.9% | mem 8.0% | START (15/20) test item "destroy_* functions do not panic" at test/destroy_test.jl:1 Worker 36: 07:54:57 | maxrss 5.9% | mem 8.0% | DONE (15/20) test item "destroy_* functions do not panic" 0.2 secs (16.8% compile), 3.55 K allocs (240.584 KB) Worker 36: 07:54:57 | maxrss 5.9% | mem 8.0% | START (16/20) test item "Using object_store_ffi_jll" at test/release_test.jl:4 Worker 36: 07:54:57 | maxrss 5.9% | mem 8.0% | DONE (16/20) test item "Using object_store_ffi_jll" <0.1 secs (<0.1% compile), 1.70 K allocs (128.392 KB) Worker 36: 07:54:57 | maxrss 5.9% | mem 8.0% | START (17/20) test item "Handle object_store_ffi panic" at test/rust_panic_test.jl:1 Worker 36: 07:55:03 | maxrss 5.9% | mem 8.0% | DONE (17/20) test item "Handle object_store_ffi panic" 5.3 secs (2.2% compile), 24.39 K allocs (1.849 MB) Worker 36: 07:55:04 | maxrss 5.9% | mem 8.0% | START (18/20) test item "SnowflakeConfig" at test/snowflake_api_tests.jl:1 Worker 36: 07:55:04 | maxrss 5.9% | mem 8.0% | DONE (18/20) test item "SnowflakeConfig" 0.4 secs (85.0% compile), 53.91 K allocs (4.130 MB) Worker 36: 07:55:05 | maxrss 5.9% | mem 8.0% | START (19/20) test item "Basic Stage exceptions" at test/snowflake_stage_exception_tests.jl:1 Worker 36: 07:55:36 | maxrss 5.9% | mem 8.0% | DONE (19/20) test item "Basic Stage exceptions" 30.6 secs (12.4% compile), 737.36 K allocs (57.600 MB) Worker 36: 07:55:37 | maxrss 5.9% | mem 8.0% | START (20/20) test item "Snowflake Stage retries" at test/snowflake_stage_exception_tests.jl:158 Worker 36: 07:57:41 | maxrss 6.1% | mem 8.2% | DONE (20/20) test item "Snowflake Stage retries" 123.7 secs (6.0% compile), 3.18 M allocs (259.069 MB) Captured Logs for test item "Snowflake Stage retries" at test/snowflake_stage_exception_tests.jl:158 on worker 36 2025-02-25T07:55:39.277715Z  WARN object_store_ffi: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 179.386724ms 2025-02-25T07:55:41.733470Z  WARN object_store_ffi: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 894.672µs 2025-02-25T07:55:43.456990Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 28.10802ms 2025-02-25T07:55:43.926302Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 751.573µs 2025-02-25T07:55:46.260325Z  WARN object_store_ffi: Generic S3 error: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 794.612µs 2025-02-25T07:55:48.912840Z  WARN object_store_ffi: Generic S3 error: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 8.242853ms 2025-02-25T07:55:50.741711Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 6.152813ms 2025-02-25T07:55:51.769148Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 607.707955ms 2025-02-25T07:55:53.202777Z  WARN object_store_ffi: Object at location xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob not found: Client error with status 404 Not Found: response body from the dummy server Caused by: 0: Object at location xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob not found: Client error with status 404 Not Found: response body from the dummy server 1: Client error with status 404 Not Found: response body from the dummy server Recent attempts (1 out of 1): reason: Code(404) after 0 retries Total retries: 0 Total Time: 12.301587ms 2025-02-25T07:55:54.050696Z  WARN object_store_ffi: Object at location blob not found: Client error with status 404 Not Found: response body from the dummy server Caused by: 0: Object at location blob not found: Client error with status 404 Not Found: response body from the dummy server 1: Client error with status 404 Not Found: response body from the dummy server Recent attempts (1 out of 1): reason: Code(404) after 0 retries Total retries: 0 Total Time: 8.137385ms 2025-02-25T07:55:55.674174Z  WARN object_store_ffi: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 458.978329ms 2025-02-25T07:55:56.260497Z  WARN object_store_ffi: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 1.03312ms 2025-02-25T07:55:56.813746Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 15.171319ms 2025-02-25T07:55:58.045128Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 583.621857ms 2025-02-25T07:55:59.194743Z  WARN object_store_ffi: Object at location xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 12.203647ms 2025-02-25T07:55:59.472210Z  WARN object_store_ffi: Object at location xox50000-s/stages/fb95b172-2466-4335-fb95-b17224664335/blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location xox50000-s/stages/fb95b172-2466-4335-fb95-b17224664335/blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 983.34µs 2025-02-25T07:56:01.456894Z  WARN object_store_ffi: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 7.58039ms 2025-02-25T07:56:02.507671Z  WARN object_store_ffi: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 5.143212ms 2025-02-25T07:56:03.104149Z  WARN object_store_ffi: Request precondition failure for path xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 194.201995ms 2025-02-25T07:56:03.688928Z  WARN object_store_ffi: Request precondition failure for path xox50000-s/stages/9c325cef-b51e-5877-9c32-5cefb51e5877/blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path xox50000-s/stages/9c325cef-b51e-5877-9c32-5cefb51e5877/blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 1.318478ms 2025-02-25T07:56:04.595751Z  WARN object_store_ffi: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 8.098735ms 2025-02-25T07:56:04.800207Z  WARN object_store_ffi: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 1.11076ms 2025-02-25T07:56:06.966695Z  WARN object_store_ffi: Generic S3 error: Client error with status 413 Payload Too Large: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 413 Payload Too Large: response body from the dummy server 1: Client error with status 413 Payload Too Large: response body from the dummy server Recent attempts (1 out of 1): reason: Code(413) after 0 retries Total retries: 0 Total Time: 8.002266ms 2025-02-25T07:56:07.752060Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 413 Payload Too Large: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 413 Payload Too Large: response body from the dummy server 1: Client error with status 413 Payload Too Large: response body from the dummy server Recent attempts (1 out of 1): reason: Code(413) after 0 retries Total retries: 0 Total Time: 957.351µs 2025-02-25T07:56:09.803682Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 12.200047ms 2025-02-25T07:56:10.807715Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 15.1488ms 2025-02-25T07:56:12.634919Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 165.420441ms 2025-02-25T07:56:13.365758Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 8.120305ms 2025-02-25T07:56:13.924205Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 1.026161ms 2025-02-25T07:56:14.954699Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 8.155954ms 2025-02-25T07:56:16.095795Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 26.264827ms 2025-02-25T07:56:16.814251Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 1.039141ms 2025-02-25T07:56:18.575581Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 226.738743ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 226.738743ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 1: Error after 2 retries in 226.738743ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 226.972121ms 2025-02-25T07:56:19.641986Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 209.886919ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/9c325cef-b51e-5877-9c32-5cefb51e5877/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 209.886919ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/9c325cef-b51e-5877-9c32-5cefb51e5877/blob) 1: Error after 2 retries in 209.886919ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/9c325cef-b51e-5877-9c32-5cefb51e5877/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/9c325cef-b51e-5877-9c32-5cefb51e5877/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 210.231975ms 2025-02-25T07:56:23.195008Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 315.416473ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 315.416473ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/dummy/dummy/blob) 1: Error after 2 retries in 315.416473ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/dummy/dummy/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/dummy/dummy/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 315.652021ms 2025-02-25T07:56:25.150398Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 239.741703ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 239.741703ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/dummy/dummy/blob) 1: Error after 2 retries in 239.741703ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/dummy/dummy/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/dummy/dummy/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 239.955191ms 2025-02-25T07:56:28.345418Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 308.694435ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 308.694435ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 1: Error after 2 retries in 308.694435ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 308.859754ms 2025-02-25T07:56:30.136521Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 293.925242ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/9c325cef-b51e-5877-9c32-5cefb51e5877/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 293.925242ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/9c325cef-b51e-5877-9c32-5cefb51e5877/blob) 1: Error after 2 retries in 293.925242ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/9c325cef-b51e-5877-9c32-5cefb51e5877/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/9c325cef-b51e-5877-9c32-5cefb51e5877/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 294.151239ms 2025-02-25T07:56:33.491775Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 228.07543ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 228.07543ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) 1: Error after 2 retries in 228.07543ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 228.215479ms 2025-02-25T07:56:36.754429Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 926.851845ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 926.851845ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) 1: Error after 2 retries in 926.851845ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/dummy/dummy/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 927.065903ms 2025-02-25T07:56:39.030068Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 239.327935ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 239.327935ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 1: Error after 2 retries in 239.327935ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 239.510154ms 2025-02-25T07:56:41.188681Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 309.832773ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/9c325cef-b51e-5877-9c32-5cefb51e5877/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 309.832773ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/9c325cef-b51e-5877-9c32-5cefb51e5877/blob) 1: Error after 2 retries in 309.832773ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/9c325cef-b51e-5877-9c32-5cefb51e5877/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/9c325cef-b51e-5877-9c32-5cefb51e5877/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 310.091481ms 2025-02-25T07:56:43.535581Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 304.835939ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8082/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 304.835939ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8082/dummy/dummy/blob) 1: Error after 2 retries in 304.835939ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8082/dummy/dummy/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8082/dummy/dummy/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 304.983738ms 2025-02-25T07:56:46.316638Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 306.077968ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8082/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 306.077968ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8082/dummy/dummy/blob) 1: Error after 2 retries in 306.077968ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8082/dummy/dummy/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8082/dummy/dummy/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 306.291226ms 2025-02-25T07:56:51.862005Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 3.302991781s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 3.302991781s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 1: Error after 2 retries in 3.302991781s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 2: error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.303179939s 2025-02-25T07:56:57.359524Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 3.270259069s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/9c836ae0-8144-8ec4-9c83-6ae081448ec4/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 3.270259069s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/9c836ae0-8144-8ec4-9c83-6ae081448ec4/blob) 1: Error after 2 retries in 3.270259069s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/9c836ae0-8144-8ec4-9c83-6ae081448ec4/blob) 2: error sending request for url (http://127.0.0.1:8081/dummy/xox50000-s/stages/9c836ae0-8144-8ec4-9c83-6ae081448ec4/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.270506327s [ Info: Listening on: 127.0.0.1:8082, thread id: 2 [ Info: Listening on: 127.0.0.1:8083, thread id: 2 [ Info: Server on 127.0.0.1:8082 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 2 [ Info: Listening on: 127.0.0.1:8083, thread id: 2 [ Info: Server on 127.0.0.1:8082 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8082, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8083, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8083, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8082, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8082, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 2 [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8082, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8083, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8083, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8083, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:691 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#25#27"{Int64, Nothing, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#25#27"{Int64, Nothing, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob HTTP/1.1 │ host: 127.0.0.1:8081 │ x-amz-date: 20250225T075618Z │ x-amz-content-sha256: UNSIGNED-PAYLOAD │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8080, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 2 [ Info: Listening on: 127.0.0.1:8080, thread id: 2 [ Info: Server on 127.0.0.1:8082 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 2 [ Info: Listening on: 127.0.0.1:8080, thread id: 2 [ Info: Server on 127.0.0.1:8082 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8082, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:691 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#25#27"{Int64, Nothing, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#25#27"{Int64, Nothing, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /dummy/dummy/blob HTTP/1.1 │ x-ms-meta-encryptiondata: {"EncryptionMode":"FullBlob","WrappedContentKey":{"KeyId":"symmKey1","EncryptedKey":"PxVfA6k3cLXFzCPZ9AOMzhc2zR+z6oKiSGZ2dDBgGWo=","Algorithm":"AES_CBC_256"},"ContentEncryptionIV":"QoeuBdXHM6yxmbeEaVJmkA==","EncryptionAgent":{"Protocol":"1.0","EncryptionAlgorithm":"AES_CBC_128"},"KeyWrappingMetadata":{"EncryptionLibrary":"Java 5.3.0"}} │ x-ms-meta-matdesc: {"smkId":"1","queryId":"1","keySize":"128"} │ x-ms-blob-type: BlockBlob │ content-length: 16 │ date: Tue, 25 Feb 2025 07:56:35 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ │ ⋮ │ 16-byte body │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8082, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:691 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{RustyObjectStore.Test.var"#5#7"{RustyObjectStore.Test.SFGatewayMock}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{RustyObjectStore.Test.var"#5#7"{RustyObjectStore.Test.SFGatewayMock}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ POST /session/heartbeat HTTP/1.1 │ authorization: ****** │ accept: */* │ host: 127.0.0.1:8080 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8082 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob HTTP/1.1 │ host: 127.0.0.1:8081 │ x-amz-date: 20250225T075648Z │ x-amz-content-sha256: UNSIGNED-PAYLOAD │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob HTTP/1.1 │ host: 127.0.0.1:8081 │ x-amz-date: 20250225T075648Z │ x-amz-content-sha256: UNSIGNED-PAYLOAD │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob HTTP/1.1 │ host: 127.0.0.1:8081 │ x-amz-date: 20250225T075648Z │ x-amz-content-sha256: UNSIGNED-PAYLOAD │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /dummy/xox50000-s/stages/9c836ae0-8144-8ec4-9c83-6ae081448ec4/blob HTTP/1.1 │ content-length: 16 │ x-amz-meta-x-amz-matdesc: {"smkId":"1","queryId":"1","keySize":"128"} │ x-amz-meta-x-amz-unencrypted-content-length: 5 │ x-amz-meta-x-amz-cek-alg: AES/CBC/PKCS5Padding │ x-amz-meta-x-amz-key: LRSvCAaoWkQdK76WUuLLE7fMwRgLWUIHynQnAkUTEXw= │ x-amz-meta-x-amz-iv: vhZ5093DcsnQQO3Plp2wyw== │ host: 127.0.0.1:8081 │ x-amz-date: 20250225T075654Z │ x-amz-content-sha256: UNSIGNED-PAYLOAD │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ │ │ ⋮ │ 16-byte body │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /dummy/xox50000-s/stages/9c836ae0-8144-8ec4-9c83-6ae081448ec4/blob HTTP/1.1 │ content-length: 16 │ x-amz-meta-x-amz-matdesc: {"smkId":"1","queryId":"1","keySize":"128"} │ x-amz-meta-x-amz-unencrypted-content-length: 5 │ x-amz-meta-x-amz-cek-alg: AES/CBC/PKCS5Padding │ x-amz-meta-x-amz-key: LRSvCAaoWkQdK76WUuLLE7fMwRgLWUIHynQnAkUTEXw= │ x-amz-meta-x-amz-iv: vhZ5093DcsnQQO3Plp2wyw== │ host: 127.0.0.1:8081 │ x-amz-date: 20250225T075654Z │ x-amz-content-sha256: UNSIGNED-PAYLOAD │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ │ │ ⋮ │ 16-byte body │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing 2025-02-25T07:57:01.756112Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 3.272247258s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 3.272247258s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) 1: Error after 2 retries in 3.272247258s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) 2: error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.272390326s 2025-02-25T07:57:06.922812Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 3.372669163s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 3.372669163s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) 1: Error after 2 retries in 3.372669163s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) 2: error sending request for url (http://127.0.0.1:8081/dummy/dummy/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.372912151s 2025-02-25T07:57:09.592744Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 273.1711ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 273.1711ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 1: Error after 2 retries in 273.1711ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 273.348469ms 2025-02-25T07:57:10.731448Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 283.651084ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/fb95b172-2466-4335-fb95-b17224664335/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 283.651084ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/fb95b172-2466-4335-fb95-b17224664335/blob) 1: Error after 2 retries in 283.651084ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/fb95b172-2466-4335-fb95-b17224664335/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/fb95b172-2466-4335-fb95-b17224664335/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 283.900151ms 2025-02-25T07:57:11.758434Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 236.697118ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 236.697118ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 1: Error after 2 retries in 236.697118ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 236.838577ms 2025-02-25T07:57:13.064678Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 240.852259ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 240.852259ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 1: Error after 2 retries in 240.852259ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 241.069437ms 2025-02-25T07:57:14.605821Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 288.537188ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 288.537188ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 1: Error after 2 retries in 288.537188ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 288.718256ms 2025-02-25T07:57:15.809513Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 225.177385ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/fb95b172-2466-4335-fb95-b17224664335/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 225.177385ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/fb95b172-2466-4335-fb95-b17224664335/blob) 1: Error after 2 retries in 225.177385ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/fb95b172-2466-4335-fb95-b17224664335/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/xox50000-s/stages/fb95b172-2466-4335-fb95-b17224664335/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 225.470381ms 2025-02-25T07:57:16.727506Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 292.772519ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 292.772519ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 1: Error after 2 retries in 292.772519ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 292.926948ms 2025-02-25T07:57:17.742559Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 299.844293ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 299.844293ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 1: Error after 2 retries in 299.844293ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 2: error sending request for url (http://127.0.0.1:8082/dummy/dummy/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 300.07568ms 2025-02-25T07:57:19.228142Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-25T07:57:20.196175Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-25T07:57:21.333136Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-25T07:57:21.333241Z  WARN object_store_ffi: io error while streaming body: Generic S3 error: error decoding response body Caused by: 0: Generic S3 error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: end of file before message length reached Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 2.783832545s 2025-02-25T07:57:22.878621Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:57:23.295765Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:57:23.785629Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:57:23.785714Z  WARN object_store_ffi: io error while streaming body: Generic MicrosoftAzure error: error decoding response body Caused by: 0: Generic MicrosoftAzure error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: end of file before message length reached Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 915.317182ms 2025-02-25T07:57:24.118878Z  WARN object_store_ffi::snowflake::client: Heartbeat failed: error sending request for url (http://127.0.0.1:8081/session/heartbeat) Caused by: 0: error sending request for url (http://127.0.0.1:8081/session/heartbeat) 1: client error (Connect) 2: tcp connect error: Connection refused (os error 111) 3: Connection refused (os error 111) 2025-02-25T07:57:25.065078Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-25T07:57:25.771104Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-25T07:57:26.915104Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-25T07:57:26.915232Z  WARN object_store_ffi: io error while streaming body: Generic S3 error: error decoding response body Caused by: 0: Generic S3 error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: Connection reset by peer (os error 104) Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 2.320493664s 2025-02-25T07:57:26.925474Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:57:27.394218Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:57:28.273460Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:57:28.273571Z  WARN object_store_ffi: io error while streaming body: Generic MicrosoftAzure error: error decoding response body Caused by: 0: Generic MicrosoftAzure error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: Connection reset by peer (os error 104) Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 1.348824474s 2025-02-25T07:57:29.792688Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-25T07:57:31.226969Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-25T07:57:32.729072Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-25T07:57:32.729185Z  WARN object_store_ffi: io error while streaming body: Generic S3 error: error decoding response body Caused by: 0: Generic S3 error: error decoding response body 1: error decoding response body 2: request or response body error 3: operation timed out Recent attempts (3 out of 3): reason: Timeout after 0 retries reason: Timeout after 0 retries reason: Timeout after 0 retries Total retries: 2 Total Time: 3.937801366s 2025-02-25T07:57:34.656642Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:57:36.315032Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:57:38.380939Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-25T07:57:38.381094Z  WARN object_store_ffi: io error while streaming body: Generic MicrosoftAzure error: error decoding response body Caused by: 0: Generic MicrosoftAzure error: error decoding response body 1: error decoding response body 2: request or response body error 3: operation timed out Recent attempts (3 out of 3): reason: Timeout after 0 retries reason: Timeout after 0 retries reason: Timeout after 0 retries Total retries: 2 Total Time: 4.725465423s 2025-02-25T07:57:39.944111Z  WARN object_store_ffi: operation was cancelled 2025-02-25T07:57:41.112272Z  WARN object_store_ffi: operation was cancelled ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /dummy/xox50000-s/stages/9c836ae0-8144-8ec4-9c83-6ae081448ec4/blob HTTP/1.1 │ content-length: 16 │ x-amz-meta-x-amz-matdesc: {"smkId":"1","queryId":"1","keySize":"128"} │ x-amz-meta-x-amz-unencrypted-content-length: 5 │ x-amz-meta-x-amz-cek-alg: AES/CBC/PKCS5Padding │ x-amz-meta-x-amz-key: LRSvCAaoWkQdK76WUuLLE7fMwRgLWUIHynQnAkUTEXw= │ x-amz-meta-x-amz-iv: vhZ5093DcsnQQO3Plp2wyw== │ host: 127.0.0.1:8081 │ x-amz-date: 20250225T075654Z │ x-amz-content-sha256: UNSIGNED-PAYLOAD │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ │ │ ⋮ │ 16-byte body │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /dummy/dummy/blob HTTP/1.1 │ content-length: 0 │ date: Tue, 25 Feb 2025 07:56:58 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /dummy/dummy/blob HTTP/1.1 │ content-length: 0 │ date: Tue, 25 Feb 2025 07:56:58 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /dummy/dummy/blob HTTP/1.1 │ content-length: 0 │ date: Tue, 25 Feb 2025 07:56:58 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /dummy/dummy/blob HTTP/1.1 │ x-ms-meta-matdesc: {"smkId":"1","queryId":"1","keySize":"128"} │ x-ms-meta-encryptiondata: {"EncryptionMode":"FullBlob","WrappedContentKey":{"KeyId":"symmKey1","EncryptedKey":"U8uEMlCsypFZbXrAIEN/1suIF8cgxPDMutoIWK50c9s=","Algorithm":"AES_CBC_256"},"ContentEncryptionIV":"QK5R9g2gLpKOlMWLpMw2zw==","EncryptionAgent":{"Protocol":"1.0","EncryptionAlgorithm":"AES_CBC_128"},"KeyWrappingMetadata":{"EncryptionLibrary":"Java 5.3.0"}} │ x-ms-blob-type: BlockBlob │ content-length: 16 │ date: Tue, 25 Feb 2025 07:57:03 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ │ ⋮ │ 16-byte body │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /dummy/dummy/blob HTTP/1.1 │ x-ms-meta-matdesc: {"smkId":"1","queryId":"1","keySize":"128"} │ x-ms-meta-encryptiondata: {"EncryptionMode":"FullBlob","WrappedContentKey":{"KeyId":"symmKey1","EncryptedKey":"U8uEMlCsypFZbXrAIEN/1suIF8cgxPDMutoIWK50c9s=","Algorithm":"AES_CBC_256"},"ContentEncryptionIV":"QK5R9g2gLpKOlMWLpMw2zw==","EncryptionAgent":{"Protocol":"1.0","EncryptionAlgorithm":"AES_CBC_128"},"KeyWrappingMetadata":{"EncryptionLibrary":"Java 5.3.0"}} │ x-ms-blob-type: BlockBlob │ content-length: 16 │ date: Tue, 25 Feb 2025 07:57:03 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ │ ⋮ │ 16-byte body │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] write │ @ ./io.jl:689 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:248 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [9] (::HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [11] invokelatest │ @ ./essentials.jl:889 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{Main.var"##Snowflake Stage retries#266".var"#29#31"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /dummy/dummy/blob HTTP/1.1 │ x-ms-meta-matdesc: {"smkId":"1","queryId":"1","keySize":"128"} │ x-ms-meta-encryptiondata: {"EncryptionMode":"FullBlob","WrappedContentKey":{"KeyId":"symmKey1","EncryptedKey":"U8uEMlCsypFZbXrAIEN/1suIF8cgxPDMutoIWK50c9s=","Algorithm":"AES_CBC_256"},"ContentEncryptionIV":"QK5R9g2gLpKOlMWLpMw2zw==","EncryptionAgent":{"Protocol":"1.0","EncryptionAlgorithm":"AES_CBC_128"},"KeyWrappingMetadata":{"EncryptionLibrary":"Java 5.3.0"}} │ x-ms-blob-type: BlockBlob │ content-length: 16 │ date: Tue, 25 Feb 2025 07:57:03 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ │ ⋮ │ 16-byte body │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 2 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 2 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 2 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 2 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 2 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 2 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 2 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8080, thread id: 2 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 2 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8083 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 2 [ Info: Listening on: 127.0.0.1:8080, thread id: 2 [ Info: Server on 127.0.0.1:8083 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 2 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8084, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8083 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Server on 127.0.0.1:8084 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 2 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8080 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Listening on: 127.0.0.1:8080, thread id: 2 [ Info: Server on 127.0.0.1:8083 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8080, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Server on 127.0.0.1:8080 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 2 [ Info: Listening on: 127.0.0.1:8080, thread id: 2 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1066 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1120 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:248 [inlined] │ [5] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [6] write │ @ ./strings/io.jl:248 [inlined] │ [7] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:793 │ [8] (::HTTP.Handlers.var"#1#2"{RustyObjectStore.Test.var"#5#7"{RustyObjectStore.Test.SFGatewayMock}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [9] #invokelatest#2 │ @ ./essentials.jl:892 [inlined] │ [10] invokelatest │ @ ./essentials.jl:889 [inlined] │ [11] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [12] (::HTTP.Servers.var"#16#17"{HTTP.Handlers.var"#1#2"{RustyObjectStore.Test.var"#5#7"{RustyObjectStore.Test.SFGatewayMock}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ POST /queries/v1/query-request?clientStartTime=1740470260&requestId=aff06416-0cf3-452a-973b-08e63fd2e940&request_guid=8b61f611-3911-42cd-b05f-dca42170fe6e HTTP/1.1 │ content-type: application/json │ accept: application/json │ authorization: ****** │ host: 127.0.0.1:8080 │ content-length: 141 │ │ {"asyncExec":false,"isInternal":false,"sequenceId":1,"sqlText":"PUT file:///tmp/whatever @TESTDATABASE.TESTSCHEMA.TESTSTAGE666235D81F93205F"}""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8080 closing Error in testset "azure" on worker 36: Test Failed at /home/pkgeval/.julia/packages/RustyObjectStore/7Ie5z/test/snowflake_stage_exception_tests.jl:732 Expression: nrequests == 1 Evaluated: 0 == 1 Worker 36: ┌ Error: handle_connection handler error. Worker 36: │ Worker 36: │ =========================== Worker 36: │ HTTP Error message: Worker 36: │ Worker 36: │ ERROR: EOFError: read end of file Worker 36: │ Stacktrace: Worker 36: │ [1] wait Worker 36: │ @ ./asyncevent.jl:159 [inlined] Worker 36: │ [2] sleep Worker 36: │ @ ./asyncevent.jl:265 [inlined] Worker 36: │ [3] (::Main.var"##Snowflake Stage retries#266".var"#19#22"{Base.RefValue{Int64}})(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) Worker 36: │ @ Main.var"##Snowflake Stage retries#266" ~/.julia/packages/RustyObjectStore/7Ie5z/test/snowflake_stage_exception_tests.jl:378 Worker 36: │ [4] #invokelatest#2 Worker 36: │ @ ./essentials.jl:892 [inlined] Worker 36: │ [5] invokelatest Worker 36: │ @ ./essentials.jl:889 [inlined] Worker 36: │ [6] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) Worker 36: │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 Worker 36: │ [7] (::HTTP.Servers.var"#16#17"{Main.var"##Snowflake Stage retries#266".var"#19#22"{Base.RefValue{Int64}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() Worker 36: │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 Worker 36: │ request = Worker 36: │ HTTP.Messages.Request: Worker 36: │ """ Worker 36: │ GET /dummy/xox50000-s/stages/28e1545c-c3a0-b022-28e1-545cc3a0b022/blob HTTP/1.1 Worker 36: │ host: 127.0.0.1:8083 Worker 36: │ x-amz-date: 20250225T075731Z Worker 36: │ x-amz-content-sha256: UNSIGNED-PAYLOAD Worker 36: │ authorization: ****** Worker 36: │ accept: */* Worker 36: │ user-agent: object_store/0.10.2 Worker 36: │ Worker 36: │ """ Worker 36: └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 Worker 36: [36] signal (15): Terminated in expression starting at none:1 epoll_wait at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) Test Summary: | Pass Fail Broken Total Time RustyObjectStore | 15170 2 8 15180 14m13.6s test | 15170 2 8 15180 test/aws_api_tests.jl | 3 3 test/aws_s3_exception_tests.jl | 163 2 165 test/azure_api_tests.jl | 5 5 test/azure_blobs_exception_tests.jl | 176 1 2 179 Basic BlobStorage exceptions | 35 35 29.0s BlobStorage retries | 141 1 2 144 55.2s 400: Bad Request | 10 10 2.8s 403: Forbidden | 10 10 2.1s 404: Not Found | 5 5 0.9s 405: Method Not Supported | 10 10 2.5s 409: Conflict | 10 10 2.2s 412: Precondition Failed | 10 10 1.8s 413: Content Too Large | 5 5 0.8s 429: Too Many Requests | 18 2 20 4.6s 502: Bad Gateway | 8 8 3.9s 503: Service Unavailable | 8 8 3.4s 504: Gateway Timeout | 8 8 2.9s Timeout | 8 8 11.1s TCP Closed | 8 8 0.7s TCP reset | 8 8 0.6s Incomplete GET body | 4 4 3.2s Incomplete GET body reset | 4 4 3.2s Incomplete GET body timeout | 4 4 4.3s Cancellation | 1 1 2 1.5s InvalidBlockList | 2 2 2.1s test/basic_unified_tests.jl | 14504 14504 test/destroy_test.jl | 2 2 test/release_test.jl | 1 1 test/rust_panic_test.jl | 1 1 test/snowflake_api_tests.jl | 3 3 test/snowflake_stage_exception_tests.jl | 312 1 4 317 Basic Stage exceptions | 33 33 31.0s Snowflake Stage retries | 279 1 4 284 2m04.1s 400: Bad Request | 20 20 7.5s 403: Forbidden | 20 20 7.0s 404: Not Found | 10 10 2.5s 405: Method Not Supported | 20 20 3.8s 409: Conflict | 20 20 4.2s 412: Precondition Failed | 20 20 2.7s 413: Content Too Large | 10 10 2.7s 429: Too Many Requests | 36 4 40 9.2s 502: Bad Gateway | 16 16 9.5s 503: Service Unavailable | 16 16 10.5s 504: Gateway Timeout | 16 16 10.5s Timeout | 16 16 20.8s TCP Closed | 16 16 4.4s TCP reset | 16 16 4.7s Incomplete GET body | 8 8 6.3s Incomplete GET body reset | 8 8 4.3s Incomplete GET body timeout | 8 8 10.2s Cancellation | 3 1 4 2.7s aws | 2 2 1.1s azure | 1 1 2 1.2s ERROR: LoadError: Some tests did not pass: 15170 passed, 2 failed, 0 errored, 8 broken. in expression starting at /home/pkgeval/.julia/packages/RustyObjectStore/7Ie5z/test/runtests.jl:4 Testing failed after 693.03s ERROR: LoadError: Package RustyObjectStore errored during testing Stacktrace: [1] pkgerror(msg::String) @ Pkg.Types /opt/julia/share/julia/stdlib/v1.10/Pkg/src/Types.jl:70 [2] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool) @ Pkg.Operations /opt/julia/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2034 [3] test @ /opt/julia/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1915 [inlined] [4] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{io::Base.PipeEndpoint}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.10/Pkg/src/API.jl:444 [5] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.PipeEndpoint, kwargs::@Kwargs{julia_args::Cmd}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.10/Pkg/src/API.jl:159 [6] test @ /opt/julia/share/julia/stdlib/v1.10/Pkg/src/API.jl:147 [inlined] [7] #test#74 @ /opt/julia/share/julia/stdlib/v1.10/Pkg/src/API.jl:146 [inlined] [8] top-level scope @ /PkgEval.jl/scripts/evaluate.jl:219 in expression starting at /PkgEval.jl/scripts/evaluate.jl:210 PkgEval failed after 929.49s: package has test failures