Package evaluation of RustyObjectStore on Julia 1.13.0-DEV.111 (ce76dbf07d*) started at 2025-02-27T23:55:14.830 ################################################################################ # Set-up # Installing PkgEval dependencies (TestEnv)... Set-up completed after 7.58s ################################################################################ # Installation # Installing RustyObjectStore... Resolving package versions... Updating `~/.julia/environments/v1.13/Project.toml` [1b5eed3d] + RustyObjectStore v0.11.1 Updating `~/.julia/environments/v1.13/Manifest.toml` [d1d4a3ce] + BitFlags v0.1.9 [85eb1798] + CloudBase v1.4.7 [944b1d66] + CodecZlib v0.7.8 [f0e56b4a] + ConcurrentUtilities v2.5.0 [ffbed154] + DocStringExtensions v0.9.3 [460bff9d] + ExceptionUnwrapping v0.1.11 [937e9308] + Figgy v1.1.1 [069b7b12] + FunctionWrappers v1.1.3 [cd3eb016] + HTTP v1.10.15 [692b3bcd] + JLLWrappers v1.7.0 [0f8b85d8] + JSON3 v1.14.1 [e6f89c97] + LoggingExtras v1.1.0 [6ac74813] + MD5 v0.2.3 [739be429] + MbedTLS v1.1.9 [4d8831e6] + OpenSSL v1.4.3 [69de0a69] + Parsers v2.8.1 [aea7be01] + PrecompileTools v1.2.1 [21216c6a] + Preferences v1.4.3 [1b5eed3d] + RustyObjectStore v0.11.1 [777ac1f9] + SimpleBufferStream v1.2.0 [856f2bd8] + StructTypes v1.11.0 [3bb67fe8] + TranscodingStreams v0.11.3 [5c2747f8] + URIs v1.5.1 [c8ffd9c3] + MbedTLS_jll v2.28.6+1 [a4b94fbf] + NodeJS_16_jll v16.15.0+0 [d59cea07] + azurite_jll v3.29.0+0 [ad83bbfc] + minio_jll v1.0.0+1 ⌅ [0e112785] + object_store_ffi_jll v0.11.1+0 [0dad84c5] + ArgTools v1.1.2 [56f22d72] + Artifacts v1.11.0 [2a0f44e3] + Base64 v1.11.0 [ade2ca70] + Dates v1.11.0 [f43a241f] + Downloads v1.7.0 [7b1f6079] + FileWatching v1.11.0 [b77e0a4c] + InteractiveUtils v1.11.0 [ac6e5ff7] + JuliaSyntaxHighlighting v1.12.0 [b27032c2] + LibCURL v0.6.4 [76f85450] + LibGit2 v1.11.0 [8f399da3] + Libdl v1.11.0 [56ddb016] + Logging v1.11.0 [d6f4376e] + Markdown v1.11.0 [a63ad114] + Mmap v1.11.0 [ca575930] + NetworkOptions v1.3.0 [44cfe95a] + Pkg v1.12.0 [de0858da] + Printf v1.11.0 [9a3f8284] + Random v1.11.0 [ea8e919c] + SHA v0.7.0 [9e88b42a] + Serialization v1.11.0 [6462fe0b] + Sockets v1.11.0 [f489334b] + StyledStrings v1.11.0 [fa267f1f] + TOML v1.0.3 [a4e569a6] + Tar v1.10.0 [8dfed614] + Test v1.11.0 [cf7118a7] + UUIDs v1.11.0 [4ec0a83e] + Unicode v1.11.0 [deac9b47] + LibCURL_jll v8.11.1+1 [e37daf67] + LibGit2_jll v1.9.0+0 [29816b5a] + LibSSH2_jll v1.11.3+1 [14a3606d] + MozillaCACerts_jll v2024.12.31 [458c3c95] + OpenSSL_jll v3.0.16+0 [83775a58] + Zlib_jll v1.3.1+2 [8e850ede] + nghttp2_jll v1.64.0+1 [3f19e933] + p7zip_jll v17.5.0+2 Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m` Installation completed after 1.52s ################################################################################ # Precompilation # Precompiling PkgEval dependencies... Precompiling package dependencies... Precompilation completed after 30.94s ################################################################################ # Testing # Testing RustyObjectStore Status `/tmp/jl_nQOAR0/Project.toml` [85eb1798] CloudBase v1.4.7 [944b1d66] CodecZlib v0.7.8 [ffbed154] DocStringExtensions v0.9.3 [cd3eb016] HTTP v1.10.15 [0f8b85d8] JSON3 v1.14.1 [817f1d60] ReTestItems v1.29.0 [1b5eed3d] RustyObjectStore v0.11.1 ⌅ [0e112785] object_store_ffi_jll v0.11.1+0 [2a0f44e3] Base64 v1.11.0 [6462fe0b] Sockets v1.11.0 [8dfed614] Test v1.11.0 Status `/tmp/jl_nQOAR0/Manifest.toml` [d1d4a3ce] BitFlags v0.1.9 [85eb1798] CloudBase v1.4.7 [944b1d66] CodecZlib v0.7.8 [f0e56b4a] ConcurrentUtilities v2.5.0 [ffbed154] DocStringExtensions v0.9.3 [460bff9d] ExceptionUnwrapping v0.1.11 [937e9308] Figgy v1.1.1 [069b7b12] FunctionWrappers v1.1.3 [cd3eb016] HTTP v1.10.15 [692b3bcd] JLLWrappers v1.7.0 [0f8b85d8] JSON3 v1.14.1 [e6f89c97] LoggingExtras v1.1.0 [6ac74813] MD5 v0.2.3 [739be429] MbedTLS v1.1.9 [4d8831e6] OpenSSL v1.4.3 [69de0a69] Parsers v2.8.1 [aea7be01] PrecompileTools v1.2.1 [21216c6a] Preferences v1.4.3 [817f1d60] ReTestItems v1.29.0 [1b5eed3d] RustyObjectStore v0.11.1 [777ac1f9] SimpleBufferStream v1.2.0 [856f2bd8] StructTypes v1.11.0 [1e6cf692] TestEnv v1.102.0 [3bb67fe8] TranscodingStreams v0.11.3 [5c2747f8] URIs v1.5.1 [c8ffd9c3] MbedTLS_jll v2.28.6+1 [a4b94fbf] NodeJS_16_jll v16.15.0+0 [d59cea07] azurite_jll v3.29.0+0 [ad83bbfc] minio_jll v1.0.0+1 ⌅ [0e112785] object_store_ffi_jll v0.11.1+0 [0dad84c5] ArgTools v1.1.2 [56f22d72] Artifacts v1.11.0 [2a0f44e3] Base64 v1.11.0 [ade2ca70] Dates v1.11.0 [f43a241f] Downloads v1.7.0 [7b1f6079] FileWatching v1.11.0 [b77e0a4c] InteractiveUtils v1.11.0 [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0 [b27032c2] LibCURL v0.6.4 [76f85450] LibGit2 v1.11.0 [8f399da3] Libdl v1.11.0 [56ddb016] Logging v1.11.0 [d6f4376e] Markdown v1.11.0 [a63ad114] Mmap v1.11.0 [ca575930] NetworkOptions v1.3.0 [44cfe95a] Pkg v1.12.0 [de0858da] Printf v1.11.0 [9a3f8284] Random v1.11.0 [ea8e919c] SHA v0.7.0 [9e88b42a] Serialization v1.11.0 [6462fe0b] Sockets v1.11.0 [f489334b] StyledStrings v1.11.0 [fa267f1f] TOML v1.0.3 [a4e569a6] Tar v1.10.0 [8dfed614] Test v1.11.0 [cf7118a7] UUIDs v1.11.0 [4ec0a83e] Unicode v1.11.0 [deac9b47] LibCURL_jll v8.11.1+1 [e37daf67] LibGit2_jll v1.9.0+0 [29816b5a] LibSSH2_jll v1.11.3+1 [14a3606d] MozillaCACerts_jll v2024.12.31 [458c3c95] OpenSSL_jll v3.0.16+0 [83775a58] Zlib_jll v1.3.1+2 [8e850ede] nghttp2_jll v1.64.0+1 [3f19e933] p7zip_jll v17.5.0+2 Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. Testing Running tests... Precompiling packages... 5969.7 ms ✓ CloudBase 12328.3 ms ✓ RustyObjectStore 2 dependencies successfully precompiled in 19 seconds. 55 already precompiled. ┌ Warning: Verbosity logging macros are deprecated as they are not compatible with juliac-compiled programs │ caller = ip:0x0 └ @ Core :-1 [ Info: Scanning for test items in project `RustyObjectStore` at paths: /home/pkgeval/.julia/packages/RustyObjectStore/7Ie5z [ Info: Finished scanning for test items in 18.23 seconds. Scheduling 20 tests on pid 25 with 1 worker processes and 2 threads per worker. [ Info: Starting test workers Worker 42: [ Info: Starting test worker 1 on pid = 42, with 2 threads [ Info: Starting running test items Worker 42: 23:57:43 | maxrss 2.2% | mem 3.5% | START ( 1/20) test item "AWSConfig" at test/aws_api_tests.jl:1 Worker 42: ┌ Warning: Verbosity logging macros are deprecated as they are not compatible with juliac-compiled programs Worker 42: │ caller = ip:0x0 Worker 42: └ @ Core :-1 Worker 42: 23:57:51 | maxrss 2.2% | mem 3.8% | DONE ( 1/20) test item "AWSConfig" 6.1 secs (70.8% compile, 12.9% recompile, 2.7% GC), 3.07 M allocs (180.990 MB) Worker 42: 23:58:22 | maxrss 2.2% | mem 3.8% | START ( 2/20) test item "Basic S3 exceptions" at test/aws_s3_exception_tests.jl:1 Worker 42: 23:59:13 | maxrss 2.4% | mem 4.2% | DONE ( 2/20) test item "Basic S3 exceptions" 48.8 secs (88.1% compile, 0.3% recompile, 0.9% GC), 16.41 M allocs (950.698 MB) ┌ Warning: Test item "Basic S3 exceptions" at test/aws_s3_exception_tests.jl:1 contains test sets without tests: │ "Delete non-existing file" └ @ ReTestItems ~/.julia/packages/ReTestItems/ecp6j/src/log_capture.jl:324 Worker 42: 00:00:34 | maxrss 2.4% | mem 4.2% | START ( 3/20) test item "AWS S3 retries" at test/aws_s3_exception_tests.jl:159 Worker 42: 00:01:18 | maxrss 2.5% | mem 4.3% | DONE ( 3/20) test item "AWS S3 retries" 42.2 secs (27.4% compile, 2.0% recompile, 0.4% GC), 5.80 M allocs (349.106 MB) Captured Logs for test item "AWS S3 retries" at test/aws_s3_exception_tests.jl:159 on worker 42 2025-02-28T00:00:40.421597Z  WARN object_store_ffi: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 2.060065s 2025-02-28T00:00:41.085321Z  WARN object_store_ffi: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 1.07634ms 2025-02-28T00:00:41.707448Z  WARN object_store_ffi: Generic S3 error: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 1.113429ms 2025-02-28T00:00:42.331217Z  WARN object_store_ffi: Generic S3 error: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 964.73µs 2025-02-28T00:00:42.980163Z  WARN object_store_ffi: Object at location blob not found: Client error with status 404 Not Found: response body from the dummy server Caused by: 0: Object at location blob not found: Client error with status 404 Not Found: response body from the dummy server 1: Client error with status 404 Not Found: response body from the dummy server Recent attempts (1 out of 1): reason: Code(404) after 0 retries Total retries: 0 Total Time: 963.741µs 2025-02-28T00:00:44.323961Z  WARN object_store_ffi: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 220.60497ms 2025-02-28T00:00:44.356388Z  WARN object_store_ffi: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 1.061419ms 2025-02-28T00:00:45.000209Z  WARN object_store_ffi: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 969.311µs 2025-02-28T00:00:45.618241Z  WARN object_store_ffi: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 989.59µs 2025-02-28T00:00:46.228311Z  WARN object_store_ffi: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 1.06035ms 2025-02-28T00:00:46.852238Z  WARN object_store_ffi: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 994.451µs 2025-02-28T00:00:47.456323Z  WARN object_store_ffi: Generic S3 error: Client error with status 413 Payload Too Large: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 413 Payload Too Large: response body from the dummy server 1: Client error with status 413 Payload Too Large: response body from the dummy server Recent attempts (1 out of 1): reason: Code(413) after 0 retries Total retries: 0 Total Time: 1.05088ms 2025-02-28T00:00:48.511247Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 1.00643ms 2025-02-28T00:00:49.131245Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 1.00223ms 2025-02-28T00:00:50.489593Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 234.3541ms 2025-02-28T00:00:50.615216Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 995.521µs 2025-02-28T00:00:51.507703Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 297.936104ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 297.936104ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/mybucket/blob) 1: Error after 2 retries in 297.936104ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/mybucket/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/mybucket/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 298.118032ms 2025-02-28T00:00:52.474360Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 301.897876ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 301.897876ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/mybucket/blob) 1: Error after 2 retries in 301.897876ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/mybucket/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/mybucket/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 302.120654ms 2025-02-28T00:00:53.410467Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 302.000005ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 302.000005ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/mybucket/blob) 1: Error after 2 retries in 302.000005ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/mybucket/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/mybucket/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 302.204413ms 2025-02-28T00:00:54.350514Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 286.030337ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 286.030337ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/mybucket/blob) 1: Error after 2 retries in 286.030337ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/mybucket/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/mybucket/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 286.250255ms 2025-02-28T00:00:55.264585Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 296.022652ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 296.022652ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) 1: Error after 2 retries in 296.022652ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 296.23227ms 2025-02-28T00:00:56.117205Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 231.777174ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 231.777174ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) 1: Error after 2 retries in 231.777174ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 231.996971ms 2025-02-28T00:01:00.468134Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 3.287029668s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 3.287029668s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/mybucket/blob) 1: Error after 2 retries in 3.287029668s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/mybucket/blob) 2: error sending request for url (http://127.0.0.1:8081/mybucket/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.287223347s 2025-02-28T00:01:04.521110Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 3.237554889s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 3.237554889s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/mybucket/blob) 1: Error after 2 retries in 3.237554889s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/mybucket/blob) 2: error sending request for url (http://127.0.0.1:8081/mybucket/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.237779087s 2025-02-28T00:01:06.279357Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 378.19652ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 378.19652ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 1: Error after 2 retries in 378.19652ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 2: error sending request for url (http://127.0.0.1:8082/mybucket/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 378.413317ms 2025-02-28T00:01:06.564348Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 257.511409ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 257.511409ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 1: Error after 2 retries in 257.511409ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 2: error sending request for url (http://127.0.0.1:8082/mybucket/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 257.741316ms 2025-02-28T00:01:07.194381Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 340.160482ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 340.160482ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 1: Error after 2 retries in 340.160482ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 2: error sending request for url (http://127.0.0.1:8082/mybucket/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 340.41535ms 2025-02-28T00:01:07.559373Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 338.732175ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 338.732175ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 1: Error after 2 retries in 338.732175ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 2: error sending request for url (http://127.0.0.1:8082/mybucket/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 338.981153ms 2025-02-28T00:01:08.072849Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-28T00:01:08.683913Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-28T00:01:09.711264Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-28T00:01:09.711382Z  WARN object_store_ffi: io error while streaming body: Generic S3 error: error decoding response body Caused by: 0: Generic S3 error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: end of file before message length reached Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 1.802828037s 2025-02-28T00:01:10.162278Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-28T00:01:10.643187Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-28T00:01:11.469472Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-28T00:01:11.469594Z  WARN object_store_ffi: io error while streaming body: Generic S3 error: error decoding response body Caused by: 0: Generic S3 error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: Connection reset by peer (os error 104) Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 1.422470878s 2025-02-28T00:01:12.945778Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-28T00:01:14.683240Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-28T00:01:16.351063Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2025-02-28T00:01:16.351185Z  WARN object_store_ffi: io error while streaming body: Generic S3 error: error decoding response body Caused by: 0: Generic S3 error: error decoding response body 1: error decoding response body 2: request or response body error 3: operation timed out Recent attempts (3 out of 3): reason: Timeout after 0 retries reason: Timeout after 0 retries reason: Timeout after 0 retries Total retries: 2 Total Time: 4.406533759s 2025-02-28T00:01:17.724695Z  WARN object_store_ffi: operation was cancelled [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 ┌ Warning: Verbosity logging macros are deprecated as they are not compatible with juliac-compiled programs │ caller = ip:0x0 └ @ Core :-1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1101 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1155 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#279".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#1 │ @ ./essentials.jl:1057 [inlined] │ [11] invokelatest │ @ ./essentials.jl:1053 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#279".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /mybucket/blob HTTP/1.1 │ host: 127.0.0.1:8081 │ x-amz-date: 20250228T000057Z │ x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1101 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1155 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#279".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#1 │ @ ./essentials.jl:1057 [inlined] │ [11] invokelatest │ @ ./essentials.jl:1053 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#279".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /mybucket/blob HTTP/1.1 │ host: 127.0.0.1:8081 │ x-amz-date: 20250228T000057Z │ x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Listening on: 127.0.0.1:8081, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1101 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1155 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#279".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#1 │ @ ./essentials.jl:1057 [inlined] │ [11] invokelatest │ @ ./essentials.jl:1053 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#279".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /mybucket/blob HTTP/1.1 │ content-length: 5 │ host: 127.0.0.1:8081 │ x-amz-date: 20250228T000101Z │ x-amz-content-sha256: 205830ca5b23bbe39ab510cfddc1dff2d9842e38b5fa7b7c48cd4ca7e44f92a1 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1101 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1155 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#279".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#1 │ @ ./essentials.jl:1057 [inlined] │ [11] invokelatest │ @ ./essentials.jl:1053 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#279".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /mybucket/blob HTTP/1.1 │ content-length: 5 │ host: 127.0.0.1:8081 │ x-amz-date: 20250228T000101Z │ x-amz-content-sha256: 205830ca5b23bbe39ab510cfddc1dff2d9842e38b5fa7b7c48cd4ca7e44f92a1 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1101 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1155 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#279".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#1 │ @ ./essentials.jl:1057 [inlined] │ [11] invokelatest │ @ ./essentials.jl:1053 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#279".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /mybucket/blob HTTP/1.1 │ content-length: 5 │ host: 127.0.0.1:8081 │ x-amz-date: 20250228T000101Z │ x-amz-content-sha256: 205830ca5b23bbe39ab510cfddc1dff2d9842e38b5fa7b7c48cd4ca7e44f92a1 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8083 closing [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing Error in testset "Timeout" on worker 42: Test Failed at /home/pkgeval/.julia/packages/RustyObjectStore/7Ie5z/test/aws_s3_exception_tests.jl:667 Expression: nrequests == 1 + max_retries Evaluated: 2 == 3 Worker 42: 00:01:36 | maxrss 2.5% | mem 4.3% | START ( 4/20) test item "AzureConfig" at test/azure_api_tests.jl:1 Worker 42: 00:01:38 | maxrss 2.5% | mem 4.3% | DONE ( 4/20) test item "AzureConfig" 0.6 secs (63.1% compile), 72.45 K allocs (4.266 MB) Worker 42: 00:02:13 | maxrss 2.5% | mem 4.3% | START ( 5/20) test item "Basic BlobStorage exceptions" at test/azure_blobs_exception_tests.jl:1 Worker 42: 00:02:51 | maxrss 2.8% | mem 4.4% | DONE ( 5/20) test item "Basic BlobStorage exceptions" 35.9 secs (75.1% compile, <0.1% recompile, 2.3% GC), 7.27 M allocs (511.420 MB) Worker 42: 00:04:23 | maxrss 2.8% | mem 4.4% | START ( 6/20) test item "BlobStorage retries" at test/azure_blobs_exception_tests.jl:213 Worker 42: 00:05:05 | maxrss 2.8% | mem 4.5% | DONE ( 6/20) test item "BlobStorage retries" 40.1 secs (14.4% compile, <0.1% recompile, 0.3% GC), 4.72 M allocs (331.966 MB) Captured Logs for test item "BlobStorage retries" at test/azure_blobs_exception_tests.jl:213 on worker 42 2025-02-28T00:04:25.803831Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 169.344687ms 2025-02-28T00:04:26.456416Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 1.160179ms 2025-02-28T00:04:27.084279Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 999.421µs 2025-02-28T00:04:27.715197Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 960.341µs 2025-02-28T00:04:28.339238Z  WARN object_store_ffi: Object at location blob not found: Client error with status 404 Not Found: response body from the dummy server Caused by: 0: Object at location blob not found: Client error with status 404 Not Found: response body from the dummy server 1: Client error with status 404 Not Found: response body from the dummy server Recent attempts (1 out of 1): reason: Code(404) after 0 retries Total retries: 0 Total Time: 1.02678ms 2025-02-28T00:04:29.615112Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 166.95108ms 2025-02-28T00:04:30.263293Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 1.04769ms 2025-02-28T00:04:30.900340Z  WARN object_store_ffi: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 1.11563ms 2025-02-28T00:04:31.559225Z  WARN object_store_ffi: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 975.311µs 2025-02-28T00:04:32.193123Z  WARN object_store_ffi: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 914.231µs 2025-02-28T00:04:32.836177Z  WARN object_store_ffi: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 971.81µs 2025-02-28T00:04:33.451318Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 413 Payload Too Large: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 413 Payload Too Large: response body from the dummy server 1: Client error with status 413 Payload Too Large: response body from the dummy server Recent attempts (1 out of 1): reason: Code(413) after 0 retries Total retries: 0 Total Time: 1.04996ms 2025-02-28T00:04:34.115276Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 1.05312ms 2025-02-28T00:04:34.146358Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 1.238258ms 2025-02-28T00:04:34.771113Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 168.879011ms 2025-02-28T00:04:35.424291Z  WARN object_store_ffi: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic MicrosoftAzure error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 1.05316ms 2025-02-28T00:04:36.293606Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 230.199637ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 230.199637ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 230.199637ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 230.388905ms 2025-02-28T00:04:37.185441Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 222.107524ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 222.107524ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 222.107524ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 222.304592ms 2025-02-28T00:04:38.079464Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 222.039914ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 222.039914ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 222.039914ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 222.225783ms 2025-02-28T00:04:38.999067Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 235.248309ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 235.248309ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 1: Error after 2 retries in 235.248309ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 235.425117ms 2025-02-28T00:04:40.369297Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 359.932181ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 359.932181ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 359.932181ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 360.123959ms 2025-02-28T00:04:41.874198Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 336.752492ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 336.752492ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 336.752492ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 336.94723ms 2025-02-28T00:04:46.318136Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 3.267467882s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 3.267467882s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 3.267467882s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.267640481s 2025-02-28T00:04:50.835137Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 3.29072834s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 3.29072834s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 1: Error after 2 retries in 3.29072834s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 2: error sending request for url (http://127.0.0.1:8081/myaccount/mycontainer/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.290918499s 2025-02-28T00:04:52.478379Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 287.249203ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 287.249203ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 1: Error after 2 retries in 287.249203ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 2: error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 287.435962ms 2025-02-28T00:04:52.745373Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 250.462934ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 250.462934ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 1: Error after 2 retries in 250.462934ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 2: error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 250.652362ms 2025-02-28T00:04:53.310345Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 346.925125ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 346.925125ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 1: Error after 2 retries in 346.925125ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 2: error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 347.117603ms 2025-02-28T00:04:53.606365Z  WARN object_store_ffi: Generic MicrosoftAzure error: Error after 2 retries in 279.733565ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) Caused by: 0: Generic MicrosoftAzure error: Error after 2 retries in 279.733565ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 1: Error after 2 retries in 279.733565ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 2: error sending request for url (http://127.0.0.1:8082/myaccount/mycontainer/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 279.953953ms 2025-02-28T00:04:54.044793Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-28T00:04:54.555748Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-28T00:04:55.563203Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-28T00:04:55.563317Z  WARN object_store_ffi: io error while streaming body: Generic MicrosoftAzure error: error decoding response body Caused by: 0: Generic MicrosoftAzure error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: end of file before message length reached Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 1.627843531s 2025-02-28T00:04:56.061848Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-28T00:04:56.529821Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-28T00:04:57.571175Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-28T00:04:57.571297Z  WARN object_store_ffi: io error while streaming body: Generic MicrosoftAzure error: error decoding response body Caused by: 0: Generic MicrosoftAzure error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: Connection reset by peer (os error 104) Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 1.62912507s 2025-02-28T00:04:58.931546Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-28T00:05:00.416516Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-28T00:05:02.465135Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic MicrosoftAzure error: error decoding response body 2025-02-28T00:05:02.465254Z  WARN object_store_ffi: io error while streaming body: Generic MicrosoftAzure error: error decoding response body Caused by: 0: Generic MicrosoftAzure error: error decoding response body 1: error decoding response body 2: request or response body error 3: operation timed out Recent attempts (3 out of 3): reason: Timeout after 0 retries reason: Timeout after 0 retries reason: Timeout after 0 retries Total retries: 2 Total Time: 4.534779877s 2025-02-28T00:05:03.783573Z  WARN object_store_ffi: operation was cancelled [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1101 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1155 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##BlobStorage retries#285".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#1 │ @ ./essentials.jl:1057 [inlined] │ [11] invokelatest │ @ ./essentials.jl:1053 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##BlobStorage retries#285".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /myaccount/mycontainer/blob HTTP/1.1 │ content-length: 0 │ date: Fri, 28 Feb 2025 00:04:43 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1101 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1155 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##BlobStorage retries#285".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#1 │ @ ./essentials.jl:1057 [inlined] │ [11] invokelatest │ @ ./essentials.jl:1053 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##BlobStorage retries#285".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /myaccount/mycontainer/blob HTTP/1.1 │ content-length: 0 │ date: Fri, 28 Feb 2025 00:04:43 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1101 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1155 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##BlobStorage retries#285".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#1 │ @ ./essentials.jl:1057 [inlined] │ [11] invokelatest │ @ ./essentials.jl:1053 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##BlobStorage retries#285".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ GET /myaccount/mycontainer/blob HTTP/1.1 │ content-length: 0 │ date: Fri, 28 Feb 2025 00:04:43 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Listening on: 127.0.0.1:8081, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1101 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1155 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##BlobStorage retries#285".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#1 │ @ ./essentials.jl:1057 [inlined] │ [11] invokelatest │ @ ./essentials.jl:1053 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##BlobStorage retries#285".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /myaccount/mycontainer/blob HTTP/1.1 │ x-ms-blob-type: BlockBlob │ content-length: 5 │ date: Fri, 28 Feb 2025 00:04:47 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1101 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1155 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##BlobStorage retries#285".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#1 │ @ ./essentials.jl:1057 [inlined] │ [11] invokelatest │ @ ./essentials.jl:1053 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##BlobStorage retries#285".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /myaccount/mycontainer/blob HTTP/1.1 │ x-ms-blob-type: BlockBlob │ content-length: 5 │ date: Fri, 28 Feb 2025 00:04:47 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1101 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1155 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/4AUPl/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/4AUPl/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:803 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##BlobStorage retries#285".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/4AUPl/src/Handlers.jl:61 │ [10] #invokelatest#1 │ @ ./essentials.jl:1057 [inlined] │ [11] invokelatest │ @ ./essentials.jl:1053 [inlined] │ [12] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:469 │ [13] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##BlobStorage retries#285".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:401 │ request = │ HTTP.Messages.Request: │ """ │ PUT /myaccount/mycontainer/blob HTTP/1.1 │ x-ms-blob-type: BlockBlob │ content-length: 5 │ date: Fri, 28 Feb 2025 00:04:47 GMT │ x-ms-version: 2023-11-03 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.10.2 │ host: 127.0.0.1:8081 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/4AUPl/src/Servers.jl:483 [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing Error in testset "TCP Closed" on worker 42: Test Failed at /home/pkgeval/.julia/packages/RustyObjectStore/7Ie5z/test/azure_blobs_exception_tests.jl:793 Expression: nrequests == 1 + max_retries Evaluated: 2 == 3 Worker 42: 00:07:17 | maxrss 2.8% | mem 4.5% | START ( 7/20) test item "Basic BlobStorage usage" at test/basic_unified_tests.jl:659 Worker 42: 00:08:50 | maxrss 5.1% | mem 7.1% | DONE ( 7/20) test item "Basic BlobStorage usage" 91.3 secs (45.0% compile, <0.1% recompile, 3.1% GC), 10.77 M allocs (2.162 GB) Worker 42: 00:11:11 | maxrss 5.1% | mem 7.1% | START ( 8/20) test item "Basic BlobStorage usage (anonymous read enabled)" at test/basic_unified_tests.jl:693 Worker 42: 00:11:21 | maxrss 5.1% | mem 5.0% | DONE ( 8/20) test item "Basic BlobStorage usage (anonymous read enabled)" 7.9 secs (16.6% compile, <0.1% recompile, 8.4% GC), 1.20 M allocs (354.265 MB) Worker 42: 00:13:27 | maxrss 5.1% | mem 5.0% | START ( 9/20) test item "Basic AWS S3 usage" at test/basic_unified_tests.jl:718 Worker 42: 00:14:48 | maxrss 5.4% | mem 6.9% | DONE ( 9/20) test item "Basic AWS S3 usage" 78.2 secs (56.3% compile, <0.1% recompile, 4.1% GC), 7.59 M allocs (1.978 GB) Worker 42: 00:17:10 | maxrss 5.4% | mem 6.9% | START (10/20) test item "Basic AWS S3 usage (anonymous read enabled)" at test/basic_unified_tests.jl:752 Worker 42: 00:17:17 | maxrss 5.4% | mem 7.0% | DONE (10/20) test item "Basic AWS S3 usage (anonymous read enabled)" 5.0 secs (26.1% compile, <0.1% recompile, 17.9% GC), 1.20 M allocs (354.033 MB) Worker 42: 00:19:22 | maxrss 5.4% | mem 7.0% | START (11/20) test item "Basic Snowflake Stage usage: AWS, non-encrypted" at test/basic_unified_tests.jl:779 Worker 42: 00:21:00 | maxrss 5.4% | mem 7.4% | DONE (11/20) test item "Basic Snowflake Stage usage: AWS, non-encrypted" 95.4 secs (72.9% compile, <0.1% recompile, 3.2% GC), 17.05 M allocs (2.256 GB) Worker 42: 00:23:22 | maxrss 5.4% | mem 7.4% | START (12/20) test item "Basic Snowflake Stage usage: AWS, encrypted" at test/basic_unified_tests.jl:796 Worker 42: 00:24:04 | maxrss 5.4% | mem 7.6% | DONE (12/20) test item "Basic Snowflake Stage usage: AWS, encrypted" 39.5 secs (24.9% compile, <0.1% recompile, 5.1% GC), 6.19 M allocs (1.416 GB) Worker 42: 2025-02-28T00:25:14.153388Z  WARN object_store_ffi::snowflake::client: Heartbeat failed: error sending request for url (http://127.0.0.1:8080/session/heartbeat) Worker 42: Caused by: Worker 42: 0: error sending request for url (http://127.0.0.1:8080/session/heartbeat) Worker 42: 1: client error (Connect) Worker 42: 2: tcp connect error: Connection refused (os error 111) Worker 42: 3: Connection refused (os error 111) Worker 42: 00:26:24 | maxrss 5.4% | mem 7.6% | START (13/20) test item "Basic Snowflake Stage usage: Azure, non-encrypted" at test/basic_unified_tests.jl:813 Worker 42: 00:27:12 | maxrss 5.5% | mem 8.0% | DONE (13/20) test item "Basic Snowflake Stage usage: Azure, non-encrypted" 45.5 secs (7.5% compile, 4.5% GC), 3.66 M allocs (1.261 GB) Worker 42: 2025-02-28T00:28:28.480560Z  WARN object_store_ffi::snowflake::client: Heartbeat failed: error sending request for url (http://127.0.0.1:8080/session/heartbeat) Worker 42: Caused by: Worker 42: 0: error sending request for url (http://127.0.0.1:8080/session/heartbeat) Worker 42: 1: client error (Connect) Worker 42: 2: tcp connect error: Connection refused (os error 111) Worker 42: 3: Connection refused (os error 111) Worker 42: 00:29:33 | maxrss 5.5% | mem 8.0% | START (14/20) test item "Basic Snowflake Stage usage: Azure, encrypted" at test/basic_unified_tests.jl:830 Worker 42: 00:30:32 | maxrss 6.1% | mem 8.7% | DONE (14/20) test item "Basic Snowflake Stage usage: Azure, encrypted" 56.9 secs (19.1% compile, 3.8% GC), 5.76 M allocs (1.392 GB) Worker 42: 00:31:03 | maxrss 6.1% | mem 8.7% | START (15/20) test item "destroy_* functions do not panic" at test/destroy_test.jl:1 Worker 42: 00:31:05 | maxrss 6.1% | mem 8.7% | DONE (15/20) test item "destroy_* functions do not panic" 0.3 secs (20.4% compile), 2.31 K allocs (122.968 KB) Worker 42: 00:31:10 | maxrss 6.1% | mem 8.7% | START (16/20) test item "Using object_store_ffi_jll" at test/release_test.jl:4 Worker 42: 00:31:11 | maxrss 6.1% | mem 8.7% | DONE (16/20) test item "Using object_store_ffi_jll" 0.2 secs (<0.1% compile), 1.51 K allocs (93.160 KB) Worker 42: 00:31:16 | maxrss 6.1% | mem 8.7% | START (17/20) test item "Handle object_store_ffi panic" at test/rust_panic_test.jl:1 Worker 42: 00:31:26 | maxrss 6.1% | mem 9.0% | DONE (17/20) test item "Handle object_store_ffi panic" 8.9 secs (5.8% compile, 0.4% recompile), 152.57 K allocs (9.078 MB) Worker 42: 2025-02-28T00:31:32.350271Z  WARN object_store_ffi::snowflake::client: Heartbeat failed: error sending request for url (http://127.0.0.1:8080/session/heartbeat) Worker 42: Caused by: Worker 42: 0: error sending request for url (http://127.0.0.1:8080/session/heartbeat) Worker 42: 1: client error (Connect) Worker 42: 2: tcp connect error: Connection refused (os error 111) Worker 42: 3: Connection refused (os error 111) Worker 42: 00:31:32 | maxrss 6.1% | mem 9.0% | START (18/20) test item "SnowflakeConfig" at test/snowflake_api_tests.jl:1 Worker 42: 00:31:34 | maxrss 6.1% | mem 9.0% | DONE (18/20) test item "SnowflakeConfig" 0.6 secs (59.5% compile), 113.95 K allocs (7.280 MB) Worker 42: 00:32:01 | maxrss 6.1% | mem 9.0% | START (19/20) test item "Basic Stage exceptions" at test/snowflake_stage_exception_tests.jl:1 Worker 42: 00:32:28 | maxrss 6.1% | mem 8.7% | DONE (19/20) test item "Basic Stage exceptions" 25.4 secs (20.1% compile, 1.7% GC), 1.27 M allocs (79.510 MB) Worker 42: 00:34:04 | maxrss 6.1% | mem 8.7% | START (20/20) test item "Snowflake Stage retries" at test/snowflake_stage_exception_tests.jl:158 Worker 42: 00:35:26 | maxrss 6.1% | mem 8.8% | DONE (20/20) test item "Snowflake Stage retries" 79.4 secs (14.2% compile, <0.1% recompile, 0.2% GC), 5.21 M allocs (331.608 MB) Worker 42: [42] signal 15: Terminated Test Summary: | Pass Fail Broken Total Time RustyObjectStore | 15170 2 8 15180 39m17.6s test | 15170 2 8 15180 test/aws_api_tests.jl | 3 3 test/aws_s3_exception_tests.jl | 162 1 2 165 Basic S3 exceptions | 23 23 50.3s AWS S3 retries | 139 1 2 142 43.4s 400: Bad Request | 10 10 5.7s 403: Forbidden | 10 10 1.2s 404: Not Found | 5 5 0.7s 405: Method Not Supported | 10 10 1.3s 409: Conflict | 10 10 1.2s 412: Precondition Failed | 10 10 1.2s 413: Content Too Large | 5 5 0.6s 429: Too Many Requests | 18 2 20 2.7s 502: Bad Gateway | 8 8 1.9s 503: Service Unavailable | 8 8 1.9s 504: Gateway Timeout | 8 8 1.8s Timeout | 7 1 8 9.0s TCP Closed | 8 8 0.9s TCP reset | 8 8 1.0s Incomplete GET body | 4 4 2.1s Incomplete GET body reset | 4 4 1.7s Incomplete GET body timeout | 4 4 4.9s Cancellation | 2 2 1.4s test/azure_api_tests.jl | 5 5 test/azure_blobs_exception_tests.jl | 176 1 2 179 Basic BlobStorage exceptions | 35 35 37.1s BlobStorage retries | 141 1 2 144 41.3s 400: Bad Request | 10 10 2.0s 403: Forbidden | 10 10 1.3s 404: Not Found | 5 5 0.7s 405: Method Not Supported | 10 10 1.9s 409: Conflict | 10 10 1.3s 412: Precondition Failed | 10 10 1.3s 413: Content Too Large | 5 5 0.6s 429: Too Many Requests | 18 2 20 1.9s 502: Bad Gateway | 8 8 1.8s 503: Service Unavailable | 8 8 2.1s 504: Gateway Timeout | 8 8 2.6s Timeout | 8 8 9.4s TCP Closed | 7 1 8 0.7s TCP reset | 8 8 0.9s Incomplete GET body | 4 4 2.0s Incomplete GET body reset | 4 4 2.0s Incomplete GET body timeout | 4 4 4.9s Cancellation | 2 2 1.3s InvalidBlockList | 2 2 0.9s test/basic_unified_tests.jl | 14504 14504 test/destroy_test.jl | 2 2 test/release_test.jl | 1 1 test/rust_panic_test.jl | 1 1 test/snowflake_api_tests.jl | 3 3 test/snowflake_stage_exception_tests.jl | 313 4 317 ERROR: LoadError: Some tests did not pass: 15170 passed, 2 failed, 0 errored, 8 broken. in expression starting at /home/pkgeval/.julia/packages/RustyObjectStore/7Ie5z/test/runtests.jl:4 Testing failed after 2228.62s ERROR: LoadError: Package RustyObjectStore errored during testing Stacktrace: [1] pkgerror(msg::String) @ Pkg.Types /opt/julia/share/julia/stdlib/v1.13/Pkg/src/Types.jl:68 [2] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool) @ Pkg.Operations /opt/julia/share/julia/stdlib/v1.13/Pkg/src/Operations.jl:2386 [3] test @ /opt/julia/share/julia/stdlib/v1.13/Pkg/src/Operations.jl:2241 [inlined] [4] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{io::IOContext{IO}}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.13/Pkg/src/API.jl:486 [5] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::IOContext{IO}, kwargs::@Kwargs{julia_args::Cmd}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.13/Pkg/src/API.jl:164 [6] test(pkgs::Vector{String}; kwargs::@Kwargs{julia_args::Cmd}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.13/Pkg/src/API.jl:152 [7] test @ /opt/julia/share/julia/stdlib/v1.13/Pkg/src/API.jl:152 [inlined] [8] #test#81 @ /opt/julia/share/julia/stdlib/v1.13/Pkg/src/API.jl:151 [inlined] [9] top-level scope @ /PkgEval.jl/scripts/evaluate.jl:219 [10] include(mod::Module, _path::String) @ Base ./Base.jl:307 [11] exec_options(opts::Base.JLOptions) @ Base ./client.jl:328 [12] _start() @ Base ./client.jl:560 in expression starting at /PkgEval.jl/scripts/evaluate.jl:210 PkgEval failed after 2485.43s: package has test failures