Package evaluation to test RustyObjectStore on Julia 1.12.4 (422f456051*) started at 2026-01-29T04:49:42.551 ################################################################################ # Set-up # Installing PkgEval dependencies (TestEnv)... Activating project at `~/.julia/environments/v1.12` Set-up completed after 9.26s ################################################################################ # Installation # Installing RustyObjectStore... Resolving package versions... Installed BitFlags ───────────── v0.1.9 Installed SimpleBufferStream ─── v1.2.0 Installed azurite_jll ────────── v3.29.0+0 Installed LoggingExtras ──────── v1.2.0 Installed minio_jll ──────────── v2.0.0+0 Installed Parsers ────────────── v2.8.3 Installed JSON3 ──────────────── v1.14.3 Installed Preferences ────────── v1.5.1 Installed ConcurrentUtilities ── v2.5.0 Installed JLLWrappers ────────── v1.7.1 Installed MD5 ────────────────── v0.2.3 Installed URIs ───────────────── v1.6.1 Installed CloudBase ──────────── v1.4.10 Installed OpenSSL ────────────── v1.6.1 Installed TranscodingStreams ─── v0.11.3 Installed PrecompileTools ────── v1.3.3 Installed FunctionWrappers ───── v1.1.3 Installed MbedTLS_jll ────────── v2.28.1010+0 Installed MbedTLS ────────────── v1.1.9 Installed object_store_ffi_jll ─ v0.12.3+0 Installed NodeJS_16_jll ──────── v16.20.2+0 Installed Figgy ──────────────── v1.1.1 Installed RustyObjectStore ───── v0.12.3 Installed HTTP ───────────────── v1.10.19 Installed CodecZlib ──────────── v0.7.8 Installed StructTypes ────────── v1.11.0 Installed ExceptionUnwrapping ── v0.1.11 Installed DocStringExtensions ── v0.9.5 Installing 5 artifacts Installed artifact MbedTLS 2.2 MiB Installed artifact azurite 15.7 MiB Installed artifact object_store_ffi 23.8 MiB Installed artifact NodeJS_16 32.0 MiB Installed artifact minio 63.6 MiB Updating `~/.julia/environments/v1.12/Project.toml` [1b5eed3d] + RustyObjectStore v0.12.3 Updating `~/.julia/environments/v1.12/Manifest.toml` [d1d4a3ce] + BitFlags v0.1.9 [85eb1798] + CloudBase v1.4.10 [944b1d66] + CodecZlib v0.7.8 [f0e56b4a] + ConcurrentUtilities v2.5.0 [ffbed154] + DocStringExtensions v0.9.5 [460bff9d] + ExceptionUnwrapping v0.1.11 [937e9308] + Figgy v1.1.1 [069b7b12] + FunctionWrappers v1.1.3 [cd3eb016] + HTTP v1.10.19 [692b3bcd] + JLLWrappers v1.7.1 [0f8b85d8] + JSON3 v1.14.3 [e6f89c97] + LoggingExtras v1.2.0 [6ac74813] + MD5 v0.2.3 [739be429] + MbedTLS v1.1.9 [4d8831e6] + OpenSSL v1.6.1 [69de0a69] + Parsers v2.8.3 [aea7be01] + PrecompileTools v1.3.3 [21216c6a] + Preferences v1.5.1 [1b5eed3d] + RustyObjectStore v0.12.3 [777ac1f9] + SimpleBufferStream v1.2.0 [856f2bd8] + StructTypes v1.11.0 [3bb67fe8] + TranscodingStreams v0.11.3 [5c2747f8] + URIs v1.6.1 [c8ffd9c3] + MbedTLS_jll v2.28.1010+0 [a4b94fbf] + NodeJS_16_jll v16.20.2+0 [d59cea07] + azurite_jll v3.29.0+0 [ad83bbfc] + minio_jll v2.0.0+0 [0e112785] + object_store_ffi_jll v0.12.3+0 [56f22d72] + Artifacts v1.11.0 [2a0f44e3] + Base64 v1.11.0 [ade2ca70] + Dates v1.11.0 [b77e0a4c] + InteractiveUtils v1.11.0 [ac6e5ff7] + JuliaSyntaxHighlighting v1.12.0 [8f399da3] + Libdl v1.11.0 [56ddb016] + Logging v1.11.0 [d6f4376e] + Markdown v1.11.0 [a63ad114] + Mmap v1.11.0 [ca575930] + NetworkOptions v1.3.0 [de0858da] + Printf v1.11.0 [9a3f8284] + Random v1.11.0 [ea8e919c] + SHA v0.7.0 [9e88b42a] + Serialization v1.11.0 [6462fe0b] + Sockets v1.11.0 [f489334b] + StyledStrings v1.11.0 [fa267f1f] + TOML v1.0.3 [8dfed614] + Test v1.11.0 [cf7118a7] + UUIDs v1.11.0 [4ec0a83e] + Unicode v1.11.0 [14a3606d] + MozillaCACerts_jll v2025.11.4 [458c3c95] + OpenSSL_jll v3.5.4+0 [83775a58] + Zlib_jll v1.3.1+2 Installation completed after 17.03s ################################################################################ # Precompilation # Precompiling PkgEval dependencies... Precompiling packages... 4583.5 ms ✓ TestEnv 1 dependency successfully precompiled in 5 seconds. 26 already precompiled. Precompiling package dependencies... Precompiling packages... 4445.1 ms ✓ FunctionWrappers 1077.6 ms ✓ DocStringExtensions 1291.3 ms ✓ URIs 788.6 ms ✓ SimpleBufferStream 872.9 ms ✓ BitFlags 1251.1 ms ✓ MD5 2280.5 ms ✓ ConcurrentUtilities 1154.7 ms ✓ TranscodingStreams 4161.7 ms ✓ StructTypes 1132.0 ms ✓ LoggingExtras 1194.2 ms ✓ Figgy 1172.5 ms ✓ Preferences 1655.5 ms ✓ ExceptionUnwrapping 10262.1 ms ✓ ReTestItems 4095.0 ms ✓ OpenSSL 943.8 ms ✓ CodecZlib 912.5 ms ✓ PrecompileTools 1173.6 ms ✓ JLLWrappers 21721.7 ms ✓ Parsers 1409.6 ms ✓ object_store_ffi_jll 1411.6 ms ✓ NodeJS_16_jll 1423.9 ms ✓ MbedTLS_jll 1426.0 ms ✓ minio_jll 18703.4 ms ✓ JSON3 1383.6 ms ✓ azurite_jll 2252.7 ms ✓ MbedTLS 33749.9 ms ✓ HTTP 5431.1 ms ✓ CloudBase 6047.9 ms ✓ RustyObjectStore 29 dependencies successfully precompiled in 135 seconds. 31 already precompiled. Precompilation completed after 153.88s ################################################################################ # Testing # Testing RustyObjectStore Status `/tmp/jl_FnE4vf/Project.toml` [85eb1798] CloudBase v1.4.10 [944b1d66] CodecZlib v0.7.8 [ffbed154] DocStringExtensions v0.9.5 [cd3eb016] HTTP v1.10.19 [0f8b85d8] JSON3 v1.14.3 [817f1d60] ReTestItems v1.35.1 [1b5eed3d] RustyObjectStore v0.12.3 [0e112785] object_store_ffi_jll v0.12.3+0 [2a0f44e3] Base64 v1.11.0 [6462fe0b] Sockets v1.11.0 [8dfed614] Test v1.11.0 Status `/tmp/jl_FnE4vf/Manifest.toml` [d1d4a3ce] BitFlags v0.1.9 [85eb1798] CloudBase v1.4.10 [944b1d66] CodecZlib v0.7.8 [f0e56b4a] ConcurrentUtilities v2.5.0 [ffbed154] DocStringExtensions v0.9.5 [460bff9d] ExceptionUnwrapping v0.1.11 [937e9308] Figgy v1.1.1 [069b7b12] FunctionWrappers v1.1.3 [cd3eb016] HTTP v1.10.19 [692b3bcd] JLLWrappers v1.7.1 [0f8b85d8] JSON3 v1.14.3 [e6f89c97] LoggingExtras v1.2.0 [6ac74813] MD5 v0.2.3 [739be429] MbedTLS v1.1.9 [4d8831e6] OpenSSL v1.6.1 [69de0a69] Parsers v2.8.3 [aea7be01] PrecompileTools v1.3.3 [21216c6a] Preferences v1.5.1 [817f1d60] ReTestItems v1.35.1 [1b5eed3d] RustyObjectStore v0.12.3 [777ac1f9] SimpleBufferStream v1.2.0 [856f2bd8] StructTypes v1.11.0 [1e6cf692] TestEnv v1.103.2 [3bb67fe8] TranscodingStreams v0.11.3 [5c2747f8] URIs v1.6.1 [c8ffd9c3] MbedTLS_jll v2.28.1010+0 [a4b94fbf] NodeJS_16_jll v16.20.2+0 [d59cea07] azurite_jll v3.29.0+0 [ad83bbfc] minio_jll v2.0.0+0 [0e112785] object_store_ffi_jll v0.12.3+0 [0dad84c5] ArgTools v1.1.2 [56f22d72] Artifacts v1.11.0 [2a0f44e3] Base64 v1.11.0 [ade2ca70] Dates v1.11.0 [f43a241f] Downloads v1.7.0 [7b1f6079] FileWatching v1.11.0 [b77e0a4c] InteractiveUtils v1.11.0 [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0 [b27032c2] LibCURL v0.6.4 [76f85450] LibGit2 v1.11.0 [8f399da3] Libdl v1.11.0 [56ddb016] Logging v1.11.0 [d6f4376e] Markdown v1.11.0 [a63ad114] Mmap v1.11.0 [ca575930] NetworkOptions v1.3.0 [44cfe95a] Pkg v1.12.1 [de0858da] Printf v1.11.0 [9a3f8284] Random v1.11.0 [ea8e919c] SHA v0.7.0 [9e88b42a] Serialization v1.11.0 [6462fe0b] Sockets v1.11.0 [f489334b] StyledStrings v1.11.0 [fa267f1f] TOML v1.0.3 [a4e569a6] Tar v1.10.0 [8dfed614] Test v1.11.0 [cf7118a7] UUIDs v1.11.0 [4ec0a83e] Unicode v1.11.0 [e66e0078] CompilerSupportLibraries_jll v1.3.0+1 [deac9b47] LibCURL_jll v8.15.0+0 [e37daf67] LibGit2_jll v1.9.0+0 [29816b5a] LibSSH2_jll v1.11.3+1 [14a3606d] MozillaCACerts_jll v2025.11.4 [458c3c95] OpenSSL_jll v3.5.4+0 [83775a58] Zlib_jll v1.3.1+2 [8e850ede] nghttp2_jll v1.64.0+1 [3f19e933] p7zip_jll v17.7.0+0 Testing Running tests... [ Info: Scanning for test items in project `RustyObjectStore` at paths: /home/pkgeval/.julia/packages/RustyObjectStore/JF6aG [ Info: Finished scanning for test items in 5.26 seconds. [ Info: Scheduling 20 tests on pid 174 with 1 worker processes and 2 threads per worker. [ Info: Starting test workers Worker 178: [ Info: Starting test worker 1 on pid = 178, with 2 threads [ Info: Starting running test items Worker 178: 04:54:13 | maxrss 2.0% | mem 4.5% | START ( 1/20) test item "AWSConfig" at test/aws_api_tests.jl:1 Worker 178: 04:54:19 | maxrss 2.0% | mem 4.7% | DONE ( 1/20) test item "AWSConfig" 4.9 secs (70.8% compile, 20.6% recompile, 9.0% GC), 2.44 M allocs (139.329 MB) Worker 178: 04:54:23 | maxrss 2.0% | mem 4.8% | START ( 2/20) test item "Basic S3 exceptions" at test/aws_s3_exception_tests.jl:1 Worker 178: 04:55:13 | maxrss 2.5% | mem 5.4% | DONE ( 2/20) test item "Basic S3 exceptions" 49.3 secs (95.4% compile, 2.1% recompile, 1.6% GC), 18.50 M allocs (1.022 GB) ┌ Warning: Test item "Basic S3 exceptions" at test/aws_s3_exception_tests.jl:1 contains test sets without tests: │ "Delete non-existing file" └ @ ReTestItems ~/.julia/packages/ReTestItems/rFUty/src/log_capture.jl:328 Worker 178: 04:55:16 | maxrss 2.5% | mem 5.4% | START ( 3/20) test item "AWS S3 retries" at test/aws_s3_exception_tests.jl:159 Worker 178: 04:56:00 | maxrss 2.5% | mem 5.4% | DONE ( 3/20) test item "AWS S3 retries" 43.3 secs (24.8% compile, 0.3% recompile, 1.1% GC), 6.63 M allocs (388.774 MB) Captured Logs for test item "AWS S3 retries" at test/aws_s3_exception_tests.jl:159 on worker 178 2026-01-29T04:55:21.008505Z  WARN object_store_ffi: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 1.746322616s 2026-01-29T04:55:21.699453Z  WARN object_store_ffi: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 400 Bad Request: response body from the dummy server 1: Client error with status 400 Bad Request: response body from the dummy server Recent attempts (1 out of 1): reason: Code(400) after 0 retries Total retries: 0 Total Time: 1.076869ms 2026-01-29T04:55:22.262444Z  WARN object_store_ffi: The operation lacked the necessary privileges to complete for path blob: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: The operation lacked the necessary privileges to complete for path blob: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 1.067468ms 2026-01-29T04:55:22.894574Z  WARN object_store_ffi: The operation lacked the necessary privileges to complete for path blob: Client error with status 403 Forbidden: response body from the dummy server Caused by: 0: The operation lacked the necessary privileges to complete for path blob: Client error with status 403 Forbidden: response body from the dummy server 1: Client error with status 403 Forbidden: response body from the dummy server Recent attempts (1 out of 1): reason: Code(403) after 0 retries Total retries: 0 Total Time: 1.142697ms 2026-01-29T04:55:23.454643Z  WARN object_store_ffi: Object at location blob not found: Client error with status 404 Not Found: response body from the dummy server Caused by: 0: Object at location blob not found: Client error with status 404 Not Found: response body from the dummy server 1: Client error with status 404 Not Found: response body from the dummy server Recent attempts (1 out of 1): reason: Code(404) after 0 retries Total retries: 0 Total Time: 1.224257ms 2026-01-29T04:55:24.795277Z  WARN object_store_ffi: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 221.961175ms 2026-01-29T04:55:25.436482Z  WARN object_store_ffi: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 405 Method Not Allowed: response body from the dummy server 1: Client error with status 405 Method Not Allowed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(405) after 0 retries Total retries: 0 Total Time: 1.067809ms 2026-01-29T04:55:25.995360Z  WARN object_store_ffi: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 978.229µs 2026-01-29T04:55:26.632548Z  WARN object_store_ffi: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server Caused by: 0: Object at location blob already exists: Client error with status 409 Conflict: response body from the dummy server 1: Client error with status 409 Conflict: response body from the dummy server Recent attempts (1 out of 1): reason: Code(409) after 0 retries Total retries: 0 Total Time: 994.02µs 2026-01-29T04:55:27.187731Z  WARN object_store_ffi: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 1.199627ms 2026-01-29T04:55:27.836560Z  WARN object_store_ffi: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server Caused by: 0: Request precondition failure for path blob: Client error with status 412 Precondition Failed: response body from the dummy server 1: Client error with status 412 Precondition Failed: response body from the dummy server Recent attempts (1 out of 1): reason: Code(412) after 0 retries Total retries: 0 Total Time: 1.149847ms 2026-01-29T04:55:28.438648Z  WARN object_store_ffi: Generic S3 error: Client error with status 413 Payload Too Large: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 413 Payload Too Large: response body from the dummy server 1: Client error with status 413 Payload Too Large: response body from the dummy server Recent attempts (1 out of 1): reason: Code(413) after 0 retries Total retries: 0 Total Time: 1.111348ms 2026-01-29T04:55:29.604499Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 1.094829ms 2026-01-29T04:55:30.265355Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 993.689µs 2026-01-29T04:55:31.573438Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 265.009639ms 2026-01-29T04:55:32.300481Z  WARN object_store_ffi: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server Caused by: 0: Generic S3 error: Client error with status 429 Too Many Requests: response body from the dummy server 1: Client error with status 429 Too Many Requests: response body from the dummy server Recent attempts (1 out of 1): reason: Code(429) after 0 retries Total retries: 0 Total Time: 1.092118ms 2026-01-29T04:55:33.811564Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 479.991083ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 479.991083ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/mybucket/blob) 1: Error after 2 retries in 479.991083ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/mybucket/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8082/mybucket/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 480.171981ms 2026-01-29T04:55:35.304232Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 538.733526ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 538.733526ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/mybucket/blob) 1: Error after 2 retries in 538.733526ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/mybucket/blob) 2: HTTP status server error (502 Bad Gateway) for url (http://127.0.0.1:8081/mybucket/blob) Recent attempts (1 out of 1): reason: Code(502) after 2 retries Total retries: 2 Total Time: 538.924964ms 2026-01-29T04:55:36.123538Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 231.962ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 231.962ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/mybucket/blob) 1: Error after 2 retries in 231.962ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/mybucket/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/mybucket/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 232.144847ms 2026-01-29T04:55:37.037673Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 254.07259ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 254.07259ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/mybucket/blob) 1: Error after 2 retries in 254.07259ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/mybucket/blob) 2: HTTP status server error (503 Service Unavailable) for url (http://127.0.0.1:8081/mybucket/blob) Recent attempts (1 out of 1): reason: Code(503) after 2 retries Total retries: 2 Total Time: 254.256838ms 2026-01-29T04:55:37.916491Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 298.967273ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 298.967273ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) 1: Error after 2 retries in 298.967273ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 299.155671ms 2026-01-29T04:55:39.011327Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 449.783595ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 449.783595ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) 1: Error after 2 retries in 449.783595ms, max_retries:2, retry_timeout:10s, source:HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) 2: HTTP status server error (504 Gateway Timeout) for url (http://127.0.0.1:8081/mybucket/blob) Recent attempts (1 out of 1): reason: Code(504) after 2 retries Total retries: 2 Total Time: 449.979083ms 2026-01-29T04:55:43.260267Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 3.233600925s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 3.233600925s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/mybucket/blob) 1: Error after 2 retries in 3.233600925s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/mybucket/blob) 2: error sending request for url (http://127.0.0.1:8081/mybucket/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.233784813s 2026-01-29T04:55:47.182674Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 3.208992681s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 3.208992681s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/mybucket/blob) 1: Error after 2 retries in 3.208992681s, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8081/mybucket/blob) 2: error sending request for url (http://127.0.0.1:8081/mybucket/blob) 3: operation timed out Recent attempts (1 out of 1): reason: Timeout after 2 retries Total retries: 2 Total Time: 3.209198649s 2026-01-29T04:55:48.926554Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 371.274813ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 371.274813ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 1: Error after 2 retries in 371.274813ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 2: error sending request for url (http://127.0.0.1:8082/mybucket/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 371.457381ms 2026-01-29T04:55:49.237532Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 279.954274ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 279.954274ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 1: Error after 2 retries in 279.954274ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 2: error sending request for url (http://127.0.0.1:8082/mybucket/blob) 3: client error (SendRequest) 4: connection closed before message completed Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 280.159771ms 2026-01-29T04:55:49.779528Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 329.24131ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 329.24131ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 1: Error after 2 retries in 329.24131ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 2: error sending request for url (http://127.0.0.1:8082/mybucket/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 329.441318ms 2026-01-29T04:55:50.075503Z  WARN object_store_ffi: Generic S3 error: Error after 2 retries in 269.720035ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) Caused by: 0: Generic S3 error: Error after 2 retries in 269.720035ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 1: Error after 2 retries in 269.720035ms, max_retries:2, retry_timeout:10s, source:error sending request for url (http://127.0.0.1:8082/mybucket/blob) 2: error sending request for url (http://127.0.0.1:8082/mybucket/blob) 3: client error (SendRequest) 4: connection error 5: Connection reset by peer (os error 104) Recent attempts (1 out of 1): reason: Io after 2 retries Total retries: 2 Total Time: 269.934103ms 2026-01-29T04:55:50.562528Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2026-01-29T04:55:51.135621Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2026-01-29T04:55:52.321293Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2026-01-29T04:55:52.321399Z  WARN object_store_ffi: io error while streaming body: Generic S3 error: error decoding response body Caused by: 0: Generic S3 error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: end of file before message length reached Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 1.893253083s 2026-01-29T04:55:52.739236Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2026-01-29T04:55:52.998158Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2026-01-29T04:55:53.919473Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2026-01-29T04:55:53.919606Z  WARN object_store_ffi: io error while streaming body: Generic S3 error: error decoding response body Caused by: 0: Generic S3 error: error decoding response body 1: error decoding response body 2: request or response body error 3: error reading a body from connection 4: Connection reset by peer (os error 104) Recent attempts (3 out of 3): reason: Io after 0 retries reason: Io after 0 retries reason: Io after 0 retries Total retries: 2 Total Time: 1.285954981s 2026-01-29T04:55:55.405150Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2026-01-29T04:55:56.927298Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2026-01-29T04:55:58.829086Z  WARN object_store_ffi::crud_ops: Error while reading body: io error while streaming body: Generic S3 error: error decoding response body 2026-01-29T04:55:58.829194Z  WARN object_store_ffi: io error while streaming body: Generic S3 error: error decoding response body Caused by: 0: Generic S3 error: error decoding response body 1: error decoding response body 2: request or response body error 3: operation timed out Recent attempts (3 out of 3): reason: Timeout after 0 retries reason: Timeout after 0 retries reason: Timeout after 0 retries Total retries: 2 Total Time: 4.459474294s 2026-01-29T04:56:00.162687Z  WARN object_store_ffi: operation was cancelled [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8082, thread id: 1 [ Info: Server on 127.0.0.1:8082 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 ┌ Warning: Verbosity logging macros are deprecated as they are not compatible with juliac-compiled programs │ caller = ip:0x0 └ @ Core :-1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/ShTJs/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/ShTJs/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:808 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#282".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/ShTJs/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#282".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:400 │ request = │ HTTP.Messages.Request: │ """ │ GET /mybucket/blob HTTP/1.1 │ host: 127.0.0.1:8081 │ x-amz-date: 20260129T045540Z │ x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/ShTJs/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/ShTJs/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:808 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#282".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/ShTJs/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#282".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:400 │ request = │ HTTP.Messages.Request: │ """ │ GET /mybucket/blob HTTP/1.1 │ host: 127.0.0.1:8081 │ x-amz-date: 20260129T045540Z │ x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ │ """ └ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:483 [ Info: Listening on: 127.0.0.1:8081, thread id: 1 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/ShTJs/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/ShTJs/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:808 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#282".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/ShTJs/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#282".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:400 │ request = │ HTTP.Messages.Request: │ """ │ PUT /mybucket/blob HTTP/1.1 │ content-length: 5 │ host: 127.0.0.1:8081 │ x-amz-date: 20260129T045543Z │ x-amz-content-sha256: 205830ca5b23bbe39ab510cfddc1dff2d9842e38b5fa7b7c48cd4ca7e44f92a1 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:483 ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/ShTJs/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/ShTJs/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:808 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#282".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/ShTJs/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#282".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:400 │ request = │ HTTP.Messages.Request: │ """ │ PUT /mybucket/blob HTTP/1.1 │ content-length: 5 │ host: 127.0.0.1:8081 │ x-amz-date: 20260129T045543Z │ x-amz-content-sha256: 205830ca5b23bbe39ab510cfddc1dff2d9842e38b5fa7b7c48cd4ca7e44f92a1 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:483 [ Info: Server on 127.0.0.1:8081 closing ┌ Error: handle_connection handler error. │ │ =========================== │ HTTP Error message: │ │ ERROR: IOError: write: broken pipe (EPIPE) │ Stacktrace: │ [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1100 │ [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64) │ @ Base ./stream.jl:1154 │ [3] unsafe_write │ @ ~/.julia/packages/HTTP/ShTJs/src/Connections.jl:132 [inlined] │ [4] write │ @ ./strings/io.jl:246 [inlined] │ [5] write │ @ ./io.jl:794 [inlined] │ [6] unsafe_write(http::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, p::Ptr{UInt8}, n::UInt64) │ @ HTTP.Streams ~/.julia/packages/HTTP/ShTJs/src/Streams.jl:95 │ [7] write │ @ ./strings/io.jl:246 [inlined] │ [8] write(io::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}, s::Base.CodeUnits{UInt8, String}) │ @ Base ./strings/basic.jl:808 │ [9] (::HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#282".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}})(stream::HTTP.Streams.Stream{HTTP.Messages.Request, HTTP.Connections.Connection{Sockets.TCPSocket}}) │ @ HTTP.Handlers ~/.julia/packages/HTTP/ShTJs/src/Handlers.jl:61 │ [10] handle_connection(f::Function, c::HTTP.Connections.Connection{Sockets.TCPSocket}, listener::HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, readtimeout::Int64, access_log::Nothing, verbose::Bool) │ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:469 │ [11] (::HTTP.Servers.var"#listenloop##0#listenloop##1"{HTTP.Handlers.var"#streamhandler##0#streamhandler##1"{Main.var"##AWS S3 retries#282".var"#test_timeout##0#test_timeout##1"{Int64, String, Base.RefValue{Int64}}}, HTTP.Servers.Listener{Nothing, Sockets.TCPServer}, Set{HTTP.Connections.Connection}, Int64, Nothing, ReentrantLock, Bool, Base.Semaphore, HTTP.Connections.Connection{Sockets.TCPSocket}})() │ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:400 │ request = │ HTTP.Messages.Request: │ """ │ PUT /mybucket/blob HTTP/1.1 │ content-length: 5 │ host: 127.0.0.1:8081 │ x-amz-date: 20260129T045543Z │ x-amz-content-sha256: 205830ca5b23bbe39ab510cfddc1dff2d9842e38b5fa7b7c48cd4ca7e44f92a1 │ authorization: ****** │ accept: */* │ user-agent: object_store/0.11.2 │ │ a,b,c""" └ @ HTTP.Servers ~/.julia/packages/HTTP/ShTJs/src/Servers.jl:483 [ Info: Listening on: 127.0.0.1:8084, thread id: 1 [ Info: Server on 127.0.0.1:8084 closing [ Info: Listening on: 127.0.0.1:8083, thread id: 1 [ Info: Server on 127.0.0.1:8083 closing [ Info: Listening on: 127.0.0.1:8084, thread id: 1 [ Info: Server on 127.0.0.1:8084 closing [ Info: Listening on: 127.0.0.1:8081, thread id: 1 [ Info: Server on 127.0.0.1:8081 closing Error in testset "Timeout" on worker 178: Test Failed at /home/pkgeval/.julia/packages/RustyObjectStore/JF6aG/test/aws_s3_exception_tests.jl:667 Expression: nrequests == 1 + max_retries Evaluated: 2 == 3 Worker 178: 04:56:07 | maxrss 2.5% | mem 5.4% | START ( 4/20) test item "AzureConfig" at test/azure_api_tests.jl:1 Worker 178: 04:56:07 | maxrss 2.5% | mem 5.4% | DONE ( 4/20) test item "AzureConfig" 0.4 secs (95.6% compile), 71.66 K allocs (4.001 MB) Worker 178: 04:56:08 | maxrss 2.5% | mem 5.4% | START ( 5/20) test item "Basic BlobStorage exceptions" at test/azure_blobs_exception_tests.jl:1 Worker 178: 04:56:42 | maxrss 2.8% | mem 5.5% | DONE ( 5/20) test item "Basic BlobStorage exceptions" 34.2 secs (74.6% compile, <0.1% recompile, 1.2% GC), 8.41 M allocs (525.842 MB) Worker 178: 04:56:50 | maxrss 2.8% | mem 5.5% | START ( 6/20) test item "BlobStorage retries" at test/azure_blobs_exception_tests.jl:321 Worker 178: 04:57:29 | maxrss 2.8% | mem 5.5% | DONE ( 6/20) test item "BlobStorage retries" 39.7 secs (15.9% compile, <0.1% recompile, 1.3% GC), 5.20 M allocs (342.554 MB) Worker 178: [ Info: Server on 127.0.0.1:8081 closing Worker 178: 04:57:36 | maxrss 2.8% | mem 5.5% | START ( 7/20) test item "Basic BlobStorage usage" at test/basic_unified_tests.jl:686 Worker 178: 04:59:05 | maxrss 4.2% | mem 7.2% | DONE ( 7/20) test item "Basic BlobStorage usage" 88.0 secs (52.4% compile, <0.1% recompile, 3.6% GC), 10.39 M allocs (1.846 GB) Worker 178: 04:59:17 | maxrss 4.2% | mem 7.2% | START ( 8/20) test item "Basic BlobStorage usage (anonymous read enabled)" at test/basic_unified_tests.jl:719 Worker 178: 04:59:23 | maxrss 4.2% | mem 6.5% | DONE ( 8/20) test item "Basic BlobStorage usage (anonymous read enabled)" 6.1 secs (15.8% compile, <0.1% recompile, 9.6% GC), 1.02 M allocs (291.964 MB) Worker 178: 04:59:27 | maxrss 4.2% | mem 6.5% | START ( 9/20) test item "Basic AWS S3 usage" at test/basic_unified_tests.jl:744 Worker 178: 05:00:38 | maxrss 4.8% | mem 7.5% | DONE ( 9/20) test item "Basic AWS S3 usage" 70.8 secs (57.3% compile, <0.1% recompile, 3.8% GC), 6.84 M allocs (1.651 GB) Worker 178: 05:00:59 | maxrss 4.8% | mem 7.5% | START (10/20) test item "Basic AWS S3 usage (anonymous read enabled)" at test/basic_unified_tests.jl:778 Worker 178: 05:01:03 | maxrss 4.8% | mem 6.8% | DONE (10/20) test item "Basic AWS S3 usage (anonymous read enabled)" 4.1 secs (32.5% compile, <0.1% recompile, 22.6% GC), 1.01 M allocs (291.131 MB) Worker 178: 05:01:07 | maxrss 4.8% | mem 6.8% | START (11/20) test item "Basic Snowflake Stage usage: AWS, non-encrypted" at test/basic_unified_tests.jl:805 Worker 178: 05:02:41 | maxrss 5.1% | mem 8.3% | DONE (11/20) test item "Basic Snowflake Stage usage: AWS, non-encrypted" 93.7 secs (70.7% compile, <0.1% recompile, 3.2% GC), 16.29 M allocs (2.169 GB) Worker 178: 05:02:53 | maxrss 5.1% | mem 8.3% | START (12/20) test item "Basic Snowflake Stage usage: AWS, encrypted" at test/basic_unified_tests.jl:822 Worker 178: 05:03:35 | maxrss 5.1% | mem 8.1% | DONE (12/20) test item "Basic Snowflake Stage usage: AWS, encrypted" 41.7 secs (28.6% compile, <0.1% recompile, 5.2% GC), 7.31 M allocs (1.452 GB) Worker 178: 05:03:55 | maxrss 5.1% | mem 8.1% | START (13/20) test item "Basic Snowflake Stage usage: Azure, non-encrypted" at test/basic_unified_tests.jl:839 Worker 178: 05:04:50 | maxrss 5.1% | mem 8.1% | DONE (13/20) test item "Basic Snowflake Stage usage: Azure, non-encrypted" 55.1 secs (7.4% compile, 4.1% GC), 4.46 M allocs (1.287 GB) Worker 178: 05:05:02 | maxrss 5.1% | mem 8.1% | START (14/20) test item "Basic Snowflake Stage usage: Azure, encrypted" at test/basic_unified_tests.jl:856 Worker 178: 05:05:55 | maxrss 5.2% | mem 8.6% | DONE (14/20) test item "Basic Snowflake Stage usage: Azure, encrypted" 52.8 secs (20.4% compile, 4.1% GC), 6.85 M allocs (1.425 GB) Worker 178: 05:06:16 | maxrss 5.2% | mem 8.6% | START (15/20) test item "destroy_* functions do not panic" at test/destroy_test.jl:1 Worker 178: 05:06:16 | maxrss 5.2% | mem 8.6% | DONE (15/20) test item "destroy_* functions do not panic" <0.1 secs (86.8% compile), 2.08 K allocs (98.784 KB) Worker 178: 05:06:17 | maxrss 5.2% | mem 8.6% | START (16/20) test item "Using object_store_ffi_jll" at test/release_test.jl:4 Worker 178: 05:06:17 | maxrss 5.2% | mem 8.6% | DONE (16/20) test item "Using object_store_ffi_jll" <0.1 secs (<0.1% compile), 1.34 K allocs (75.024 KB) Worker 178: 05:06:18 | maxrss 5.2% | mem 8.6% | START (17/20) test item "Handle object_store_ffi panic" at test/rust_panic_test.jl:1 Worker 178: 05:06:24 | maxrss 5.2% | mem 8.6% | DONE (17/20) test item "Handle object_store_ffi panic" 6.2 secs (7.5% compile), 154.28 K allocs (8.619 MB) Worker 178: 05:06:25 | maxrss 5.2% | mem 8.6% | START (18/20) test item "SnowflakeConfig" at test/snowflake_api_tests.jl:1 Worker 178: 05:06:25 | maxrss 5.2% | mem 8.6% | DONE (18/20) test item "SnowflakeConfig" 0.3 secs (96.8% compile), 104.60 K allocs (6.349 MB) Worker 178: 05:06:26 | maxrss 5.2% | mem 8.6% | START (19/20) test item "Basic Stage exceptions" at test/snowflake_stage_exception_tests.jl:1 Worker 178: 05:06:52 | maxrss 5.2% | mem 8.7% | DONE (19/20) test item "Basic Stage exceptions" 25.8 secs (17.1% compile), 1.50 M allocs (87.907 MB) Worker 178: 2026-01-29T05:06:56.853424Z  WARN object_store_ffi::snowflake::client: Heartbeat failed: error sending request for url (http://127.0.0.1:8081/session/heartbeat) Worker 178: Caused by: Worker 178: 0: error sending request for url (http://127.0.0.1:8081/session/heartbeat) Worker 178: 1: client error (Connect) Worker 178: 2: tcp connect error: Connection refused (os error 111) Worker 178: 3: Connection refused (os error 111) Worker 178: 05:07:01 | maxrss 5.2% | mem 8.7% | START (20/20) test item "Snowflake Stage retries" at test/snowflake_stage_exception_tests.jl:158 Worker 178: 05:08:19 | maxrss 5.2% | mem 7.5% | DONE (20/20) test item "Snowflake Stage retries" 78.5 secs (14.1% compile, <0.1% recompile, 0.7% GC), 5.63 M allocs (365.698 MB) [ Info: All tests on worker 1 completed. Closing Worker(pid=178). Worker 178: [178] signal 15: Terminated [ Tests Completed: 20/20 test items were run. Test Summary: | Pass Fail Broken Total Time RustyObjectStore | 15233 1 8 15242 14m48.8s test | 15233 1 8 15242 test/aws_api_tests.jl | 3 3 test/aws_s3_exception_tests.jl | 162 1 2 165 Basic S3 exceptions | 23 23 49.7s AWS S3 retries | 139 1 2 142 43.3s 400: Bad Request | 10 10 5.1s 403: Forbidden | 10 10 1.2s 404: Not Found | 5 5 0.7s 405: Method Not Supported | 10 10 1.9s 409: Conflict | 10 10 1.2s 412: Precondition Failed | 10 10 1.2s 413: Content Too Large | 5 5 0.7s 429: Too Many Requests | 18 2 20 3.7s 502: Bad Gateway | 8 8 2.6s 503: Service Unavailable | 8 8 1.7s 504: Gateway Timeout | 8 8 2.0s Timeout | 7 1 8 8.8s TCP Closed | 8 8 0.9s TCP reset | 8 8 0.8s Incomplete GET body | 4 4 2.2s Incomplete GET body reset | 4 4 1.6s Incomplete GET body timeout | 4 4 4.9s Cancellation | 2 2 1.3s test/azure_api_tests.jl | 5 5 test/azure_blobs_exception_tests.jl | 183 2 185 test/basic_unified_tests.jl | 14560 14560 test/destroy_test.jl | 2 2 test/release_test.jl | 1 1 test/rust_panic_test.jl | 1 1 test/snowflake_api_tests.jl | 3 3 test/snowflake_stage_exception_tests.jl | 313 4 317 ERROR: LoadError: Some tests did not pass: 15233 passed, 1 failed, 0 errored, 8 broken. in expression starting at /home/pkgeval/.julia/packages/RustyObjectStore/JF6aG/test/runtests.jl:4 Testing failed after 749.04s ERROR: LoadError: Package RustyObjectStore errored during testing Stacktrace: [1] pkgerror(msg::String) @ Pkg.Types /opt/julia/share/julia/stdlib/v1.12/Pkg/src/Types.jl:68 [2] test(ctx::Pkg.Types.Context, pkgs::Vector{PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool) @ Pkg.Operations /opt/julia/share/julia/stdlib/v1.12/Pkg/src/Operations.jl:2535 [3] test @ /opt/julia/share/julia/stdlib/v1.12/Pkg/src/Operations.jl:2384 [inlined] [4] test(ctx::Pkg.Types.Context, pkgs::Vector{PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{io::IOContext{IO}}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.12/Pkg/src/API.jl:538 [5] test(pkgs::Vector{PackageSpec}; io::IOContext{IO}, kwargs::@Kwargs{julia_args::Cmd}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.12/Pkg/src/API.jl:169 [6] test(pkgs::Vector{String}; kwargs::@Kwargs{julia_args::Cmd}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.12/Pkg/src/API.jl:157 [7] test @ /opt/julia/share/julia/stdlib/v1.12/Pkg/src/API.jl:157 [inlined] [8] #test#81 @ /opt/julia/share/julia/stdlib/v1.12/Pkg/src/API.jl:156 [inlined] [9] top-level scope @ /PkgEval.jl/scripts/evaluate.jl:223 [10] include(mod::Module, _path::String) @ Base ./Base.jl:306 [11] exec_options(opts::Base.JLOptions) @ Base ./client.jl:317 [12] _start() @ Base ./client.jl:550 in expression starting at /PkgEval.jl/scripts/evaluate.jl:214 PkgEval failed after 1166.27s: package has test failures