Package evaluation to test RAGTools on Julia 1.11.8 (29b3528cce*) started at 2026-01-20T07:52:06.610 ################################################################################ # Set-up # Installing PkgEval dependencies (TestEnv)... Activating project at `~/.julia/environments/v1.11` Set-up completed after 8.55s ################################################################################ # Installation # Installing RAGTools... Resolving package versions... Installed HTML_Entities ─ v1.0.2 Updating `~/.julia/environments/v1.11/Project.toml` [16ddad29] + RAGTools v0.10.0 Updating `~/.julia/environments/v1.11/Manifest.toml` [1520ce14] + AbstractTrees v0.4.5 [d1d4a3ce] + BitFlags v0.1.9 [944b1d66] + CodecZlib v0.7.8 [f0e56b4a] + ConcurrentUtilities v2.5.0 [124859b0] + DataDeps v0.7.13 [460bff9d] + ExceptionUnwrapping v0.1.11 [7693890a] + HTML_Entities v1.0.2 [cd3eb016] + HTTP v1.10.19 [692b3bcd] + JLLWrappers v1.7.1 [682c06a0] + JSON v1.4.0 [0f8b85d8] + JSON3 v1.14.3 [8ef0a80b] + Languages v0.4.7 [e6f89c97] + LoggingExtras v1.2.0 [739be429] + MbedTLS v1.1.9 ⌅ [e9f21f70] + OpenAI v0.11.0 [4d8831e6] + OpenSSL v1.6.1 [69de0a69] + Parsers v2.8.3 ⌅ [aea7be01] + PrecompileTools v1.2.1 [21216c6a] + Preferences v1.5.1 [92933f4c] + ProgressMeter v1.11.0 ⌅ [670122d1] + PromptingTools v0.87.0 [16ddad29] + RAGTools v0.10.0 [189a3867] + Reexport v1.2.2 [05181044] + RelocatableFolders v1.0.1 [6c6a2e73] + Scratch v1.3.0 [777ac1f9] + SimpleBufferStream v1.2.0 [fb8f903a] + Snowball v0.1.1 [9700d1a9] + StrTables v1.0.1 ⌅ [c1b9e933] + StreamCallbacks v0.6.2 [856f2bd8] + StructTypes v1.11.0 [ec057cc2] + StructUtils v2.6.2 [3bb67fe8] + TranscodingStreams v0.11.3 [5c2747f8] + URIs v1.6.1 [796a5d58] + WordTokenizers v0.5.6 [458c3c95] + OpenSSL_jll v3.5.4+0 [88f46535] + Snowball_jll v2.2.0+0 [0dad84c5] + ArgTools v1.1.2 [56f22d72] + Artifacts v1.11.0 [2a0f44e3] + Base64 v1.11.0 [ade2ca70] + Dates v1.11.0 [8ba89e20] + Distributed v1.11.0 [f43a241f] + Downloads v1.6.0 [7b1f6079] + FileWatching v1.11.0 [b77e0a4c] + InteractiveUtils v1.11.0 [b27032c2] + LibCURL v0.6.4 [76f85450] + LibGit2 v1.11.0 [8f399da3] + Libdl v1.11.0 [37e2e46d] + LinearAlgebra v1.11.0 [56ddb016] + Logging v1.11.0 [d6f4376e] + Markdown v1.11.0 [a63ad114] + Mmap v1.11.0 [ca575930] + NetworkOptions v1.2.0 [44cfe95a] + Pkg v1.11.0 [de0858da] + Printf v1.11.0 [3fa0cd96] + REPL v1.11.0 [9a3f8284] + Random v1.11.0 [ea8e919c] + SHA v0.7.0 [9e88b42a] + Serialization v1.11.0 [6462fe0b] + Sockets v1.11.0 [2f01184e] + SparseArrays v1.11.0 [f489334b] + StyledStrings v1.11.0 [fa267f1f] + TOML v1.0.3 [a4e569a6] + Tar v1.10.0 [8dfed614] + Test v1.11.0 [cf7118a7] + UUIDs v1.11.0 [4ec0a83e] + Unicode v1.11.0 [e66e0078] + CompilerSupportLibraries_jll v1.1.1+0 [deac9b47] + LibCURL_jll v8.6.0+0 [e37daf67] + LibGit2_jll v1.7.2+0 [29816b5a] + LibSSH2_jll v1.11.0+1 [c8ffd9c3] + MbedTLS_jll v2.28.6+0 [14a3606d] + MozillaCACerts_jll v2023.12.12 [4536629a] + OpenBLAS_jll v0.3.27+1 [bea87d4a] + SuiteSparse_jll v7.7.0+0 [83775a58] + Zlib_jll v1.2.13+1 [8e850b90] + libblastrampoline_jll v5.11.0+0 [8e850ede] + nghttp2_jll v1.59.0+0 [3f19e933] + p7zip_jll v17.4.0+2 Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m` Building HTML_Entities → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/781d638b8892cd1a1e867f51863d8881c0a62834/build.log` Installation completed after 9.6s ################################################################################ # Precompilation # Precompiling PkgEval dependencies... Precompiling package dependencies... Precompiling project... 35788.2 ms ✓ PromptingTools 13036.4 ms ✓ PromptingTools → MarkdownPromptingToolsExt 120618.0 ms ✓ RAGTools 3 dependencies successfully precompiled in 170 seconds. 71 already precompiled. Precompilation completed after 187.45s ################################################################################ # Testing # Testing RAGTools Status `/tmp/jl_nHFlBz/Project.toml` [1520ce14] AbstractTrees v0.4.5 [4c88cf16] Aqua v0.8.14 [cd3eb016] HTTP v1.10.19 [0f8b85d8] JSON3 v1.14.3 [92933f4c] ProgressMeter v1.11.0 ⌅ [670122d1] PromptingTools v0.87.0 [16ddad29] RAGTools v0.10.0 [fb8f903a] Snowball v0.1.1 [37e2e46d] LinearAlgebra v1.11.0 [9a3f8284] Random v1.11.0 [2f01184e] SparseArrays v1.11.0 [8dfed614] Test v1.11.0 [4ec0a83e] Unicode v1.11.0 Status `/tmp/jl_nHFlBz/Manifest.toml` [1520ce14] AbstractTrees v0.4.5 [4c88cf16] Aqua v0.8.14 [d1d4a3ce] BitFlags v0.1.9 [944b1d66] CodecZlib v0.7.8 [34da2185] Compat v4.18.1 [f0e56b4a] ConcurrentUtilities v2.5.0 [124859b0] DataDeps v0.7.13 [460bff9d] ExceptionUnwrapping v0.1.11 [7693890a] HTML_Entities v1.0.2 [cd3eb016] HTTP v1.10.19 [692b3bcd] JLLWrappers v1.7.1 [682c06a0] JSON v1.4.0 [0f8b85d8] JSON3 v1.14.3 [8ef0a80b] Languages v0.4.7 [e6f89c97] LoggingExtras v1.2.0 [739be429] MbedTLS v1.1.9 ⌅ [e9f21f70] OpenAI v0.11.0 [4d8831e6] OpenSSL v1.6.1 [69de0a69] Parsers v2.8.3 ⌅ [aea7be01] PrecompileTools v1.2.1 [21216c6a] Preferences v1.5.1 [92933f4c] ProgressMeter v1.11.0 ⌅ [670122d1] PromptingTools v0.87.0 [16ddad29] RAGTools v0.10.0 [189a3867] Reexport v1.2.2 [05181044] RelocatableFolders v1.0.1 [6c6a2e73] Scratch v1.3.0 [777ac1f9] SimpleBufferStream v1.2.0 [fb8f903a] Snowball v0.1.1 [9700d1a9] StrTables v1.0.1 ⌅ [c1b9e933] StreamCallbacks v0.6.2 [856f2bd8] StructTypes v1.11.0 [ec057cc2] StructUtils v2.6.2 [3bb67fe8] TranscodingStreams v0.11.3 [5c2747f8] URIs v1.6.1 [796a5d58] WordTokenizers v0.5.6 [458c3c95] OpenSSL_jll v3.5.4+0 [88f46535] Snowball_jll v2.2.0+0 [0dad84c5] ArgTools v1.1.2 [56f22d72] Artifacts v1.11.0 [2a0f44e3] Base64 v1.11.0 [ade2ca70] Dates v1.11.0 [8ba89e20] Distributed v1.11.0 [f43a241f] Downloads v1.6.0 [7b1f6079] FileWatching v1.11.0 [b77e0a4c] InteractiveUtils v1.11.0 [b27032c2] LibCURL v0.6.4 [76f85450] LibGit2 v1.11.0 [8f399da3] Libdl v1.11.0 [37e2e46d] LinearAlgebra v1.11.0 [56ddb016] Logging v1.11.0 [d6f4376e] Markdown v1.11.0 [a63ad114] Mmap v1.11.0 [ca575930] NetworkOptions v1.2.0 [44cfe95a] Pkg v1.11.0 [de0858da] Printf v1.11.0 [3fa0cd96] REPL v1.11.0 [9a3f8284] Random v1.11.0 [ea8e919c] SHA v0.7.0 [9e88b42a] Serialization v1.11.0 [6462fe0b] Sockets v1.11.0 [2f01184e] SparseArrays v1.11.0 [f489334b] StyledStrings v1.11.0 [fa267f1f] TOML v1.0.3 [a4e569a6] Tar v1.10.0 [8dfed614] Test v1.11.0 [cf7118a7] UUIDs v1.11.0 [4ec0a83e] Unicode v1.11.0 [e66e0078] CompilerSupportLibraries_jll v1.1.1+0 [deac9b47] LibCURL_jll v8.6.0+0 [e37daf67] LibGit2_jll v1.7.2+0 [29816b5a] LibSSH2_jll v1.11.0+1 [c8ffd9c3] MbedTLS_jll v2.28.6+0 [14a3606d] MozillaCACerts_jll v2023.12.12 [4536629a] OpenBLAS_jll v0.3.27+1 [bea87d4a] SuiteSparse_jll v7.7.0+0 [83775a58] Zlib_jll v1.2.13+1 [8e850b90] libblastrampoline_jll v5.11.0+0 [8e850ede] nghttp2_jll v1.59.0+0 [3f19e933] p7zip_jll v17.4.0+2 Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. Testing Running tests... [ Info: Embedding 2 documents... [ Info: Done embedding. Total cost: $0.0 [ Info: Embedding 2 documents... [ Info: Truncate_dimension set to 0. Skipping truncation [ Info: Done embedding. Total cost: $0.0 [ Info: Embedding 2 documents... [ Info: Done embedding. Total cost: $0.0 [ Info: Embedding 2 documents... [ Info: Done embedding. Total cost: $0.0 [ Info: Done processing DocumentTermMatrix. [ Info: Done processing DocumentTermMatrix. [ Info: Done processing DocumentTermMatrix. [ Info: Done processing DocumentTermMatrix. [ Info: Done processing DocumentTermMatrix. [ Info: Done processing DocumentTermMatrix. [ Info: Done processing DocumentTermMatrix. [ Info: Extracting metadata from 1 documents... [ Info: Done extracting the tags. Total cost: $0.0 ┌ Warning: Model `mock-emb` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 ┌ Warning: Model `mock-meta` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Index built! (cost: $0.0) [ Info: Index built! (cost: $0.0) [ Info: Index built! (cost: $0.0) [ Info: Index built! (cost: $0.0) [ Info: Index built! [ Info: Index built! (cost: $0.0) ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Tokens: 3 in 0.0 seconds [ Info: Tokens: 3 in 0.0 seconds [ Info: Tokens: 3 in 0.0 seconds ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Tokens: 3 in 0.0 seconds ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Tokens: 3 in 0.0 seconds ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Tokens: 3 in 0.0 seconds ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Ranking chunks in positions 1 to 2 [ Info: Tokens: 3 in 0.0 seconds [ Info: Ranking chunks in positions 1 to 3 [ Info: Tokens: 3 in 0.0 seconds [ Info: Ranking chunks in positions 2 to 4 [ Info: Tokens: 3 in 0.0 seconds ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Ranking chunks in positions 1 to 3 [ Info: Tokens: 3 in 0.0 seconds ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Ranking chunks in positions 1 to 3 [ Info: Tokens: 3 in 0.0 seconds ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Ranking chunks in positions 1 to 3 [ Info: Tokens: 3 in 0.0 seconds ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Round 1 of 1 of ranking process. [ Info: Final ranking done. Tokens: 3, Cost: 0.0, Time: 0.0s [ Info: Round 1 of 1 of ranking process. [ Info: Final ranking done. Tokens: 3, Cost: 0.0, Time: 0.0s [ Info: Round 1 of 1 of ranking process. [ Info: Final ranking done. Tokens: 3, Cost: 0.0, Time: 0.0s [ Info: Round 1 of 1 of ranking process. [ Info: Final ranking done. Tokens: 3, Cost: 0.0, Time: 0.0s ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Round 1 of 1 of ranking process. [ Info: Final ranking done. Tokens: 3, Cost: 0.0, Time: 0.0s ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Round 1 of 1 of ranking process. [ Info: Final ranking done. Tokens: 3, Cost: 0.0, Time: 0.0s ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Round 1 of 1 of ranking process. [ Info: Final ranking done. Tokens: 3, Cost: 0.0, Time: 0.0s ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Tokens: 3 in 0.0 seconds [ Info: Tokens: 3 in 0.0 seconds ┌ Warning: BM25: `max_score` is zero, so scores are not normalized. Returning unnormalized scores (all zero). └ @ RAGTools ~/.julia/packages/RAGTools/TNS7E/src/bm25.jl:198 ┌ Warning: BM25: `max_score` is zero, so scores are not normalized. Returning unnormalized scores (all zero). └ @ RAGTools ~/.julia/packages/RAGTools/TNS7E/src/bm25.jl:198 ┌ Warning: Model `mock-emb` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 ┌ Warning: Model `mock-meta` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Retrieval done. Identified 4 chunks, total cost: $0.0. [ Info: Retrieval done. Identified 2 chunks, total cost: $0.0. [ Info: Retrieval done. Identified 2 chunks, total cost: $0.0. [ Info: Retrieval done. Identified 4 chunks, total cost: $0.0. [ Info: Index built! [ Info: Retrieval done. Identified 5 chunks, total cost: $0.0. ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Done generating the answer. Cost: $0.0 ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Done refining the answer. Cost: $0.0 ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 ┌ Warning: Model `mock-emb` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 ┌ Warning: Model `mock-meta` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Tokens: 2 in 0.5 seconds [ Info: Tokens: 53 in 0.0 seconds (Metadata: tool_calls) [ Info: Tokens: 31 in 0.0 seconds [ Info: Done with RAG. Total cost: $0.0 [ Info: Done with RAG. Total cost: $0.0 [ Info: Done with RAG. Total cost: $0.0 WARNING: Method definition (::Type{Main.Random123Annotater})() in module Main at /home/pkgeval/.julia/packages/RAGTools/TNS7E/test/annotation.jl:127 overwritten at /home/pkgeval/.julia/packages/RAGTools/TNS7E/test/annotation.jl:184. WARNING: Method definition (::Type{Main.Random123Annotater})() in module Main at /home/pkgeval/.julia/packages/RAGTools/TNS7E/test/annotation.jl:184 overwritten at /home/pkgeval/.julia/packages/RAGTools/TNS7E/test/annotation.jl:271. WARNING: Method definition (::Type{Main.Random123AnnotatedNode})() in module Main at /home/pkgeval/.julia/packages/RAGTools/TNS7E/test/annotation.jl:52 overwritten at /home/pkgeval/.julia/packages/RAGTools/TNS7E/test/annotation.jl:272. WARNING: Method definition (::Type{Main.Random123Annotater})() in module Main at /home/pkgeval/.julia/packages/RAGTools/TNS7E/test/annotation.jl:271 overwritten at /home/pkgeval/.julia/packages/RAGTools/TNS7E/test/annotation.jl:353. ┌ Warning: Model `mock-emb` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 ┌ Warning: Model `mock-meta` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 ┌ Warning: Model `mock-gen` already registered! It will be overwritten. └ @ PromptingTools ~/.julia/packages/PromptingTools/G0lho/src/user_preferences.jl:372 [ Info: Tokens: 77 in 2.3 seconds (Metadata: tool_calls) [ Info: Tokens: 77 in 3.0 seconds (Metadata: tool_calls) [ Info: Tokens: 77 in 3.0 seconds (Metadata: tool_calls) [ Info: Q&A Sets built! (cost: $0.0) [ Info: Done with RAG. Total cost: $0.0 [ Info: Tokens: 189 in 1.1 seconds (Metadata: tool_calls) [ Info: Done with RAG. Total cost: $0.0 [ Info: Done with RAG. Total cost: $0.0 [ Info: Done with RAG. Total cost: $0.0 [ Info: Tokens: 189 in 0.0 seconds (Metadata: tool_calls) [ Info: Tokens: 189 in 0.0 seconds (Metadata: tool_calls) [ Info: Tokens: 189 in 0.0 seconds (Metadata: tool_calls) [ Info: QA Evaluations complete (1.0 evals successful)! Test Summary: | Pass Total Time RAGTools.jl | 1137 1137 10m39.8s Testing RAGTools tests passed Testing completed after 680.3s PkgEval succeeded after 898.85s