Package evaluation to test DocumenterDescriptions on Julia 1.14.0-DEV.2309 (6e1a27e459*) started at 2026-06-06T22:03:29.050 ################################################################################ # Set-up # Installing PkgEval dependencies (TestEnv)... Activating project at `~/.julia/environments/v1.14` Set-up completed after 17.31s ################################################################################ # Installation # Installing DocumenterDescriptions... Resolving package versions... Installed ConcurrentUtilities ───── v2.5.1 Installed AbstractTrees ─────────── v0.4.5 Installed StreamCallbacks ───────── v0.6.2 Installed URIs ──────────────────── v1.6.1 Installed CodecZlib ─────────────── v0.7.8 Installed ExceptionUnwrapping ───── v0.1.11 Installed OpenSSH_jll ───────────── v10.3.1+0 Installed TranscodingStreams ────── v0.11.3 Installed Git_LFS_jll ───────────── v3.7.1+0 Installed MbedTLS ───────────────── v1.1.10 Installed StructUtils ───────────── v2.8.2 Installed OpenSSL ───────────────── v1.6.1 Installed LoggingExtras ─────────── v1.2.0 Installed MarkdownAST ───────────── v0.1.3 Installed Expat_jll ─────────────── v2.8.1+0 Installed JSON3 ─────────────────── v1.14.3 Installed PrecompileTools ───────── v1.3.4 Installed LazilyInitializedFields ─ v1.3.0 Installed HTTP ──────────────────── v1.11.0 Installed BitFlags ──────────────── v0.1.10 Installed JSON ──────────────────── v1.6.1 Installed DocumenterDescriptions ── v0.1.0 Installed OpenAI ────────────────── v0.9.1 Installed Documenter ────────────── v1.17.0 Installed Parsers ───────────────── v2.8.5 Installed SimpleBufferStream ────── v1.2.0 Installed Git_jll ───────────────── v2.54.0+0 Installed RegistryInstances ─────── v0.1.0 Installed Libiconv_jll ──────────── v1.18.0+0 Installed MbedTLS_jll ───────────── v2.28.1010+0 Installed IOCapture ─────────────── v1.0.0 Installed Git ───────────────────── v1.5.0 Installed Preferences ───────────── v1.5.2 Installed JLLWrappers ───────────── v1.8.0 Installed StructTypes ───────────── v1.11.0 Installed DocStringExtensions ───── v0.9.5 Installed ANSIColoredPrinters ───── v0.0.1 Installed PromptingTools ────────── v0.73.0 Installing 6 artifacts Installed artifact Expat 288.6 KiB Installed artifact MbedTLS 2.2 MiB Installed artifact Libiconv 1.9 MiB Installed artifact OpenSSH 4.1 MiB Installed artifact Git_LFS 9.7 MiB Installed artifact Git 50.6 MiB Updating `~/.julia/environments/v1.14/Project.toml` [e9159221] + DocumenterDescriptions v0.1.0 Updating `~/.julia/environments/v1.14/Manifest.toml` [a4c015fc] + ANSIColoredPrinters v0.0.1 [1520ce14] + AbstractTrees v0.4.5 [d1d4a3ce] + BitFlags v0.1.10 [944b1d66] + CodecZlib v0.7.8 [f0e56b4a] + ConcurrentUtilities v2.5.1 [ffbed154] + DocStringExtensions v0.9.5 [e30172f5] + Documenter v1.17.0 [e9159221] + DocumenterDescriptions v0.1.0 [460bff9d] + ExceptionUnwrapping v0.1.11 [d7ba0133] + Git v1.5.0 ⌅ [cd3eb016] + HTTP v1.11.0 [b5f81e59] + IOCapture v1.0.0 [692b3bcd] + JLLWrappers v1.8.0 [682c06a0] + JSON v1.6.1 [0f8b85d8] + JSON3 v1.14.3 [0e77f7df] + LazilyInitializedFields v1.3.0 [e6f89c97] + LoggingExtras v1.2.0 [d0879d2d] + MarkdownAST v0.1.3 [739be429] + MbedTLS v1.1.10 ⌅ [e9f21f70] + OpenAI v0.9.1 [4d8831e6] + OpenSSL v1.6.1 [69de0a69] + Parsers v2.8.5 [aea7be01] + PrecompileTools v1.3.4 [21216c6a] + Preferences v1.5.2 ⌅ [670122d1] + PromptingTools v0.73.0 [2792f1a3] + RegistryInstances v0.1.0 [777ac1f9] + SimpleBufferStream v1.2.0 ⌅ [c1b9e933] + StreamCallbacks v0.6.2 [856f2bd8] + StructTypes v1.11.0 [ec057cc2] + StructUtils v2.8.2 [3bb67fe8] + TranscodingStreams v0.11.3 [5c2747f8] + URIs v1.6.1 [2e619515] + Expat_jll v2.8.1+0 [020c3dae] + Git_LFS_jll v3.7.1+0 [f8c6e375] + Git_jll v2.54.0+0 [94ce4f54] + Libiconv_jll v1.18.0+0 [c8ffd9c3] + MbedTLS_jll v2.28.1010+0 [9bd350c2] + OpenSSH_jll v10.3.1+0 [0dad84c5] + ArgTools v1.1.2 [56f22d72] + Artifacts v1.11.0 [2a0f44e3] + Base64 v1.11.0 [ade2ca70] + Dates v1.11.0 [f43a241f] + Downloads v1.7.0 [7b1f6079] + FileWatching v1.11.0 [b77e0a4c] + InteractiveUtils v1.11.0 [ac6e5ff7] + JuliaSyntaxHighlighting v1.13.0 [b27032c2] + LibCURL v1.0.0 [76f85450] + LibGit2 v1.11.0 [8f399da3] + Libdl v1.11.0 [56ddb016] + Logging v1.11.0 [d6f4376e] + Markdown v1.11.0 [a63ad114] + Mmap v1.11.0 [ca575930] + NetworkOptions v1.3.0 [44cfe95a] + Pkg v1.14.0 [de0858da] + Printf v1.11.0 [3fa0cd96] + REPL v1.11.0 [9a3f8284] + Random v1.11.0 [ea8e919c] + SHA v1.13.0 [9e88b42a] + Serialization v1.11.0 [6462fe0b] + Sockets v1.11.0 [f489334b] + StyledStrings v1.13.0 [fa267f1f] + TOML v1.0.3 [a4e569a6] + Tar v1.10.0 [8dfed614] + Test v1.11.0 [cf7118a7] + UUIDs v1.11.0 [4ec0a83e] + Unicode v1.11.0 [e66e0078] + CompilerSupportLibraries_jll v1.5.2+0 [deac9b47] + LibCURL_jll v8.20.0+1 [e37daf67] + LibGit2_jll v1.9.4+0 [29816b5a] + LibSSH2_jll v1.11.101+0 [14a3606d] + MozillaCACerts_jll v2026.5.14 [458c3c95] + OpenSSL_jll v3.5.6+0 [efcefdf7] + PCRE2_jll v10.47.0+0 [83775a58] + Zlib_jll v1.3.2+0 [3161d3a3] + Zstd_jll v1.5.7+1 [8e850ede] + nghttp2_jll v1.69.0+0 [3f19e933] + p7zip_jll v17.8.0+0 Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m` Installation completed after 13.62s ################################################################################ # Precompilation # Precompiling PkgEval dependencies... Precompiling project... 6.6 s ✓ TestEnv 1 dependency successfully precompiled in 7 seconds. 27 already precompiled. Precompiling package dependencies... Precompiling project... 1.3 s ✓ ANSIColoredPrinters 39.7 s ✓ URIs 0.9 s ✓ LazilyInitializedFields 39.5 s ✓ BitFlags 1.1 s ✓ DocStringExtensions 75.9 s ✓ AbstractTrees 55.2 s ✓ TranscodingStreams 0.7 s ✓ SimpleBufferStream 0.8 s ✓ IOCapture 1.7 s ✓ ConcurrentUtilities 3.6 s ✓ StructTypes 2.7 s ✓ LoggingExtras 3.2 s ✓ StructUtils 2.9 s ✓ Preferences 39.6 s ✓ ExceptionUnwrapping 7.8 s ✓ RegistryInstances 81.4 s ✓ OpenSSL 21.8 s ✓ MarkdownAST 20.3 s ✓ CodecZlib 3.4 s ✓ JLLWrappers 3.0 s ✓ PrecompileTools 3.6 s ✓ Libiconv_jll 3.7 s ✓ Git_LFS_jll 3.6 s ✓ MbedTLS_jll 3.7 s ✓ OpenSSH_jll 3.8 s ✓ Expat_jll 39.4 s ✓ Parsers 4.1 s ✓ MbedTLS 3.8 s ✓ Git_jll 43.5 s ✓ JSON3 108.5 s ✓ JSON ┌ Info: Ignoring an error that occurred during the precompilation workload │ exception = │ HTTP.RequestError: │ HTTP.Request: │ HTTP.Messages.Request: │ """ │ GET / HTTP/1.1 │ Host: localhost:57814 │ Accept: */* │ User-Agent: HTTP.jl/1.14.0-DEV.2309 │ Content-Length: 0 │ Accept-Encoding: gzip │ │ """Underlying error: │ MethodError: objects of type ReentrantLock are not callable. │ In case you did not try calling it explicitly, check if a ReentrantLock has been passed as an argument to a method that expects a callable instead. │ The object of type `ReentrantLock` exists, but no method is defined for this combination of argument types when trying to treat it as a callable object. │ Stacktrace: │ [1] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/ConnectionRequest.jl:143 [inlined] │ [2] (::HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, socket_type_tls::Nothing, readtimeout::Int64, connect_timeout::Int64, logerrors::Bool, logtag::Nothing, closeimmediately::Bool, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64}) │ @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/Y97L1/src/clientlayers/ConnectionRequest.jl:51 │ [3] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/ConnectionRequest.jl:60 [inlined] │ [4] (::Base.var"#45#46"{Base.var"#47#48"{ExponentialBackOff, HTTP.RetryRequest.var"#retrylayer##kw_body#manageretries#0##->###0"{HTTP.Messages.Request, Int64, typeof(HTTP.RetryRequest.FALSE), Base.RefValue{Int64}}, HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(args::HTTP.Messages.Request; kwargs::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64}) │ @ Base error.jl:311 │ [5] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/RetryRequest.jl:75 [inlined] │ [6] (::HTTP.RetryRequest.var"#retrylayer#manageretries##0"{HTTP.RetryRequest.var"#retrylayer##kw_body#manageretries#0##0"{HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_delays::ExponentialBackOff, retry_check::Function, retry_non_idempotent::Bool, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64}) │ @ HTTP.RetryRequest ~/.julia/packages/HTTP/Y97L1/src/clientlayers/RetryRequest.jl:10 │ [7] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/RetryRequest.jl:30 [inlined] │ [8] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/CookieRequest.jl:42 [inlined] │ [9] (::HTTP.CookieRequest.var"#cookielayer#managecookies##0"{HTTP.CookieRequest.var"#cookielayer##kw_body#managecookies#0##0"{HTTP.RetryRequest.var"#retrylayer#manageretries##0"{HTTP.RetryRequest.var"#retrylayer##kw_body#manageretries#0##0"{HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64}) │ @ HTTP.CookieRequest ~/.julia/packages/HTTP/Y97L1/src/clientlayers/CookieRequest.jl:11 │ [10] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/CookieRequest.jl:19 [inlined] │ [11] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/HeadersRequest.jl:71 [inlined] │ [12] (::HTTP.HeadersRequest.var"#headerslayer#defaultheaders##0"{HTTP.HeadersRequest.var"#headerslayer##kw_body#defaultheaders#0##0"{HTTP.CookieRequest.var"#cookielayer#managecookies##0"{HTTP.CookieRequest.var"#cookielayer##kw_body#managecookies#0##0"{HTTP.RetryRequest.var"#retrylayer#manageretries##0"{HTTP.RetryRequest.var"#retrylayer##kw_body#manageretries#0##0"{HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}})(req::HTTP.Messages.Request; iofunction::Nothing, decompress::Nothing, basicauth::Bool, detect_content_type::Bool, canonicalize_headers::Bool, kw::@Kwargs{verbose::Int64}) │ @ HTTP.HeadersRequest ~/.julia/packages/HTTP/Y97L1/src/clientlayers/HeadersRequest.jl:8 │ [13] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/HeadersRequest.jl:14 [inlined] │ [14] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/RedirectRequest.jl:25 [inlined] │ [15] (::HTTP.RedirectRequest.var"#redirectlayer#redirects##0"{HTTP.RedirectRequest.var"#redirectlayer##kw_body#redirects#0##0"{HTTP.HeadersRequest.var"#headerslayer#defaultheaders##0"{HTTP.HeadersRequest.var"#headerslayer##kw_body#defaultheaders#0##0"{HTTP.CookieRequest.var"#cookielayer#managecookies##0"{HTTP.CookieRequest.var"#cookielayer##kw_body#managecookies#0##0"{HTTP.RetryRequest.var"#retrylayer#manageretries##0"{HTTP.RetryRequest.var"#retrylayer##kw_body#manageretries#0##0"{HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Nothing, kw::@Kwargs{verbose::Int64}) │ @ HTTP.RedirectRequest ~/.julia/packages/HTTP/Y97L1/src/clientlayers/RedirectRequest.jl:8 │ [16] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/RedirectRequest.jl:14 [inlined] │ [17] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/MessageRequest.jl:30 [inlined] │ [18] (::HTTP.MessageRequest.var"#messagelayer#makerequest##0"{HTTP.MessageRequest.var"#messagelayer##kw_body#makerequest#0##0"{HTTP.RedirectRequest.var"#redirectlayer#redirects##0"{HTTP.RedirectRequest.var"#redirectlayer##kw_body#redirects#0##0"{HTTP.HeadersRequest.var"#headerslayer#defaultheaders##0"{HTTP.HeadersRequest.var"#headerslayer##kw_body#defaultheaders#0##0"{HTTP.CookieRequest.var"#cookielayer#managecookies##0"{HTTP.CookieRequest.var"#cookielayer##kw_body#managecookies#0##0"{HTTP.RetryRequest.var"#retrylayer#manageretries##0"{HTTP.RetryRequest.var"#retrylayer##kw_body#manageretries#0##0"{HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}})(method::String, url::URIs.URI, headers::Nothing, body::Vector{UInt8}; copyheaders::Bool, response_stream::Nothing, http_version::HTTP.Strings.HTTPVersion, verbose::Int64, kw::@Kwargs{}) │ @ HTTP.MessageRequest ~/.julia/packages/HTTP/Y97L1/src/clientlayers/MessageRequest.jl:17 │ [19] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/MessageRequest.jl:24 [inlined] │ [20] (::HTTP.MessageRequest.var"#messagelayer#makerequest##0"{HTTP.MessageRequest.var"#messagelayer##kw_body#makerequest#0##0"{HTTP.RedirectRequest.var"#redirectlayer#redirects##0"{HTTP.RedirectRequest.var"#redirectlayer##kw_body#redirects#0##0"{HTTP.HeadersRequest.var"#headerslayer#defaultheaders##0"{HTTP.HeadersRequest.var"#headerslayer##kw_body#defaultheaders#0##0"{HTTP.CookieRequest.var"#cookielayer#managecookies##0"{HTTP.CookieRequest.var"#cookielayer##kw_body#managecookies#0##0"{HTTP.RetryRequest.var"#retrylayer#manageretries##0"{HTTP.RetryRequest.var"#retrylayer##kw_body#manageretries#0##0"{HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}})(method::String, url::URIs.URI, headers::Nothing, body::Vector{UInt8}) │ @ HTTP.MessageRequest ~/.julia/packages/HTTP/Y97L1/src/clientlayers/MessageRequest.jl:17 │ [21] request(stack::Function, method::String, url::String, h::Nothing, b::Vector{UInt8}, q::Nothing; headers::Nothing, body::Vector{UInt8}, query::Nothing, kw::@Kwargs{}) │ @ HTTP ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:463 │ [22] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:321 [inlined] │ [23] request(method::String, url::String, h::Nothing, b::Vector{UInt8}; headers::Nothing, body::Vector{UInt8}, query::Nothing, observelayers::Bool, kw::@Kwargs{}) │ @ HTTP ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:88 [inlined] │ [24] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:321 [inlined] │ [25] request(method::String, url::String, h::Nothing, b::Vector{UInt8}) │ @ HTTP ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:88 [inlined] │ [26] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:321 [inlined] │ [27] request(method::String, url::String) │ @ HTTP ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:88 [inlined] │ [28] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:526 [inlined] │ [29] get(a::String; kw::@Kwargs{}) │ @ HTTP ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:521 [inlined] │ [30] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:526 [inlined] │ [31] get(a::String) │ @ HTTP ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:521 [inlined] │ [32] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/precompile.jl:36 [inlined] │ [33] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/precompile.jl:34 [inlined] │ [34] (::HTTP.var"##->###3"{String})() │ @ HTTP ~/.julia/packages/HTTP/Y97L1/src/precompile.jl:4 [inlined] │ [35] withenv(::HTTP.var"##->###3"{String}, ::Pair{String, Union{Nothing, String}}, ::Vararg{Pair{String, Union{Nothing, String}}}) │ @ Base env.jl:265 │ [36] macro expansion │ @ ~/.julia/packages/PrecompileTools/QUxvR/src/workloads.jl:75 [inlined] │ [37] macro expansion │ @ ~/.julia/packages/HTTP/Y97L1/src/precompile.jl:34 [inlined] │ [38] top-level scope │ @ ~/.julia/packages/HTTP/Y97L1/src/precompile.jl:4 │ [39] include(mapexpr::Function, mod::Module, _path::String) │ @ Base Base.jl:327 │ [40] top-level scope │ @ ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:651 │ [41] include(mod::Module, _path::String) │ @ Base Base.jl:326 │ [42] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing) │ @ Base loading.jl:3296 │ [43] top-level scope │ @ stdin:5 │ [44] eval(m::Module, e::Any) │ @ Core boot.jl:521 │ [45] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) │ @ Base loading.jl:3132 │ [46] include_string(m::Module, txt::String, fname::String) │ @ Base loading.jl:3142 [inlined] │ [47] exec_options(opts::Base.JLOptions) │ @ Base client.jl:353 │ [48] _start() └ @ Base client.jl:596 149.5 s ✓ HTTP 2.5 s ✓ Git 51.9 s ✓ StreamCallbacks 67.1 s ✓ OpenAI 228.8 s ✓ Documenter ┌ Warning: OPENAI_API_KEY variable not set! OpenAI models will not be available - set API key directly via `PromptingTools.OPENAI_API_KEY=`! └ @ PromptingTools ~/.julia/packages/PromptingTools/wvgCI/src/user_preferences.jl:189 ┌ Info: JuliaLowering threw given input: │ code = │ :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947 =# Core.@doc " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" function retrieve(retriever::AbstractRetriever, index::AbstractDocumentIndex, question::AbstractString; verbose::Integer = 1, top_k::Integer = 100, top_n::Integer = 5, api_kwargs::NamedTuple = NamedTuple(), rephraser::AbstractRephraser = retriever.rephraser, rephraser_kwargs::NamedTuple = NamedTuple(), embedder::AbstractEmbedder = retriever.embedder, embedder_kwargs::NamedTuple = NamedTuple(), processor::AbstractProcessor = retriever.processor, processor_kwargs::NamedTuple = NamedTuple(), finder::AbstractSimilarityFinder = retriever.finder, finder_kwargs::NamedTuple = NamedTuple(), tagger::AbstractTagger = retriever.tagger, tagger_kwargs::NamedTuple = NamedTuple(), filter::AbstractTagFilter = retriever.filter, filter_kwargs::NamedTuple = NamedTuple(), reranker::AbstractReranker = retriever.reranker, reranker_kwargs::NamedTuple = NamedTuple(), cost_tracker = Threads.Atomic{Float64}(0.0), kwargs...) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1059 =# │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1083 =# │ rephraser_kwargs_ = if isempty(api_kwargs) │ rephraser_kwargs │ else │ merge(rephraser_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1085 =# │ rephrased_questions = rephrase(rephraser, question; verbose = verbose > 1, cost_tracker, rephraser_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1089 =# │ embeddings = if HasEmbeddings(index) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1090 =# │ embedder_kwargs_ = if isempty(api_kwargs) │ embedder_kwargs │ else │ merge(embedder_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1092 =# │ embeddings = get_embeddings(embedder, rephrased_questions; verbose = verbose > 1, cost_tracker, embedder_kwargs_...) │ else │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1095 =# │ embeddings = hcat([Float32[] for x = rephrased_questions]...) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1099 =# │ keywords = if HasKeywords(index) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1101 =# │ keywords = get_keywords(processor, rephrased_questions; verbose = verbose > 1, processor_kwargs..., return_keywords = true) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1104 =# │ verbose >= 1 && (keywords isa AbstractVector{<:AbstractVector{<:AbstractString}} || #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1105 =# @warn("Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: $(typeof(processor))).")) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1106 =# │ keywords │ else │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1108 =# │ [String[] for x = rephrased_questions] │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1111 =# │ finder_kwargs_ = if isempty(api_kwargs) │ finder_kwargs │ else │ merge(finder_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1113 =# │ emb_candidates = find_closest(finder, index, embeddings, keywords; verbose = verbose > 1, top_k, finder_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1117 =# │ tagger_kwargs_ = if isempty(api_kwargs) │ tagger_kwargs │ else │ merge(tagger_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1119 =# │ tags = get_tags(tagger, rephrased_questions; verbose = verbose > 1, cost_tracker, tagger_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1122 =# │ filter_kwargs_ = if isempty(api_kwargs) │ filter_kwargs │ else │ merge(filter_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1124 =# │ tag_candidates = find_tags(filter, index, tags; verbose = verbose > 1, filter_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1129 =# │ filtered_candidates = if isnothing(tag_candidates) │ emb_candidates │ else │ emb_candidates & tag_candidates │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1136 =# │ reranker_kwargs_ = if isempty(api_kwargs) │ reranker_kwargs │ else │ merge(reranker_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1138 =# │ reranked_candidates = rerank(reranker, index, question, filtered_candidates; top_n, verbose = verbose > 1, cost_tracker, reranker_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1141 =# │ verbose > 0 && #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1142 =# @info("Retrieval done. Identified $(length(positions(reranked_candidates))) chunks, total cost: \$$(round(cost_tracker[], digits = 2)).") │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1145 =# │ result = RAGResult(; question, answer = nothing, rephrased_questions, final_answer = nothing, context = collect(index[reranked_candidates, :chunks, sorted = true]), sources = collect(index[reranked_candidates, :sources, sorted = true]), emb_candidates, tag_candidates, filtered_candidates, reranked_candidates) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1158 =# │ return result │ end) │ st0 = │ SyntaxTree with attributes mod,kind,var_id,toplevel_pure,scope_type,macro_source,name_val,syntax_flags,meta,scope_layer,value,jl_source,is_toplevel_thunk,source,__macro_ctx__ │ [macrocall] │ │ @doc :: Identifier │ mod │ :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947 =#) :: Value │ │ " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" :: Value │ │ [function] │ │ [call] │ │ retrieve :: Identifier │ │ [parameters] │ │ [kw] │ │ [::] │ │ verbose :: Identifier │ │ Integer :: Identifier │ │ 1 :: Value │ │ [kw] │ │ [::] │ │ top_k :: Identifier │ │ Integer :: Identifier │ │ 100 :: Value │ │ [kw] │ │ [::] │ │ top_n :: Identifier │ │ Integer :: Identifier │ │ 5 :: Value │ │ [kw] │ │ [::] │ │ api_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ rephraser :: Identifier │ │ AbstractRephraser :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ rephraser :: Identifier │ │ [kw] │ │ [::] │ │ rephraser_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ embedder :: Identifier │ │ AbstractEmbedder :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ embedder :: Identifier │ │ [kw] │ │ [::] │ │ embedder_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ processor :: Identifier │ │ AbstractProcessor :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ processor :: Identifier │ │ [kw] │ │ [::] │ │ processor_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ finder :: Identifier │ │ AbstractSimilarityFinder :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ finder :: Identifier │ │ [kw] │ │ [::] │ │ finder_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ tagger :: Identifier │ │ AbstractTagger :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ tagger :: Identifier │ │ [kw] │ │ [::] │ │ tagger_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ filter :: Identifier │ │ AbstractTagFilter :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ filter :: Identifier │ │ [kw] │ │ [::] │ │ filter_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ reranker :: Identifier │ │ AbstractReranker :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ reranker :: Identifier │ │ [kw] │ │ [::] │ │ reranker_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ cost_tracker :: Identifier │ │ [call] │ │ [curly] │ │ [.] │ │ Threads :: Identifier │ │ [inert] │ │ Atomic :: Identifier │ │ Float64 :: Identifier │ │ 0.0 :: Value │ │ [...] │ │ kwargs :: Identifier │ │ [::] │ │ retriever :: Identifier │ │ AbstractRetriever :: Identifier │ │ [::] │ │ index :: Identifier │ │ AbstractDocumentIndex :: Identifier │ │ [::] │ │ question :: Identifier │ │ AbstractString :: Identifier │ │ [block] │ │ [=] │ │ rephraser_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ rephraser_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ rephraser_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ rephrased_questions :: Identifier │ │ [call] │ │ rephrase :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ rephraser_kwargs_ :: Identifier │ │ rephraser :: Identifier │ │ question :: Identifier │ │ [=] │ │ embeddings :: Identifier │ │ [if] │ │ [call] │ │ HasEmbeddings :: Identifier │ │ index :: Identifier │ │ [block] │ │ [=] │ │ embedder_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ embedder_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ embedder_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ embeddings :: Identifier │ │ [call] │ │ get_embeddings :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ embedder_kwargs_ :: Identifier │ │ embedder :: Identifier │ │ rephrased_questions :: Identifier │ │ [block] │ │ [=] │ │ embeddings :: Identifier │ │ [call] │ │ hcat :: Identifier │ │ [...] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ Float32 :: Identifier │ │ [=] │ │ x :: Identifier │ │ rephrased_questions :: Identifier │ │ [=] │ │ keywords :: Identifier │ │ [if] │ │ [call] │ │ HasKeywords :: Identifier │ │ index :: Identifier │ │ [block] │ │ [=] │ │ keywords :: Identifier │ │ [call] │ │ get_keywords :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ [...] │ │ processor_kwargs :: Identifier │ │ [kw] │ │ return_keywords :: Identifier │ │ true :: Value │ │ processor :: Identifier │ │ rephrased_questions :: Identifier │ │ [&&] │ │ [call] │ │ >= :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ [||] │ │ [call] │ │ isa :: Identifier │ │ keywords :: Identifier │ │ [curly] │ │ AbstractVector :: Identifier │ │ [<:] │ │ [curly] │ │ AbstractVector :: Identifier │ │ [<:] │ │ AbstractString :: Identifier │ │ [macrocall] │ │ @warn :: Identifier │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1105 =#) :: Value │ │ [string] │ │ "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " :: Value │ │ [call] │ │ typeof :: Identifier │ │ processor :: Identifier │ │ ")." :: Value │ │ keywords :: Identifier │ │ [block] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ String :: Identifier │ │ [=] │ │ x :: Identifier │ │ rephrased_questions :: Identifier │ │ [=] │ │ finder_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ finder_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ finder_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ emb_candidates :: Identifier │ │ [call] │ │ find_closest :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ top_k :: Identifier │ │ [...] │ │ finder_kwargs_ :: Identifier │ │ finder :: Identifier │ │ index :: Identifier │ │ embeddings :: Identifier │ │ keywords :: Identifier │ │ [=] │ │ tagger_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ tagger_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ tagger_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ tags :: Identifier │ │ [call] │ │ get_tags :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ tagger_kwargs_ :: Identifier │ │ tagger :: Identifier │ │ rephrased_questions :: Identifier │ │ [=] │ │ filter_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ filter_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ filter_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ tag_candidates :: Identifier │ │ [call] │ │ find_tags :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ [...] │ │ filter_kwargs_ :: Identifier │ │ filter :: Identifier │ │ index :: Identifier │ │ tags :: Identifier │ │ [=] │ │ filtered_candidates :: Identifier │ │ [if] │ │ [call] │ │ isnothing :: Identifier │ │ tag_candidates :: Identifier │ │ emb_candidates :: Identifier │ │ [call] │ │ & :: Identifier │ │ emb_candidates :: Identifier │ │ tag_candidates :: Identifier │ │ [=] │ │ reranker_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ reranker_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ reranker_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ reranked_candidates :: Identifier │ │ [call] │ │ rerank :: Identifier │ │ [parameters] │ │ top_n :: Identifier │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ reranker_kwargs_ :: Identifier │ │ reranker :: Identifier │ │ index :: Identifier │ │ question :: Identifier │ │ filtered_candidates :: Identifier │ │ [&&] │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 0 :: Value │ │ [macrocall] │ │ @info :: Identifier │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1142 =#) :: Value │ │ [string] │ │ "Retrieval done. Identified " :: Value │ │ [call] │ │ length :: Identifier │ │ [call] │ │ positions :: Identifier │ │ reranked_candidates :: Identifier │ │ " chunks, total cost: \$" :: Value │ │ [call] │ │ round :: Identifier │ │ [ref] │ │ cost_tracker :: Identifier │ │ [kw] │ │ digits :: Identifier │ │ 2 :: Value │ │ "." :: Value │ │ [=] │ │ result :: Identifier │ │ [call] │ │ RAGResult :: Identifier │ │ [parameters] │ │ question :: Identifier │ │ [kw] │ │ answer :: Identifier │ │ nothing :: Identifier │ │ rephrased_questions :: Identifier │ │ [kw] │ │ final_answer :: Identifier │ │ nothing :: Identifier │ │ [kw] │ │ context :: Identifier │ │ [call] │ │ collect :: Identifier │ │ [ref] │ │ index :: Identifier │ │ reranked_candidates :: Identifier │ │ [inert] │ │ chunks :: Identifier │ │ [kw] │ │ sorted :: Identifier │ │ true :: Value │ │ [kw] │ │ sources :: Identifier │ │ [call] │ │ collect :: Identifier │ │ [ref] │ │ index :: Identifier │ │ reranked_candidates :: Identifier │ │ [inert] │ │ sources :: Identifier │ │ [kw] │ │ sorted :: Identifier │ │ true :: Value │ │ emb_candidates :: Identifier │ │ tag_candidates :: Identifier │ │ filtered_candidates :: Identifier │ │ reranked_candidates :: Identifier │ │ [return] │ │ result :: Identifier │ │ │ st1 = │ SyntaxTree with attributes mod,kind,var_id,toplevel_pure,scope_type,macro_source,name_val,syntax_flags,meta,scope_layer,value,jl_source,is_toplevel_thunk,source │ [block] │ │ [=] │ │ val :: Identifier │ scope_layer=3 │ [function] │ │ [call] │ │ retrieve :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ [::] │ │ verbose :: Identifier │ scope_layer=1 │ Integer :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [kw] │ │ [::] │ │ top_k :: Identifier │ scope_layer=1 │ Integer :: Identifier │ scope_layer=1 │ 100 :: Value │ macro_source=485 │ [kw] │ │ [::] │ │ top_n :: Identifier │ scope_layer=1 │ Integer :: Identifier │ scope_layer=1 │ 5 :: Value │ macro_source=485 │ [kw] │ │ [::] │ │ api_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ rephraser :: Identifier │ scope_layer=1 │ AbstractRephraser :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ rephraser :: Identifier │ │ [kw] │ │ [::] │ │ rephraser_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ embedder :: Identifier │ scope_layer=1 │ AbstractEmbedder :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ embedder :: Identifier │ │ [kw] │ │ [::] │ │ embedder_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ processor :: Identifier │ scope_layer=1 │ AbstractProcessor :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ processor :: Identifier │ │ [kw] │ │ [::] │ │ processor_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ finder :: Identifier │ scope_layer=1 │ AbstractSimilarityFinder :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ finder :: Identifier │ │ [kw] │ │ [::] │ │ finder_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ tagger :: Identifier │ scope_layer=1 │ AbstractTagger :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ tagger :: Identifier │ │ [kw] │ │ [::] │ │ tagger_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ filter :: Identifier │ scope_layer=1 │ AbstractTagFilter :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ filter :: Identifier │ │ [kw] │ │ [::] │ │ filter_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ reranker :: Identifier │ scope_layer=1 │ AbstractReranker :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ reranker :: Identifier │ │ [kw] │ │ [::] │ │ reranker_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ cost_tracker :: Identifier │ scope_layer=1 │ [call] │ │ [curly] │ │ [.] │ │ Threads :: Identifier │ scope_layer=1 │ [inert] │ │ Atomic :: Identifier │ │ Float64 :: Identifier │ scope_layer=1 │ 0.0 :: Value │ macro_source=485 │ [...] │ │ kwargs :: Identifier │ scope_layer=1 │ [::] │ │ retriever :: Identifier │ scope_layer=1 │ AbstractRetriever :: Identifier │ scope_layer=1 │ [::] │ │ index :: Identifier │ scope_layer=1 │ AbstractDocumentIndex :: Identifier │ scope_layer=1 │ [::] │ │ question :: Identifier │ scope_layer=1 │ AbstractString :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ rephraser_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ rephraser_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ rephraser_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ rephrased_questions :: Identifier │ scope_layer=1 │ [call] │ │ rephrase :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ rephraser_kwargs_ :: Identifier │ scope_layer=1 │ rephraser :: Identifier │ scope_layer=1 │ question :: Identifier │ scope_layer=1 │ [=] │ │ embeddings :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ HasEmbeddings :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ embedder_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ embedder_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ embedder_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ embeddings :: Identifier │ scope_layer=1 │ [call] │ │ get_embeddings :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ embedder_kwargs_ :: Identifier │ scope_layer=1 │ embedder :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ embeddings :: Identifier │ scope_layer=1 │ [call] │ │ hcat :: Identifier │ scope_layer=1 │ [...] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ Float32 :: Identifier │ scope_layer=1 │ [=] │ │ x :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [=] │ │ keywords :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ HasKeywords :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ keywords :: Identifier │ scope_layer=1 │ [call] │ │ get_keywords :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [...] │ │ processor_kwargs :: Identifier │ scope_layer=1 │ [kw] │ │ return_keywords :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ processor :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [&&] │ │ [call] │ │ >= :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [||] │ │ [call] │ │ isa :: Identifier │ scope_layer=1 │ keywords :: Identifier │ scope_layer=1 │ [curly] │ │ AbstractVector :: Identifier │ scope_layer=1 │ [<:] │ scope_layer=1 │ [curly] │ │ AbstractVector :: Identifier │ scope_layer=1 │ [<:] │ scope_layer=1 │ AbstractString :: Identifier │ scope_layer=1 │ [block] │ │ [let] │ │ [block] │ macro_source=485 │ [block] │ │ [=] │ │ #1281#level :: Identifier │ scope_layer=1 │ Warn :: Identifier │ mod,scope_layer=1 │ [=] │ │ #1282#std_level :: Identifier │ scope_layer=1 │ #1281#level :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ >= :: Identifier │ mod,scope_layer=1 │ [.] │ │ #1282#std_level :: Identifier │ scope_layer=1 │ [inert] │ │ level :: Identifier │ │ [ref] │ macro_source=485 │ Base.Threads.Atomic{Int32}(-1000) :: Value │ macro_source=485 │ [block] │ │ [=] │ │ #1283#group :: Identifier │ scope_layer=1 │ [inert] │ │ retrieval :: Identifier │ │ [=] │ │ #1284#_module :: Identifier │ scope_layer=1 │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [=] │ │ #1285#logger :: Identifier │ scope_layer=1 │ [call] │ │ Base.CoreLogging.current_logger_for_env :: Value │ macro_source=485 │ #1282#std_level :: Identifier │ scope_layer=1 │ #1283#group :: Identifier │ scope_layer=1 │ #1284#_module :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ ! :: Identifier │ mod,scope_layer=1 │ [call] │ │ === :: Identifier │ mod,scope_layer=1 │ #1285#logger :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1286#id :: Identifier │ scope_layer=1 │ [inert] │ │ PromptingTools_Experimental_RAGTools_7aa4399b :: Identifier │ │ [if] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.shouldlog :: Value │ macro_source=485 │ #1285#logger :: Identifier │ scope_layer=1 │ #1281#level :: Identifier │ scope_layer=1 │ #1284#_module :: Identifier │ scope_layer=1 │ #1283#group :: Identifier │ scope_layer=1 │ #1286#id :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ #1287#file :: Identifier │ scope_layer=1 │ "/home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ [if] │ │ [call] │ │ isa :: Identifier │ mod,scope_layer=1 │ #1287#file :: Identifier │ scope_layer=1 │ String :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1287#file :: Identifier │ scope_layer=1 │ [call] │ │ [.] │ │ Base :: Identifier │ mod,scope_layer=1 │ [inert] │ │ fixup_stdlib_path :: Identifier │ │ #1287#file :: Identifier │ scope_layer=1 │ [=] │ │ #1288#line :: Identifier │ scope_layer=1 │ 1105 :: Value │ macro_source=485 │ [local] │ │ #1289#msg :: Identifier │ scope_layer=1 │ #1290#kwargs :: Identifier │ scope_layer=1 │ [if] │ │ [block] │ │ [try] │ │ [block] │ │ [=] │ │ #1289#msg :: Identifier │ scope_layer=1 │ [string] │ │ "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " :: Value │ macro_source=485 │ [call] │ │ typeof :: Identifier │ scope_layer=1 │ processor :: Identifier │ scope_layer=1 │ ")." :: Value │ macro_source=485 │ [=] │ │ #1290#kwargs :: Identifier │ scope_layer=1 │ [tuple] │ macro_source=485 │ [parameters] │ macro_source=485 │ true :: Value │ macro_source=485 │ #1303#err :: Identifier │ scope_layer=1 │ [block] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.logging_error :: Value │ macro_source=485 │ #1285#logger :: Identifier │ scope_layer=1 │ #1281#level :: Identifier │ scope_layer=1 │ #1284#_module :: Identifier │ scope_layer=1 │ #1283#group :: Identifier │ scope_layer=1 │ #1286#id :: Identifier │ scope_layer=1 │ #1287#file :: Identifier │ scope_layer=1 │ #1288#line :: Identifier │ scope_layer=1 │ #1303#err :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ false :: Value │ macro_source=485 │ [block] │ │ [if] │ │ [isdefined] │ │ #1289#msg :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [if] │ │ [isdefined] │ │ #1290#kwargs :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [call] │ │ Base.CoreLogging.handle_message_nothrow :: Value │ macro_source=485 │ [parameters] │ │ [...] │ │ #1290#kwargs :: Identifier │ scope_layer=1 │ #1285#logger :: Identifier │ scope_layer=1 │ #1281#level :: Identifier │ scope_layer=1 │ #1289#msg :: Identifier │ scope_layer=1 │ #1284#_module :: Identifier │ scope_layer=1 │ #1283#group :: Identifier │ scope_layer=1 │ #1286#id :: Identifier │ scope_layer=1 │ #1287#file :: Identifier │ scope_layer=1 │ #1288#line :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ keywords :: Identifier │ scope_layer=1 │ [block] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ String :: Identifier │ scope_layer=1 │ [=] │ │ x :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [=] │ │ finder_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ finder_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ finder_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ emb_candidates :: Identifier │ scope_layer=1 │ [call] │ │ find_closest :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ top_k :: Identifier │ scope_layer=1 │ [...] │ │ finder_kwargs_ :: Identifier │ scope_layer=1 │ finder :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ embeddings :: Identifier │ scope_layer=1 │ keywords :: Identifier │ scope_layer=1 │ [=] │ │ tagger_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ tagger_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ tagger_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ tags :: Identifier │ scope_layer=1 │ [call] │ │ get_tags :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ tagger_kwargs_ :: Identifier │ scope_layer=1 │ tagger :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [=] │ │ filter_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ filter_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ filter_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ tag_candidates :: Identifier │ scope_layer=1 │ [call] │ │ find_tags :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [...] │ │ filter_kwargs_ :: Identifier │ scope_layer=1 │ filter :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ tags :: Identifier │ scope_layer=1 │ [=] │ │ filtered_candidates :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isnothing :: Identifier │ scope_layer=1 │ tag_candidates :: Identifier │ scope_layer=1 │ emb_candidates :: Identifier │ scope_layer=1 │ [call] │ │ & :: Identifier │ scope_layer=1 │ emb_candidates :: Identifier │ scope_layer=1 │ tag_candidates :: Identifier │ scope_layer=1 │ [=] │ │ reranker_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ reranker_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ reranker_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ reranked_candidates :: Identifier │ scope_layer=1 │ [call] │ │ rerank :: Identifier │ scope_layer=1 │ [parameters] │ │ top_n :: Identifier │ scope_layer=1 │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ reranker_kwargs_ :: Identifier │ scope_layer=1 │ reranker :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ question :: Identifier │ scope_layer=1 │ filtered_candidates :: Identifier │ scope_layer=1 │ [&&] │ │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 0 :: Value │ macro_source=485 │ [block] │ │ [let] │ │ [block] │ macro_source=485 │ [block] │ │ [=] │ │ #1304#level :: Identifier │ scope_layer=1 │ Info :: Identifier │ mod,scope_layer=1 │ [=] │ │ #1305#std_level :: Identifier │ scope_layer=1 │ #1304#level :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ >= :: Identifier │ mod,scope_layer=1 │ [.] │ │ #1305#std_level :: Identifier │ scope_layer=1 │ [inert] │ │ level :: Identifier │ │ [ref] │ macro_source=485 │ Base.Threads.Atomic{Int32}(-1000) :: Value │ macro_source=485 │ [block] │ │ [=] │ │ #1306#group :: Identifier │ scope_layer=1 │ [inert] │ │ retrieval :: Identifier │ │ [=] │ │ #1307#_module :: Identifier │ scope_layer=1 │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [=] │ │ #1308#logger :: Identifier │ scope_layer=1 │ [call] │ │ Base.CoreLogging.current_logger_for_env :: Value │ macro_source=485 │ #1305#std_level :: Identifier │ scope_layer=1 │ #1306#group :: Identifier │ scope_layer=1 │ #1307#_module :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ ! :: Identifier │ mod,scope_layer=1 │ [call] │ │ === :: Identifier │ mod,scope_layer=1 │ #1308#logger :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1309#id :: Identifier │ scope_layer=1 │ [inert] │ │ PromptingTools_Experimental_RAGTools_30253e51 :: Identifier │ │ [if] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.shouldlog :: Value │ macro_source=485 │ #1308#logger :: Identifier │ scope_layer=1 │ #1304#level :: Identifier │ scope_layer=1 │ #1307#_module :: Identifier │ scope_layer=1 │ #1306#group :: Identifier │ scope_layer=1 │ #1309#id :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ #1310#file :: Identifier │ scope_layer=1 │ "/home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ [if] │ │ [call] │ │ isa :: Identifier │ mod,scope_layer=1 │ #1310#file :: Identifier │ scope_layer=1 │ String :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1310#file :: Identifier │ scope_layer=1 │ [call] │ │ [.] │ │ Base :: Identifier │ mod,scope_layer=1 │ [inert] │ │ fixup_stdlib_path :: Identifier │ │ #1310#file :: Identifier │ scope_layer=1 │ [=] │ │ #1311#line :: Identifier │ scope_layer=1 │ 1142 :: Value │ macro_source=485 │ [local] │ │ #1312#msg :: Identifier │ scope_layer=1 │ #1313#kwargs :: Identifier │ scope_layer=1 │ [if] │ │ [block] │ │ [try] │ │ [block] │ │ [=] │ │ #1312#msg :: Identifier │ scope_layer=1 │ [string] │ │ "Retrieval done. Identified " :: Value │ macro_source=485 │ [call] │ │ length :: Identifier │ scope_layer=1 │ [call] │ │ positions :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ " chunks, total cost: \$" :: Value │ macro_source=485 │ [call] │ │ round :: Identifier │ scope_layer=1 │ [ref] │ │ cost_tracker :: Identifier │ scope_layer=1 │ [kw] │ │ digits :: Identifier │ scope_layer=1 │ 2 :: Value │ macro_source=485 │ "." :: Value │ macro_source=485 │ [=] │ │ #1313#kwargs :: Identifier │ scope_layer=1 │ [tuple] │ macro_source=485 │ [parameters] │ macro_source=485 │ true :: Value │ macro_source=485 │ #1326#err :: Identifier │ scope_layer=1 │ [block] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.logging_error :: Value │ macro_source=485 │ #1308#logger :: Identifier │ scope_layer=1 │ #1304#level :: Identifier │ scope_layer=1 │ #1307#_module :: Identifier │ scope_layer=1 │ #1306#group :: Identifier │ scope_layer=1 │ #1309#id :: Identifier │ scope_layer=1 │ #1310#file :: Identifier │ scope_layer=1 │ #1311#line :: Identifier │ scope_layer=1 │ #1326#err :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ false :: Value │ macro_source=485 │ [block] │ │ [if] │ │ [isdefined] │ │ #1312#msg :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [if] │ │ [isdefined] │ │ #1313#kwargs :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [call] │ │ Base.CoreLogging.handle_message_nothrow :: Value │ macro_source=485 │ [parameters] │ │ [...] │ │ #1313#kwargs :: Identifier │ scope_layer=1 │ #1308#logger :: Identifier │ scope_layer=1 │ #1304#level :: Identifier │ scope_layer=1 │ #1312#msg :: Identifier │ scope_layer=1 │ #1307#_module :: Identifier │ scope_layer=1 │ #1306#group :: Identifier │ scope_layer=1 │ #1309#id :: Identifier │ scope_layer=1 │ #1310#file :: Identifier │ scope_layer=1 │ #1311#line :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ [=] │ │ result :: Identifier │ scope_layer=1 │ [call] │ │ RAGResult :: Identifier │ scope_layer=1 │ [parameters] │ │ question :: Identifier │ scope_layer=1 │ [kw] │ │ answer :: Identifier │ scope_layer=1 │ nothing :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [kw] │ │ final_answer :: Identifier │ scope_layer=1 │ nothing :: Identifier │ scope_layer=1 │ [kw] │ │ context :: Identifier │ scope_layer=1 │ [call] │ │ collect :: Identifier │ scope_layer=1 │ [ref] │ │ index :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ [inert] │ │ chunks :: Identifier │ │ [kw] │ │ sorted :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ [kw] │ │ sources :: Identifier │ scope_layer=1 │ [call] │ │ collect :: Identifier │ scope_layer=1 │ [ref] │ │ index :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ [inert] │ │ sources :: Identifier │ │ [kw] │ │ sorted :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ emb_candidates :: Identifier │ scope_layer=1 │ tag_candidates :: Identifier │ scope_layer=1 │ filtered_candidates :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ [return] │ │ result :: Identifier │ scope_layer=1 │ [call] │ │ Base.Docs.doc! :: Value │ macro_source=485 │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [call] │ │ Base.Docs.Binding :: Value │ macro_source=485 │ PromptingTools.Experimental.RAGTools :: Value │ │ [inert] │ jl_source=L65 │ retrieve :: Identifier │ │ [call] │ macro_source=485 │ Base.Docs.docstr :: Value │ macro_source=485 │ [call] │ macro_source=485 │ Core.svec :: Value │ macro_source=485 │ " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" :: Value │ macro_source=485 │ [call] │ macro_source=485 │ Dict{Symbol, Any} :: Value │ macro_source=485 │ :path => "/home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ :linenumber => 947 :: Value │ macro_source=485 │ :module => PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [curly] │ │ Union :: Identifier │ scope_layer=1 │ [curly] │ │ Tuple :: Identifier │ scope_layer=1 │ AbstractRetriever :: Identifier │ scope_layer=1 │ AbstractDocumentIndex :: Identifier │ scope_layer=1 │ AbstractString :: Identifier │ scope_layer=1 │ val :: Identifier │ scope_layer=3 │ │ file = "/home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl" │ line = 947 └ mod = PromptingTools.Experimental.RAGTools ERROR: LoadError: LoweringError: #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1145 =# - invalid syntax: unknown form `kw` or number of arguments 2 Expression:  (kw sorted true) Containing expressions:  (kw context (call collect (ref index reranked_candidates (inert chunks) (kw sorted true))))  Detailed provenance:  (kw sorted true)  └─ (kw sorted true)  ├─ @ /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1145  └─ (macrocall @doc :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947 =#) " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" (function (call retrieve (parameters (kw (:: verbose Integer) 1) (kw (:: top_k Integer) 100) (kw (:: top_n Integer) 5) (kw (:: api_kwargs NamedTuple) (call NamedTuple)) (kw (:: rephraser AbstractRephraser) (. retriever (inert rephraser))) (kw (:: rephraser_kwargs NamedTuple) (call NamedTuple)) (kw (:: embedder AbstractEmbedder) (. retriever (inert embedder))) (kw (:: embedder_kwargs NamedTuple) (call NamedTuple)) (kw (:: processor AbstractProcessor) (. retriever (inert processor))) (kw (:: processor_kwargs NamedTuple) (call NamedTuple)) (kw (:: finder AbstractSimilarityFinder) (. retriever (inert finder))) (kw (:: finder_kwargs NamedTuple) (call NamedTuple)) (kw (:: tagger AbstractTagger) (. retriever (inert tagger))) (kw (:: tagger_kwargs NamedTuple) (call NamedTuple)) (kw (:: filter AbstractTagFilter) (. retriever (inert filter))) (kw (:: filter_kwargs NamedTuple) (call NamedTuple)) (kw (:: reranker AbstractReranker) (. retriever (inert reranker))) (kw (:: reranker_kwargs NamedTuple) (call NamedTuple)) (kw cost_tracker (call (curly (. Threads (inert Atomic)) Float64) 0.0)) (... kwargs)) (:: retriever AbstractRetriever) (:: index AbstractDocumentIndex) (:: question AbstractString)) (block (= rephraser_kwargs_ (if (call isempty api_kwargs) rephraser_kwargs (call merge rephraser_kwargs (tuple (parameters api_kwargs))))) (= rephrased_questions (call rephrase (parameters (kw verbose (call > verbose 1)) cost_tracker (... rephraser_kwargs_)) rephraser question)) (= embeddings (if (call HasEmbeddings index) (block (= embedder_kwargs_ (if (call isempty api_kwargs) embedder_kwargs (call merge embedder_kwargs (tuple (parameters api_kwargs))))) (= embeddings (call get_embeddings (parameters (kw verbose (call > verbose 1)) cost_tracker (... embedder_kwargs_)) embedder rephrased_questions))) (block (= embeddings (call hcat (... (comprehension (generator (ref Float32) (= x rephrased_questions))))))))) (= keywords (if (call HasKeywords index) (block (= keywords (call get_keywords (parameters (kw verbose (call > verbose 1)) (... processor_kwargs) (kw return_keywords true)) processor rephrased_questions)) (&& (call >= verbose 1) (|| (call isa keywords (curly AbstractVector (<: (curly AbstractVector (<: AbstractString))))) (macrocall @warn :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1105 =#) (string "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " (call typeof processor) ").")))) keywords) (block (comprehension (generator (ref String) (= x rephrased_questions)))))) (= finder_kwargs_ (if (call isempty api_kwargs) finder_kwargs (call merge finder_kwargs (tuple (parameters api_kwargs))))) (= emb_candidates (call find_closest (parameters (kw verbose (call > verbose 1)) top_k (... finder_kwargs_)) finder index embeddings keywords)) (= tagger_kwargs_ (if (call isempty api_kwargs) tagger_kwargs (call merge tagger_kwargs (tuple (parameters api_kwargs))))) (= tags (call get_tags (parameters (kw verbose (call > verbose 1)) cost_tracker (... tagger_kwargs_)) tagger rephrased_questions)) (= filter_kwargs_ (if (call isempty api_kwargs) filter_kwargs (call merge filter_kwargs (tuple (parameters api_kwargs))))) (= tag_candidates (call find_tags (parameters (kw verbose (call > verbose 1)) (... filter_kwargs_)) filter index tags)) (= filtered_candidates (if (call isnothing tag_candidates) emb_candidates (call & emb_candidates tag_candidates))) (= reranker_kwargs_ (if (call isempty api_kwargs) reranker_kwargs (call merge reranker_kwargs (tuple (parameters api_kwargs))))) (= reranked_candidates (call rerank (parameters top_n (kw verbose (call > verbose 1)) cost_tracker (... reranker_kwargs_)) reranker index question filtered_candidates)) (&& (call > verbose 0) (macrocall @info :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1142 =#) (string "Retrieval done. Identified " (call length (call positions reranked_candidates)) " chunks, total cost: \$" (call round (ref cost_tracker) (kw digits 2)) "."))) (= result (call RAGResult (parameters question (kw answer nothing) rephrased_questions (kw final_answer nothing) (kw context (call collect (ref index reranked_candidates (inert chunks) (kw sorted true)))) (kw sources (call collect (ref index reranked_candidates (inert sources) (kw sorted true)))) emb_candidates tag_candidates filtered_candidates reranked_candidates))) (return result))))  └─ @ /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947  Stacktrace:  [1] expand_forms_2(ctx::Base.JuliaLowering.MacroExpansionContext{Dict{Symbol, Dict{Int64, Any}}}, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}})  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/desugaring.jl:4448  [2] core_lowering_hook(code::Any, mod::Module, file::String, line::UInt64, world::UInt64, _warn::Bool)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/hooks.jl:30  [3] include(mapexpr::Function, mod::Module, _path::String)  @ Base Base.jl:327  [4] top-level scope  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:1  [5] macro expansion  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:47 [inlined]  [6] eval(m::Module, e::Any)  @ Core boot.jl:521  [7] _eval(mod::Module, iter::Base.JuliaLowering.LoweringIterator{Dict{Symbol, Dict{Int64, Any}}}; soft_scope::Nothing)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:545  [8] eval(mod::Module, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}; macro_world::UInt64, soft_scope::Nothing, opts::@Kwargs{expr_compat_mode::Bool})  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:518 [inlined]  [9] top-level scope  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:1  [10] macro expansion  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:1 [inlined]  [11] include(mapexpr::Function, mod::Module, _path::String)  @ Base Base.jl:327  [12] top-level scope  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:1  [13] macro expansion  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:18 [inlined]  [14] eval(m::Module, e::Any)  @ Core boot.jl:521  [15] _eval(mod::Module, iter::Base.JuliaLowering.LoweringIterator{Dict{Symbol, Dict{Int64, Any}}}; soft_scope::Nothing)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:545  [16] eval(mod::Module, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}; macro_world::UInt64, soft_scope::Nothing, opts::@Kwargs{expr_compat_mode::Bool})  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:518 [inlined]  [17] top-level scope  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:1  [18] macro expansion  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:1 [inlined]  [19] include(mapexpr::Function, mod::Module, _path::String)  @ Base Base.jl:327  [20] top-level scope  @ ~/.julia/packages/PromptingTools/wvgCI/src/PromptingTools.jl:120  [21] include(mod::Module, _path::String)  @ Base Base.jl:326  [22] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)  @ Base loading.jl:3296  [23] top-level scope  @ stdin:5  [24] eval(m::Module, e::Any)  @ Core boot.jl:521  [25] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)  @ Base loading.jl:3132  [26] include_string(m::Module, txt::String, fname::String)  @ Base loading.jl:3142 [inlined]  [27] exec_options(opts::Base.JLOptions)  @ Base client.jl:353  [28] _start()  @ Base client.jl:596 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:1 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:1 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/PromptingTools.jl:1 in expression starting at stdin:5 ✗ PromptingTools ERROR: LoadError: Precompiled image Base.PkgId(Base.UUID("670122d1-24a8-4d70-bfce-740807c42192"), "PromptingTools") not available with flags CacheFlags(; use_pkgimages=false, debug_level=1, check_bounds=1, inline=true, opt_level=0) Stacktrace:  [1] error(s::String)  @ Base error.jl:56  [2] __require_prelocked(pkg::Base.PkgId, env::String)  @ Base loading.jl:2837  [3] _require_prelocked(uuidkey::Base.PkgId, env::String)  @ Base loading.jl:2685  [4] macro expansion  @ loading.jl:2599 [inlined]  [5] macro expansion  @ lock.jl:376 [inlined]  [6] __require(into::Module, mod::Symbol)  @ Base loading.jl:2563  [7] require(into::Module, mod::Symbol)  @ Base loading.jl:2539 [inlined]  [8] eval_import_path(at::Module, from::Nothing, path::Expr, keyword::String)  @ Base module.jl:36 [inlined]  [9] eval_import_path_all(at::Module, path::Expr, keyword::String)  @ Base module.jl:60  [10] _eval_using(to::Module, path::Expr, flags::UInt8)  @ Base module.jl:137 [inlined]  [11] eval_using(to::Module, path::Expr)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/runtime.jl:207  [12] top-level scope  @ ~/.julia/packages/PromptingTools/wvgCI/ext/MarkdownPromptingToolsExt.jl:3  [13] include(mod::Module, _path::String)  @ Base Base.jl:326  [14] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)  @ Base loading.jl:3296  [15] top-level scope  @ stdin:5  [16] eval(m::Module, e::Any)  @ Core boot.jl:521  [17] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)  @ Base loading.jl:3132  [18] include_string(m::Module, txt::String, fname::String)  @ Base loading.jl:3142 [inlined]  [19] exec_options(opts::Base.JLOptions)  @ Base client.jl:353  [20] _start()  @ Base client.jl:596 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/ext/MarkdownPromptingToolsExt.jl:1 in expression starting at stdin:5 ✗ PromptingTools → MarkdownPromptingToolsExt ERROR: LoadError: Precompiled image Base.PkgId(Base.UUID("670122d1-24a8-4d70-bfce-740807c42192"), "PromptingTools") not available with flags CacheFlags(; use_pkgimages=false, debug_level=1, check_bounds=1, inline=true, opt_level=0) Stacktrace:  [1] error(s::String)  @ Base error.jl:56  [2] __require_prelocked(pkg::Base.PkgId, env::String)  @ Base loading.jl:2837  [3] _require_prelocked(uuidkey::Base.PkgId, env::String)  @ Base loading.jl:2685  [4] macro expansion  @ loading.jl:2599 [inlined]  [5] macro expansion  @ lock.jl:376 [inlined]  [6] __require(into::Module, mod::Symbol)  @ Base loading.jl:2563  [7] require(into::Module, mod::Symbol)  @ Base loading.jl:2539 [inlined]  [8] eval_import_path(at::Module, from::Nothing, path::Expr, keyword::String)  @ Base module.jl:36 [inlined]  [9] _eval_import(imported::Bool, to::Module, from::Nothing, paths::Expr)  @ Base module.jl:111  [10] eval_import(imported::Bool, to::Module, from::Nothing, paths::Expr)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/runtime.jl:195  [11] top-level scope  @ ~/.julia/packages/DocumenterDescriptions/kTXBL/src/DocumenterDescriptions.jl:4  [12] include(mod::Module, _path::String)  @ Base Base.jl:326  [13] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)  @ Base loading.jl:3296  [14] top-level scope  @ stdin:5  [15] eval(m::Module, e::Any)  @ Core boot.jl:521  [16] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)  @ Base loading.jl:3132  [17] include_string(m::Module, txt::String, fname::String)  @ Base loading.jl:3142 [inlined]  [18] exec_options(opts::Base.JLOptions)  @ Base client.jl:353  [19] _start()  @ Base client.jl:596 in expression starting at /home/pkgeval/.julia/packages/DocumenterDescriptions/kTXBL/src/DocumenterDescriptions.jl:1 in expression starting at stdin:5 ✗ DocumenterDescriptions 36 dependencies successfully precompiled in 1316 seconds. 33 already precompiled. 1 dependency had output during precompilation: ┌ HTTP │ ┌ Info: Ignoring an error that occurred during the precompilation workload │ │ exception = │ │ HTTP.RequestError: │ │ HTTP.Request: │ │ HTTP.Messages.Request: │ │ """ │ │ GET / HTTP/1.1 │ │ Host: localhost:57814 │ │ Accept: */* │ │ User-Agent: HTTP.jl/1.14.0-DEV.2309 │ │ Content-Length: 0 │ │ Accept-Encoding: gzip │ │ │ │ """Underlying error: │ │ MethodError: objects of type ReentrantLock are not callable. │ │ In case you did not try calling it explicitly, check if a ReentrantLock has been passed as an argument to a method that expects a callable instead. │ │ The object of type `ReentrantLock` exists, but no method is defined for this combination of argument types when trying to treat it as a callable object. │ │ Stacktrace: │ │ [1] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/ConnectionRequest.jl:143 [inlined] │ │ [2] (::HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, socket_type_tls::Nothing, readtimeout::Int64, connect_timeout::Int64, logerrors::Bool, logtag::Nothing, closeimmediately::Bool, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64}) │ │ @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/Y97L1/src/clientlayers/ConnectionRequest.jl:51 │ │ [3] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/ConnectionRequest.jl:60 [inlined] │ │ [4] (::Base.var"#45#46"{Base.var"#47#48"{ExponentialBackOff, HTTP.RetryRequest.var"#retrylayer##kw_body#manageretries#0##->###0"{HTTP.Messages.Request, Int64, typeof(HTTP.RetryRequest.FALSE), Base.RefValue{Int64}}, HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(args::HTTP.Messages.Request; kwargs::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64}) │ │ @ Base error.jl:311 │ │ [5] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/RetryRequest.jl:75 [inlined] │ │ [6] (::HTTP.RetryRequest.var"#retrylayer#manageretries##0"{HTTP.RetryRequest.var"#retrylayer##kw_body#manageretries#0##0"{HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}})(req::HTTP.Messages.Request; retry::Bool, retries::Int64, retry_delays::ExponentialBackOff, retry_check::Function, retry_non_idempotent::Bool, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64}) │ │ @ HTTP.RetryRequest ~/.julia/packages/HTTP/Y97L1/src/clientlayers/RetryRequest.jl:10 │ │ [7] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/RetryRequest.jl:30 [inlined] │ │ [8] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/CookieRequest.jl:42 [inlined] │ │ [9] (::HTTP.CookieRequest.var"#cookielayer#managecookies##0"{HTTP.CookieRequest.var"#cookielayer##kw_body#managecookies#0##0"{HTTP.RetryRequest.var"#retrylayer#manageretries##0"{HTTP.RetryRequest.var"#retrylayer##kw_body#manageretries#0##0"{HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}})(req::HTTP.Messages.Request; cookies::Bool, cookiejar::HTTP.Cookies.CookieJar, kw::@Kwargs{iofunction::Nothing, decompress::Nothing, verbose::Int64}) │ │ @ HTTP.CookieRequest ~/.julia/packages/HTTP/Y97L1/src/clientlayers/CookieRequest.jl:11 │ │ [10] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/CookieRequest.jl:19 [inlined] │ │ [11] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/HeadersRequest.jl:71 [inlined] │ │ [12] (::HTTP.HeadersRequest.var"#headerslayer#defaultheaders##0"{HTTP.HeadersRequest.var"#headerslayer##kw_body#defaultheaders#0##0"{HTTP.CookieRequest.var"#cookielayer#managecookies##0"{HTTP.CookieRequest.var"#cookielayer##kw_body#managecookies#0##0"{HTTP.RetryRequest.var"#retrylayer#manageretries##0"{HTTP.RetryRequest.var"#retrylayer##kw_body#manageretries#0##0"{HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}})(req::HTTP.Messages.Request; iofunction::Nothing, decompress::Nothing, basicauth::Bool, detect_content_type::Bool, canonicalize_headers::Bool, kw::@Kwargs{verbose::Int64}) │ │ @ HTTP.HeadersRequest ~/.julia/packages/HTTP/Y97L1/src/clientlayers/HeadersRequest.jl:8 │ │ [13] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/HeadersRequest.jl:14 [inlined] │ │ [14] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/RedirectRequest.jl:25 [inlined] │ │ [15] (::HTTP.RedirectRequest.var"#redirectlayer#redirects##0"{HTTP.RedirectRequest.var"#redirectlayer##kw_body#redirects#0##0"{HTTP.HeadersRequest.var"#headerslayer#defaultheaders##0"{HTTP.HeadersRequest.var"#headerslayer##kw_body#defaultheaders#0##0"{HTTP.CookieRequest.var"#cookielayer#managecookies##0"{HTTP.CookieRequest.var"#cookielayer##kw_body#managecookies#0##0"{HTTP.RetryRequest.var"#retrylayer#manageretries##0"{HTTP.RetryRequest.var"#retrylayer##kw_body#manageretries#0##0"{HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}})(req::HTTP.Messages.Request; redirect::Bool, redirect_limit::Int64, redirect_method::Nothing, forwardheaders::Bool, response_stream::Nothing, kw::@Kwargs{verbose::Int64}) │ │ @ HTTP.RedirectRequest ~/.julia/packages/HTTP/Y97L1/src/clientlayers/RedirectRequest.jl:8 │ │ [16] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/RedirectRequest.jl:14 [inlined] │ │ [17] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/MessageRequest.jl:30 [inlined] │ │ [18] (::HTTP.MessageRequest.var"#messagelayer#makerequest##0"{HTTP.MessageRequest.var"#messagelayer##kw_body#makerequest#0##0"{HTTP.RedirectRequest.var"#redirectlayer#redirects##0"{HTTP.RedirectRequest.var"#redirectlayer##kw_body#redirects#0##0"{HTTP.HeadersRequest.var"#headerslayer#defaultheaders##0"{HTTP.HeadersRequest.var"#headerslayer##kw_body#defaultheaders#0##0"{HTTP.CookieRequest.var"#cookielayer#managecookies##0"{HTTP.CookieRequest.var"#cookielayer##kw_body#managecookies#0##0"{HTTP.RetryRequest.var"#retrylayer#manageretries##0"{HTTP.RetryRequest.var"#retrylayer##kw_body#manageretries#0##0"{HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}})(method::String, url::URIs.URI, headers::Nothing, body::Vector{UInt8}; copyheaders::Bool, response_stream::Nothing, http_version::HTTP.Strings.HTTPVersion, verbose::Int64, kw::@Kwargs{}) │ │ @ HTTP.MessageRequest ~/.julia/packages/HTTP/Y97L1/src/clientlayers/MessageRequest.jl:17 │ │ [19] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/clientlayers/MessageRequest.jl:24 [inlined] │ │ [20] (::HTTP.MessageRequest.var"#messagelayer#makerequest##0"{HTTP.MessageRequest.var"#messagelayer##kw_body#makerequest#0##0"{HTTP.RedirectRequest.var"#redirectlayer#redirects##0"{HTTP.RedirectRequest.var"#redirectlayer##kw_body#redirects#0##0"{HTTP.HeadersRequest.var"#headerslayer#defaultheaders##0"{HTTP.HeadersRequest.var"#headerslayer##kw_body#defaultheaders#0##0"{HTTP.CookieRequest.var"#cookielayer#managecookies##0"{HTTP.CookieRequest.var"#cookielayer##kw_body#managecookies#0##0"{HTTP.RetryRequest.var"#retrylayer#manageretries##0"{HTTP.RetryRequest.var"#retrylayer##kw_body#manageretries#0##0"{HTTP.ConnectionRequest.var"#connectionlayer#connections##0"{HTTP.ConnectionRequest.var"#connectionlayer##kw_body#connections#0##0"{HTTP.TimeoutRequest.var"#timeoutlayer#timeouts##0"{HTTP.TimeoutRequest.var"#timeoutlayer##kw_body#timeouts#0##0"{HTTP.ExceptionRequest.var"#exceptionlayer#exceptions##0"{HTTP.ExceptionRequest.var"#exceptionlayer##kw_body#exceptions#0##0"{typeof(HTTP.StreamRequest.streamlayer)}}}}}}}}}}}}}}}})(method::String, url::URIs.URI, headers::Nothing, body::Vector{UInt8}) │ │ @ HTTP.MessageRequest ~/.julia/packages/HTTP/Y97L1/src/clientlayers/MessageRequest.jl:17 │ │ [21] request(stack::Function, method::String, url::String, h::Nothing, b::Vector{UInt8}, q::Nothing; headers::Nothing, body::Vector{UInt8}, query::Nothing, kw::@Kwargs{}) │ │ @ HTTP ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:463 │ │ [22] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:321 [inlined] │ │ [23] request(method::String, url::String, h::Nothing, b::Vector{UInt8}; headers::Nothing, body::Vector{UInt8}, query::Nothing, observelayers::Bool, kw::@Kwargs{}) │ │ @ HTTP ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:88 [inlined] │ │ [24] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:321 [inlined] │ │ [25] request(method::String, url::String, h::Nothing, b::Vector{UInt8}) │ │ @ HTTP ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:88 [inlined] │ │ [26] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:321 [inlined] │ │ [27] request(method::String, url::String) │ │ @ HTTP ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:88 [inlined] │ │ [28] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:526 [inlined] │ │ [29] get(a::String; kw::@Kwargs{}) │ │ @ HTTP ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:521 [inlined] │ │ [30] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:526 [inlined] │ │ [31] get(a::String) │ │ @ HTTP ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:521 [inlined] │ │ [32] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/precompile.jl:36 [inlined] │ │ [33] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/precompile.jl:34 [inlined] │ │ [34] (::HTTP.var"##->###3"{String})() │ │ @ HTTP ~/.julia/packages/HTTP/Y97L1/src/precompile.jl:4 [inlined] │ │ [35] withenv(::HTTP.var"##->###3"{String}, ::Pair{String, Union{Nothing, String}}, ::Vararg{Pair{String, Union{Nothing, String}}}) │ │ @ Base env.jl:265 │ │ [36] macro expansion │ │ @ ~/.julia/packages/PrecompileTools/QUxvR/src/workloads.jl:75 [inlined] │ │ [37] macro expansion │ │ @ ~/.julia/packages/HTTP/Y97L1/src/precompile.jl:34 [inlined] │ │ [38] top-level scope │ │ @ ~/.julia/packages/HTTP/Y97L1/src/precompile.jl:4 │ │ [39] include(mapexpr::Function, mod::Module, _path::String) │ │ @ Base Base.jl:327 │ │ [40] top-level scope │ │ @ ~/.julia/packages/HTTP/Y97L1/src/HTTP.jl:651 │ │ [41] include(mod::Module, _path::String) │ │ @ Base Base.jl:326 │ │ [42] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing) │ │ @ Base loading.jl:3296 │ │ [43] top-level scope │ │ @ stdin:5 │ │ [44] eval(m::Module, e::Any) │ │ @ Core boot.jl:521 │ │ [45] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) │ │ @ Base loading.jl:3132 │ │ [46] include_string(m::Module, txt::String, fname::String) │ │ @ Base loading.jl:3142 [inlined] │ │ [47] exec_options(opts::Base.JLOptions) │ │ @ Base client.jl:353 │ │ [48] _start() │ └ @ Base client.jl:596 └ Precompilation completed after 1354.34s ################################################################################ # Testing # Testing DocumenterDescriptions Status `/tmp/jl_cMFmL0/Project.toml` [e30172f5] Documenter v1.17.0 [e9159221] DocumenterDescriptions v0.1.0 ⌅ [670122d1] PromptingTools v0.73.0 [fa267f1f] TOML v1.0.3 [8dfed614] Test v1.11.0 Status `/tmp/jl_cMFmL0/Manifest.toml` [a4c015fc] ANSIColoredPrinters v0.0.1 [1520ce14] AbstractTrees v0.4.5 [d1d4a3ce] BitFlags v0.1.10 [944b1d66] CodecZlib v0.7.8 [f0e56b4a] ConcurrentUtilities v2.5.1 [ffbed154] DocStringExtensions v0.9.5 [e30172f5] Documenter v1.17.0 [e9159221] DocumenterDescriptions v0.1.0 [460bff9d] ExceptionUnwrapping v0.1.11 [d7ba0133] Git v1.5.0 ⌅ [cd3eb016] HTTP v1.11.0 [b5f81e59] IOCapture v1.0.0 [692b3bcd] JLLWrappers v1.8.0 [682c06a0] JSON v1.6.1 [0f8b85d8] JSON3 v1.14.3 [0e77f7df] LazilyInitializedFields v1.3.0 [e6f89c97] LoggingExtras v1.2.0 [d0879d2d] MarkdownAST v0.1.3 [739be429] MbedTLS v1.1.10 ⌅ [e9f21f70] OpenAI v0.9.1 [4d8831e6] OpenSSL v1.6.1 [69de0a69] Parsers v2.8.5 [aea7be01] PrecompileTools v1.3.4 [21216c6a] Preferences v1.5.2 ⌅ [670122d1] PromptingTools v0.73.0 [2792f1a3] RegistryInstances v0.1.0 [777ac1f9] SimpleBufferStream v1.2.0 ⌅ [c1b9e933] StreamCallbacks v0.6.2 [856f2bd8] StructTypes v1.11.0 [ec057cc2] StructUtils v2.8.2 [3bb67fe8] TranscodingStreams v0.11.3 [5c2747f8] URIs v1.6.1 [2e619515] Expat_jll v2.8.1+0 [020c3dae] Git_LFS_jll v3.7.1+0 [f8c6e375] Git_jll v2.54.0+0 [94ce4f54] Libiconv_jll v1.18.0+0 [c8ffd9c3] MbedTLS_jll v2.28.1010+0 [9bd350c2] OpenSSH_jll v10.3.1+0 [0dad84c5] ArgTools v1.1.2 [56f22d72] Artifacts v1.11.0 [2a0f44e3] Base64 v1.11.0 [ade2ca70] Dates v1.11.0 [f43a241f] Downloads v1.7.0 [7b1f6079] FileWatching v1.11.0 [b77e0a4c] InteractiveUtils v1.11.0 [ac6e5ff7] JuliaSyntaxHighlighting v1.13.0 [b27032c2] LibCURL v1.0.0 [76f85450] LibGit2 v1.11.0 [8f399da3] Libdl v1.11.0 [56ddb016] Logging v1.11.0 [d6f4376e] Markdown v1.11.0 [a63ad114] Mmap v1.11.0 [ca575930] NetworkOptions v1.3.0 [44cfe95a] Pkg v1.14.0 [de0858da] Printf v1.11.0 [3fa0cd96] REPL v1.11.0 [9a3f8284] Random v1.11.0 [ea8e919c] SHA v1.13.0 [9e88b42a] Serialization v1.11.0 [6462fe0b] Sockets v1.11.0 [f489334b] StyledStrings v1.13.0 [fa267f1f] TOML v1.0.3 [a4e569a6] Tar v1.10.0 [8dfed614] Test v1.11.0 [cf7118a7] UUIDs v1.11.0 [4ec0a83e] Unicode v1.11.0 [e66e0078] CompilerSupportLibraries_jll v1.5.2+0 [deac9b47] LibCURL_jll v8.20.0+1 [e37daf67] LibGit2_jll v1.9.4+0 [29816b5a] LibSSH2_jll v1.11.101+0 [14a3606d] MozillaCACerts_jll v2026.5.14 [458c3c95] OpenSSL_jll v3.5.6+0 [efcefdf7] PCRE2_jll v10.47.0+0 [83775a58] Zlib_jll v1.3.2+0 [3161d3a3] Zstd_jll v1.5.7+1 [8e850ede] nghttp2_jll v1.69.0+0 [3f19e933] p7zip_jll v17.8.0+0 Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. Testing Running tests... ┌ Warning: OPENAI_API_KEY variable not set! OpenAI models will not be available - set API key directly via `PromptingTools.OPENAI_API_KEY=`! └ @ PromptingTools ~/.julia/packages/PromptingTools/wvgCI/src/user_preferences.jl:189 ┌ Info: JuliaLowering threw given input: │ code = │ :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947 =# Core.@doc " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" function retrieve(retriever::AbstractRetriever, index::AbstractDocumentIndex, question::AbstractString; verbose::Integer = 1, top_k::Integer = 100, top_n::Integer = 5, api_kwargs::NamedTuple = NamedTuple(), rephraser::AbstractRephraser = retriever.rephraser, rephraser_kwargs::NamedTuple = NamedTuple(), embedder::AbstractEmbedder = retriever.embedder, embedder_kwargs::NamedTuple = NamedTuple(), processor::AbstractProcessor = retriever.processor, processor_kwargs::NamedTuple = NamedTuple(), finder::AbstractSimilarityFinder = retriever.finder, finder_kwargs::NamedTuple = NamedTuple(), tagger::AbstractTagger = retriever.tagger, tagger_kwargs::NamedTuple = NamedTuple(), filter::AbstractTagFilter = retriever.filter, filter_kwargs::NamedTuple = NamedTuple(), reranker::AbstractReranker = retriever.reranker, reranker_kwargs::NamedTuple = NamedTuple(), cost_tracker = Threads.Atomic{Float64}(0.0), kwargs...) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1059 =# │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1083 =# │ rephraser_kwargs_ = if isempty(api_kwargs) │ rephraser_kwargs │ else │ merge(rephraser_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1085 =# │ rephrased_questions = rephrase(rephraser, question; verbose = verbose > 1, cost_tracker, rephraser_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1089 =# │ embeddings = if HasEmbeddings(index) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1090 =# │ embedder_kwargs_ = if isempty(api_kwargs) │ embedder_kwargs │ else │ merge(embedder_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1092 =# │ embeddings = get_embeddings(embedder, rephrased_questions; verbose = verbose > 1, cost_tracker, embedder_kwargs_...) │ else │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1095 =# │ embeddings = hcat([Float32[] for x = rephrased_questions]...) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1099 =# │ keywords = if HasKeywords(index) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1101 =# │ keywords = get_keywords(processor, rephrased_questions; verbose = verbose > 1, processor_kwargs..., return_keywords = true) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1104 =# │ verbose >= 1 && (keywords isa AbstractVector{<:AbstractVector{<:AbstractString}} || #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1105 =# @warn("Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: $(typeof(processor))).")) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1106 =# │ keywords │ else │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1108 =# │ [String[] for x = rephrased_questions] │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1111 =# │ finder_kwargs_ = if isempty(api_kwargs) │ finder_kwargs │ else │ merge(finder_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1113 =# │ emb_candidates = find_closest(finder, index, embeddings, keywords; verbose = verbose > 1, top_k, finder_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1117 =# │ tagger_kwargs_ = if isempty(api_kwargs) │ tagger_kwargs │ else │ merge(tagger_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1119 =# │ tags = get_tags(tagger, rephrased_questions; verbose = verbose > 1, cost_tracker, tagger_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1122 =# │ filter_kwargs_ = if isempty(api_kwargs) │ filter_kwargs │ else │ merge(filter_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1124 =# │ tag_candidates = find_tags(filter, index, tags; verbose = verbose > 1, filter_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1129 =# │ filtered_candidates = if isnothing(tag_candidates) │ emb_candidates │ else │ emb_candidates & tag_candidates │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1136 =# │ reranker_kwargs_ = if isempty(api_kwargs) │ reranker_kwargs │ else │ merge(reranker_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1138 =# │ reranked_candidates = rerank(reranker, index, question, filtered_candidates; top_n, verbose = verbose > 1, cost_tracker, reranker_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1141 =# │ verbose > 0 && #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1142 =# @info("Retrieval done. Identified $(length(positions(reranked_candidates))) chunks, total cost: \$$(round(cost_tracker[], digits = 2)).") │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1145 =# │ result = RAGResult(; question, answer = nothing, rephrased_questions, final_answer = nothing, context = collect(index[reranked_candidates, :chunks, sorted = true]), sources = collect(index[reranked_candidates, :sources, sorted = true]), emb_candidates, tag_candidates, filtered_candidates, reranked_candidates) │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1158 =# │ return result │ end) │ st0 = │ SyntaxTree with attributes mod,kind,var_id,toplevel_pure,scope_type,macro_source,name_val,syntax_flags,meta,scope_layer,value,jl_source,is_toplevel_thunk,source,__macro_ctx__ │ [macrocall] │ │ @doc :: Identifier │ mod │ :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947 =#) :: Value │ │ " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" :: Value │ │ [function] │ │ [call] │ │ retrieve :: Identifier │ │ [parameters] │ │ [kw] │ │ [::] │ │ verbose :: Identifier │ │ Integer :: Identifier │ │ 1 :: Value │ │ [kw] │ │ [::] │ │ top_k :: Identifier │ │ Integer :: Identifier │ │ 100 :: Value │ │ [kw] │ │ [::] │ │ top_n :: Identifier │ │ Integer :: Identifier │ │ 5 :: Value │ │ [kw] │ │ [::] │ │ api_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ rephraser :: Identifier │ │ AbstractRephraser :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ rephraser :: Identifier │ │ [kw] │ │ [::] │ │ rephraser_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ embedder :: Identifier │ │ AbstractEmbedder :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ embedder :: Identifier │ │ [kw] │ │ [::] │ │ embedder_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ processor :: Identifier │ │ AbstractProcessor :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ processor :: Identifier │ │ [kw] │ │ [::] │ │ processor_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ finder :: Identifier │ │ AbstractSimilarityFinder :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ finder :: Identifier │ │ [kw] │ │ [::] │ │ finder_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ tagger :: Identifier │ │ AbstractTagger :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ tagger :: Identifier │ │ [kw] │ │ [::] │ │ tagger_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ filter :: Identifier │ │ AbstractTagFilter :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ filter :: Identifier │ │ [kw] │ │ [::] │ │ filter_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ reranker :: Identifier │ │ AbstractReranker :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ reranker :: Identifier │ │ [kw] │ │ [::] │ │ reranker_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ cost_tracker :: Identifier │ │ [call] │ │ [curly] │ │ [.] │ │ Threads :: Identifier │ │ [inert] │ │ Atomic :: Identifier │ │ Float64 :: Identifier │ │ 0.0 :: Value │ │ [...] │ │ kwargs :: Identifier │ │ [::] │ │ retriever :: Identifier │ │ AbstractRetriever :: Identifier │ │ [::] │ │ index :: Identifier │ │ AbstractDocumentIndex :: Identifier │ │ [::] │ │ question :: Identifier │ │ AbstractString :: Identifier │ │ [block] │ │ [=] │ │ rephraser_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ rephraser_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ rephraser_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ rephrased_questions :: Identifier │ │ [call] │ │ rephrase :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ rephraser_kwargs_ :: Identifier │ │ rephraser :: Identifier │ │ question :: Identifier │ │ [=] │ │ embeddings :: Identifier │ │ [if] │ │ [call] │ │ HasEmbeddings :: Identifier │ │ index :: Identifier │ │ [block] │ │ [=] │ │ embedder_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ embedder_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ embedder_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ embeddings :: Identifier │ │ [call] │ │ get_embeddings :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ embedder_kwargs_ :: Identifier │ │ embedder :: Identifier │ │ rephrased_questions :: Identifier │ │ [block] │ │ [=] │ │ embeddings :: Identifier │ │ [call] │ │ hcat :: Identifier │ │ [...] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ Float32 :: Identifier │ │ [=] │ │ x :: Identifier │ │ rephrased_questions :: Identifier │ │ [=] │ │ keywords :: Identifier │ │ [if] │ │ [call] │ │ HasKeywords :: Identifier │ │ index :: Identifier │ │ [block] │ │ [=] │ │ keywords :: Identifier │ │ [call] │ │ get_keywords :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ [...] │ │ processor_kwargs :: Identifier │ │ [kw] │ │ return_keywords :: Identifier │ │ true :: Value │ │ processor :: Identifier │ │ rephrased_questions :: Identifier │ │ [&&] │ │ [call] │ │ >= :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ [||] │ │ [call] │ │ isa :: Identifier │ │ keywords :: Identifier │ │ [curly] │ │ AbstractVector :: Identifier │ │ [<:] │ │ [curly] │ │ AbstractVector :: Identifier │ │ [<:] │ │ AbstractString :: Identifier │ │ [macrocall] │ │ @warn :: Identifier │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1105 =#) :: Value │ │ [string] │ │ "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " :: Value │ │ [call] │ │ typeof :: Identifier │ │ processor :: Identifier │ │ ")." :: Value │ │ keywords :: Identifier │ │ [block] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ String :: Identifier │ │ [=] │ │ x :: Identifier │ │ rephrased_questions :: Identifier │ │ [=] │ │ finder_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ finder_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ finder_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ emb_candidates :: Identifier │ │ [call] │ │ find_closest :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ top_k :: Identifier │ │ [...] │ │ finder_kwargs_ :: Identifier │ │ finder :: Identifier │ │ index :: Identifier │ │ embeddings :: Identifier │ │ keywords :: Identifier │ │ [=] │ │ tagger_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ tagger_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ tagger_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ tags :: Identifier │ │ [call] │ │ get_tags :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ tagger_kwargs_ :: Identifier │ │ tagger :: Identifier │ │ rephrased_questions :: Identifier │ │ [=] │ │ filter_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ filter_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ filter_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ tag_candidates :: Identifier │ │ [call] │ │ find_tags :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ [...] │ │ filter_kwargs_ :: Identifier │ │ filter :: Identifier │ │ index :: Identifier │ │ tags :: Identifier │ │ [=] │ │ filtered_candidates :: Identifier │ │ [if] │ │ [call] │ │ isnothing :: Identifier │ │ tag_candidates :: Identifier │ │ emb_candidates :: Identifier │ │ [call] │ │ & :: Identifier │ │ emb_candidates :: Identifier │ │ tag_candidates :: Identifier │ │ [=] │ │ reranker_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ reranker_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ reranker_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ reranked_candidates :: Identifier │ │ [call] │ │ rerank :: Identifier │ │ [parameters] │ │ top_n :: Identifier │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ reranker_kwargs_ :: Identifier │ │ reranker :: Identifier │ │ index :: Identifier │ │ question :: Identifier │ │ filtered_candidates :: Identifier │ │ [&&] │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 0 :: Value │ │ [macrocall] │ │ @info :: Identifier │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1142 =#) :: Value │ │ [string] │ │ "Retrieval done. Identified " :: Value │ │ [call] │ │ length :: Identifier │ │ [call] │ │ positions :: Identifier │ │ reranked_candidates :: Identifier │ │ " chunks, total cost: \$" :: Value │ │ [call] │ │ round :: Identifier │ │ [ref] │ │ cost_tracker :: Identifier │ │ [kw] │ │ digits :: Identifier │ │ 2 :: Value │ │ "." :: Value │ │ [=] │ │ result :: Identifier │ │ [call] │ │ RAGResult :: Identifier │ │ [parameters] │ │ question :: Identifier │ │ [kw] │ │ answer :: Identifier │ │ nothing :: Identifier │ │ rephrased_questions :: Identifier │ │ [kw] │ │ final_answer :: Identifier │ │ nothing :: Identifier │ │ [kw] │ │ context :: Identifier │ │ [call] │ │ collect :: Identifier │ │ [ref] │ │ index :: Identifier │ │ reranked_candidates :: Identifier │ │ [inert] │ │ chunks :: Identifier │ │ [kw] │ │ sorted :: Identifier │ │ true :: Value │ │ [kw] │ │ sources :: Identifier │ │ [call] │ │ collect :: Identifier │ │ [ref] │ │ index :: Identifier │ │ reranked_candidates :: Identifier │ │ [inert] │ │ sources :: Identifier │ │ [kw] │ │ sorted :: Identifier │ │ true :: Value │ │ emb_candidates :: Identifier │ │ tag_candidates :: Identifier │ │ filtered_candidates :: Identifier │ │ reranked_candidates :: Identifier │ │ [return] │ │ result :: Identifier │ │ │ st1 = │ SyntaxTree with attributes mod,kind,var_id,toplevel_pure,scope_type,macro_source,name_val,syntax_flags,meta,scope_layer,value,jl_source,is_toplevel_thunk,source │ [block] │ │ [=] │ │ val :: Identifier │ scope_layer=3 │ [function] │ │ [call] │ │ retrieve :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ [::] │ │ verbose :: Identifier │ scope_layer=1 │ Integer :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [kw] │ │ [::] │ │ top_k :: Identifier │ scope_layer=1 │ Integer :: Identifier │ scope_layer=1 │ 100 :: Value │ macro_source=485 │ [kw] │ │ [::] │ │ top_n :: Identifier │ scope_layer=1 │ Integer :: Identifier │ scope_layer=1 │ 5 :: Value │ macro_source=485 │ [kw] │ │ [::] │ │ api_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ rephraser :: Identifier │ scope_layer=1 │ AbstractRephraser :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ rephraser :: Identifier │ │ [kw] │ │ [::] │ │ rephraser_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ embedder :: Identifier │ scope_layer=1 │ AbstractEmbedder :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ embedder :: Identifier │ │ [kw] │ │ [::] │ │ embedder_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ processor :: Identifier │ scope_layer=1 │ AbstractProcessor :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ processor :: Identifier │ │ [kw] │ │ [::] │ │ processor_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ finder :: Identifier │ scope_layer=1 │ AbstractSimilarityFinder :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ finder :: Identifier │ │ [kw] │ │ [::] │ │ finder_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ tagger :: Identifier │ scope_layer=1 │ AbstractTagger :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ tagger :: Identifier │ │ [kw] │ │ [::] │ │ tagger_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ filter :: Identifier │ scope_layer=1 │ AbstractTagFilter :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ filter :: Identifier │ │ [kw] │ │ [::] │ │ filter_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ reranker :: Identifier │ scope_layer=1 │ AbstractReranker :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ reranker :: Identifier │ │ [kw] │ │ [::] │ │ reranker_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ cost_tracker :: Identifier │ scope_layer=1 │ [call] │ │ [curly] │ │ [.] │ │ Threads :: Identifier │ scope_layer=1 │ [inert] │ │ Atomic :: Identifier │ │ Float64 :: Identifier │ scope_layer=1 │ 0.0 :: Value │ macro_source=485 │ [...] │ │ kwargs :: Identifier │ scope_layer=1 │ [::] │ │ retriever :: Identifier │ scope_layer=1 │ AbstractRetriever :: Identifier │ scope_layer=1 │ [::] │ │ index :: Identifier │ scope_layer=1 │ AbstractDocumentIndex :: Identifier │ scope_layer=1 │ [::] │ │ question :: Identifier │ scope_layer=1 │ AbstractString :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ rephraser_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ rephraser_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ rephraser_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ rephrased_questions :: Identifier │ scope_layer=1 │ [call] │ │ rephrase :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ rephraser_kwargs_ :: Identifier │ scope_layer=1 │ rephraser :: Identifier │ scope_layer=1 │ question :: Identifier │ scope_layer=1 │ [=] │ │ embeddings :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ HasEmbeddings :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ embedder_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ embedder_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ embedder_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ embeddings :: Identifier │ scope_layer=1 │ [call] │ │ get_embeddings :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ embedder_kwargs_ :: Identifier │ scope_layer=1 │ embedder :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ embeddings :: Identifier │ scope_layer=1 │ [call] │ │ hcat :: Identifier │ scope_layer=1 │ [...] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ Float32 :: Identifier │ scope_layer=1 │ [=] │ │ x :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [=] │ │ keywords :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ HasKeywords :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ keywords :: Identifier │ scope_layer=1 │ [call] │ │ get_keywords :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [...] │ │ processor_kwargs :: Identifier │ scope_layer=1 │ [kw] │ │ return_keywords :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ processor :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [&&] │ │ [call] │ │ >= :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [||] │ │ [call] │ │ isa :: Identifier │ scope_layer=1 │ keywords :: Identifier │ scope_layer=1 │ [curly] │ │ AbstractVector :: Identifier │ scope_layer=1 │ [<:] │ scope_layer=1 │ [curly] │ │ AbstractVector :: Identifier │ scope_layer=1 │ [<:] │ scope_layer=1 │ AbstractString :: Identifier │ scope_layer=1 │ [block] │ │ [let] │ │ [block] │ macro_source=485 │ [block] │ │ [=] │ │ #1281#level :: Identifier │ scope_layer=1 │ Warn :: Identifier │ mod,scope_layer=1 │ [=] │ │ #1282#std_level :: Identifier │ scope_layer=1 │ #1281#level :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ >= :: Identifier │ mod,scope_layer=1 │ [.] │ │ #1282#std_level :: Identifier │ scope_layer=1 │ [inert] │ │ level :: Identifier │ │ [ref] │ macro_source=485 │ Base.Threads.Atomic{Int32}(-1000) :: Value │ macro_source=485 │ [block] │ │ [=] │ │ #1283#group :: Identifier │ scope_layer=1 │ [inert] │ │ retrieval :: Identifier │ │ [=] │ │ #1284#_module :: Identifier │ scope_layer=1 │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [=] │ │ #1285#logger :: Identifier │ scope_layer=1 │ [call] │ │ Base.CoreLogging.current_logger_for_env :: Value │ macro_source=485 │ #1282#std_level :: Identifier │ scope_layer=1 │ #1283#group :: Identifier │ scope_layer=1 │ #1284#_module :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ ! :: Identifier │ mod,scope_layer=1 │ [call] │ │ === :: Identifier │ mod,scope_layer=1 │ #1285#logger :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1286#id :: Identifier │ scope_layer=1 │ [inert] │ │ PromptingTools_Experimental_RAGTools_7aa4399b :: Identifier │ │ [if] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.shouldlog :: Value │ macro_source=485 │ #1285#logger :: Identifier │ scope_layer=1 │ #1281#level :: Identifier │ scope_layer=1 │ #1284#_module :: Identifier │ scope_layer=1 │ #1283#group :: Identifier │ scope_layer=1 │ #1286#id :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ #1287#file :: Identifier │ scope_layer=1 │ "/home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ [if] │ │ [call] │ │ isa :: Identifier │ mod,scope_layer=1 │ #1287#file :: Identifier │ scope_layer=1 │ String :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1287#file :: Identifier │ scope_layer=1 │ [call] │ │ [.] │ │ Base :: Identifier │ mod,scope_layer=1 │ [inert] │ │ fixup_stdlib_path :: Identifier │ │ #1287#file :: Identifier │ scope_layer=1 │ [=] │ │ #1288#line :: Identifier │ scope_layer=1 │ 1105 :: Value │ macro_source=485 │ [local] │ │ #1289#msg :: Identifier │ scope_layer=1 │ #1290#kwargs :: Identifier │ scope_layer=1 │ [if] │ │ [block] │ │ [try] │ │ [block] │ │ [=] │ │ #1289#msg :: Identifier │ scope_layer=1 │ [string] │ │ "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " :: Value │ macro_source=485 │ [call] │ │ typeof :: Identifier │ scope_layer=1 │ processor :: Identifier │ scope_layer=1 │ ")." :: Value │ macro_source=485 │ [=] │ │ #1290#kwargs :: Identifier │ scope_layer=1 │ [tuple] │ macro_source=485 │ [parameters] │ macro_source=485 │ true :: Value │ macro_source=485 │ #1303#err :: Identifier │ scope_layer=1 │ [block] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.logging_error :: Value │ macro_source=485 │ #1285#logger :: Identifier │ scope_layer=1 │ #1281#level :: Identifier │ scope_layer=1 │ #1284#_module :: Identifier │ scope_layer=1 │ #1283#group :: Identifier │ scope_layer=1 │ #1286#id :: Identifier │ scope_layer=1 │ #1287#file :: Identifier │ scope_layer=1 │ #1288#line :: Identifier │ scope_layer=1 │ #1303#err :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ false :: Value │ macro_source=485 │ [block] │ │ [if] │ │ [isdefined] │ │ #1289#msg :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [if] │ │ [isdefined] │ │ #1290#kwargs :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [call] │ │ Base.CoreLogging.handle_message_nothrow :: Value │ macro_source=485 │ [parameters] │ │ [...] │ │ #1290#kwargs :: Identifier │ scope_layer=1 │ #1285#logger :: Identifier │ scope_layer=1 │ #1281#level :: Identifier │ scope_layer=1 │ #1289#msg :: Identifier │ scope_layer=1 │ #1284#_module :: Identifier │ scope_layer=1 │ #1283#group :: Identifier │ scope_layer=1 │ #1286#id :: Identifier │ scope_layer=1 │ #1287#file :: Identifier │ scope_layer=1 │ #1288#line :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ keywords :: Identifier │ scope_layer=1 │ [block] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ String :: Identifier │ scope_layer=1 │ [=] │ │ x :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [=] │ │ finder_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ finder_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ finder_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ emb_candidates :: Identifier │ scope_layer=1 │ [call] │ │ find_closest :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ top_k :: Identifier │ scope_layer=1 │ [...] │ │ finder_kwargs_ :: Identifier │ scope_layer=1 │ finder :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ embeddings :: Identifier │ scope_layer=1 │ keywords :: Identifier │ scope_layer=1 │ [=] │ │ tagger_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ tagger_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ tagger_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ tags :: Identifier │ scope_layer=1 │ [call] │ │ get_tags :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ tagger_kwargs_ :: Identifier │ scope_layer=1 │ tagger :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [=] │ │ filter_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ filter_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ filter_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ tag_candidates :: Identifier │ scope_layer=1 │ [call] │ │ find_tags :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [...] │ │ filter_kwargs_ :: Identifier │ scope_layer=1 │ filter :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ tags :: Identifier │ scope_layer=1 │ [=] │ │ filtered_candidates :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isnothing :: Identifier │ scope_layer=1 │ tag_candidates :: Identifier │ scope_layer=1 │ emb_candidates :: Identifier │ scope_layer=1 │ [call] │ │ & :: Identifier │ scope_layer=1 │ emb_candidates :: Identifier │ scope_layer=1 │ tag_candidates :: Identifier │ scope_layer=1 │ [=] │ │ reranker_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ reranker_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ reranker_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ reranked_candidates :: Identifier │ scope_layer=1 │ [call] │ │ rerank :: Identifier │ scope_layer=1 │ [parameters] │ │ top_n :: Identifier │ scope_layer=1 │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ reranker_kwargs_ :: Identifier │ scope_layer=1 │ reranker :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ question :: Identifier │ scope_layer=1 │ filtered_candidates :: Identifier │ scope_layer=1 │ [&&] │ │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 0 :: Value │ macro_source=485 │ [block] │ │ [let] │ │ [block] │ macro_source=485 │ [block] │ │ [=] │ │ #1304#level :: Identifier │ scope_layer=1 │ Info :: Identifier │ mod,scope_layer=1 │ [=] │ │ #1305#std_level :: Identifier │ scope_layer=1 │ #1304#level :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ >= :: Identifier │ mod,scope_layer=1 │ [.] │ │ #1305#std_level :: Identifier │ scope_layer=1 │ [inert] │ │ level :: Identifier │ │ [ref] │ macro_source=485 │ Base.Threads.Atomic{Int32}(-1000) :: Value │ macro_source=485 │ [block] │ │ [=] │ │ #1306#group :: Identifier │ scope_layer=1 │ [inert] │ │ retrieval :: Identifier │ │ [=] │ │ #1307#_module :: Identifier │ scope_layer=1 │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [=] │ │ #1308#logger :: Identifier │ scope_layer=1 │ [call] │ │ Base.CoreLogging.current_logger_for_env :: Value │ macro_source=485 │ #1305#std_level :: Identifier │ scope_layer=1 │ #1306#group :: Identifier │ scope_layer=1 │ #1307#_module :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ ! :: Identifier │ mod,scope_layer=1 │ [call] │ │ === :: Identifier │ mod,scope_layer=1 │ #1308#logger :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1309#id :: Identifier │ scope_layer=1 │ [inert] │ │ PromptingTools_Experimental_RAGTools_30253e51 :: Identifier │ │ [if] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.shouldlog :: Value │ macro_source=485 │ #1308#logger :: Identifier │ scope_layer=1 │ #1304#level :: Identifier │ scope_layer=1 │ #1307#_module :: Identifier │ scope_layer=1 │ #1306#group :: Identifier │ scope_layer=1 │ #1309#id :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ #1310#file :: Identifier │ scope_layer=1 │ "/home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ [if] │ │ [call] │ │ isa :: Identifier │ mod,scope_layer=1 │ #1310#file :: Identifier │ scope_layer=1 │ String :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1310#file :: Identifier │ scope_layer=1 │ [call] │ │ [.] │ │ Base :: Identifier │ mod,scope_layer=1 │ [inert] │ │ fixup_stdlib_path :: Identifier │ │ #1310#file :: Identifier │ scope_layer=1 │ [=] │ │ #1311#line :: Identifier │ scope_layer=1 │ 1142 :: Value │ macro_source=485 │ [local] │ │ #1312#msg :: Identifier │ scope_layer=1 │ #1313#kwargs :: Identifier │ scope_layer=1 │ [if] │ │ [block] │ │ [try] │ │ [block] │ │ [=] │ │ #1312#msg :: Identifier │ scope_layer=1 │ [string] │ │ "Retrieval done. Identified " :: Value │ macro_source=485 │ [call] │ │ length :: Identifier │ scope_layer=1 │ [call] │ │ positions :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ " chunks, total cost: \$" :: Value │ macro_source=485 │ [call] │ │ round :: Identifier │ scope_layer=1 │ [ref] │ │ cost_tracker :: Identifier │ scope_layer=1 │ [kw] │ │ digits :: Identifier │ scope_layer=1 │ 2 :: Value │ macro_source=485 │ "." :: Value │ macro_source=485 │ [=] │ │ #1313#kwargs :: Identifier │ scope_layer=1 │ [tuple] │ macro_source=485 │ [parameters] │ macro_source=485 │ true :: Value │ macro_source=485 │ #1326#err :: Identifier │ scope_layer=1 │ [block] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.logging_error :: Value │ macro_source=485 │ #1308#logger :: Identifier │ scope_layer=1 │ #1304#level :: Identifier │ scope_layer=1 │ #1307#_module :: Identifier │ scope_layer=1 │ #1306#group :: Identifier │ scope_layer=1 │ #1309#id :: Identifier │ scope_layer=1 │ #1310#file :: Identifier │ scope_layer=1 │ #1311#line :: Identifier │ scope_layer=1 │ #1326#err :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ false :: Value │ macro_source=485 │ [block] │ │ [if] │ │ [isdefined] │ │ #1312#msg :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [if] │ │ [isdefined] │ │ #1313#kwargs :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [call] │ │ Base.CoreLogging.handle_message_nothrow :: Value │ macro_source=485 │ [parameters] │ │ [...] │ │ #1313#kwargs :: Identifier │ scope_layer=1 │ #1308#logger :: Identifier │ scope_layer=1 │ #1304#level :: Identifier │ scope_layer=1 │ #1312#msg :: Identifier │ scope_layer=1 │ #1307#_module :: Identifier │ scope_layer=1 │ #1306#group :: Identifier │ scope_layer=1 │ #1309#id :: Identifier │ scope_layer=1 │ #1310#file :: Identifier │ scope_layer=1 │ #1311#line :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ [=] │ │ result :: Identifier │ scope_layer=1 │ [call] │ │ RAGResult :: Identifier │ scope_layer=1 │ [parameters] │ │ question :: Identifier │ scope_layer=1 │ [kw] │ │ answer :: Identifier │ scope_layer=1 │ nothing :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [kw] │ │ final_answer :: Identifier │ scope_layer=1 │ nothing :: Identifier │ scope_layer=1 │ [kw] │ │ context :: Identifier │ scope_layer=1 │ [call] │ │ collect :: Identifier │ scope_layer=1 │ [ref] │ │ index :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ [inert] │ │ chunks :: Identifier │ │ [kw] │ │ sorted :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ [kw] │ │ sources :: Identifier │ scope_layer=1 │ [call] │ │ collect :: Identifier │ scope_layer=1 │ [ref] │ │ index :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ [inert] │ │ sources :: Identifier │ │ [kw] │ │ sorted :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ emb_candidates :: Identifier │ scope_layer=1 │ tag_candidates :: Identifier │ scope_layer=1 │ filtered_candidates :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ [return] │ │ result :: Identifier │ scope_layer=1 │ [call] │ │ Base.Docs.doc! :: Value │ macro_source=485 │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [call] │ │ Base.Docs.Binding :: Value │ macro_source=485 │ PromptingTools.Experimental.RAGTools :: Value │ │ [inert] │ jl_source=L65 │ retrieve :: Identifier │ │ [call] │ macro_source=485 │ Base.Docs.docstr :: Value │ macro_source=485 │ [call] │ macro_source=485 │ Core.svec :: Value │ macro_source=485 │ " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" :: Value │ macro_source=485 │ [call] │ macro_source=485 │ Dict{Symbol, Any} :: Value │ macro_source=485 │ :path => "/home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ :linenumber => 947 :: Value │ macro_source=485 │ :module => PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [curly] │ │ Union :: Identifier │ scope_layer=1 │ [curly] │ │ Tuple :: Identifier │ scope_layer=1 │ AbstractRetriever :: Identifier │ scope_layer=1 │ AbstractDocumentIndex :: Identifier │ scope_layer=1 │ AbstractString :: Identifier │ scope_layer=1 │ val :: Identifier │ scope_layer=3 │ │ file = "/home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl" │ line = 947 └ mod = PromptingTools.Experimental.RAGTools ERROR: LoadError: LoweringError: #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1145 =# - invalid syntax: unknown form `kw` or number of arguments 2 Expression:  (kw sorted true) Containing expressions:  (kw context (call collect (ref index reranked_candidates (inert chunks) (kw sorted true))))  Detailed provenance:  (kw sorted true)  └─ (kw sorted true)  ├─ @ /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1145  └─ (macrocall @doc :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947 =#) " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" (function (call retrieve (parameters (kw (:: verbose Integer) 1) (kw (:: top_k Integer) 100) (kw (:: top_n Integer) 5) (kw (:: api_kwargs NamedTuple) (call NamedTuple)) (kw (:: rephraser AbstractRephraser) (. retriever (inert rephraser))) (kw (:: rephraser_kwargs NamedTuple) (call NamedTuple)) (kw (:: embedder AbstractEmbedder) (. retriever (inert embedder))) (kw (:: embedder_kwargs NamedTuple) (call NamedTuple)) (kw (:: processor AbstractProcessor) (. retriever (inert processor))) (kw (:: processor_kwargs NamedTuple) (call NamedTuple)) (kw (:: finder AbstractSimilarityFinder) (. retriever (inert finder))) (kw (:: finder_kwargs NamedTuple) (call NamedTuple)) (kw (:: tagger AbstractTagger) (. retriever (inert tagger))) (kw (:: tagger_kwargs NamedTuple) (call NamedTuple)) (kw (:: filter AbstractTagFilter) (. retriever (inert filter))) (kw (:: filter_kwargs NamedTuple) (call NamedTuple)) (kw (:: reranker AbstractReranker) (. retriever (inert reranker))) (kw (:: reranker_kwargs NamedTuple) (call NamedTuple)) (kw cost_tracker (call (curly (. Threads (inert Atomic)) Float64) 0.0)) (... kwargs)) (:: retriever AbstractRetriever) (:: index AbstractDocumentIndex) (:: question AbstractString)) (block (= rephraser_kwargs_ (if (call isempty api_kwargs) rephraser_kwargs (call merge rephraser_kwargs (tuple (parameters api_kwargs))))) (= rephrased_questions (call rephrase (parameters (kw verbose (call > verbose 1)) cost_tracker (... rephraser_kwargs_)) rephraser question)) (= embeddings (if (call HasEmbeddings index) (block (= embedder_kwargs_ (if (call isempty api_kwargs) embedder_kwargs (call merge embedder_kwargs (tuple (parameters api_kwargs))))) (= embeddings (call get_embeddings (parameters (kw verbose (call > verbose 1)) cost_tracker (... embedder_kwargs_)) embedder rephrased_questions))) (block (= embeddings (call hcat (... (comprehension (generator (ref Float32) (= x rephrased_questions))))))))) (= keywords (if (call HasKeywords index) (block (= keywords (call get_keywords (parameters (kw verbose (call > verbose 1)) (... processor_kwargs) (kw return_keywords true)) processor rephrased_questions)) (&& (call >= verbose 1) (|| (call isa keywords (curly AbstractVector (<: (curly AbstractVector (<: AbstractString))))) (macrocall @warn :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1105 =#) (string "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " (call typeof processor) ").")))) keywords) (block (comprehension (generator (ref String) (= x rephrased_questions)))))) (= finder_kwargs_ (if (call isempty api_kwargs) finder_kwargs (call merge finder_kwargs (tuple (parameters api_kwargs))))) (= emb_candidates (call find_closest (parameters (kw verbose (call > verbose 1)) top_k (... finder_kwargs_)) finder index embeddings keywords)) (= tagger_kwargs_ (if (call isempty api_kwargs) tagger_kwargs (call merge tagger_kwargs (tuple (parameters api_kwargs))))) (= tags (call get_tags (parameters (kw verbose (call > verbose 1)) cost_tracker (... tagger_kwargs_)) tagger rephrased_questions)) (= filter_kwargs_ (if (call isempty api_kwargs) filter_kwargs (call merge filter_kwargs (tuple (parameters api_kwargs))))) (= tag_candidates (call find_tags (parameters (kw verbose (call > verbose 1)) (... filter_kwargs_)) filter index tags)) (= filtered_candidates (if (call isnothing tag_candidates) emb_candidates (call & emb_candidates tag_candidates))) (= reranker_kwargs_ (if (call isempty api_kwargs) reranker_kwargs (call merge reranker_kwargs (tuple (parameters api_kwargs))))) (= reranked_candidates (call rerank (parameters top_n (kw verbose (call > verbose 1)) cost_tracker (... reranker_kwargs_)) reranker index question filtered_candidates)) (&& (call > verbose 0) (macrocall @info :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1142 =#) (string "Retrieval done. Identified " (call length (call positions reranked_candidates)) " chunks, total cost: \$" (call round (ref cost_tracker) (kw digits 2)) "."))) (= result (call RAGResult (parameters question (kw answer nothing) rephrased_questions (kw final_answer nothing) (kw context (call collect (ref index reranked_candidates (inert chunks) (kw sorted true)))) (kw sources (call collect (ref index reranked_candidates (inert sources) (kw sorted true)))) emb_candidates tag_candidates filtered_candidates reranked_candidates))) (return result))))  └─ @ /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947  Stacktrace:  [1] expand_forms_2(ctx::Base.JuliaLowering.MacroExpansionContext{Dict{Symbol, Dict{Int64, Any}}}, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}})  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/desugaring.jl:4448  [2] core_lowering_hook(code::Any, mod::Module, file::String, line::UInt64, world::UInt64, _warn::Bool)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/hooks.jl:30  [3] include(mapexpr::Function, mod::Module, _path::String)  @ Base Base.jl:327  [4] top-level scope  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:1  [5] macro expansion  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:47 [inlined]  [6] eval(m::Module, e::Any)  @ Core boot.jl:521  [7] _eval(mod::Module, iter::Base.JuliaLowering.LoweringIterator{Dict{Symbol, Dict{Int64, Any}}}; soft_scope::Nothing)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:545  [8] eval(mod::Module, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}; macro_world::UInt64, soft_scope::Nothing, opts::@Kwargs{expr_compat_mode::Bool})  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:518 [inlined]  [9] top-level scope  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:1  [10] macro expansion  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:1 [inlined]  [11] include(mapexpr::Function, mod::Module, _path::String)  @ Base Base.jl:327  [12] top-level scope  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:1  [13] macro expansion  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:18 [inlined]  [14] eval(m::Module, e::Any)  @ Core boot.jl:521  [15] _eval(mod::Module, iter::Base.JuliaLowering.LoweringIterator{Dict{Symbol, Dict{Int64, Any}}}; soft_scope::Nothing)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:545  [16] eval(mod::Module, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}; macro_world::UInt64, soft_scope::Nothing, opts::@Kwargs{expr_compat_mode::Bool})  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:518 [inlined]  [17] top-level scope  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:1  [18] macro expansion  @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:1 [inlined]  [19] include(mapexpr::Function, mod::Module, _path::String)  @ Base Base.jl:327  [20] top-level scope  @ ~/.julia/packages/PromptingTools/wvgCI/src/PromptingTools.jl:120  [21] include(mod::Module, _path::String)  @ Base Base.jl:326  [22] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)  @ Base loading.jl:3296  [23] top-level scope  @ stdin:5  [24] eval(m::Module, e::Any)  @ Core boot.jl:521  [25] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)  @ Base loading.jl:3132  [26] include_string(m::Module, txt::String, fname::String)  @ Base loading.jl:3142 [inlined]  [27] exec_options(opts::Base.JLOptions)  @ Base client.jl:353  [28] _start()  @ Base client.jl:596 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:1 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:1 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/PromptingTools.jl:1 in expression starting at stdin:5 ERROR: LoadError: Precompiled image Base.PkgId(Base.UUID("670122d1-24a8-4d70-bfce-740807c42192"), "PromptingTools") not available with flags CacheFlags(; use_pkgimages=false, debug_level=1, check_bounds=1, inline=true, opt_level=0) Stacktrace:  [1] error(s::String)  @ Base error.jl:56  [2] __require_prelocked(pkg::Base.PkgId, env::String)  @ Base loading.jl:2837  [3] _require_prelocked(uuidkey::Base.PkgId, env::String)  @ Base loading.jl:2685  [4] macro expansion  @ loading.jl:2599 [inlined]  [5] macro expansion  @ lock.jl:376 [inlined]  [6] __require(into::Module, mod::Symbol)  @ Base loading.jl:2563  [7] require(into::Module, mod::Symbol)  @ Base loading.jl:2539 [inlined]  [8] eval_import_path(at::Module, from::Nothing, path::Expr, keyword::String)  @ Base module.jl:36 [inlined]  [9] eval_import_path_all(at::Module, path::Expr, keyword::String)  @ Base module.jl:60  [10] _eval_using(to::Module, path::Expr, flags::UInt8)  @ Base module.jl:137 [inlined]  [11] eval_using(to::Module, path::Expr)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/runtime.jl:207  [12] top-level scope  @ ~/.julia/packages/PromptingTools/wvgCI/ext/MarkdownPromptingToolsExt.jl:3  [13] include(mod::Module, _path::String)  @ Base Base.jl:326  [14] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)  @ Base loading.jl:3296  [15] top-level scope  @ stdin:5  [16] eval(m::Module, e::Any)  @ Core boot.jl:521  [17] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)  @ Base loading.jl:3132  [18] include_string(m::Module, txt::String, fname::String)  @ Base loading.jl:3142 [inlined]  [19] exec_options(opts::Base.JLOptions)  @ Base client.jl:353  [20] _start()  @ Base client.jl:596 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/ext/MarkdownPromptingToolsExt.jl:1 in expression starting at stdin:5 ERROR: LoadError: Precompiled image Base.PkgId(Base.UUID("670122d1-24a8-4d70-bfce-740807c42192"), "PromptingTools") not available with flags CacheFlags(; use_pkgimages=false, debug_level=1, check_bounds=1, inline=true, opt_level=0) Stacktrace:  [1] error(s::String)  @ Base error.jl:56  [2] __require_prelocked(pkg::Base.PkgId, env::String)  @ Base loading.jl:2837  [3] _require_prelocked(uuidkey::Base.PkgId, env::String)  @ Base loading.jl:2685  [4] macro expansion  @ loading.jl:2599 [inlined]  [5] macro expansion  @ lock.jl:376 [inlined]  [6] __require(into::Module, mod::Symbol)  @ Base loading.jl:2563  [7] require(into::Module, mod::Symbol)  @ Base loading.jl:2539 [inlined]  [8] eval_import_path(at::Module, from::Nothing, path::Expr, keyword::String)  @ Base module.jl:36 [inlined]  [9] _eval_import(imported::Bool, to::Module, from::Nothing, paths::Expr)  @ Base module.jl:111  [10] eval_import(imported::Bool, to::Module, from::Nothing, paths::Expr)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/runtime.jl:195  [11] top-level scope  @ ~/.julia/packages/DocumenterDescriptions/kTXBL/src/DocumenterDescriptions.jl:4  [12] include(mod::Module, _path::String)  @ Base Base.jl:326  [13] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)  @ Base loading.jl:3296  [14] top-level scope  @ stdin:5  [15] eval(m::Module, e::Any)  @ Core boot.jl:521  [16] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)  @ Base loading.jl:3132  [17] include_string(m::Module, txt::String, fname::String)  @ Base loading.jl:3142 [inlined]  [18] exec_options(opts::Base.JLOptions)  @ Base client.jl:353  [19] _start()  @ Base client.jl:596 in expression starting at /home/pkgeval/.julia/packages/DocumenterDescriptions/kTXBL/src/DocumenterDescriptions.jl:1 in expression starting at stdin:5 3 dependencies had output during precompilation: ┌ DocumenterDescriptions │ [Output was shown above] └ ┌ PromptingTools → MarkdownPromptingToolsExt │ ERROR: LoadError: Precompiled image Base.PkgId(Base.UUID("670122d1-24a8-4d70-bfce-740807c42192"), "PromptingTools") not available with flags CacheFlags(; use_pkgimages=false, debug_level=1, check_bounds=1, inline=true, opt_level=0) │ Stacktrace: │ [1] error(s::String) │ @ Base error.jl:56 │ [2] __require_prelocked(pkg::Base.PkgId, env::String) │ @ Base loading.jl:2837 │ [3] _require_prelocked(uuidkey::Base.PkgId, env::String) │ @ Base loading.jl:2685 │ [4] macro expansion │ @ loading.jl:2599 [inlined] │ [5] macro expansion │ @ lock.jl:376 [inlined] │ [6] __require(into::Module, mod::Symbol) │ @ Base loading.jl:2563 │ [7] require(into::Module, mod::Symbol) │ @ Base loading.jl:2539 [inlined] │ [8] eval_import_path(at::Module, from::Nothing, path::Expr, keyword::String) │ @ Base module.jl:36 [inlined] │ [9] eval_import_path_all(at::Module, path::Expr, keyword::String) │ @ Base module.jl:60 │ [10] _eval_using(to::Module, path::Expr, flags::UInt8) │ @ Base module.jl:137 [inlined] │ [11] eval_using(to::Module, path::Expr) │ @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/runtime.jl:207 │ [12] top-level scope │ @ ~/.julia/packages/PromptingTools/wvgCI/ext/MarkdownPromptingToolsExt.jl:3 │ [13] include(mod::Module, _path::String) │ @ Base Base.jl:326 │ [14] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing) │ @ Base loading.jl:3296 │ [15] top-level scope │ @ stdin:5 │ [16] eval(m::Module, e::Any) │ @ Core boot.jl:521 │ [17] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) │ @ Base loading.jl:3132 │ [18] include_string(m::Module, txt::String, fname::String) │ @ Base loading.jl:3142 [inlined] │ [19] exec_options(opts::Base.JLOptions) │ @ Base client.jl:353 │ [20] _start() │ @ Base client.jl:596 │ in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/ext/MarkdownPromptingToolsExt.jl:1 │ in expression starting at stdin:5 └ ┌ PromptingTools │ ┌ Warning: OPENAI_API_KEY variable not set! OpenAI models will not be available - set API key directly via `PromptingTools.OPENAI_API_KEY=`! │ └ @ PromptingTools ~/.julia/packages/PromptingTools/wvgCI/src/user_preferences.jl:189 │ ┌ Info: JuliaLowering threw given input: │ │ code = │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947 =# Core.@doc " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" function retrieve(retriever::AbstractRetriever, index::AbstractDocumentIndex, question::AbstractString; verbose::Integer = 1, top_k::Integer = 100, top_n::Integer = 5, api_kwargs::NamedTuple = NamedTuple(), rephraser::AbstractRephraser = retriever.rephraser, rephraser_kwargs::NamedTuple = NamedTuple(), embedder::AbstractEmbedder = retriever.embedder, embedder_kwargs::NamedTuple = NamedTuple(), processor::AbstractProcessor = retriever.processor, processor_kwargs::NamedTuple = NamedTuple(), finder::AbstractSimilarityFinder = retriever.finder, finder_kwargs::NamedTuple = NamedTuple(), tagger::AbstractTagger = retriever.tagger, tagger_kwargs::NamedTuple = NamedTuple(), filter::AbstractTagFilter = retriever.filter, filter_kwargs::NamedTuple = NamedTuple(), reranker::AbstractReranker = retriever.reranker, reranker_kwargs::NamedTuple = NamedTuple(), cost_tracker = Threads.Atomic{Float64}(0.0), kwargs...) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1059 =# │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1083 =# │ │ rephraser_kwargs_ = if isempty(api_kwargs) │ │ rephraser_kwargs │ │ else │ │ merge(rephraser_kwargs, (; api_kwargs)) │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1085 =# │ │ rephrased_questions = rephrase(rephraser, question; verbose = verbose > 1, cost_tracker, rephraser_kwargs_...) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1089 =# │ │ embeddings = if HasEmbeddings(index) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1090 =# │ │ embedder_kwargs_ = if isempty(api_kwargs) │ │ embedder_kwargs │ │ else │ │ merge(embedder_kwargs, (; api_kwargs)) │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1092 =# │ │ embeddings = get_embeddings(embedder, rephrased_questions; verbose = verbose > 1, cost_tracker, embedder_kwargs_...) │ │ else │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1095 =# │ │ embeddings = hcat([Float32[] for x = rephrased_questions]...) │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1099 =# │ │ keywords = if HasKeywords(index) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1101 =# │ │ keywords = get_keywords(processor, rephrased_questions; verbose = verbose > 1, processor_kwargs..., return_keywords = true) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1104 =# │ │ verbose >= 1 && (keywords isa AbstractVector{<:AbstractVector{<:AbstractString}} || #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1105 =# @warn("Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: $(typeof(processor))).")) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1106 =# │ │ keywords │ │ else │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1108 =# │ │ [String[] for x = rephrased_questions] │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1111 =# │ │ finder_kwargs_ = if isempty(api_kwargs) │ │ finder_kwargs │ │ else │ │ merge(finder_kwargs, (; api_kwargs)) │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1113 =# │ │ emb_candidates = find_closest(finder, index, embeddings, keywords; verbose = verbose > 1, top_k, finder_kwargs_...) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1117 =# │ │ tagger_kwargs_ = if isempty(api_kwargs) │ │ tagger_kwargs │ │ else │ │ merge(tagger_kwargs, (; api_kwargs)) │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1119 =# │ │ tags = get_tags(tagger, rephrased_questions; verbose = verbose > 1, cost_tracker, tagger_kwargs_...) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1122 =# │ │ filter_kwargs_ = if isempty(api_kwargs) │ │ filter_kwargs │ │ else │ │ merge(filter_kwargs, (; api_kwargs)) │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1124 =# │ │ tag_candidates = find_tags(filter, index, tags; verbose = verbose > 1, filter_kwargs_...) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1129 =# │ │ filtered_candidates = if isnothing(tag_candidates) │ │ emb_candidates │ │ else │ │ emb_candidates & tag_candidates │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1136 =# │ │ reranker_kwargs_ = if isempty(api_kwargs) │ │ reranker_kwargs │ │ else │ │ merge(reranker_kwargs, (; api_kwargs)) │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1138 =# │ │ reranked_candidates = rerank(reranker, index, question, filtered_candidates; top_n, verbose = verbose > 1, cost_tracker, reranker_kwargs_...) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1141 =# │ │ verbose > 0 && #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1142 =# @info("Retrieval done. Identified $(length(positions(reranked_candidates))) chunks, total cost: \$$(round(cost_tracker[], digits = 2)).") │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1145 =# │ │ result = RAGResult(; question, answer = nothing, rephrased_questions, final_answer = nothing, context = collect(index[reranked_candidates, :chunks, sorted = true]), sources = collect(index[reranked_candidates, :sources, sorted = true]), emb_candidates, tag_candidates, filtered_candidates, reranked_candidates) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1158 =# │ │ return result │ │ end) │ │ st0 = │ │ SyntaxTree with attributes mod,kind,var_id,toplevel_pure,scope_type,macro_source,name_val,syntax_flags,meta,scope_layer,value,jl_source,is_toplevel_thunk,source,__macro_ctx__ │ │ [macrocall] │ │ │ @doc :: Identifier │ mod │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947 =#) :: Value │ │ │ " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" :: Value │ │ │ [function] │ │ │ [call] │ │ │ retrieve :: Identifier │ │ │ [parameters] │ │ │ [kw] │ │ │ [::] │ │ │ verbose :: Identifier │ │ │ Integer :: Identifier │ │ │ 1 :: Value │ │ │ [kw] │ │ │ [::] │ │ │ top_k :: Identifier │ │ │ Integer :: Identifier │ │ │ 100 :: Value │ │ │ [kw] │ │ │ [::] │ │ │ top_n :: Identifier │ │ │ Integer :: Identifier │ │ │ 5 :: Value │ │ │ [kw] │ │ │ [::] │ │ │ api_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ rephraser :: Identifier │ │ │ AbstractRephraser :: Identifier │ │ │ [.] │ │ │ retriever :: Identifier │ │ │ [inert] │ │ │ rephraser :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ rephraser_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ embedder :: Identifier │ │ │ AbstractEmbedder :: Identifier │ │ │ [.] │ │ │ retriever :: Identifier │ │ │ [inert] │ │ │ embedder :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ embedder_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ processor :: Identifier │ │ │ AbstractProcessor :: Identifier │ │ │ [.] │ │ │ retriever :: Identifier │ │ │ [inert] │ │ │ processor :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ processor_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ finder :: Identifier │ │ │ AbstractSimilarityFinder :: Identifier │ │ │ [.] │ │ │ retriever :: Identifier │ │ │ [inert] │ │ │ finder :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ finder_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ tagger :: Identifier │ │ │ AbstractTagger :: Identifier │ │ │ [.] │ │ │ retriever :: Identifier │ │ │ [inert] │ │ │ tagger :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ tagger_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ filter :: Identifier │ │ │ AbstractTagFilter :: Identifier │ │ │ [.] │ │ │ retriever :: Identifier │ │ │ [inert] │ │ │ filter :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ filter_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ reranker :: Identifier │ │ │ AbstractReranker :: Identifier │ │ │ [.] │ │ │ retriever :: Identifier │ │ │ [inert] │ │ │ reranker :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ reranker_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ cost_tracker :: Identifier │ │ │ [call] │ │ │ [curly] │ │ │ [.] │ │ │ Threads :: Identifier │ │ │ [inert] │ │ │ Atomic :: Identifier │ │ │ Float64 :: Identifier │ │ │ 0.0 :: Value │ │ │ [...] │ │ │ kwargs :: Identifier │ │ │ [::] │ │ │ retriever :: Identifier │ │ │ AbstractRetriever :: Identifier │ │ │ [::] │ │ │ index :: Identifier │ │ │ AbstractDocumentIndex :: Identifier │ │ │ [::] │ │ │ question :: Identifier │ │ │ AbstractString :: Identifier │ │ │ [block] │ │ │ [=] │ │ │ rephraser_kwargs_ :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ │ │ api_kwargs :: Identifier │ │ │ rephraser_kwargs :: Identifier │ │ │ [call] │ │ │ merge :: Identifier │ │ │ rephraser_kwargs :: Identifier │ │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ │ │ [=] │ │ │ rephrased_questions :: Identifier │ │ │ [call] │ │ │ rephrase :: Identifier │ │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ cost_tracker :: Identifier │ │ │ [...] │ │ │ rephraser_kwargs_ :: Identifier │ │ │ rephraser :: Identifier │ │ │ question :: Identifier │ │ │ [=] │ │ │ embeddings :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ HasEmbeddings :: Identifier │ │ │ index :: Identifier │ │ │ [block] │ │ │ [=] │ │ │ embedder_kwargs_ :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ │ │ api_kwargs :: Identifier │ │ │ embedder_kwargs :: Identifier │ │ │ [call] │ │ │ merge :: Identifier │ │ │ embedder_kwargs :: Identifier │ │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ │ │ [=] │ │ │ embeddings :: Identifier │ │ │ [call] │ │ │ get_embeddings :: Identifier │ │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ cost_tracker :: Identifier │ │ │ [...] │ │ │ embedder_kwargs_ :: Identifier │ │ │ embedder :: Identifier │ │ │ rephrased_questions :: Identifier │ │ │ [block] │ │ │ [=] │ │ │ embeddings :: Identifier │ │ │ [call] │ │ │ hcat :: Identifier │ │ │ [...] │ │ │ [comprehension] │ │ │ [generator] │ │ │ [ref] │ │ │ Float32 :: Identifier │ │ │ [=] │ │ │ x :: Identifier │ │ │ rephrased_questions :: Identifier │ │ │ [=] │ │ │ keywords :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ HasKeywords :: Identifier │ │ │ index :: Identifier │ │ │ [block] │ │ │ [=] │ │ │ keywords :: Identifier │ │ │ [call] │ │ │ get_keywords :: Identifier │ │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ [...] │ │ │ processor_kwargs :: Identifier │ │ │ [kw] │ │ │ return_keywords :: Identifier │ │ │ true :: Value │ │ │ processor :: Identifier │ │ │ rephrased_questions :: Identifier │ │ │ [&&] │ │ │ [call] │ │ │ >= :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ [||] │ │ │ [call] │ │ │ isa :: Identifier │ │ │ keywords :: Identifier │ │ │ [curly] │ │ │ AbstractVector :: Identifier │ │ │ [<:] │ │ │ [curly] │ │ │ AbstractVector :: Identifier │ │ │ [<:] │ │ │ AbstractString :: Identifier │ │ │ [macrocall] │ │ │ @warn :: Identifier │ │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1105 =#) :: Value │ │ │ [string] │ │ │ "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " :: Value │ │ │ [call] │ │ │ typeof :: Identifier │ │ │ processor :: Identifier │ │ │ ")." :: Value │ │ │ keywords :: Identifier │ │ │ [block] │ │ │ [comprehension] │ │ │ [generator] │ │ │ [ref] │ │ │ String :: Identifier │ │ │ [=] │ │ │ x :: Identifier │ │ │ rephrased_questions :: Identifier │ │ │ [=] │ │ │ finder_kwargs_ :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ │ │ api_kwargs :: Identifier │ │ │ finder_kwargs :: Identifier │ │ │ [call] │ │ │ merge :: Identifier │ │ │ finder_kwargs :: Identifier │ │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ │ │ [=] │ │ │ emb_candidates :: Identifier │ │ │ [call] │ │ │ find_closest :: Identifier │ │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ top_k :: Identifier │ │ │ [...] │ │ │ finder_kwargs_ :: Identifier │ │ │ finder :: Identifier │ │ │ index :: Identifier │ │ │ embeddings :: Identifier │ │ │ keywords :: Identifier │ │ │ [=] │ │ │ tagger_kwargs_ :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ │ │ api_kwargs :: Identifier │ │ │ tagger_kwargs :: Identifier │ │ │ [call] │ │ │ merge :: Identifier │ │ │ tagger_kwargs :: Identifier │ │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ │ │ [=] │ │ │ tags :: Identifier │ │ │ [call] │ │ │ get_tags :: Identifier │ │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ cost_tracker :: Identifier │ │ │ [...] │ │ │ tagger_kwargs_ :: Identifier │ │ │ tagger :: Identifier │ │ │ rephrased_questions :: Identifier │ │ │ [=] │ │ │ filter_kwargs_ :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ │ │ api_kwargs :: Identifier │ │ │ filter_kwargs :: Identifier │ │ │ [call] │ │ │ merge :: Identifier │ │ │ filter_kwargs :: Identifier │ │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ │ │ [=] │ │ │ tag_candidates :: Identifier │ │ │ [call] │ │ │ find_tags :: Identifier │ │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ [...] │ │ │ filter_kwargs_ :: Identifier │ │ │ filter :: Identifier │ │ │ index :: Identifier │ │ │ tags :: Identifier │ │ │ [=] │ │ │ filtered_candidates :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ isnothing :: Identifier │ │ │ tag_candidates :: Identifier │ │ │ emb_candidates :: Identifier │ │ │ [call] │ │ │ & :: Identifier │ │ │ emb_candidates :: Identifier │ │ │ tag_candidates :: Identifier │ │ │ [=] │ │ │ reranker_kwargs_ :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ │ │ api_kwargs :: Identifier │ │ │ reranker_kwargs :: Identifier │ │ │ [call] │ │ │ merge :: Identifier │ │ │ reranker_kwargs :: Identifier │ │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ │ │ [=] │ │ │ reranked_candidates :: Identifier │ │ │ [call] │ │ │ rerank :: Identifier │ │ │ [parameters] │ │ │ top_n :: Identifier │ │ │ [kw] │ │ │ verbose :: Identifier │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ cost_tracker :: Identifier │ │ │ [...] │ │ │ reranker_kwargs_ :: Identifier │ │ │ reranker :: Identifier │ │ │ index :: Identifier │ │ │ question :: Identifier │ │ │ filtered_candidates :: Identifier │ │ │ [&&] │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 0 :: Value │ │ │ [macrocall] │ │ │ @info :: Identifier │ │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1142 =#) :: Value │ │ │ [string] │ │ │ "Retrieval done. Identified " :: Value │ │ │ [call] │ │ │ length :: Identifier │ │ │ [call] │ │ │ positions :: Identifier │ │ │ reranked_candidates :: Identifier │ │ │ " chunks, total cost: \$" :: Value │ │ │ [call] │ │ │ round :: Identifier │ │ │ [ref] │ │ │ cost_tracker :: Identifier │ │ │ [kw] │ │ │ digits :: Identifier │ │ │ 2 :: Value │ │ │ "." :: Value │ │ │ [=] │ │ │ result :: Identifier │ │ │ [call] │ │ │ RAGResult :: Identifier │ │ │ [parameters] │ │ │ question :: Identifier │ │ │ [kw] │ │ │ answer :: Identifier │ │ │ nothing :: Identifier │ │ │ rephrased_questions :: Identifier │ │ │ [kw] │ │ │ final_answer :: Identifier │ │ │ nothing :: Identifier │ │ │ [kw] │ │ │ context :: Identifier │ │ │ [call] │ │ │ collect :: Identifier │ │ │ [ref] │ │ │ index :: Identifier │ │ │ reranked_candidates :: Identifier │ │ │ [inert] │ │ │ chunks :: Identifier │ │ │ [kw] │ │ │ sorted :: Identifier │ │ │ true :: Value │ │ │ [kw] │ │ │ sources :: Identifier │ │ │ [call] │ │ │ collect :: Identifier │ │ │ [ref] │ │ │ index :: Identifier │ │ │ reranked_candidates :: Identifier │ │ │ [inert] │ │ │ sources :: Identifier │ │ │ [kw] │ │ │ sorted :: Identifier │ │ │ true :: Value │ │ │ emb_candidates :: Identifier │ │ │ tag_candidates :: Identifier │ │ │ filtered_candidates :: Identifier │ │ │ reranked_candidates :: Identifier │ │ │ [return] │ │ │ result :: Identifier │ │ │ │ │ st1 = │ │ SyntaxTree with attributes mod,kind,var_id,toplevel_pure,scope_type,macro_source,name_val,syntax_flags,meta,scope_layer,value,jl_source,is_toplevel_thunk,source │ │ [block] │ │ │ [=] │ │ │ val :: Identifier │ scope_layer=3 │ │ [function] │ │ │ [call] │ │ │ retrieve :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ [kw] │ │ │ [::] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ Integer :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ [kw] │ │ │ [::] │ │ │ top_k :: Identifier │ scope_layer=1 │ │ Integer :: Identifier │ scope_layer=1 │ │ 100 :: Value │ macro_source=485 │ │ [kw] │ │ │ [::] │ │ │ top_n :: Identifier │ scope_layer=1 │ │ Integer :: Identifier │ scope_layer=1 │ │ 5 :: Value │ macro_source=485 │ │ [kw] │ │ │ [::] │ │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ [::] │ │ │ rephraser :: Identifier │ scope_layer=1 │ │ AbstractRephraser :: Identifier │ scope_layer=1 │ │ [.] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ rephraser :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ rephraser_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ [::] │ │ │ embedder :: Identifier │ scope_layer=1 │ │ AbstractEmbedder :: Identifier │ scope_layer=1 │ │ [.] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ embedder :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ embedder_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ [::] │ │ │ processor :: Identifier │ scope_layer=1 │ │ AbstractProcessor :: Identifier │ scope_layer=1 │ │ [.] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ processor :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ processor_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ [::] │ │ │ finder :: Identifier │ scope_layer=1 │ │ AbstractSimilarityFinder :: Identifier │ scope_layer=1 │ │ [.] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ finder :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ finder_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ [::] │ │ │ tagger :: Identifier │ scope_layer=1 │ │ AbstractTagger :: Identifier │ scope_layer=1 │ │ [.] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ tagger :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ tagger_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ [::] │ │ │ filter :: Identifier │ scope_layer=1 │ │ AbstractTagFilter :: Identifier │ scope_layer=1 │ │ [.] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ filter :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ filter_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ [::] │ │ │ reranker :: Identifier │ scope_layer=1 │ │ AbstractReranker :: Identifier │ scope_layer=1 │ │ [.] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ reranker :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ reranker_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ cost_tracker :: Identifier │ scope_layer=1 │ │ [call] │ │ │ [curly] │ │ │ [.] │ │ │ Threads :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ Atomic :: Identifier │ │ │ Float64 :: Identifier │ scope_layer=1 │ │ 0.0 :: Value │ macro_source=485 │ │ [...] │ │ │ kwargs :: Identifier │ scope_layer=1 │ │ [::] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ AbstractRetriever :: Identifier │ scope_layer=1 │ │ [::] │ │ │ index :: Identifier │ scope_layer=1 │ │ AbstractDocumentIndex :: Identifier │ scope_layer=1 │ │ [::] │ │ │ question :: Identifier │ scope_layer=1 │ │ AbstractString :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ rephraser_kwargs_ :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ scope_layer=1 │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ rephraser_kwargs :: Identifier │ scope_layer=1 │ │ [call] │ │ │ merge :: Identifier │ scope_layer=1 │ │ rephraser_kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ [=] │ │ │ rephrased_questions :: Identifier │ scope_layer=1 │ │ [call] │ │ │ rephrase :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ cost_tracker :: Identifier │ scope_layer=1 │ │ [...] │ │ │ rephraser_kwargs_ :: Identifier │ scope_layer=1 │ │ rephraser :: Identifier │ scope_layer=1 │ │ question :: Identifier │ scope_layer=1 │ │ [=] │ │ │ embeddings :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ HasEmbeddings :: Identifier │ scope_layer=1 │ │ index :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ embedder_kwargs_ :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ scope_layer=1 │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ embedder_kwargs :: Identifier │ scope_layer=1 │ │ [call] │ │ │ merge :: Identifier │ scope_layer=1 │ │ embedder_kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ [=] │ │ │ embeddings :: Identifier │ scope_layer=1 │ │ [call] │ │ │ get_embeddings :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ cost_tracker :: Identifier │ scope_layer=1 │ │ [...] │ │ │ embedder_kwargs_ :: Identifier │ scope_layer=1 │ │ embedder :: Identifier │ scope_layer=1 │ │ rephrased_questions :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ embeddings :: Identifier │ scope_layer=1 │ │ [call] │ │ │ hcat :: Identifier │ scope_layer=1 │ │ [...] │ │ │ [comprehension] │ │ │ [generator] │ │ │ [ref] │ │ │ Float32 :: Identifier │ scope_layer=1 │ │ [=] │ │ │ x :: Identifier │ scope_layer=1 │ │ rephrased_questions :: Identifier │ scope_layer=1 │ │ [=] │ │ │ keywords :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ HasKeywords :: Identifier │ scope_layer=1 │ │ index :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ keywords :: Identifier │ scope_layer=1 │ │ [call] │ │ │ get_keywords :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ [...] │ │ │ processor_kwargs :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ return_keywords :: Identifier │ scope_layer=1 │ │ true :: Value │ macro_source=485 │ │ processor :: Identifier │ scope_layer=1 │ │ rephrased_questions :: Identifier │ scope_layer=1 │ │ [&&] │ │ │ [call] │ │ │ >= :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ [||] │ │ │ [call] │ │ │ isa :: Identifier │ scope_layer=1 │ │ keywords :: Identifier │ scope_layer=1 │ │ [curly] │ │ │ AbstractVector :: Identifier │ scope_layer=1 │ │ [<:] │ scope_layer=1 │ │ [curly] │ │ │ AbstractVector :: Identifier │ scope_layer=1 │ │ [<:] │ scope_layer=1 │ │ AbstractString :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [let] │ │ │ [block] │ macro_source=485 │ │ [block] │ │ │ [=] │ │ │ #1281#level :: Identifier │ scope_layer=1 │ │ Warn :: Identifier │ mod,scope_layer=1 │ │ [=] │ │ │ #1282#std_level :: Identifier │ scope_layer=1 │ │ #1281#level :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ >= :: Identifier │ mod,scope_layer=1 │ │ [.] │ │ │ #1282#std_level :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ level :: Identifier │ │ │ [ref] │ macro_source=485 │ │ Base.Threads.Atomic{Int32}(-1000) :: Value │ macro_source=485 │ │ [block] │ │ │ [=] │ │ │ #1283#group :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ retrieval :: Identifier │ │ │ [=] │ │ │ #1284#_module :: Identifier │ scope_layer=1 │ │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ │ [=] │ │ │ #1285#logger :: Identifier │ scope_layer=1 │ │ [call] │ │ │ Base.CoreLogging.current_logger_for_env :: Value │ macro_source=485 │ │ #1282#std_level :: Identifier │ scope_layer=1 │ │ #1283#group :: Identifier │ scope_layer=1 │ │ #1284#_module :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ ! :: Identifier │ mod,scope_layer=1 │ │ [call] │ │ │ === :: Identifier │ mod,scope_layer=1 │ │ #1285#logger :: Identifier │ scope_layer=1 │ │ nothing :: Identifier │ mod,scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ #1286#id :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ PromptingTools_Experimental_RAGTools_7aa4399b :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ │ Base.CoreLogging.shouldlog :: Value │ macro_source=485 │ │ #1285#logger :: Identifier │ scope_layer=1 │ │ #1281#level :: Identifier │ scope_layer=1 │ │ #1284#_module :: Identifier │ scope_layer=1 │ │ #1283#group :: Identifier │ scope_layer=1 │ │ #1286#id :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ #1287#file :: Identifier │ scope_layer=1 │ │ "/home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ │ [if] │ │ │ [call] │ │ │ isa :: Identifier │ mod,scope_layer=1 │ │ #1287#file :: Identifier │ scope_layer=1 │ │ String :: Identifier │ mod,scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ #1287#file :: Identifier │ scope_layer=1 │ │ [call] │ │ │ [.] │ │ │ Base :: Identifier │ mod,scope_layer=1 │ │ [inert] │ │ │ fixup_stdlib_path :: Identifier │ │ │ #1287#file :: Identifier │ scope_layer=1 │ │ [=] │ │ │ #1288#line :: Identifier │ scope_layer=1 │ │ 1105 :: Value │ macro_source=485 │ │ [local] │ │ │ #1289#msg :: Identifier │ scope_layer=1 │ │ #1290#kwargs :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [block] │ │ │ [try] │ │ │ [block] │ │ │ [=] │ │ │ #1289#msg :: Identifier │ scope_layer=1 │ │ [string] │ │ │ "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " :: Value │ macro_source=485 │ │ [call] │ │ │ typeof :: Identifier │ scope_layer=1 │ │ processor :: Identifier │ scope_layer=1 │ │ ")." :: Value │ macro_source=485 │ │ [=] │ │ │ #1290#kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ macro_source=485 │ │ [parameters] │ macro_source=485 │ │ true :: Value │ macro_source=485 │ │ #1303#err :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [call] │ │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ │ Base.CoreLogging.logging_error :: Value │ macro_source=485 │ │ #1285#logger :: Identifier │ scope_layer=1 │ │ #1281#level :: Identifier │ scope_layer=1 │ │ #1284#_module :: Identifier │ scope_layer=1 │ │ #1283#group :: Identifier │ scope_layer=1 │ │ #1286#id :: Identifier │ scope_layer=1 │ │ #1287#file :: Identifier │ scope_layer=1 │ │ #1288#line :: Identifier │ scope_layer=1 │ │ #1303#err :: Identifier │ scope_layer=1 │ │ true :: Value │ macro_source=485 │ │ false :: Value │ macro_source=485 │ │ [block] │ │ │ [if] │ │ │ [isdefined] │ │ │ #1289#msg :: Identifier │ scope_layer=1 │ │ nothing :: Value │ macro_source=485 │ │ [call] │ │ │ throw :: Identifier │ mod,scope_layer=1 │ │ [call] │ │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ │ [if] │ │ │ [isdefined] │ │ │ #1290#kwargs :: Identifier │ scope_layer=1 │ │ nothing :: Value │ macro_source=485 │ │ [call] │ │ │ throw :: Identifier │ mod,scope_layer=1 │ │ [call] │ │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ │ [call] │ │ │ Base.CoreLogging.handle_message_nothrow :: Value │ macro_source=485 │ │ [parameters] │ │ │ [...] │ │ │ #1290#kwargs :: Identifier │ scope_layer=1 │ │ #1285#logger :: Identifier │ scope_layer=1 │ │ #1281#level :: Identifier │ scope_layer=1 │ │ #1289#msg :: Identifier │ scope_layer=1 │ │ #1284#_module :: Identifier │ scope_layer=1 │ │ #1283#group :: Identifier │ scope_layer=1 │ │ #1286#id :: Identifier │ scope_layer=1 │ │ #1287#file :: Identifier │ scope_layer=1 │ │ #1288#line :: Identifier │ scope_layer=1 │ │ nothing :: Identifier │ mod,scope_layer=1 │ │ keywords :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [comprehension] │ │ │ [generator] │ │ │ [ref] │ │ │ String :: Identifier │ scope_layer=1 │ │ [=] │ │ │ x :: Identifier │ scope_layer=1 │ │ rephrased_questions :: Identifier │ scope_layer=1 │ │ [=] │ │ │ finder_kwargs_ :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ scope_layer=1 │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ finder_kwargs :: Identifier │ scope_layer=1 │ │ [call] │ │ │ merge :: Identifier │ scope_layer=1 │ │ finder_kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ [=] │ │ │ emb_candidates :: Identifier │ scope_layer=1 │ │ [call] │ │ │ find_closest :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ top_k :: Identifier │ scope_layer=1 │ │ [...] │ │ │ finder_kwargs_ :: Identifier │ scope_layer=1 │ │ finder :: Identifier │ scope_layer=1 │ │ index :: Identifier │ scope_layer=1 │ │ embeddings :: Identifier │ scope_layer=1 │ │ keywords :: Identifier │ scope_layer=1 │ │ [=] │ │ │ tagger_kwargs_ :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ scope_layer=1 │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ tagger_kwargs :: Identifier │ scope_layer=1 │ │ [call] │ │ │ merge :: Identifier │ scope_layer=1 │ │ tagger_kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ [=] │ │ │ tags :: Identifier │ scope_layer=1 │ │ [call] │ │ │ get_tags :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ cost_tracker :: Identifier │ scope_layer=1 │ │ [...] │ │ │ tagger_kwargs_ :: Identifier │ scope_layer=1 │ │ tagger :: Identifier │ scope_layer=1 │ │ rephrased_questions :: Identifier │ scope_layer=1 │ │ [=] │ │ │ filter_kwargs_ :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ scope_layer=1 │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ filter_kwargs :: Identifier │ scope_layer=1 │ │ [call] │ │ │ merge :: Identifier │ scope_layer=1 │ │ filter_kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ [=] │ │ │ tag_candidates :: Identifier │ scope_layer=1 │ │ [call] │ │ │ find_tags :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ [...] │ │ │ filter_kwargs_ :: Identifier │ scope_layer=1 │ │ filter :: Identifier │ scope_layer=1 │ │ index :: Identifier │ scope_layer=1 │ │ tags :: Identifier │ scope_layer=1 │ │ [=] │ │ │ filtered_candidates :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ isnothing :: Identifier │ scope_layer=1 │ │ tag_candidates :: Identifier │ scope_layer=1 │ │ emb_candidates :: Identifier │ scope_layer=1 │ │ [call] │ │ │ & :: Identifier │ scope_layer=1 │ │ emb_candidates :: Identifier │ scope_layer=1 │ │ tag_candidates :: Identifier │ scope_layer=1 │ │ [=] │ │ │ reranker_kwargs_ :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ scope_layer=1 │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ reranker_kwargs :: Identifier │ scope_layer=1 │ │ [call] │ │ │ merge :: Identifier │ scope_layer=1 │ │ reranker_kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ [=] │ │ │ reranked_candidates :: Identifier │ scope_layer=1 │ │ [call] │ │ │ rerank :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ top_n :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ cost_tracker :: Identifier │ scope_layer=1 │ │ [...] │ │ │ reranker_kwargs_ :: Identifier │ scope_layer=1 │ │ reranker :: Identifier │ scope_layer=1 │ │ index :: Identifier │ scope_layer=1 │ │ question :: Identifier │ scope_layer=1 │ │ filtered_candidates :: Identifier │ scope_layer=1 │ │ [&&] │ │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 0 :: Value │ macro_source=485 │ │ [block] │ │ │ [let] │ │ │ [block] │ macro_source=485 │ │ [block] │ │ │ [=] │ │ │ #1304#level :: Identifier │ scope_layer=1 │ │ Info :: Identifier │ mod,scope_layer=1 │ │ [=] │ │ │ #1305#std_level :: Identifier │ scope_layer=1 │ │ #1304#level :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ >= :: Identifier │ mod,scope_layer=1 │ │ [.] │ │ │ #1305#std_level :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ level :: Identifier │ │ │ [ref] │ macro_source=485 │ │ Base.Threads.Atomic{Int32}(-1000) :: Value │ macro_source=485 │ │ [block] │ │ │ [=] │ │ │ #1306#group :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ retrieval :: Identifier │ │ │ [=] │ │ │ #1307#_module :: Identifier │ scope_layer=1 │ │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ │ [=] │ │ │ #1308#logger :: Identifier │ scope_layer=1 │ │ [call] │ │ │ Base.CoreLogging.current_logger_for_env :: Value │ macro_source=485 │ │ #1305#std_level :: Identifier │ scope_layer=1 │ │ #1306#group :: Identifier │ scope_layer=1 │ │ #1307#_module :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ ! :: Identifier │ mod,scope_layer=1 │ │ [call] │ │ │ === :: Identifier │ mod,scope_layer=1 │ │ #1308#logger :: Identifier │ scope_layer=1 │ │ nothing :: Identifier │ mod,scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ #1309#id :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ PromptingTools_Experimental_RAGTools_30253e51 :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ │ Base.CoreLogging.shouldlog :: Value │ macro_source=485 │ │ #1308#logger :: Identifier │ scope_layer=1 │ │ #1304#level :: Identifier │ scope_layer=1 │ │ #1307#_module :: Identifier │ scope_layer=1 │ │ #1306#group :: Identifier │ scope_layer=1 │ │ #1309#id :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ #1310#file :: Identifier │ scope_layer=1 │ │ "/home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ │ [if] │ │ │ [call] │ │ │ isa :: Identifier │ mod,scope_layer=1 │ │ #1310#file :: Identifier │ scope_layer=1 │ │ String :: Identifier │ mod,scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ #1310#file :: Identifier │ scope_layer=1 │ │ [call] │ │ │ [.] │ │ │ Base :: Identifier │ mod,scope_layer=1 │ │ [inert] │ │ │ fixup_stdlib_path :: Identifier │ │ │ #1310#file :: Identifier │ scope_layer=1 │ │ [=] │ │ │ #1311#line :: Identifier │ scope_layer=1 │ │ 1142 :: Value │ macro_source=485 │ │ [local] │ │ │ #1312#msg :: Identifier │ scope_layer=1 │ │ #1313#kwargs :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [block] │ │ │ [try] │ │ │ [block] │ │ │ [=] │ │ │ #1312#msg :: Identifier │ scope_layer=1 │ │ [string] │ │ │ "Retrieval done. Identified " :: Value │ macro_source=485 │ │ [call] │ │ │ length :: Identifier │ scope_layer=1 │ │ [call] │ │ │ positions :: Identifier │ scope_layer=1 │ │ reranked_candidates :: Identifier │ scope_layer=1 │ │ " chunks, total cost: \$" :: Value │ macro_source=485 │ │ [call] │ │ │ round :: Identifier │ scope_layer=1 │ │ [ref] │ │ │ cost_tracker :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ digits :: Identifier │ scope_layer=1 │ │ 2 :: Value │ macro_source=485 │ │ "." :: Value │ macro_source=485 │ │ [=] │ │ │ #1313#kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ macro_source=485 │ │ [parameters] │ macro_source=485 │ │ true :: Value │ macro_source=485 │ │ #1326#err :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [call] │ │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ │ Base.CoreLogging.logging_error :: Value │ macro_source=485 │ │ #1308#logger :: Identifier │ scope_layer=1 │ │ #1304#level :: Identifier │ scope_layer=1 │ │ #1307#_module :: Identifier │ scope_layer=1 │ │ #1306#group :: Identifier │ scope_layer=1 │ │ #1309#id :: Identifier │ scope_layer=1 │ │ #1310#file :: Identifier │ scope_layer=1 │ │ #1311#line :: Identifier │ scope_layer=1 │ │ #1326#err :: Identifier │ scope_layer=1 │ │ true :: Value │ macro_source=485 │ │ false :: Value │ macro_source=485 │ │ [block] │ │ │ [if] │ │ │ [isdefined] │ │ │ #1312#msg :: Identifier │ scope_layer=1 │ │ nothing :: Value │ macro_source=485 │ │ [call] │ │ │ throw :: Identifier │ mod,scope_layer=1 │ │ [call] │ │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ │ [if] │ │ │ [isdefined] │ │ │ #1313#kwargs :: Identifier │ scope_layer=1 │ │ nothing :: Value │ macro_source=485 │ │ [call] │ │ │ throw :: Identifier │ mod,scope_layer=1 │ │ [call] │ │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ │ [call] │ │ │ Base.CoreLogging.handle_message_nothrow :: Value │ macro_source=485 │ │ [parameters] │ │ │ [...] │ │ │ #1313#kwargs :: Identifier │ scope_layer=1 │ │ #1308#logger :: Identifier │ scope_layer=1 │ │ #1304#level :: Identifier │ scope_layer=1 │ │ #1312#msg :: Identifier │ scope_layer=1 │ │ #1307#_module :: Identifier │ scope_layer=1 │ │ #1306#group :: Identifier │ scope_layer=1 │ │ #1309#id :: Identifier │ scope_layer=1 │ │ #1310#file :: Identifier │ scope_layer=1 │ │ #1311#line :: Identifier │ scope_layer=1 │ │ nothing :: Identifier │ mod,scope_layer=1 │ │ [=] │ │ │ result :: Identifier │ scope_layer=1 │ │ [call] │ │ │ RAGResult :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ question :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ answer :: Identifier │ scope_layer=1 │ │ nothing :: Identifier │ scope_layer=1 │ │ rephrased_questions :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ final_answer :: Identifier │ scope_layer=1 │ │ nothing :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ context :: Identifier │ scope_layer=1 │ │ [call] │ │ │ collect :: Identifier │ scope_layer=1 │ │ [ref] │ │ │ index :: Identifier │ scope_layer=1 │ │ reranked_candidates :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ chunks :: Identifier │ │ │ [kw] │ │ │ sorted :: Identifier │ scope_layer=1 │ │ true :: Value │ macro_source=485 │ │ [kw] │ │ │ sources :: Identifier │ scope_layer=1 │ │ [call] │ │ │ collect :: Identifier │ scope_layer=1 │ │ [ref] │ │ │ index :: Identifier │ scope_layer=1 │ │ reranked_candidates :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ sources :: Identifier │ │ │ [kw] │ │ │ sorted :: Identifier │ scope_layer=1 │ │ true :: Value │ macro_source=485 │ │ emb_candidates :: Identifier │ scope_layer=1 │ │ tag_candidates :: Identifier │ scope_layer=1 │ │ filtered_candidates :: Identifier │ scope_layer=1 │ │ reranked_candidates :: Identifier │ scope_layer=1 │ │ [return] │ │ │ result :: Identifier │ scope_layer=1 │ │ [call] │ │ │ Base.Docs.doc! :: Value │ macro_source=485 │ │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ │ [call] │ │ │ Base.Docs.Binding :: Value │ macro_source=485 │ │ PromptingTools.Experimental.RAGTools :: Value │ │ │ [inert] │ jl_source=L65 │ │ retrieve :: Identifier │ │ │ [call] │ macro_source=485 │ │ Base.Docs.docstr :: Value │ macro_source=485 │ │ [call] │ macro_source=485 │ │ Core.svec :: Value │ macro_source=485 │ │ " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" :: Value │ macro_source=485 │ │ [call] │ macro_source=485 │ │ Dict{Symbol, Any} :: Value │ macro_source=485 │ │ :path => "/home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ │ :linenumber => 947 :: Value │ macro_source=485 │ │ :module => PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ │ [curly] │ │ │ Union :: Identifier │ scope_layer=1 │ │ [curly] │ │ │ Tuple :: Identifier │ scope_layer=1 │ │ AbstractRetriever :: Identifier │ scope_layer=1 │ │ AbstractDocumentIndex :: Identifier │ scope_layer=1 │ │ AbstractString :: Identifier │ scope_layer=1 │ │ val :: Identifier │ scope_layer=3 │ │ │ │ file = "/home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl" │ │ line = 947 │ └ mod = PromptingTools.Experimental.RAGTools │ ERROR: LoadError: LoweringError: │ #= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1145 =# - invalid syntax: unknown form `kw` or number of arguments 2 │ Expression: │ (kw sorted true) │ Containing expressions: │ (kw context (call collect (ref index reranked_candidates (inert chunks) (kw sorted true)))) │ │ Detailed provenance: │ (kw sorted true) │ └─ (kw sorted true) │ ├─ @ /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1145 │ └─ (macrocall @doc :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947 =#) " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" (function (call retrieve (parameters (kw (:: verbose Integer) 1) (kw (:: top_k Integer) 100) (kw (:: top_n Integer) 5) (kw (:: api_kwargs NamedTuple) (call NamedTuple)) (kw (:: rephraser AbstractRephraser) (. retriever (inert rephraser))) (kw (:: rephraser_kwargs NamedTuple) (call NamedTuple)) (kw (:: embedder AbstractEmbedder) (. retriever (inert embedder))) (kw (:: embedder_kwargs NamedTuple) (call NamedTuple)) (kw (:: processor AbstractProcessor) (. retriever (inert processor))) (kw (:: processor_kwargs NamedTuple) (call NamedTuple)) (kw (:: finder AbstractSimilarityFinder) (. retriever (inert finder))) (kw (:: finder_kwargs NamedTuple) (call NamedTuple)) (kw (:: tagger AbstractTagger) (. retriever (inert tagger))) (kw (:: tagger_kwargs NamedTuple) (call NamedTuple)) (kw (:: filter AbstractTagFilter) (. retriever (inert filter))) (kw (:: filter_kwargs NamedTuple) (call NamedTuple)) (kw (:: reranker AbstractReranker) (. retriever (inert reranker))) (kw (:: reranker_kwargs NamedTuple) (call NamedTuple)) (kw cost_tracker (call (curly (. Threads (inert Atomic)) Float64) 0.0)) (... kwargs)) (:: retriever AbstractRetriever) (:: index AbstractDocumentIndex) (:: question AbstractString)) (block (= rephraser_kwargs_ (if (call isempty api_kwargs) rephraser_kwargs (call merge rephraser_kwargs (tuple (parameters api_kwargs))))) (= rephrased_questions (call rephrase (parameters (kw verbose (call > verbose 1)) cost_tracker (... rephraser_kwargs_)) rephraser question)) (= embeddings (if (call HasEmbeddings index) (block (= embedder_kwargs_ (if (call isempty api_kwargs) embedder_kwargs (call merge embedder_kwargs (tuple (parameters api_kwargs))))) (= embeddings (call get_embeddings (parameters (kw verbose (call > verbose 1)) cost_tracker (... embedder_kwargs_)) embedder rephrased_questions))) (block (= embeddings (call hcat (... (comprehension (generator (ref Float32) (= x rephrased_questions))))))))) (= keywords (if (call HasKeywords index) (block (= keywords (call get_keywords (parameters (kw verbose (call > verbose 1)) (... processor_kwargs) (kw return_keywords true)) processor rephrased_questions)) (&& (call >= verbose 1) (|| (call isa keywords (curly AbstractVector (<: (curly AbstractVector (<: AbstractString))))) (macrocall @warn :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1105 =#) (string "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " (call typeof processor) ").")))) keywords) (block (comprehension (generator (ref String) (= x rephrased_questions)))))) (= finder_kwargs_ (if (call isempty api_kwargs) finder_kwargs (call merge finder_kwargs (tuple (parameters api_kwargs))))) (= emb_candidates (call find_closest (parameters (kw verbose (call > verbose 1)) top_k (... finder_kwargs_)) finder index embeddings keywords)) (= tagger_kwargs_ (if (call isempty api_kwargs) tagger_kwargs (call merge tagger_kwargs (tuple (parameters api_kwargs))))) (= tags (call get_tags (parameters (kw verbose (call > verbose 1)) cost_tracker (... tagger_kwargs_)) tagger rephrased_questions)) (= filter_kwargs_ (if (call isempty api_kwargs) filter_kwargs (call merge filter_kwargs (tuple (parameters api_kwargs))))) (= tag_candidates (call find_tags (parameters (kw verbose (call > verbose 1)) (... filter_kwargs_)) filter index tags)) (= filtered_candidates (if (call isnothing tag_candidates) emb_candidates (call & emb_candidates tag_candidates))) (= reranker_kwargs_ (if (call isempty api_kwargs) reranker_kwargs (call merge reranker_kwargs (tuple (parameters api_kwargs))))) (= reranked_candidates (call rerank (parameters top_n (kw verbose (call > verbose 1)) cost_tracker (... reranker_kwargs_)) reranker index question filtered_candidates)) (&& (call > verbose 0) (macrocall @info :(#= /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:1142 =#) (string "Retrieval done. Identified " (call length (call positions reranked_candidates)) " chunks, total cost: \$" (call round (ref cost_tracker) (kw digits 2)) "."))) (= result (call RAGResult (parameters question (kw answer nothing) rephrased_questions (kw final_answer nothing) (kw context (call collect (ref index reranked_candidates (inert chunks) (kw sorted true)))) (kw sources (call collect (ref index reranked_candidates (inert sources) (kw sorted true)))) emb_candidates tag_candidates filtered_candidates reranked_candidates))) (return result)))) │ └─ @ /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947 │ │ Stacktrace: │ [1] expand_forms_2(ctx::Base.JuliaLowering.MacroExpansionContext{Dict{Symbol, Dict{Int64, Any}}}, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}) │ @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/desugaring.jl:4448 │ [2] core_lowering_hook(code::Any, mod::Module, file::String, line::UInt64, world::UInt64, _warn::Bool) │ @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/hooks.jl:30 │ [3] include(mapexpr::Function, mod::Module, _path::String) │ @ Base Base.jl:327 │ [4] top-level scope │ @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:1 │ [5] macro expansion │ @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:47 [inlined] │ [6] eval(m::Module, e::Any) │ @ Core boot.jl:521 │ [7] _eval(mod::Module, iter::Base.JuliaLowering.LoweringIterator{Dict{Symbol, Dict{Int64, Any}}}; soft_scope::Nothing) │ @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:545 │ [8] eval(mod::Module, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}; macro_world::UInt64, soft_scope::Nothing, opts::@Kwargs{expr_compat_mode::Bool}) │ @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:518 [inlined] │ [9] top-level scope │ @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:1 │ [10] macro expansion │ @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:1 [inlined] │ [11] include(mapexpr::Function, mod::Module, _path::String) │ @ Base Base.jl:327 │ [12] top-level scope │ @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:1 │ [13] macro expansion │ @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:18 [inlined] │ [14] eval(m::Module, e::Any) │ @ Core boot.jl:521 │ [15] _eval(mod::Module, iter::Base.JuliaLowering.LoweringIterator{Dict{Symbol, Dict{Int64, Any}}}; soft_scope::Nothing) │ @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:545 │ [16] eval(mod::Module, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}; macro_world::UInt64, soft_scope::Nothing, opts::@Kwargs{expr_compat_mode::Bool}) │ @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:518 [inlined] │ [17] top-level scope │ @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:1 │ [18] macro expansion │ @ ~/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:1 [inlined] │ [19] include(mapexpr::Function, mod::Module, _path::String) │ @ Base Base.jl:327 │ [20] top-level scope │ @ ~/.julia/packages/PromptingTools/wvgCI/src/PromptingTools.jl:120 │ [21] include(mod::Module, _path::String) │ @ Base Base.jl:326 │ [22] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing) │ @ Base loading.jl:3296 │ [23] top-level scope │ @ stdin:5 │ [24] eval(m::Module, e::Any) │ @ Core boot.jl:521 │ [25] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) │ @ Base loading.jl:3132 │ [26] include_string(m::Module, txt::String, fname::String) │ @ Base loading.jl:3142 [inlined] │ [27] exec_options(opts::Base.JLOptions) │ @ Base client.jl:353 │ [28] _start() │ @ Base client.jl:596 │ in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/retrieval.jl:947 │ in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/RAGTools/RAGTools.jl:1 │ in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/Experimental/Experimental.jl:1 │ in expression starting at /home/pkgeval/.julia/packages/PromptingTools/wvgCI/src/PromptingTools.jl:1 │ in expression starting at stdin:5 └ ERROR: LoadError: The following 3 packages failed to precompile: DocumenterDescriptions Failed to precompile DocumenterDescriptions [e9159221-f9c0-48a0-ba53-9ba151c71a1d] to "/home/pkgeval/.julia/compiled/v1.14/DocumenterDescriptions/jl_6CzQS2" (ProcessExited(1)). PromptingTools → MarkdownPromptingToolsExt Failed to precompile MarkdownPromptingToolsExt [d0be87b4-40ad-5214-b203-6d62ab8a77ad] to "/home/pkgeval/.julia/compiled/v1.14/MarkdownPromptingToolsExt/jl_E2SeJf" (ProcessExited(1)). PromptingTools Failed to precompile PromptingTools [670122d1-24a8-4d70-bfce-740807c42192] to "/home/pkgeval/.julia/compiled/v1.14/PromptingTools/jl_cPhIRt" (ProcessExited(1)). in expression starting at /home/pkgeval/.julia/packages/DocumenterDescriptions/kTXBL/test/runtests.jl:1 Testing failed after 210.94s ERROR: LoadError: Package DocumenterDescriptions errored during testing Stacktrace: [1] pkgerror(msg::String) @ Pkg.Types /opt/julia/share/julia/stdlib/v1.14/Pkg/src/Types.jl:68 [2] test(ctx::Pkg.Types.Context, pkgs::Vector{PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool) @ Pkg.Operations /opt/julia/share/julia/stdlib/v1.14/Pkg/src/Operations.jl:3247 [3] test(ctx::Pkg.Types.Context, pkgs::Vector{PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{io::IOContext{IO}}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.14/Pkg/src/API.jl:587 [4] test(pkgs::Vector{PackageSpec}; io::IOContext{IO}, kwargs::@Kwargs{julia_args::Cmd}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.14/Pkg/src/API.jl:172 [5] test(pkgs::Vector{String}; kwargs::@Kwargs{julia_args::Cmd}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.14/Pkg/src/API.jl:160 [6] test(pkg::String; kwargs::@Kwargs{julia_args::Cmd}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.14/Pkg/src/API.jl:159 [inlined] [7] top-level scope @ /PkgEval.jl/scripts/evaluate.jl:223 [8] include(mod::Module, _path::String) @ Base Base.jl:326 [9] exec_options(opts::Base.JLOptions) @ Base client.jl:355 [10] _start() @ Base client.jl:596 in expression starting at /PkgEval.jl/scripts/evaluate.jl:214 PkgEval failed after 1622.44s: package fails to precompile