Package evaluation to load SwarmAgents on Julia 1.14.0-DEV.2373 (cf67ecc88f*) started at 2026-06-13T16:07:09.624 ################################################################################ # Set-up # Set-up completed after 0.14s ################################################################################ # Installation # Installing SwarmAgents... Resolving package versions... Installed ExceptionUnwrapping ─ v0.1.11 Installed AbstractTrees ─────── v0.4.5 Installed ConcurrentUtilities ─ v2.5.1 Installed CodecZlib ─────────── v0.7.8 Installed URIs ──────────────── v1.6.1 Installed MbedTLS ───────────── v1.1.10 Installed TranscodingStreams ── v0.11.3 Installed PrecompileTools ───── v1.3.4 Installed BitFlags ──────────── v0.1.10 Installed OpenAI ────────────── v0.9.1 Installed OpenSSL ───────────── v1.6.1 Installed JSON3 ─────────────── v1.14.3 Installed Parsers ───────────── v2.8.5 Installed LoggingExtras ─────── v1.2.0 Installed HTTP ──────────────── v1.11.0 Installed MbedTLS_jll ───────── v2.28.1010+0 Installed SimpleBufferStream ── v1.2.0 Installed StructTypes ───────── v1.11.0 Installed JLLWrappers ───────── v1.8.0 Installed Preferences ───────── v1.5.2 Installed SwarmAgents ───────── v0.1.0 Installed PromptingTools ────── v0.59.1 Installing 1 artifacts Installed artifact MbedTLS 2.2 MiB Updating `~/.julia/environments/v1.14/Project.toml` [6fce2c4d] + SwarmAgents v0.1.0 Updating `~/.julia/environments/v1.14/Manifest.toml` [1520ce14] + AbstractTrees v0.4.5 [d1d4a3ce] + BitFlags v0.1.10 [944b1d66] + CodecZlib v0.7.8 [f0e56b4a] + ConcurrentUtilities v2.5.1 [460bff9d] + ExceptionUnwrapping v0.1.11 ⌅ [cd3eb016] + HTTP v1.11.0 [692b3bcd] + JLLWrappers v1.8.0 [0f8b85d8] + JSON3 v1.14.3 [e6f89c97] + LoggingExtras v1.2.0 [739be429] + MbedTLS v1.1.10 ⌅ [e9f21f70] + OpenAI v0.9.1 [4d8831e6] + OpenSSL v1.6.1 [69de0a69] + Parsers v2.8.5 [aea7be01] + PrecompileTools v1.3.4 [21216c6a] + Preferences v1.5.2 ⌅ [670122d1] + PromptingTools v0.59.1 [777ac1f9] + SimpleBufferStream v1.2.0 [856f2bd8] + StructTypes v1.11.0 [6fce2c4d] + SwarmAgents v0.1.0 [3bb67fe8] + TranscodingStreams v0.11.3 [5c2747f8] + URIs v1.6.1 [c8ffd9c3] + MbedTLS_jll v2.28.1010+0 [0dad84c5] + ArgTools v1.2.0 [56f22d72] + Artifacts v1.11.0 [2a0f44e3] + Base64 v1.11.0 [ade2ca70] + Dates v1.11.0 [f43a241f] + Downloads v1.7.0 [7b1f6079] + FileWatching v1.11.0 [b77e0a4c] + InteractiveUtils v1.11.0 [ac6e5ff7] + JuliaSyntaxHighlighting v1.13.0 [b27032c2] + LibCURL v1.0.0 [76f85450] + LibGit2 v1.11.0 [8f399da3] + Libdl v1.11.0 [56ddb016] + Logging v1.11.0 [d6f4376e] + Markdown v1.11.0 [a63ad114] + Mmap v1.11.0 [ca575930] + NetworkOptions v1.3.0 [44cfe95a] + Pkg v1.14.0 [de0858da] + Printf v1.11.0 [3fa0cd96] + REPL v1.11.0 [9a3f8284] + Random v1.11.0 [ea8e919c] + SHA v1.13.0 [9e88b42a] + Serialization v1.11.0 [6462fe0b] + Sockets v1.11.0 [f489334b] + StyledStrings v1.13.0 [fa267f1f] + TOML v1.0.3 [a4e569a6] + Tar v1.10.0 [8dfed614] + Test v1.11.0 [cf7118a7] + UUIDs v1.11.0 [4ec0a83e] + Unicode v1.11.0 [e66e0078] + CompilerSupportLibraries_jll v1.5.2+0 [deac9b47] + LibCURL_jll v8.20.0+1 [e37daf67] + LibGit2_jll v1.9.4+0 [29816b5a] + LibSSH2_jll v1.11.101+0 [14a3606d] + MozillaCACerts_jll v2026.5.14 [458c3c95] + OpenSSL_jll v3.5.7+0 [efcefdf7] + PCRE2_jll v10.47.0+0 [83775a58] + Zlib_jll v1.3.2+0 [3161d3a3] + Zstd_jll v1.5.7+1 [8e850ede] + nghttp2_jll v1.69.0+0 [3f19e933] + p7zip_jll v17.8.0+0 Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m` Installation completed after 12.2s ################################################################################ # Precompilation # Precompiling PkgEval dependencies... Project No packages added to or removed from `~/.julia/environments/pkgeval/Project.toml` Manifest No packages added to or removed from `~/.julia/environments/pkgeval/Manifest.toml` Precompiling package dependencies... Precompiling project... 34.1 s ✓ URIs 33.8 s ✓ BitFlags 64.4 s ✓ AbstractTrees 47.2 s ✓ TranscodingStreams 0.7 s ✓ SimpleBufferStream 1.4 s ✓ ConcurrentUtilities 3.2 s ✓ StructTypes 2.3 s ✓ LoggingExtras 2.5 s ✓ Preferences 35.0 s ✓ ExceptionUnwrapping 70.3 s ✓ OpenSSL 17.4 s ✓ CodecZlib 2.7 s ✓ JLLWrappers 2.4 s ✓ PrecompileTools 3.0 s ✓ MbedTLS_jll 32.5 s ✓ Parsers 3.5 s ✓ MbedTLS 36.0 s ✓ JSON3 123.0 s ✓ HTTP 59.2 s ✓ OpenAI ┌ Warning: OPENAI_API_KEY variable not set! OpenAI models will not be available - set API key directly via `PromptingTools.OPENAI_API_KEY=`! └ @ PromptingTools ~/.julia/packages/PromptingTools/aTolC/src/user_preferences.jl:181 ┌ Info: JuliaLowering threw given input: │ code = │ :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946 =# Core.@doc " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" function retrieve(retriever::AbstractRetriever, index::AbstractDocumentIndex, question::AbstractString; verbose::Integer = 1, top_k::Integer = 100, top_n::Integer = 5, api_kwargs::NamedTuple = NamedTuple(), rephraser::AbstractRephraser = retriever.rephraser, rephraser_kwargs::NamedTuple = NamedTuple(), embedder::AbstractEmbedder = retriever.embedder, embedder_kwargs::NamedTuple = NamedTuple(), processor::AbstractProcessor = retriever.processor, processor_kwargs::NamedTuple = NamedTuple(), finder::AbstractSimilarityFinder = retriever.finder, finder_kwargs::NamedTuple = NamedTuple(), tagger::AbstractTagger = retriever.tagger, tagger_kwargs::NamedTuple = NamedTuple(), filter::AbstractTagFilter = retriever.filter, filter_kwargs::NamedTuple = NamedTuple(), reranker::AbstractReranker = retriever.reranker, reranker_kwargs::NamedTuple = NamedTuple(), cost_tracker = Threads.Atomic{Float64}(0.0), kwargs...) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1058 =# │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1082 =# │ rephraser_kwargs_ = if isempty(api_kwargs) │ rephraser_kwargs │ else │ merge(rephraser_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1084 =# │ rephrased_questions = rephrase(rephraser, question; verbose = verbose > 1, cost_tracker, rephraser_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1088 =# │ embeddings = if HasEmbeddings(index) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1089 =# │ embedder_kwargs_ = if isempty(api_kwargs) │ embedder_kwargs │ else │ merge(embedder_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1091 =# │ embeddings = get_embeddings(embedder, rephrased_questions; verbose = verbose > 1, cost_tracker, embedder_kwargs_...) │ else │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1094 =# │ embeddings = hcat([Float32[] for x = rephrased_questions]...) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1098 =# │ keywords = if HasKeywords(index) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1100 =# │ keywords = get_keywords(processor, rephrased_questions; verbose = verbose > 1, processor_kwargs..., return_keywords = true) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1103 =# │ verbose >= 1 && (keywords isa AbstractVector{<:AbstractVector{<:AbstractString}} || #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1104 =# @warn("Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: $(typeof(processor))).")) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1105 =# │ keywords │ else │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1107 =# │ [String[] for x = rephrased_questions] │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1110 =# │ finder_kwargs_ = if isempty(api_kwargs) │ finder_kwargs │ else │ merge(finder_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1112 =# │ emb_candidates = find_closest(finder, index, embeddings, keywords; verbose = verbose > 1, top_k, finder_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1116 =# │ tagger_kwargs_ = if isempty(api_kwargs) │ tagger_kwargs │ else │ merge(tagger_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1118 =# │ tags = get_tags(tagger, rephrased_questions; verbose = verbose > 1, cost_tracker, tagger_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1121 =# │ filter_kwargs_ = if isempty(api_kwargs) │ filter_kwargs │ else │ merge(filter_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1123 =# │ tag_candidates = find_tags(filter, index, tags; verbose = verbose > 1, filter_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1128 =# │ filtered_candidates = if isnothing(tag_candidates) │ emb_candidates │ else │ emb_candidates & tag_candidates │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1135 =# │ reranker_kwargs_ = if isempty(api_kwargs) │ reranker_kwargs │ else │ merge(reranker_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1137 =# │ reranked_candidates = rerank(reranker, index, question, filtered_candidates; top_n, verbose = verbose > 1, cost_tracker, reranker_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1140 =# │ verbose > 0 && #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1141 =# @info("Retrieval done. Identified $(length(positions(reranked_candidates))) chunks, total cost: \$$(round(cost_tracker[], digits = 2)).") │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1144 =# │ result = RAGResult(; question, answer = nothing, rephrased_questions, final_answer = nothing, context = collect(index[reranked_candidates, :chunks, sorted = true]), sources = collect(index[reranked_candidates, :sources, sorted = true]), emb_candidates, tag_candidates, filtered_candidates, reranked_candidates) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1157 =# │ return result │ end) │ st0 = │ SyntaxTree with attributes mod,kind,var_id,toplevel_pure,scope_type,macro_source,name_val,syntax_flags,meta,scope_layer,value,jl_source,is_toplevel_thunk,source,__macro_ctx__ │ [macrocall] │ │ @doc :: Identifier │ mod │ :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946 =#) :: Value │ │ " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" :: Value │ │ [function] │ │ [call] │ │ retrieve :: Identifier │ │ [parameters] │ │ [kw] │ │ [::] │ │ verbose :: Identifier │ │ Integer :: Identifier │ │ 1 :: Value │ │ [kw] │ │ [::] │ │ top_k :: Identifier │ │ Integer :: Identifier │ │ 100 :: Value │ │ [kw] │ │ [::] │ │ top_n :: Identifier │ │ Integer :: Identifier │ │ 5 :: Value │ │ [kw] │ │ [::] │ │ api_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ rephraser :: Identifier │ │ AbstractRephraser :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ rephraser :: Identifier │ │ [kw] │ │ [::] │ │ rephraser_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ embedder :: Identifier │ │ AbstractEmbedder :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ embedder :: Identifier │ │ [kw] │ │ [::] │ │ embedder_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ processor :: Identifier │ │ AbstractProcessor :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ processor :: Identifier │ │ [kw] │ │ [::] │ │ processor_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ finder :: Identifier │ │ AbstractSimilarityFinder :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ finder :: Identifier │ │ [kw] │ │ [::] │ │ finder_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ tagger :: Identifier │ │ AbstractTagger :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ tagger :: Identifier │ │ [kw] │ │ [::] │ │ tagger_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ filter :: Identifier │ │ AbstractTagFilter :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ filter :: Identifier │ │ [kw] │ │ [::] │ │ filter_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ reranker :: Identifier │ │ AbstractReranker :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ reranker :: Identifier │ │ [kw] │ │ [::] │ │ reranker_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ cost_tracker :: Identifier │ │ [call] │ │ [curly] │ │ [.] │ │ Threads :: Identifier │ │ [inert] │ │ Atomic :: Identifier │ │ Float64 :: Identifier │ │ 0.0 :: Value │ │ [...] │ │ kwargs :: Identifier │ │ [::] │ │ retriever :: Identifier │ │ AbstractRetriever :: Identifier │ │ [::] │ │ index :: Identifier │ │ AbstractDocumentIndex :: Identifier │ │ [::] │ │ question :: Identifier │ │ AbstractString :: Identifier │ │ [block] │ │ [=] │ │ rephraser_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ rephraser_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ rephraser_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ rephrased_questions :: Identifier │ │ [call] │ │ rephrase :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ rephraser_kwargs_ :: Identifier │ │ rephraser :: Identifier │ │ question :: Identifier │ │ [=] │ │ embeddings :: Identifier │ │ [if] │ │ [call] │ │ HasEmbeddings :: Identifier │ │ index :: Identifier │ │ [block] │ │ [=] │ │ embedder_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ embedder_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ embedder_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ embeddings :: Identifier │ │ [call] │ │ get_embeddings :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ embedder_kwargs_ :: Identifier │ │ embedder :: Identifier │ │ rephrased_questions :: Identifier │ │ [block] │ │ [=] │ │ embeddings :: Identifier │ │ [call] │ │ hcat :: Identifier │ │ [...] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ Float32 :: Identifier │ │ [=] │ │ x :: Identifier │ │ rephrased_questions :: Identifier │ │ [=] │ │ keywords :: Identifier │ │ [if] │ │ [call] │ │ HasKeywords :: Identifier │ │ index :: Identifier │ │ [block] │ │ [=] │ │ keywords :: Identifier │ │ [call] │ │ get_keywords :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ [...] │ │ processor_kwargs :: Identifier │ │ [kw] │ │ return_keywords :: Identifier │ │ true :: Value │ │ processor :: Identifier │ │ rephrased_questions :: Identifier │ │ [&&] │ │ [call] │ │ >= :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ [||] │ │ [call] │ │ isa :: Identifier │ │ keywords :: Identifier │ │ [curly] │ │ AbstractVector :: Identifier │ │ [<:] │ │ [curly] │ │ AbstractVector :: Identifier │ │ [<:] │ │ AbstractString :: Identifier │ │ [macrocall] │ │ @warn :: Identifier │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1104 =#) :: Value │ │ [string] │ │ "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " :: Value │ │ [call] │ │ typeof :: Identifier │ │ processor :: Identifier │ │ ")." :: Value │ │ keywords :: Identifier │ │ [block] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ String :: Identifier │ │ [=] │ │ x :: Identifier │ │ rephrased_questions :: Identifier │ │ [=] │ │ finder_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ finder_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ finder_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ emb_candidates :: Identifier │ │ [call] │ │ find_closest :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ top_k :: Identifier │ │ [...] │ │ finder_kwargs_ :: Identifier │ │ finder :: Identifier │ │ index :: Identifier │ │ embeddings :: Identifier │ │ keywords :: Identifier │ │ [=] │ │ tagger_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ tagger_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ tagger_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ tags :: Identifier │ │ [call] │ │ get_tags :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ tagger_kwargs_ :: Identifier │ │ tagger :: Identifier │ │ rephrased_questions :: Identifier │ │ [=] │ │ filter_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ filter_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ filter_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ tag_candidates :: Identifier │ │ [call] │ │ find_tags :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ [...] │ │ filter_kwargs_ :: Identifier │ │ filter :: Identifier │ │ index :: Identifier │ │ tags :: Identifier │ │ [=] │ │ filtered_candidates :: Identifier │ │ [if] │ │ [call] │ │ isnothing :: Identifier │ │ tag_candidates :: Identifier │ │ emb_candidates :: Identifier │ │ [call] │ │ & :: Identifier │ │ emb_candidates :: Identifier │ │ tag_candidates :: Identifier │ │ [=] │ │ reranker_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ reranker_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ reranker_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ reranked_candidates :: Identifier │ │ [call] │ │ rerank :: Identifier │ │ [parameters] │ │ top_n :: Identifier │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ reranker_kwargs_ :: Identifier │ │ reranker :: Identifier │ │ index :: Identifier │ │ question :: Identifier │ │ filtered_candidates :: Identifier │ │ [&&] │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 0 :: Value │ │ [macrocall] │ │ @info :: Identifier │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1141 =#) :: Value │ │ [string] │ │ "Retrieval done. Identified " :: Value │ │ [call] │ │ length :: Identifier │ │ [call] │ │ positions :: Identifier │ │ reranked_candidates :: Identifier │ │ " chunks, total cost: \$" :: Value │ │ [call] │ │ round :: Identifier │ │ [ref] │ │ cost_tracker :: Identifier │ │ [kw] │ │ digits :: Identifier │ │ 2 :: Value │ │ "." :: Value │ │ [=] │ │ result :: Identifier │ │ [call] │ │ RAGResult :: Identifier │ │ [parameters] │ │ question :: Identifier │ │ [kw] │ │ answer :: Identifier │ │ nothing :: Identifier │ │ rephrased_questions :: Identifier │ │ [kw] │ │ final_answer :: Identifier │ │ nothing :: Identifier │ │ [kw] │ │ context :: Identifier │ │ [call] │ │ collect :: Identifier │ │ [ref] │ │ index :: Identifier │ │ reranked_candidates :: Identifier │ │ [inert] │ │ chunks :: Identifier │ │ [kw] │ │ sorted :: Identifier │ │ true :: Value │ │ [kw] │ │ sources :: Identifier │ │ [call] │ │ collect :: Identifier │ │ [ref] │ │ index :: Identifier │ │ reranked_candidates :: Identifier │ │ [inert] │ │ sources :: Identifier │ │ [kw] │ │ sorted :: Identifier │ │ true :: Value │ │ emb_candidates :: Identifier │ │ tag_candidates :: Identifier │ │ filtered_candidates :: Identifier │ │ reranked_candidates :: Identifier │ │ [return] │ │ result :: Identifier │ │ │ st1 = │ SyntaxTree with attributes mod,kind,var_id,toplevel_pure,scope_type,macro_source,name_val,syntax_flags,meta,scope_layer,value,jl_source,is_toplevel_thunk,source │ [block] │ │ [=] │ │ val :: Identifier │ scope_layer=3 │ [function] │ │ [call] │ │ retrieve :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ [::] │ │ verbose :: Identifier │ scope_layer=1 │ Integer :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [kw] │ │ [::] │ │ top_k :: Identifier │ scope_layer=1 │ Integer :: Identifier │ scope_layer=1 │ 100 :: Value │ macro_source=485 │ [kw] │ │ [::] │ │ top_n :: Identifier │ scope_layer=1 │ Integer :: Identifier │ scope_layer=1 │ 5 :: Value │ macro_source=485 │ [kw] │ │ [::] │ │ api_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ rephraser :: Identifier │ scope_layer=1 │ AbstractRephraser :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ rephraser :: Identifier │ │ [kw] │ │ [::] │ │ rephraser_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ embedder :: Identifier │ scope_layer=1 │ AbstractEmbedder :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ embedder :: Identifier │ │ [kw] │ │ [::] │ │ embedder_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ processor :: Identifier │ scope_layer=1 │ AbstractProcessor :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ processor :: Identifier │ │ [kw] │ │ [::] │ │ processor_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ finder :: Identifier │ scope_layer=1 │ AbstractSimilarityFinder :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ finder :: Identifier │ │ [kw] │ │ [::] │ │ finder_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ tagger :: Identifier │ scope_layer=1 │ AbstractTagger :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ tagger :: Identifier │ │ [kw] │ │ [::] │ │ tagger_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ filter :: Identifier │ scope_layer=1 │ AbstractTagFilter :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ filter :: Identifier │ │ [kw] │ │ [::] │ │ filter_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ reranker :: Identifier │ scope_layer=1 │ AbstractReranker :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ reranker :: Identifier │ │ [kw] │ │ [::] │ │ reranker_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ cost_tracker :: Identifier │ scope_layer=1 │ [call] │ │ [curly] │ │ [.] │ │ Threads :: Identifier │ scope_layer=1 │ [inert] │ │ Atomic :: Identifier │ │ Float64 :: Identifier │ scope_layer=1 │ 0.0 :: Value │ macro_source=485 │ [...] │ │ kwargs :: Identifier │ scope_layer=1 │ [::] │ │ retriever :: Identifier │ scope_layer=1 │ AbstractRetriever :: Identifier │ scope_layer=1 │ [::] │ │ index :: Identifier │ scope_layer=1 │ AbstractDocumentIndex :: Identifier │ scope_layer=1 │ [::] │ │ question :: Identifier │ scope_layer=1 │ AbstractString :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ rephraser_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ rephraser_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ rephraser_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ rephrased_questions :: Identifier │ scope_layer=1 │ [call] │ │ rephrase :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ rephraser_kwargs_ :: Identifier │ scope_layer=1 │ rephraser :: Identifier │ scope_layer=1 │ question :: Identifier │ scope_layer=1 │ [=] │ │ embeddings :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ HasEmbeddings :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ embedder_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ embedder_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ embedder_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ embeddings :: Identifier │ scope_layer=1 │ [call] │ │ get_embeddings :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ embedder_kwargs_ :: Identifier │ scope_layer=1 │ embedder :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ embeddings :: Identifier │ scope_layer=1 │ [call] │ │ hcat :: Identifier │ scope_layer=1 │ [...] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ Float32 :: Identifier │ scope_layer=1 │ [=] │ │ x :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [=] │ │ keywords :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ HasKeywords :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ keywords :: Identifier │ scope_layer=1 │ [call] │ │ get_keywords :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [...] │ │ processor_kwargs :: Identifier │ scope_layer=1 │ [kw] │ │ return_keywords :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ processor :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [&&] │ │ [call] │ │ >= :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [||] │ │ [call] │ │ isa :: Identifier │ scope_layer=1 │ keywords :: Identifier │ scope_layer=1 │ [curly] │ │ AbstractVector :: Identifier │ scope_layer=1 │ [<:] │ scope_layer=1 │ [curly] │ │ AbstractVector :: Identifier │ scope_layer=1 │ [<:] │ scope_layer=1 │ AbstractString :: Identifier │ scope_layer=1 │ [block] │ │ [let] │ │ [block] │ macro_source=485 │ [block] │ │ [=] │ │ #1131#level :: Identifier │ scope_layer=1 │ Warn :: Identifier │ mod,scope_layer=1 │ [=] │ │ #1132#std_level :: Identifier │ scope_layer=1 │ #1131#level :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ >= :: Identifier │ mod,scope_layer=1 │ [.] │ │ #1132#std_level :: Identifier │ scope_layer=1 │ [inert] │ │ level :: Identifier │ │ [ref] │ macro_source=485 │ Base.Threads.Atomic{Int32}(-1000) :: Value │ macro_source=485 │ [block] │ │ [=] │ │ #1133#group :: Identifier │ scope_layer=1 │ [inert] │ │ retrieval :: Identifier │ │ [=] │ │ #1134#_module :: Identifier │ scope_layer=1 │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [=] │ │ #1135#logger :: Identifier │ scope_layer=1 │ [call] │ │ Base.CoreLogging.current_logger_for_env :: Value │ macro_source=485 │ #1132#std_level :: Identifier │ scope_layer=1 │ #1133#group :: Identifier │ scope_layer=1 │ #1134#_module :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ ! :: Identifier │ mod,scope_layer=1 │ [call] │ │ === :: Identifier │ mod,scope_layer=1 │ #1135#logger :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1136#id :: Identifier │ scope_layer=1 │ [inert] │ │ PromptingTools_Experimental_RAGTools_7aa4399b :: Identifier │ │ [if] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.shouldlog :: Value │ macro_source=485 │ #1135#logger :: Identifier │ scope_layer=1 │ #1131#level :: Identifier │ scope_layer=1 │ #1134#_module :: Identifier │ scope_layer=1 │ #1133#group :: Identifier │ scope_layer=1 │ #1136#id :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ #1137#file :: Identifier │ scope_layer=1 │ "/home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ [if] │ │ [call] │ │ isa :: Identifier │ mod,scope_layer=1 │ #1137#file :: Identifier │ scope_layer=1 │ String :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1137#file :: Identifier │ scope_layer=1 │ [call] │ │ [.] │ │ Base :: Identifier │ mod,scope_layer=1 │ [inert] │ │ fixup_stdlib_path :: Identifier │ │ #1137#file :: Identifier │ scope_layer=1 │ [=] │ │ #1138#line :: Identifier │ scope_layer=1 │ 1104 :: Value │ macro_source=485 │ [local] │ │ #1139#msg :: Identifier │ scope_layer=1 │ #1140#kwargs :: Identifier │ scope_layer=1 │ [if] │ │ [block] │ │ [try] │ │ [block] │ │ [=] │ │ #1139#msg :: Identifier │ scope_layer=1 │ [string] │ │ "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " :: Value │ macro_source=485 │ [call] │ │ typeof :: Identifier │ scope_layer=1 │ processor :: Identifier │ scope_layer=1 │ ")." :: Value │ macro_source=485 │ [=] │ │ #1140#kwargs :: Identifier │ scope_layer=1 │ [tuple] │ macro_source=485 │ [parameters] │ macro_source=485 │ true :: Value │ macro_source=485 │ #1153#err :: Identifier │ scope_layer=1 │ [block] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.logging_error :: Value │ macro_source=485 │ #1135#logger :: Identifier │ scope_layer=1 │ #1131#level :: Identifier │ scope_layer=1 │ #1134#_module :: Identifier │ scope_layer=1 │ #1133#group :: Identifier │ scope_layer=1 │ #1136#id :: Identifier │ scope_layer=1 │ #1137#file :: Identifier │ scope_layer=1 │ #1138#line :: Identifier │ scope_layer=1 │ #1153#err :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ false :: Value │ macro_source=485 │ [block] │ │ [if] │ │ [isdefined] │ │ #1139#msg :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [if] │ │ [isdefined] │ │ #1140#kwargs :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [call] │ │ Base.CoreLogging.handle_message_nothrow :: Value │ macro_source=485 │ [parameters] │ │ [...] │ │ #1140#kwargs :: Identifier │ scope_layer=1 │ #1135#logger :: Identifier │ scope_layer=1 │ #1131#level :: Identifier │ scope_layer=1 │ #1139#msg :: Identifier │ scope_layer=1 │ #1134#_module :: Identifier │ scope_layer=1 │ #1133#group :: Identifier │ scope_layer=1 │ #1136#id :: Identifier │ scope_layer=1 │ #1137#file :: Identifier │ scope_layer=1 │ #1138#line :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ keywords :: Identifier │ scope_layer=1 │ [block] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ String :: Identifier │ scope_layer=1 │ [=] │ │ x :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [=] │ │ finder_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ finder_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ finder_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ emb_candidates :: Identifier │ scope_layer=1 │ [call] │ │ find_closest :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ top_k :: Identifier │ scope_layer=1 │ [...] │ │ finder_kwargs_ :: Identifier │ scope_layer=1 │ finder :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ embeddings :: Identifier │ scope_layer=1 │ keywords :: Identifier │ scope_layer=1 │ [=] │ │ tagger_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ tagger_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ tagger_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ tags :: Identifier │ scope_layer=1 │ [call] │ │ get_tags :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ tagger_kwargs_ :: Identifier │ scope_layer=1 │ tagger :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [=] │ │ filter_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ filter_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ filter_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ tag_candidates :: Identifier │ scope_layer=1 │ [call] │ │ find_tags :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [...] │ │ filter_kwargs_ :: Identifier │ scope_layer=1 │ filter :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ tags :: Identifier │ scope_layer=1 │ [=] │ │ filtered_candidates :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isnothing :: Identifier │ scope_layer=1 │ tag_candidates :: Identifier │ scope_layer=1 │ emb_candidates :: Identifier │ scope_layer=1 │ [call] │ │ & :: Identifier │ scope_layer=1 │ emb_candidates :: Identifier │ scope_layer=1 │ tag_candidates :: Identifier │ scope_layer=1 │ [=] │ │ reranker_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ reranker_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ reranker_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ reranked_candidates :: Identifier │ scope_layer=1 │ [call] │ │ rerank :: Identifier │ scope_layer=1 │ [parameters] │ │ top_n :: Identifier │ scope_layer=1 │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ reranker_kwargs_ :: Identifier │ scope_layer=1 │ reranker :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ question :: Identifier │ scope_layer=1 │ filtered_candidates :: Identifier │ scope_layer=1 │ [&&] │ │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 0 :: Value │ macro_source=485 │ [block] │ │ [let] │ │ [block] │ macro_source=485 │ [block] │ │ [=] │ │ #1154#level :: Identifier │ scope_layer=1 │ Info :: Identifier │ mod,scope_layer=1 │ [=] │ │ #1155#std_level :: Identifier │ scope_layer=1 │ #1154#level :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ >= :: Identifier │ mod,scope_layer=1 │ [.] │ │ #1155#std_level :: Identifier │ scope_layer=1 │ [inert] │ │ level :: Identifier │ │ [ref] │ macro_source=485 │ Base.Threads.Atomic{Int32}(-1000) :: Value │ macro_source=485 │ [block] │ │ [=] │ │ #1156#group :: Identifier │ scope_layer=1 │ [inert] │ │ retrieval :: Identifier │ │ [=] │ │ #1157#_module :: Identifier │ scope_layer=1 │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [=] │ │ #1158#logger :: Identifier │ scope_layer=1 │ [call] │ │ Base.CoreLogging.current_logger_for_env :: Value │ macro_source=485 │ #1155#std_level :: Identifier │ scope_layer=1 │ #1156#group :: Identifier │ scope_layer=1 │ #1157#_module :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ ! :: Identifier │ mod,scope_layer=1 │ [call] │ │ === :: Identifier │ mod,scope_layer=1 │ #1158#logger :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1159#id :: Identifier │ scope_layer=1 │ [inert] │ │ PromptingTools_Experimental_RAGTools_30253e51 :: Identifier │ │ [if] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.shouldlog :: Value │ macro_source=485 │ #1158#logger :: Identifier │ scope_layer=1 │ #1154#level :: Identifier │ scope_layer=1 │ #1157#_module :: Identifier │ scope_layer=1 │ #1156#group :: Identifier │ scope_layer=1 │ #1159#id :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ #1160#file :: Identifier │ scope_layer=1 │ "/home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ [if] │ │ [call] │ │ isa :: Identifier │ mod,scope_layer=1 │ #1160#file :: Identifier │ scope_layer=1 │ String :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1160#file :: Identifier │ scope_layer=1 │ [call] │ │ [.] │ │ Base :: Identifier │ mod,scope_layer=1 │ [inert] │ │ fixup_stdlib_path :: Identifier │ │ #1160#file :: Identifier │ scope_layer=1 │ [=] │ │ #1161#line :: Identifier │ scope_layer=1 │ 1141 :: Value │ macro_source=485 │ [local] │ │ #1162#msg :: Identifier │ scope_layer=1 │ #1163#kwargs :: Identifier │ scope_layer=1 │ [if] │ │ [block] │ │ [try] │ │ [block] │ │ [=] │ │ #1162#msg :: Identifier │ scope_layer=1 │ [string] │ │ "Retrieval done. Identified " :: Value │ macro_source=485 │ [call] │ │ length :: Identifier │ scope_layer=1 │ [call] │ │ positions :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ " chunks, total cost: \$" :: Value │ macro_source=485 │ [call] │ │ round :: Identifier │ scope_layer=1 │ [ref] │ │ cost_tracker :: Identifier │ scope_layer=1 │ [kw] │ │ digits :: Identifier │ scope_layer=1 │ 2 :: Value │ macro_source=485 │ "." :: Value │ macro_source=485 │ [=] │ │ #1163#kwargs :: Identifier │ scope_layer=1 │ [tuple] │ macro_source=485 │ [parameters] │ macro_source=485 │ true :: Value │ macro_source=485 │ #1176#err :: Identifier │ scope_layer=1 │ [block] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.logging_error :: Value │ macro_source=485 │ #1158#logger :: Identifier │ scope_layer=1 │ #1154#level :: Identifier │ scope_layer=1 │ #1157#_module :: Identifier │ scope_layer=1 │ #1156#group :: Identifier │ scope_layer=1 │ #1159#id :: Identifier │ scope_layer=1 │ #1160#file :: Identifier │ scope_layer=1 │ #1161#line :: Identifier │ scope_layer=1 │ #1176#err :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ false :: Value │ macro_source=485 │ [block] │ │ [if] │ │ [isdefined] │ │ #1162#msg :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [if] │ │ [isdefined] │ │ #1163#kwargs :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [call] │ │ Base.CoreLogging.handle_message_nothrow :: Value │ macro_source=485 │ [parameters] │ │ [...] │ │ #1163#kwargs :: Identifier │ scope_layer=1 │ #1158#logger :: Identifier │ scope_layer=1 │ #1154#level :: Identifier │ scope_layer=1 │ #1162#msg :: Identifier │ scope_layer=1 │ #1157#_module :: Identifier │ scope_layer=1 │ #1156#group :: Identifier │ scope_layer=1 │ #1159#id :: Identifier │ scope_layer=1 │ #1160#file :: Identifier │ scope_layer=1 │ #1161#line :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ [=] │ │ result :: Identifier │ scope_layer=1 │ [call] │ │ RAGResult :: Identifier │ scope_layer=1 │ [parameters] │ │ question :: Identifier │ scope_layer=1 │ [kw] │ │ answer :: Identifier │ scope_layer=1 │ nothing :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [kw] │ │ final_answer :: Identifier │ scope_layer=1 │ nothing :: Identifier │ scope_layer=1 │ [kw] │ │ context :: Identifier │ scope_layer=1 │ [call] │ │ collect :: Identifier │ scope_layer=1 │ [ref] │ │ index :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ [inert] │ │ chunks :: Identifier │ │ [kw] │ │ sorted :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ [kw] │ │ sources :: Identifier │ scope_layer=1 │ [call] │ │ collect :: Identifier │ scope_layer=1 │ [ref] │ │ index :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ [inert] │ │ sources :: Identifier │ │ [kw] │ │ sorted :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ emb_candidates :: Identifier │ scope_layer=1 │ tag_candidates :: Identifier │ scope_layer=1 │ filtered_candidates :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ [return] │ │ result :: Identifier │ scope_layer=1 │ [call] │ │ Base.Docs.doc! :: Value │ macro_source=485 │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [call] │ │ Base.Docs.Binding :: Value │ macro_source=485 │ PromptingTools.Experimental.RAGTools :: Value │ │ [inert] │ jl_source=L65 │ retrieve :: Identifier │ │ [call] │ macro_source=485 │ Base.Docs.docstr :: Value │ macro_source=485 │ [call] │ macro_source=485 │ Core.svec :: Value │ macro_source=485 │ " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" :: Value │ macro_source=485 │ [call] │ macro_source=485 │ Dict{Symbol, Any} :: Value │ macro_source=485 │ :path => "/home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ :linenumber => 946 :: Value │ macro_source=485 │ :module => PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [curly] │ │ Union :: Identifier │ scope_layer=1 │ [curly] │ │ Tuple :: Identifier │ scope_layer=1 │ AbstractRetriever :: Identifier │ scope_layer=1 │ AbstractDocumentIndex :: Identifier │ scope_layer=1 │ AbstractString :: Identifier │ scope_layer=1 │ val :: Identifier │ scope_layer=3 │ │ file = "/home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl" │ line = 946 └ mod = PromptingTools.Experimental.RAGTools ERROR: LoadError: LoweringError: #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1144 =# - invalid syntax: unknown form `kw` or number of arguments 2 Expression:  (kw sorted true) Containing expressions:  (kw context (call collect (ref index reranked_candidates (inert chunks) (kw sorted true))))  Detailed provenance:  (kw sorted true)  └─ (kw sorted true)  ├─ @ /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1144  └─ (macrocall @doc :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946 =#) " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" (function (call retrieve (parameters (kw (:: verbose Integer) 1) (kw (:: top_k Integer) 100) (kw (:: top_n Integer) 5) (kw (:: api_kwargs NamedTuple) (call NamedTuple)) (kw (:: rephraser AbstractRephraser) (. retriever (inert rephraser))) (kw (:: rephraser_kwargs NamedTuple) (call NamedTuple)) (kw (:: embedder AbstractEmbedder) (. retriever (inert embedder))) (kw (:: embedder_kwargs NamedTuple) (call NamedTuple)) (kw (:: processor AbstractProcessor) (. retriever (inert processor))) (kw (:: processor_kwargs NamedTuple) (call NamedTuple)) (kw (:: finder AbstractSimilarityFinder) (. retriever (inert finder))) (kw (:: finder_kwargs NamedTuple) (call NamedTuple)) (kw (:: tagger AbstractTagger) (. retriever (inert tagger))) (kw (:: tagger_kwargs NamedTuple) (call NamedTuple)) (kw (:: filter AbstractTagFilter) (. retriever (inert filter))) (kw (:: filter_kwargs NamedTuple) (call NamedTuple)) (kw (:: reranker AbstractReranker) (. retriever (inert reranker))) (kw (:: reranker_kwargs NamedTuple) (call NamedTuple)) (kw cost_tracker (call (curly (. Threads (inert Atomic)) Float64) 0.0)) (... kwargs)) (:: retriever AbstractRetriever) (:: index AbstractDocumentIndex) (:: question AbstractString)) (block (= rephraser_kwargs_ (if (call isempty api_kwargs) rephraser_kwargs (call merge rephraser_kwargs (tuple (parameters api_kwargs))))) (= rephrased_questions (call rephrase (parameters (kw verbose (call > verbose 1)) cost_tracker (... rephraser_kwargs_)) rephraser question)) (= embeddings (if (call HasEmbeddings index) (block (= embedder_kwargs_ (if (call isempty api_kwargs) embedder_kwargs (call merge embedder_kwargs (tuple (parameters api_kwargs))))) (= embeddings (call get_embeddings (parameters (kw verbose (call > verbose 1)) cost_tracker (... embedder_kwargs_)) embedder rephrased_questions))) (block (= embeddings (call hcat (... (comprehension (generator (ref Float32) (= x rephrased_questions))))))))) (= keywords (if (call HasKeywords index) (block (= keywords (call get_keywords (parameters (kw verbose (call > verbose 1)) (... processor_kwargs) (kw return_keywords true)) processor rephrased_questions)) (&& (call >= verbose 1) (|| (call isa keywords (curly AbstractVector (<: (curly AbstractVector (<: AbstractString))))) (macrocall @warn :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1104 =#) (string "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " (call typeof processor) ").")))) keywords) (block (comprehension (generator (ref String) (= x rephrased_questions)))))) (= finder_kwargs_ (if (call isempty api_kwargs) finder_kwargs (call merge finder_kwargs (tuple (parameters api_kwargs))))) (= emb_candidates (call find_closest (parameters (kw verbose (call > verbose 1)) top_k (... finder_kwargs_)) finder index embeddings keywords)) (= tagger_kwargs_ (if (call isempty api_kwargs) tagger_kwargs (call merge tagger_kwargs (tuple (parameters api_kwargs))))) (= tags (call get_tags (parameters (kw verbose (call > verbose 1)) cost_tracker (... tagger_kwargs_)) tagger rephrased_questions)) (= filter_kwargs_ (if (call isempty api_kwargs) filter_kwargs (call merge filter_kwargs (tuple (parameters api_kwargs))))) (= tag_candidates (call find_tags (parameters (kw verbose (call > verbose 1)) (... filter_kwargs_)) filter index tags)) (= filtered_candidates (if (call isnothing tag_candidates) emb_candidates (call & emb_candidates tag_candidates))) (= reranker_kwargs_ (if (call isempty api_kwargs) reranker_kwargs (call merge reranker_kwargs (tuple (parameters api_kwargs))))) (= reranked_candidates (call rerank (parameters top_n (kw verbose (call > verbose 1)) cost_tracker (... reranker_kwargs_)) reranker index question filtered_candidates)) (&& (call > verbose 0) (macrocall @info :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1141 =#) (string "Retrieval done. Identified " (call length (call positions reranked_candidates)) " chunks, total cost: \$" (call round (ref cost_tracker) (kw digits 2)) "."))) (= result (call RAGResult (parameters question (kw answer nothing) rephrased_questions (kw final_answer nothing) (kw context (call collect (ref index reranked_candidates (inert chunks) (kw sorted true)))) (kw sources (call collect (ref index reranked_candidates (inert sources) (kw sorted true)))) emb_candidates tag_candidates filtered_candidates reranked_candidates))) (return result))))  └─ @ /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946  Stacktrace:  [1] expand_forms_2(ctx::Base.JuliaLowering.MacroExpansionContext{Dict{Symbol, Dict{Int64, Any}}}, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}})  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/desugaring.jl:4440  [2] core_lowering_hook(code::Any, mod::Module, file::String, line::UInt64, world::UInt64, _warn::Bool)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/hooks.jl:30  [3] include(mapexpr::Function, mod::Module, _path::String)  @ Base Base.jl:326  [4] top-level scope  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:1  [5] macro expansion  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:47 [inlined]  [6] eval(m::Module, e::Any)  @ Core boot.jl:522  [7] _eval(mod::Module, iter::Base.JuliaLowering.LoweringIterator{Dict{Symbol, Dict{Int64, Any}}}; soft_scope::Nothing)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:547  [8] eval(mod::Module, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}; macro_world::UInt64, soft_scope::Nothing, opts::@Kwargs{expr_compat_mode::Bool})  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:520 [inlined]  [9] top-level scope  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:1  [10] macro expansion  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:1 [inlined]  [11] include(mapexpr::Function, mod::Module, _path::String)  @ Base Base.jl:326  [12] top-level scope  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:1  [13] macro expansion  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:18 [inlined]  [14] eval(m::Module, e::Any)  @ Core boot.jl:522  [15] _eval(mod::Module, iter::Base.JuliaLowering.LoweringIterator{Dict{Symbol, Dict{Int64, Any}}}; soft_scope::Nothing)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:547  [16] eval(mod::Module, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}; macro_world::UInt64, soft_scope::Nothing, opts::@Kwargs{expr_compat_mode::Bool})  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:520 [inlined]  [17] top-level scope  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:1  [18] macro expansion  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:1 [inlined]  [19] include(mapexpr::Function, mod::Module, _path::String)  @ Base Base.jl:326  [20] top-level scope  @ ~/.julia/packages/PromptingTools/aTolC/src/PromptingTools.jl:108  [21] include(mod::Module, _path::String)  @ Base Base.jl:325  [22] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)  @ Base loading.jl:3296  [23] top-level scope  @ stdin:5  [24] eval(m::Module, e::Any)  @ Core boot.jl:522  [25] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)  @ Base loading.jl:3132  [26] include_string(m::Module, txt::String, fname::String)  @ Base loading.jl:3142 [inlined]  [27] exec_options(opts::Base.JLOptions)  @ Base client.jl:353  [28] _start()  @ Base client.jl:596 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:1 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:1 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/PromptingTools.jl:1 in expression starting at stdin:5 ✗ PromptingTools ERROR: LoadError: Precompiled image Base.PkgId(Base.UUID("670122d1-24a8-4d70-bfce-740807c42192"), "PromptingTools") not available with flags CacheFlags(; use_pkgimages=false, debug_level=1, check_bounds=1, inline=true, opt_level=0) Stacktrace:  [1] error(s::String)  @ Base error.jl:56  [2] __require_prelocked(pkg::Base.PkgId, env::String)  @ Base loading.jl:2837  [3] _require_prelocked(uuidkey::Base.PkgId, env::String)  @ Base loading.jl:2685  [4] macro expansion  @ loading.jl:2599 [inlined]  [5] macro expansion  @ lock.jl:376 [inlined]  [6] __require(into::Module, mod::Symbol)  @ Base loading.jl:2563  [7] require(into::Module, mod::Symbol)  @ Base loading.jl:2539 [inlined]  [8] eval_import_path(at::Module, from::Nothing, path::Expr, keyword::String)  @ Base module.jl:36 [inlined]  [9] eval_import_path_all(at::Module, path::Expr, keyword::String)  @ Base module.jl:60  [10] _eval_using(to::Module, path::Expr, flags::UInt8)  @ Base module.jl:137 [inlined]  [11] eval_using(to::Module, path::Expr)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/runtime.jl:207  [12] top-level scope  @ ~/.julia/packages/PromptingTools/aTolC/ext/MarkdownPromptingToolsExt.jl:3  [13] include(mod::Module, _path::String)  @ Base Base.jl:325  [14] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)  @ Base loading.jl:3296  [15] top-level scope  @ stdin:5  [16] eval(m::Module, e::Any)  @ Core boot.jl:522  [17] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)  @ Base loading.jl:3132  [18] include_string(m::Module, txt::String, fname::String)  @ Base loading.jl:3142 [inlined]  [19] exec_options(opts::Base.JLOptions)  @ Base client.jl:353  [20] _start()  @ Base client.jl:596 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/ext/MarkdownPromptingToolsExt.jl:1 in expression starting at stdin:5 ✗ PromptingTools → MarkdownPromptingToolsExt ERROR: LoadError: Precompiled image Base.PkgId(Base.UUID("670122d1-24a8-4d70-bfce-740807c42192"), "PromptingTools") not available with flags CacheFlags(; use_pkgimages=false, debug_level=1, check_bounds=1, inline=true, opt_level=0) Stacktrace:  [1] error(s::String)  @ Base error.jl:56  [2] __require_prelocked(pkg::Base.PkgId, env::String)  @ Base loading.jl:2837  [3] _require_prelocked(uuidkey::Base.PkgId, env::String)  @ Base loading.jl:2685  [4] macro expansion  @ loading.jl:2599 [inlined]  [5] macro expansion  @ lock.jl:376 [inlined]  [6] __require(into::Module, mod::Symbol)  @ Base loading.jl:2563  [7] require(into::Module, mod::Symbol)  @ Base loading.jl:2539 [inlined]  [8] eval_import_path(at::Module, from::Nothing, path::Expr, keyword::String)  @ Base module.jl:36 [inlined]  [9] eval_import_path_all(at::Module, path::Expr, keyword::String)  @ Base module.jl:60  [10] _eval_using(to::Module, path::Expr, flags::UInt8)  @ Base module.jl:137 [inlined]  [11] eval_using(to::Module, path::Expr)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/runtime.jl:207  [12] top-level scope  @ ~/.julia/packages/SwarmAgents/qtRuA/src/SwarmAgents.jl:4  [13] include(mod::Module, _path::String)  @ Base Base.jl:325  [14] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)  @ Base loading.jl:3296  [15] top-level scope  @ stdin:5  [16] eval(m::Module, e::Any)  @ Core boot.jl:522  [17] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)  @ Base loading.jl:3132  [18] include_string(m::Module, txt::String, fname::String)  @ Base loading.jl:3142 [inlined]  [19] exec_options(opts::Base.JLOptions)  @ Base client.jl:353  [20] _start()  @ Base client.jl:596 in expression starting at /home/pkgeval/.julia/packages/SwarmAgents/qtRuA/src/SwarmAgents.jl:1 in expression starting at stdin:5 ✗ SwarmAgents 20 dependencies successfully precompiled in 707 seconds. 33 already precompiled. Precompilation completed after 725.96s ################################################################################ # Loading # Loading SwarmAgents... ┌ Warning: OPENAI_API_KEY variable not set! OpenAI models will not be available - set API key directly via `PromptingTools.OPENAI_API_KEY=`! └ @ PromptingTools ~/.julia/packages/PromptingTools/aTolC/src/user_preferences.jl:181 ┌ Info: JuliaLowering threw given input: │ code = │ :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946 =# Core.@doc " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" function retrieve(retriever::AbstractRetriever, index::AbstractDocumentIndex, question::AbstractString; verbose::Integer = 1, top_k::Integer = 100, top_n::Integer = 5, api_kwargs::NamedTuple = NamedTuple(), rephraser::AbstractRephraser = retriever.rephraser, rephraser_kwargs::NamedTuple = NamedTuple(), embedder::AbstractEmbedder = retriever.embedder, embedder_kwargs::NamedTuple = NamedTuple(), processor::AbstractProcessor = retriever.processor, processor_kwargs::NamedTuple = NamedTuple(), finder::AbstractSimilarityFinder = retriever.finder, finder_kwargs::NamedTuple = NamedTuple(), tagger::AbstractTagger = retriever.tagger, tagger_kwargs::NamedTuple = NamedTuple(), filter::AbstractTagFilter = retriever.filter, filter_kwargs::NamedTuple = NamedTuple(), reranker::AbstractReranker = retriever.reranker, reranker_kwargs::NamedTuple = NamedTuple(), cost_tracker = Threads.Atomic{Float64}(0.0), kwargs...) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1058 =# │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1082 =# │ rephraser_kwargs_ = if isempty(api_kwargs) │ rephraser_kwargs │ else │ merge(rephraser_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1084 =# │ rephrased_questions = rephrase(rephraser, question; verbose = verbose > 1, cost_tracker, rephraser_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1088 =# │ embeddings = if HasEmbeddings(index) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1089 =# │ embedder_kwargs_ = if isempty(api_kwargs) │ embedder_kwargs │ else │ merge(embedder_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1091 =# │ embeddings = get_embeddings(embedder, rephrased_questions; verbose = verbose > 1, cost_tracker, embedder_kwargs_...) │ else │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1094 =# │ embeddings = hcat([Float32[] for x = rephrased_questions]...) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1098 =# │ keywords = if HasKeywords(index) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1100 =# │ keywords = get_keywords(processor, rephrased_questions; verbose = verbose > 1, processor_kwargs..., return_keywords = true) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1103 =# │ verbose >= 1 && (keywords isa AbstractVector{<:AbstractVector{<:AbstractString}} || #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1104 =# @warn("Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: $(typeof(processor))).")) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1105 =# │ keywords │ else │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1107 =# │ [String[] for x = rephrased_questions] │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1110 =# │ finder_kwargs_ = if isempty(api_kwargs) │ finder_kwargs │ else │ merge(finder_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1112 =# │ emb_candidates = find_closest(finder, index, embeddings, keywords; verbose = verbose > 1, top_k, finder_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1116 =# │ tagger_kwargs_ = if isempty(api_kwargs) │ tagger_kwargs │ else │ merge(tagger_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1118 =# │ tags = get_tags(tagger, rephrased_questions; verbose = verbose > 1, cost_tracker, tagger_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1121 =# │ filter_kwargs_ = if isempty(api_kwargs) │ filter_kwargs │ else │ merge(filter_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1123 =# │ tag_candidates = find_tags(filter, index, tags; verbose = verbose > 1, filter_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1128 =# │ filtered_candidates = if isnothing(tag_candidates) │ emb_candidates │ else │ emb_candidates & tag_candidates │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1135 =# │ reranker_kwargs_ = if isempty(api_kwargs) │ reranker_kwargs │ else │ merge(reranker_kwargs, (; api_kwargs)) │ end │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1137 =# │ reranked_candidates = rerank(reranker, index, question, filtered_candidates; top_n, verbose = verbose > 1, cost_tracker, reranker_kwargs_...) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1140 =# │ verbose > 0 && #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1141 =# @info("Retrieval done. Identified $(length(positions(reranked_candidates))) chunks, total cost: \$$(round(cost_tracker[], digits = 2)).") │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1144 =# │ result = RAGResult(; question, answer = nothing, rephrased_questions, final_answer = nothing, context = collect(index[reranked_candidates, :chunks, sorted = true]), sources = collect(index[reranked_candidates, :sources, sorted = true]), emb_candidates, tag_candidates, filtered_candidates, reranked_candidates) │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1157 =# │ return result │ end) │ st0 = │ SyntaxTree with attributes mod,kind,var_id,toplevel_pure,scope_type,macro_source,name_val,syntax_flags,meta,scope_layer,value,jl_source,is_toplevel_thunk,source,__macro_ctx__ │ [macrocall] │ │ @doc :: Identifier │ mod │ :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946 =#) :: Value │ │ " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" :: Value │ │ [function] │ │ [call] │ │ retrieve :: Identifier │ │ [parameters] │ │ [kw] │ │ [::] │ │ verbose :: Identifier │ │ Integer :: Identifier │ │ 1 :: Value │ │ [kw] │ │ [::] │ │ top_k :: Identifier │ │ Integer :: Identifier │ │ 100 :: Value │ │ [kw] │ │ [::] │ │ top_n :: Identifier │ │ Integer :: Identifier │ │ 5 :: Value │ │ [kw] │ │ [::] │ │ api_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ rephraser :: Identifier │ │ AbstractRephraser :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ rephraser :: Identifier │ │ [kw] │ │ [::] │ │ rephraser_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ embedder :: Identifier │ │ AbstractEmbedder :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ embedder :: Identifier │ │ [kw] │ │ [::] │ │ embedder_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ processor :: Identifier │ │ AbstractProcessor :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ processor :: Identifier │ │ [kw] │ │ [::] │ │ processor_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ finder :: Identifier │ │ AbstractSimilarityFinder :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ finder :: Identifier │ │ [kw] │ │ [::] │ │ finder_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ tagger :: Identifier │ │ AbstractTagger :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ tagger :: Identifier │ │ [kw] │ │ [::] │ │ tagger_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ filter :: Identifier │ │ AbstractTagFilter :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ filter :: Identifier │ │ [kw] │ │ [::] │ │ filter_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ [::] │ │ reranker :: Identifier │ │ AbstractReranker :: Identifier │ │ [.] │ │ retriever :: Identifier │ │ [inert] │ │ reranker :: Identifier │ │ [kw] │ │ [::] │ │ reranker_kwargs :: Identifier │ │ NamedTuple :: Identifier │ │ [call] │ │ NamedTuple :: Identifier │ │ [kw] │ │ cost_tracker :: Identifier │ │ [call] │ │ [curly] │ │ [.] │ │ Threads :: Identifier │ │ [inert] │ │ Atomic :: Identifier │ │ Float64 :: Identifier │ │ 0.0 :: Value │ │ [...] │ │ kwargs :: Identifier │ │ [::] │ │ retriever :: Identifier │ │ AbstractRetriever :: Identifier │ │ [::] │ │ index :: Identifier │ │ AbstractDocumentIndex :: Identifier │ │ [::] │ │ question :: Identifier │ │ AbstractString :: Identifier │ │ [block] │ │ [=] │ │ rephraser_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ rephraser_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ rephraser_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ rephrased_questions :: Identifier │ │ [call] │ │ rephrase :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ rephraser_kwargs_ :: Identifier │ │ rephraser :: Identifier │ │ question :: Identifier │ │ [=] │ │ embeddings :: Identifier │ │ [if] │ │ [call] │ │ HasEmbeddings :: Identifier │ │ index :: Identifier │ │ [block] │ │ [=] │ │ embedder_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ embedder_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ embedder_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ embeddings :: Identifier │ │ [call] │ │ get_embeddings :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ embedder_kwargs_ :: Identifier │ │ embedder :: Identifier │ │ rephrased_questions :: Identifier │ │ [block] │ │ [=] │ │ embeddings :: Identifier │ │ [call] │ │ hcat :: Identifier │ │ [...] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ Float32 :: Identifier │ │ [=] │ │ x :: Identifier │ │ rephrased_questions :: Identifier │ │ [=] │ │ keywords :: Identifier │ │ [if] │ │ [call] │ │ HasKeywords :: Identifier │ │ index :: Identifier │ │ [block] │ │ [=] │ │ keywords :: Identifier │ │ [call] │ │ get_keywords :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ [...] │ │ processor_kwargs :: Identifier │ │ [kw] │ │ return_keywords :: Identifier │ │ true :: Value │ │ processor :: Identifier │ │ rephrased_questions :: Identifier │ │ [&&] │ │ [call] │ │ >= :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ [||] │ │ [call] │ │ isa :: Identifier │ │ keywords :: Identifier │ │ [curly] │ │ AbstractVector :: Identifier │ │ [<:] │ │ [curly] │ │ AbstractVector :: Identifier │ │ [<:] │ │ AbstractString :: Identifier │ │ [macrocall] │ │ @warn :: Identifier │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1104 =#) :: Value │ │ [string] │ │ "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " :: Value │ │ [call] │ │ typeof :: Identifier │ │ processor :: Identifier │ │ ")." :: Value │ │ keywords :: Identifier │ │ [block] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ String :: Identifier │ │ [=] │ │ x :: Identifier │ │ rephrased_questions :: Identifier │ │ [=] │ │ finder_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ finder_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ finder_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ emb_candidates :: Identifier │ │ [call] │ │ find_closest :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ top_k :: Identifier │ │ [...] │ │ finder_kwargs_ :: Identifier │ │ finder :: Identifier │ │ index :: Identifier │ │ embeddings :: Identifier │ │ keywords :: Identifier │ │ [=] │ │ tagger_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ tagger_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ tagger_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ tags :: Identifier │ │ [call] │ │ get_tags :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ tagger_kwargs_ :: Identifier │ │ tagger :: Identifier │ │ rephrased_questions :: Identifier │ │ [=] │ │ filter_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ filter_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ filter_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ tag_candidates :: Identifier │ │ [call] │ │ find_tags :: Identifier │ │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ [...] │ │ filter_kwargs_ :: Identifier │ │ filter :: Identifier │ │ index :: Identifier │ │ tags :: Identifier │ │ [=] │ │ filtered_candidates :: Identifier │ │ [if] │ │ [call] │ │ isnothing :: Identifier │ │ tag_candidates :: Identifier │ │ emb_candidates :: Identifier │ │ [call] │ │ & :: Identifier │ │ emb_candidates :: Identifier │ │ tag_candidates :: Identifier │ │ [=] │ │ reranker_kwargs_ :: Identifier │ │ [if] │ │ [call] │ │ isempty :: Identifier │ │ api_kwargs :: Identifier │ │ reranker_kwargs :: Identifier │ │ [call] │ │ merge :: Identifier │ │ reranker_kwargs :: Identifier │ │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ │ [=] │ │ reranked_candidates :: Identifier │ │ [call] │ │ rerank :: Identifier │ │ [parameters] │ │ top_n :: Identifier │ │ [kw] │ │ verbose :: Identifier │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 1 :: Value │ │ cost_tracker :: Identifier │ │ [...] │ │ reranker_kwargs_ :: Identifier │ │ reranker :: Identifier │ │ index :: Identifier │ │ question :: Identifier │ │ filtered_candidates :: Identifier │ │ [&&] │ │ [call] │ │ > :: Identifier │ │ verbose :: Identifier │ │ 0 :: Value │ │ [macrocall] │ │ @info :: Identifier │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1141 =#) :: Value │ │ [string] │ │ "Retrieval done. Identified " :: Value │ │ [call] │ │ length :: Identifier │ │ [call] │ │ positions :: Identifier │ │ reranked_candidates :: Identifier │ │ " chunks, total cost: \$" :: Value │ │ [call] │ │ round :: Identifier │ │ [ref] │ │ cost_tracker :: Identifier │ │ [kw] │ │ digits :: Identifier │ │ 2 :: Value │ │ "." :: Value │ │ [=] │ │ result :: Identifier │ │ [call] │ │ RAGResult :: Identifier │ │ [parameters] │ │ question :: Identifier │ │ [kw] │ │ answer :: Identifier │ │ nothing :: Identifier │ │ rephrased_questions :: Identifier │ │ [kw] │ │ final_answer :: Identifier │ │ nothing :: Identifier │ │ [kw] │ │ context :: Identifier │ │ [call] │ │ collect :: Identifier │ │ [ref] │ │ index :: Identifier │ │ reranked_candidates :: Identifier │ │ [inert] │ │ chunks :: Identifier │ │ [kw] │ │ sorted :: Identifier │ │ true :: Value │ │ [kw] │ │ sources :: Identifier │ │ [call] │ │ collect :: Identifier │ │ [ref] │ │ index :: Identifier │ │ reranked_candidates :: Identifier │ │ [inert] │ │ sources :: Identifier │ │ [kw] │ │ sorted :: Identifier │ │ true :: Value │ │ emb_candidates :: Identifier │ │ tag_candidates :: Identifier │ │ filtered_candidates :: Identifier │ │ reranked_candidates :: Identifier │ │ [return] │ │ result :: Identifier │ │ │ st1 = │ SyntaxTree with attributes mod,kind,var_id,toplevel_pure,scope_type,macro_source,name_val,syntax_flags,meta,scope_layer,value,jl_source,is_toplevel_thunk,source │ [block] │ │ [=] │ │ val :: Identifier │ scope_layer=3 │ [function] │ │ [call] │ │ retrieve :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ [::] │ │ verbose :: Identifier │ scope_layer=1 │ Integer :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [kw] │ │ [::] │ │ top_k :: Identifier │ scope_layer=1 │ Integer :: Identifier │ scope_layer=1 │ 100 :: Value │ macro_source=485 │ [kw] │ │ [::] │ │ top_n :: Identifier │ scope_layer=1 │ Integer :: Identifier │ scope_layer=1 │ 5 :: Value │ macro_source=485 │ [kw] │ │ [::] │ │ api_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ rephraser :: Identifier │ scope_layer=1 │ AbstractRephraser :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ rephraser :: Identifier │ │ [kw] │ │ [::] │ │ rephraser_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ embedder :: Identifier │ scope_layer=1 │ AbstractEmbedder :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ embedder :: Identifier │ │ [kw] │ │ [::] │ │ embedder_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ processor :: Identifier │ scope_layer=1 │ AbstractProcessor :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ processor :: Identifier │ │ [kw] │ │ [::] │ │ processor_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ finder :: Identifier │ scope_layer=1 │ AbstractSimilarityFinder :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ finder :: Identifier │ │ [kw] │ │ [::] │ │ finder_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ tagger :: Identifier │ scope_layer=1 │ AbstractTagger :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ tagger :: Identifier │ │ [kw] │ │ [::] │ │ tagger_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ filter :: Identifier │ scope_layer=1 │ AbstractTagFilter :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ filter :: Identifier │ │ [kw] │ │ [::] │ │ filter_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ [::] │ │ reranker :: Identifier │ scope_layer=1 │ AbstractReranker :: Identifier │ scope_layer=1 │ [.] │ │ retriever :: Identifier │ scope_layer=1 │ [inert] │ │ reranker :: Identifier │ │ [kw] │ │ [::] │ │ reranker_kwargs :: Identifier │ scope_layer=1 │ NamedTuple :: Identifier │ scope_layer=1 │ [call] │ │ NamedTuple :: Identifier │ scope_layer=1 │ [kw] │ │ cost_tracker :: Identifier │ scope_layer=1 │ [call] │ │ [curly] │ │ [.] │ │ Threads :: Identifier │ scope_layer=1 │ [inert] │ │ Atomic :: Identifier │ │ Float64 :: Identifier │ scope_layer=1 │ 0.0 :: Value │ macro_source=485 │ [...] │ │ kwargs :: Identifier │ scope_layer=1 │ [::] │ │ retriever :: Identifier │ scope_layer=1 │ AbstractRetriever :: Identifier │ scope_layer=1 │ [::] │ │ index :: Identifier │ scope_layer=1 │ AbstractDocumentIndex :: Identifier │ scope_layer=1 │ [::] │ │ question :: Identifier │ scope_layer=1 │ AbstractString :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ rephraser_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ rephraser_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ rephraser_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ rephrased_questions :: Identifier │ scope_layer=1 │ [call] │ │ rephrase :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ rephraser_kwargs_ :: Identifier │ scope_layer=1 │ rephraser :: Identifier │ scope_layer=1 │ question :: Identifier │ scope_layer=1 │ [=] │ │ embeddings :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ HasEmbeddings :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ embedder_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ embedder_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ embedder_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ embeddings :: Identifier │ scope_layer=1 │ [call] │ │ get_embeddings :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ embedder_kwargs_ :: Identifier │ scope_layer=1 │ embedder :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ embeddings :: Identifier │ scope_layer=1 │ [call] │ │ hcat :: Identifier │ scope_layer=1 │ [...] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ Float32 :: Identifier │ scope_layer=1 │ [=] │ │ x :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [=] │ │ keywords :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ HasKeywords :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ keywords :: Identifier │ scope_layer=1 │ [call] │ │ get_keywords :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [...] │ │ processor_kwargs :: Identifier │ scope_layer=1 │ [kw] │ │ return_keywords :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ processor :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [&&] │ │ [call] │ │ >= :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [||] │ │ [call] │ │ isa :: Identifier │ scope_layer=1 │ keywords :: Identifier │ scope_layer=1 │ [curly] │ │ AbstractVector :: Identifier │ scope_layer=1 │ [<:] │ scope_layer=1 │ [curly] │ │ AbstractVector :: Identifier │ scope_layer=1 │ [<:] │ scope_layer=1 │ AbstractString :: Identifier │ scope_layer=1 │ [block] │ │ [let] │ │ [block] │ macro_source=485 │ [block] │ │ [=] │ │ #1131#level :: Identifier │ scope_layer=1 │ Warn :: Identifier │ mod,scope_layer=1 │ [=] │ │ #1132#std_level :: Identifier │ scope_layer=1 │ #1131#level :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ >= :: Identifier │ mod,scope_layer=1 │ [.] │ │ #1132#std_level :: Identifier │ scope_layer=1 │ [inert] │ │ level :: Identifier │ │ [ref] │ macro_source=485 │ Base.Threads.Atomic{Int32}(-1000) :: Value │ macro_source=485 │ [block] │ │ [=] │ │ #1133#group :: Identifier │ scope_layer=1 │ [inert] │ │ retrieval :: Identifier │ │ [=] │ │ #1134#_module :: Identifier │ scope_layer=1 │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [=] │ │ #1135#logger :: Identifier │ scope_layer=1 │ [call] │ │ Base.CoreLogging.current_logger_for_env :: Value │ macro_source=485 │ #1132#std_level :: Identifier │ scope_layer=1 │ #1133#group :: Identifier │ scope_layer=1 │ #1134#_module :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ ! :: Identifier │ mod,scope_layer=1 │ [call] │ │ === :: Identifier │ mod,scope_layer=1 │ #1135#logger :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1136#id :: Identifier │ scope_layer=1 │ [inert] │ │ PromptingTools_Experimental_RAGTools_7aa4399b :: Identifier │ │ [if] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.shouldlog :: Value │ macro_source=485 │ #1135#logger :: Identifier │ scope_layer=1 │ #1131#level :: Identifier │ scope_layer=1 │ #1134#_module :: Identifier │ scope_layer=1 │ #1133#group :: Identifier │ scope_layer=1 │ #1136#id :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ #1137#file :: Identifier │ scope_layer=1 │ "/home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ [if] │ │ [call] │ │ isa :: Identifier │ mod,scope_layer=1 │ #1137#file :: Identifier │ scope_layer=1 │ String :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1137#file :: Identifier │ scope_layer=1 │ [call] │ │ [.] │ │ Base :: Identifier │ mod,scope_layer=1 │ [inert] │ │ fixup_stdlib_path :: Identifier │ │ #1137#file :: Identifier │ scope_layer=1 │ [=] │ │ #1138#line :: Identifier │ scope_layer=1 │ 1104 :: Value │ macro_source=485 │ [local] │ │ #1139#msg :: Identifier │ scope_layer=1 │ #1140#kwargs :: Identifier │ scope_layer=1 │ [if] │ │ [block] │ │ [try] │ │ [block] │ │ [=] │ │ #1139#msg :: Identifier │ scope_layer=1 │ [string] │ │ "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " :: Value │ macro_source=485 │ [call] │ │ typeof :: Identifier │ scope_layer=1 │ processor :: Identifier │ scope_layer=1 │ ")." :: Value │ macro_source=485 │ [=] │ │ #1140#kwargs :: Identifier │ scope_layer=1 │ [tuple] │ macro_source=485 │ [parameters] │ macro_source=485 │ true :: Value │ macro_source=485 │ #1153#err :: Identifier │ scope_layer=1 │ [block] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.logging_error :: Value │ macro_source=485 │ #1135#logger :: Identifier │ scope_layer=1 │ #1131#level :: Identifier │ scope_layer=1 │ #1134#_module :: Identifier │ scope_layer=1 │ #1133#group :: Identifier │ scope_layer=1 │ #1136#id :: Identifier │ scope_layer=1 │ #1137#file :: Identifier │ scope_layer=1 │ #1138#line :: Identifier │ scope_layer=1 │ #1153#err :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ false :: Value │ macro_source=485 │ [block] │ │ [if] │ │ [isdefined] │ │ #1139#msg :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [if] │ │ [isdefined] │ │ #1140#kwargs :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [call] │ │ Base.CoreLogging.handle_message_nothrow :: Value │ macro_source=485 │ [parameters] │ │ [...] │ │ #1140#kwargs :: Identifier │ scope_layer=1 │ #1135#logger :: Identifier │ scope_layer=1 │ #1131#level :: Identifier │ scope_layer=1 │ #1139#msg :: Identifier │ scope_layer=1 │ #1134#_module :: Identifier │ scope_layer=1 │ #1133#group :: Identifier │ scope_layer=1 │ #1136#id :: Identifier │ scope_layer=1 │ #1137#file :: Identifier │ scope_layer=1 │ #1138#line :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ keywords :: Identifier │ scope_layer=1 │ [block] │ │ [comprehension] │ │ [generator] │ │ [ref] │ │ String :: Identifier │ scope_layer=1 │ [=] │ │ x :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [=] │ │ finder_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ finder_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ finder_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ emb_candidates :: Identifier │ scope_layer=1 │ [call] │ │ find_closest :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ top_k :: Identifier │ scope_layer=1 │ [...] │ │ finder_kwargs_ :: Identifier │ scope_layer=1 │ finder :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ embeddings :: Identifier │ scope_layer=1 │ keywords :: Identifier │ scope_layer=1 │ [=] │ │ tagger_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ tagger_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ tagger_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ tags :: Identifier │ scope_layer=1 │ [call] │ │ get_tags :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ tagger_kwargs_ :: Identifier │ scope_layer=1 │ tagger :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [=] │ │ filter_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ filter_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ filter_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ tag_candidates :: Identifier │ scope_layer=1 │ [call] │ │ find_tags :: Identifier │ scope_layer=1 │ [parameters] │ │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ [...] │ │ filter_kwargs_ :: Identifier │ scope_layer=1 │ filter :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ tags :: Identifier │ scope_layer=1 │ [=] │ │ filtered_candidates :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isnothing :: Identifier │ scope_layer=1 │ tag_candidates :: Identifier │ scope_layer=1 │ emb_candidates :: Identifier │ scope_layer=1 │ [call] │ │ & :: Identifier │ scope_layer=1 │ emb_candidates :: Identifier │ scope_layer=1 │ tag_candidates :: Identifier │ scope_layer=1 │ [=] │ │ reranker_kwargs_ :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ isempty :: Identifier │ scope_layer=1 │ api_kwargs :: Identifier │ scope_layer=1 │ reranker_kwargs :: Identifier │ scope_layer=1 │ [call] │ │ merge :: Identifier │ scope_layer=1 │ reranker_kwargs :: Identifier │ scope_layer=1 │ [tuple] │ │ [parameters] │ │ api_kwargs :: Identifier │ scope_layer=1 │ [=] │ │ reranked_candidates :: Identifier │ scope_layer=1 │ [call] │ │ rerank :: Identifier │ scope_layer=1 │ [parameters] │ │ top_n :: Identifier │ scope_layer=1 │ [kw] │ │ verbose :: Identifier │ scope_layer=1 │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 1 :: Value │ macro_source=485 │ cost_tracker :: Identifier │ scope_layer=1 │ [...] │ │ reranker_kwargs_ :: Identifier │ scope_layer=1 │ reranker :: Identifier │ scope_layer=1 │ index :: Identifier │ scope_layer=1 │ question :: Identifier │ scope_layer=1 │ filtered_candidates :: Identifier │ scope_layer=1 │ [&&] │ │ [call] │ │ > :: Identifier │ scope_layer=1 │ verbose :: Identifier │ scope_layer=1 │ 0 :: Value │ macro_source=485 │ [block] │ │ [let] │ │ [block] │ macro_source=485 │ [block] │ │ [=] │ │ #1154#level :: Identifier │ scope_layer=1 │ Info :: Identifier │ mod,scope_layer=1 │ [=] │ │ #1155#std_level :: Identifier │ scope_layer=1 │ #1154#level :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ >= :: Identifier │ mod,scope_layer=1 │ [.] │ │ #1155#std_level :: Identifier │ scope_layer=1 │ [inert] │ │ level :: Identifier │ │ [ref] │ macro_source=485 │ Base.Threads.Atomic{Int32}(-1000) :: Value │ macro_source=485 │ [block] │ │ [=] │ │ #1156#group :: Identifier │ scope_layer=1 │ [inert] │ │ retrieval :: Identifier │ │ [=] │ │ #1157#_module :: Identifier │ scope_layer=1 │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [=] │ │ #1158#logger :: Identifier │ scope_layer=1 │ [call] │ │ Base.CoreLogging.current_logger_for_env :: Value │ macro_source=485 │ #1155#std_level :: Identifier │ scope_layer=1 │ #1156#group :: Identifier │ scope_layer=1 │ #1157#_module :: Identifier │ scope_layer=1 │ [if] │ │ [call] │ │ ! :: Identifier │ mod,scope_layer=1 │ [call] │ │ === :: Identifier │ mod,scope_layer=1 │ #1158#logger :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1159#id :: Identifier │ scope_layer=1 │ [inert] │ │ PromptingTools_Experimental_RAGTools_30253e51 :: Identifier │ │ [if] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.shouldlog :: Value │ macro_source=485 │ #1158#logger :: Identifier │ scope_layer=1 │ #1154#level :: Identifier │ scope_layer=1 │ #1157#_module :: Identifier │ scope_layer=1 │ #1156#group :: Identifier │ scope_layer=1 │ #1159#id :: Identifier │ scope_layer=1 │ [block] │ │ [=] │ │ #1160#file :: Identifier │ scope_layer=1 │ "/home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ [if] │ │ [call] │ │ isa :: Identifier │ mod,scope_layer=1 │ #1160#file :: Identifier │ scope_layer=1 │ String :: Identifier │ mod,scope_layer=1 │ [block] │ │ [=] │ │ #1160#file :: Identifier │ scope_layer=1 │ [call] │ │ [.] │ │ Base :: Identifier │ mod,scope_layer=1 │ [inert] │ │ fixup_stdlib_path :: Identifier │ │ #1160#file :: Identifier │ scope_layer=1 │ [=] │ │ #1161#line :: Identifier │ scope_layer=1 │ 1141 :: Value │ macro_source=485 │ [local] │ │ #1162#msg :: Identifier │ scope_layer=1 │ #1163#kwargs :: Identifier │ scope_layer=1 │ [if] │ │ [block] │ │ [try] │ │ [block] │ │ [=] │ │ #1162#msg :: Identifier │ scope_layer=1 │ [string] │ │ "Retrieval done. Identified " :: Value │ macro_source=485 │ [call] │ │ length :: Identifier │ scope_layer=1 │ [call] │ │ positions :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ " chunks, total cost: \$" :: Value │ macro_source=485 │ [call] │ │ round :: Identifier │ scope_layer=1 │ [ref] │ │ cost_tracker :: Identifier │ scope_layer=1 │ [kw] │ │ digits :: Identifier │ scope_layer=1 │ 2 :: Value │ macro_source=485 │ "." :: Value │ macro_source=485 │ [=] │ │ #1163#kwargs :: Identifier │ scope_layer=1 │ [tuple] │ macro_source=485 │ [parameters] │ macro_source=485 │ true :: Value │ macro_source=485 │ #1176#err :: Identifier │ scope_layer=1 │ [block] │ │ [call] │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ Base.CoreLogging.logging_error :: Value │ macro_source=485 │ #1158#logger :: Identifier │ scope_layer=1 │ #1154#level :: Identifier │ scope_layer=1 │ #1157#_module :: Identifier │ scope_layer=1 │ #1156#group :: Identifier │ scope_layer=1 │ #1159#id :: Identifier │ scope_layer=1 │ #1160#file :: Identifier │ scope_layer=1 │ #1161#line :: Identifier │ scope_layer=1 │ #1176#err :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ false :: Value │ macro_source=485 │ [block] │ │ [if] │ │ [isdefined] │ │ #1162#msg :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [if] │ │ [isdefined] │ │ #1163#kwargs :: Identifier │ scope_layer=1 │ nothing :: Value │ macro_source=485 │ [call] │ │ throw :: Identifier │ mod,scope_layer=1 │ [call] │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ [call] │ │ Base.CoreLogging.handle_message_nothrow :: Value │ macro_source=485 │ [parameters] │ │ [...] │ │ #1163#kwargs :: Identifier │ scope_layer=1 │ #1158#logger :: Identifier │ scope_layer=1 │ #1154#level :: Identifier │ scope_layer=1 │ #1162#msg :: Identifier │ scope_layer=1 │ #1157#_module :: Identifier │ scope_layer=1 │ #1156#group :: Identifier │ scope_layer=1 │ #1159#id :: Identifier │ scope_layer=1 │ #1160#file :: Identifier │ scope_layer=1 │ #1161#line :: Identifier │ scope_layer=1 │ nothing :: Identifier │ mod,scope_layer=1 │ [=] │ │ result :: Identifier │ scope_layer=1 │ [call] │ │ RAGResult :: Identifier │ scope_layer=1 │ [parameters] │ │ question :: Identifier │ scope_layer=1 │ [kw] │ │ answer :: Identifier │ scope_layer=1 │ nothing :: Identifier │ scope_layer=1 │ rephrased_questions :: Identifier │ scope_layer=1 │ [kw] │ │ final_answer :: Identifier │ scope_layer=1 │ nothing :: Identifier │ scope_layer=1 │ [kw] │ │ context :: Identifier │ scope_layer=1 │ [call] │ │ collect :: Identifier │ scope_layer=1 │ [ref] │ │ index :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ [inert] │ │ chunks :: Identifier │ │ [kw] │ │ sorted :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ [kw] │ │ sources :: Identifier │ scope_layer=1 │ [call] │ │ collect :: Identifier │ scope_layer=1 │ [ref] │ │ index :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ [inert] │ │ sources :: Identifier │ │ [kw] │ │ sorted :: Identifier │ scope_layer=1 │ true :: Value │ macro_source=485 │ emb_candidates :: Identifier │ scope_layer=1 │ tag_candidates :: Identifier │ scope_layer=1 │ filtered_candidates :: Identifier │ scope_layer=1 │ reranked_candidates :: Identifier │ scope_layer=1 │ [return] │ │ result :: Identifier │ scope_layer=1 │ [call] │ │ Base.Docs.doc! :: Value │ macro_source=485 │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [call] │ │ Base.Docs.Binding :: Value │ macro_source=485 │ PromptingTools.Experimental.RAGTools :: Value │ │ [inert] │ jl_source=L65 │ retrieve :: Identifier │ │ [call] │ macro_source=485 │ Base.Docs.docstr :: Value │ macro_source=485 │ [call] │ macro_source=485 │ Core.svec :: Value │ macro_source=485 │ " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" :: Value │ macro_source=485 │ [call] │ macro_source=485 │ Dict{Symbol, Any} :: Value │ macro_source=485 │ :path => "/home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ :linenumber => 946 :: Value │ macro_source=485 │ :module => PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ [curly] │ │ Union :: Identifier │ scope_layer=1 │ [curly] │ │ Tuple :: Identifier │ scope_layer=1 │ AbstractRetriever :: Identifier │ scope_layer=1 │ AbstractDocumentIndex :: Identifier │ scope_layer=1 │ AbstractString :: Identifier │ scope_layer=1 │ val :: Identifier │ scope_layer=3 │ │ file = "/home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl" │ line = 946 └ mod = PromptingTools.Experimental.RAGTools ERROR: LoadError: LoweringError: #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1144 =# - invalid syntax: unknown form `kw` or number of arguments 2 Expression:  (kw sorted true) Containing expressions:  (kw context (call collect (ref index reranked_candidates (inert chunks) (kw sorted true))))  Detailed provenance:  (kw sorted true)  └─ (kw sorted true)  ├─ @ /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1144  └─ (macrocall @doc :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946 =#) " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" (function (call retrieve (parameters (kw (:: verbose Integer) 1) (kw (:: top_k Integer) 100) (kw (:: top_n Integer) 5) (kw (:: api_kwargs NamedTuple) (call NamedTuple)) (kw (:: rephraser AbstractRephraser) (. retriever (inert rephraser))) (kw (:: rephraser_kwargs NamedTuple) (call NamedTuple)) (kw (:: embedder AbstractEmbedder) (. retriever (inert embedder))) (kw (:: embedder_kwargs NamedTuple) (call NamedTuple)) (kw (:: processor AbstractProcessor) (. retriever (inert processor))) (kw (:: processor_kwargs NamedTuple) (call NamedTuple)) (kw (:: finder AbstractSimilarityFinder) (. retriever (inert finder))) (kw (:: finder_kwargs NamedTuple) (call NamedTuple)) (kw (:: tagger AbstractTagger) (. retriever (inert tagger))) (kw (:: tagger_kwargs NamedTuple) (call NamedTuple)) (kw (:: filter AbstractTagFilter) (. retriever (inert filter))) (kw (:: filter_kwargs NamedTuple) (call NamedTuple)) (kw (:: reranker AbstractReranker) (. retriever (inert reranker))) (kw (:: reranker_kwargs NamedTuple) (call NamedTuple)) (kw cost_tracker (call (curly (. Threads (inert Atomic)) Float64) 0.0)) (... kwargs)) (:: retriever AbstractRetriever) (:: index AbstractDocumentIndex) (:: question AbstractString)) (block (= rephraser_kwargs_ (if (call isempty api_kwargs) rephraser_kwargs (call merge rephraser_kwargs (tuple (parameters api_kwargs))))) (= rephrased_questions (call rephrase (parameters (kw verbose (call > verbose 1)) cost_tracker (... rephraser_kwargs_)) rephraser question)) (= embeddings (if (call HasEmbeddings index) (block (= embedder_kwargs_ (if (call isempty api_kwargs) embedder_kwargs (call merge embedder_kwargs (tuple (parameters api_kwargs))))) (= embeddings (call get_embeddings (parameters (kw verbose (call > verbose 1)) cost_tracker (... embedder_kwargs_)) embedder rephrased_questions))) (block (= embeddings (call hcat (... (comprehension (generator (ref Float32) (= x rephrased_questions))))))))) (= keywords (if (call HasKeywords index) (block (= keywords (call get_keywords (parameters (kw verbose (call > verbose 1)) (... processor_kwargs) (kw return_keywords true)) processor rephrased_questions)) (&& (call >= verbose 1) (|| (call isa keywords (curly AbstractVector (<: (curly AbstractVector (<: AbstractString))))) (macrocall @warn :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1104 =#) (string "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " (call typeof processor) ").")))) keywords) (block (comprehension (generator (ref String) (= x rephrased_questions)))))) (= finder_kwargs_ (if (call isempty api_kwargs) finder_kwargs (call merge finder_kwargs (tuple (parameters api_kwargs))))) (= emb_candidates (call find_closest (parameters (kw verbose (call > verbose 1)) top_k (... finder_kwargs_)) finder index embeddings keywords)) (= tagger_kwargs_ (if (call isempty api_kwargs) tagger_kwargs (call merge tagger_kwargs (tuple (parameters api_kwargs))))) (= tags (call get_tags (parameters (kw verbose (call > verbose 1)) cost_tracker (... tagger_kwargs_)) tagger rephrased_questions)) (= filter_kwargs_ (if (call isempty api_kwargs) filter_kwargs (call merge filter_kwargs (tuple (parameters api_kwargs))))) (= tag_candidates (call find_tags (parameters (kw verbose (call > verbose 1)) (... filter_kwargs_)) filter index tags)) (= filtered_candidates (if (call isnothing tag_candidates) emb_candidates (call & emb_candidates tag_candidates))) (= reranker_kwargs_ (if (call isempty api_kwargs) reranker_kwargs (call merge reranker_kwargs (tuple (parameters api_kwargs))))) (= reranked_candidates (call rerank (parameters top_n (kw verbose (call > verbose 1)) cost_tracker (... reranker_kwargs_)) reranker index question filtered_candidates)) (&& (call > verbose 0) (macrocall @info :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1141 =#) (string "Retrieval done. Identified " (call length (call positions reranked_candidates)) " chunks, total cost: \$" (call round (ref cost_tracker) (kw digits 2)) "."))) (= result (call RAGResult (parameters question (kw answer nothing) rephrased_questions (kw final_answer nothing) (kw context (call collect (ref index reranked_candidates (inert chunks) (kw sorted true)))) (kw sources (call collect (ref index reranked_candidates (inert sources) (kw sorted true)))) emb_candidates tag_candidates filtered_candidates reranked_candidates))) (return result))))  └─ @ /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946  Stacktrace:  [1] expand_forms_2(ctx::Base.JuliaLowering.MacroExpansionContext{Dict{Symbol, Dict{Int64, Any}}}, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}})  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/desugaring.jl:4440  [2] core_lowering_hook(code::Any, mod::Module, file::String, line::UInt64, world::UInt64, _warn::Bool)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/hooks.jl:30  [3] include(mapexpr::Function, mod::Module, _path::String)  @ Base Base.jl:326  [4] top-level scope  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:1  [5] macro expansion  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:47 [inlined]  [6] eval(m::Module, e::Any)  @ Core boot.jl:522  [7] _eval(mod::Module, iter::Base.JuliaLowering.LoweringIterator{Dict{Symbol, Dict{Int64, Any}}}; soft_scope::Nothing)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:547  [8] eval(mod::Module, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}; macro_world::UInt64, soft_scope::Nothing, opts::@Kwargs{expr_compat_mode::Bool})  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:520 [inlined]  [9] top-level scope  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:1  [10] macro expansion  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:1 [inlined]  [11] include(mapexpr::Function, mod::Module, _path::String)  @ Base Base.jl:326  [12] top-level scope  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:1  [13] macro expansion  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:18 [inlined]  [14] eval(m::Module, e::Any)  @ Core boot.jl:522  [15] _eval(mod::Module, iter::Base.JuliaLowering.LoweringIterator{Dict{Symbol, Dict{Int64, Any}}}; soft_scope::Nothing)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:547  [16] eval(mod::Module, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}; macro_world::UInt64, soft_scope::Nothing, opts::@Kwargs{expr_compat_mode::Bool})  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:520 [inlined]  [17] top-level scope  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:1  [18] macro expansion  @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:1 [inlined]  [19] include(mapexpr::Function, mod::Module, _path::String)  @ Base Base.jl:326  [20] top-level scope  @ ~/.julia/packages/PromptingTools/aTolC/src/PromptingTools.jl:108  [21] include(mod::Module, _path::String)  @ Base Base.jl:325  [22] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)  @ Base loading.jl:3296  [23] top-level scope  @ stdin:5  [24] eval(m::Module, e::Any)  @ Core boot.jl:522  [25] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)  @ Base loading.jl:3132  [26] include_string(m::Module, txt::String, fname::String)  @ Base loading.jl:3142 [inlined]  [27] exec_options(opts::Base.JLOptions)  @ Base client.jl:353  [28] _start()  @ Base client.jl:596 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:1 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:1 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/PromptingTools.jl:1 in expression starting at stdin:5 ERROR: LoadError: Precompiled image Base.PkgId(Base.UUID("670122d1-24a8-4d70-bfce-740807c42192"), "PromptingTools") not available with flags CacheFlags(; use_pkgimages=false, debug_level=1, check_bounds=1, inline=true, opt_level=0) Stacktrace:  [1] error(s::String)  @ Base error.jl:56  [2] __require_prelocked(pkg::Base.PkgId, env::String)  @ Base loading.jl:2837  [3] _require_prelocked(uuidkey::Base.PkgId, env::String)  @ Base loading.jl:2685  [4] macro expansion  @ loading.jl:2599 [inlined]  [5] macro expansion  @ lock.jl:376 [inlined]  [6] __require(into::Module, mod::Symbol)  @ Base loading.jl:2563  [7] require(into::Module, mod::Symbol)  @ Base loading.jl:2539 [inlined]  [8] eval_import_path(at::Module, from::Nothing, path::Expr, keyword::String)  @ Base module.jl:36 [inlined]  [9] eval_import_path_all(at::Module, path::Expr, keyword::String)  @ Base module.jl:60  [10] _eval_using(to::Module, path::Expr, flags::UInt8)  @ Base module.jl:137 [inlined]  [11] eval_using(to::Module, path::Expr)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/runtime.jl:207  [12] top-level scope  @ ~/.julia/packages/PromptingTools/aTolC/ext/MarkdownPromptingToolsExt.jl:3  [13] include(mod::Module, _path::String)  @ Base Base.jl:325  [14] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)  @ Base loading.jl:3296  [15] top-level scope  @ stdin:5  [16] eval(m::Module, e::Any)  @ Core boot.jl:522  [17] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)  @ Base loading.jl:3132  [18] include_string(m::Module, txt::String, fname::String)  @ Base loading.jl:3142 [inlined]  [19] exec_options(opts::Base.JLOptions)  @ Base client.jl:353  [20] _start()  @ Base client.jl:596 in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/ext/MarkdownPromptingToolsExt.jl:1 in expression starting at stdin:5 ERROR: LoadError: Precompiled image Base.PkgId(Base.UUID("670122d1-24a8-4d70-bfce-740807c42192"), "PromptingTools") not available with flags CacheFlags(; use_pkgimages=false, debug_level=1, check_bounds=1, inline=true, opt_level=0) Stacktrace:  [1] error(s::String)  @ Base error.jl:56  [2] __require_prelocked(pkg::Base.PkgId, env::String)  @ Base loading.jl:2837  [3] _require_prelocked(uuidkey::Base.PkgId, env::String)  @ Base loading.jl:2685  [4] macro expansion  @ loading.jl:2599 [inlined]  [5] macro expansion  @ lock.jl:376 [inlined]  [6] __require(into::Module, mod::Symbol)  @ Base loading.jl:2563  [7] require(into::Module, mod::Symbol)  @ Base loading.jl:2539 [inlined]  [8] eval_import_path(at::Module, from::Nothing, path::Expr, keyword::String)  @ Base module.jl:36 [inlined]  [9] eval_import_path_all(at::Module, path::Expr, keyword::String)  @ Base module.jl:60  [10] _eval_using(to::Module, path::Expr, flags::UInt8)  @ Base module.jl:137 [inlined]  [11] eval_using(to::Module, path::Expr)  @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/runtime.jl:207  [12] top-level scope  @ ~/.julia/packages/SwarmAgents/qtRuA/src/SwarmAgents.jl:4  [13] include(mod::Module, _path::String)  @ Base Base.jl:325  [14] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)  @ Base loading.jl:3296  [15] top-level scope  @ stdin:5  [16] eval(m::Module, e::Any)  @ Core boot.jl:522  [17] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)  @ Base loading.jl:3132  [18] include_string(m::Module, txt::String, fname::String)  @ Base loading.jl:3142 [inlined]  [19] exec_options(opts::Base.JLOptions)  @ Base client.jl:353  [20] _start()  @ Base client.jl:596 in expression starting at /home/pkgeval/.julia/packages/SwarmAgents/qtRuA/src/SwarmAgents.jl:1 in expression starting at stdin:5 3 dependencies had output during precompilation: ┌ PromptingTools → MarkdownPromptingToolsExt │ ERROR: LoadError: Precompiled image Base.PkgId(Base.UUID("670122d1-24a8-4d70-bfce-740807c42192"), "PromptingTools") not available with flags CacheFlags(; use_pkgimages=false, debug_level=1, check_bounds=1, inline=true, opt_level=0) │ Stacktrace: │ [1] error(s::String) │ @ Base error.jl:56 │ [2] __require_prelocked(pkg::Base.PkgId, env::String) │ @ Base loading.jl:2837 │ [3] _require_prelocked(uuidkey::Base.PkgId, env::String) │ @ Base loading.jl:2685 │ [4] macro expansion │ @ loading.jl:2599 [inlined] │ [5] macro expansion │ @ lock.jl:376 [inlined] │ [6] __require(into::Module, mod::Symbol) │ @ Base loading.jl:2563 │ [7] require(into::Module, mod::Symbol) │ @ Base loading.jl:2539 [inlined] │ [8] eval_import_path(at::Module, from::Nothing, path::Expr, keyword::String) │ @ Base module.jl:36 [inlined] │ [9] eval_import_path_all(at::Module, path::Expr, keyword::String) │ @ Base module.jl:60 │ [10] _eval_using(to::Module, path::Expr, flags::UInt8) │ @ Base module.jl:137 [inlined] │ [11] eval_using(to::Module, path::Expr) │ @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/runtime.jl:207 │ [12] top-level scope │ @ ~/.julia/packages/PromptingTools/aTolC/ext/MarkdownPromptingToolsExt.jl:3 │ [13] include(mod::Module, _path::String) │ @ Base Base.jl:325 │ [14] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing) │ @ Base loading.jl:3296 │ [15] top-level scope │ @ stdin:5 │ [16] eval(m::Module, e::Any) │ @ Core boot.jl:522 │ [17] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) │ @ Base loading.jl:3132 │ [18] include_string(m::Module, txt::String, fname::String) │ @ Base loading.jl:3142 [inlined] │ [19] exec_options(opts::Base.JLOptions) │ @ Base client.jl:353 │ [20] _start() │ @ Base client.jl:596 │ in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/ext/MarkdownPromptingToolsExt.jl:1 │ in expression starting at stdin:5 └ ┌ SwarmAgents │ [Output was shown above] └ ┌ PromptingTools │ ┌ Warning: OPENAI_API_KEY variable not set! OpenAI models will not be available - set API key directly via `PromptingTools.OPENAI_API_KEY=`! │ └ @ PromptingTools ~/.julia/packages/PromptingTools/aTolC/src/user_preferences.jl:181 │ ┌ Info: JuliaLowering threw given input: │ │ code = │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946 =# Core.@doc " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" function retrieve(retriever::AbstractRetriever, index::AbstractDocumentIndex, question::AbstractString; verbose::Integer = 1, top_k::Integer = 100, top_n::Integer = 5, api_kwargs::NamedTuple = NamedTuple(), rephraser::AbstractRephraser = retriever.rephraser, rephraser_kwargs::NamedTuple = NamedTuple(), embedder::AbstractEmbedder = retriever.embedder, embedder_kwargs::NamedTuple = NamedTuple(), processor::AbstractProcessor = retriever.processor, processor_kwargs::NamedTuple = NamedTuple(), finder::AbstractSimilarityFinder = retriever.finder, finder_kwargs::NamedTuple = NamedTuple(), tagger::AbstractTagger = retriever.tagger, tagger_kwargs::NamedTuple = NamedTuple(), filter::AbstractTagFilter = retriever.filter, filter_kwargs::NamedTuple = NamedTuple(), reranker::AbstractReranker = retriever.reranker, reranker_kwargs::NamedTuple = NamedTuple(), cost_tracker = Threads.Atomic{Float64}(0.0), kwargs...) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1058 =# │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1082 =# │ │ rephraser_kwargs_ = if isempty(api_kwargs) │ │ rephraser_kwargs │ │ else │ │ merge(rephraser_kwargs, (; api_kwargs)) │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1084 =# │ │ rephrased_questions = rephrase(rephraser, question; verbose = verbose > 1, cost_tracker, rephraser_kwargs_...) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1088 =# │ │ embeddings = if HasEmbeddings(index) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1089 =# │ │ embedder_kwargs_ = if isempty(api_kwargs) │ │ embedder_kwargs │ │ else │ │ merge(embedder_kwargs, (; api_kwargs)) │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1091 =# │ │ embeddings = get_embeddings(embedder, rephrased_questions; verbose = verbose > 1, cost_tracker, embedder_kwargs_...) │ │ else │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1094 =# │ │ embeddings = hcat([Float32[] for x = rephrased_questions]...) │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1098 =# │ │ keywords = if HasKeywords(index) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1100 =# │ │ keywords = get_keywords(processor, rephrased_questions; verbose = verbose > 1, processor_kwargs..., return_keywords = true) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1103 =# │ │ verbose >= 1 && (keywords isa AbstractVector{<:AbstractVector{<:AbstractString}} || #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1104 =# @warn("Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: $(typeof(processor))).")) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1105 =# │ │ keywords │ │ else │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1107 =# │ │ [String[] for x = rephrased_questions] │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1110 =# │ │ finder_kwargs_ = if isempty(api_kwargs) │ │ finder_kwargs │ │ else │ │ merge(finder_kwargs, (; api_kwargs)) │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1112 =# │ │ emb_candidates = find_closest(finder, index, embeddings, keywords; verbose = verbose > 1, top_k, finder_kwargs_...) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1116 =# │ │ tagger_kwargs_ = if isempty(api_kwargs) │ │ tagger_kwargs │ │ else │ │ merge(tagger_kwargs, (; api_kwargs)) │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1118 =# │ │ tags = get_tags(tagger, rephrased_questions; verbose = verbose > 1, cost_tracker, tagger_kwargs_...) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1121 =# │ │ filter_kwargs_ = if isempty(api_kwargs) │ │ filter_kwargs │ │ else │ │ merge(filter_kwargs, (; api_kwargs)) │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1123 =# │ │ tag_candidates = find_tags(filter, index, tags; verbose = verbose > 1, filter_kwargs_...) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1128 =# │ │ filtered_candidates = if isnothing(tag_candidates) │ │ emb_candidates │ │ else │ │ emb_candidates & tag_candidates │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1135 =# │ │ reranker_kwargs_ = if isempty(api_kwargs) │ │ reranker_kwargs │ │ else │ │ merge(reranker_kwargs, (; api_kwargs)) │ │ end │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1137 =# │ │ reranked_candidates = rerank(reranker, index, question, filtered_candidates; top_n, verbose = verbose > 1, cost_tracker, reranker_kwargs_...) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1140 =# │ │ verbose > 0 && #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1141 =# @info("Retrieval done. Identified $(length(positions(reranked_candidates))) chunks, total cost: \$$(round(cost_tracker[], digits = 2)).") │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1144 =# │ │ result = RAGResult(; question, answer = nothing, rephrased_questions, final_answer = nothing, context = collect(index[reranked_candidates, :chunks, sorted = true]), sources = collect(index[reranked_candidates, :sources, sorted = true]), emb_candidates, tag_candidates, filtered_candidates, reranked_candidates) │ │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1157 =# │ │ return result │ │ end) │ │ st0 = │ │ SyntaxTree with attributes mod,kind,var_id,toplevel_pure,scope_type,macro_source,name_val,syntax_flags,meta,scope_layer,value,jl_source,is_toplevel_thunk,source,__macro_ctx__ │ │ [macrocall] │ │ │ @doc :: Identifier │ mod │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946 =#) :: Value │ │ │ " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" :: Value │ │ │ [function] │ │ │ [call] │ │ │ retrieve :: Identifier │ │ │ [parameters] │ │ │ [kw] │ │ │ [::] │ │ │ verbose :: Identifier │ │ │ Integer :: Identifier │ │ │ 1 :: Value │ │ │ [kw] │ │ │ [::] │ │ │ top_k :: Identifier │ │ │ Integer :: Identifier │ │ │ 100 :: Value │ │ │ [kw] │ │ │ [::] │ │ │ top_n :: Identifier │ │ │ Integer :: Identifier │ │ │ 5 :: Value │ │ │ [kw] │ │ │ [::] │ │ │ api_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ rephraser :: Identifier │ │ │ AbstractRephraser :: Identifier │ │ │ [.] │ │ │ retriever :: Identifier │ │ │ [inert] │ │ │ rephraser :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ rephraser_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ embedder :: Identifier │ │ │ AbstractEmbedder :: Identifier │ │ │ [.] │ │ │ retriever :: Identifier │ │ │ [inert] │ │ │ embedder :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ embedder_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ processor :: Identifier │ │ │ AbstractProcessor :: Identifier │ │ │ [.] │ │ │ retriever :: Identifier │ │ │ [inert] │ │ │ processor :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ processor_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ finder :: Identifier │ │ │ AbstractSimilarityFinder :: Identifier │ │ │ [.] │ │ │ retriever :: Identifier │ │ │ [inert] │ │ │ finder :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ finder_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ tagger :: Identifier │ │ │ AbstractTagger :: Identifier │ │ │ [.] │ │ │ retriever :: Identifier │ │ │ [inert] │ │ │ tagger :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ tagger_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ filter :: Identifier │ │ │ AbstractTagFilter :: Identifier │ │ │ [.] │ │ │ retriever :: Identifier │ │ │ [inert] │ │ │ filter :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ filter_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ reranker :: Identifier │ │ │ AbstractReranker :: Identifier │ │ │ [.] │ │ │ retriever :: Identifier │ │ │ [inert] │ │ │ reranker :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ reranker_kwargs :: Identifier │ │ │ NamedTuple :: Identifier │ │ │ [call] │ │ │ NamedTuple :: Identifier │ │ │ [kw] │ │ │ cost_tracker :: Identifier │ │ │ [call] │ │ │ [curly] │ │ │ [.] │ │ │ Threads :: Identifier │ │ │ [inert] │ │ │ Atomic :: Identifier │ │ │ Float64 :: Identifier │ │ │ 0.0 :: Value │ │ │ [...] │ │ │ kwargs :: Identifier │ │ │ [::] │ │ │ retriever :: Identifier │ │ │ AbstractRetriever :: Identifier │ │ │ [::] │ │ │ index :: Identifier │ │ │ AbstractDocumentIndex :: Identifier │ │ │ [::] │ │ │ question :: Identifier │ │ │ AbstractString :: Identifier │ │ │ [block] │ │ │ [=] │ │ │ rephraser_kwargs_ :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ │ │ api_kwargs :: Identifier │ │ │ rephraser_kwargs :: Identifier │ │ │ [call] │ │ │ merge :: Identifier │ │ │ rephraser_kwargs :: Identifier │ │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ │ │ [=] │ │ │ rephrased_questions :: Identifier │ │ │ [call] │ │ │ rephrase :: Identifier │ │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ cost_tracker :: Identifier │ │ │ [...] │ │ │ rephraser_kwargs_ :: Identifier │ │ │ rephraser :: Identifier │ │ │ question :: Identifier │ │ │ [=] │ │ │ embeddings :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ HasEmbeddings :: Identifier │ │ │ index :: Identifier │ │ │ [block] │ │ │ [=] │ │ │ embedder_kwargs_ :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ │ │ api_kwargs :: Identifier │ │ │ embedder_kwargs :: Identifier │ │ │ [call] │ │ │ merge :: Identifier │ │ │ embedder_kwargs :: Identifier │ │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ │ │ [=] │ │ │ embeddings :: Identifier │ │ │ [call] │ │ │ get_embeddings :: Identifier │ │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ cost_tracker :: Identifier │ │ │ [...] │ │ │ embedder_kwargs_ :: Identifier │ │ │ embedder :: Identifier │ │ │ rephrased_questions :: Identifier │ │ │ [block] │ │ │ [=] │ │ │ embeddings :: Identifier │ │ │ [call] │ │ │ hcat :: Identifier │ │ │ [...] │ │ │ [comprehension] │ │ │ [generator] │ │ │ [ref] │ │ │ Float32 :: Identifier │ │ │ [=] │ │ │ x :: Identifier │ │ │ rephrased_questions :: Identifier │ │ │ [=] │ │ │ keywords :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ HasKeywords :: Identifier │ │ │ index :: Identifier │ │ │ [block] │ │ │ [=] │ │ │ keywords :: Identifier │ │ │ [call] │ │ │ get_keywords :: Identifier │ │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ [...] │ │ │ processor_kwargs :: Identifier │ │ │ [kw] │ │ │ return_keywords :: Identifier │ │ │ true :: Value │ │ │ processor :: Identifier │ │ │ rephrased_questions :: Identifier │ │ │ [&&] │ │ │ [call] │ │ │ >= :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ [||] │ │ │ [call] │ │ │ isa :: Identifier │ │ │ keywords :: Identifier │ │ │ [curly] │ │ │ AbstractVector :: Identifier │ │ │ [<:] │ │ │ [curly] │ │ │ AbstractVector :: Identifier │ │ │ [<:] │ │ │ AbstractString :: Identifier │ │ │ [macrocall] │ │ │ @warn :: Identifier │ │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1104 =#) :: Value │ │ │ [string] │ │ │ "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " :: Value │ │ │ [call] │ │ │ typeof :: Identifier │ │ │ processor :: Identifier │ │ │ ")." :: Value │ │ │ keywords :: Identifier │ │ │ [block] │ │ │ [comprehension] │ │ │ [generator] │ │ │ [ref] │ │ │ String :: Identifier │ │ │ [=] │ │ │ x :: Identifier │ │ │ rephrased_questions :: Identifier │ │ │ [=] │ │ │ finder_kwargs_ :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ │ │ api_kwargs :: Identifier │ │ │ finder_kwargs :: Identifier │ │ │ [call] │ │ │ merge :: Identifier │ │ │ finder_kwargs :: Identifier │ │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ │ │ [=] │ │ │ emb_candidates :: Identifier │ │ │ [call] │ │ │ find_closest :: Identifier │ │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ top_k :: Identifier │ │ │ [...] │ │ │ finder_kwargs_ :: Identifier │ │ │ finder :: Identifier │ │ │ index :: Identifier │ │ │ embeddings :: Identifier │ │ │ keywords :: Identifier │ │ │ [=] │ │ │ tagger_kwargs_ :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ │ │ api_kwargs :: Identifier │ │ │ tagger_kwargs :: Identifier │ │ │ [call] │ │ │ merge :: Identifier │ │ │ tagger_kwargs :: Identifier │ │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ │ │ [=] │ │ │ tags :: Identifier │ │ │ [call] │ │ │ get_tags :: Identifier │ │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ cost_tracker :: Identifier │ │ │ [...] │ │ │ tagger_kwargs_ :: Identifier │ │ │ tagger :: Identifier │ │ │ rephrased_questions :: Identifier │ │ │ [=] │ │ │ filter_kwargs_ :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ │ │ api_kwargs :: Identifier │ │ │ filter_kwargs :: Identifier │ │ │ [call] │ │ │ merge :: Identifier │ │ │ filter_kwargs :: Identifier │ │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ │ │ [=] │ │ │ tag_candidates :: Identifier │ │ │ [call] │ │ │ find_tags :: Identifier │ │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ [...] │ │ │ filter_kwargs_ :: Identifier │ │ │ filter :: Identifier │ │ │ index :: Identifier │ │ │ tags :: Identifier │ │ │ [=] │ │ │ filtered_candidates :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ isnothing :: Identifier │ │ │ tag_candidates :: Identifier │ │ │ emb_candidates :: Identifier │ │ │ [call] │ │ │ & :: Identifier │ │ │ emb_candidates :: Identifier │ │ │ tag_candidates :: Identifier │ │ │ [=] │ │ │ reranker_kwargs_ :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ │ │ api_kwargs :: Identifier │ │ │ reranker_kwargs :: Identifier │ │ │ [call] │ │ │ merge :: Identifier │ │ │ reranker_kwargs :: Identifier │ │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ │ │ [=] │ │ │ reranked_candidates :: Identifier │ │ │ [call] │ │ │ rerank :: Identifier │ │ │ [parameters] │ │ │ top_n :: Identifier │ │ │ [kw] │ │ │ verbose :: Identifier │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 1 :: Value │ │ │ cost_tracker :: Identifier │ │ │ [...] │ │ │ reranker_kwargs_ :: Identifier │ │ │ reranker :: Identifier │ │ │ index :: Identifier │ │ │ question :: Identifier │ │ │ filtered_candidates :: Identifier │ │ │ [&&] │ │ │ [call] │ │ │ > :: Identifier │ │ │ verbose :: Identifier │ │ │ 0 :: Value │ │ │ [macrocall] │ │ │ @info :: Identifier │ │ │ :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1141 =#) :: Value │ │ │ [string] │ │ │ "Retrieval done. Identified " :: Value │ │ │ [call] │ │ │ length :: Identifier │ │ │ [call] │ │ │ positions :: Identifier │ │ │ reranked_candidates :: Identifier │ │ │ " chunks, total cost: \$" :: Value │ │ │ [call] │ │ │ round :: Identifier │ │ │ [ref] │ │ │ cost_tracker :: Identifier │ │ │ [kw] │ │ │ digits :: Identifier │ │ │ 2 :: Value │ │ │ "." :: Value │ │ │ [=] │ │ │ result :: Identifier │ │ │ [call] │ │ │ RAGResult :: Identifier │ │ │ [parameters] │ │ │ question :: Identifier │ │ │ [kw] │ │ │ answer :: Identifier │ │ │ nothing :: Identifier │ │ │ rephrased_questions :: Identifier │ │ │ [kw] │ │ │ final_answer :: Identifier │ │ │ nothing :: Identifier │ │ │ [kw] │ │ │ context :: Identifier │ │ │ [call] │ │ │ collect :: Identifier │ │ │ [ref] │ │ │ index :: Identifier │ │ │ reranked_candidates :: Identifier │ │ │ [inert] │ │ │ chunks :: Identifier │ │ │ [kw] │ │ │ sorted :: Identifier │ │ │ true :: Value │ │ │ [kw] │ │ │ sources :: Identifier │ │ │ [call] │ │ │ collect :: Identifier │ │ │ [ref] │ │ │ index :: Identifier │ │ │ reranked_candidates :: Identifier │ │ │ [inert] │ │ │ sources :: Identifier │ │ │ [kw] │ │ │ sorted :: Identifier │ │ │ true :: Value │ │ │ emb_candidates :: Identifier │ │ │ tag_candidates :: Identifier │ │ │ filtered_candidates :: Identifier │ │ │ reranked_candidates :: Identifier │ │ │ [return] │ │ │ result :: Identifier │ │ │ │ │ st1 = │ │ SyntaxTree with attributes mod,kind,var_id,toplevel_pure,scope_type,macro_source,name_val,syntax_flags,meta,scope_layer,value,jl_source,is_toplevel_thunk,source │ │ [block] │ │ │ [=] │ │ │ val :: Identifier │ scope_layer=3 │ │ [function] │ │ │ [call] │ │ │ retrieve :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ [kw] │ │ │ [::] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ Integer :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ [kw] │ │ │ [::] │ │ │ top_k :: Identifier │ scope_layer=1 │ │ Integer :: Identifier │ scope_layer=1 │ │ 100 :: Value │ macro_source=485 │ │ [kw] │ │ │ [::] │ │ │ top_n :: Identifier │ scope_layer=1 │ │ Integer :: Identifier │ scope_layer=1 │ │ 5 :: Value │ macro_source=485 │ │ [kw] │ │ │ [::] │ │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ [::] │ │ │ rephraser :: Identifier │ scope_layer=1 │ │ AbstractRephraser :: Identifier │ scope_layer=1 │ │ [.] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ rephraser :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ rephraser_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ [::] │ │ │ embedder :: Identifier │ scope_layer=1 │ │ AbstractEmbedder :: Identifier │ scope_layer=1 │ │ [.] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ embedder :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ embedder_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ [::] │ │ │ processor :: Identifier │ scope_layer=1 │ │ AbstractProcessor :: Identifier │ scope_layer=1 │ │ [.] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ processor :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ processor_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ [::] │ │ │ finder :: Identifier │ scope_layer=1 │ │ AbstractSimilarityFinder :: Identifier │ scope_layer=1 │ │ [.] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ finder :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ finder_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ [::] │ │ │ tagger :: Identifier │ scope_layer=1 │ │ AbstractTagger :: Identifier │ scope_layer=1 │ │ [.] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ tagger :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ tagger_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ [::] │ │ │ filter :: Identifier │ scope_layer=1 │ │ AbstractTagFilter :: Identifier │ scope_layer=1 │ │ [.] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ filter :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ filter_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ [::] │ │ │ reranker :: Identifier │ scope_layer=1 │ │ AbstractReranker :: Identifier │ scope_layer=1 │ │ [.] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ reranker :: Identifier │ │ │ [kw] │ │ │ [::] │ │ │ reranker_kwargs :: Identifier │ scope_layer=1 │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [call] │ │ │ NamedTuple :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ cost_tracker :: Identifier │ scope_layer=1 │ │ [call] │ │ │ [curly] │ │ │ [.] │ │ │ Threads :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ Atomic :: Identifier │ │ │ Float64 :: Identifier │ scope_layer=1 │ │ 0.0 :: Value │ macro_source=485 │ │ [...] │ │ │ kwargs :: Identifier │ scope_layer=1 │ │ [::] │ │ │ retriever :: Identifier │ scope_layer=1 │ │ AbstractRetriever :: Identifier │ scope_layer=1 │ │ [::] │ │ │ index :: Identifier │ scope_layer=1 │ │ AbstractDocumentIndex :: Identifier │ scope_layer=1 │ │ [::] │ │ │ question :: Identifier │ scope_layer=1 │ │ AbstractString :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ rephraser_kwargs_ :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ scope_layer=1 │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ rephraser_kwargs :: Identifier │ scope_layer=1 │ │ [call] │ │ │ merge :: Identifier │ scope_layer=1 │ │ rephraser_kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ [=] │ │ │ rephrased_questions :: Identifier │ scope_layer=1 │ │ [call] │ │ │ rephrase :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ cost_tracker :: Identifier │ scope_layer=1 │ │ [...] │ │ │ rephraser_kwargs_ :: Identifier │ scope_layer=1 │ │ rephraser :: Identifier │ scope_layer=1 │ │ question :: Identifier │ scope_layer=1 │ │ [=] │ │ │ embeddings :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ HasEmbeddings :: Identifier │ scope_layer=1 │ │ index :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ embedder_kwargs_ :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ scope_layer=1 │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ embedder_kwargs :: Identifier │ scope_layer=1 │ │ [call] │ │ │ merge :: Identifier │ scope_layer=1 │ │ embedder_kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ [=] │ │ │ embeddings :: Identifier │ scope_layer=1 │ │ [call] │ │ │ get_embeddings :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ cost_tracker :: Identifier │ scope_layer=1 │ │ [...] │ │ │ embedder_kwargs_ :: Identifier │ scope_layer=1 │ │ embedder :: Identifier │ scope_layer=1 │ │ rephrased_questions :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ embeddings :: Identifier │ scope_layer=1 │ │ [call] │ │ │ hcat :: Identifier │ scope_layer=1 │ │ [...] │ │ │ [comprehension] │ │ │ [generator] │ │ │ [ref] │ │ │ Float32 :: Identifier │ scope_layer=1 │ │ [=] │ │ │ x :: Identifier │ scope_layer=1 │ │ rephrased_questions :: Identifier │ scope_layer=1 │ │ [=] │ │ │ keywords :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ HasKeywords :: Identifier │ scope_layer=1 │ │ index :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ keywords :: Identifier │ scope_layer=1 │ │ [call] │ │ │ get_keywords :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ [...] │ │ │ processor_kwargs :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ return_keywords :: Identifier │ scope_layer=1 │ │ true :: Value │ macro_source=485 │ │ processor :: Identifier │ scope_layer=1 │ │ rephrased_questions :: Identifier │ scope_layer=1 │ │ [&&] │ │ │ [call] │ │ │ >= :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ [||] │ │ │ [call] │ │ │ isa :: Identifier │ scope_layer=1 │ │ keywords :: Identifier │ scope_layer=1 │ │ [curly] │ │ │ AbstractVector :: Identifier │ scope_layer=1 │ │ [<:] │ scope_layer=1 │ │ [curly] │ │ │ AbstractVector :: Identifier │ scope_layer=1 │ │ [<:] │ scope_layer=1 │ │ AbstractString :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [let] │ │ │ [block] │ macro_source=485 │ │ [block] │ │ │ [=] │ │ │ #1131#level :: Identifier │ scope_layer=1 │ │ Warn :: Identifier │ mod,scope_layer=1 │ │ [=] │ │ │ #1132#std_level :: Identifier │ scope_layer=1 │ │ #1131#level :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ >= :: Identifier │ mod,scope_layer=1 │ │ [.] │ │ │ #1132#std_level :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ level :: Identifier │ │ │ [ref] │ macro_source=485 │ │ Base.Threads.Atomic{Int32}(-1000) :: Value │ macro_source=485 │ │ [block] │ │ │ [=] │ │ │ #1133#group :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ retrieval :: Identifier │ │ │ [=] │ │ │ #1134#_module :: Identifier │ scope_layer=1 │ │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ │ [=] │ │ │ #1135#logger :: Identifier │ scope_layer=1 │ │ [call] │ │ │ Base.CoreLogging.current_logger_for_env :: Value │ macro_source=485 │ │ #1132#std_level :: Identifier │ scope_layer=1 │ │ #1133#group :: Identifier │ scope_layer=1 │ │ #1134#_module :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ ! :: Identifier │ mod,scope_layer=1 │ │ [call] │ │ │ === :: Identifier │ mod,scope_layer=1 │ │ #1135#logger :: Identifier │ scope_layer=1 │ │ nothing :: Identifier │ mod,scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ #1136#id :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ PromptingTools_Experimental_RAGTools_7aa4399b :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ │ Base.CoreLogging.shouldlog :: Value │ macro_source=485 │ │ #1135#logger :: Identifier │ scope_layer=1 │ │ #1131#level :: Identifier │ scope_layer=1 │ │ #1134#_module :: Identifier │ scope_layer=1 │ │ #1133#group :: Identifier │ scope_layer=1 │ │ #1136#id :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ #1137#file :: Identifier │ scope_layer=1 │ │ "/home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ │ [if] │ │ │ [call] │ │ │ isa :: Identifier │ mod,scope_layer=1 │ │ #1137#file :: Identifier │ scope_layer=1 │ │ String :: Identifier │ mod,scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ #1137#file :: Identifier │ scope_layer=1 │ │ [call] │ │ │ [.] │ │ │ Base :: Identifier │ mod,scope_layer=1 │ │ [inert] │ │ │ fixup_stdlib_path :: Identifier │ │ │ #1137#file :: Identifier │ scope_layer=1 │ │ [=] │ │ │ #1138#line :: Identifier │ scope_layer=1 │ │ 1104 :: Value │ macro_source=485 │ │ [local] │ │ │ #1139#msg :: Identifier │ scope_layer=1 │ │ #1140#kwargs :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [block] │ │ │ [try] │ │ │ [block] │ │ │ [=] │ │ │ #1139#msg :: Identifier │ scope_layer=1 │ │ [string] │ │ │ "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " :: Value │ macro_source=485 │ │ [call] │ │ │ typeof :: Identifier │ scope_layer=1 │ │ processor :: Identifier │ scope_layer=1 │ │ ")." :: Value │ macro_source=485 │ │ [=] │ │ │ #1140#kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ macro_source=485 │ │ [parameters] │ macro_source=485 │ │ true :: Value │ macro_source=485 │ │ #1153#err :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [call] │ │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ │ Base.CoreLogging.logging_error :: Value │ macro_source=485 │ │ #1135#logger :: Identifier │ scope_layer=1 │ │ #1131#level :: Identifier │ scope_layer=1 │ │ #1134#_module :: Identifier │ scope_layer=1 │ │ #1133#group :: Identifier │ scope_layer=1 │ │ #1136#id :: Identifier │ scope_layer=1 │ │ #1137#file :: Identifier │ scope_layer=1 │ │ #1138#line :: Identifier │ scope_layer=1 │ │ #1153#err :: Identifier │ scope_layer=1 │ │ true :: Value │ macro_source=485 │ │ false :: Value │ macro_source=485 │ │ [block] │ │ │ [if] │ │ │ [isdefined] │ │ │ #1139#msg :: Identifier │ scope_layer=1 │ │ nothing :: Value │ macro_source=485 │ │ [call] │ │ │ throw :: Identifier │ mod,scope_layer=1 │ │ [call] │ │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ │ [if] │ │ │ [isdefined] │ │ │ #1140#kwargs :: Identifier │ scope_layer=1 │ │ nothing :: Value │ macro_source=485 │ │ [call] │ │ │ throw :: Identifier │ mod,scope_layer=1 │ │ [call] │ │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ │ [call] │ │ │ Base.CoreLogging.handle_message_nothrow :: Value │ macro_source=485 │ │ [parameters] │ │ │ [...] │ │ │ #1140#kwargs :: Identifier │ scope_layer=1 │ │ #1135#logger :: Identifier │ scope_layer=1 │ │ #1131#level :: Identifier │ scope_layer=1 │ │ #1139#msg :: Identifier │ scope_layer=1 │ │ #1134#_module :: Identifier │ scope_layer=1 │ │ #1133#group :: Identifier │ scope_layer=1 │ │ #1136#id :: Identifier │ scope_layer=1 │ │ #1137#file :: Identifier │ scope_layer=1 │ │ #1138#line :: Identifier │ scope_layer=1 │ │ nothing :: Identifier │ mod,scope_layer=1 │ │ keywords :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [comprehension] │ │ │ [generator] │ │ │ [ref] │ │ │ String :: Identifier │ scope_layer=1 │ │ [=] │ │ │ x :: Identifier │ scope_layer=1 │ │ rephrased_questions :: Identifier │ scope_layer=1 │ │ [=] │ │ │ finder_kwargs_ :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ scope_layer=1 │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ finder_kwargs :: Identifier │ scope_layer=1 │ │ [call] │ │ │ merge :: Identifier │ scope_layer=1 │ │ finder_kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ [=] │ │ │ emb_candidates :: Identifier │ scope_layer=1 │ │ [call] │ │ │ find_closest :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ top_k :: Identifier │ scope_layer=1 │ │ [...] │ │ │ finder_kwargs_ :: Identifier │ scope_layer=1 │ │ finder :: Identifier │ scope_layer=1 │ │ index :: Identifier │ scope_layer=1 │ │ embeddings :: Identifier │ scope_layer=1 │ │ keywords :: Identifier │ scope_layer=1 │ │ [=] │ │ │ tagger_kwargs_ :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ scope_layer=1 │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ tagger_kwargs :: Identifier │ scope_layer=1 │ │ [call] │ │ │ merge :: Identifier │ scope_layer=1 │ │ tagger_kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ [=] │ │ │ tags :: Identifier │ scope_layer=1 │ │ [call] │ │ │ get_tags :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ cost_tracker :: Identifier │ scope_layer=1 │ │ [...] │ │ │ tagger_kwargs_ :: Identifier │ scope_layer=1 │ │ tagger :: Identifier │ scope_layer=1 │ │ rephrased_questions :: Identifier │ scope_layer=1 │ │ [=] │ │ │ filter_kwargs_ :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ scope_layer=1 │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ filter_kwargs :: Identifier │ scope_layer=1 │ │ [call] │ │ │ merge :: Identifier │ scope_layer=1 │ │ filter_kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ [=] │ │ │ tag_candidates :: Identifier │ scope_layer=1 │ │ [call] │ │ │ find_tags :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ [kw] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ [...] │ │ │ filter_kwargs_ :: Identifier │ scope_layer=1 │ │ filter :: Identifier │ scope_layer=1 │ │ index :: Identifier │ scope_layer=1 │ │ tags :: Identifier │ scope_layer=1 │ │ [=] │ │ │ filtered_candidates :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ isnothing :: Identifier │ scope_layer=1 │ │ tag_candidates :: Identifier │ scope_layer=1 │ │ emb_candidates :: Identifier │ scope_layer=1 │ │ [call] │ │ │ & :: Identifier │ scope_layer=1 │ │ emb_candidates :: Identifier │ scope_layer=1 │ │ tag_candidates :: Identifier │ scope_layer=1 │ │ [=] │ │ │ reranker_kwargs_ :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ isempty :: Identifier │ scope_layer=1 │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ reranker_kwargs :: Identifier │ scope_layer=1 │ │ [call] │ │ │ merge :: Identifier │ scope_layer=1 │ │ reranker_kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ │ │ [parameters] │ │ │ api_kwargs :: Identifier │ scope_layer=1 │ │ [=] │ │ │ reranked_candidates :: Identifier │ scope_layer=1 │ │ [call] │ │ │ rerank :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ top_n :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ verbose :: Identifier │ scope_layer=1 │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 1 :: Value │ macro_source=485 │ │ cost_tracker :: Identifier │ scope_layer=1 │ │ [...] │ │ │ reranker_kwargs_ :: Identifier │ scope_layer=1 │ │ reranker :: Identifier │ scope_layer=1 │ │ index :: Identifier │ scope_layer=1 │ │ question :: Identifier │ scope_layer=1 │ │ filtered_candidates :: Identifier │ scope_layer=1 │ │ [&&] │ │ │ [call] │ │ │ > :: Identifier │ scope_layer=1 │ │ verbose :: Identifier │ scope_layer=1 │ │ 0 :: Value │ macro_source=485 │ │ [block] │ │ │ [let] │ │ │ [block] │ macro_source=485 │ │ [block] │ │ │ [=] │ │ │ #1154#level :: Identifier │ scope_layer=1 │ │ Info :: Identifier │ mod,scope_layer=1 │ │ [=] │ │ │ #1155#std_level :: Identifier │ scope_layer=1 │ │ #1154#level :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ >= :: Identifier │ mod,scope_layer=1 │ │ [.] │ │ │ #1155#std_level :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ level :: Identifier │ │ │ [ref] │ macro_source=485 │ │ Base.Threads.Atomic{Int32}(-1000) :: Value │ macro_source=485 │ │ [block] │ │ │ [=] │ │ │ #1156#group :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ retrieval :: Identifier │ │ │ [=] │ │ │ #1157#_module :: Identifier │ scope_layer=1 │ │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ │ [=] │ │ │ #1158#logger :: Identifier │ scope_layer=1 │ │ [call] │ │ │ Base.CoreLogging.current_logger_for_env :: Value │ macro_source=485 │ │ #1155#std_level :: Identifier │ scope_layer=1 │ │ #1156#group :: Identifier │ scope_layer=1 │ │ #1157#_module :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [call] │ │ │ ! :: Identifier │ mod,scope_layer=1 │ │ [call] │ │ │ === :: Identifier │ mod,scope_layer=1 │ │ #1158#logger :: Identifier │ scope_layer=1 │ │ nothing :: Identifier │ mod,scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ #1159#id :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ PromptingTools_Experimental_RAGTools_30253e51 :: Identifier │ │ │ [if] │ │ │ [call] │ │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ │ Base.CoreLogging.shouldlog :: Value │ macro_source=485 │ │ #1158#logger :: Identifier │ scope_layer=1 │ │ #1154#level :: Identifier │ scope_layer=1 │ │ #1157#_module :: Identifier │ scope_layer=1 │ │ #1156#group :: Identifier │ scope_layer=1 │ │ #1159#id :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ #1160#file :: Identifier │ scope_layer=1 │ │ "/home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ │ [if] │ │ │ [call] │ │ │ isa :: Identifier │ mod,scope_layer=1 │ │ #1160#file :: Identifier │ scope_layer=1 │ │ String :: Identifier │ mod,scope_layer=1 │ │ [block] │ │ │ [=] │ │ │ #1160#file :: Identifier │ scope_layer=1 │ │ [call] │ │ │ [.] │ │ │ Base :: Identifier │ mod,scope_layer=1 │ │ [inert] │ │ │ fixup_stdlib_path :: Identifier │ │ │ #1160#file :: Identifier │ scope_layer=1 │ │ [=] │ │ │ #1161#line :: Identifier │ scope_layer=1 │ │ 1141 :: Value │ macro_source=485 │ │ [local] │ │ │ #1162#msg :: Identifier │ scope_layer=1 │ │ #1163#kwargs :: Identifier │ scope_layer=1 │ │ [if] │ │ │ [block] │ │ │ [try] │ │ │ [block] │ │ │ [=] │ │ │ #1162#msg :: Identifier │ scope_layer=1 │ │ [string] │ │ │ "Retrieval done. Identified " :: Value │ macro_source=485 │ │ [call] │ │ │ length :: Identifier │ scope_layer=1 │ │ [call] │ │ │ positions :: Identifier │ scope_layer=1 │ │ reranked_candidates :: Identifier │ scope_layer=1 │ │ " chunks, total cost: \$" :: Value │ macro_source=485 │ │ [call] │ │ │ round :: Identifier │ scope_layer=1 │ │ [ref] │ │ │ cost_tracker :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ digits :: Identifier │ scope_layer=1 │ │ 2 :: Value │ macro_source=485 │ │ "." :: Value │ macro_source=485 │ │ [=] │ │ │ #1163#kwargs :: Identifier │ scope_layer=1 │ │ [tuple] │ macro_source=485 │ │ [parameters] │ macro_source=485 │ │ true :: Value │ macro_source=485 │ │ #1176#err :: Identifier │ scope_layer=1 │ │ [block] │ │ │ [call] │ │ │ invokelatest :: Identifier │ mod,scope_layer=1 │ │ Base.CoreLogging.logging_error :: Value │ macro_source=485 │ │ #1158#logger :: Identifier │ scope_layer=1 │ │ #1154#level :: Identifier │ scope_layer=1 │ │ #1157#_module :: Identifier │ scope_layer=1 │ │ #1156#group :: Identifier │ scope_layer=1 │ │ #1159#id :: Identifier │ scope_layer=1 │ │ #1160#file :: Identifier │ scope_layer=1 │ │ #1161#line :: Identifier │ scope_layer=1 │ │ #1176#err :: Identifier │ scope_layer=1 │ │ true :: Value │ macro_source=485 │ │ false :: Value │ macro_source=485 │ │ [block] │ │ │ [if] │ │ │ [isdefined] │ │ │ #1162#msg :: Identifier │ scope_layer=1 │ │ nothing :: Value │ macro_source=485 │ │ [call] │ │ │ throw :: Identifier │ mod,scope_layer=1 │ │ [call] │ │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ │ [if] │ │ │ [isdefined] │ │ │ #1163#kwargs :: Identifier │ scope_layer=1 │ │ nothing :: Value │ macro_source=485 │ │ [call] │ │ │ throw :: Identifier │ mod,scope_layer=1 │ │ [call] │ │ │ AssertionError :: Identifier │ mod,scope_layer=1 │ │ "Assertion to tell the compiler about the definedness of this variable" :: Value │ macro_source=485 │ │ [call] │ │ │ Base.CoreLogging.handle_message_nothrow :: Value │ macro_source=485 │ │ [parameters] │ │ │ [...] │ │ │ #1163#kwargs :: Identifier │ scope_layer=1 │ │ #1158#logger :: Identifier │ scope_layer=1 │ │ #1154#level :: Identifier │ scope_layer=1 │ │ #1162#msg :: Identifier │ scope_layer=1 │ │ #1157#_module :: Identifier │ scope_layer=1 │ │ #1156#group :: Identifier │ scope_layer=1 │ │ #1159#id :: Identifier │ scope_layer=1 │ │ #1160#file :: Identifier │ scope_layer=1 │ │ #1161#line :: Identifier │ scope_layer=1 │ │ nothing :: Identifier │ mod,scope_layer=1 │ │ [=] │ │ │ result :: Identifier │ scope_layer=1 │ │ [call] │ │ │ RAGResult :: Identifier │ scope_layer=1 │ │ [parameters] │ │ │ question :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ answer :: Identifier │ scope_layer=1 │ │ nothing :: Identifier │ scope_layer=1 │ │ rephrased_questions :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ final_answer :: Identifier │ scope_layer=1 │ │ nothing :: Identifier │ scope_layer=1 │ │ [kw] │ │ │ context :: Identifier │ scope_layer=1 │ │ [call] │ │ │ collect :: Identifier │ scope_layer=1 │ │ [ref] │ │ │ index :: Identifier │ scope_layer=1 │ │ reranked_candidates :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ chunks :: Identifier │ │ │ [kw] │ │ │ sorted :: Identifier │ scope_layer=1 │ │ true :: Value │ macro_source=485 │ │ [kw] │ │ │ sources :: Identifier │ scope_layer=1 │ │ [call] │ │ │ collect :: Identifier │ scope_layer=1 │ │ [ref] │ │ │ index :: Identifier │ scope_layer=1 │ │ reranked_candidates :: Identifier │ scope_layer=1 │ │ [inert] │ │ │ sources :: Identifier │ │ │ [kw] │ │ │ sorted :: Identifier │ scope_layer=1 │ │ true :: Value │ macro_source=485 │ │ emb_candidates :: Identifier │ scope_layer=1 │ │ tag_candidates :: Identifier │ scope_layer=1 │ │ filtered_candidates :: Identifier │ scope_layer=1 │ │ reranked_candidates :: Identifier │ scope_layer=1 │ │ [return] │ │ │ result :: Identifier │ scope_layer=1 │ │ [call] │ │ │ Base.Docs.doc! :: Value │ macro_source=485 │ │ PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ │ [call] │ │ │ Base.Docs.Binding :: Value │ macro_source=485 │ │ PromptingTools.Experimental.RAGTools :: Value │ │ │ [inert] │ jl_source=L65 │ │ retrieve :: Identifier │ │ │ [call] │ macro_source=485 │ │ Base.Docs.docstr :: Value │ macro_source=485 │ │ [call] │ macro_source=485 │ │ Core.svec :: Value │ macro_source=485 │ │ " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" :: Value │ macro_source=485 │ │ [call] │ macro_source=485 │ │ Dict{Symbol, Any} :: Value │ macro_source=485 │ │ :path => "/home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl" :: Value │ macro_source=485 │ │ :linenumber => 946 :: Value │ macro_source=485 │ │ :module => PromptingTools.Experimental.RAGTools :: Value │ macro_source=485 │ │ [curly] │ │ │ Union :: Identifier │ scope_layer=1 │ │ [curly] │ │ │ Tuple :: Identifier │ scope_layer=1 │ │ AbstractRetriever :: Identifier │ scope_layer=1 │ │ AbstractDocumentIndex :: Identifier │ scope_layer=1 │ │ AbstractString :: Identifier │ scope_layer=1 │ │ val :: Identifier │ scope_layer=3 │ │ │ │ file = "/home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl" │ │ line = 946 │ └ mod = PromptingTools.Experimental.RAGTools │ ERROR: LoadError: LoweringError: │ #= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1144 =# - invalid syntax: unknown form `kw` or number of arguments 2 │ Expression: │ (kw sorted true) │ Containing expressions: │ (kw context (call collect (ref index reranked_candidates (inert chunks) (kw sorted true)))) │ │ Detailed provenance: │ (kw sorted true) │ └─ (kw sorted true) │ ├─ @ /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1144 │ └─ (macrocall @doc :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946 =#) " retrieve(retriever::AbstractRetriever,\n index::AbstractChunkIndex,\n question::AbstractString;\n verbose::Integer = 1,\n top_k::Integer = 100,\n top_n::Integer = 5,\n api_kwargs::NamedTuple = NamedTuple(),\n rephraser::AbstractRephraser = retriever.rephraser,\n rephraser_kwargs::NamedTuple = NamedTuple(),\n embedder::AbstractEmbedder = retriever.embedder,\n embedder_kwargs::NamedTuple = NamedTuple(),\n processor::AbstractProcessor = retriever.processor,\n processor_kwargs::NamedTuple = NamedTuple(),\n finder::AbstractSimilarityFinder = retriever.finder,\n finder_kwargs::NamedTuple = NamedTuple(),\n tagger::AbstractTagger = retriever.tagger,\n tagger_kwargs::NamedTuple = NamedTuple(),\n filter::AbstractTagFilter = retriever.filter,\n filter_kwargs::NamedTuple = NamedTuple(),\n reranker::AbstractReranker = retriever.reranker,\n reranker_kwargs::NamedTuple = NamedTuple(),\n cost_tracker = Threads.Atomic{Float64}(0.0),\n kwargs...)\n\nRetrieves the most relevant chunks from the index for the given question and returns them in the `RAGResult` object.\n\nThis is the main entry point for the retrieval stage of the RAG pipeline. It is often followed by `generate!` step.\n\nNotes:\n- The default flow is `build_context!` -> `answer!` -> `refine!` -> `postprocess!`.\n\nThe arguments correspond to the steps of the retrieval process (rephrasing, embedding, finding similar docs, tagging, filtering by tags, reranking).\nYou can customize each step by providing a new custom type that dispatches the corresponding function, \n eg, create your own type `struct MyReranker<:AbstractReranker end` and define the custom method for it `rerank(::MyReranker,...) = ...`.\n\nNote: Discover available retrieval sub-types for each step with `subtypes(AbstractRephraser)` and similar for other abstract types.\n\nIf you're using locally-hosted models, you can pass the `api_kwargs` with the `url` field set to the model's URL and make sure to provide corresponding \n `model` kwargs to `rephraser`, `embedder`, and `tagger` to use the custom models (they make AI calls).\n\n# Arguments\n- `retriever`: The retrieval method to use. Default is `SimpleRetriever` but could be `AdvancedRetriever` for more advanced retrieval.\n- `index`: The index that holds the chunks and sources to be retrieved from.\n- `question`: The question to be used for the retrieval.\n- `verbose`: If `>0`, it prints out verbose logging. Default is `1`. If you set it to `2`, it will print out logs for each sub-function.\n- `top_k`: The TOTAL number of closest chunks to return from `find_closest`. Default is `100`.\n If there are multiple rephrased questions, the number of chunks per each item will be `top_k ÷ number_of_rephrased_questions`.\n- `top_n`: The TOTAL number of most relevant chunks to return for the context (from `rerank` step). Default is `5`.\n- `api_kwargs`: Additional keyword arguments to be passed to the API calls (shared by all `ai*` calls).\n- `rephraser`: Transform the question into one or more questions. Default is `retriever.rephraser`.\n- `rephraser_kwargs`: Additional keyword arguments to be passed to the rephraser.\n - `model`: The model to use for rephrasing. Default is `PT.MODEL_CHAT`.\n - `template`: The rephrasing template to use. Default is `:RAGQueryOptimizer` or `:RAGQueryHyDE` (depending on the `rephraser` selected).\n- `embedder`: The embedding method to use. Default is `retriever.embedder`.\n- `embedder_kwargs`: Additional keyword arguments to be passed to the embedder.\n- `processor`: The processor method to use when using Keyword-based index. Default is `retriever.processor`.\n- `processor_kwargs`: Additional keyword arguments to be passed to the processor.\n- `finder`: The similarity search method to use. Default is `retriever.finder`, often `CosineSimilarity`.\n- `finder_kwargs`: Additional keyword arguments to be passed to the similarity finder.\n- `tagger`: The tag generating method to use. Default is `retriever.tagger`.\n- `tagger_kwargs`: Additional keyword arguments to be passed to the tagger. Noteworthy arguments:\n - `tags`: Directly provide the tags to use for filtering (can be String, Regex, or Vector{String}). Useful for `tagger = PassthroughTagger`.\n- `filter`: The tag matching method to use. Default is `retriever.filter`.\n- `filter_kwargs`: Additional keyword arguments to be passed to the tag filter.\n- `reranker`: The reranking method to use. Default is `retriever.reranker`.\n- `reranker_kwargs`: Additional keyword arguments to be passed to the reranker.\n - `model`: The model to use for reranking. Default is `rerank-english-v2.0` if you use `reranker = CohereReranker()`.\n- `cost_tracker`: An atomic counter to track the cost of the retrieval. Default is `Threads.Atomic{Float64}(0.0)`.\n\nSee also: `SimpleRetriever`, `AdvancedRetriever`, `build_index`, `rephrase`, `get_embeddings`, `get_keywords`, `find_closest`, `get_tags`, `find_tags`, `rerank`, `RAGResult`.\n\n# Examples\n\nFind the 5 most relevant chunks from the index for the given question.\n```julia\n# assumes you have an existing index `index`\nretriever = SimpleRetriever()\n\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n\n# or use the default retriever (same as above)\nresult = retrieve(retriever,\n index,\n \"What is the capital of France?\",\n top_n = 5)\n```\n\nApply more advanced retrieval with question rephrasing and reranking (requires `COHERE_API_KEY`).\nWe will obtain top 100 chunks from embeddings (`top_k`) and top 5 chunks from reranking (`top_n`).\n\n```julia\nretriever = AdvancedRetriever()\n\nresult = retrieve(retriever, index, question; top_k=100, top_n=5)\n```\n\nYou can use the `retriever` to customize your retrieval strategy or directly change the strategy types in the `retrieve` kwargs!\n\nExample of using locally-hosted model hosted on `localhost:8080`:\n```julia\nretriever = SimpleRetriever()\nresult = retrieve(retriever, index, question;\n rephraser_kwargs = (; model = \"custom\"),\n embedder_kwargs = (; model = \"custom\"),\n tagger_kwargs = (; model = \"custom\"), api_kwargs = (;\n url = \"http://localhost:8080\"))\n```\n" (function (call retrieve (parameters (kw (:: verbose Integer) 1) (kw (:: top_k Integer) 100) (kw (:: top_n Integer) 5) (kw (:: api_kwargs NamedTuple) (call NamedTuple)) (kw (:: rephraser AbstractRephraser) (. retriever (inert rephraser))) (kw (:: rephraser_kwargs NamedTuple) (call NamedTuple)) (kw (:: embedder AbstractEmbedder) (. retriever (inert embedder))) (kw (:: embedder_kwargs NamedTuple) (call NamedTuple)) (kw (:: processor AbstractProcessor) (. retriever (inert processor))) (kw (:: processor_kwargs NamedTuple) (call NamedTuple)) (kw (:: finder AbstractSimilarityFinder) (. retriever (inert finder))) (kw (:: finder_kwargs NamedTuple) (call NamedTuple)) (kw (:: tagger AbstractTagger) (. retriever (inert tagger))) (kw (:: tagger_kwargs NamedTuple) (call NamedTuple)) (kw (:: filter AbstractTagFilter) (. retriever (inert filter))) (kw (:: filter_kwargs NamedTuple) (call NamedTuple)) (kw (:: reranker AbstractReranker) (. retriever (inert reranker))) (kw (:: reranker_kwargs NamedTuple) (call NamedTuple)) (kw cost_tracker (call (curly (. Threads (inert Atomic)) Float64) 0.0)) (... kwargs)) (:: retriever AbstractRetriever) (:: index AbstractDocumentIndex) (:: question AbstractString)) (block (= rephraser_kwargs_ (if (call isempty api_kwargs) rephraser_kwargs (call merge rephraser_kwargs (tuple (parameters api_kwargs))))) (= rephrased_questions (call rephrase (parameters (kw verbose (call > verbose 1)) cost_tracker (... rephraser_kwargs_)) rephraser question)) (= embeddings (if (call HasEmbeddings index) (block (= embedder_kwargs_ (if (call isempty api_kwargs) embedder_kwargs (call merge embedder_kwargs (tuple (parameters api_kwargs))))) (= embeddings (call get_embeddings (parameters (kw verbose (call > verbose 1)) cost_tracker (... embedder_kwargs_)) embedder rephrased_questions))) (block (= embeddings (call hcat (... (comprehension (generator (ref Float32) (= x rephrased_questions))))))))) (= keywords (if (call HasKeywords index) (block (= keywords (call get_keywords (parameters (kw verbose (call > verbose 1)) (... processor_kwargs) (kw return_keywords true)) processor rephrased_questions)) (&& (call >= verbose 1) (|| (call isa keywords (curly AbstractVector (<: (curly AbstractVector (<: AbstractString))))) (macrocall @warn :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1104 =#) (string "Processed Keywords is not a vector of tokenized queries. Have you used the correct processor? (provided: " (call typeof processor) ").")))) keywords) (block (comprehension (generator (ref String) (= x rephrased_questions)))))) (= finder_kwargs_ (if (call isempty api_kwargs) finder_kwargs (call merge finder_kwargs (tuple (parameters api_kwargs))))) (= emb_candidates (call find_closest (parameters (kw verbose (call > verbose 1)) top_k (... finder_kwargs_)) finder index embeddings keywords)) (= tagger_kwargs_ (if (call isempty api_kwargs) tagger_kwargs (call merge tagger_kwargs (tuple (parameters api_kwargs))))) (= tags (call get_tags (parameters (kw verbose (call > verbose 1)) cost_tracker (... tagger_kwargs_)) tagger rephrased_questions)) (= filter_kwargs_ (if (call isempty api_kwargs) filter_kwargs (call merge filter_kwargs (tuple (parameters api_kwargs))))) (= tag_candidates (call find_tags (parameters (kw verbose (call > verbose 1)) (... filter_kwargs_)) filter index tags)) (= filtered_candidates (if (call isnothing tag_candidates) emb_candidates (call & emb_candidates tag_candidates))) (= reranker_kwargs_ (if (call isempty api_kwargs) reranker_kwargs (call merge reranker_kwargs (tuple (parameters api_kwargs))))) (= reranked_candidates (call rerank (parameters top_n (kw verbose (call > verbose 1)) cost_tracker (... reranker_kwargs_)) reranker index question filtered_candidates)) (&& (call > verbose 0) (macrocall @info :(#= /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:1141 =#) (string "Retrieval done. Identified " (call length (call positions reranked_candidates)) " chunks, total cost: \$" (call round (ref cost_tracker) (kw digits 2)) "."))) (= result (call RAGResult (parameters question (kw answer nothing) rephrased_questions (kw final_answer nothing) (kw context (call collect (ref index reranked_candidates (inert chunks) (kw sorted true)))) (kw sources (call collect (ref index reranked_candidates (inert sources) (kw sorted true)))) emb_candidates tag_candidates filtered_candidates reranked_candidates))) (return result)))) │ └─ @ /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946 │ │ Stacktrace: │ [1] expand_forms_2(ctx::Base.JuliaLowering.MacroExpansionContext{Dict{Symbol, Dict{Int64, Any}}}, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}) │ @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/desugaring.jl:4440 │ [2] core_lowering_hook(code::Any, mod::Module, file::String, line::UInt64, world::UInt64, _warn::Bool) │ @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/hooks.jl:30 │ [3] include(mapexpr::Function, mod::Module, _path::String) │ @ Base Base.jl:326 │ [4] top-level scope │ @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:1 │ [5] macro expansion │ @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:47 [inlined] │ [6] eval(m::Module, e::Any) │ @ Core boot.jl:522 │ [7] _eval(mod::Module, iter::Base.JuliaLowering.LoweringIterator{Dict{Symbol, Dict{Int64, Any}}}; soft_scope::Nothing) │ @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:547 │ [8] eval(mod::Module, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}; macro_world::UInt64, soft_scope::Nothing, opts::@Kwargs{expr_compat_mode::Bool}) │ @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:520 [inlined] │ [9] top-level scope │ @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:1 │ [10] macro expansion │ @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:1 [inlined] │ [11] include(mapexpr::Function, mod::Module, _path::String) │ @ Base Base.jl:326 │ [12] top-level scope │ @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:1 │ [13] macro expansion │ @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:18 [inlined] │ [14] eval(m::Module, e::Any) │ @ Core boot.jl:522 │ [15] _eval(mod::Module, iter::Base.JuliaLowering.LoweringIterator{Dict{Symbol, Dict{Int64, Any}}}; soft_scope::Nothing) │ @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:547 │ [16] eval(mod::Module, ex::Base.JuliaSyntax.SyntaxTree{Dict{Symbol, Dict{Int64, Any}}}; macro_world::UInt64, soft_scope::Nothing, opts::@Kwargs{expr_compat_mode::Bool}) │ @ Base.JuliaLowering /source/usr/share/julia/JuliaLowering/src/eval.jl:520 [inlined] │ [17] top-level scope │ @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:1 │ [18] macro expansion │ @ ~/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:1 [inlined] │ [19] include(mapexpr::Function, mod::Module, _path::String) │ @ Base Base.jl:326 │ [20] top-level scope │ @ ~/.julia/packages/PromptingTools/aTolC/src/PromptingTools.jl:108 │ [21] include(mod::Module, _path::String) │ @ Base Base.jl:325 │ [22] include_package_for_output(pkg::Base.PkgId, input::String, syntax_version::VersionNumber, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing) │ @ Base loading.jl:3296 │ [23] top-level scope │ @ stdin:5 │ [24] eval(m::Module, e::Any) │ @ Core boot.jl:522 │ [25] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) │ @ Base loading.jl:3132 │ [26] include_string(m::Module, txt::String, fname::String) │ @ Base loading.jl:3142 [inlined] │ [27] exec_options(opts::Base.JLOptions) │ @ Base client.jl:353 │ [28] _start() │ @ Base client.jl:596 │ in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/retrieval.jl:946 │ in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/RAGTools/RAGTools.jl:1 │ in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/Experimental/Experimental.jl:1 │ in expression starting at /home/pkgeval/.julia/packages/PromptingTools/aTolC/src/PromptingTools.jl:1 │ in expression starting at stdin:5 └ ERROR: The following 3 packages failed to precompile: PromptingTools → MarkdownPromptingToolsExt Failed to precompile MarkdownPromptingToolsExt [d0be87b4-40ad-5214-b203-6d62ab8a77ad] to "/home/pkgeval/.julia/compiled/v1.14/MarkdownPromptingToolsExt/jl_qcclWh" (ProcessExited(1)). SwarmAgents Failed to precompile SwarmAgents [6fce2c4d-9d52-4112-b2c3-c0b80c6b8436] to "/home/pkgeval/.julia/compiled/v1.14/SwarmAgents/jl_YOoypE" (ProcessExited(1)). PromptingTools Failed to precompile PromptingTools [670122d1-24a8-4d70-bfce-740807c42192] to "/home/pkgeval/.julia/compiled/v1.14/PromptingTools/jl_tOdUkW" (ProcessExited(1)). Loading failed after 140.53s ERROR: LoadError: failed process: Process(`/opt/julia/bin/julia -C native -J/opt/julia/lib/julia/sys.so -g1 --check-bounds=yes --inline=yes --check-bounds=yes --pkgimages=existing -e 'using SwarmAgents'`, ProcessExited(1)) [1] Stacktrace: [1] pipeline_error(proc::Base.Process) @ Base process.jl:612 [inlined] [2] run(::Cmd; wait::Bool) @ Base process.jl:525 [3] run(::Cmd) @ Base process.jl:522 [4] top-level scope @ /PkgEval.jl/scripts/evaluate.jl:197 [5] include(mod::Module, _path::String) @ Base Base.jl:325 [6] exec_options(opts::Base.JLOptions) @ Base client.jl:355 [7] _start() @ Base client.jl:596 in expression starting at /PkgEval.jl/scripts/evaluate.jl:188 PkgEval failed after 899.27s: package fails to precompile