Package evaluation to test CommonMark on Julia 1.12.7-DEV.42 (6f510b6086*) started at 2026-06-19T10:20:48.025 ################################################################################ # Set-up # Installing PkgEval dependencies (TestEnv)... Activating project at `~/.julia/environments/v1.12` Set-up completed after 8.71s ################################################################################ # Installation # Installing CommonMark... Resolving package versions... Updating `~/.julia/environments/v1.12/Project.toml` [a80b9123] + CommonMark v1.0.2 Updating `~/.julia/environments/v1.12/Manifest.toml` [a80b9123] + CommonMark v1.0.2 [aea7be01] + PrecompileTools v1.3.4 [21216c6a] + Preferences v1.5.2 [ade2ca70] + Dates v1.11.0 [de0858da] + Printf v1.11.0 [fa267f1f] + TOML v1.0.3 [4ec0a83e] + Unicode v1.11.0 Installation completed after 5.9s ################################################################################ # Precompilation # Precompiling PkgEval dependencies... Precompiling package dependencies... Precompiling packages... 1545.0 ms ✓ pandoc_jll 1 dependency successfully precompiled in 2 seconds. 75 already precompiled. 24 dependencies precompiled but different versions are currently loaded (ArgTools, Base64, Dates, Downloads, JuliaSyntaxHighlighting, LibCURL, LibCURL_jll, LibGit2, LibGit2_jll, LibSSH2_jll, Logging, Markdown, MozillaCACerts_jll, NetworkOptions, OpenSSL_jll, Pkg, Printf, StyledStrings, TOML, Tar, UUIDs, Zlib_jll, nghttp2_jll and p7zip_jll). Restart julia to access the new versions. Otherwise, 31 dependents of these packages may trigger further precompilation to work with the unexpected versions. Precompilation completed after 21.89s ################################################################################ # Testing # Testing CommonMark Status `/tmp/jl_t5EiWg/Project.toml` [a80b9123] CommonMark v1.0.2 [682c06a0] JSON v1.6.1 [d0879d2d] MarkdownAST v0.1.3 [aea7be01] PrecompileTools v1.3.4 [324d217c] ReferenceTests v0.10.6 [f8b46487] TestItemRunner v1.1.5 [ddb6d928] YAML v0.4.16 [c5432543] pandoc_jll v3.6.4+0 [d6f4376e] Markdown v1.11.0 [44cfe95a] Pkg v1.12.1 [8dfed614] Test v1.11.0 Status `/tmp/jl_t5EiWg/Manifest.toml` [1520ce14] AbstractTrees v0.4.5 [3da002f7] ColorTypes v0.12.1 [c3611d14] ColorVectorSpace v0.11.0 [5ae59095] Colors v0.13.1 [a80b9123] CommonMark v1.0.2 [a8cc5b0e] Crayons v4.1.1 [ab62b9b5] DeepDiffs v1.2.0 [b4f34e82] Distances v0.10.12 [5789e2e9] FileIO v1.19.0 ⌅ [53c48c17] FixedPointNumbers v0.8.6 [c817782e] ImageBase v0.1.7 [a09fc81d] ImageCore v0.10.5 [9b13fd28] IndirectArrays v1.0.0 [692b3bcd] JLLWrappers v1.8.0 [682c06a0] JSON v1.6.1 [dbb5928d] MappedArrays v0.4.3 [d0879d2d] MarkdownAST v0.1.3 [e94cdb99] MosaicViews v0.3.4 [6fe1bfb0] OffsetArrays v1.17.0 [5432bcbf] PaddedViews v0.5.12 [69de0a69] Parsers v2.8.6 [aea7be01] PrecompileTools v1.3.4 [21216c6a] Preferences v1.5.2 [189a3867] Reexport v1.2.2 [324d217c] ReferenceTests v0.10.6 [ae029012] Requires v1.3.1 [45858cf5] Sixel v0.1.5 [cae243ae] StackViews v0.1.2 [10745b16] Statistics v1.11.1 [82ae8749] StatsAPI v1.8.0 [69024149] StringEncodings v0.3.7 [ec057cc2] StructUtils v2.8.2 [62fd8b95] TensorCore v0.1.1 [f8b46487] TestItemRunner v1.1.5 [1c621080] TestItems v1.0.0 [c8c2cc18] XTermColors v0.2.1 [ddb6d928] YAML v0.4.16 [aacddb02] JpegTurbo_jll v3.1.5+0 [94ce4f54] Libiconv_jll v1.18.0+0 [b53b4c65] libpng_jll v1.6.58+0 [075b6546] libsixel_jll v1.10.5+0 [c5432543] pandoc_jll v3.6.4+0 [0dad84c5] ArgTools v1.1.2 [56f22d72] Artifacts v1.11.0 [2a0f44e3] Base64 v1.11.0 [ade2ca70] Dates v1.11.0 [f43a241f] Downloads v1.7.0 [7b1f6079] FileWatching v1.11.0 [b77e0a4c] InteractiveUtils v1.11.0 [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0 [b27032c2] LibCURL v0.6.4 [76f85450] LibGit2 v1.11.0 [8f399da3] Libdl v1.11.0 [37e2e46d] LinearAlgebra v1.12.0 [56ddb016] Logging v1.11.0 [d6f4376e] Markdown v1.11.0 [ca575930] NetworkOptions v1.3.0 [44cfe95a] Pkg v1.12.1 [de0858da] Printf v1.11.0 [3fa0cd96] REPL v1.11.0 [9a3f8284] Random v1.11.0 [ea8e919c] SHA v0.7.0 [9e88b42a] Serialization v1.11.0 [6462fe0b] Sockets v1.11.0 [f489334b] StyledStrings v1.11.0 [fa267f1f] TOML v1.0.3 [a4e569a6] Tar v1.10.0 [8dfed614] Test v1.11.0 [cf7118a7] UUIDs v1.11.0 [4ec0a83e] Unicode v1.11.0 [e66e0078] CompilerSupportLibraries_jll v1.3.0+1 [deac9b47] LibCURL_jll v8.15.0+0 [e37daf67] LibGit2_jll v1.9.0+0 [29816b5a] LibSSH2_jll v1.11.3+1 [14a3606d] MozillaCACerts_jll v2025.11.4 [4536629a] OpenBLAS_jll v0.3.29+0 [458c3c95] OpenSSL_jll v3.5.6+0 [83775a58] Zlib_jll v1.3.1+2 [8e850b90] libblastrampoline_jll v5.15.0+0 [8e850ede] nghttp2_jll v1.64.0+1 [3f19e933] p7zip_jll v17.7.0+0 Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. Testing Running tests... Precompiling packages... 3896.9 ms ✓ XTermColors 78941.2 ms ✓ ReferenceTests 2 dependencies successfully precompiled in 83 seconds. 60 already precompiled. 13 dependencies precompiled but different versions are currently loaded (Base64, Dates, InteractiveUtils, JuliaSyntaxHighlighting, Logging, Markdown, PrecompileTools, Preferences, Printf, Serialization, StyledStrings, TOML and Test). Restart julia to access the new versions. Otherwise, 15 dependents of these packages may trigger further precompilation to work with the unexpected versions. ┌ Info: Reference Test for "multiple_refs_same_label.txt" failed: │ - DIFF ------------------------ │ """ │ - A \e[34;4mcool ref\e[39;24m and \e[34;4mhere\e[39;24m. │ + A cool ref and here. │ │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/references/referencelinks/multiple_refs_same_label.txt" └ actual = "/tmp/jl_MIO1Vt/multiple_refs_same_label.txt" referencelinks: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/referencelinks.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference(filename::String, raw_actual::Text{String}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [6] test_all_formats(test_dir::String, base_name::String, ast::CommonMark.Node, reference_dir::String; formats::Vector{Symbol}, env::Nothing, transform::Nothing) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:41 [7] test_all_formats @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:14 [inlined] [8] #test_all_formats##2 @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:9 [inlined] [9] (::Main.var"##277".Utilities.var"#test_all_formats##0#test_all_formats##1"{Main.var"##277".Utilities.var"#test_all_formats##2#test_all_formats##3"{String}})(base_name::String, ast::CommonMark.Node, reference_dir::String) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:8 [10] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/extensions/referencelinks.jl:185 [11] eval(m::Module, e::Any) @ Core ./boot.jl:489 [12] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [13] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [14] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [15] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [16] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [17] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [18] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [19] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [20] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [21] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [22] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [23] top-level scope @ none:6 [24] eval(m::Module, e::Any) @ Core ./boot.jl:489 [25] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [26] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/referencelinks.jl:185 ┌ Info: Reference Test for "internal_pipes.txt" failed: │ - DIFF ------------------------ │ """ │ ┏━━━┯━━━━┯━━━┓ │ - ┃ 1 │ 10 │ \e[36m|\e[39m ┃ │ + ┃ 1 │ 10 │ | ┃ │ ┠───┼────┼───┨ │ - ┃ \e[3m|\e[23m │ \e[32m|\e[39m │ ┃ │ + ┃ | │ | │ ┃ │ ┃ 1 │ 2 │ 3 ┃ │ ┗━━━┷━━━━┷━━━┛ │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/references/tables/internal_pipes.txt" └ actual = "/tmp/jl_TukloR/internal_pipes.txt" tables: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/tables.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference(filename::String, raw_actual::Text{String}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [6] test_all_formats(test_dir::String, base_name::String, ast::CommonMark.Node, reference_dir::String; formats::Vector{Symbol}, env::Nothing, transform::Nothing) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:41 [7] test_all_formats @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:14 [inlined] [8] #test_all_formats##2 @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:9 [inlined] [9] (::Main.var"##277".Utilities.var"#test_all_formats##0#test_all_formats##1"{Main.var"##277".Utilities.var"#test_all_formats##2#test_all_formats##3"{String}})(base_name::String, ast::CommonMark.Node, reference_dir::String) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:8 [10] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/extensions/tables.jl:52 [11] eval(m::Module, e::Any) @ Core ./boot.jl:489 [12] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [13] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [14] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [15] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [16] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [17] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [18] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [19] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [20] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [21] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [22] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [23] top-level scope @ none:6 [24] eval(m::Module, e::Any) @ Core ./boot.jl:489 [25] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [26] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/tables.jl:52 ┌ Info: Reference Test for "inline_raw.txt" failed: │ - DIFF ------------------------ │ """ │ - \e[90mhtml\e[39m\e[90mlatex\e[39m\e[90mtypst\e[39m │ + htmllatextypst │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/references/raw/inline_raw.txt" └ actual = "/tmp/jl_vcEczh/inline_raw.txt" raw_content: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/raw.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference(filename::String, raw_actual::Text{String}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [6] test_all_formats(test_dir::String, base_name::String, ast::CommonMark.Node, reference_dir::String; formats::Vector{Symbol}, env::Nothing, transform::Nothing) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:41 [7] test_all_formats @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:14 [inlined] [8] #test_all_formats##2 @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:9 [inlined] [9] (::Main.var"##277".Utilities.var"#test_all_formats##0#test_all_formats##1"{Main.var"##277".Utilities.var"#test_all_formats##2#test_all_formats##3"{String}})(base_name::String, ast::CommonMark.Node, reference_dir::String) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:8 [10] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/extensions/raw.jl:13 [11] eval(m::Module, e::Any) @ Core ./boot.jl:489 [12] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [13] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [14] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [15] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [16] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [17] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [18] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [19] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [20] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [21] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [22] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [23] top-level scope @ none:6 [24] eval(m::Module, e::Any) @ Core ./boot.jl:489 [25] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [26] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/raw.jl:13 Precompiling packages... 4889.0 ms ✓ pandoc_jll 1 dependency successfully precompiled in 5 seconds. 6 already precompiled. 6 dependencies precompiled but different versions are currently loaded (Dates, JLLWrappers, Preferences, Printf, TOML and Unicode). Restart julia to access the new versions. Otherwise, 1 dependent of these packages may trigger further precompilation to work with the unexpected versions. ┌ Info: Reference Test for "link_only.txt" failed: │ - DIFF ------------------------ │ """ │ - text\e[31m[^1]\e[39m │ + text[^1] │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/references/footnotes/link_only.txt" └ actual = "/tmp/jl_ibvOyS/link_only.txt" footnotes: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/footnotes.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference(filename::String, raw_actual::Text{String}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [6] test_all_formats(test_dir::String, base_name::String, ast::CommonMark.Node, reference_dir::String; formats::Vector{Symbol}, env::Nothing, transform::Nothing) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:41 [7] test_all_formats @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:14 [inlined] [8] #test_all_formats##2 @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:9 [inlined] [9] (::Main.var"##277".Utilities.var"#test_all_formats##0#test_all_formats##1"{Main.var"##277".Utilities.var"#test_all_formats##2#test_all_formats##3"{String}})(base_name::String, ast::CommonMark.Node, reference_dir::String) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:8 [10] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/extensions/footnotes.jl:10 [11] eval(m::Module, e::Any) @ Core ./boot.jl:489 [12] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [13] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [14] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [15] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [16] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [17] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [18] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [19] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [20] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [21] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [22] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [23] top-level scope @ none:6 [24] eval(m::Module, e::Any) @ Core ./boot.jl:489 [25] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [26] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/footnotes.jl:10 ┌ Info: Reference Test for "link.txt" failed: │ - DIFF ------------------------ │ """ │ - \e[34;4mlink\e[39;24m │ + link │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/references/smartlinks/link.txt" └ actual = "/tmp/jl_3nTRA6/link.txt" smart_links: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/smartlinks.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference(filename::String, raw_actual::Text{String}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [6] test_all_formats(test_dir::String, base_name::String, ast::CommonMark.Node, reference_dir::String; formats::Vector{Symbol}, env::Dict{String, Any}, transform::Function) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:41 [7] test_all_formats @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:14 [inlined] [8] #test_all_formats##2 @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:9 [inlined] [9] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/extensions/smartlinks.jl:35 [10] eval(m::Module, e::Any) @ Core ./boot.jl:489 [11] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [12] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [13] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [14] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [15] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [16] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [17] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [18] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [19] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [20] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [21] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [22] top-level scope @ none:6 [23] eval(m::Module, e::Any) @ Core ./boot.jl:489 [24] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [25] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/smartlinks.jl:35 Precompiling packages... 1104.2 ms ✓ StringEncodings 5550.0 ms ✓ YAML 2 dependencies successfully precompiled in 7 seconds. 8 already precompiled. 7 dependencies precompiled but different versions are currently loaded (Base64, Dates, JLLWrappers, Preferences, Printf, TOML and Unicode). Restart julia to access the new versions. Otherwise, 3 dependents of these packages may trigger further precompilation to work with the unexpected versions. ┌ Info: Reference Test for "basic_warning.txt" failed: │ - DIFF ------------------------ │ """ │ - \e[39;1m┌ warning ─────────────────────────────────────────────────────────────────────\e[22m │ - \e[39;1m│\e[22m This is a warning. │ - \e[39;1m└──────────────────────────────────────────────────────────────────────────────\e[22m │ + ┌ warning ───────────────────────────────────────────────────────────────────── │ + │ This is a warning. │ + └────────────────────────────────────────────────────────────────────────────── │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/references/fenceddivs/basic_warning.txt" └ actual = "/tmp/jl_CtT29l/basic_warning.txt" fenceddivs: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/fenceddivs.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference(filename::String, raw_actual::Text{String}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [6] test_all_formats(test_dir::String, base_name::String, ast::CommonMark.Node, reference_dir::String; formats::Vector{Symbol}, env::Nothing, transform::Nothing) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:41 [7] test_all_formats @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:14 [inlined] [8] #test_all_formats##2 @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:9 [inlined] [9] (::Main.var"##277".Utilities.var"#test_all_formats##0#test_all_formats##1"{Main.var"##277".Utilities.var"#test_all_formats##2#test_all_formats##3"{String}})(base_name::String, ast::CommonMark.Node, reference_dir::String) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:8 [10] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/extensions/fenceddivs.jl:75 [11] eval(m::Module, e::Any) @ Core ./boot.jl:489 [12] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [13] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [14] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [15] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [16] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [17] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [18] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [19] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [20] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [21] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [22] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [23] top-level scope @ none:6 [24] eval(m::Module, e::Any) @ Core ./boot.jl:489 [25] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [26] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/fenceddivs.jl:75 ┌ Info: Reference Test for "note_basic.txt" failed: │ - DIFF ------------------------ │ """ │ - \e[34;1m┌ Note ────────────────────────────────────────────────────────────────────────\e[39;22m │ - \e[34;1m│\e[39;22m Basic note content. │ - \e[34;1m└──────────────────────────────────────────────────────────────────────────────\e[39;22m │ + ┌ Note ──────────────────────────────────────────────────────────────────────── │ + │ Basic note content. │ + └────────────────────────────────────────────────────────────────────────────── │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/references/github_alerts/note_basic.txt" └ actual = "/tmp/jl_NqKARS/note_basic.txt" github_alerts: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/github_alerts.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference(filename::String, raw_actual::Text{String}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [6] test_all_formats(test_dir::String, base_name::String, ast::CommonMark.Node, reference_dir::String; formats::Vector{Symbol}, env::Nothing, transform::Nothing) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:41 [7] test_all_formats @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:14 [inlined] [8] #test_all_formats##2 @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:9 [inlined] [9] (::Main.var"##277".Utilities.var"#test_all_formats##0#test_all_formats##1"{Main.var"##277".Utilities.var"#test_all_formats##2#test_all_formats##3"{String}})(base_name::String, ast::CommonMark.Node, reference_dir::String) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:8 [10] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/extensions/github_alerts.jl:95 [11] eval(m::Module, e::Any) @ Core ./boot.jl:489 [12] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [13] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [14] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [15] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [16] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [17] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [18] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [19] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [20] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [21] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [22] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [23] top-level scope @ none:6 [24] eval(m::Module, e::Any) @ Core ./boot.jl:489 [25] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [26] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/github_alerts.jl:95 ┌ Info: Reference Test for "basic.txt" failed: │ - DIFF ------------------------ │ """ │ - ‘some’ \e[33minterpolation\e[39m \e[33minterpolation\e[39m │ + ‘some’ interpolation interpolation │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/references/interpolation/basic.txt" └ actual = "/tmp/jl_2kso9o/basic.txt" interpolation: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/interpolation.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference(filename::String, raw_actual::Text{String}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [6] test_all_formats(test_dir::String, base_name::String, ast::CommonMark.Node, reference_dir::String; formats::Vector{Symbol}, env::Nothing, transform::Nothing) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:41 [7] test_all_formats @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:14 [inlined] [8] #test_all_formats##2 @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:9 [inlined] [9] (::Main.var"##277".Utilities.var"#test_all_formats##0#test_all_formats##1"{Main.var"##277".Utilities.var"#test_all_formats##2#test_all_formats##3"{String}})(base_name::String, ast::CommonMark.Node, reference_dir::String) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:8 [10] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/extensions/interpolation.jl:23 [11] eval(m::Module, e::Any) @ Core ./boot.jl:489 [12] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [13] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [14] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [15] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [16] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [17] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [18] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [19] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [20] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [21] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [22] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [23] top-level scope @ none:6 [24] eval(m::Module, e::Any) @ Core ./boot.jl:489 [25] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [26] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/interpolation.jl:23 ┌ Info: Reference Test for "tight_single.txt" failed: │ - DIFF ------------------------ │ """ │ - \e[1mTerm\e[22m │ + Term │ Definition │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/references/definitionlists/tight_single.txt" └ actual = "/tmp/jl_zdRsNI/tight_single.txt" definitionlists: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/definitionlists.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference(filename::String, raw_actual::Text{String}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [6] test_all_formats(test_dir::String, base_name::String, ast::CommonMark.Node, reference_dir::String; formats::Vector{Symbol}, env::Nothing, transform::Nothing) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:41 [7] test_all_formats @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:14 [inlined] [8] #test_all_formats##2 @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:9 [inlined] [9] (::Main.var"##277".Utilities.var"#test_all_formats##0#test_all_formats##1"{Main.var"##277".Utilities.var"#test_all_formats##2#test_all_formats##3"{String}})(base_name::String, ast::CommonMark.Node, reference_dir::String) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:8 [10] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/extensions/definitionlists.jl:10 [11] eval(m::Module, e::Any) @ Core ./boot.jl:489 [12] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [13] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [14] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [15] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [16] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [17] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [18] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [19] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [20] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [21] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [22] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [23] top-level scope @ none:6 [24] eval(m::Module, e::Any) @ Core ./boot.jl:489 [25] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [26] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/definitionlists.jl:10 ┌ Info: Reference Test for "code.txt" failed: │ - DIFF ------------------------ │ """ │ - \e[36mcode\e[39m │ + code │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/writers/references/term/code.txt" └ actual = "/tmp/jl_MboSXT/code.txt" terminal_writer: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/writers/term.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference @ ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [inlined] [6] _test_single_format(test_dir::String, filename::String, text::String, parser::CommonMark.Parser, format_func::typeof(CommonMark.term)) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:62 [7] (::Main.var"##277".Utilities.var"#test_single_format##0#test_single_format##1"{String, CommonMark.Parser})(filename::String, text::String, format::Function) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:47 [8] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/writers/term.jl:10 [9] eval(m::Module, e::Any) @ Core ./boot.jl:489 [10] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [11] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [12] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [13] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [14] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [15] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [16] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [17] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [18] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [19] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [20] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [21] top-level scope @ none:6 [22] eval(m::Module, e::Any) @ Core ./boot.jl:489 [23] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [24] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/writers/term.jl:10 ┌ Info: Reference Test for "multiple_extensions.txt" failed: │ - DIFF ------------------------ │ """ │ - \e[34;1m#\e[39;22m Integration Tests │ + # Integration Tests │ │ - \e[31;1m┌ Danger ──────────────────────────────────────────────────────────────────────\e[39;22m │ - \e[31;1m│\e[39;22m Warning (\e[31m@cite\e[39m) admonition\e[31m[^1]\e[39m… │ - \e[31;1m│\e[39;22m  │ - \e[31;1m│\e[39;22m \e[35m│\e[39m \e[90mmaths\e[39m │ - \e[31;1m│\e[39;22m  │ - \e[31;1m│\e[39;22m \e[31m┌ [^1] ──────────────────────────────────────────────────────────────────────\e[39m │ - \e[31;1m│\e[39;22m \e[31m│\e[39m 1 │ - \e[31;1m│\e[39;22m \e[31m│\e[39m  │ - \e[31;1m│\e[39;22m \e[31m│\e[39m footnote content │ - \e[31;1m│\e[39;22m \e[31m└────────────────────────────────────────────────────────────────────────────\e[39m │ - \e[31;1m└──────────────────────────────────────────────────────────────────────────────\e[39;22m │ + ┌ Danger ────────────────────────────────────────────────────────────────────── │ + │ Warning (@cite) admonition[^1]… │ + │  │ + │ │ maths │ + │  │ + │ ┌ [^1] ────────────────────────────────────────────────────────────────────── │ + │ │ 1 │ + │ │  │ + │ │ footnote content │ + │ └──────────────────────────────────────────────────────────────────────────── │ + └────────────────────────────────────────────────────────────────────────────── │ │ - \e[33;1m┌ Warning ─────────────────────────────────────────────────────────────────────\e[39;22m │ - \e[33;1m│\e[39;22m “Warning” (\e[31m@cite\e[39m) admonition\e[31m[^2]\e[39m. │ - \e[33;1m│\e[39;22m  │ - \e[33;1m│\e[39;22m ┏━━━━━━━━━┓ │ - \e[33;1m│\e[39;22m ┃ table ┃ │ - \e[33;1m│\e[39;22m ┠─────────┨ │ - \e[33;1m│\e[39;22m ┃ content ┃ │ - \e[33;1m│\e[39;22m ┗━━━━━━━━━┛ │ - \e[33;1m│\e[39;22m \e[31m┌ [^2] ──────────────────────────────────────────────────────────────────────\e[39m │ - \e[33;1m│\e[39;22m \e[31m│\e[39m 2 │ - \e[33;1m│\e[39;22m \e[31m└────────────────────────────────────────────────────────────────────────────\e[39m │ - \e[33;1m└──────────────────────────────────────────────────────────────────────────────\e[39;22m │ + ┌ Warning ───────────────────────────────────────────────────────────────────── │ + │ “Warning” (@cite) admonition[^2]. │ + │  │ + │ ┏━━━━━━━━━┓ │ + │ ┃ table ┃ │ + │ ┠─────────┨ │ + │ ┃ content ┃ │ + │ ┗━━━━━━━━━┛ │ + │ ┌ [^2] ────────────────────────────────────────────────────────────────────── │ + │ │ 2 │ + │ └──────────────────────────────────────────────────────────────────────────── │ + └────────────────────────────────────────────────────────────────────────────── │ │ - \e[36;1m┌ Info ────────────────────────────────────────────────────────────────────────\e[39;22m │ - \e[36;1m│\e[39;22m ‘Tip’ (\e[31m@cite\e[39m) ‘admonition’\e[31m[^3]\e[39m. │ - \e[36;1m│\e[39;22m  │ - \e[36;1m│\e[39;22m \e[31m┌ [^3] ──────────────────────────────────────────────────────────────────────\e[39m │ - \e[36;1m│\e[39;22m \e[31m│\e[39m \e[35mx\e[39m │ - \e[36;1m│\e[39;22m \e[31m└────────────────────────────────────────────────────────────────────────────\e[39m │ - \e[36;1m└──────────────────────────────────────────────────────────────────────────────\e[39;22m │ + ┌ Info ──────────────────────────────────────────────────────────────────────── │ + │ ‘Tip’ (@cite) ‘admonition’[^3]. │ + │  │ + │ ┌ [^3] ────────────────────────────────────────────────────────────────────── │ + │ │ x │ + │ └──────────────────────────────────────────────────────────────────────────── │ + └────────────────────────────────────────────────────────────────────────────── │ │ - \e[36;1m┌ Note ────────────────────────────────────────────────────────────────────────\e[39;22m │ - \e[36;1m│\e[39;22m Note (\e[31m@cite\e[39m) “admonition”\e[31m[^4]\e[39m. │ - \e[36;1m│\e[39;22m  │ - \e[36;1m│\e[39;22m \e[90mlatex\e[39m │ - \e[36;1m│\e[39;22m  │ - \e[36;1m│\e[39;22m \e[31m┌ [^4] ──────────────────────────────────────────────────────────────────────\e[39m │ - \e[36;1m│\e[39;22m \e[31m│\e[39m 4 │ - \e[36;1m│\e[39;22m \e[31m└────────────────────────────────────────────────────────────────────────────\e[39m │ - \e[36;1m└──────────────────────────────────────────────────────────────────────────────\e[39;22m │ + ┌ Note ──────────────────────────────────────────────────────────────────────── │ + │ Note (@cite) “admonition”[^4]. │ + │  │ + │ latex │ + │  │ + │ ┌ [^4] ────────────────────────────────────────────────────────────────────── │ + │ │ 4 │ + │ └──────────────────────────────────────────────────────────────────────────── │ + └────────────────────────────────────────────────────────────────────────────── │ │ - \e[32;1m┌ Tip ─────────────────────────────────────────────────────────────────────────\e[39;22m │ - \e[32;1m│\e[39;22m Tip (\e[31m@cite\e[39m) admonition\e[31m[^5]\e[39m. │ - \e[32;1m│\e[39;22m  │ - \e[32;1m│\e[39;22m \e[31m┌ [^5] ──────────────────────────────────────────────────────────────────────\e[39m │ - \e[32;1m│\e[39;22m \e[31m│\e[39m 5 │ - \e[32;1m│\e[39;22m \e[31m└────────────────────────────────────────────────────────────────────────────\e[39m │ - \e[32;1m└──────────────────────────────────────────────────────────────────────────────\e[39;22m │ + ┌ Tip ───────────────────────────────────────────────────────────────────────── │ + │ Tip (@cite) admonition[^5]. │ + │  │ + │ ┌ [^5] ────────────────────────────────────────────────────────────────────── │ + │ │ 5 │ + │ └──────────────────────────────────────────────────────────────────────────── │ + └────────────────────────────────────────────────────────────────────────────── │ │ - \e[34;1m##\e[39;22m References │ + ## References │ │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/references/integration/multiple_extensions.txt" └ actual = "/tmp/jl_St9b5w/multiple_extensions.txt" multiple_extensions: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/integration.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference(filename::String, raw_actual::Text{String}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [6] test_all_formats(test_dir::String, base_name::String, ast::CommonMark.Node, reference_dir::String; formats::Vector{Symbol}, env::Nothing, transform::Nothing) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:41 [7] test_all_formats @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:14 [inlined] [8] #test_all_formats##2 @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:9 [inlined] [9] (::Main.var"##277".Utilities.var"#test_all_formats##0#test_all_formats##1"{Main.var"##277".Utilities.var"#test_all_formats##2#test_all_formats##3"{String}})(base_name::String, ast::CommonMark.Node, reference_dir::String) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:8 [10] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/integration.jl:25 [11] eval(m::Module, e::Any) @ Core ./boot.jl:489 [12] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [13] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [14] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [15] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [16] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [17] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [18] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [19] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [20] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [21] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [22] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [23] top-level scope @ none:6 [24] eval(m::Module, e::Any) @ Core ./boot.jl:489 [25] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [26] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/integration.jl:25 ┌ Info: Reference Test for "inline_math.txt" failed: │ - DIFF ------------------------ │ """ │ - Some \e[35mmath\e[39m. │ + Some math. │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/references/math/inline_math.txt" └ actual = "/tmp/jl_4P0IF3/inline_math.txt" math: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/math.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference(filename::String, raw_actual::Text{String}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [6] test_all_formats(test_dir::String, base_name::String, ast::CommonMark.Node, reference_dir::String; formats::Vector{Symbol}, env::Nothing, transform::Nothing) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:41 [7] test_all_formats @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:14 [inlined] [8] #test_all_formats##2 @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:9 [inlined] [9] (::Main.var"##277".Utilities.var"#test_all_formats##0#test_all_formats##1"{Main.var"##277".Utilities.var"#test_all_formats##2#test_all_formats##3"{String}})(base_name::String, ast::CommonMark.Node, reference_dir::String) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:8 [10] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/extensions/math.jl:13 [11] eval(m::Module, e::Any) @ Core ./boot.jl:489 [12] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [13] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [14] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [15] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [16] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [17] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [18] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [19] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [20] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [21] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [22] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [23] top-level scope @ none:6 [24] eval(m::Module, e::Any) @ Core ./boot.jl:489 [25] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [26] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/math.jl:13 ┌ Info: Reference Test for "unbracketed_unknown.txt" failed: │ - DIFF ------------------------ │ """ │ - \e[31m@unknown\e[39m │ + @unknown │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/references/citations/unbracketed_unknown.txt" └ actual = "/tmp/jl_Pvj7yi/unbracketed_unknown.txt" citations: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/citations.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference(filename::String, raw_actual::Text{String}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [6] test_all_formats(test_dir::String, base_name::String, ast::CommonMark.Node, reference_dir::String; formats::Vector{Symbol}, env::Dict{String, Any}, transform::Nothing) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:41 [7] test_all_formats @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:14 [inlined] [8] #test_all_formats##2 @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:9 [inlined] [9] test(bib::Vector{Any}, ast::CommonMark.Node, base_name::String) @ Main.var"##326" ~/.julia/packages/CommonMark/adJOy/test/extensions/citations.jl:13 [10] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/extensions/citations.jl:19 [11] eval(m::Module, e::Any) @ Core ./boot.jl:489 [12] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [13] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [14] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [15] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [16] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [17] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [18] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [19] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [20] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [21] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [22] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [23] top-level scope @ none:6 [24] eval(m::Module, e::Any) @ Core ./boot.jl:489 [25] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [26] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/citations.jl:19 ┌ Info: Reference Test for "custom_highlighter.txt" failed: │ - DIFF ------------------------ │ """ │ - \e[36m│\e[39m \e[90mNO TERM HIGHLIGHTING\e[39m │ + │ NO TERM HIGHLIGHTING │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/references/highlights/custom_highlighter.txt" └ actual = "/tmp/jl_VzP08i/custom_highlighter.txt" highlights: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/highlights.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference(filename::String, raw_actual::Text{String}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [6] test_all_formats(test_dir::String, base_name::String, ast::CommonMark.Node, reference_dir::String; formats::Vector{Symbol}, env::Nothing, transform::Function) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:41 [7] test_all_formats @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:14 [inlined] [8] #test_all_formats##2 @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:9 [inlined] [9] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/extensions/highlights.jl:41 [10] eval(m::Module, e::Any) @ Core ./boot.jl:489 [11] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [12] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [13] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [14] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [15] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [16] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [17] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [18] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [19] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [20] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [21] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [22] top-level scope @ none:6 [23] eval(m::Module, e::Any) @ Core ./boot.jl:489 [24] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [25] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/highlights.jl:41 ┌ Info: Reference Test for "warning_basic.txt" failed: │ - DIFF ------------------------ │ """ │ - \e[33;1m┌ Warning ─────────────────────────────────────────────────────────────────────\e[39;22m │ - \e[33;1m│\e[39;22m text │ - \e[33;1m└──────────────────────────────────────────────────────────────────────────────\e[39;22m │ + ┌ Warning ───────────────────────────────────────────────────────────────────── │ + │ text │ + └────────────────────────────────────────────────────────────────────────────── │ """ │ ------------------------------- │ reference = "/home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/references/admonitions/warning_basic.txt" └ actual = "/tmp/jl_NFjgim/warning_basic.txt" admonitions: Error During Test at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/admonitions.jl:2 Got exception outside of a @test LoadError: To update the reference images either run the tests interactively with 'include("test/runtests.jl")', or to force-update all failing reference images set the environment variable `JULIA_REFERENCETESTS_UPDATE` to "true" and re-run the tests via Pkg. Stacktrace: [1] error(s::String) @ Base ./error.jl:44 [2] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing; kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:165 [3] test_reference(reference_file::FileIO.File{FileIO.DataFormat{:TXT}, String}, raw_actual::Text{String}, equiv::Nothing, rendermode::Nothing) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:113 [4] test_reference(filename::String, raw_actual::Text{String}; by::Nothing, render::Nothing, format::Nothing, kw::@Kwargs{}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:110 [5] test_reference(filename::String, raw_actual::Text{String}) @ ReferenceTests ~/.julia/packages/ReferenceTests/QAKab/src/test_reference.jl:105 [6] test_all_formats(test_dir::String, base_name::String, ast::CommonMark.Node, reference_dir::String; formats::Vector{Symbol}, env::Nothing, transform::Nothing) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:41 [7] test_all_formats @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:14 [inlined] [8] #test_all_formats##2 @ ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:9 [inlined] [9] (::Main.var"##277".Utilities.var"#test_all_formats##0#test_all_formats##1"{Main.var"##277".Utilities.var"#test_all_formats##2#test_all_formats##3"{String}})(base_name::String, ast::CommonMark.Node, reference_dir::String) @ Main.var"##277".Utilities ~/.julia/packages/CommonMark/adJOy/test/test_utils.jl:8 [10] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/extensions/admonitions.jl:10 [11] eval(m::Module, e::Any) @ Core ./boot.jl:489 [12] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String) @ Base ./loading.jl:2873 [13] include_string(m::Module, txt::String, fname::String) @ Base ./loading.jl:2883 [14] #run_testitem##6 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:119 [inlined] [15] withpath(f::TestItemRunner.var"#run_testitem##6#run_testitem##7"{String, String, Module}, path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/vendored_code.jl:7 [16] #run_testitem##4 @ ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:118 [inlined] [17] cd(f::TestItemRunner.var"#run_testitem##4#run_testitem##5"{String, String, Module}, dir::String) @ Base.Filesystem ./file.jl:112 [18] run_testitem(filepath::String, use_default_usings::Bool, setups::Vector{Symbol}, package_name::String, original_code::String, line::Int64, column::Int64, test_setup_module_set::TestItemRunner.TestSetupModuleSet, testsetups::Dict{Symbol, Any}) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:117 [19] run_tests(path::String; filter::Nothing, verbose::Bool) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:231 [20] run_tests(path::String) @ TestItemRunner ~/.julia/packages/TestItemRunner/GnoVt/src/TestItemRunner.jl:134 [21] top-level scope @ ~/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 [22] include(mapexpr::Function, mod::Module, _path::String) @ Base ./Base.jl:307 [23] top-level scope @ none:6 [24] eval(m::Module, e::Any) @ Core ./boot.jl:489 [25] exec_options(opts::Base.JLOptions) @ Base ./client.jl:283 [26] _start() @ Base ./client.jl:550 in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/extensions/admonitions.jl:10 Test Summary: | Pass Error Total Time Package | 1985 15 2000 4m51.4s test/extensions/autoidentifiers.jl | 16 16 12.9s test/writers/notebook.jl | 18 18 2m00.6s test/roundtrip.jl | 4 4 14.5s test/extensions/shortcodes.jl | 67 67 8.3s test/readers/stdlib.jl | 53 53 5.5s test/spec_compliance.jl | 655 655 7.3s test/ambiguities.jl | 1 1 0.8s test/extensions/tasklist.jl | 20 20 0.2s test/extensions/referencelinks.jl | 73 1 74 9.0s referencelinks | 73 1 74 9.0s test/readers/json.jl | 45 45 10.0s test/stdlib_compat.jl | 56 56 2.8s test/extensions/tables.jl | 19 1 20 1.5s tables | 19 1 20 1.5s test/normalize_uri.jl | 9 9 0.0s test/extensions/raw.jl | 3 1 4 12.3s raw_content | 3 1 4 12.3s test/writers/json.jl | 46 46 8.6s test/extensions/footnotes.jl | 3 1 4 0.3s footnotes | 3 1 4 0.3s test/spec.jl | 2 2 0.1s test/extensions/smartlinks.jl | 3 1 4 2.4s smart_links | 3 1 4 2.4s test/extensions/strikethrough.jl | 15 15 0.0s test/extensions/mark.jl | 22 22 0.1s test/extensions/subscript.jl | 20 20 0.1s test/extensions/frontmatter.jl | 14 14 19.5s test/extensions/delimiter_interactions.jl | 26 26 0.1s test/unicodes.jl | 10 10 0.0s test/writers/markdown.jl | 72 72 0.1s test/extensions/fenceddivs.jl | 21 1 22 0.6s fenceddivs | 21 1 22 0.6s test/writers/typst.jl | 23 23 0.0s test/extensions/attributes.jl | 94 94 0.5s test/extensions/superscript.jl | 25 25 0.1s test/extensions/github_alerts.jl | 27 1 28 0.7s github_alerts | 27 1 28 0.7s test/parsing.jl | 75 75 8.5s test/extensions/typography.jl | 5 5 0.0s test/extensions/interpolation.jl | 13 1 14 0.2s interpolation | 13 1 14 0.2s test/extensions/definitionlists.jl | 3 1 4 1.1s definitionlists | 3 1 4 1.1s test/builders.jl | 101 101 3.3s test/ast.jl | 9 9 0.1s test/transform.jl | 21 21 4.3s test/sourcepos.jl | 6 6 0.4s test/writers/term.jl | 3 1 4 0.6s terminal_writer | 1 1 0.0s term_wrapping | 3 3 0.5s test/integration.jl | 3 1 4 1.7s multiple_extensions | 3 1 4 1.7s test/extensions/math.jl | 3 1 4 0.0s math | 3 1 4 0.0s test/writers/html.jl | 25 25 0.0s test/readers/markdownast.jl | 82 82 13.4s test/extensions/citations.jl | 3 1 4 0.9s citations | 3 1 4 0.9s test/extensions/highlights.jl | 2 1 3 0.9s highlights | 2 1 3 0.9s test/samples.jl | 26 26 1.1s test/extensions/grid_tables.jl | 117 117 15.9s test/extensions/admonitions.jl | 3 1 4 0.0s admonitions | 3 1 4 0.0s test/writers/latex.jl | 23 23 0.0s ERROR: LoadError: Some tests did not pass: 1985 passed, 0 failed, 15 errored, 0 broken. in expression starting at /home/pkgeval/.julia/packages/CommonMark/adJOy/test/runtests.jl:2 Testing failed after 341.57s ERROR: LoadError: Package CommonMark errored during testing Stacktrace: [1] pkgerror(msg::String) @ Pkg.Types /opt/julia/share/julia/stdlib/v1.12/Pkg/src/Types.jl:68 [2] test(ctx::Pkg.Types.Context, pkgs::Vector{PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool) @ Pkg.Operations /opt/julia/share/julia/stdlib/v1.12/Pkg/src/Operations.jl:2538 [3] test @ /opt/julia/share/julia/stdlib/v1.12/Pkg/src/Operations.jl:2387 [inlined] [4] test(ctx::Pkg.Types.Context, pkgs::Vector{PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{io::IOContext{IO}}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.12/Pkg/src/API.jl:552 [5] test(pkgs::Vector{PackageSpec}; io::IOContext{IO}, kwargs::@Kwargs{julia_args::Cmd}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.12/Pkg/src/API.jl:169 [6] test(pkgs::Vector{String}; kwargs::@Kwargs{julia_args::Cmd}) @ Pkg.API /opt/julia/share/julia/stdlib/v1.12/Pkg/src/API.jl:157 [7] test @ /opt/julia/share/julia/stdlib/v1.12/Pkg/src/API.jl:157 [inlined] [8] #test#81 @ /opt/julia/share/julia/stdlib/v1.12/Pkg/src/API.jl:156 [inlined] [9] top-level scope @ /PkgEval.jl/scripts/evaluate.jl:223 [10] include(mod::Module, _path::String) @ Base ./Base.jl:306 [11] exec_options(opts::Base.JLOptions) @ Base ./client.jl:317 [12] _start() @ Base ./client.jl:550 in expression starting at /PkgEval.jl/scripts/evaluate.jl:214 PkgEval failed after 409.12s: package tests unexpectedly errored