Strange error when evaluating the Jacobian for certain systems

[TorkelE]

Seems to occur in relatively niche situations, hence it hasn't been caught. However, it have recently appeared in the Catalyst test (causing them to fail).

using Catalyst, ModelingToolkit

#Prepares the model.
rs = @reaction_network begin
    p1 * X3^2 + p2, X3 + Y3 --> Z3
end
u0 = [u => 1.0 for u in unknowns(rs)]
ps = [p => 1.0 for p in parameters(rs)]

# First Jacobian evaluation (fine)
osys = complete(convert(ODESystem, rs))
prob = ODEProblem(osys, u0, (0.0, 1.0), ps; jac = true)
J = zeros(length(u0), length(u0))
prob.f.jac(J, prob.u0, prob.p, 0.0)

# Second Jacobian evaluation (fails).
prob = ODEProblem(osys, u0, (0.0, 1.0), ps; jac = true)
J = zeros(length(u0), length(u0))
prob.f.jac(J, prob.u0, prob.p, 0.0) # Error

Gives a

ERROR: MethodError: Cannot `convert` an object of type SymbolicUtils.BasicSymbolic{Float64} to an object of type Float64
The function `convert` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  (::Type{T})(::SymbolicUtils.Symbolic) where T<:Union{AbstractFloat, Integer, Complex{<:AbstractFloat}, Complex{<:Integer}}
   @ Symbolics ~/.julia/packages/Symbolics/bKp7i/src/Symbolics.jl:178
  convert(::Type{T}, ::Unitful.Quantity) where T<:Real
   @ Unitful ~/.julia/packages/Unitful/nwwOk/src/conversion.jl:158
  convert(::Type{T}, ::Unitful.Level) where T<:Real
   @ Unitful ~/.julia/packages/Unitful/nwwOk/src/logarithm.jl:23
  ...

[ChrisRackauckas]

I assume this is hash consing related.

[AayushSabharwal]

Yeah maybe the caching is wrong in a way our tests don't catch

[hersle]

Is the cause likely to be the same as for JuliaSymbolics/Symbolics.jl#1470?