Upgrade syntax to sqlquery here

c42f · c42f · commit 15c73df48708 · 2021-02-05T16:04:01.000+10:00
This leads to less verbosity in the need for delimiters: sql`x` is shorter than @query("x"). Crucially, editor syntax highlighers seem to highlight all $ interpolations within Cmd-style backticks, so we still get syntax highlighting with this in contrast to a normal string macro. Now that we have more control over parsing, upgrade the parsing rules so that SQL single-quoted strings are handled specifically and allow $ to be written without escapes.
diff --git a/README.md b/README.md
@@ -1,35 +1,49 @@
 # SqlStrings
 
-The main thing provided here is the `@sql` macro to allow queries to be
+This package provides the `@sql_cmd` macro to allow SQL query strings to be
 constructed by normal-looking string interpolation but without danger of SQL
 injection attacks.
 
-Note that `@sql` does not parse the text as SQL source as this wouldn't be
-usable across databases and would be a lot of work. Instead, it keeps any
-literal SQL text you write as-is and only treats the Julia-level string
-interpolations specially.
+[![Little Bobby Tables](https://imgs.xkcd.com/comics/exploits_of_a_mom.png)](https://xkcd.com/327)
 
-Use `runquery` to execute queries generated by `@sql`.
+`@sql_cmd` is quite simple — it understands only the basic rules of SQL
+quoting and Julia string interpolation, but does no other parsing of the source
+text. In this sense it is quite similar to `Base.Cmd` - it keeps any literal
+SQL text you write as-is and captures the Julia-level string interpolations
+in a safe way.
 
 ## Simple usage
 
+To use with a given database backend, you'll need a small amount of integration
+code. In the examples below we'll use with LibPQ.jl and a `runquery()` function
+(hopefully integration will be automatic in future).
+
+```julia
+import LibPQ
+
+runquery(conn, sql::SqlStrings.Sql)
+    query, args = SqlStrings.prepare(sql)
+    LibPQ.execute(conn, query, args)
+end
+```
+
 Creating a table and inserting some values
 
 ```julia
 conn = LibPQ.connection(your_connection_string)
 
-runquery(conn, @sql "create table foo (email text, userid integer)")
+runquery(conn, sql`CREATE TABLE foo (email text, userid integer)`)
 
 for (email,id) in [ ("admin@example.com", 1)
                     ("foo@example.com",   2)]
-    runquery(conn, @sql "insert into foo values ($email, $id)")
+    runquery(conn, sql`INSERT INTO foo VALUES ($email, $id)`)
 end
 ```
 
 Thence:
 
-```
-julia> runquery(conn, @sql "select * from foo") |> DataFrame
+```julia
+julia> runquery(conn, sql`SELECT * FROM foo`) |> DataFrame
 2×2 DataFrame
  Row │ email              userid
      │ String?            Int32?
@@ -47,7 +61,7 @@ occasion:
 
 ```julia
 email_and_id = ("bar@example.com", 3)
-runquery(conn, @sql "insert into foo values ($(email_and_id...))")
+runquery(conn, sql`INSERT INTO foo VALUES ($(email_and_id...))`)
 ```
 
 ## Howto: Using the `in` operator with a Julia collection
@@ -56,7 +70,7 @@ There's two ways to do this. First, using `in` and splatting syntax
 
 ```julia
 julia> ids = (1,2)
-       runquery(conn, @sql "select * from foo where userid in ($(ids...))") |> DataFrame
+       runquery(conn, sql`SELECT * FROM foo WHERE userid IN ($(ids...))`) |> DataFrame
        2×2 DataFrame
         Row │ email              userid
             │ String?            Int32?
@@ -67,9 +81,9 @@ julia> ids = (1,2)
 
 Second, using the SQL `any` operator and simply passing a single SQL array parameter:
 
-```
+```julia
 julia> ids = [1,2]
-       runquery(conn, @sql "select * from foo where userid = any($ids)") |> DataFrame
+       runquery(conn, sql`SELECT * FROM foo WHERE userid = any($ids)`) |> DataFrame
        2×2 DataFrame
         Row │ email              userid
             │ String?            Int32?
@@ -81,7 +95,7 @@ julia> ids = [1,2]
 ## Howto: Building up a query from fragments
 
 On occasion you might want to dynamically build up a complicated query from
-fragments of SQL source text. To do this, the result of `@sql` can be
+fragments of SQL source text. To do this, the result of `@sql_cmd` can be
 interpolated into a larger query as follows.
 
 ```julia
@@ -91,16 +105,16 @@ some_condition = true
 
 x = 100
 x = 20
-# Example of an optional clauses - use empty @sql() to disable it.
-and_clause = some_condition ? @sql("and y=$y") : @sql()
+# Example of an optional clauses - use empty sql` to disable it.
+and_clause = some_condition ? sql`AND y=$y` : sql``
 
-# Interpolation of values produces SQL parameters; interpolating @sql
+# Interpolation of values produces SQL parameters; interpolating sql`
 # fragments adds them to the query.
-q = @sql "select * from table where x=$x $and_clause"
+q = sql`SELECT * FROM table WHERE x=$x $and_clause`
 runquery(conn, q)
 ```
 
-A word of warning that constructing SQl logic with Julia-level logic can make
+A word of warning that constructing SQL logic with Julia-level logic can make
 the code quite hard to understand. It can be worth considering writing one
 larger SQL query which does more of the logic on the SQL side.
 
diff --git a/src/SqlStrings.jl b/src/SqlStrings.jl
@@ -1,11 +1,11 @@
 module SqlStrings
 
-export @sql
+export @sql_cmd
 
 """
     Literal(str)
 
-Literal string argument to `@sql`. These are literal query fragments of SQL
+Literal string argument to `@sql_cmd`. These are literal query fragments of SQL
 source text.
 """
 struct Literal
@@ -19,8 +19,9 @@ Literal(val) = Literal(string(val))
 """
     Sql
 
-A query or query-fragment which keeps track of interpolations and will pass
-them as SQL query parameters. Construct this type with the `@sql` macro.
+An SQL query or query-fragment which keeps track of interpolations and will
+pass them as SQL query parameters. Construct this type with the `@sql_cmd`
+macro.
 """
 struct Sql
     args::Vector
@@ -54,12 +55,77 @@ function process_args!(processed, splat::SplatArgs, args...)
     return process_args!(processed, args...)
 end
 
+function parse_interpolations(str, allow_dollars_in_strings)
+    args = []
+    i = 1
+    literal_start = i
+    literal_end = 0
+    in_singlequote = false
+    prev_was_backslash = false
+    while i <= lastindex(str)
+        c = str[i]
+        if !allow_dollars_in_strings && in_singlequote && c == '$'
+            error("""Interpolated arguments should not be quoted, but found quoting in sql`$str`
+                     subexpression starting at `$(str[i:end])`""")
+        end
+        if c == '$' && !in_singlequote
+            if prev_was_backslash
+                literal_end = prevind(str, literal_end, 1)
+                if literal_start <= literal_end
+                    push!(args, Literal(str[literal_start:literal_end]))
+                end
+                literal_start = i
+                i = nextind(str, i)
+            else
+                if literal_start <= literal_end
+                    push!(args, Literal(str[literal_start:literal_end]))
+                end
+                (interpolated_arg, i) = Meta.parse(str, i+1; greedy=false)
+                if Meta.isexpr(interpolated_arg, :...)
+                    push!(args, :(SplatArgs($(esc(interpolated_arg.args[1])))))
+                else
+                    push!(args, esc(interpolated_arg))
+                end
+                literal_start = i
+            end
+        else
+            if c == '\''
+                # We assume standard SQL which uses '' rather than \' for
+                # escaping quotes.
+                in_singlequote = !in_singlequote
+            end
+            literal_end = i
+            i = nextind(str, i)
+        end
+        prev_was_backslash = c == '\\'
+    end
+    if literal_start <= literal_end
+        push!(args, Literal(str[literal_start:literal_end]))
+    end
+    args
+end
+
+"""
+    allow_dollars_in_strings[] = true
+
+Set this parsing option to `false` to disallow dollars inside SQL strings, for
+example disallowing the `'\$s'` in
+
+    sql`select * from foo where s = '\$s'`
+
+When converting code from plain string-based interpolation, it's helpful to
+have a macro-expansion-time sanity check that no manual quoting of interpolated
+arguments remains.
+"""
+const allow_dollars_in_strings = Ref(true)
+
 """
     sql`SOME SQL ... \$var`
     sql`SOME SQL ... \$(var...)`
+    sql`SOME SQL ... 'A \$literal string'`
     sql``
 
-The `@sql` macro is a tool for tracking SQL query strings together with
+The `@sql_cmd` macro is a tool for tracking SQL query strings together with
 their parameters, but without interpolating the parameters into the query
 string directly. Instead, interpolations like `\$x` will result in the value of
 `x` being passed as a query parameter. If you've got a collection of values to
@@ -68,69 +134,53 @@ within the interpolation, for example `insert into foo values(\$(x...))`.
 
 Use this rather than direct string interpolation to prevent SQL injection
 attacks and allow systematic conversion of Julia types into their SQL
-equivalents.
+equivalents via the database layer, rather than via `string()`.
 
 Empty query fragments can be generated with ```sql`` ``` which is useful if you
-must dynamically generate SQL code based on conditionals (however also consider
-embedding any conditionals on the SQL side rather than in the Julia code.)
+must dynamically generate SQL code based on conditionals. However you should
+also consider embedding any conditionals on the SQL side rather than in the
+Julia code.
+
+*Interpolations are ignored* inside standard SQL Strings with a single quote,
+so using `'A \$literal string'` will include `\$literal` rather than the value
+of the variable `literal`. If converting code from using raw strings, you may
+have needed to quote interpolations. In that case you can check your conversion
+by setting `SqlStrings.allow_dollars_in_strings[] = false`.
+
+If you need to include a literal `\$` in the SQL code outside a string, you can
+escape it with `\\\$`.
 """
-macro sql(ex)
-    if ex isa String
-        args = [Literal(ex)]
-    elseif ex isa Expr && ex.head == :string
-        args = []
-        for (i,arg) in enumerate(ex.args)
-            if arg isa String
-                push!(args, Literal(arg))
-            else
-                # Sanity check: arguments should not be quoted
-                prev_quote = i > 1               && ex.args[i-1] isa String && endswith(ex.args[i-1], '\'')
-                next_quote = i < length(ex.args) && ex.args[i+1] isa String && startswith(ex.args[i+1], '\'')
-                if prev_quote || next_quote
-                    error("""Interpolated arguments should not be quoted, but found quoting in subexpression
-                             $(Expr(:string, ex.args[i-1:i+1]...))""")
-                end
-                if Meta.isexpr(arg, :...)
-                    push!(args, :(SplatArgs($(esc(arg.args[1])))))
-                else
-                    push!(args, esc(arg))
-                end
-            end
-        end
-    else
-        error("Unexpected expression passed to @sql: `$ex`")
-    end
+macro sql_cmd(str)
+    args = parse_interpolations(str, allow_dollars_in_strings[])
     quote
         Sql(process_args!([], $(args...)))
     end
 end
 
-macro sql()
-    Sql([])
-end
-
 function Base.:*(x::Sql, y::Sql)
     Sql(vcat(x.args, [Literal(" ")], y.args))
 end
 
-function _prepare(query::Sql)
+default_placeholder_string(i) = "\$$i"
+
+function prepare(sql::Sql, to_placeholder = default_placeholder_string)
     querystr = ""
     arg_values = []
     i = 1
-    for arg in query.args
+    for arg in sql.args
         if arg isa Literal
             querystr *= arg.fragment
         else
-            querystr *= "\$$i"
+            querystr *= to_placeholder(i)
             push!(arg_values, arg)
             i += 1
         end
     end
     querystr, arg_values
 end
 
-function Base.show(io::IO, query::Sql)
-    query, arg_values = _prepare(query)
+function Base.show(io::IO, sql::Sql)
+    query, arg_values = prepare(sql)
     print(io, query)
     if !isempty(arg_values)
         args_str = join(["\$$i = $(repr(val))" for (i,val) in enumerate(arg_values)], "\n  ")
diff --git a/test/runtests.jl b/test/runtests.jl
@@ -2,46 +2,55 @@ using SqlStrings
 using Test
 using UUIDs
 
-querystr(q) = SqlStrings._prepare(q)[1]
-queryargs(q) = SqlStrings._prepare(q)[2]
+querystr(q) = SqlStrings.prepare(q)[1]
+queryargs(q) = SqlStrings.prepare(q)[2]
 
 @testset "SqlStrings.jl" begin
     x = 1
     y = 2
-    q1 = @sql "select a where b=$x and c=$(x+y)"
+    q1 = sql`select a where b=$x and c=$(x+y)`
     @test querystr(q1) == raw"select a where b=$1 and c=$2"
     @test queryargs(q1) == [x, x+y]
 
     # Test concatenation
-    q2 = @sql("select a") * @sql("where b=$x")
+    q2 = sql`select a` * sql`where b=$x`
     @test querystr(q2) == raw"select a where b=$1"
     @test queryargs(q2) == [x,]
 
     # Test that interpolating queries into queries works
-    where_clause = @sql "where b=$x"
-    and_clause = @sql "and c=$y"
-    empty_clause = @sql()
-    q3 = @sql "select a $where_clause $and_clause $empty_clause"
+    where_clause = sql`where b=$x`
+    and_clause = sql`and c=$y`
+    empty_clause = sql``
+    q3 = sql`select a $where_clause $and_clause $empty_clause`
     @test querystr(q3) == raw"select a where b=$1 and c=$2 "
     @test queryargs(q3) == [x, y]
 
     # On occasion, we need to interpolate in a literal string rather than use a
     # parameter. Test that interpolating Literal works for this case.
     column = "x"
-    @test querystr(@sql "select $(SqlStrings.Literal(column)) from a") ==
+    @test querystr(sql`select $(SqlStrings.Literal(column)) from a`) ==
         raw"select x from a"
 
-    # Test that erroneously adding quoting produces an error message
-    @test_throws LoadError @macroexpand @sql "select $y where x = '$x'"
-
     # Test splatting syntax
     z = [1,"hi"]
-    q4 = @sql "insert into foo values($(z...))"
+    q4 = sql`insert into foo values($(z...))`
     @test querystr(q4) == raw"insert into foo values($1,$2)"
     @test queryargs(q4) == z
 
     # Test that Literal turns values into strings
     @test SqlStrings.Literal(:col_name).fragment == "col_name"
     @test SqlStrings.Literal(1).fragment == "1"
+
+    # Test dollars inside SQL strings - the $x here should be a literal.
+    q5 = sql`select $y where x = '$x'`
+    @test querystr(q5) == raw"select $1 where x = '$x'"
+    @test queryargs(q5) == [y,]
+    # Escaping of $
+    q6 = sql`some literal \$a`
+    @test querystr(q6) == raw"some literal $a"
+
+    SqlStrings.allow_dollars_in_strings[] = false
+    @test_throws LoadError @macroexpand sql`select $y where x = '$x'`
+    SqlStrings.allow_dollars_in_strings[] = true
 end
 
