#sql-parser #parser-generator #parser #sql #scanner #tokenizer #tokenize

libsql-sqlite3-parser

SQL parser (as understood by SQLite) (libsql fork)

2 releases

0.11.1 Mar 7, 2024
0.11.0 Jan 4, 2024

#218 in Parser implementations

Download history 268/week @ 2024-01-05 293/week @ 2024-01-12 346/week @ 2024-01-19 500/week @ 2024-01-26 164/week @ 2024-02-02 320/week @ 2024-02-09 458/week @ 2024-02-16 505/week @ 2024-02-23 1102/week @ 2024-03-01 1141/week @ 2024-03-08 1297/week @ 2024-03-15 1036/week @ 2024-03-22 1570/week @ 2024-03-29 988/week @ 2024-04-05

5,034 downloads per month
Used in libsql

Apache-2.0/MIT

450KB
11K SLoC

Rust 5.5K SLoC // 0.1% comments C 4.5K SLoC // 0.2% comments Happy 1.5K SLoC

Build Status Latest Version Docs dependency status

LEMON parser generator modified to generate Rust code.

Lemon source and SQLite3 grammar were last synced as of May 2022.

Unsupported

Unsupported Grammar syntax

  • %token_destructor: Code to execute to destroy token data
  • %default_destructor: Code for the default non-terminal destructor
  • %destructor: Code which executes whenever this symbol is popped from the stack during error processing

https://www.codeproject.com/Articles/1056460/Generating-a-High-Speed-Parser-Part-Lemon https://www.sqlite.org/lemon.html

SQLite

SQLite lexer and SQLite parser have been ported from C to Rust. The parser generates an AST.

Lexer/Parser:

  • Keep track of position (line, column).
  • Streamable (stop at the end of statement).
  • Resumable (restart after the end of statement).

Lexer and parser have been tested with the following scripts:

TODO:

Unsupported by Rust

  • #line directive

API change

  • No ParseAlloc/ParseFree anymore

Features not tested

  • NDEBUG
  • YYNOERRORRECOVERY
  • YYERRORSYMBOL

To be fixed

  • RHS are moved. Maybe it is not a problem if they are always used once. Just add a check in lemon...
  • %extra_argument is not supported.
  • Terminal symbols generated by lemon should be dumped in a specified file.

Raison d'être

  • lemon_rust does the same thing but with an old version of lemon. And it seems not possible to use yystack as a stack because items may be access randomly and the top+1 item can be used.

  • lalrpop would be the perfect alternative but it does not support fallback/streaming (see this issue) and compilation/generation is slow.

Dependencies

~1.3–2MB
~35K SLoC