postgresql-cst-parser

Note: This parser is not an official PostgreSQL project but an independent, unofficial tool.

Overview

postgresql-cst-parser is a PostgreSQL-specific Concrete Syntax Tree (CST) parser developed in Pure Rust. This document describes the parser's features, development motivation, usage, and implementation details.

Key Features

• PostgreSQL 17 Support: Supports the latest PostgreSQL 17 syntax.
• Structured CST Output: The generated CST strictly follows the structure defined in PostgreSQL's gram.y file.
• Utilizing cstree: Uses the cstree crate for building syntax trees.
• PL/pgSQL: Currently not supported.

Development Motivation

This project was developed because we needed a library that can be used from Rust, supports all syntax, and (being written in Pure Rust) can be used with wasm-bindgen.

Usage

You can use it as follows:

use postgresql_cst_parser::{parse, syntax_kind::SyntaxKind};

fn main() {
    // Parse SQL query and get the syntax tree
    let sql = "SELECT tbl.a as a, tbl.b from TBL tbl WHERE tbl.a > 0;";
    let root = parse(sql).unwrap();

    // Example 1: Extract all column references from the query
    let column_refs: Vec<String> = root
        .descendants()
        .filter(|node| node.kind() == SyntaxKind::columnref)
        .map(|node| node.text().to_string())
        .collect();

    println!("Column references: {:?}", column_refs); // ["tbl.a", "tbl.b", "tbl.a"]

    // Example 2: Find the WHERE condition
    if let Some(where_clause) = root
        .descendants()
        .find(|node| node.kind() == SyntaxKind::where_clause)
    {
        println!("WHERE condition: {}", where_clause.text());
    }

    // Example 3: Get the selected table name
    if let Some(relation_expr) = root
        .descendants()
        .find(|node| node.kind() == SyntaxKind::relation_expr)
    {
        if let Some(name_node) = relation_expr
            .descendants()
            .find(|node| node.kind() == SyntaxKind::ColId)
        {
            println!("Table name: {}", name_node.text());
        }
    }

    // Example 4: Parse complex SQL and extract specific nodes
    let complex_sql = "WITH data AS (SELECT id, value FROM source WHERE value > 10) 
                       SELECT d.id, d.value, COUNT(*) OVER (PARTITION BY d.id) 
                       FROM data d JOIN other o ON d.id = o.id 
                       ORDER BY d.value DESC LIMIT 10;";

    let complex_root = parse(complex_sql).unwrap();

    // Extract CTEs (Common Table Expressions)
    let ctes: Vec<_> = complex_root
        .descendants()
        .filter(|node| node.kind() == SyntaxKind::common_table_expr)
        .collect();

    // Extract window functions
    let window_funcs: Vec<_> = complex_root
        .descendants()
        .filter(|node| node.kind() == SyntaxKind::over_clause)
        .collect();

    println!("Number of CTEs: {}", ctes.len());
    println!("Number of window functions: {}", window_funcs.len());
}

Example of the generated syntax tree:

SELECT tbl.a as a from TBL tbl;

Root@0..31
  parse_toplevel@0..31
    stmtmulti@0..31
      stmtmulti@0..30
        toplevel_stmt@0..30
          stmt@0..30
            SelectStmt@0..30
              select_no_parens@0..30
                simple_select@0..30
                  SELECT@0..6 "SELECT"
                  Whitespace@6..7 " "
                  opt_target_list@7..17
                    target_list@7..17
                      target_el@7..17
                        a_expr@7..12
                          c_expr@7..12
                            columnref@7..12
                              ColId@7..10
                                IDENT@7..10 "tbl"
                              indirection@10..12
                                indirection_el@10..12
                                  Dot@10..11 "."
                                  attr_name@11..12
                                    ColLabel@11..12
                                      IDENT@11..12 "a"
                        Whitespace@12..13 " "
                        AS@13..15 "as"
                        Whitespace@15..16 " "
                        ColLabel@16..17
                          IDENT@16..17 "a"
                  Whitespace@17..18 " "
                  from_clause@18..30
                    FROM@18..22 "from"
                    Whitespace@22..23 " "
                    from_list@23..30
                      table_ref@23..30
                        relation_expr@23..26
                          qualified_name@23..26
                            ColId@23..26
                              IDENT@23..26 "TBL"
                        Whitespace@26..27 " "
                        opt_alias_clause@27..30
                          alias_clause@27..30
                            ColId@27..30
                              IDENT@27..30 "tbl"
      Semicolon@30..31 ";"

If you'd like to try this parser directly, you can experience it online here.

Implementation

This implementation uses PostgreSQL's scan.l and gram.y with patches from libpg_query applied. scan.l has been further rewritten for Rust, and based on scan.l and gram.y, a syntax parsing table has been created to build the parser.

License

• kwlist.h, parser.c, scan.l, gram.y are under the PostgreSQL License.
• lexer_ported.rs and generated.rs contain code ported from PostgreSQL, so the ported parts are under the PostgreSQL License.
• This project applies patches from libpg_query to scan.l and gram.y, but the patches themselves are not included in this repository.
• Other files are published under the MIT License.