#odbc #database #sql #arrow

arrow-odbc

Read/Write Apache Arrow arrays from/to ODBC data sources

39 releases (16 breaking)

Uses new Rust 2021

new 0.17.1 Jun 28, 2022
0.16.0 Jun 19, 2022
0.9.1 Mar 22, 2022
0.6.4 Dec 29, 2021
0.6.1 Nov 30, 2021

#270 in Database interfaces

Download history 51/week @ 2022-03-12 164/week @ 2022-03-19 174/week @ 2022-03-26 108/week @ 2022-04-02 61/week @ 2022-04-09 48/week @ 2022-04-16 123/week @ 2022-04-23 205/week @ 2022-04-30 354/week @ 2022-05-07 557/week @ 2022-05-14 133/week @ 2022-05-21 305/week @ 2022-05-28 366/week @ 2022-06-04 142/week @ 2022-06-11 241/week @ 2022-06-18 120/week @ 2022-06-25

915 downloads per month
Used in arrow-flightsql-odbc

MIT license

86KB
1.5K SLoC

arrow-odbc

Docs Licence Crates.io

Fill Apache Arrow arrays from ODBC data sources. This crate is build on top of the arrow and odbc-api crate and enables you to read the data of an ODBC data source as sequence of Apache Arrow record batches. This crate can be used to insert the contens of Arrow record batches into a database table, too.

About Arrow

Apache Arrow defines a language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware like CPUs and GPUs. The Arrow memory format also supports zero-copy reads for lightning-fast data access without serialization overhead.

About ODBC

ODBC (Open DataBase Connectivity) is a standard which enables you to access data from a wide variaty of data sources using SQL.

Usage

use arrow_odbc::{odbc_api::Environment, OdbcReader};

const CONNECTION_STRING: &str = "\
    Driver={ODBC Driver 17 for SQL Server};\
    Server=localhost;\
    UID=SA;\
    PWD=My@Test@Password1;\
";

fn main() -> Result<(), anyhow::Error> {

    let odbc_environment = Environment::new()?;
    
    // Connect with database.
    let connection = odbc_environment.connect_with_connection_string(CONNECTION_STRING)?;

    // This SQL statement does not require any arguments.
    let parameters = ();

    // Execute query and create result set
    let cursor = connection
        .execute("SELECT * FROM MyTable", parameters)?
        .expect("SELECT statement must produce a cursor");

    // Each batch shall only consist of maximum 10.000 rows.
    let max_batch_size = 10_000;

    // Read result set as arrow batches. Infer Arrow types automatically using the meta
    // information of `cursor`.
    let arrow_record_batches = OdbcReader::new(cursor, max_batch_size)?;

    for batch in arrow_record_batches {
        // ... process batch ...
    }
    Ok(())
}

Matching of ODBC to Arrow types then querying

ODBC Arrow
Numeric(p <= 38) Decimal
Decimal(p <= 38) Decimal
Integer Int32
SmallInt Int16
Real Float32
Float(p <=24) Float32
Double Float64
Float(p > 24) Float64
Date Date32
LongVarbinary Binary
Timestamp(p = 0) TimestampSecond
Timestamp(p: 1..3) TimestampMilliSecond
Timestamp(p: 4..6) TimestampMicroSecond
Timestamp(p >= 7 ) TimestampNanoSecond
BigInt Int64
TinyInt Int8
Bit Boolean
Varbinary Binary
Binary FixedSizedBinary
All others Utf8

Matching of Arrow to ODBC types then inserting

Arrow ODBC
Utf8 VarChar
Decimal(p, s = 0) VarChar(p + 1)
Decimal(p, s != 0) VarChar(p + 2)
Int8 TinyInt
Int16 SmallInt
Int32 Integer
Int64 BigInt
Float16 Real
Float32 Real
Float64 Double
Timestamp s Timestamp(7)
Timestamp ms Timestamp(7)
Timestamp us Timestamp(7)
Timestamp ns Timestamp(7)
Date32 Date
Date64 Date
Time32 s Time
Time32 ms VarChar(12)
Time64 us VarChar(15)
Time64 ns VarChar(16)
Binary Varbinary
FixedBinary(l) Varbinary(l)
All others Unsupported

The mapping for insertion is not the optimal yet, but before spending a lot of work on improving it I was curious that usecase would pop up for users. So if something does not work, but maybe could provided a better mapping of Arrow to ODBC types, feel free to open an issue. If you do so please give a lot of context of what you are trying to do.

Supported Arrow types

Appart from the afformentioned Arrow types Uint8 is also supported if specifying the Arrow schema directly.

Dependencies

~9–13MB
~267K SLoC