#odbc #parquet #sql

app odbc2parquet

Query an ODBC data source and store the result in a Parquet file

10 unstable releases (3 breaking)

new 0.4.0 Nov 21, 2020
0.3.0 Nov 11, 2020
0.2.1 Oct 30, 2020
0.1.5 Sep 27, 2020

#141 in Command line utilities

Download history 31/week @ 2020-09-02 35/week @ 2020-09-09 8/week @ 2020-09-16 20/week @ 2020-09-23 13/week @ 2020-09-30 12/week @ 2020-10-14 12/week @ 2020-10-21 42/week @ 2020-10-28 20/week @ 2020-11-04 24/week @ 2020-11-11 16/week @ 2020-11-18

74 downloads per month

MIT license

33KB
715 lines

ODBC to Parquet

A command line tool to query an ODBC data source and write the result into a parquet file.

  • Low memory requirements. Only holds one batch at a time in memory.
  • Fast. Makes efficient use of ODBC bulk reads, to lower IO overhead.
  • Flexible. Query any ODBC data source you have a driver for. MySQL, MS SQL, Excel, ...

Mapping of types

The tool queries the ODBC Data source for type information and maps it to parquet type as such:

ODBC SQL Type Parquet Logical Type
Decimal(p=0..18, s=0) Decimal(p,s)
Numeric(p=0..18, s=0) Decimal(p,s)
Bit Boolean
Double Double
Real Float
Float Float
Tiny Integer Int8
Small Integer Int16
Integer Int32
Big Int Int64
Date Date
Timestamp Timestamp Microseconds
All others Utf8 Byte Array

p is short for precision. s is short for scale. Intervals are inclusive the last element.

Installation

Download binary from GitHub

https://github.com/pacman82/odbc2parquet/releases/latest

Note: Download the 32 Bit version if you want to connect to data sources using 32 Bit drivers and download the 64 Bit version if you want to connect via 64 Bit drivers. It won't work vice versa.

Via Cargo

If you have a rust nightly toolchain installed, you can install this tool via cargo.

cargo +nightly install odbc2parquet

Usage

Query using connection string

odbc2parquet query --connection-string "Driver={ODBC Driver 17 for SQL Server};Server=localhost;UID=SA;PWD=<YourStrong@Passw0rd>;" out.par  "SELECT * FROM Birthdays"

Query using data source name

odbc2parquet query --dsn my_db --password "<YourStrong@Passw0rd>" --user "SA" out.par1 "SELECT * FROM Birthdays"

Use parameters in query

odbc2parquet query --connection-string "Driver={ODBC Driver 17 for SQL Server};Server=localhost;UID=SA;PWD=<YourStrong@Passw0rd>;" out.par  "SELECT * FROM Birthdays WHERE year > ? and year < ?" 1990 2010

Use odbc2parquet --help to see all option.

Dependencies

~24MB
~503K SLoC