bigtable_rs

A simple Rust library for working with Google Bigtable Data API V2 .

Disclaimer - might be ready for production use

This library might be production ready. Any contribution or help is highly appreciated.

I had the idea to build a client like this and got some input from others see here and porting code from here (thanks to @mvines).

Code for read_rows parsing logic is tested by porting Google's Java client tests code with json test cases as raw input.

Introduction

Current idea is to make this library very light weighted, and you assemble requests based on Google Bigtable V2 protobuf schema and send the requests via tonic gRPC over HTTP/2. So the user have the flexibility of creating any type of Bigtable request and use this client to talk to Bigtable service.

Compiled Bigtable API proto as Rust code is also included in the repo here so users don't need to compile from proto again.

The returned row values from Bigtable is parsed by this library.

Supported interfaces towards Bigtable:

• ReadRows
• SampleRowKeys
• MutateRow
• MutateRows

For other gRPC APIs/methods, one should be able to use the gRCP client directly and assemble any customized request you need to interact with Bigtable service via building the Protobuf messages (already complied as rs files and included here). See this example.

gcp_auth is used, which supports:

• Application Default Credentials
• Connection authenticated via Google service account key json file (by setting GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json environment parameter)
• Default service account by retrieving a token from the gcloud metadata server

You can use the library as follows:

[dependencies]
bigtable_rs = "0.2.16"
tokio = { version = "1.0", features = ["rt-multi-thread"] }
env_logger = "0.11.1"

Documentation is on crate.io. See examples folders for more examples. The following example showing how to do a key range scan

use bigtable_rs::bigtable;
use bigtable_rs::google::bigtable::v2::row_filter::{Chain, Filter};
use bigtable_rs::google::bigtable::v2::row_range::{EndKey, StartKey};
use bigtable_rs::google::bigtable::v2::{ReadRowsRequest, RowFilter, RowRange, RowSet};
use env_logger;
use std::error::Error;
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error>> {
    env_logger::init();

    let project_id = "project-id";
    let instance_name = "instance-1";
    let table_name = "table-1";
    let channel_size = 4;
    let timeout = Duration::from_secs(10);

    let key_start: String = "key1".to_owned();
    let key_end: String = "key4".to_owned();

    // make a bigtable client
    let connection = bigtable::BigTableConnection::new(
        project_id,
        instance_name,
        true,
        channel_size,
        Some(timeout),
    )
        .await?;
    let mut bigtable = connection.client();

    // prepare a ReadRowsRequest
    let request = ReadRowsRequest {
        table_name: bigtable.get_full_table_name(table_name),
        rows_limit: 10,
        rows: Some(RowSet {
            row_keys: vec![], // use this field to put keys for reading specific rows
            row_ranges: vec![RowRange {
                start_key: Some(StartKey::StartKeyClosed(key_start.into_bytes())),
                end_key: Some(EndKey::EndKeyOpen(key_end.into_bytes())),
            }],
        }),
        filter: Some(RowFilter {
            filter: Some(Filter::Chain(Chain {
                filters: vec![
                    RowFilter {
                        filter: Some(Filter::FamilyNameRegexFilter("cf1".to_owned())),
                    },
                    RowFilter {
                        filter: Some(Filter::ColumnQualifierRegexFilter("c1".as_bytes().to_vec())),
                    },
                    RowFilter {
                        filter: Some(Filter::CellsPerColumnLimitFilter(2)),
                    },
                ],
            })),
        }),
        ..ReadRowsRequest::default()
    };

    // calling bigtable API to get results
    let response = bigtable.read_rows(request).await?;

    // simply print results for example usage
    response.into_iter().for_each(|(key, data)| {
        println!("------------\n{}", String::from_utf8(key.clone()).unwrap());
        data.into_iter().for_each(|row_cell| {
            println!(
                "    [{}:{}] \"{}\" @ {}",
                row_cell.family_name,
                String::from_utf8(row_cell.qualifier).unwrap(),
                String::from_utf8(row_cell.value).unwrap(),
                row_cell.timestamp_micros
            )
        })
    });

    Ok(())
}

Play with example code

To start develop or test the example code above, install cbt tool if you haven't, then start a test Bigtable instance locally and insert some test data like this:

# at one terminal, start a bigtable insatnce locally
. start_bigtable_local.sh

# at another terminal, load some data into it
. start_load_table_local.sh

Then run an example with this command:

BIGTABLE_EMULATOR_HOST=localhost:8086 RUST_LOG=bigtable_rs=trace cargo run --bin simple_read

If you see error related to google authentication or gcp project not found, then remember to set environment parameter BIGTABLE_EMULATOR_HOST=localhost:8086 so to let the example's client connect onto the local emulator.

To run it against real Bigtable instance:

GOOGLE_APPLICATION_CREDENTIALS=<path_to_key>/service_account_key.json cargo run --bin simple_read

Or if you want to use your gcloud auth login auth then simply run the client without any special env settings:

cargo run --bin simple_read

See examples folders for more examples.

Development

Clone this repo (and checkout the submodules if rebuilding of the Googleapis proto is needed), and run cargo build.

cargo build

# Or set BUILD_BIGTABLE_RS_GOOGLE_PROTO=true to enable rebuild of google protos
# BUILD_BIGTABLE_RS_GOOGLE_PROTO=true cargo build

See bigtable_rs/src/build.rs for more details.

Running tests:

cargo test -- --nocapture

rustup component add llvm-tools-preview
cargo install grcov
mkdir -p target/coverage/html

CARGO_INCREMENTAL=0 RUSTFLAGS='-Cinstrument-coverage' RUSTDOCFLAGS='-Cinstrument-coverage' cargo test

grcov . --binary-path ./target/debug/deps/ -s . -t html --branch --ignore-not-existing --ignore '../*' --ignore "/*" --ignore "bigtable_rs/src/google/*" --keep-only "bigtable_rs/src/*" -o target/coverage/html