htsget-config

Configuration for htsget-rs and relevant crates.

Overview

This crate is used to configure htsget-rs by using a config file or reading environment variables.

Usage

For running htsget-rs as an application

To configure htsget-rs, a TOML config file can be used. It also supports reading config from environment variables. Any config options set by environment variables override values in the config file. For some of the more deeply nested config options, it may be more ergonomic to use a config file rather than environment variables.

The configuration consists of multiple parts, config for the ticket server, config for the data server, service-info config, and config for the resolvers.

Ticket server config

The ticket server responds to htsget requests by returning a set of URL tickets that the client must fetch and concatenate. To configure the ticket server, set the following options:

TLS is supported by setting the ticket_server_key and ticket_server_cert options. An example of config for the ticket server:

ticket_server_addr = '127.0.0.1:8080'
ticket_server_cors_allow_credentials = false
ticket_server_cors_allow_origins = 'Mirror'
ticket_server_cors_allow_headers = ['Content-Type']
ticket_server_cors_allow_methods = ['GET', 'POST']
ticket_server_cors_max_age = 86400
ticket_server_cors_expose_headers = []

Local data server config

The local data server responds to tickets produced by the ticket server by serving local filesystem data. To configure the data server, set the following options:

TLS is supported by setting the data_server_key and data_server_cert options. An example of config for the data server:

data_server_addr = '127.0.0.1:8081'
data_server_local_path = 'data'
data_server_serve_at = '/data'
data_server_key = 'key.pem'
data_server_cert = 'cert.pem'
data_server_cors_allow_credentials = false
data_server_cors_allow_origins = 'Mirror'
data_server_cors_allow_headers = ['Content-Type']
data_server_cors_allow_methods = ['GET', 'POST']
data_server_cors_max_age = 86400
data_server_cors_expose_headers = []

Sometimes it may be useful to disable the data server as all responses to the ticket server will be handled elsewhere, such as with an AWS S3 data server.

To disable the data server, set the following option:

data_server_enabled = false

Service info config

The service info config controls what is returned when the service-info path is queried.
To configure the service-info, set the following options:

An example of config for the service info:

id = 'id'
name = 'name'
version = '0.1'
organization_name = 'name'
organization_url = 'https://example.com/'
contact_url = 'mailto:nobody@example.com'
documentation_url = 'https://example.com/'
created_at = '2022-01-01T12:00:00Z'
updated_at = '2022-01-01T12:00:00Z'
environment = 'dev'

Resolvers

The resolvers component of htsget-rs is used to map query IDs to the location of the resource. Each query that htsget-rs receives is 'resolved' to a location, which a data server can respond with. A query ID is matched with a regex, and is then mapped with a substitution string that has access to the regex capture groups. Resolvers are configured in an array, where the first matching resolver is resolver used to map the ID.

To create a resolver, add a [[resolvers]] array of tables, and set the following options:

For example, below is a regex option which matches a / between two groups, and inserts an additional data inbetween the groups with the substitution_string.

[[resolvers]]
regex = '(?P<group1>.*?)/(?P<group2>.*)'
substitution_string = '$group1/data/$group2'

For more information about regex options see the regex crate.

Each resolver also maps to a certain storage backend. This storage backend can be used to set query IDs which are served from local storage, from S3-style bucket storage, or from HTTP URLs. To set the storage backend for a resolver, add a [resolvers.storage] table. Some storage backends require feature flags to be set when compiling htsget-rs.

To use LocalStorage, set storage = 'Local'. This will derive the values for the fields below from the data_server config:

To use S3Storage, build htsget-rs with the s3-storage feature enabled, and set storage = 'S3'. This will derive the value for bucket from the regex component of the resolvers:

UrlStorage is another storage backend which can be used to serve data from a remote HTTP URL. When using this storage backend, htsget-rs will fetch data from a url which is set in the config. It will also forward any headers received with the initial query, which is useful for authentication. To use UrlStorage, build htsget-rs with the url-storage feature enabled, and set the following options under [resolvers.storage]:

When using UrlStorage, the following requests will be made to the url.

• GET request to fetch only the headers of the data file (e.g. GET /data.bam, with Range: bytes=0-<end_of_bam_header>).
• GET request to fetch the entire index file (e.g. GET /data.bam.bai).
• HEAD request on the data file to get its length (e.g. HEAD /data.bam).

All headers received in the initial query will be included when making these requests.

For example, a resolvers value of:

[[resolvers]]
regex = '^(example_bucket)/(?P<key>.*)$'
substitution_string = '$key'
storage = 'S3'

Will use "example_bucket" as the S3 bucket if that resolver matches, because this is the first capture group in the regex. Note, to use this feature, at least one capture group must be defined in the regex.

Note, all the values for S3Storage or LocalStorage can be also be set manually by adding a [resolvers.storage] table. For example, to manually set the config for LocalStorage:

[[resolvers]]
regex = '.*'
substitution_string = '$0'

[resolvers.storage]
scheme = 'Http'
authority = '127.0.0.1:8081'
local_path = 'data'
path_prefix = '/data'

or, to manually set the config for S3Storage:

[[resolvers]]
regex = '.*'
substitution_string = '$0'

[resolvers.storage]
bucket = 'bucket'

UrlStorage can only be specified manually.

There are additional examples of config files located under examples/config-files.

Note

By default, when htsget-rs is compiled with the s3-storage feature flag, storage = 'S3' is used when no storage options are specified. Otherwise, storage = 'Local' is used when no storage options are specified. Compilation includes the s3-storage feature flag by default, so in order to have storage = 'Local' as the default, --no-default-features can be passed to cargo.

Allow guard

Additionally, the resolver component has a feature, which allows resolving IDs based on the other fields present in a query. This is useful as allows the resolver to match an ID, if a particular set of query parameters are also present. For example, a resolver can be set to only resolve IDs if the format is also BAM.

This component can be configured by setting the [resolver.allow_guard] table with. The following options are available to restrict which queries are resolved by a resolver:

An example of a fully configured resolver:

[[resolvers]]
regex = '.*'
substitution_string = '$0'

[resolvers.storage]
bucket = 'bucket'

[resolvers.allow_guard]
allow_reference_names = ['chr1']
allow_fields = ['QNAME']
allow_tags = ['RG']
allow_formats = ['BAM']
allow_classes = ['body']
allow_interval_start = 100
allow_interval_end = 1000

In this example, the resolver will only match the query ID if the query is for chr1 with positions between 100 and 1000.

TLS

TLS can be configured for the ticket server, data server, or the url storage client. These options read private keys and certificates from PEM-formatted files. Certificates must be in X.509 format and private keys can be RSA, PKCS8, or SEC1 (EC) encoded. The following options are available:

When used by the ticket and data servers, key and cert enable TLS, and when used with the url storage client, they enable client authentication. The root store is only used by the url storage client. Note, the url storage client always allows TLS, however the default configuration performs no client authentication and uses the native root certificate store.

For example, TLS for the ticket server can be enabled by specifying the key and cert options:

ticket_server_tls.cert = "cert.pem"
ticket_server_tls.key = "key.pem"

Further TLS examples are available under examples/config-files.

Config file location

The htsget-rs binaries (htsget-actix and htsget-lambda) support some command line options. The config file location can be specified by setting the --config option:

cargo run -p htsget-actix -- --config "config.toml"

The config can also be read from an environment variable:

export HTSGET_CONFIG="config.toml"

If no config file is specified, the default configuration is used. Further, the default configuration file can be printed to stdout by passing the --print-default-config flag:

cargo run -p htsget-actix -- --print-default-config

Use the --help flag to see more details on command line options.

Log formatting

The Tracing crate is used extensively by htsget-rs is for logging functionality. The RUST_LOG variable is read to configure the level that trace logs are emitted.

For example, the following indicates trace level for all htsget crates, and info level for all other crates:

export RUST_LOG='info,htsget_lambda=trace,htsget_lambda=trace,htsget_config=trace,htsget_http=trace,htsget_search=trace,htsget_test=trace'

See here for more information on setting this variable.

The style of formatting can be configured by setting the following option:

See here for more information on how these values look.

Configuring htsget-rs with environment variables

All the htsget-rs config options can be set by environment variables, which is convenient for runtimes such as AWS Lambda. The ticket server, data server and service info options are flattened and can be set directly using environment variable. It is not recommended to set the resolvers using environment variables, however it can be done by setting a single environment variable which contains a list of structures, where a key name and value pair is used to set the nested options.

Environment variables will override options set in the config file. Note, arrays are delimited with [ and ] in environment variables, and items are separated by commas.

The following environment variables - corresponding to the TOML config - are available:

In order to use HTSGET_RESOLVERS, the entire resolver config array must be set. The nested array of resolvers structure can be set using name key and value pairs, for example:

export HTSGET_RESOLVERS="[{
    regex=regex,
    substitution_string=substitution_string,
    storage={
        bucket=bucket
    },
    allow_guard={
        allow_reference_names=[chr1],
        allow_fields=[QNAME],
        allow_tags=[RG],
        allow_formats=[BAM],
        allow_classes=[body],
        allow_interval_start=100,
        allow_interval_end=1000
    }  
}]"

Similar to the data_server option, the data server can be disabled by setting the equivalent environment variable:

export HTSGET_DATA_SERVER_ENABLED=false

MinIO

Operating a local object storage like MinIO can be easily achieved by leveraging the endpoint directive as shown below:

[[resolvers]]
regex = ".*"
substitution_string = "$0"

[resolvers.storage]
bucket = 'bucket'
endpoint = "http://127.0.0.1:9000"

This will have htsget-rs behaving like the native AWS CLI, i.e:

mkdir /tmp/test
minio server /tmp/test
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
aws s3 mb --endpoint-url=http://localhost:9000 s3://bucket/
aws s3 cp --recursive --endpoint-url=http://localhost:9000 htsget-rs/data/bam s3://bucket/
cargo run -p htsget-actix -- --config ~/.htsget-rs/config.toml

# On another session/terminal
curl http://localhost:8080/reads/htsnexus_test_NA12878

Please don't run the example above as-is in production systems ;)

As a library

This crate reads config files and environment variables using figment, and accepts command-line arguments using clap. The main function for this is from_config, which is used to obtain the Config struct. The crate also contains the regex_resolver abstraction, which is used for matching a query ID with regex, and changing it by using a substitution string.

Feature flags

This crate has the following features:

• s3-storage: used to enable S3Storage functionality.
• url-storage: used to enable UrlStorage functionality.

License

This project is licensed under the MIT license.