6 stable releases
2.2.3 | May 30, 2021 |
---|---|
2.1.0 | May 30, 2021 |
2.0.0 |
|
1.0.3 |
|
0.0.1 |
|
#1175 in HTTP server
34 downloads per month
49KB
801 lines
Files - CDN Origin Server for Cloud Storage
Purpose
The purpose of this service is to be your CDN Origin Server for Cloud Storage for any JSON client including JavaScript front-ends like React with fetch.
Files supports aws s3, vultr, wasabi, yandex, and digital ocean.
Files is built to be simple and blazing fast with JWT verification, bucket management, and object management.
To use this service you need to have a running broker server.
Features
- Very performant with almost no CPU and memory usage
- Supports bucket and object management
- Supports cloud storage providers: aws s3, vultr, wasabi, yandex, and digital ocean
- Supports being an origin server for Bunny.net, CDN77, Cloudflare, or other CDN providers
- Under 1000 lines of code
- Supports CORS
- Multi-tenant
- Supports JWT authentication
- Supports JWT caching with expiry checking to minimize verify API calls
- Supports SSL - full end-to-end encryption
- JSON API
- Auto-provision and renews SSL cert via LetsEncrypt or use your own SSL cert
- Uses user authorization scoping
- CLI Application - zn
- Built on broker
Use
- create a user on broker with the following scopes - for full permissions
files:full
or granular permissionsfiles:provider, files:get_object, files:put_object, files:delete_object, files:create_bucket, files:list_bucket, or files:delete_bucket
- login to broker and get a JWT
- attach the JWT as an Authorization: Bearer {token} to the following JSON API endpoints
- Add your credentials for your provider(s) - the creds are just for the specific user that creates them
- Valid providers are
aws
,wasabi
,yandex
,vultr
, anddo
- Valid regions are:
Provider | Region |
---|---|
wasabi | wa-us-east-1 |
wasabi | wa-us-east-2 |
wasabi | wa-us-west-1 |
wasabi | wa-eu-central-1 |
aws | us-east-1 |
aws | us-east-2 |
aws | us-west-1 |
aws | us-west-2 |
aws | ca-central-1 |
aws | ap-south-1 |
aws | ap-northeast-1 |
aws | ap-northeast-2 |
aws | ap-northeast-3 |
aws | cn-north-1 |
aws | cn-northwest-1 |
aws | eu-north-1 |
aws | eu-central-1 |
aws | eu-west-1 |
aws | eu-west-2 |
aws | eu-west-3 |
aws | me-south-1 |
aws | sa-east-1 |
do | nyc3 |
do | ams3 |
do | spg1 |
do | fra1 |
vultr | ewr1 |
yandex | ru-central1 |
Add/Update Provider
POST /provider
- authenticated endpoint (Authorization: Bearer {jwt}) example:
{
"name": "aws",
"access_key": "AKIAIOSFODNN7EXAMPLE",
"secret_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}
will return: 200
or 500
or 400
or 401
Get URL Object
GET /files/{region}/{bucket}/{object_path}
- authenticated endpoint (Authorization: Bearer {jwt}) example:
{
"object_path": "/test.pdf",
"region": "wa-us-east-1",
"bucket": "test"
}
will return: 200
or 500
or 400
or 401
200 - will return a file stream with the proper content-type
Get Object
POST /get
- authenticated endpoint (Authorization: Bearer {jwt}) example:
{
"object_path": "/test.pdf",
"region": "wa-us-east-1",
"bucket": "test"
}
will return: 200
or 500
or 400
or 401
200 - will return a file stream with the proper content-type
Put Object
POST /put
- authenticated endpoint (Authorization: Bearer {jwt}) example:
{
"object_path": "/test.pdf",
"region": "wa-us-east-1",
"bucket": "test",
"file": "dGhpc2lzYXN0cmluZw=="
}
file
is a base64 encoded binary file - this is doable with the FileReader class in the browser
will return: the provider's status code (e.g. 200
, 400
)
- note: this action will overwrite a file if the file is the same name
Delete Object
POST /del
- authenticated endpoint (Authorization: Bearer {jwt}) example:
{
"object_path": "/test.pdf",
"region": "wa-us-east-1",
"bucket": "test"
}
will return: the provider's status code (e.g. 204
, 409
)
Create Bucket
POST /create_bucket
- authenticated endpoint (Authorization: Bearer {jwt}) example:
{
"region": "wa-us-east-1",
"bucket": "test"
}
will return: 200
with no body response or the provider's status code (e.g. 409
) and the provider's error
{
"error": "provider's response as an xml string"
}
- note: if the bucket already exists the request will return 200
Delete Bucket
POST /delete_bucket
- authenticated endpoint (Authorization: Bearer {jwt}) example:
{
"region": "wa-us-east-1",
"bucket": "test"
}
will return: the provider's status code (e.g. 204
, 409
)
- note: all objects in the bucket have to be deleted before the bucket can be deleted (based on provider's rules)
List Bucket
POST /list_bucket
- authenticated endpoint (Authorization: Bearer {jwt}) example:
{
"region": "wa-us-east-1",
"bucket": "test",
"prefix": "/",
"delimiter": "/"
}
delimiter
is an optional field
will return: 200
, 400
, 500
, 401
200 - will return an array of objects
{
"objects": [{
"key": "/test.pdf",
"e_tag": "33a64df551425fcc55e4d42a148795d9f25f89d4",
"storage_class": "STANDARD",
"size": 100,
"last_modified": "2021-04-14T22:34:04.000Z"
}]
}
Health Check
GET or HEAD /
- public endpoint
will return: 200
Install
cargo install files
- the
origin
can be passed in as a flag - default*
- the
port
can be passed in as a flag - default9999
- can only be set for unsecure connections - the
secure
flag for https can be true or false - defaultfalse
- the
auto_cert
flag for an autorenewing LetsEncrypt SSL cert can be true or false - requires a resolvable domain - defaulttrue
- the
key_path
flag whenauto_cert
isfalse
to set the SSL key path for your own cert - defaultcerts/private_key.pem
- the
cert_path
flag whenauto_cert
isfalse
to set the SSL cert path for your own cert - defaultcerts/chain.pem
- the
certs
flag is the storage path of LetsEncrypt certs - defaultcerts
- the
db
flag is the path where the embedded database will be saved - defaultdb
- the
domain
flag is the domain name (e.g. api.broker.com) of the domain you want to register with LetsEncrypt - must be fully resolvable - the
broker
flag is the broker domain/ip/port of the broker server - defaulthttp://localhost:8080
- production example:
./files --secure="true" --domain="files.broker.com" --broker="https://broker.broker.com"
Service
There is an example systemctl
service for Ubuntu called files.service
in the code
TechStack
Inspiration
Dependencies
~78MB
~1.5M SLoC