Lib.rs

› Web programming › HTTP server | Authentication | Database implementations
#text-search #full-text-search #nlp #web-services #service #jwt #language

app portal

Full Text Search Web Service

by Bevan Hunt

1 stable release

3.0.1 May 1, 2021
3.0.0 Apr 17, 2021
2.1.2 Apr 16, 2021
1.1.5 Apr 12, 2021
0.0.3 Apr 3, 2021

#292 in Database implementations

22 downloads per month

MIT license

34KB
571 lines

Portal - Full Text Search Web Service

Purpose

The purpose of this service is to be your full text search web service for any JSON client including JavaScript front-ends like React with fetch.

Portal is a competitor to ElasticSearch, MeiliSearch, AWS CloudSearch, AWS ElasticSearch Service, and Algolia.

Portal is built to be simple and blazing fast with JWT verification, indexing, deindexing, search, and word suggest.

Indexing is batch indexing by default.

To use this service you need to have a running sonic server and a broker server.

Features

  • Very performant with almost no CPU and memory usage
  • Supports full-text search and word suggestions in 87 natural languages
  • Under 1000 lines of code
  • Supports CORS
  • Supports JWT authentication
  • Supports JWT caching with expiry checking to minimize verify API calls
  • Multi-tenant
  • Supports SSL - full end-to-end encryption
  • JSON API
  • Auto-provision and renews SSL cert via LetsEncrypt or use your own SSL cert
  • Uses user authorization scoping
  • Built on sonic and broker

Use

  1. create a user on broker with the following scopes - for full permissions portal:full or granular permissions portal:create_bucket, portal:delete_bucket, portal:index, portal:deindex, portal:search, or portal:suggest
  2. login to broker and get a JWT
  3. attach the JWT as an Authorization: Bearer {token} to the following JSON API endpoints
  4. create a bucket

Create Bucket

POST /create_bucket
  • authenticated endpoint (Authorization: Bearer {jwt}) example:
{
    "collection": "coffee",
    "name": "tenant_1"
}

will return: 200 or 500 or 400 or 401

  • note: bucket names must be globally unique

Delete Bucket

POST /delete_bucket
  • authenticated endpoint (Authorization: Bearer {jwt}) example:
{
    "collection": "coffee",
    "name": "tenant_1"
}

will return: 200 or 500 or 400 or 401

Index

POST /index
  • authenticated endpoint (Authorization: Bearer {jwt}) example:
{
    "items": [{
        "collection": "coffee", 
        "bucket": "tenant_1", 
        "id": "49e28aae-88d4-4c19-86d8-51f2c9f11039", 
        "data": {
            "name": "roasted",
            "image": "https://img.com/bucket/123/123.jpg"
        },
        "locale": "eng",
        "indexes": ["name"]
    }]
}

will return: 200 or 500 or 400 or 401

  • note: only buckets the user has created can be used in the index call

  • note: locale is an optional field of an ISO 639-3 locale code - if not defined the locale will be auto-detected

  • Supported Locales:

    • afr => Afrikaans
    • aka => Akan
    • amh => Amharic
    • ara => Arabic
    • azj => Azerbaijani (North)
    • bel => Belarusian
    • ben => Bengali
    • bho => Bhojpuri
    • bul => Bulgarian
    • cat => Catalan, Valencian
    • ceb => Cebuano
    • ces => Czech
    • cmn => Mandarin Chinese
    • dan => Danish
    • deu => German
    • ell => Greek (Modern)
    • eng => English
    • epo => Esperanto
    • est => Estonian
    • fin => Finnish
    • fra => French
    • guj => Gujarati
    • hat => Haitian, Haitian Creole
    • hau => Hausa
    • heb => Hebrew
    • hin => Hindi
    • hrv => Croatian
    • hun => Hungarian
    • ibo => Igbo
    • ilo => Iloko
    • ind => Indonesian
    • ita => Italian
    • jav => Javanese
    • jpn => Japanese
    • kan => Kannada
    • kat => Georgian
    • khm => Khmer, Central Khmer
    • kin => Kinyarwanda
    • kor => Korean
    • kur => Kurdish
    • lat => Latin
    • lav => Latvian
    • lit => Lithuanian
    • mai => Maithili
    • mal => Malayalam
    • mar => Marathi
    • mkd => Macedonian
    • mlg => Malagasy
    • mya => Burmese
    • nep => Nepali (macrolanguage)
    • nld => Dutch, Flemish
    • nno => Norwegian Nynorsk
    • nob => Norwegian Bokmål
    • nya => Chewa, Chichewa, Nyanja
    • ori => Oriya (macrolanguage)
    • orm => Oromo
    • pan => Panjabi, Punjabi
    • pes => Iranian Persian
    • pol => Polish
    • por => Portuguese
    • ron => Moldavian, Moldovan, Romanian
    • run => Rundi
    • rus => Russian
    • sin => Sinhala, Sinhalese
    • skr => Saraiki, Seraiki
    • slk => Slovak
    • slv => Slovenian
    • sna => Shona
    • som => Somali
    • spa => Castilian, Spanish
    • srp => Serbian
    • swe => Swedish
    • tam => Tamil
    • tel => Telugu
    • tgl => Tagalog
    • tha => Thai
    • tir => Tigrinya
    • tuk => Turkmen
    • tur => Turkish
    • uig => Uighur, Uyghur
    • ukr => Ukrainian
    • urd => Urdu
    • uzb => Uzbek
    • vie => Vietnamese
    • ydd => Eastern Yiddish
    • yor => Yoruba
    • zul => Zulu
POST /search
  • authenticated endpoint (Authorization: Bearer {jwt})
{
    "collection": "coffee", 
    "bucket": "tenant_1", 
    "query": "roasted",
    "limit": 10,
    "offset": 10
}
  • limit and offset are optional fields

will return: 200 or 500 or 400 or 401

200 - will return an array of objects

[
    {
        "collection": "coffee", 
        "bucket": "tenant_1", 
        "id": "49e28aae-88d4-4c19-86d8-51f2c9f11039", 
        "data": {
            "name": "roasted",
            "image": "https://img.com/bucket/123/123.jpg"
        },
        "locale": "eng",
        "indexes": ["name"]
    }
]

Word Suggest

POST /suggest
  • authenticated endpoint (Authorization: Bearer {jwt})
{
    "collection": "coffee", 
    "bucket": "tenant_1", 
    "query": "r",
    "limit": 10
}
  • note: limit is an optional field

will return: 200 or 500 or 400 or 401

{
    "suggestions": ["roasted"]
}

Deindex

POST /deindex
  • authenticated endpoint (Authorization: Bearer {jwt})
{
    "ids": ["49e28aae-88d4-4c19-86d8-51f2c9f11039"]
}

will return: 200 or 500 or 400 or 401

Health Check

GET or HEAD /
  • public endpoint

will return: 200

Install

cargo install portal

  • the origin can be passed in as a flag - default *
  • the port can be passed in as a flag - default 8888 - can only be set for unsecure connections
  • the secure flag for https can be true or false - default false
  • the auto_cert flag for an autorenewing LetsEncrypt SSL cert can be true or false - requires a resolvable domain - default true
  • the key_path flag when auto_cert is false to set the SSL key path for your own cert - default certs/private_key.pem
  • the cert_path flag when auto_cert is false to set the SSL cert path for your own cert - default certs/chain.pem
  • the certs flag is the storage path of LetsEncrypt certs - default certs
  • the db flag is the path where the embedded database will be saved - default db
  • the domain flag is the domain name (e.g. api.broker.com) of the domain you want to register with LetsEncrypt - must be fully resolvable
  • the sonic_server flag is the sonic domain/ip/port of the sonic server - default localhost:1491
  • the sonic_password flag is the sonic password set in the sonic config file - default SecretPassword
  • the broker flag is the broker domain/ip/port of the broker server - default http://localhost:8080
  • production example: ./portal --secure="true" --domain="index.broker.com" --sonic_server="sonic.broker.com" --sonic_password="wj34T%$Dx" --broker="https://broker.broker.com"

Service

There is an example systemctl service for Ubuntu called portal.service in the code

TechStack

Inspiration

Dependencies

~68MB
~1.5M SLoC