ASIMOV Bright Data Module

ASIMOV module for data import powered by the Bright Data web data platform.

✨ Features

• Imports structured data from Airbnb, Amazon, Crunchbase, eBay, Facebook, Google, Indeed, Instagram, LinkedIn, Walmart, X (aka Twitter), Yahoo, and YouTube.
• Collects the raw JSON data via the Bright Data API (requires an API key).
• Constructs a semantic knowledge graph based on the KNOW ontology.
• Supports plain JSON output as well as RDF output in the form of JSON-LD.
• Distributed as a standalone static binary with zero runtime dependencies.

🛠️ Prerequisites

• Rust 1.85+ (2024 edition) if building from source code

⬇️ Installation

Installation from PyPI

pip install -U asimov-brightdata-module

Installation from RubyGems

gem install asimov-brightdata-module

Installation from NPM

npm install -g asimov-brightdata-module

Installation from Source Code

cargo install asimov-brightdata-module

👉 Examples

export BRIGHTDATA_API_KEY="..."

Fetching X Profiles

asimov-brightdata-fetcher https://x.com/bright_init   # JSON
asimov-brightdata-importer https://x.com/bright_init  # JSON-LD

Fetching LinkedIn Profiles

asimov-brightdata-fetcher https://www.linkedin.com/in/orlenchner/
asimov-brightdata-fetcher https://www.linkedin.com/company/bright-data/

Fetching Crunchbase Profiles

asimov-brightdata-fetcher https://www.crunchbase.com/organization/brightdata

Fetching Amazon Products

asimov-brightdata-fetcher https://www.amazon.com/Master-Algorithm-Ultimate-Learning-Machine/dp/0465094279

⚙ Configuration

Environment Variables

• BRIGHTDATA_API_KEY: (required) the Bright Data API key to use

📚 Reference

Installed Binaries

• asimov-brightdata-cataloger: discovers entities via the Bright Data API (not implemented yet)
• asimov-brightdata-fetcher: collects JSON data from the Bright Data API
• asimov-brightdata-importer: collects and transforms JSON into JSON-LD (not implemented yet)

Supported Datasets

👨‍💻 Development

git clone https://github.com/asimov-modules/asimov-brightdata-module.git