#parallel #ui #async #stack-overflow #stackoverflow #cli

bin+lib falion

An open source, programmed in rust, privacy focused tool for scraping programming resources (like stackoverflow) fast, efficient and asynchronous/parallel using the CLI or GUI

4 releases (stable)

new 1.0.2 Mar 18, 2024
1.0.0 Mar 14, 2024
0.2.0 Apr 22, 2022

#191 in Web programming

Download history 1/week @ 2024-02-15 4/week @ 2024-02-22 1/week @ 2024-02-29 264/week @ 2024-03-14

269 downloads per month

MIT and GPL-3.0-only…

4.5MB
4K SLoC

Contributors Forks Stargazers Issues MIT License Crates.io


Logo

An open source, programmed in rust, privacy focused tool for scraping programming resources (like stackoverflow) fast, efficient and asynchronous/parallel using the CLI or GUI.

Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents
  1. About The Project
  2. 🏁 Getting Started
  3. 🪧 Usage
  4. 🗾 Road Map
  5. 💁 Contributing
  6. ❓ FAQ
  7. 📟 Semver
  8. 🔰 Security
  9. 🪪 License
  10. 📧 Contact
  11. 📄 Template
  12. 🦾 Acknowledgments

🪽 About The Project

Falion is creative open-source tool created in Rust with the aim of improving programming resource scraping. It prioritizes privacy, speed, and efficiency, targeting sources like StackOverflow, GitHub Gists, and more.

The project aims to solve the challenges of accessing information quickly and privately in the programming world, using DuckDuckGo for secure, private search results. HTTPS-only connections & no JavaScript, plain HTML.

Falion is lightning-fast and efficient, with asynchronous result retrieval and parallel scraping. It takes only about 1-1.5 seconds to obtain five results from each resource. Meaning, by the time you see the results everything is already loaded and ready to be accessed.

It offers user-centric interfaces, including a Command Line Interface (CLI) with VIM-like key bindings (where applicable) and a Graphical User Interface (GUI).

Cross-platform support ensures that it runs on Linux, MacOS, and Windows.

It can be used as a standalone tool or integrated into Rust projects as a crate, enhancing development capabilities.


Key Highlights

Privacy First: It utilizes DuckDuckGo for results, stick to HTTPS-only connections, and keep it tracker-free.

Blazing Speed: Thanks to asynchronous result retrieval and parallel scraping, you get your answers in a snap.

User-Centric Interfaces: Choose between a VIM-like (where applicable) CLI or a lightweight GUI for your preferred experience.

Cross-Platform: Works seamlessly on Linux, MacOS, and Windows.

No Rate Limitations: Scraping without interruptions under normal use.

Versatile: Use it as a stand-alone tool or integrate it as a crate into your Rust projects.


Supported Resources

These are generic resources that work for most popular languages. In the future there will be more added + custom resources support. Also request resources here.

  • StackOverflow is a programming Q&A website.
  • StackExchange is a network of community-driven Q&A websites.
  • GitHub Gists are a platform for sharing and collaborating on code snippets.
  • GeeksForGeeks is an online learning platform for computer science and programming.
  • DuckDuckGo Results are just your normal results rendered as text, not as nice as the above, as a last resort.

If you've got any questions please give the FAQ a check.

(back to top)

💡 Features

  • Scrapes programming resources
    • StackOverflow - No need to say anything
    • StackExchange- index of different forums
    • Github Gists - for code snippets and guides
    • GeeksForGeeks - has various good programming resources
    • DuckDuckGo - just so you don't need to leave the program you also have html versions of the basic results
  • Privacy focused
    • DuckDuckGo used for results
    • HTTPs only requests
    • Only fetches HTML, no unnecessary tracking
  • Incredibly fast and efficient
    • Results are retrieved asynchronously
    • Resources are scraped in parallel
    • Instantly view a page, by the time results appear the content is already loaded
    • Takes about 1 second to 1 and a half to get the results (& content). 5 results each resource
    • Leverages rust's performance
    • Many optimizations done
    • Ligthweight on resources (GUI and CLI)
  • CLI interface
    • Fast and render optimized interface
    • VIM like key binds (where applicable)
    • Simple and intuitive
  • GUI interface
    • Ligthweight interface
    • Simple
    • Responsive
  • Cross-platform
    • Linux
    • MacOS
    • Windows
  • Doesn't get you rate limited, under normal use
  • Project can also be used as a crate
  • Results are cached for up to 7 days
  • Web UI
  • Custom resources support
  • Configurable
    • Config file editable from the program

(back to top)

🎥 Video showcase

Showcase

(back to top)

🍔 Built with

(back to top)

The stock libraries and these awesome third-party ones

(back to top)

🏁 Getting Started

🏃 Running The Program

Platform Versions
Windows 7 and above
Linux Anything really
MacOS 10.15 and above
All Platforms All the above

(back to top)


🪟 Windows

Portable EXE

  1. Go to the Releases Tab and download the falion-windows.exe file (might have to click show all).
  2. Double click the exe you just downloaded and there you go the program works.
  3. Might wanna take a look at the Usage Tab if you don't understand something about it.
  4. Also you might wanna make a desktop shortcut and one to place in the start-menu

Installer

  1. Go to the Releases Tab and download the falion-windows-installer.exe file (might have to click show all).
  2. Double click the installer and go through it as you would with any other installer.
  3. If you look now in the start menu (or on the desktop if you ticked create desktop shortcut) you are gonna see a shortcut for falion, just run it like any other program.
  4. Might wanna take a look at the Usage Tab if you don't understand something about it.

(back to top)


🐧 Linux

⚠️ Dependencies: openssl, most likely already installed.

(back to top)

Portable Bin

  1. Go to the Releases Tab and download the falion-linux file.
  2. Double click the bin you just downloaded and there you go the program works.
  3. Might wanna take a look at the Usage Tab if you don't understand something about it.
  4. Also you might wanna download falion-linux.tar.gz file instead and extract linux/desktop/falion.desktop to $HOME/.local/share/applications, linux/desktop/icons to $HOME/.local/share/icons in order to have a start-menu shortcut and falion to /usr/bin/.

AppImage

  1. Go to the Releases Tab and download the falion-linux.AppImage file.
  2. Double click the AppImage you download and there you go the program just works. You may want to install AppImageLauncher if you don't have it already, so when you start the AppImage you'll get a prompt asking if you want to integrate and run it and if you do so it will appear just as if you installed it.
  3. Might wanna take a look at the Usage Tab if you don't understand something about it.

AUR

The PKGs are:

  • falion (for stable)
  • falion-bin (for precompiled)
  • falion-git (to compile from the latest source code commit)

a. if you have an AUR manager (like paru or yay, which you should)

  1. Just like with any other AUR pkg choose your prefered type and you can run the following command for example.

    paru -Sy falion
    
  2. Search for falion in your app launcher and launch it.

  3. Might wanna take a look at the Usage Tab if you don't understand something about it.

b. Manually cloning and building it from AUR

  1. First install the basic build dependencies, if you don't already have them:

    sudo pacman -Sy gcc base-devel --needed
    
  2. Then clone the build script

    git clone https://aur.archlinux.org/falion.git # or falion-bin & falion-git
    
  3. Cd into the new cloned repository and run the following to build the package

    makepkg
    
  4. In order to install the package run the following (where * is just an any other characters place holder)

    sudo pacman -U falion-*.pkg.tar.zst
    

Nix File

You are using NixOS, don't worry I got you bro.

  1. Go to the Releases Tab and download the falion-linux.nix file.
  2. If you use flakes then put it in your pkgs folder, and up-top add your tag (like my). If you don't just add the code in your default.nix file and install it this way.
  3. Might wanna take a look at the Usage Tab if you don't understand something about it.

Deb File

You should use the app image. This does not provide a desktop file, you'll have to run it from the command-line. It's here just as another means if needed. I will try to make a ppa eventually.

  1. Go to the Releases Tab and download the falion-linux.deb file.

  2. Open a terminal in the folder where your download is and run the following command:

    sudo dpkg -i falion-linux.deb
    
  3. Run falion in the terminal and there it is, the app.

  4. Might wanna take a look at the Usage Tab if you don't understand something about it.

  5. Also you might wanna download falion-linux.tar.gz file instead and extract linux/desktop/falion.desktop to $HOME/.local/share/applications and linux/desktop/icons to $HOME/.local/share/icons in order to have a start-menu shortcut.

(back to top)


🍎 macOS

(back to top)

Portable binary

  1. Go to the Releases Tab and download the falion-macos file.
  2. Double click the bin you just downloaded and there you go the program works.
  3. Might wanna take a look at the Usage Tab if you don't understand something about it.
  4. Also you might wanna place it somewhere special and create a shortuct

App Folder

Damn simillar to portable binary, only real difference is this has an icon.

  1. Go to the Releases Tab and download the falion-macos-app.tar.gz file.

  2. Use your archive manager or run in the terminal the following command:

    tar -xzf falion-macos-app.tar.gz
    
  3. Double clikc the app folder you just downloaded and there you go the program works.

  4. Might wanna take a look at the Usage Tab if you don't understand something about it.

DMG Installer

Works just like any other dmg installer you've used.

  1. Go to the Releases Tab and download the falion-macos-installer.dmg file.
  2. Double click to run the dmg.
  3. Drag the app folder over the Applications folder.
  4. Done, you've just installed the app, should see it in launchpad now.
  5. Might wanna take a look at the Usage Tab if you don't understand something about it.

Homebrew

Note this method doesn't come with a desktop entry. You'll have to run the falion command or just create a shortcut yourself, it's really easy.

  1. You will need to have homebrew installed, if you don't have it installed run the following command:

    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
    
  2. You'll need to add my tap repository, run the following command for that:

    brew tap Obscurely/tap
    
  3. Install the pkg.

    brew install falion
    
  4. Might wanna take a look at the Usage Tab if you don't understand something about it.

(back to top)


🚉 All Platforms

This method will work across any Linux distribution, Windows 10/11 and macOS (Big Sur+ tested).

  1. Install rust, either using the official rustup installer or any pkg manager you may use. (There is also a shell.nix file in the repository if you use nix)

  2. Run the following command in your terminal of choice:

    cargo install falion
    
  3. Make sure you have .cargo/bin in path, for linux and macOS you would need to add the following line in your terminal RC file (e.g $HOME/.zshrc)

    export PATH=$HOME/.cargo/bin:$PATH # This is for Linux & macOS, look below for Windows.
    

    On windows it should work automatically (restart if just installed), if not you can follow this guide for how to add something to path. The cargo bin folder will be Users\{your-user}\.cargo\bin

  4. You may want to create a symlink on Linux & macOS or create a shortcut if you are on Windows to the bin file for easy access.

  5. In order to update run the install command again, and you can now follow usage for more information on how to use it.

  6. Also you might wanna create a shortcut and place it in the start-menu acording to your platform.

(back to top)


🛠 Compilation

This program only uses cross platform libraries. The following steps require that you have rust installed, check their official installation page or use any pkg manager you may want. (There is also a shell.nix file in the repository if you use nix).

  1. Clone this repository on your PC, you can use "git clone", if you have git installed, like this:

    git clone https://github.com/Obscurely/falion.git
    

    Otherwise in the right up side of the repository page you will see a download button, download the repository as zip and extract it in a folder

  2. Open a new terminal/cmd window in the folder you extracted the repository in, if you can't right click on the folder and open it there do:

    cd the/path
    

    and you will get there.

  3. From there run this compile command in the terminal:

    cargo build --release
    

    It will take a bit depending on your system because of executable speed optimizations, but be patient.

  4. Done, navigate to target/release and grab only the "falion" file from there.

(back to top)

🪧 Usage

In the provided video it's presented everything you should know on how to use falion. I also think the UI is intuitive enough, but I obviously can't have an unbiased opinion or a first look experience, so here you go.

Basics

CLI key binds

Note: where '..' is used it means from that to that like '1..5' would mean from 1 to 5.

**Main menu:**
[1..5] = Access that resource.
SHIFT + [1..5] = Go to the next element in the list of that resource.
ALT + [1..5] = Go to the previous element in the list of that resource.
n = Move to the next element in the list of every resource.
SHIFT + n = Move back to the previous element in the list of every resource.
CTRL + c = Clear terminal and exit.

**Sub menus for the resources:**
n = Move to the next element in the content list (like questions & answers).
SHIFT + n = Move back to the previous element in the content list.
q = Go back to the main menu.
CTRL + c = Clear terminal and exit.

How to use GUI

  1. Write your query in the search box and press enter or click the button besides it.
  2. Browse through the resources
    1. The arrows besides the resources are for moving one element forwards or backwards individually.
    2. The arrows at the bottom are for moving one elment forwards or backwards all the resources.
  3. Click on a button containing the title to move into the result view for that result.
  4. If it's a dynamic result use the arrows to move, if not just scroll through.
  5. In the upper left corner there is a back button to take you back to the main menu.

Advanced

There are few cli options you might wanna know

  1. --verbose to make logging verbose
  2. --disable-logs to disable writting log files
  3. --ui to run the ui from the cli
  4. --help to display the help message
  5. --keybinds to display the keybinds list for the cli

(back to top)

🗾 Road Map

The roadmap (kanban board) is located up top in the projects tab or at this link.

(back to top)

💁 Contributing

Contributions are always welcome!

Please read CONTRIBUTING.md and CODE_OF_CONDUCT.md, but to generalise it the basic process is: edit a file you want, do a pull request, I will look at it and if the change makes sense and is a good one I will accept it and that's it. Also be kind and respectful.

(back to top)

❓FAQ

  1. Does Falion support scraping of non-English content on supported resources like StackOverflow or GitHub Gists?

    No. For one DuckDuckGO struggles with anything, but english and second I have configured the client to only accept english because most of the programming related stuff is in english.

  2. Can I contribute and if so, what's the process?

    First refer to Contributing. Second as long as you write good, clean, fast, documented/commented code and it helps with the project I will be more than happy to accept your pull request.

  3. What privacy measures does Falion take when scraping content from resources?

    It uses DuckDuckGO for search results, queries the least number of urls possible and only html pages with no added finger prints to the client. The only thing to worry about is your IP address which most people can get a new one with a router restart so it holds no value.

  4. Is running Falion on older hardware slower than advertised?

    I have ran it in a 1 core Windows vm on my 6 years old cpu (ryzen 7 2700) with turbo disabled, 2Gb of ram and the performance seemed to be about the same (at most half a sec slower). As long as your total network latency is low and the speed above like 100Mbits/s it's going to be mostly as advertised. Regardless it will be faster than using your browser.

  5. How can I report a bug or request a feature?

    Just go here.

  6. What can the falion crate do for my project?

    All the methods used to get results from the programming resources or duckduckgo are exposed. So if you want to get search results from any of those in a highly efficient way this is the crate for you.

(back to top)

📟 Semver

falion is following Semantic Versioning 2.0.

(back to top)

🔰 Security

If you happen to find any issues please carefully read SECURITY.md and strictly follow everything stated.

(back to top)

🪪 License

Is under MIT so stick to the license conditions and have fun :)

(back to top)

📧 Contact

Either post an issue in the Issues Tab or contact me at this email address if you have more to say: obscurely.message@protonmail.com

(back to top)

📄 Template

This repository is based on this template Obscurely/RustTemplate

🦾 Acknowledgments

(back to top)

ko-fi

Dependencies

~22–40MB
~647K SLoC