#search-engine #real-time #ai #indexing #integration #update #processing

yanked infini-pizza

The Next-Gen Real-Time Search & AI-Native Innovation Engine

1 unstable release

0.1.2 May 23, 2023
0.1.1 May 23, 2023
0.1.0 May 23, 2023

#80 in #indexing

25 downloads per month

AGPL-3.0-or-later

22KB
532 lines

INFINI Search, Code Name: Pizza

The Next-Gen Real-Time Search & AI-Native Innovation Engine.

Major Features of INFINI Pizza:

  • True real-time: Get search results instantly after insertion, no need to refresh anymore.
  • Support partial update in place: No longer pull and push back the entire document again.
  • High performance: Extremely fast with high throughput and low latency, hardware reduced.
  • High scalability: Supports very large-scale clusters, beyond petabytes.
  • An in-memory index and columnar store using object storage for persistence.
  • Native integration with LLMs and ML, empowering AI-Native enterprise innovation.

Design with storage and computation separation, and also storage and indexing separation. This feature emphasizes the decoupling of storage and computation, as well as the separation of storage and indexing. It enables efficient and scalable data processing by allowing independent management and optimization of storage resources, computational resources, and indexing strategies.

The documents are persisted as Parquet files in object storage. Native integration with other big data systems through object storage and the standard Parquet format.

The philosophy of INFINI Pizza is that indexes should be designed per use case, and should not attempt to fit every use case with a single index. Therefore, we introduced Views, which allow combining different document sources into a single index or separating a document into different layers of indexes for different use cases.

Native integration with LLMs (Language Models) and ML (Machine Learning) technologies is a key aspect of INFINI Pizza, providing powerful capabilities for AI-Native enterprise innovation. By seamlessly integrating with LLMs and ML frameworks, INFINI Pizza enables advanced natural language processing, machine learning, and data analytics directly within the search and data retrieval pipeline.

This integration empowers enterprises to leverage the full potential of AI technologies, such as sentiment analysis, language understanding, recommendation systems, and predictive analytics, to enhance their search and data-driven applications. By harnessing the power of LLMs and ML models, INFINI Pizza facilitates more accurate and context-aware search results, personalized recommendations, intelligent query understanding, and other AI-driven functionalities.

The native integration with LLMs and ML in INFINI Pizza eliminates the need for separate infrastructure or complex integration efforts. It provides a seamless and efficient environment for enterprises to leverage cutting-edge AI capabilities within their search and data processing workflows. This integration opens up new possibilities for AI-driven innovation and unlocks the full potential of enterprise data for enhanced decision-making, customer experiences, and business insights.

To learn more about INFINI Pizza, please visit: http://infinilabs.com/

Dependencies

~17–28MB
~480K SLoC