Can I add, update or remove documents from an existing Lingo4G index?

Yes. With Lingo4G version 1.7.0 or later you can add, update and delete without re-indexing the whole collection.

Does Lingo4G come as an Elasticsearch or Solr plugin?

No. Lingo4G is standalone software that manges its own index. The index contains Lingo4G-specific data that is not typically present in indices created by enterprise search engines. Therefore, Lingo4G will have to run in parallel to an Elasticsearch or Solr instance you might already have. As a result, the same will be stored separately in the two systems.

Is Lingo4G an end-user product?

No. Lingo4G was designed as a software component rather than a complete end-user application. The primary use case for Lingo4G is integration into larger software suites. Therefore, some programming experience will be required to get started and to use Lingo4G.

Having said that, Lingo4G comes with a GUI application, called Lingo4G Explorer, meant to allow rapid experiments and tuning of topic extraction and clustering. If Lingo4G Explorer meets the needs of your end users, feel free to use it. One thing to bear in mind is that our primary focus is on the underlying topic extraction and clustering algorithms, developing and extending Lingo4G Explorer is a lower priority.

What are the system requirements for Lingo4G?

Lingo4G can run on any platform supporting Java 17 or later. While processing cannot currently be distributed to multiple machines, a high-end workstation with fast SSD storage should be capable of handling collections of several tens of gigabytes. For most data sets not exceeding gigabytes, any computer with 4GB of memory and some disk space will be sufficient. We very much recommend using SSD drives to store Lingo4G indices. Please see the Requirements section of Lingo4G manual for more details.

What does near real-time processing mean?

Near real-time processing means that in most cases Lingo4G will be able to extract topics for a subset of documents in your index within seconds, regardless of the size of the subset. Clustering of tens or hundreds of thousands of documents may take several minutes.

To achieve near real-time performance during analysis, Lingo4G needs to index your collection first. Indexing is a one-time process where the initial text processing (tokenization, label extraction) is applied. On modern hardware, indexing speed is in the order of 200–500 MB per minute.

The performance of both indexing and analysis is crucially dependent on the technology used to store Lingo4G index. To maximize performance and CPU utilization, use fast SSD drives.

What is the largest collection Lingo4G can handle?

On modern hardware with a high-core-count CPU and fast SSD storage, Lingo4G can handle collections reaching hundreds of gigabytes or a terabyte of text.

If you'd like to test Lingo4G on such a large data set, Lingo4G comes with built-in support for indexing patent grant and application documents available from US Patent and Trademark Office. The collection is currently about 500 GB of text.

One important factor to consider is that currently Lingo4G does not offer distributed processing. This means that the maximum reasonable size of the project will be limited by the amount of RAM, disk space and processing power available on a single virtual or physical server.

Which languages does Lingo4G support?

Currently, Lingo4G can only process English text. If you'd like to apply Lingo4G to content written in a different language, please contact us.

Can I use the dotAtlas map visualization component in my application?

The dotAtlas map visualization component shipping with Lingo4G Explorer is currently pre-release software. It's been battle-tested for months by early adopters, but lacks finalized API and documentation.

If you'd like to try integrating dotAtlas into your software, please let us know. We'll be happy to share the pre-release version along with code examples and initial guidance.

We will not charge any extra fees for the pre-release versions of dotAtlas. Once it enters the official product suite, the use of dotAtlas will require a license fee similar to the one that applies for Carrot Search FoamTree.

What are the applications of Lingo4G?

The natural use case is exploration of large volumes of human-readable text, such as scientific papers, business or legal documents.

You can combine Lingo4G basic text processing operations to build, for example, the following functionalities:

• Recommending tags for a new document based on the tags of the documents existing in the index.

• Example-based document search: finding documents similar to the example seed document the user provides, based on keyword or semantic vector similarity.

• 2D map visualization of documents where semantically-similar documents occupy the same area of the map. Colors of the document marker on the map indicate the cluster to which the document belongs. Salient phrases extracted from documents describe the densely-populated areas of the map.

• Finding documents with highly overlapping content, which may suggest the documents are duplicates or plagiarised copies. Lingo4G can perform such a process efficiently for millions of documents at once.

I'd like to cluster search results fetched from public search engines. Is Lingo4G suitable for this?

No, the only engine that fits here is Lingo3G.

Before clustering, Lingo4G needs to access the entire collection of documents in advance to create an index. In case of public search engines, the underlying collection is not usually available as a whole, which rules out Lingo4G*.

Lingo3G, on the other hand, was designed to cluster data in one pass — you'd simply fetch the text of search results from the search engine, feed the text to Lingo3G and receive the search results clustered.

I have a fast-growing collection of social conversations and I'd like to detect emerging topics in that data. Is Lingo4G or Lingo3G suitable for the task?

While neither Lingo4G nor Lingo3G have a specific feature of emerging topic discovery, they will likely be useful components of such systems.

Currently, Lingo4G's fast incremental indexing option does not discover new phrases introduced by new documents. To allow Lingo4G to pick up new phrases from newly-added conversations, features would have to be periodically re-generated based on the whole collection. This may not be a problem if it is acceptable for the new phrases to enter analysis on, for example, daily basis.

If it is required for the new content to be considered immediately after it arrives, the only applicable engine is currently Lingo3G, possibly coupled with Apache Solr or Elasticsearch.

I have a 50 GB collection of patent applications. I want to extract topics for up to 1000-document subsets of that collection. Which engine is better in this case?

Both Lingo4G and Lingo3G are suitable in this case.

Lingo4G will create an index for your collection, which will give you the ability not only to extract topics, but also to search your collection, perform conventional document clustering, produce 2d document maps and complete more advanced text mining tasks, such as classification.

In case of Lingo3G, the easiest way to process your collection would be to index it in, for example, Apache Solr or Elasticsearch. Then, depending on the architecture of your software, you would either fetch documents from your search engine and feed them to Lingo3G Java/C# API for clustering, or have your search results clustered directly inside the search engine through Solr or Elasticsearch plugins.

I have a collection of 1 million research papers and I'd like to extract topics for up to 200k-document subsets at once. Which engine can handle that task?

Only Lingo4G can handle topic extraction for such large subsets of your collection. Its on-disk index makes it possible to analyze hundreds of thousands of documents within a limited memory footprint.

Lingo3G with its in-memory processing model is not suitable for inputs exceeding 10 MB of text.