RadLab — Blog

Are crypto mining farms frozen?

2026-01-02T00:00:00Z

Today we’re looking at a slightly different “fairy‑tale”. Not from a production‑side technical perspective, but perhaps a proposal for a partial solution to the problem that’s been discussed in the media—how else we might use “frozen” cryptocurrency mines 😉 for example,LINK..

Given the demand for computing power required to train models, not every graphics card will be suitable for this purpose (you need cards with a very large amount of VRAM). In this post we’ll show how to turn a USB drive into a ready‑made computational environment, to which we’ll attach an LLM Router as a communication layer for generative models.

If you already have a ready environment for cryptocurrency mining, you can safely skip the HiveOS Installation step. If you don’t, going through this step helps to prepare the system. In the description a USB drive is used as the system medium, because HiveOS is installed by default on USB.

Intro

How can llm‑router help solve this problem? The proposal is actually quite obvious and simple: use the resources of a cryptocurrency mining farm as a provider for offering inference services for generative models. In other words, if you already have a ready‑made infrastructure, this post will show you how to leverage it to run generative models. As a result, you’ll have a locally operating service that can launch models such as gpt‑oss or gemma.

What can you do with this setup? For example, you could offer a paid service that performs computations (inference) on those models.

HiveOS Installation

The first step is to install a ready‑made operating system with the graphics‑card drivers already configured. For example, you can use HiveOS (the standard Ubuntu desktop is also an acceptable option if you don’t want to deal with installing GPU‑driver libraries yourself). The installation process is described in detail on the official site https://hiveon.com/install – the currently available version is based on Ubuntu 22.04.

We select the GPU (or the beta driver version for the adventurous ;‑)) and install it according to the How to Write Image guide. In our example we performed the installation on a USB stick using Ubuntu’s built‑in image‑writing tool (you can also use balenaEtcher).

After the installation the default user is user and the password is 1. Of course, it’s good practice to change the default password to something more complex (once you’re logged in to the GPU machine run passwd).

OS configuration

We’ll skip the part where you set up the HiveOS server as a mining RIG (cryptocurrency mining and process monitoring through the HiveOS interface). Instead, we’ll take advantage of the already‑configured system and install a few basic system libraries that are required to run a local instance of LLM Router together with a model served via Ollama.

The local provider can be any of the following – Ollama, vLLM, llama.cpp – it doesn’t matter; LLM Router is simply an interface that talks to whatever local model‑provider protocol you choose.

After logging into the machine that has HiveOS installed: ssh user@IP.LOKALNE.MASZYNY you will see information about the loaded drivers and the available graphics cards, e.g.:

In our case there are three RTX 3090 cards available, each with 24 GB of VRAM. Detailed information about limits, temperatures, etc., can be viewed with the nvidia‑info command (for NVIDIA GPUs).

Now we only need to install the basic packages pip(for installing dependencies) and git(for cloning the repository):

# Installation on the system (if it isn’t already present) git + pip
apt install git python3-pip -y

In fact, those two additional packages are enough for the operating system to be fully configured to run LLM Router.

Installing LLM Router

In this example we show how to run the latest version of LLM Router natively (i.e., not from a Docker image) directly from the main (main) branch.

# Cloning the repository llm-router
git clone https://github.com/radlab-dev-group/llm-router

# Installation of dependencies and the API
cd llm-router/
pip install -r requirements.txt
pip install .[api]

That’s it—in short: download, install, and start with just three commands :). Now, by running: ./run-rest-api-gunicorn.sh we expose a ready‑made API using Gunicorn. At this point no models are attached yet, but the network service (REST API) is already up and running.

View the startup log [CLICK]+

root@HiveOS:/home/user/llm-router# ./run-rest-api-gunicorn.sh
2026-01-01 17:59:14,058 INFO main: Starting LLM‑Router API with gunicorn
2026-01-01 17:59:14,065 DEBUG llm_router_api.core.lb.provider_strategy_facade: [provider-monitor] keys to check: []
2026-01-01 17:59:14,072 INFO llm_router_api.core.lb.provider_strategy_facade: [Load balancing] Strategy FirstAvailableStrategy
2026-01-01 17:59:14,072 DEBUG llm_router_api.core.monitor.services_monitor: [services-monitor] thread started
2026-01-01 17:59:14,077 DEBUG llm_router_api.register.auto_loader: Instantiating OpenAICompletionHandlerWOApi
2026-01-01 17:59:14,078 DEBUG llm_router_api.register.auto_loader: Instantiating AnswerBasedOnTheContext
2026-01-01 17:59:14,078 DEBUG llm_router_api.register.auto_loader: Instantiating OpenAICompletionHandler
2026-01-01 17:59:14,078 DEBUG llm_router_api.register.auto_loader: Instantiating OpenAIModelsV1Handler
2026-01-01 17:59:14,078 DEBUG llm_router_api.register.auto_loader: Instantiating ApiVersion
2026-01-01 17:59:14,078 INFO llm_router_api.endpoints.endpoint_i: -> Running LLM-Router version: 0.4.4
2026-01-01 17:59:14,078 DEBUG llm_router_api.register.auto_loader: Instantiating OpenAIResponsesV1Handler
2026-01-01 17:59:14,078 DEBUG llm_router_api.register.auto_loader: Instantiating Ping
2026-01-01 17:59:14,078 DEBUG llm_router_api.register.auto_loader: Instantiating OpenAIResponsesHandler
2026-01-01 17:59:14,078 DEBUG llm_router_api.register.auto_loader: Instantiating ConversationWithModel
2026-01-01 17:59:14,078 DEBUG llm_router_api.register.auto_loader: Instantiating FullArticleFromTexts
2026-01-01 17:59:14,078 DEBUG llm_router_api.register.auto_loader: Instantiating ExtendedConversationWithModel
2026-01-01 17:59:14,078 DEBUG llm_router_api.register.auto_loader: Instantiating GenerateNewsFromTextHandler
2026-01-01 17:59:14,078 DEBUG llm_router_api.register.auto_loader: Instantiating TranslateTexts
2026-01-01 17:59:14,078 DEBUG llm_router_api.register.auto_loader: Instantiating SimplifyTexts
2026-01-01 17:59:14,079 DEBUG llm_router_api.register.auto_loader: Instantiating OllamaTagsHandler
2026-01-01 17:59:14,079 DEBUG llm_router_api.register.auto_loader: Instantiating GenerateQuestionsFromTexts
2026-01-01 17:59:14,079 DEBUG llm_router_api.register.auto_loader: Instantiating LLMStudioChatV0Handler
2026-01-01 17:59:14,079 DEBUG llm_router_api.register.auto_loader: Instantiating OllamaHomeHandler
2026-01-01 17:59:14,079 DEBUG llm_router_api.register.auto_loader: Instantiating FastTextMasking
2026-01-01 17:59:14,079 DEBUG llm_router_api.register.auto_loader: Instantiating VllmChatCompletion
2026-01-01 17:59:14,079 DEBUG llm_router_api.register.auto_loader: Instantiating LmStudioModelsHandler
2026-01-01 17:59:14,079 DEBUG llm_router_api.register.auto_loader: Instantiating OpenAIModelsHandler
2026-01-01 17:59:14,079 DEBUG llm_router_api.register.auto_loader: Instantiating OllamaChatHandler
2026-01-01 17:59:14,079 INFO llm_router_api.register.register: Registered endpoint POST /chat/completions (OpenAICompletionHandlerWOApi)
2026-01-01 17:59:14,079 INFO llm_router_api.register.register: Registered endpoint POST /api/generative_answer (AnswerBasedOnTheContext)
2026-01-01 17:59:14,080 INFO llm_router_api.register.register: Registered endpoint POST /api/chat/completions (OpenAICompletionHandler)
2026-01-01 17:59:14,080 INFO llm_router_api.register.register: Registered endpoint GET /v1/models (OpenAIModelsV1Handler)
2026-01-01 17:59:14,080 INFO llm_router_api.register.register: Registered endpoint GET /api/version (ApiVersion)
2026-01-01 17:59:14,080 INFO llm_router_api.register.register: Registered endpoint POST /v1/responses (OpenAIResponsesV1Handler)
2026-01-01 17:59:14,080 INFO llm_router_api.register.register: Registered endpoint GET /api/ping (Ping)
2026-01-01 17:59:14,081 INFO llm_router_api.register.register: Registered endpoint POST /responses (OpenAIResponsesHandler)
2026-01-01 17:59:14,081 INFO llm_router_api.register.register: Registered endpoint POST /api/conversation_with_model (ConversationWithModel)
2026-01-01 17:59:14,081 INFO llm_router_api.register.register: Registered endpoint POST /api/create_full_article_from_texts (FullArticleFromTexts)
2026-01-01 17:59:14,081 INFO llm_router_api.register.register: Registered endpoint POST /api/extended_conversation_with_model (ExtendedConversationWithModel)
2026-01-01 17:59:14,081 INFO llm_router_api.register.register: Registered endpoint POST /api/generate_article_from_text (GenerateNewsFromTextHandler)
2026-01-01 17:59:14,081 INFO llm_router_api.register.register: Registered endpoint POST /api/translate (TranslateTexts)
2026-01-01 17:59:14,082 INFO llm_router_api.register.register: Registered endpoint POST /api/simplify_text (SimplifyTexts)
2026-01-01 17:59:14,082 INFO llm_router_api.register.register: Registered endpoint GET /api/tags (OllamaTagsHandler)
2026-01-01 17:59:14,082 INFO llm_router_api.register.register: Registered endpoint POST /api/generate_questions (GenerateQuestionsFromTexts)
2026-01-01 17:59:14,082 INFO llm_router_api.register.register: Registered endpoint POST /api/v0/chat/completions (LLMStudioChatV0Handler)
2026-01-01 17:59:14,082 INFO llm_router_api.register.register: Registered endpoint GET / (OllamaHomeHandler)
2026-01-01 17:59:14,082 INFO llm_router_api.register.register: Registered endpoint POST /api/fast_text_mask (FastTextMasking)
2026-01-01 17:59:14,083 INFO llm_router_api.register.register: Registered endpoint POST /v1/chat/completions (VllmChatCompletion)
2026-01-01 17:59:14,083 INFO llm_router_api.register.register: Registered endpoint GET /api/v0/models (LmStudioModelsHandler)
2026-01-01 17:59:14,083 INFO llm_router_api.register.register: Registered endpoint GET /models (OpenAIModelsHandler)
2026-01-01 17:59:14,083 INFO llm_router_api.register.register: Registered endpoint POST /api/chat (OllamaChatHandler)
2026-01-01 17:59:14,083 INFO llm_router_api.core.metrics: [Prometheus] preparing metrics request hooks
2026-01-01 17:59:14,083 INFO llm_router_api.core.metrics: [Prometheus] registering metrics request hooks
[2026-01-01 17:59:14 +0000] [147288] [INFO] Starting gunicorn 23.0.0
[2026-01-01 17:59:14 +0000] [147288] [INFO] Listening at: http://0.0.0.0:8080 (147288)
[2026-01-01 17:59:14 +0000] [147288] [INFO] Using worker: gthread
[2026-01-01 17:59:14 +0000] [147296] [INFO] Booting worker with pid: 147296
[2026-01-01 17:59:14 +0000] [147297] [INFO] Booting worker with pid: 147297
[2026-01-01 17:59:14 +0000] [147298] [INFO] Booting worker with pid: 147298
[2026-01-01 17:59:14 +0000] [147299] [INFO] Booting worker with pid: 147299
2026-01-01 17:59:19,066 DEBUG llm_router_api.core.lb.provider_strategy_facade: [provider-monitor] keys to check: []

The log contains information from DEBUG mode, so there’s quite a lot of it. This is the default logging level, which can be changed at any time.

Sample configuration: Ollama + gpt‑oss:120b

Now we’ll move on to installing Ollama with the 120‑billion‑parameter model (gpt‑oss:120b) made available. The model is quite large, requiring three RTX cards with 24 GB of VRAM each. Ollama can run this model very well on three cards while maintaining almost full context‑length support.

We stop the running LLM Router instance (CTRL +C) and install Ollama—the installation is straightforward; just invoke the official installation script from the console:

curl -fsSL https://ollama.com/install.sh | sh

After a short while (depending on your internet connection), Ollama will be ready and you need to load a local model onto it. A detailed guide on how to do this locally is available on the model’s page: https://ollama.com/library/gpt-oss. All you have to do is:

ollama run gpt-oss:120b

After the download finishes, the model is immediately available in the local Ollama instance.

NOTE!

The gpt-oss:120b model available through Ollama is a quantized version of OpenAI’s gpt‑oss‑120b. Despite the heavy quantization, the model is still very large—it occupies roughly 65 GB on disk (and the same amount in GPU memory). Consequently, the USB drive must be sufficiently large to store the model locally (or you can mount an external drive that already contains the model).

OPTION: If the model has already been downloaded and is available over the network, you can attach the model from a network drive instead of copying it to a USB stick. Below is an example of how to run the model on three GPUs when the model is stored on a network share (e.g., via nfs) from a directory /mnt/data2/llms/models/ollama/.ollama/models/:

CUDA_VISIBLE_DEVICES=0,1,2 \
  OLLAMA_NOPRUNE=true \
  OLLAMA_CONTEXT_LENGTH=128000 \
  OLLAMA_MODELS=/mnt/data2/llms/models/ollama/.ollama/models/ \
  OLLAMA_HOST=0.0.0.0 \
  ollama serve

To see which models are available in Ollama, simply run the command ollama list. For example, in our setup (with a mounted network drive):

root@HiveOS:/home/user/llm-router# ollama list
NAME                           ID              SIZE      MODIFIED
dolphin-mistral:latest         5dc8c5a2be65    4.1 GB    5 days ago
dolphin3:8b                    d5ab9ae8e1f2    4.9 GB    6 days ago
devstral-2:latest              524a6607f0f5    74 GB     6 days ago
devstral-2:123b                524a6607f0f5    74 GB     6 days ago
deepseek-r1:70b                d37b54d01a76    42 GB     4 weeks ago
gemini-3-pro-preview:latest    91a1db042ba1    -         4 weeks ago
gpt-oss:120b-ctx32k-rope       c265a9f5b5de    65 GB     7 weeks ago
gpt-oss:120b-ctx250k           964592b444fd    65 GB     7 weeks ago
glm-4.6:cloud                  05277b76269f    -         8 weeks ago
qwen3-coder:30b                06c1097efce0    18 GB     2 months ago
qwen3:235b                     72840bddff91    142 GB    3 months ago
gpt-oss:120b                   f7f8e2f8f4e0    65 GB     4 months ago
gpt-oss:20b                    aa4295ac10c3    13 GB     4 months ago

LLM Router Configuration for gpt‑oss:120b

Alright, we now have the system set up, the router downloaded, and the model loaded into Ollama. It’s time to connect the model to the router. The previous commands start the model on the same machine where the llm‑router is running. To simplify the demonstration, we also provide a ready‑made configuration file that can be plugged into the router (it will overwrite the default configuration file):

You can either edit the startup script run‑rest‑api‑gunicorn.sh and change the default path to the configuration (the variable LLM_ROUTER_MODELS_CONFIG ),
or, even more simply, pass the path to the prepared configuration when launching the router process. In other words, just issue the following command:

LLM_ROUTER_MODELS_CONFIG=resources/configs/specific/models-config-local-ollama.json ./run-rest-api-gunicorn.sh

That’s basically it 🙂 You can now query the router by specifying the model name. Here’s an example using curl with JSON formatting via jq:

curl -X POST http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "gpt-oss:120b", "messages": [{"role": "user", "content": "Cześć"}]}' | jq

which should produce a result similar to:

Outro

These few steps are enough to get the LLM Router up and running on typical cryptocurrency‑mining hardware and operating systems, leveraging those resources. Of course, this is just an example; the repository contains many more model configurations and ways to connect to various local providers.

Miners, this is your second chance! 🙂

Happy New Year! Take care of your data and avoid sending sensitive information to the cloud!

Can a knowledge base be easily introduced for GenAI?

2025-12-27T00:00:00Z

Absolutely—just plug langchain_rag into the llm-router 😉

Intro

Hello! Today we are showing additional capabilities of the LLM Router in the form of plugins, actually just one plugin langchain_rag. This is a plugin that creates a local knowledge base from the content provided during the indexing process. It uses well‑known solutions such as langchain to create a RAG system and fais as a vector knowledge base. Then the created knowledge base can be attached to any query in the LLM Router, without modifying the application that currently uses generative models. What does this approach provide?

The ability to automatically enrich the context passed to the generative model — i.e., the standard RAG approach. It is enough to index the documents you choose using the available CLI command llm-router-rag-langchain index [OPTIONS] and all those contents are available to the generative model when generating a response.

What does this provide? The ability to automatically enrich the context passed to the generative model — i.e., the standard RAG approach. It is enough to index the documents you choose using the available CLI command llm-router-rag-langchain index [OPTIONS], and all those contents are available to the generative model when generating a response.

Indexation

Data indexing is the step in which, from a given directory, files with specified extensions are stored in a local knowledge base. The knowledge‑base plugin uses a loaded embedder model (by default configured asgoogle/embeddinggemma-300m) to create vector representations of chunked texts and saves them to a locally created FAISS vector database. It is the user who chooses which documents to use for context enrichment and in which collections they should be placed. This is an important step because the accuracy of the generative model’s responses depends on the content of this database. Despite its importance, the process is very simple. After installing llm‑router, a command for handling the plugin’s knowledge base is available: llm-router-rag-langchain. The command is installed in the CLI, so it is available from any router directory.

In today’s example we used all the descriptions from the main repository and sub‑repositories of LLM Router as the knowledge base:

Main llm-routera repository: md and txt files;
Plugins repository (including the described plugin) – files in md , txt format;
Services handling guardrails and maskers plugins – md and txt ;
Repository with example tools using llm‑router – md and txt files;
Dedicated graphical interfaces for Anonymizer and Configs Manager – also md and txt files;
llm-router.cloud website – html , md , and txt files;

For simplicity we added a basic Bash script in which you only need to change the file types and indexing paths and, without modifying any other settings from the main router directory, issue the command [here full script]:

bash scripts/llm-router-rag-langchain-index.sh

Because of this, a local knowledge base will be saved in the default directory: workdir/plugins/utils/rag/langchain/sample_collection/, which can then be attached as the knowledge base in the llm‑router.

Running the knowledge base in LLM Router

The knowledge base is loaded by default from the directory where the LLM Router is started, but the storage/load location can be set arbitrarily with the environment variable LLM_ROUTER_LANGCHAIN_RAG_PERSIST_DIR. For the LLM Router to use the created knowledge base, the langchain_rag plugin must be activated and the path to the created knowledge base must be set. After starting the router with such a configuration, every query to the generative model will be enriched with context from the attached knowledge base. Activating the plugin simply means exporting the environment variable with utils‑type plugins when launching the router (the default scipt allows modification).

export LLM_ROUTER_UTILS_PLUGINS_PIPELINE="langchain_rag"

And that’s basically it—if you don’t modify parameters such as context size or the embedder, there’s nothing you need to change or configure—just index your own data 🙂 How it looks in practice. Below, in the screenshot, is the model’s response without the attached langchain_rag plugin, with the indexed content:

And the response with the plugin attached:

Outro

And to wrap up, here’s a console video showing how to index the mentioned content:

and switching the router to operate in knowledge‑base mode (the beginning of the video shows the router running without a knowledge base, the second part shows it running with an attached knowledge base):

Starring: OpenWeb UI with the LLM‑Router attached (with the langchain_rag plugin disabled and enabled). We encourage you to download, try it out, and share your impressions of using it 🙂

Related Products: Knowledge-base context augmentation is natively supported via the plugin ecosystem of LLM Router. You can also explore live demonstrations on RDL Playground AI.

Is it possible to break the laws of nature?

2025-12-27T00:00:00Z

I don’t know, although I have a feeling […]

Hello everyone! Today we’re back after two months of intensive work on publishing the LLM Router. Here’s the earlier post: click.. A small project has turned into a sizable project, and its development—beyond just the coding side—has drawn in new contributors. In addition to Michała and Bartka, Krzysiek and Maciek. have joined the effort. We’re building a solid set of skills … 😉

In today’s post we explain how to configure the LLM Router using the models that are available under the Apache 2.0 license. A huge thank‑you goes to the Speakleash foundation, which, as part of its activities, has released two models that are perfect for use in the router. In this article we show how to plug in the models Bielik-11B-v2.3-Instruct “Bielik” and models Bielik-Guard-0.1B-v1.0 “Sójka” models so that they work together in the LLM Router. We also demonstrate an example of how 8 Bieliks and 5 Bielik-Guards can cooperate within a virtual ecosystem. 🙂

Eagles and jays in one nest

In nature the Eagle Bielik is a solitary predator that only pairs up during the breeding season. It does not allow foreign birds onto its territory. In the context of the jays the relationship is simple: predator – prey. However… that is “just” nature, and in the digital world we can do more.

Metaphor of the Natural Ecosystem

In the digital world the name eagles is taken by the language model Bielik-11B-v2.3-Instruct, while the jays is represented by Bielik-Guard-0.1B-v1.0, a guard‑rail type model. The nest is the LLM Router, which in its ecosystem imposes order and harmony, and the predator‑prey relationship disappears. On the contrary, these “species” start to form large flocks together, and unwanted wildlife is kept out.

In the conglomerate of Instruct and Guardrail models, supported by sensitive‑data masking techniques, we obtain an ideal solution for controlled use of generative language models. A locally‑run router lets you control the content sent to generative models. The “prey” from the natural world becomes the defender: the language model communicates with the application, while background masking mechanisms obscure sensitive data.

This conglomerate is exactly the solution described today in the LLM Router 😉

A few words about LLM Router

“The full description of LLM Router can be found on the home page llm-router.cloud and in the previous blog post.”

LLM Router is an on‑premises solution whose purpose is to control and optimise traffic to generative models. Unlike most routers, we do not optimise for API‑call cost; on the contrary, given a pool of providers (both local and cloud‑based) we aim to distribute the traffic so that response times are as short as possible.

During the whole process additional steps are performed, such as checking the outgoing content for ethical compliance and for the presence of sensitive data. Ethical correctness of the transmitted content is enforced by guard‑rail mechanisms: when an incident is detected the conversation/message is automatically blocked, an audit record is created, no payload is sent to the generative model, and the application receives a notification that the content has been blocked and the dialogue cannot continue.

Masking mechanisms run in the background and are responsible for obscuring sensitive information before it reaches the generative model. This mechanism does not block the message; instead it replaces the payload on‑the‑fly with a masked version, logs an audit entry for the detected incident, and forwards the masked message to the generative model.

All of this is configurable via the appropriate settings of the launched router.

Eagles and jays in LLM Router

To enable the aforementioned ecosystem to operate correctly, we must properly prepare this environment. For this purpose we need several components:

llm-router – the core traffic‑coordination ecosystem (a detailed description is available on gitgubie end www );
llm-router-services – services in which guardrail‑type models are run (example with Bielik-Guard );
model Bielik-11B-v2.3-Instruct – a Polish generative model;
model Bielik-Guard-0.1B-v1.0 – a guardrail model operating on Polish‑language texts;

Next, the individual components simply need to be assembled in one place – the LLM Router configuration. To keep this entry from becoming too long, we have uploaded all configurations to GitHub: [click], where we also placed a complete description of all steps. In short, however:

Start the serwice with guard (five “jays” simultaneously monitor the content);
Launch Bielik on vLLM (8 providers with the same model), on GPUs cuda:0 , cuda:1 , cuda:2 ;
Configure and start the LLM Router;

In the provided example we initiated the process of auditing and blocking prohibited content, as well as auditing and masking sensitive data using the built‑in fast_masker mechanism (description of the masking rules and their implementations in the router plugins).

Overview of the ecosystem

Below are screenshots of nvitopa running on each host, along with a view of the API console.

Host Lab2 with three “Bielik” models running (in vLLM)

Host Lab3 also has three “Bielik” (vLLM) models running.

Host Lab4 with two “Bielik” models (vLLM)

Host with router API (4 gunicorn workers, 16 threads each) and five “Sójki” (5 gunicorn workers on cuda:0)

The environment configured this way is ready to handle dozens of requests 🙂 For performance testing we prepared a simple application whose job is to translate a given text into Polish. For this we used the built‑in endpoint from llm‑router /api/translateand the dataset to translate, which is marmikpandya/mental-health available on Hugging Face. To run the test we used the run-text-translator.sh script from the utilsami repository; after installing the library the CLI commandtranslate-texts becomes available.

See: translate-texts –help

$ translate-texts --help
usage: translate-texts [-h] --llm-router-host LLM_ROUTER_HOST --model MODEL --dataset-path DATASET_PATH [--dataset-type {json,jsonl}] [--accept-field ACCEPT_FIELD] [--num-workers NUM_WORKERS] [--batch-size BATCH_SIZE]

options:
  -h, --help            show this help message and exit
  --llm-router-host LLM_ROUTER_HOST
                        Base URL of the LLM router service (e.g., http://localhost:port)
  --model MODEL         Model name to use for translation (e.g., speakleash/Bielik-11B-v2.3-Instruct)
  --dataset-path DATASET_PATH
                        Path to a dataset file. This option can be provided multiple times to process several files.
  --dataset-type {json,jsonl}
                        Explicit type of dataset files (json or jsonl). If omitted, the type is inferred from each file's extension.
  --accept-field ACCEPT_FIELD
                        Name of a field to retain from each record. Can be supplied multiple times; if omitted all fields are kept.
  --num-workers NUM_WORKERS
                        Number of worker threads for parallel translation (default: 1 – runs sequentially).
  --batch-size BATCH_SIZE
                        How many texts to send in a single request to the router (default: 8).

In the video below we demonstrate the router’s operation together with the models mentioned earlier.

First 0–3 seconds: a view of the API machine (top part) from the llm‑router, the lower‑left area shows the running service with the jays “Sójki”, and the lower‑right area displays an nvitop view of the GPU cores.
Seconds 3–6: a view of the Lab4 machine with two eagles Bielik instances (vLLMs running on the left side, and a GPU‑utilization monitor on the right).
Seconds 6–10: a view of the Lab3 machine with three eagles Bielik instances.
Seconds 11–13: a similar view showing the load on the Lab2 machine (three Bielik instances).
Subsequent seconds: switching between the machines, and the final segment returns to a view of the API machine.

During operation, the masking and prohibited‑content detection mechanism caught the cases it masked, blocked, and audited to a file (the logs contain entries like [AUDIT]************). The audited content is written to a dedicated directory (separate from the main logs) and encrypted with a GPG public key. The data can be decrypted only with the corresponding private key. In the video below we show how the logs (files with the .audit extension) are stored and how to decrypt them using the decryption script provided in the llm‑router repository (scripts/decrypt_auditor_logs.sh). Here is the result of running the command.

$ bash scripts/decrypt_auditor_logs.sh

with an additional password set on the decryption key

Outro

In this way order has been restored in the virtual world. The “Sójki” guard the content sent to generative models, the “Bieliki” generate the output, the masking mechanisms hide sensitive information, and the auditors continuously monitor everything for security.

We certainly encourage you to download and test the solution! Everything is available under the Apache 2.0 license. Happy masking! 😉

Llm-Router- a connector between an application and generative models

2025-10-24T00:00:00Z

This time, we’re introducing a project that we’ve been using internally for a while. Given its high versatility, it might be useful to others as well. We encourage you to try it out and report any issues you encounter 🙂

In the following part, we’ll briefly describe the contents of the repository, how to run the ready-made image (yes, just download and run :)), and an example configuration. In the README on Github, we covered the project from a technical angle; here, we’re focusing more on usage.

We’ve publicly shared the llm-router project on GitHub. This solution works like a router in the networking world, with one key difference: routing in llm-router happens between the user and generative models. Just as a router receives a message, processes it, knows where to forward it, and sends a response back to the user, our llm-router receives a query from the user specifying which model they’d like to use. It then routes the query to the appropriate model (it knows their locations), processes the model’s output, and returns the final response to the user. In short, the user doesn’t need to worry about where a model is running—they just need to know the model’s name and send the query to llm-router.

Supernova explosion

What can llm-router be used for? It’s a solution available via a REST API that manages traffic between an application and a generative model. It does this in such a way that the user/application doesn’t need to know the model’s location, connection parameters, and, more importantly, doesn’t store private keys for external generative model services (like OpenAI or Google). This is because all the model handling logic resides on the llm-router side. You just need to connect a list of external models to it, and the router will take care of the appropriate routing. It doesn’t matter whether the model is running locally, for instance using VLLM, LMStudio, or Ollama, or if it’s publicly available in the cloud.

Currently, all APIs that implement the Ollama or OpenAI standard are supported. Moreover, the router can handle traffic between different standards (including streaming). Even if a user has been using Ollama so far, they can stick with that standard and, without modifying their application code, use llm-router to send a query to an API like VLLM. This greatly simplifies deploying models in production.

Additionally, the predefined endpoints (click), which can be freely extended and new ones added, significantly streamline the software development process. The same functionality that would have to be coded into the application (and would only be available there) can now be implemented on the llm-router side, as a central access node. This simplifies introducing fixes, new features, and managing them. These dedicated endpoints can be thought of as agents performing specific functions.

Configuration flexibility also allows for naming models arbitrarily. For example, from the application’s perspective, the model it queries might be named model_generatywny, and this name is then resolved to the correct model and host by the llm-router (the recording below shows an example of how this can work – the application doesn’t know which model it’s running).

A box of building blocks

And inside the box, there are two smaller boxes: one with an assembled toy and the other with actual building blocks. Let’s start with the toy, which is a publicly available image on quay that can be downloaded and run without building. We only need to configure the models we have access to or are currently using. The process of downloading and running the image is very simple: just have Docker installed on your system and download the ready-made image from our Quay repository. To download the image, just run:

docker run -p 5555:8080 quay.io/radlab/llm-router:rc1

and update to the latest version:

docker pull quay.io/radlab/llm-router:rc1

After downloading, the image will run automatically and expose the llm-router API in its simplest Flask-based version on port 5555. The flag -p 5555:8080 means we’re forwarding port 8080 from inside the Docker container (where the router runs) to port 5555, accessible externally. The base server is Flask, so before scaling up (especially for streaming), we recommend reviewing configuration via environment variables and considering deployment with Gunicorn (with a set worker count, log level, etc.). To run llm-router with a custom config, simply mount it into the image to override the default config located at /srv/llm-router/resources/configs/models-config.json. Pass the -v parameter to Docker like this:

-v sciezka/do/lokalnego/pliku/models.json:/srv/llm-router/resources/configs/models-config.json

Now, the building blocks… The instructions are available in the README (along with a Polish description featuring an example advanced endpoint you can add to llm-router yourself), so we won’t repeat that here. However, the key takeaway is how the configuration file is structured. Below, in “Something for the eye,” we’ve included a sample JSON configuration for three models:

google/vllm : This is the google/gemma-3-12b-it model running on the local network ( http://192.168.100.79:7000/ ), launched via VLLM.
google/chmura : This is actually the gemini-2.5-flash-lite model accessible through Google’s cloud API https://generativelanguage.googleapis.com/v1beta/openai , for which you’ll need to insert your API key ( PUT YOUR API KEY HERE ).
openai/model : This is the gpt-oss:120b model running on the local network http://192.168.100.66:11434 , launched via Ollama.

Something for the Eyes

The video below contains several screens: the top left corner shows a view of VLLM with google/gemma3-12b-it, the bottom left corner shows Ollama with the gpt-oss:120b model, the bottom right corner is a console with the running llm-router, and the top right corner is the AI Assistant plugin for PyCharm. By default, the plugin allows connecting to one model provider, whereas connecting the llm-router allows injecting any standard, even one not natively supported by the application. Additionally, the video shows how you can change model names so that the application side doesn’t know which model is underneath. Meanwhile, in the console at the bottom, you can see how the llm-router receives requests and routes them to the appropriate host:

The configuration used to start Llm-Router:

{
  "google_models": {
    "google/vllm": {
      "api_host": "http://192.168.100.79:7000/",
      "api_token": "",
      "api_type": "vllm",
      "input_size": 4096,
      "model_path": "google/gemma-3-12b-it"
    },
    "google/chmura": {
      "api_host": "https://generativelanguage.googleapis.com/v1beta/openai/",
      "api_token": "PUT YOUR API KEY HERE",
      "api_type": "openai",
      "input_size": 512000,
      "model_path": "gemini-2.5-flash-lite"
    }
  },
  "openai_models": {
    "openai/model": {
      "api_host": "http://192.168.100.66:11434",
      "api_token": "",
      "api_type": "ollama",
      "input_size": 256000,
      "model_path": "gpt-oss:120b"
    }
  },
  "active_models": {
    "google_models": [
      "google/vllm",
      "google/chmura"
    ],
    "openai_models": [
      "openai/model"
    ]
  }
}

And the full script with Docker startup (what is run in the terminal in the video — i.e., full configuration and change of all default parameters :)):

#!/bin/bash

PWD=$(pwd)

docker run \
  -p 5555:8080 \
  -e LLM_ROUTER_TIMEOUT=500 \
  -e LLM_ROUTER_IN_DEBUG=1 \
  -e LLM_ROUTER_MINIMUM=1 \
  -e LLM_ROUTER_EP_PREFIX="/api" \
  -e LLM_ROUTER_SERVER_TYPE=gunicorn \
  -e LLM_ROUTER_SERVER_PORT=8080 \
  -e LLM_ROUTER_SERVER_WORKERS_COUNT=4 \
  -e LLM_ROUTER_DEFAULT_EP_LANGUAGE="pl" \
  -e LLM_ROUTER_LOG_FILENAME="llm-proxy-rest.log" \
  -e LLM_ROUTER_EXTERNAL_TIMEOUT=300 \
  -e LLM_ROUTER_MODELS_CONFIG=/srv/cfg.json \
  -e LLM_ROUTER_PROMPTS_DIR="/srv/prompts" \
  -v "${PWD}/resources/configs/models-config-names.json":/srv/cfg.json \
  -v "${PWD}/resources/prompts":/srv/prompts \
  quay.io/radlab/llm-router:rc1

Logout

One of the places where Llm-Router is running is our playground.

Every generative model functionality is programmed as a dedicated endpoint in llm-router. We’re sharing the code under the Apache 2.0 license—so you can use it commercially and non-commercially. Enjoy! 😊

We encourage you to try it out and share your suggestions and feedback.

Update (v1.1+ Release): The planned features (6 advanced Load Balancing strategies, Redis-backed sliding-window Rate Limiting, Prometheus metrics, and the PII / NASK & Sójka guardrail plugin pipeline) are now fully available! Explore the full LLM Router product page.

We make the codes publicly available – open source.

2025-09-28T00:00:00Z

Hello! Today we will describe the technical aspects. Over the past month, we have been working on publishing some of our solutions. Today we would like to present a few of them. Some of them are working mechanisms on the playground, others are ideas implemented “on the side.” We have returned to the idea of developing solutions completely publicly on GitHub. Therefore, we invite you to visit our GitHub profile, which we briefly present in this post 😉

Below are the projects with brief descriptions. Each of them has a fairly extensive README file on Github.

relgat – projector (“from the side”)

The repository (clic) contains a proprietary implementation of link-prediction mechanisms based on semantic graphs. The idea was to recreate semantic relationships and enable the creation of new representations based on existing ones. We assume that nodes in the graph have a semantic description (e.g., using embedding), while edges have no description but are distinguishable from each other. The edge label in such a graph denotes the name of the relation.

The method enables learning transformation matrices for edges in the graph based on the node environment. The model simultaneously learns the transformations of each relation separately and learns to define nodes based on those existing in the graph. A node is defined by its neighborhood (input); we assume that what enters a given node defines it. Unlike standard RelGAT, we do not learn the entire network—we learn transformations of the base space to the same space in such a way that for [A -> rel1 -> B], [C -> rel2 -> B] we look for transformations rel1 i rel2 o that the combination of embeddings A i C gives embedding B. By additionally using evaluation by metrics such as hits@X as loss functions, we additionally learn link prediction. This will make it possible to transform any vector X from the learned space) using a transformation matrix M (e.g., by activating specific relations) in such a way as to obtain vector X‘ in the same space, but after transformation by relations.

We have already completed a series of experiments (the charts show the latest stable training sessions) and so far (according to the charts) it looks promising 😉

plwordnet (“from the side”)

This repository (clic) can be used to operate on the Polish and English Wordnet (Wordnet – Polish, Princeton Wordnet – English). Although it is not currently used only for browsing wordnet’s, it is primarily a mechanism for creating data sets. A method for creating embeddings for semantic graph nodes is implemented there. It contains the entire process from downloading Wordnet, preparing data for further processing, to: creating

a semantic embedder,
embedding representations for the meanings of PLWordnet and Princeton Wordnet using this embedder.

On the huggingface platform we have published the first version of the model (radlab/semantic-euro-bert-encoder-v1). It is a bilingual embedder model in which we used EuroBERT as the language model. Creating embedding representations is a multi-step process. First, representations are created for lexical units. Then, for those units for which no representation could be created but which are found in a synset that contains some representations, a (fake) representation is created for the unit. Then, for each synset that has at least one representation of a unit, embedding representations are created as weighted averages of the embedding’s of the units. After this operation, not all units and synsets have definitions, which is why the relgat-projector is to be used, among other things, to transform a node without a representation but with some environment (synset or unit).

ml-utils (“from the side”)

Earlier libraries are based on ml-utils. This is a library that simplifies the processing process in the context of machine learning. It contains general classes and methods that are independent of the project, but their functionality can be shared with other projects. Thanks to ml-utils, you can, among other things:

Easily manage the logging of results to Weights & Biases,
manage prompts for generative models using directory-based names,
support any OpenAPI-compliant generative model API, create queues, and cache responses to reduce the number of model queries.

Additionally, general-purpose modules are available, such as env parsing and creating unified loggers, as well as a module for downloading data from Wikipedia, which is used in plwordnet to enrich the context of built embeddings.

Modules from the playground

In the clusterer repository, we’ve included a module responsible for creating clusters on the Playground. The types of information visible in the Information Browser or Information Explorer are created using this module. A more detailed description of the method is available on the blog in the linked articles. We’ve also released the code for the graph visualizer, currently running at https://graph.playground.radlab.dev/, which presents continuous and discontinuous information graphs. We’ve also included the radlab-playground-ui repository with the Playground interface code as a Streamlite application.

Summary

All code is available under the Apache 2.0 open source license and is free for commercial and non-commercial use. End. 😉

Information Explorer- not a hammer, but a scalpel

2025-09-17T00:00:00Z

Today’s blog post will be short, but the attachment is quite long 🙂

Intro i Outro

We encourage you to take a look behind the scenes at how the Information Browser and Information Explorer work 😉 For this occasion, we have prepared a technical report with a detailed description of how our solutions work. In the report, we present precise definitions of information and its types, i.e., daily, continuous, and discontinuous information. We have introduced definitions of information graphs for information and presented the process of their construction, analysis, and extraction of elements that make up the information browser and information explorer. The data that feeds these methods is, of course, news from the News stream. We have also launched a public GitHuba, where we have posted both the pdf and LaTeX codes of the report.

Combo

When reading, we recommend reading the entire document from start to finish, without missing any details. The report shows how to extract information from the data written between the lines. Here is a preview of the PDF file:

Pobierz PDF

Eksplorator_Informacji-Raport-techniczny-v1.0-20250714

Pobierz

Related Products: Graph-based entity discovery and clustering algorithms are deployed in Radar Informacji and the analytics modules of RDL Playground AI.

3c Polarization – model from plg on HF

2025-09-26T00:00:00Z

Hello 😉 Today, we are presenting and publishing an element of the playground’s. It is a classification model whose task is to determine the polarity of texts. We are publishing the model on our huggingface 😉 We define polarities as:

positive — the model assigns this class if the information in the texts can evoke positive emotions/feelings;
negative — if the information in the texts evokes negative feelings;
ambivalent/neutral — if the information is neutral or evokes conflicting feelings at the same time;

Process: data and learning

The model was developed in two stages. However, unlike the standard approach (starting with annotation), we began by training the model based on existing annotations (hm… that’s nothing new… yes and no…). However, when looking for a dataset for polarization, most often these are datasets related to opinions on a given topic. In our case, it is not about determining the polarization of opinions (e.g., This product can be thrown in the trash! or I recommend this doctor because he has a very good approach to patients!), but about determining the polarization of information that the media (websites) feed us.

So we had to look elsewhere 😉 The choice was not so obvious, but it reflected the idea of polarization of information rather than opinion… and we settled on PlWordNet Emo. PlWordNet Emo is a selected part of PlWordNet’s marked with emotions (and more) (the markings are at the level of word meanings) — I recommend familiarizing yourself with what EmoPlWordNet is — it is a very valuable source of information. After a few technical steps, we transformed the examples of usage into meanings with emotional descriptions, and then into an approximate set of emotional polarization (reduction of emotions and granulation to 3 – positive, negative, ambivalent). Below are a few examples from the set, after conversion:

We used this collection to train the polarity3c-zero model, which was immediately used in the decision support process during annotation. The model provided real-time annotation suggestions for two annotators. This resulted in a collection of approximately 3,500 manual annotations, which were used for the final training of the model.

The final model is a simple architecture in which a simple classification layer (ClassificationHead) is added above the language model, which is trained to determine polarity. Classification layer architecture:

  (classifier): RobertaClassificationHead(
    (dense): Linear(in_features=1024, out_features=1024, bias=True)
    (dropout): Dropout(p=0.1, inplace=False)
    (out_proj): Linear(in_features=1024, out_features=3, bias=True)
  )

1024 on the first dense layer of Linear is the output of the base model polish-roberta-large-v2, the output of the polarity3c model, is the layer marked (out_proj), which has 3 features at the output, i.e., our detected classes. The charts below show the basic metrics from training this model:

Launch and testing

The easiest way to run the model is to use the transformers library with a pipeline’a.

from transformers import pipeline

classifier = pipeline(model="radlab/polarity-3c", task="text-classification")

The use of the model is equally simple:

 classifier("Po upadku reżimu Asada w Syrii, mieszkańcy, borykający się z ubóstwem, zaczęli tłumnie poszukiwać skarbów, zachęceni legendami o zakopanych bogactwach i dostępnością wykrywaczy metali, które stały się popularnym towarem. Mimo, że działalność ta jest nielegalna, rząd przymyka oko, a sprzedawcy oferują urządzenia nawet dla dzieci. Poszukiwacze skupiają się na obszarach historycznych, wierząc w legendy o skarbach ukrytych przez starożytne cywilizacje i wojska osmańskie, choć eksperci ostrzegają przed fałszywymi monetami i kradzieżą artefaktów z muzeów.")

In response, we will receive the value:

[{'label': 'ambivalent', 'score': 0.9994786381721497}]

To read the full confidence distribution of the model, simply add the option specifying how many labels with the highest probability you want to receive. In our case, we have 3 labels, so we add the option top_k=3 to theclassifier to receive information about all classes:

 classifier("Po upadku reżimu Asada w Syrii, mieszkańcy, borykający się z ubóstwem, zaczęli tłumnie poszukiwać skarbów, zachęceni legendami o zakopanych bogactwach i dostępnością wykrywaczy metali, które stały się popularnym towarem. Mimo, że działalność ta jest nielegalna, rząd przymyka oko, a sprzedawcy oferują urządzenia nawet dla dzieci. Poszukiwacze skupiają się na obszarach historycznych, wierząc w legendy o skarbach ukrytych przez starożytne cywilizacje i wojska osmańskie, choć eksperci ostrzegają przed fałszywymi monetami i kradzieżą artefaktów z muzeów.", top_k=3)

And here’s the way out:

[{'label': 'ambivalent', 'score': 0.9994786381721497},
 {'label': 'negative', 'score': 0.0002675618161447346},
 {'label': 'positive', 'score': 0.0002538080152589828}]

Outro

The model has been available on our playground’s since July last year. We collect data on its performance, which can be viewed in the statistics. Information about polarization is also added to each new stream using this model. The model is, of course, available for free on our HF: here is the model.

Related solutions: This model powers sentiment and polarity analytics in Radar Informacji and the live news stream on RDL Playground AI.

Unlock the potential of your data with our ML/NLP. Bon appetit! 😉

Information Browser

2025-09-26T00:00:00Z

From a marketing perspective, it would be good to start this post with something like:

Today, we present a revolutionary, fully automated solution that will allow you to analyze media trends in real time. Want to find out what people are writing about? What’s trending? Yes! You’ve come to the right place. Today is your chance to take advantage of this opportunity….

Easy…. It’s a bit like Telemango these days… it’s not appropriate 😉 So we’ll start as usual.

What is today’s post about?

We have released a new feature on the playground called Information Browser, and it does exactly what its name suggests—it allows you to browse information over time.

What is information? We understand information to be a concise thematic group of news items from websites that implicitly relate to the same situation/topic/event… What appears in the news, what the media write about, what is trending. It is what people read about on a given day/time period, only collected in one place, summarized, and named. The name of the information is not predefined in any way; it is determined on the basis of emerging news items grouped into compact information group.. The name of the information may be different every day—it depends on what is being written (and read ;-)).

The Browser operates on the basis of information found on popular news websites, which we analyze on a regular basis and present the results of these analyses in the playground. The Browser presents the process of automatic analysis of information from these websites. It is a mechanism that presents the most important information from a given day, names this information, and writes a short summary in which you can learn in brief what the detected information is about. In addition, it will present a number of related (let’s call them) triggers for this information:

will show the exact addresses of the websites that publish this information;
will show the sources of information dissemination using a diagram visualizing the percentage share of a specific portal in sharing this information;
will present an analysis of the polarization of this information (it is worth noting the correlation between the name of the information and the polarization of the information);
use a calendar to enable tracking and analysis of information over time (although it may have a different name);

How does it work?

From a technical point of view, it is a mix of different methods and optimizations. The mechanism examines the differences between texts and, based on their similarity, selects the news items that are most similar to each other from the perspective of reduced embedding for one information group. The mechanism for separating information clusters adjusts its operation to the amount of data it analyzes. There are days when there are about 600 news items, but there are also days when there are twice as many. Based on multiple distributions, the mechanism selects the one that best fits the additional conditions for data division, e.g., it does not allow divisions below 5 information groups or above 60, and additionally takes into account the golden mean, i.e., targeting 20 topics per day, while optimizing the number of rejected examples. From the perspective of the two months reviewed, the media most often publishes between 20 and 30 pieces of information per day.

For whom: “as different as chalk and cheese”

You could say: For everyone… From the perspective of someone looking for specific information, this solution allows them to access dozens or hundreds of news items from a specific day with specific information in one place. If someone is interested in sporting events and there is a lot of information about sporting events on a given day, they will receive a summary of many events (including those from foreign sources). For example, suggestion number 1 (number 0 appears by default) on May 26, 2025, shows the information “Sports – successes and changes.”, whose summary looks like this (click to enlarge):

In addition, we receive addresses of related pages with information containing this description:

Kliknij aby zobaczyć zdjęcie

From an analyst’s perspective, linking information about sources of propagation, i.e., the graph:

The graph shows where the information comes from and how much it is dominated by a single data source. And while in the case of sports, this may be a less important variable, in information related to, for example, politics or modern technologies, the distribution of the origin of the pages may show the process of introducing information into circulation. Analyzing this information over time using a calendar allows you to track it and draw conclusions. It is important to remember that information is named individually on a daily basis, so names may vary depending on the day. Therefore, human interaction is (still) required to track specific information over time 😉 In addition, we receive a summary of the distribution of emotional polarization in the form of a diagram:

When analyzing such a diagram, it is worth paying attention to the correlation between the name of the information and the polarization/tone of the text (positive, negative, ambivalent-neutral information). The name of the category clearly indicates what the polarization distribution may look like. Our model, which has been operating on the playground for a very long time and is used to evaluate texts in real time, is responsible for detecting polarization. Based on the decisions of this model, the share of a specific tone in a particular piece of information is presented (this is not the tone of the summary article, but of the news items containing the information). In addition, we obtain accurate data on the number of news items:

And linking the number of news items in the information with the number of news items throughout the day (and this variable is accessible):

allows you to determine the relevance of information in the context of the entire day…. The rest can be left to the imagination 😉

Outro

Why only one day? No problem, the mechanism works based on time intervals, analyzing information from the specified interval, not for the specified day. The mechanism itself is universal, and for analyses covering more than a week, it requires slightly more powerful hardware (and, surprisingly, it is RAM and a processor, not a GPU :)), the analysis of about a thousand news items from a given day takes 15 to 28 minutes, so not that long. We are currently analyzing days up to January 1, 2025. After this process, it will be possible to track information from January 1, 2025, to the day on which it will be tracked. The mechanism works on the basis of automated one-day analyses, which is why a summary of the previous day’s information appears every day. After the one-day analysis, we plan to introduce weekly analyses.

We encourage you to browse the Information browser, analyze it, and draw conclusions. The browser is, of course, entirely non-profit, without ads or logging in 😉

Interesting fact

Content Supervisor about Browser [click]

Summary of this post using the content supervisor from Public Chat 😉 Here’s what gemma wrote about the Browser:

Conversation available to read in Public Chat, just load the hash:

text 7ivUpPCTNPR0d8gGbPT9hh5nD1Y8yMapJSOISOTg7mFJvryf3BgzGdT03Bsf0Ui3gWymAkciIWRxPJnHslFwnP0FtYzrSi8xj7AOqxT5lcqQ1waaYadNUIOoOJAqu9wP

Explore related solutions: Learn about the complete 8-stage algorithmic pipeline and architecture in the Radar Informacji product overview and try the live demo on RDL Playground AI.

Unlock the potential of your data with our ML/NLP. Bon appetit! 😉

pLLama3.1 8B — medium-sized and even small GenAI for Polish

2025-10-24T00:00:00Z

Yes… it was intentional to misspell not-in-or-de-r (it’s not a mistake) as a preview of the next version of the model 🙂 Between pLlamą3 and pLlamą3.2, there was also a version called pLLama3.1 — and that’s what today’s post is about.

pLLama – generated using AI

We recently wrote:

Intro

Another version of Meta’s model that speaks better Polish! In this post, we present the pLLama3.1 model in 8B architecture. This time, we have published the –content model, which is a less talkative version of the model, the –chat model as a more talkative version, the pure –base model after LoR, but we have also made available an experimental model with the transfer of adaptive layers from the pLLama3 model to the pLLama3.1 model (more on that in a future post :)). All models are of course available on huggingface (full collection of 3.1 models):

Model radlab/pLLama3.1-8B-content : this is an SFT model, and DPO provides short and concise answers.
Model radlab/pLLama3.1-8B-chat model is a more talkative version of the model (after SFT and DPO), ideal for chatting.
Model radlab/pLLama3.1-8B-base-ft-16bit is a model directly after SFT with LoRa.
Eksperimental model radlab/pLLama-L31-adapters-MIX-SFT-DPO with transfer of adaptive layers between models.

Dataset, learning, and graphs

All versions of the pLLama3.1 models have one parent, namely Llama-3.1-8B-Instruct from Meta.AI in the 8B architecture. This model already writes quite well in its basic version in Polish, but it still has a long way to go before it can write (at least) correctly. Hence the idea: let’s train this model too 🙂

For training, we used a dataset developed for earlier models. To briefly recap, this is over 650k instructions generated automatically based on available datasets. We converted each set (e.g., marked with parts of speech) into system prompts in the form of questions and answers. This resulted in a set of instructions that we used as data to train the model. We also developed a set of instructions for DPO based on available language corpora, including NKJP and KPWr.

Due to hardware limitations, we performed SFT using adaptive layers (LoRA) with 16-bit quantization (bits and bytes). Similarly, in DPO, the base model was loaded as a quantized version of pLLama3.1 after SFT, and the adaptive layer was also retrained during DPO. As a reminder, in our case, during SFT, we skew the model weights towards the Polish language, while in the DPO process, we teach the model to generate text correctly in terms of grammar and punctuation. We trained the models on single RTX 4090 graphics cards (24 GB). SFT (LoRA) lasted 14d 2h 3m and 52s. DPO lasted 2d 15h 2m and 42s. The difference between the –content and –chat models is a matter of a different way of combining the adaptation layer with the model (but this will be discussed in a separate post). Below are two graphs, the first showing the loss function value in SFT, and the second showing the chosen and rejected logit values during DPO.

Loss function during SFT with Lora:

In the DPO process:

Outro

How does version 3.1 differ from the pLLama3 model? First and foremost, in terms of context size. The pLLama3 model had a limited context of 8k tokens. The pLLama3.1 version has a context length of 128k tokens (this results from the base model, i.e., in pLLama 3, we trained Llama3 from Meta, while in pLLama3.1, we trained the LLama3.1 model from Meta). In RAG tests, pLLama3.1 performs significantly better than version 3, mainly due to its larger context. At the moment, when answering a question, the model is able to accommodate much more information, which significantly reduces the frequency of its querying, greatly reduces the problem of too frequent querying of the model, dividing the data to the size of the input (and, incidentally, reduces the problem of compiling the final answer in RAG based on partial answers with a small context).

And as an addition to HuggingFace, we have a model that we call L31 (pLLama-L31-adapters-MIX-SFT-DPO) as an experiment in mixing models.

That’s all for today, thanks!

Unlock the potential of your data with our ML/NLP. Bon appetite! 😉

pLlama3.2 (1B + 3B) – small GenAI for Polish

2025-10-24T00:00:00Z

pLLama – generated using AI

Intro

Today a very short post. A post about a model, small by today’s standards. You’ve probably heard about Llama 3.2 from Meta AI? Meta recently released models for content generation (Instruct) and images (Vision-Instruct).

Unfortunately, the Vision-Instruct models are not available in the European Union, and by extension, in our country.

So… what are we left with? We are left with teaching small text models in 1B and 3B architecture in Polish 😉

Data and Training

For further training, we first used the fine-tuning technique, and then in the DPO process, we trained both models for language correction. The data was exactly the same as for pLLama3 (Click to read the article). This applies to both the fine-tuning and DPO. The only change was a small adjustment in the learning hyperparameters (batch size, learning rate). The number of epochs in FT was 5, and the number of steps in DPO was 50k. And… oh yes, training in 16-bit 🙂

See for yourself the loss function chart for both the training and evaluation parts (it’s the same fine-tuning process for the Polish language):

And that’s exactly the difference between them 🙂

Feelings

Since the 1B and 3B models are distilled versions of existing models, likely trained mostly on English data, I was curious about the result of fine-tuning and DPO training for these models. The 1B model (after both FT and DPO) is, without a doubt, the least capable model in our collection—but Meta’s own benchmarks show exactly the same. As for the 3B model, it’s a completely different tier than the 1B. One could say it performs as expected.

Comparing them to larger models somewhat misses the point, because with the 1B and 3B, Meta focused on making them runnable on less powerful hardware, even on a smartphone 😉. While the 1B model struggles, the 3B model can be surprisingly effective.

Models for Download

We invite you to our HuggingFace page, where we have created a collection with the pLLama3.2 Models. All models are, of course, publicly available for free:

1B architecture models: radlab/pLLama3.2-1B – the model after fine-tuning only, and radlab/pLLama3.2-1B-DPO after FT + DPO
3B architecture models: radlab/pLLama3.2-3B the model after fine-tuning only, and radlab/pLLama3.2-3B-DPO after FT + DPO

Bon appetit 🙂

What next?

Next will be not-in-or-de-r: pLLama3.1-8B (the green graph) and L31 as an interesting mix…

pLlama3 (8B + 70B) – GenAI for Polish

2025-10-24T00:00:00Z

pLLama – generated using AI

Intro

In the last post, we mentioned the GenaAI model… Yes! Today we want to introduce you to our new model, or rather a family of models further trained on the Polish language: pLLama3-8B-chat, pLLama3-8B-creator, and pLLama3-70B. This time, we decided to further train models from the giants to make them handle language Polish.

First up were Llama-3-8B-Instruct and Llama-3-70B-Instruct from Meta.ai. This is an instruction model that can carry out commands given in the model’s prompt. The original model does not handle Polish very well. Sometimes it can write something in Polish, but it is difficult to force it to communicate fluently in language Polish. Interestingly, it can surprisingly well understand Polish commands, but it cannot construct a correct response.

We are making the 8B architecture models available in two configurations:

radlab/pLLama3-8B-creator , a model that gives fairly short, specific answers to user queries;
radlab/pLLama3-8B-chat – a model that is a talkative version, reflecting the behavior of the original meta-llama/Meta-Llama-3-8B-Instruct model..

Dataset

The fact that the original model could understand Polish commands was the spark that ignited our work on the model. We hypothesized that it would suffice to appropriately fine-tune the language layer while interfering as little as possible with the instruction layer.

However, as is often the case with large language models, the first problem arose: where to obtain data (especially available as instruction/chat conversation sets!) that would be sufficiently abundant and of the highest possible quality? For the Polish language, there is actually only one publicly available dataset for training an instruction model: alpaca-dolly-chrisociepa-instruction-only-polish. However, the size of this dataset did not allow for sufficient fine-tuning of the model.

Therefore, we developed a method to create a dataset structured as instructions, using other datasets available online. We prepared a set of approximately 650k Polish instructions that we fed to the model during training.

Additionally, we developed a training set for the DPO process, which contained 100k examples where we taught the model to select correctly written versions of texts over those containing language errors. The DPO training examples included a correctly written linguistic form (chosen) and an incorrectly written form (rejected) of the same text. It is worth noting that the DPO data was built in reverse – namely, for very high-quality texts (chosen), we prepared their corrupted counterpart (rejected).

Training Process

The training process was divided into two stages:

Fine-tuning (FT) on a set of 650k Polish-language instructions, with the fine-tuning duration set to 5 epochs.
After the FT stage, we further trained the model using DPO on 100k instructions for correct Polish writing; in this case, we set the training duration to 15k steps.

Fine-tuning the model on 650k was aimed at shifting the language layer of the fine-tuned model toward the Polish language. Unfortunately, due to the imperfect instruction set

Fun fact: the instruction set was generated ~95% automatically — using an early version of the trained model (in the 70B architecture) iteratively to assist in creating the set that we used to train this target model…

the model started actually speaking Polish and responding to instructions, but it sometimes made quite a lot of typos. That’s why we decided to further train the model with DPO. In DPO, we primarily wanted to achieve an improvement in the model’s language layer compared to the FT model. The optimization involved teaching the model to select the correct linguistic form and reject the incorrect one.

The graph below shows the loss function during the entire training process (8B model). The sudden drop in value at the beginning of the function may indicate a fairly quick adaptation of the model to the Polish language. And the entire, long-lasting training process is, de facto, refining the already existing linguistic details.

We trained the 8B models for 5 epochs, and the entire training process lasted over 19 days and 14 hours on a single RTX 4090 card.

Detailed metrics after completed fine-tuning (8B model):

Metric	Value
eval/loss	0.8690009713172913
eval/runtime	464.5158
eval/samples_per_second	8.611
total_flos	46121863674517000000
train_loss	0.8724352801788304
train_runtime	1695168.0431
train_samples_per_second	1.758
train/epoch	5
train/grad_norm	0.17023593187332153
train/learning_rate	6.584723441615452e-8
train/loss	0.8293

Metrics after the training process (fine-tuning)

And the DPO metrics are as follows:

eval/logits/chosen: 0.1370937079191208
eval/logits/rejected: 0.07430506497621536
eval/logps/chosen: -454.11962890625
eval/logps/rejected :-764.1261596679688
eval/loss: 0.05717926099896431
eval/rewards/accuracies: 0.9372459053993224
eval/rewards/chosen: -26.75682830810547
eval/rewards/margins: 32.37759780883789
eval/rewards/rejected: -59.134429931640625
eval/runtime: 1,386.3177
eval/samples_per_second: 2.838
eval/steps_per_second: 1.42

The graph below shows the loss function during the entire training process of the model in the 70B architecture.

We also trained the 70B model for 5 epochs, and the entire training process lasted over 1 month and 5 days on a single RTX A6000 ADA 48GB card, while the metrics after completed fine-tuning:

eval/loss:0.7297297716140747
eval/runtime:6,364.4589
eval/samples_per_second:0.628
eval/steps_per_second:0.628

What after training?

Our models have one very interesting feature. At the moment, they can read text in any language and respond in Polish. That’s why one of them, which we named pLLama3-8B-creator , we employed to write articles for a certain project, which we’ll tell you about soon. Another one, pLLama3-8B-chat, also has its part in it 🙂 Enjoy!

Huggingface

Of course! We make the model available for free on our Hugging Face. The individual models:

radlab/pLLama3-8B-chat at: https://huggingface.co/radlab/pLLama3-8B-chat
radlab/pLLama3-8B-creator available: https://huggingface.co/radlab/pLLama3-8B-creator
radlab/pLLama3-70B available: https://huggingface.co/radlab/pLLama3-70B

Bon appetit and welcome! 🙂

RAG (problems?)

2025-09-26T00:00:00Z

Today’s post will be a little different than usual. This time, we are not presenting a new model, but a description of the RAG method. The method is widely known, so this post will focus more on the problems that arise when implementing systems that use RAG than on the description of the method itself.

To explain RAG in a few words, it is an information retrieval method combined (in the results presentation layer) with a generative model/instruction model. The search mechanism and the reordering of search results are responsible for finding text fragments that contain the answer to the user’s question. The text fragments are then applied to the generative model in such a way that the model can use them to answer the user’s question. And that’s it in a nutshell 😉

So what can RAG be used for? Since it is a search method, it can obviously be used for all kinds of search engines. However, an important aspect that distinguishes RAG from the standard method of searching document collections is the introduction of a generative model (GenAI) to provide the final answer to the user. Thanks to the accuracy of the search engine itself and the ability of the GenAI model to answer questions based on context, RAG is ideal for searching and summarizing information scattered across multiple documents. Additionally, thanks to GenAI and its purpose of conducting chat conversations, we have the ability to influence the final form of the answer when asking a question. For example, imagine that we have a collection of documents in which we gather texts with different user opinions about various products. In this case, RAG would be ideal for answering the question:

What are the positive and negative opinions about product XYZ? Give three examples of positive and negative opinions and suggest responses to customer comments.

As you can see in the example, you can provide RAG not only with a search term, but also with detailed instructions on how to respond.

In fig. 1, we have broken down the RAG process into smaller and more specific steps. There are six main modules that symbolize issues throughout the RAG pipeline. The minimum path in such a pipeline is marked with a red dotted line and a red arrow. The green line and green arrow show the optimal path in the pipeline, which can be called RAG. In this post, we have briefly described the individual steps, and their detailed elaborations will be published in subsequent posts.

Fig. 1 RAG components (own work)

RAG in pieces

Data

Data is a fundamental element of any system that uses machine learning (ML) methods. Depending on the problem that such an ML system solves, it must be provided with appropriate data. In the case of RAG- data refers to all kinds of texts in which answers can be found. It may therefore seem that nowadays, when you can find everything on the internet, a lack of data should not be a problem… This is partly true, but for the most part it is not 😉

It is the abundance of data that introduces the problem of its location, availability under the appropriate license, and its quality, however understood. The selection of the origin of information from specific sources, the quality of the data itself, its readability, and detail are very important elements of the whole puzzle. Add to this the various file formats in which the information is stored (including pdf), and we are faced with the task of integrating different sources into a common format. It is important to remember that if we enter data into the system that contains false information, or contains too little information, or is not detailed enough, then a system using the RAG method will provide answers based on this data, which may be incorrect.

It is worth emphasizing once again that the final response of the system is generated on the basis of the data provided by the search engine. Therefore, if the search engine contains unreliable data, false and inaccurate information, then we will receive such responses. That is why it is worth taking the time to carefully select the set/create the collection of documents on which we will be working, and to take care of data preprocessing.

It is therefore safe to say that the quality of responses from such a system is strongly determined by the quality of the data used by that system.

Data Preprocessing

A very important element of the RAG system is data preprocessing, i.e., its initial processing. Before the data enters the further process, it is worth introducing a certain standard for its quality. The aforementioned diversity of data records causes problems with reading them correctly. Automatic processing of data downloaded from websites, docx files, and pdf introduces a lot of noise into the data. From the perspective of the RAG system, it is important that the data contains relevant information. When data is loaded automatically, for example from a pdf file, we will read not only the relevant fragments, but also a whole lot of text, including chapter headings, fragmented tabular data, or worse — completely unreadable data, such as this (e.g., when there are images in the pdf file that are also difficult to OCR):

4kV�#4T��##�_#�p��W#�_�|3��-�tg�\S�x�b�X��t�]*���\&/#��/�P�

Of course, it is difficult to do anything about this situation, but it is worth detecting such files and not allowing the data to pass further, already at the pre-processing stage. In our previous post , we presented another problem with pdf files. After reading them, we get heavily distorted data, to quote an example from the post:

Strona...Tytuł...%6%z%50%|.....Zdecydowana0większość0czerwonych.karłów$należy#do,typu,widmowego%M,%ale%zalicza^się^do^nich^także^wieleHgwiazdHpóźnychHpodtypówHtypuHwidmowegoHKHorazHrzadkoHwystępujące,,najsłabsze,gwiazdy,typu,L.,Maksimum%intensywności%emitowanego%światła%przypada%w%zakresie%światła%czerwonego%lub%nawet%bliskiej%podczerwieni.

From the perspective of further processing without proper cleaning of such data, this only causes problems. Indexing text that contains such illegible data will make it equally illegible for the search methods themselves. At this stage, it is important to be able to assess the data in terms of its quality and, as far as possible, bring it to a form close to ideal 😉 Is it worth introducing ML models for assessment and improvement at this stage?– Of course yes! Here is an example of how our T5 model works, cleaning texts from such situations:

Zdecydowana większość czerwonych karłów należy do typu widmowego M, ale zalicza się do nich także wiele gwiazd późnych podtypów typu widmowego K oraz rzadko występujące, najsłabsze gwiazdy typu L. Maksimum intensywności emitowanego światła przypada w zakresie światła czerwonego lub nawet bliskiej podczerwieni.

Maybe not perfect, but this form is good 😉

What to do if your documents contain a lot of tables, drawings, or charts? It is worth considering using models that can interpret them and generate a description that can then be used to enrich the original text.

(…) It is also important to identify good and bad quality data and standardize it for further processing.

Data indexing

This is a very important process in the entire RAG. After preliminary data processing, the data must be saved in a database, where it will then be searched. At this stage, it is worth introducing parallel text indexing in both relational and vector databases. The relational database will help to arrange a longer document in its original structure, while the vector database will enable semantic searching. A very important element in the indexing process is the method of indexing. It is at this stage that we decide which information from the previous step will be indexed, and where and how.

First of all, we need to consider the purpose of the RAG system (specific field, specialist agent, general information) and select an appropriate model (embedder) that will “understand” the indexed content well enough. It is the embedder that creates vector representations of texts that are indexed in vector databases (e.g., Milvus), which are then searched. If, at this stage, the transformation of text into a vector is inaccurate, in other words, if we choose the wrong embedder model for the content we are indexing, this may result in poorer/less accurate quality in the comparison process. An indexing error can therefore be propagated to further processing steps. It is also important to choose the appropriate length of texts that are indexed as a single fragment. If the text is too long, it is worth dividing it into smaller parts called chunks — that is, one text is divided into several smaller fragments.

What if we index highly specific content for which no embedders are available? Of course, we are faced with the problem of finding another, more suitable model or training such a model from scratch. In one of our previous posts, we presented our preliminary embedder model, which was trained on web content.

(…) It is also important to select or train a specialist embedder and ensure appropriate indexing in databases.

Search

One could say that things are somewhat simplified here, because at the search stage we use the same mechanisms to translate text into vector representation as at the indexing stage… Nothing could be further from the truth. This is only the case when we limit the search process to counting the similarity of vectors representing the query to chunks in the database. Most often, the cosine of the angle between the query vector and the vectors stored in the vector database is calculated. These are the vectors that we stored during the data indexing stage. However, as we mentioned earlier, it depends on what information we index, and where and how.

If, at the indexing stage, we enrich the information about the indexed fragments with additional metadata, it is at this step that we are able to preliminarily filter out those chunks that are 100% incorrect. For example, if we enrich the description of a chunk with its category, then at this stage, before semantic comparison using a search engine, we can limit the search space only to those that come from the specified category.

Therefore, it is worth considering the option of limiting semantic search by additional dimensions. Will additional ML methods be useful for this? Of course they will. It is just a matter of imagination and the purpose of a given model.

(…) Additional meta-information about chunks can be very helpful during search, e.g., limiting results only to documents from a specific category.

Changing the order of ranking results

Okay, but what if, after filtering and searching for information, we still have trouble choosing which of the found chunks is the most relevant? This is where the process of reranking the order of results returned by the search engine comes in handy. One of the standard approaches is to use cross-encoders models, which, if properly trained, can determine the degree of relevance of the query to the chunk found. Of course, as in the case of an embedder, if the reranker model is not properly matched to the specifics of the data, the result may be the opposite of what is expected 😉

What should you do in such a situation? There are two main approaches:

after gaining a better understanding of the field and conducting more thorough research, you can make a more informed choice of a different one,
or you can train a specialized reranker yourself, tailored to the problem.

Both approaches have advantages and disadvantages. The first one is faster, but (a) there is no guarantee that there is a model that we need, (b) if we agree on a model, we accept its accuracy in our problem. The approach involving training or retraining our own model will likely be much more costly due to the time needed to develop it. However, there is a good chance that such a model will be much more accurate than the available ones, which will improve the quality of the final RAG system. So what should we choose? It depends on the situation… 😉

Of course, in the reranking process, we are not limited to operating only on cross-encoder models. At this stage, we get potential fragments that contain the answer, so only our imagination limits us here in terms of what can be done… Is it worth using specialized models here? Absolutely.

(…) Matching the relevance of the chunk to the given question is a very important step. An inappropriate change in the ranking order of documents may result in texts containing important information being rejected just before they are submitted to GenAI, which can significantly spoil the final result.

Generating responses

And finally, we are here, at the stage that makes the biggest impression on the end user. It is only at this stage that GenAI comes into play—as an element that constructs the response, which is presented as an answer to the RAG system user.

All of the steps mentioned above constitute a stream of data preprocessing, searching, selection, and determining the relevance of partial results to the user’s query. It is these steps that largely determine whether the final answer will be relevant or not. Incorrectly selected data at this stage will cause the GenAI model to hallucinate. It is also important that the GenAI model we use to generate responses is also properly trained in the task of answering questions. When we talk about GenAI, we tend to think of instruction models. It is therefore important that such a GenAI not only understands the language and is able to respond in that language, but also responds appropriately to instructions. Its task will be to summarize all the fragments/chunks returned from the previous steps. So it is important that such a model is primarily able to respond to instructions for answering questions based on the context provided in the form of chunks.

Instruction models are usually very large models, starting with the smaller 8B (billions), through 70B, 405B, and more parameters. This brings us to another question — should we use an on-premises model or a cloud model? Of course, it all depends on the purpose of the system and the ability to transfer data via the internet to external services such as ChatGPT. If you choose a locally queried generative model, the question remains, as in all previous steps… the right model for the problem at hand 🙂 And what model? That’s a separate story.

(…) The previous steps are important, but equally important is the right choice of GenAI, which will generate the final response. It can ruin the whole impression.

Conclusion

Let’s put it all together 😉

It is safe to say that the quality of responses from such a system is strongly determined by the quality of the data used by the system. It is also important to identify good and bad quality data and standardize it for further processing. It is also important to select or train a specialized embedder and ensure proper indexing in databases. Additional meta-information about chunks can be very helpful during searches, e.g., limiting results to documents from a specific category. Matching the relevance of a chunk to a given question is a very important step. Inappropriate changes to the ranking of documents may result in texts with important information being rejected just before they are submitted to GenAI, which can significantly spoil the final result. The previous steps are important, but equally important is the appropriate selection of GenAI, which will generate the final answer. It is GenAI that can spoil the whole impression.

PPS.

And soon we will publish our GenAI model — pLLama, an LLama model trained in Polish 😉

Tools for RAG Architecture: To manage request routing across local/cloud model engines and protect enterprise prompts against personal data leaks, check out LLM Router and PII Masker.

Unlock the potential of your data with our ML/NLP. Bon appetit! 😉

Bi-encoder and cross-encoder model

2025-09-26T00:00:00Z

The possibilities offered by generative models are enormous, as evidenced by the success of OpenAI and its flagship product, ChatGPT. Generative models based on transformer architecture are on par with humans in terms of content creation, whether in the form of text, images, or more complex animations. Despite their powerful generative capabilities, they have one fundamental drawback: updating information in these models. When querying such a model for information from the everyday world that goes beyond the training set, it is virtually impossible for the model to provide the correct answer. One solution to this problem is an approach based on Retrieval Augmented Generation (RAG).

Retrieval Augmented Generation

The idea behind RAG is to separate the information layer from the generative model. The information layer is a form of database where information is stored, while the generative model is used to provide answers based on information retrieved from the database. What does this approach offer? A lot. 😉

It introduces a breakthrough in NLP, where the previously used Retrieval Information approaches (replaced by new mechanisms), combined with well-functioning generative models, are able to provide the latest information. Since this information is stored in a database, it can be updated on an ongoing basis. When a piece of information becomes outdated, it can simply be removed from the database. As a result, when providing an answer, the generative model will not receive this information as context for its response, which will limit the model’s hallucinations.

Machinery

So what is needed to implement the RAG approach? In the simplest case, it is a few mechanisms:

a mechanism that convert text into vector form (hereinafter referred to as an embedder);
a database that allows indexed texts to be stored and searched (e.g., Milvus );
a generative model that will provide answers based on the context provided (e.g., LLama3 from MetaAI ).

In more complex cases, not only the information retrieval process is controlled, but also the collection of texts returned by the search engine.

Embedder and reranker model

In today’s post, we would like to present our two models. The first one is an embedder, that can be used to transform text into vector form. The second one is a mechanism that reranking the results returned by a semantic search engine. The technical difference between an embedder (bi-encoder) and a reranker (cross-encoder) is very well presented in this article. In a nutshell: we train the bi-encoder model on a semantic similarity process, where the evaluation function is the indicated semantic distance, in our case the cosine angle measure. In the case of a cross-encoder, the model is trained on the value of the loss function, resulting from the correlation of the model’s responses to the test set.

We used the sentence-transformers library to build these models. The loss function for the bi-encoder model is the cosine distance, and for the cross-encoder it is CECorrelationEvaluator. As a dataset, we used existing information retrieval datasets, e.g., ipipan/maupqa, which we enriched with our radlab/polish-sts-dataset. Of course, in both cases, these datasets had to be transformed accordingly and adapted to the problem of training the bi-encoder and cross-encoder.

“bi-encoder”

The bi-encoder model trained for 1 day and 9 hours on a single NVIDIA GeForce RTX 4090 card. The Pearson and Spearman correlation values, measured on the evaluation set, are as follows:

What is most interesting from our perspective is the correlation for Cosine-Similariy. The correlation values are high, and in both cases, they can be interpreted as a strong and fairly strong correlation. This means that the model’s responses strongly correlate with the test data. The published bi-encoder model has an averaged pooling layer. It can be used to create embeddings from text and compare them semantically. Link to the model on huggigface: radlab/polish-bi-encoder-mean.

Sample code to load the model using sentence-transformers, which will create a vector representation for the specified texts:

from sentence_transformers import SentenceTransformer
sentences = ['Ala ma kota i psa, widzi dzisiaj też śnieg', 'Ewa ma białe zęby']

model = SentenceTransformer('radlab/polish-bi-encoder-mean')
embeddings = model.encode(sentences)
print(embeddings)

“cross-encoder”

We have also made available a cross-encoder model that returns the relevance value of two texts. It can, of course, be used as a reranker, e.g., in a semantic search engine. What might such a process look like? As usual- it’s simple 🙂

As a result of a semantic search, we get a set of texts that are ranked according to the similarity of one text fragment to another – the one being searched for. This set is the perfect input for a model whose goal is not only to determine the similarity between two text fragments, but also to determine which fragment better matches the searched text.

We have also made the finished reranking model available on huggingface in the radlab/polish-cross-encoder repository. This model was trained on a single NVIDIA GeForce RTX 4090 card for 3 days and 10 hours. Pearson and Spearman correlation on the test set:

Pearson Correlation : 0.9360742942528038
Spearman Correlation : 0.8718174291678207

The values of both correlations, as in the case of the bi-encoder, are interpreted as a strong and fairly strong correlation between the model’s responses and the test set.

Sample code (using sentence-transformers) that assigns responses to questions:

from sentence_transformers.cross_encoder import CrossEncoder

model_path = "radlab/polish-cross-encoder"
model = CrossEncoder(model_path)

questions = [
    "Jaką mamy dziś pogodę? bo Andrzej nic nie mówił.",
    "Gdzie jedzie Andrzej? Bo wczoraj był w Warszawie.",
    "Czy oskarżony się zgadza z przedstawionym wyrokiem?",
]
answers = [
    "Pan Andrzej siedzi w pociągu i jedzie do Wiednia. Ogląda na telefonie zabawne filmiki.",
    "Poada deszcz i jest wilgotno, jednak wczoraj było słonecznie.",
    "Wyrok jest prawomocny i nie podlega dalszym rozważaniom.",
]
for question in questions:
    context_with_question = [(s, question) for s in answers]
    results = sorted(
        {
            idx: r for idx, r in enumerate(model.predict(context_with_question))
        }.items(),
        key=lambda x: x[1],
        reverse=True,
    )

    print(f"QUESTION: {question}")
    print("ANSWERS (sorted):")
    for idx, score in results:
        print(f"\t[{score}]\t{answers[idx]}")
    print("")

The result of calling the above code should be similar to:

QUESTION: Jaką mamy dziś pogodę? bo Andrzej nic nie mówił.
ANSWERS (sorted):
        [0.016749681904911995]  Poada deszcz i jest wilgotno, jednak wczoraj było słonecznie.
        [0.01602918468415737]   Pan Andrzej siedzi w pociągu i jedzie do Wiednia. Ogląda na telefonie zabawne filmiki.
        [0.016013670712709427]  Wyrok jest prawomocny i nie podlega dalszym rozważaniom.

QUESTION: Gdzie jedzie Andrzej? Bo wczoraj był w Warszawie.
ANSWERS (sorted):
        [0.5997582674026489]    Pan Andrzej siedzi w pociągu i jedzie do Wiednia. Ogląda na telefonie zabawne filmiki.
        [0.4528200924396515]    Wyrok jest prawomocny i nie podlega dalszym rozważaniom.
        [0.17350871860980988]   Poada deszcz i jest wilgotno, jednak wczoraj było słonecznie.

QUESTION: Czy oskarżony się zgadza z przedstawionym wyrokiem?
ANSWERS (sorted):
        [0.8431766629219055]    Wyrok jest prawomocny i nie podlega dalszym rozważaniom.
        [0.6823258996009827]    Poada deszcz i jest wilgotno, jednak wczoraj było słonecznie.
        [0.558414101600647]     Pan Andrzej siedzi w pociągu i jedzie do Wiednia. Ogląda na telefonie zabawne filmiki.

Outro

Both models presented today can be easily used to build a system based on the RAG approach. They can also be used as independently operating models. However, it is worth noting that the performance measured by the cross-encoder model’s operating time is incomparably lower than that of the bi-encoder model. Therefore, in the final solution, it is recommended to combine both models into a single pipeline. In the first step, based on text embeddings, the most similar fragments are searched for in the database, and then the question is matched to a limited set of answers based on the returned fragments.

Of course, for more models, visit our huggingface.

Unlock the potential of your data with our ML/NLP. Bon appetit! 😉

Text denoiser

2025-09-24T00:00:00Z

Currently, the world of natural language processing is dominated by solutions based on transformer architecture models. The diversity of these models has practically dominated every area of NLP. Regardless of their architecture and purpose, however, they have one thing in common: a high demand for good-quality data, which is used to train these models.

The problem to be solved

So where can the data be obtained? Of course, it depends on the specific case and purpose of the model. And data, as data, is often stored in various formats, including one of the least accessible for automatic processing, namely pdf. The concept of saving in pdf format is primarily to present the saved content so that the graphic form is the same in any reader on any device. However, when automatically processing these files, problems with reading the content may be encountered.

An example of a problem with text that occurs after reading a poor-quality pdf file:

Strona...Tytuł...%6%z%50%|.....Zdecydowana0większość0czerwonych.karłów$należy#do,typu,widmowego%M,%ale%zalicza^się^do^nich^także^wieleHgwiazdHpóźnychHpodtypówHtypuHwidmowegoHKHorazHrzadkoHwystępujące,,najsłabsze,gwiazdy,typu,L.,Maksimum%intensywności%emitowanego%światła%przypada%w%zakresie%światła%czerwonego%lub%nawet%bliskiej%podczerwieni.

The correct form of this text is:

Zdecydowana większość czerwonych karłów należy do typu widmowego M, ale zalicza się do nich także wiele gwiazd późnych podtypów typu widmowego K oraz rzadko występujące, najsłabsze gwiazdy typu L. Maksimum intensywności emitowanego światła przypada w zakresie światła czerwonego lub nawet bliskiej podczerwieni.

Solution to the problem

Seeing many problems that are repetitive on a large scale, we developed a method of noise generator to data that resembles the problems encountered.The method is based on conditional probabilities and maps the distribution of problems from mass-processed texts. The developed collection contains 1 M texts, from which we randomly selected 100K for the noise removal model training process.

Training the noise remover

We used a pre-trained base model in the T5 allegro/plt5-base architecture, which we trained on the data cleaning problem. Training the model took 5 hours and 30 minutes on a single NVIDIA GeForce RTX 4090 card. Loss function graph for the evaluation set:

Example of operation

We have made the trained model available on our huggingface repository radlab/polish-denoiser-t5-base.

The input to the model is text in the form: denoise: <text to noise remover>. For the example above, it is:

denoise: Strona...Tytuł...%6%z%50%|.....Zdecydowana0większość0czerwonych.karłów$należy#do,typu,widmowego%M,%ale%zalicza^się^do^nich^także^wieleHgwiazdHpóźnychHpodtypówHtypuHwidmowegoHKHorazHrzadkoHwystępujące,,najsłabsze,gwiazdy,typu,L.,Maksimum%intensywności%emitowanego%światła%przypada%w%zakresie%światła%czerwonego%lub%nawet%bliskiej%podczerwieni.

The output from the model looks like this:

Zdecydowana większość czerwonych karłów należy do typu widmowego M, ale zalicza się do nich także wiele gwiazd późnych podtypów typu widmowego K oraz rzadko występujące, najsławsze gwiazdy typu L. Maksimum intensywności emitowanego światła przypada w zakresie światła czerwonego lub nawet bliskiej podczerwieni.

Sample code that allows you to run the model for the text from the example above:

from transformers import T5ForConditionalGeneration, T5Tokenizer

def do_inference(text, model, tokenizer):
    input_text = f"denoise: {text}"
    inputs = tokenizer.encode(
        input_text,
        return_tensors="pt",
        max_length=256,
        padding="max_length",
        truncation=True,
    )

    corrected_ids = model.generate(
        inputs,
        max_length=256,
        num_beams=5,
        early_stopping=True,
    )

    corrected_sentence = tokenizer.decode(corrected_ids[0], skip_special_tokens=True)
    return corrected_sentence

model = T5ForConditionalGeneration.from_pretrained("radlab/polish-denoiser-t5-base")
tokenizer = T5Tokenizer.from_pretrained("radlab/polish-denoiser-t5-base")

text_str = "Strona...Tytuł...%6%z%50%|.....Zdecydowana0większość0czerwonych.karłów$należy#do,typu,widmowego%M,%ale%zalicza^się^do^nich^także^wieleHgwiazdHpóźnychHpodtypówHtypuHwidmowegoHKHorazHrzadkoHwystępujące,,najsłabsze,gwiazdy,typu,L.,Maksimum%intensywności%emitowanego%światła%przypada%w%zakresie%światła%czerwonego%lub%nawet%bliskiej%podczerwieni."

print(do_inference(text_str, model, tokenizer))

Feel free to use it, bon appétit 🙂

Prompt Sanitization & Security: For production-grade personal data anonymization and masking before inference, explore PII Masker and the LLM Router gateway.

Extraction QA – our model polish-qa-v2

2025-09-24T00:00:00Z

In the field of natural language processing, innovative solutions are constantly emerging that enable precise answers to questions in different languages. We present the polish-qa-v2 model, which represents a step forward in our research on matching questions with answers using large language models.

The polish-qa-v2 model is an example of an extractive QA model that specializes in extracting the most relevant fragment of text to provide a precise answer to a question. Unlike generative models, which create answers from scratch, extractive models limit themselves to selecting existing text fragments as answers to questions.

Example of use

from transformers import pipeline
model_path = "radlab/polish-qa-v2"
question_answerer = pipeline(
  "question-answering",
  model=model_path
)
question = "Jakie silniki posiadał okręt?"
context = """Okręt był napędzany przez trzy trzycylindrowe maszyny parowe potrójnego rozprężania, które napędzały poprzez wały napędowe trzy śruby napędowe (dwie trójskrzydłowe
zewnętrzne o średnicy 4,5 metra i czteroskrzydłową o średnicy 4,2 metra).
Para była dostarczana przez cztery kotły wodnorurkowe typu Marine,
wyposażone łącznie w osiem palenisk i osiem kotłów cylindrycznych,
które miały łącznie 32 paleniska. Ciśnienie robocze kotłów wynosiło 12 at,
a ich łączna powierzchnia grzewcza 3560 m². Wszystkie kotły były opalane węglem,
którego normalny zapas wynosił 650, a maksymalny 1070 ton.
Nominalna moc siłowni wynosiła 13 000 KM (maksymalnie 13 922 KM przy 108 obr./min),
co pozwalało na osiągnięcie prędkości maksymalnej od 17,5 do 17,6 węzła.
Zasięg wynosił 3420 mil morskich przy prędkości 10 węzłów. Zużycie węgla przy mocy 10 000 KM
wynosiło około 11 ton na godzinę, a przy mocy maksymalnej 16 ton na godzinę.
"""
print(
  question_answerer(
    question=question,
    context=context.replace("\n", " ")
  )
)

After executing the above code, the following result is obtained:

{
  'score': 0.612459123134613,
  'start': 25,
  'end': 84,
  'answer': ' trzy trzycylindrowe maszyny parowe potrójnego rozprężania,'
}

The experiment is based on the sdadas/polish-roberta-large-v2 model, which we trained on the clarin-pl/poquad dataset. Training took 2 hours on a single RTX 3090 Ti graphics card.

polish-qa-v2 is available at https://huggingface.co/radlab/polish-qa-v2, where you can test it without having to run it locally. For those interested, we provide the entire model for download at the same link.

We invite you to read.

RadLab — Blog

Are crypto mining farms frozen?

Intro

HiveOS Installation

OS configuration

Installing LLM Router

View the startup log [CLICK]+

Sample configuration: Ollama + gpt‑oss:120b

LLM Router Configuration for gpt‑oss:120b

Outro

Can a knowledge base be easily introduced for GenAI?

Intro

Indexation

Running the knowledge base in LLM Router

Outro

Is it possible to break the laws of nature?

Eagles and jays in one nest

Metaphor of the Natural Ecosystem

A few words about LLM Router

Eagles and jays in LLM Router

Overview of the ecosystem

Outro

Llm-Router- a connector between an application and generative models

Login

Supernova explosion

A box of building blocks

Something for the Eyes

Logout

We make the codes publicly available – open source.

relgat – projector (“from the side”)

plwordnet (“from the side”)

ml-utils (“from the side”)

Modules from the playground

Summary

Information Explorer- not a hammer, but a scalpel

Intro i Outro

Combo

3c Polarization – model from plg on HF

Process: data and learning

Launch and testing

Outro

Unlock the potential of your data with our ML/NLP. Bon appetit! 😉

Information Browser

What is today’s post about?

How does it work?

For whom: “as different as chalk and cheese”

Outro

Unlock the potential of your data with our ML/NLP. Bon appetit! 😉

pLLama3.1 8B — medium-sized and even small GenAI for Polish

Intro

Dataset, learning, and graphs

Outro

Unlock the potential of your data with our ML/NLP. Bon appetite! 😉

pLlama3.2 (1B + 3B) – small GenAI for Polish

Intro

Data and Training

Feelings

Models for Download

What next?

pLlama3 (8B + 70B) – GenAI for Polish

Intro

Dataset

Training Process

What after training?

Huggingface

RAG (problems?)

RAG in pieces

Data

Data Preprocessing

Data indexing

Search

Changing the order of ranking results

Generating responses

Conclusion

Unlock the potential of your data with our ML/NLP. Bon appetit! 😉

Bi-encoder and cross-encoder model

Retrieval Augmented Generation

Machinery

Embedder and reranker model

“bi-encoder”

Installing LLM Router

View the startup log [CLICK]+

LLM Router Configuration for gpt‑oss:120b