Models

Compare classical methods and transformers for text processing.

Logistic Regression

ENTrained

Accuracy: 62.67%
Macro F1: 37.25%
F1 weighted: 60.36%
Precision weighted: 59.68%

Linear SVM

ENTrained

Accuracy: 62.57%
Macro F1: 37.06%
F1 weighted: 60.15%
Precision weighted: 59.29%

Multilingual BERT

ENTrained

Accuracy: 62.51%
Macro F1: 37.91%
F1 weighted: 60.07%
Precision weighted: 60.98%

Logistic Regression

A classical statistical method: a linear model that estimates the probability of each class.

TF-IDF · word (1,2) + char (3,5)

EN · Trained

Accuracy: 62.67%
Macro F1: 37.25%
F1 weighted: 60.36%
Precision weighted: 59.68%

Protocol: SGD: official train / dev / test partitions, all dataset files. Only user turns that start or change the intent are used (confirmations, answers to follow-up questions and thanks are excluded). Equivalent API versions are merged into one class; Services subtypes are kept. Classes absent from train remain in test. Exact duplicate texts removed.

Test examples: 9912 · Macro F1: 37.25%

Classes seen in training: Accuracy 78.58% · Macro F1 74.49% · examples from classes absent in training: 2007 / 9912 (11)

Samples train / dev / test: 37201 / 5324 / 9912

Slot filling · train lexicon, exact span and type: Precision 2.5% · Recall 2.0% · F1 2.2%

Trained on: 25 Oct 2026, 10:17 UTC

Metrics apply only to the stated dataset, language and partition. Macro F1 gives each class equal weight.

sklearn.linear_model.LogisticRegression

Linear SVM

A support vector machine for separating classes.

TF-IDF · word (1,2) + char (3,5)

EN · Trained

Accuracy: 62.57%
Macro F1: 37.06%
F1 weighted: 60.15%
Precision weighted: 59.29%

Test examples: 9912 · Macro F1: 37.06%

Classes seen in training: Accuracy 78.46% · Macro F1 74.13% · examples from classes absent in training: 2007 / 9912 (11)

Samples train / dev / test: 37201 / 5324 / 9912

Slot filling · train lexicon, exact span and type: Precision 2.5% · Recall 2.0% · F1 2.2%

Trained on: 25 Oct 2026, 10:18 UTC

Metrics apply only to the stated dataset, language and partition. Macro F1 gives each class equal weight.

sklearn.svm.LinearSVC

Multilingual BERT

A multilingual transformer for working with multiple languages.

Transformer · contextual embeddings

EN · Trained

Accuracy: 62.51%
Macro F1: 37.91%
F1 weighted: 60.07%
Precision weighted: 60.98%

Test examples: 9912 · Macro F1: 37.91%

Classes seen in training: Accuracy 78.38% · Macro F1 74.37% · examples from classes absent in training: 2007 / 9912 (11)

Samples train / dev / test: 37201 / 5324 / 9912

Slot filling · trained mBERT tagger: Precision 44.3% · Recall 48.0% · F1 46.1% · lexicon on the same data: F1 6.1%

Trained on: 25 Oct 2026, 16:33 UTC

Metrics apply only to the stated dataset, language and partition. Macro F1 gives each class equal weight.

google-bert/bert-base-multilingual-cased

Models

Logistic Regression

Linear SVM

Multilingual BERT

Logistic Regression

Linear SVM

Multilingual BERT

Model comparison

Intent recognition

Entity recognition and sentiment

People, places and organizations

Control check · n=4

Sentiment

Control check · n=9