Natural Language Processing

1. Fundamentals of text processing
Introduction to NLP. Pattern matching and regular expressions for searching patterns in text. Text tokenization. Stemming and lemmatization. Sentence segmentation. Edit distance, measuring similarity between words.
2. Statistical language models (N-grams)
N-grams: unigrams, bigrams, trigrams. Calculating sequence probabilities. Perplexity.
3. Text classification – Naive Bayes
The principle of the Naive Bayes classifier. Training the model on data. Evaluation metrics: Precision, Recall, F1 score. Examples: sentiment analysis, spam detection.
4. Text classification – Logistic regression
Linear and logistic regression. Representation of text using features. Calculation of weights using gradient descent. Prevention of overfitting. Multinomial logistic regression.
5. Vector model and Word Embeddings
Traditional vector model: TF-IDF. Word Embeddings: Word2vec, Skip-gram, Continuous bag of words. Properties of embeddings. Bias in data and models.
6. Introduction to neural networks for NLP
Perceptron. Feedforward neural networks. Activation functions (ReLU, sigmoid). Network training. Use for text classification.
7. Recurrent Neural Networks (RNN and LSTM)
The principle of recurrent neural networks (RNN) and their internal state (memory). The problem of vanishing and exploding gradients. LSTM (Long Short-Term Memory) and GRU cells. Bidirectional and layered RNN architecture.
8. Encoder-Decoder Model and Attention Mechanism
Encoder-Decoder architecture. Attention mechanism.
9. Transformer Architecture
Self-Attention. Multi-Head Attention. Positional Encoding. Transformer block structure (encoder and decoder).
10. Large Language Models (LLM)
GPT (Generative Pre-trained Transformer). Pre-training principle. Text generation and sampling techniques.
11. Bidirectional BERT Models and Fine Tuning
BERT (Bidirectional Encoder Representations from Transformers). Masked Language Modeling (MLM). Fine tuning: adapting a pre-trained model to specific tasks (classification, entity recognition - NER).
12. The future of NLP
Prompting: Zero-shot, one-shot, and few-shot learning. Chain-of-Thought prompting for complex tasks. RLHF (Reinforcement Learning from Human Feedback).