Natural Language Processing

- Fundamentals of text processing
Introduction to NLP. Pattern matching and regular expressions for searching patterns in text. Text tokenization. Stemming and lemmatization. Sentence segmentation. Edit distance, measuring similarity between words.

- Statistical language models (N-grams)
N-grams: unigrams, bigrams, trigrams. Calculating sequence probabilities. Perplexity.

- Text classification – Naive Bayes
The principle of the Naive Bayes classifier. Training the model on data. Evaluation metrics: Precision, Recall, F1 score. Examples: sentiment analysis, spam detection.

- Text classification – Logistic regression
Linear and logistic regression. Representation of text using features. Calculation of weights using gradient descent. Prevention of overfitting. Multinomial logistic regression.

- Vector model and Word Embeddings
Traditional vector model: TF-IDF. Word Embeddings: Word2vec, Skip-gram, Continuous bag of words. Properties of embeddings. Bias in data and models.

- Introduction to neural networks for NLP
Perceptron. Feedforward neural networks. Activation functions (ReLU, sigmoid). Network training. Use for text classification.

- Recurrent Neural Networks (RNN and LSTM)
The principle of recurrent neural networks (RNN) and their internal state (memory). The problem of vanishing and exploding gradients. LSTM (Long Short-Term Memory) and GRU cells. Bidirectional and layered RNN architecture.

- Encoder-Decoder Model and Attention Mechanism
Encoder-Decoder architecture. Attention mechanism.

- Transformer Architecture
Self-Attention. Multi-Head Attention. Positional Encoding. Transformer block structure (encoder and decoder).

- Large Language Models (LLM)
GPT (Generative Pre-trained Transformer). Pre-training principle. Text generation and sampling techniques.

- Bidirectional BERT Models and Fine Tuning
BERT (Bidirectional Encoder Representations from Transformers). Masked Language Modeling (MLM). Fine tuning: adapting a pre-trained model to specific tasks (classification, entity recognition - NER).

- The future of NLP
Prompting: Zero-shot, one-shot, and few-shot learning. Chain-of-Thought prompting for complex tasks. RLHF (Reinforcement Learning from Human Feedback).