Category Archives: Theory

Deep Learning 22: Diffusion Models (2)

Previously, we introduced Autoencoders and Hierarchical Variational Autoencoders (HVAEs). In this post, we will cover the details of Denoising Diffusion Probabilistic Models (DDPM). Diffusion Models We can treat DDPM as a restricted HVAE. Here, each only depends on . In DDPM, we do not have parameters to add noises, and it is a predefined GaussianContinue reading “Deep Learning 22: Diffusion Models (2)”

Deep Learning 21: Diffusion Models (1)

Background: Autoencoders   We would let the latent variables form a normal distribution, and we measure how similar the generated data and the truth data . The loss is defined as: The first term is the reconstruction loss on the generated data and truth data, and the second term is the restriction of the variationalContinue reading “Deep Learning 21: Diffusion Models (1)”

Deep Learning 20: graph batching in PyTorch Geometric

PyG or PyTorch Geometric is a deep learning framework for GNNs (graph neural networks). When dealing with graphs, to speed up the computation, we need to do the batch computation, even though the graphs are in different “shapes”. In PyG, it is possible to pack the data in batches. According to the documentation. “Adjacency matricesContinue reading “Deep Learning 20: graph batching in PyTorch Geometric”

AAN.how: All About NLP —— New 2022 Release: over 24,000 resources and over 7,000 lecture notes!

🎉 We are pleased to release the new, 2022 version of the AAN database with over 24,000 resources and over 7,000 lecture notes. Visit aan.how! 2022 TutorialBank This update includes 19,765 manually collected resources with valid URLs, meta-data, organized by topic. We also release an extra batch of 5,001 resources. These resources have valid URLs butContinue reading “AAN.how: All About NLP —— New 2022 Release: over 24,000 resources and over 7,000 lecture notes!”

Deep Learning 19: Training MLM on any pre-trained BERT models

MLM, masked language modeling, is an important task for trianing a BERT model. In the orignal BERT paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, it is one of the main tasks of how BERT was pre-trained. So if you have your own corpus, it is possible to train MLM on any pre-trainedContinue reading “Deep Learning 19: Training MLM on any pre-trained BERT models”

Slideshare (11): Graph Neural Networks and Applications in NLP

Check out my class talk slides about Graph Neural Networks and their applications in NLP! Covered materials: Semi-Supervised Classification with Graph Convolutional Networks Variational Graph Auto-Encoders Graph Attention Networks Graph Convolutional Networks for Text Classification (AAAI 2019) Heterogeneous Graph Neural Networks for Extractive Document Summarization (ACL 2020) A Graph-based Coarse-to-fine Method for Unsupervised Bilingual LexiconContinue reading “Slideshare (11): Graph Neural Networks and Applications in NLP”

Paddle Serving: model-as-a-service! Triggered by a single command line, deployment finishes in 10 minutes

To bridge the gap between Paddle Serving and PaddlePaddle framework, we release the new service of PaddleServing: Model As A Service (MAAS) online in Github. With the help of the new service, when a PaddlePaddle model is trained, users now can obtain the corresponding inference model at the same time, making it possible to deployContinue reading “Paddle Serving: model-as-a-service! Triggered by a single command line, deployment finishes in 10 minutes”

Slideshare (9): Transfer learning tutorial 2.0

Please check my notes for an updated version about Transfer Learning with NLP tasks: Irene_TransferLearning_2020Spring A brief outline: Transfer Learning with word embedding Pre-BERT times BERT and its variants Understanding, reducing BERT Transfer Learning in the real world   Also, visit my A brief Introduction on Transfer Learning notes from 2019 Spring.