MLM, masked language modeling, is an important task for trianing a BERT model. In the orignal BERT paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, it is one of the main tasks of how BERT was pre-trained. So if you have your own corpus, it is possible to train MLM on any pre-trainedContinue reading “Deep Learning 19: Training MLM on any pre-trained BERT models”
Category Archives: Theory
Slideshare (11): Graph Neural Networks and Applications in NLP
Check out my class talk slides about Graph Neural Networks and their applications in NLP! Covered materials: Semi-Supervised Classification with Graph Convolutional Networks Variational Graph Auto-Encoders Graph Attention Networks Graph Convolutional Networks for Text Classification (AAAI 2019) Heterogeneous Graph Neural Networks for Extractive Document Summarization (ACL 2020) A Graph-based Coarse-to-fine Method for Unsupervised Bilingual LexiconContinue reading “Slideshare (11): Graph Neural Networks and Applications in NLP”
Slideshare (10): Reading notes bout Text Generation, BERT, Knowledge Distillation…
Please check my notes for the Spring Semester 2020: Topics covered: Text Generation BERT understanding Knowledge Distillation NLP Applications EHR+Translation
Paddle Serving: model-as-a-service! Triggered by a single command line, deployment finishes in 10 minutes
To bridge the gap between Paddle Serving and PaddlePaddle framework, we release the new service of PaddleServing: Model As A Service (MAAS) online in Github. With the help of the new service, when a PaddlePaddle model is trained, users now can obtain the corresponding inference model at the same time, making it possible to deployContinue reading “Paddle Serving: model-as-a-service! Triggered by a single command line, deployment finishes in 10 minutes”
Slideshare (9): Transfer learning tutorial 2.0
Please check my notes for an updated version about Transfer Learning with NLP tasks: Irene_TransferLearning_2020Spring A brief outline: Transfer Learning with word embedding Pre-BERT times BERT and its variants Understanding, reducing BERT Transfer Learning in the real world Also, visit my A brief Introduction on Transfer Learning notes from 2019 Spring.
Slideshare (8): Graph Neural Network Paper reading notes
The pdf contains some notes on several papers, some of which only have 1-2 slide pages. Again, red fonts are my thoughts and insights. I wish my notes can help readers to better understand the new concepts and get inspired. Click to view: GNN_notes_Fall2019.pdf
Prepare for the Interviews!
For the past few years after my Master’s, I did many jobs, long term, short term, internship, or full-time. I also had too many interviews, some of them I failed. Together with my friends, we had collected many materials, including basic algorithms, popular questions, basic machine learning knowledge, and deep learning knowledge. Then I organizedContinue reading “Prepare for the Interviews!”
Understanding Variational Graph Auto-Encoders
Variational Auto-Encoders My post about Auto-encoder. For Variational Auto-Encoders (VAE) (from paper Auto-Encoding Variational Bayes), we actually add latent variables to the existing Autoencoders. The main idea is, we want to restrict the parameters from a known distribution. Why we want this? We wish the generative model to provide more “creative” things. If the modelContinue reading “Understanding Variational Graph Auto-Encoders”
Understanding Graph Convolutional Networks
Why Graphs? Graph Convolution Networks (GCNs) [0] deal with graphs where the data form with a graph structure. A typical graph is represented as G(V, E), where V is the collection of all the nodes and Eis the collection of all the edges.
Slideshare (4): A brief Introduction on Transfer Learning
Please check my notes for Transfer Learning introduction! Transfer Learning
ireneli.eu 