[Notice] This list is not being maintained anymore because of the overwhelming amount of deep learning papers published every day since 2017.
A curated list of the most cited deep learning papers (2012-2016)
We believe that there exist classic deep learning papers which are worth reading regardless of their application domain. Rather than providing overwhelming amount of papers, We would like to provide a curated list of the awesome deep learning papers which are considered as must-reads in certain research domains.
Before this list, there exist other awesome deep learning lists, for example, Deep Vision and Awesome Recurrent Neural Networks. Also, after this list comes out, another awesome list for deep learning beginners, called Deep Learning Papers Reading Roadmap, has been created and loved by many deep learning researchers.
Although the Roadmap List includes lots of important deep learning papers, it feels overwhelming for me to read them all. As I mentioned in the introduction, I believe that seminal works can give us lessons regardless of their application domain. Thus, I would like to introduce top 100 deep learning papers here as a good starting point of overviewing deep learning researches.
To get the news for newly released papers everyday, follow my twitter or facebook page!
(Citation criteria) - < 6 months : New Papers (by discussion) - 2016 : +60 citations or “More Papers from 2016” - 2015 : +200 citations - 2014 : +400 citations - 2013 : +600 citations - 2012 : +800 citations - ~2012 : Old Papers (by discussion)
Please note that we prefer seminal deep learning papers that can be applied to various researches rather than application papers. For that reason, some papers that meet the criteria may not be accepted while others can be. It depends on the impact of the paper, applicability to other researches scarcity of the research domain, and so on.
We need your contributions!
If you have any suggestions (missing papers, new papers, key researchers or typos), please feel free to edit and pull a request. (Please read the contributing guide for further instructions, though just letting me know the title of papers can also be a big contribution to us.)
(Update) You can download all top-100 papers with this and collect all authors’ names with this. Also, bib file for all top-100 papers are available. Thanks, doodhwala, Sven and grepinsight!
(More than Top 100)
Newly published papers (< 6 months) which are worth reading - MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (2017), Andrew G. Howard et al. [pdf] - Convolutional Sequence to Sequence Learning (2017), Jonas Gehring et al. [pdf] - A Knowledge-Grounded Neural Conversation Model (2017), Marjan Ghazvininejad et al. [pdf] - Accurate, Large Minibatch SGD:Training ImageNet in 1 Hour (2017), Priya Goyal et al. [pdf] - TACOTRON: Towards end-to-end speech synthesis (2017), Y. Wang et al. [pdf] - Deep Photo Style Transfer (2017), F. Luan et al. [pdf] - Evolution Strategies as a Scalable Alternative to Reinforcement Learning (2017), T. Salimans et al. [pdf] - Deformable Convolutional Networks (2017), J. Dai et al. [pdf] - Mask R-CNN (2017), K. He et al. [pdf] - Learning to discover cross-domain relations with generative adversarial networks (2017), T. Kim et al. [pdf] - Deep voice: Real-time neural text-to-speech (2017), S. Arik et al., [pdf] - PixelNet: Representation of the pixels, by the pixels, and for the pixels (2017), A. Bansal et al. [pdf] - Batch renormalization: Towards reducing minibatch dependence in batch-normalized models (2017), S. Ioffe. [pdf] - Wasserstein GAN (2017), M. Arjovsky et al. [pdf] - Understanding deep learning requires rethinking generalization (2017), C. Zhang et al. [pdf] - Least squares generative adversarial networks (2016), X. Mao et al. [pdf]
Classic papers published before 2012 - An analysis of single-layer networks in unsupervised feature learning (2011), A. Coates et al. [pdf] - Deep sparse rectifier neural networks (2011), X. Glorot et al. [pdf] - Natural language processing (almost) from scratch (2011), R. Collobert et al. [pdf] - Recurrent neural network based language model (2010), T. Mikolov et al. [pdf] - Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion (2010), P. Vincent et al. [pdf] - Learning mid-level features for recognition (2010), Y. Boureau [pdf] - A practical guide to training restricted boltzmann machines (2010), G. Hinton [pdf] - Understanding the difficulty of training deep feedforward neural networks (2010), X. Glorot and Y. Bengio [pdf] - Why does unsupervised pre-training help deep learning (2010), D. Erhan et al. [pdf] - Learning deep architectures for AI (2009), Y. Bengio. [pdf] - Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations (2009), H. Lee et al. [pdf] - Greedy layer-wise training of deep networks (2007), Y. Bengio et al. [pdf] - Reducing the dimensionality of data with neural networks, G. Hinton and R. Salakhutdinov. [pdf] - A fast learning algorithm for deep belief nets (2006), G. Hinton et al. [pdf] - Gradient-based learning applied to document recognition (1998), Y. LeCun et al. [pdf] - Long short-term memory (1997), S. Hochreiter and J. Schmidhuber. [pdf]
(Lectures) - CS231n, Convolutional Neural Networks for Visual Recognition, Stanford University [web] - CS224d, Deep Learning for Natural Language Processing, Stanford University [web] - Oxford Deep NLP 2017, Deep Learning for Natural Language Processing, University of Oxford [web]
(Tutorials) - NIPS 2016 Tutorials, Long Beach [web] - ICML 2016 Tutorials, New York City [web] - ICLR 2016 Videos, San Juan [web] - Deep Learning Summer School 2016, Montreal [web] - Bay Area Deep Learning School 2016, Stanford [web]
(Blogs) - OpenAI [web] - Distill [web] - Andrej Karpathy Blog [web] - Colah’s Blog [Web] - WildML [Web] - FastML [web] - TheMorningPaper [web]
(2016) - A character-level decoder without explicit segmentation for neural machine translation (2016), J. Chung et al. [pdf] - Dermatologist-level classification of skin cancer with deep neural networks (2017), A. Esteva et al. [html] - Weakly supervised object localization with multi-fold multiple instance learning (2017), R. Gokberk et al. [pdf] - Brain tumor segmentation with deep neural networks (2017), M. Havaei et al. [pdf] - Professor Forcing: A New Algorithm for Training Recurrent Networks (2016), A. Lamb et al. [pdf] - Adversarially learned inference (2016), V. Dumoulin et al. [web][pdf] - Understanding convolutional neural networks (2016), J. Koushik [pdf] - Taking the human out of the loop: A review of bayesian optimization (2016), B. Shahriari et al. [pdf] - Adaptive computation time for recurrent neural networks (2016), A. Graves [pdf] - Densely connected convolutional networks (2016), G. Huang et al. [pdf] - Region-based convolutional networks for accurate object detection and segmentation (2016), R. Girshick et al. - Continuous deep q-learning with model-based acceleration (2016), S. Gu et al. [pdf] - A thorough examination of the cnn/daily mail reading comprehension task (2016), D. Chen et al. [pdf] - Achieving open vocabulary neural machine translation with hybrid word-character models, M. Luong and C. Manning. [pdf] - Very Deep Convolutional Networks for Natural Language Processing (2016), A. Conneau et al. [pdf] - Bag of tricks for efficient text classification (2016), A. Joulin et al. [pdf] - Efficient piecewise training of deep structured models for semantic segmentation (2016), G. Lin et al. [pdf] - Learning to compose neural networks for question answering (2016), J. Andreas et al. [pdf] - Perceptual losses for real-time style transfer and super-resolution (2016), J. Johnson et al. [pdf] - Reading text in the wild with convolutional neural networks (2016), M. Jaderberg et al. [pdf] - What makes for effective detection proposals? (2016), J. Hosang et al. [pdf] - Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks (2016), S. Bell et al. [pdf]. - Instance-aware semantic segmentation via multi-task network cascades (2016), J. Dai et al. [pdf] - Conditional image generation with pixelcnn decoders (2016), A. van den Oord et al. [pdf] - Deep networks with stochastic depth (2016), G. Huang et al., [pdf] - Consistency and Fluctuations For Stochastic Gradient Langevin Dynamics (2016), Yee Whye Teh et al. [pdf]
(2015) - Ask your neurons: A neural-based approach to answering questions about images (2015), M. Malinowski et al. [pdf] - Exploring models and data for image question answering (2015), M. Ren et al. [pdf] - Are you talking to a machine? dataset and methods for multilingual image question (2015), H. Gao et al. [pdf] - Mind’s eye: A recurrent visual representation for image caption generation (2015), X. Chen and C. Zitnick. [pdf] - From captions to visual concepts and back (2015), H. Fang et al. [pdf]. - Towards AI-complete question answering: A set of prerequisite toy tasks (2015), J. Weston et al. [pdf] - Ask me anything: Dynamic memory networks for natural language processing (2015), A. Kumar et al. [pdf] - Unsupervised learning of video representations using LSTMs (2015), N. Srivastava et al. [pdf] - Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding (2015), S. Han et al. [pdf] - Improved semantic representations from tree-structured long short-term memory networks (2015), K. Tai et al. [pdf] - Character-aware neural language models (2015), Y. Kim et al. [pdf] - Grammar as a foreign language (2015), O. Vinyals et al. [pdf] - Trust Region Policy Optimization (2015), J. Schulman et al. [pdf] - Beyond short snippents: Deep networks for video classification (2015) [pdf] - Learning Deconvolution Network for Semantic Segmentation (2015), H. Noh et al. [pdf] - Learning spatiotemporal features with 3d convolutional networks (2015), D. Tran et al. [pdf] - Understanding neural networks through deep visualization (2015), J. Yosinski et al. [pdf] - An Empirical Exploration of Recurrent Network Architectures (2015), R. Jozefowicz et al. [pdf] - Deep generative image models using a laplacian pyramid of adversarial networks (2015), E.Denton et al. [pdf] - Gated Feedback Recurrent Neural Networks (2015), J. Chung et al. [pdf] - Fast and accurate deep network learning by exponential linear units (ELUS) (2015), D. Clevert et al. [pdf] - Pointer networks (2015), O. Vinyals et al. [pdf] - Visualizing and Understanding Recurrent Networks (2015), A. Karpathy et al. [pdf] - Attention-based models for speech recognition (2015), J. Chorowski et al. [pdf] - End-to-end memory networks (2015), S. Sukbaatar et al. [pdf] - Describing videos by exploiting temporal structure (2015), L. Yao et al. [pdf] - A neural conversational model (2015), O. Vinyals and Q. Le. [pdf] - Improving distributional similarity with lessons learned from word embeddings, O. Levy et al. [[pdf]] (https://www.transacl.org/ojs/index.php/tacl/article/download/570/124) - Transition-Based Dependency Parsing with Stack Long Short-Term Memory (2015), C. Dyer et al. [pdf] - Improved Transition-Based Parsing by Modeling Characters instead of Words with LSTMs (2015), M. Ballesteros et al. [pdf] - Finding function in form: Compositional character models for open vocabulary word representation (2015), W. Ling et al. [pdf]
(~2014) - DeepPose: Human pose estimation via deep neural networks (2014), A. Toshev and C. Szegedy [pdf] - Learning a Deep Convolutional Network for Image Super-Resolution (2014, C. Dong et al. [pdf] - Recurrent models of visual attention (2014), V. Mnih et al. [pdf] - Empirical evaluation of gated recurrent neural networks on sequence modeling (2014), J. Chung et al. [pdf] - Addressing the rare word problem in neural machine translation (2014), M. Luong et al. [pdf] - On the properties of neural machine translation: Encoder-decoder approaches (2014), K. Cho et. al. - Recurrent neural network regularization (2014), W. Zaremba et al. [pdf] - Intriguing properties of neural networks (2014), C. Szegedy et al. [pdf] - Towards end-to-end speech recognition with recurrent neural networks (2014), A. Graves and N. Jaitly. [pdf] - Scalable object detection using deep neural networks (2014), D. Erhan et al. [pdf] - On the importance of initialization and momentum in deep learning (2013), I. Sutskever et al. [pdf] - Regularization of neural networks using dropconnect (2013), L. Wan et al. [pdf] - Learning Hierarchical Features for Scene Labeling (2013), C. Farabet et al. [pdf] - Linguistic Regularities in Continuous Space Word Representations (2013), T. Mikolov et al. [pdf] - Large scale distributed deep networks (2012), J. Dean et al. [pdf] - A Fast and Accurate Dependency Parser using Neural Networks. Chen and Manning. [pdf]
Thank you for all your contributions. Please make sure to read the contributing guide before you make a pull request.
To the extent possible under law, Terry T. Um has waived all copyright and related or neighboring rights to this work.