PyTorch Tutorial for Deep Learning Researchers


This repository provides tutorial code for deep learning researchers to learn PyTorch. In the tutorial, most of the models were implemented with less than 30 lines of code. Before starting this tutorial, it is recommended to finish Official Pytorch Tutorial.


Table of Contents

1. Basics

2. Intermediate

3. Advanced

4. Utilities


Getting Started

$ git clone https://github.com/yunjey/pytorch-tutorial.git
$ cd pytorch-tutorial/tutorials/PATH_TO_PROJECT
$ python main.py

Dependencies

Comments
  • [Image Captioning] out of memory immediately after training starts?

    [Image Captioning] out of memory immediately after training starts?

    What size cards are these networks tested and trained on? I just tried running "09 - Image Captioning" and I immediately get errors. I am testing this on a Titan X with 12 GB of memory:

    Namespace(batch_size=128, caption_path='./data/annotations/captions_train2014.json', crop_size=224, embed_size=256, hidden_size=512, image_dir='./data/resized2014', learning_rate=0.001, log_step=10, model_path='./models/', num_epochs=5, num_layers=1, num_workers=2, save_step=1000, vocab_path='./data/vocab.pkl')
    loading annotations into memory...
    Done (t=0.89s)
    creating index...
    index created!
    THCudaCheck FAIL file=/b/wheel/pytorch-src/torch/lib/THC/generic/THCStorage.cu line=66 error=2 : out of memory
    Traceback (most recent call last):
      File "train.py", line 119, in <module>
        main(args)
      File "train.py", line 66, in main
        features = encoder(images)
      File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
        result = self.forward(*input, **kwargs)
      File "/root/pytorch/pytorch-tutorial/tutorials/09 - Image Captioning/model.py", line 26, in forward
        features = self.resnet(images)
      File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
        result = self.forward(*input, **kwargs)
      File "/root/.venv/local/lib/python2.7/site-packages/torchvision/models/resnet.py", line 146, in forward
        x = self.layer3(x)
      File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
        result = self.forward(*input, **kwargs)
      File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 64, in forward
        input = module(input)
      File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
        result = self.forward(*input, **kwargs)
      File "/root/.venv/local/lib/python2.7/site-packages/torchvision/models/resnet.py", line 85, in forward
        out = self.bn3(out)
      File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
        result = self.forward(*input, **kwargs)
      File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/modules/batchnorm.py", line 43, in forward
        self.training, self.momentum, self.eps)
      File "/root/.venv/local/lib/python2.7/site-packages/torch/nn/functional.py", line 463, in batch_norm
        return f(input, weight, bias)
    RuntimeError: cuda runtime error (2) : out of memory at /b/wheel/pytorch-src/torch/lib/THC/generic/THCStorage.cu:66
    Command exited with non-zero status 1
    
  • Loss Calculation in Image Captioning

    Loss Calculation in Image Captioning

    Hi , am new to PyTorch in model.py you do ,

    embeddings = self.embed(captions)
    embeddings = torch.cat((features.unsqueeze(1), embeddings), 1)
    

    that means you are only passing the context(from the EncoderCNN ) as input to the first time step, hence your outputs will always be 1+captions size as you are running the lstm for captions+1 number of time steps in the decoder , but looks like you have targets and outputs of same size. What am I missing here ? Thanks in advance .

  • Train Image Captioning Code on Other Datasets

    Train Image Captioning Code on Other Datasets

    @yunjey How we can train your image captioning code on another dataset? If we prepare a JSON file like (http://msvocds.blob.core.windows.net/annotations-1-0-3/captions_train-val2014.zip) the code can be trained on a new dataset?

  • [Image Captioning] High GPU memory usage when using pytorch 0.11 for image captioning

    [Image Captioning] High GPU memory usage when using pytorch 0.11 for image captioning

    Hi, I am running the 09 image captioning model. When I used the pytorch 0.10, I can set the batchsize to be 512 (takes around 8G for the GPU memory). But when I upgrade to pytorch 0.11, the maximum batchsize I can set is only around 30 (takes around 11G GPU memory). I am confused since the codebase is mainly the basic CNN and lstm models. Do you know where the problem comes from? Have you tried to upgrade to pytorch 0.11 to run it?

    By the way, I am using Tesla K40c GPU. pytorch 0.10: 0.1.10+ac9245a pytorch 0.11: same problem with (0.1.11_5 and 0.1.11+b13b701)

  • Size mismatch error occurs when using pretrained ResNet model.

    Size mismatch error occurs when using pretrained ResNet model.

    Hello @yunjey . Nice to meet you and thanks for your work. I'm studying pytorch with your source.

    I think the resolution of input image shoud be changed "256x256" to "224x224".


    • Environment

      • hardware: MacBook Pro (Retina, 15-inch, Early 2013)
      • os: macOS Sierra 10.12.6
    • Version

      • torch==0.3.0.post4
      • torchvision==0.2.0
    • Code in error: pytorch-tutorial/tutorials/01-basics/pytorch_basics/main.py line 152-154

      # For test.
      images = Variable(torch.randn(10, 3, 256, 256)) # => randn(10, 3, 224, 224)
      outputs = resnet(images)
      
    • Error message

    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-7-55315f838129> in <module>()
          1 images = Variable(torch.randn(10, 3, 256, 256))
    ----> 2 outputs = resnet(images)
          3 print (outputs.size())   # (10, 100)
    
    ~/.virtualenvs/fira/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
        323         for hook in self._forward_pre_hooks.values():
        324             hook(self, input)
    --> 325         result = self.forward(*input, **kwargs)
        326         for hook in self._forward_hooks.values():
        327             hook_result = hook(self, input, result)
    
    ~/.virtualenvs/fira/lib/python3.6/site-packages/torchvision/models/resnet.py in forward(self, x)
        149         x = self.avgpool(x)
        150         x = x.view(x.size(0), -1)
    --> 151         x = self.fc(x)
        152 
        153         return x
    
    ~/.virtualenvs/fira/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
        323         for hook in self._forward_pre_hooks.values():
        324             hook(self, input)
    --> 325         result = self.forward(*input, **kwargs)
        326         for hook in self._forward_hooks.values():
        327             hook_result = hook(self, input, result)
    
    ~/.virtualenvs/fira/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
         53 
         54     def forward(self, input):
    ---> 55         return F.linear(input, self.weight, self.bias)
         56 
         57     def __repr__(self):
    
    ~/.virtualenvs/fira/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)
        833     if input.dim() == 2 and bias is not None:
        834         # fused op is marginally faster
    --> 835         return torch.addmm(bias, input, weight.t())
        836 
        837     output = input.matmul(weight.t())
    
    RuntimeError: size mismatch, m1: [10 x 2048], m2: [512 x 100] at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:1416
    
  • i have a problem of image caption,can i use another language?

    i have a problem of image caption,can i use another language?

    I try to train model by using my own datasets, but i have a problem of character encoding, i try to change the caption into Chinese, but the vab.pkl has many ' S'\xe7\xba\xa6' ' and when i use my trained model,it always gives /</start/>/</unk/>/</end/>/ ,so, could you please tell me how to solve this problem?Thank you very much.

  • [DCGAN] Deep Convolutional Generative Adversarial Network

    [DCGAN] Deep Convolutional Generative Adversarial Network

    When I follow the tutorials to train this network, I can not get effective image results. What should I do??

    I made the following modifications

    Save the sampled images

    fake_images = fake_images.view(fake_images.size(0), 3, 32, 32) torchvision.utils.save_image(fake_images.data, './sample/fake_samples_%d.png'

    qq 20170520170512

  •  No such file or directory: 'models/encoder-2-1000.ckpt'

    No such file or directory: 'models/encoder-2-1000.ckpt'

    I downloaded the pretrained model to use. But I don't have the models/encoder-2 1000.ckpt' file. What I have from the model.zip are just "decoder-5 3000.pkl" and "encoder-5 3000.pkl".

  • [Image captioning] pack_packed wrong lengths

    [Image captioning] pack_packed wrong lengths

    In the Image Captioning tutorial in the DecoderRNN:

    def forward(self, features, captions, lengths):
      embeddings = self.embed(captions)
      embeddings = torch.cat((features.unsqueeze(1), embeddings), 1)
      packed = pack_padded_sequence(embeddings, lengths, batch_first=True)
      hiddens, _ = self.lstm(packed)
      outputs = self.linear(hiddens[0])
      return outputs
    

    shouldn't the lenghts inside pack_padded be lenghts+1 to take into account the increased length due to the features added with cat?

    e.g. (assuming numbers are captions, e is the embedding, f are the features and batch_size = 4) if

    embeds:
    e(126)  e(1214)  e(14)    e(4033)
    e(126)  e(6)     e(84)    e(4033)
    e(126)  e(3002)  e(4033)  e(0)
    e(126)  e(3002)  e(4033)  e(0)
    

    has lengths = [4,4,3,3] then

    embeds_cat:
    f_0  e(126)  e(1214)  e(14)    e(4033)
    f_1  e(126)  e(6)     e(84)    e(4033)
    f_2  e(126)  e(3002)  e(4033)  e(0)
    f_3  e(126)  e(3002)  e(4033)  e(0) 
    

    should have lengths = [5,5,4,4], right?

  • No scalar data was found.

    No scalar data was found.

    Hi, yunnjey. thanks for your work. I run your demo, but it doesn't seem to work. image what's strange is that there is tf events under the log files. so is there any thing wrong?

  • Error with Image captioning

    Error with Image captioning

    When I Run the evaluation command python sample.py --image='png/example.png'

    I got this error

    Traceback (most recent call last): File "sample.py", line 97, in main(args) File "sample.py", line 61, in main sampled_ids = decoder.sample(feature) File "/users/PAS1273/osu8235/pytorch/pytorch-tutorial/tutorials/03-advanced/image_captioning/model.py", line 62, in sample hiddens, states = self.lstm(inputs, states) # (batch_size, 1, hidden_size), File "/users/PAS1273/osu8235/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in call result = self.forward(*input, **kwargs) File "/users/PAS1273/osu8235/.local/lib/python2.7/site-packages/torch/nn/modules/rnn.py", line 162, in forward output, hidden = func(input, self.all_weights, hx) File "/users/PAS1273/osu8235/.local/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 351, in forward return func(input, *fargs, **fkwargs) File "/users/PAS1273/osu8235/.local/lib/python2.7/site-packages/torch/autograd/function.py", line 284, in _do_forward flat_output = super(NestedIOFunction, self)._do_forward(*flat_input) File "/users/PAS1273/osu8235/.local/lib/python2.7/site-packages/torch/autograd/function.py", line 306, in forward result = self.forward_extended(*nested_tensors) File "/users/PAS1273/osu8235/.local/lib/python2.7/site-packages/torch/nn/_functions/rnn.py", line 293, in forward_extended cudnn.rnn.forward(self, input, hx, weight, output, hy) File "/users/PAS1273/osu8235/.local/lib/python2.7/site-packages/torch/backends/cudnn/rnn.py", line 208, in forward 'input must have 3 dimensions, got {}'.format(input.dim())) RuntimeError: input must have 3 dimensions, got 2

  • some question about the position of 'optimizer.zero_grad()'

    some question about the position of 'optimizer.zero_grad()'

    I think the correct way the code the training is that

        optimizer.zero_grad()
        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)
        
        # Backward and optimize
        loss.backward()
        optimizer.step()
    

    not that

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)
        
        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    
  • main.py failed

    main.py failed

    https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/01-basics/pytorch_basics/main.py:

    [email protected]:~/dev-learn/gpu/pytorch/tutorial# python3 pytorch-main.py
    tensor(2.)
    tensor(1.)
    tensor(1.)
    w:  Parameter containing:
    tensor([[-0.4404, -0.1190,  0.4829],
            [ 0.4956, -0.3760,  0.1435]], requires_grad=True)
    b:  Parameter containing:
    tensor([0.4760, 0.5684], requires_grad=True)
    loss:  1.0722367763519287
    dL/dw:  tensor([[-0.4775,  0.0949,  0.0065],
            [ 0.0698, -0.4295, -0.0664]])
    dL/db:  tensor([0.3632, 0.3266])
    loss after 1 step optimization:  1.0655767917633057
    Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ../../data/cifar-10-python.tar.gz
    170499072it [00:32, 5230791.96it/s]
    Extracting ../../data/cifar-10-python.tar.gz to ../../data/
    torch.Size([3, 32, 32])
    6
    Traceback (most recent call last):
      File "pytorch-main.py", line 157, in <module>
        shuffle=True)
      File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 270, in __init__
        sampler = RandomSampler(dataset, generator=generator)  # type: ignore[arg-type]
      File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/sampler.py", line 103, in __init__
        "value, but got num_samples={}".format(self.num_samples))
    ValueError: num_samples should be a positive integer value, but got num_samples=0
    [email protected]:~/dev-learn/gpu/pytorch/tutorial#
    
    
  • ValueError: num_samples should be a positive integer value, but got num_samples=0

    ValueError: num_samples should be a positive integer value, but got num_samples=0

    Traceback (most recent call last): File "D:/PycharmWorkspace/pytorch-tutorial/tutorials/01-basics/pytorch_basics/main.py", line 154, in train_loader = torch.utils.data.DataLoader(dataset=custom_dataset, File "D:\anaconda3\envs\torch\lib\site-packages\torch\utils\data\dataloader.py", line 262, in init sampler = RandomSampler(dataset, generator=generator) # type: ignore File "D:\anaconda3\envs\torch\lib\site-packages\torch\utils\data\sampler.py", line 103, in init raise ValueError("num_samples should be a positive integer " ValueError: num_samples should be a positive integer value, but got num_samples=0

  • AttributeError: module 'torch.nn' has no attribute 'linear'

    AttributeError: module 'torch.nn' has no attribute 'linear'

    #=============================================#

    2.Basic autogard example2

    #=============================================#

    Create tensors of shape (10,3) and (10,2)

    x = torch.randn(10, 3) y = torch.randn(10, 2)

    Build a fully connected layer

    linear = nn.linear(3, 2) print('w:', linear.weight) print('b:', linear.bias)

    Build loss function and optimizer

    loss = nn.MSELoss() optimizer = torch.optim.SGD(linear.parameters(), lr=0.01)

    Please help answer this question, thank you very much!

Open source guides/codes for mastering deep learning to deploying deep learning in production in PyTorch, Python, C++ and more.
Open source guides/codes for mastering deep learning to deploying deep learning in production in PyTorch, Python, C++ and more.

Deep Learning Materials by Deep Learning Wizard Start Learning Now Please head to www.deeplearningwizard.com to start learning! It is mobile/tablet fr

Jun 25, 2022
Deep Learning (with PyTorch)

Deep Learning (with PyTorch) This notebook repository now has a companion website, where all the course material can be found in video and textual for

Jun 27, 2022
Pytorch implementations of various Deep NLP models in cs-224n(Stanford Univ)

DeepNLP-models-Pytorch Pytorch implementations of various Deep NLP models in cs-224n(Stanford Univ: NLP with Deep Learning) This is not for Pytorch be

Jun 27, 2022
Interactive deep learning book with multi-framework code, math, and discussions. Adopted at 200 universities.
Interactive deep learning book with multi-framework code, math, and discussions. Adopted at 200 universities.

D2L.ai: Interactive Deep Learning Book with Multi-Framework Code, Math, and Discussions Book website | STAT 157 Course at UC Berkeley | Latest version

Jun 30, 2022
A collection of various deep learning architectures, models, and tips

Deep Learning Models A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks. Traditiona

Jun 27, 2022
A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.

PyTorch Examples WARNING: if you fork this repo, github actions will run daily on it. To disable this, go to /examples/settings/actions and Disable Ac

Jun 26, 2022
A scalable template for PyTorch projects, with examples in Image Segmentation, Object classification, GANs and Reinforcement Learning.
A scalable template for PyTorch projects, with examples in Image Segmentation, Object classification, GANs and Reinforcement Learning.

PyTorch Project Template is being sponsored by the following tool; please help to support us by taking a look and signing up to a free trial PyTorch P

Jun 26, 2022
PyTorch tutorials.

PyTorch Tutorials All the tutorials are now presented as sphinx style documentation at: https://pytorch.org/tutorials Contributing We use sphinx-galle

Jul 2, 2022
C++ Implementation of PyTorch Tutorials for Everyone
C++ Implementation of PyTorch Tutorials for Everyone

C++ Implementation of PyTorch Tutorials for Everyone OS (Compiler)\LibTorch 1.9.0 macOS (clang 10.0, 11.0, 12.0) Linux (gcc 8, 9, 10, 11) Windows (msv

Jul 5, 2022
Simple examples to introduce PyTorch

This repository introduces the fundamental concepts of PyTorch through self-contained examples. At its core, PyTorch provides two main features: An n-

Jun 27, 2022
Minimal tutorials for PyTorch

Minimal tutorials for PyTorch adapted from Alec Radford's Theano tutorials. Tensor multiplication Linear Regression Logistic Regression Neural Network

May 14, 2022
PyTorch Implementation of Fully Convolutional Networks. (Training code to reproduce the original result is available.)
PyTorch Implementation of Fully Convolutional Networks. (Training code to reproduce the original result is available.)

pytorch-fcn PyTorch implementation of Fully Convolutional Networks. Requirements pytorch >= 0.2.0 torchvision >= 0.1.8 fcn >= 6.1.5 Pillow scipy tqdm

Jun 27, 2022
Simple PyTorch Tutorials Zero to ALL!

PyTorchZeroToAll Quick 3~4 day lecture materials for HKUST students. Video Lectures: (RNN TBA) Youtube Bilibili Slides Lecture Slides @GoogleDrive If

Jun 27, 2022
PyTorch tutorials and best practices.

Effective PyTorch Table of Contents Part I: PyTorch Fundamentals PyTorch basics Encapsulate your model with Modules Broadcasting the good and the ugly

Jun 30, 2022
Some example scripts on pytorch

pytorch-practice Some example scripts on pytorch CONLL 2000 Chunking task Uses BiLSTM CRF loss with char CNN embeddings. To run use: cd data/conll2000

Jun 8, 2022
ConvNet training using pytorch

Convolutional networks using PyTorch This is a complete training example for Deep Convolutional Networks on various datasets (ImageNet, Cifar10, Cifar

Jun 18, 2022
simple generative adversarial network (GAN) using PyTorch

Generative Adversarial Networks (GANs) in PyTorch Running Run the sample code by typing: ./gan_pytorch.py ...and you'll train two nets to battle it o

Jun 14, 2020
Torch Containers simplified in PyTorch
Torch Containers simplified in PyTorch

pytorch-containers This repository aims to help former Torchies more seamlessly transition to the "Containerless" world of PyTorch by providing a list

Apr 25, 2022
Example of network fine-tuning in pytorch for the kaggle competition Dogs vs. Cats Redux: Kernels Edition

Example of network fine-tuning in pytorch for the kaggle competition Dogs vs. Cats Redux: Kernels Edition Currently

Apr 26, 2022