Future's blog
0%

PyTorch Practical Hand-Written Modules Basics 1--Tensor basics (attributes, creation)

Posted on Edited on In

This series would not be a general PyTorch introduction or detailed tutorials. Instead, this would be a very practical introduction to some common basics needed for Implementing Hand-Written Modules.

This is the First Section of this series, we would like to introduce some tensor basics, including: tensor attributes, tensor creation, and some other things. All the things I mentioned will be practical, but not exhaustive.

1. Tensor attributes

We introduce 5 key attributes for torch.tensor a here:

1.1 a.shape

  • a.shape: Returns the shape of a. The return type is torch.Size. Example:
1
2
3
4
a = torch.randn(10, 20) # create a 10x20 tensor
a.shape
# OUTPUT
torch.Size([10, 20])

The torch.Size object supports some tricks:

1
2
3
4
5
# unpack
h, w = a.shape
h, w
# OUTPUT
(10, 20)
1
2
3
4
# unpack in function calls
print(*a.shape)
# OUTPUT
10 20

1.2 a.ndim

  • a.ndim: Returns number of dimensions of a.

It looks like len(a.shape). It also has a function version, called a.ndimension()

1
2
3
4
5
6
7
a.ndim
a.ndimension()
len(a.shape)
# OUTPUT
2
2
2

1.3 a.device

  • a.device: Returns where the a locates.
1
2
3
a.device
# OUTPUT
device(type='cpu')

Convert to CUDA by using a = a.to('cuda:0'). Convert back to CPU by using a = a.to('cpu') or a = a.cpu().

1.4 a.dtype

  • a.dtype: Returns the data type of a.

The data type of tensor a. It’s very important in PyTorch! Usually, the data type would be torch.float32 or torch.int64. Some data type convert method:

1
2
3
4
5
# to float32
f = a.float()
f.dtype
# OUTPUT
torch.float32
1
2
3
4
5
# to int64
l = a.long()
l.dtype
# OUTPUT
torch.int64
1
2
3
4
5
# to int32
i = a.int()
i.dtype
# OUTPUT
torch.int32
1
2
3
4
5
# Also, we can use .to() as well:
f = a.to(torch.float32)
f.dtype
# OUTPUT
torch.float32

1.5 a.numel

  • a.numel(): Returns number of elements in a. Usually used in counting number of parameters in the model.
1
2
3
a.numel()
# OUTPUT
200  # it's 10*20!
1
2
3
4
5
import torchvision
model = torchvision.models.resnet50()
sum([p.numel() for p in model.parameters()])
# OUTPUT
25557032

2. Tensor creation

PyTorch tensors plays key role in writing deep learning programs. Usually, tensor are from two types: data and auxiliary variables (e.g., masks).

2.1 From data

For the data tensor, they are usually converted from other packages, such as numpy. We have several methods to convert it to torch.tensor.

  • torch.tensor(arr) Returns a deep copy of arr, i.e., the storage data is independent with arr. (Very memory and time consuming, not recommended for most cases)

  • torch.from_numpy(arr) Returns a shallow copy tensor, i.e., the storage data is shared with arr.

  • torch.as_tensor(arr, dtype=..., device=...) If dtype and device is same as arr, then it behaves like torch.from_numpy() function, shallow copy. Otherwise, it acts like torch.tensor(), deep copy. So using the function is recommended.

2.2 Special tensors

For the special tensors, PyTorch provides some common methods:

  • Linear:

We have torch.linspace and torch.arange. They are easy to understand. Please see the docs linspace and arange.

  • Random:
1
2
3
torch.randn(1, 2) # normal distribution, shape 1x2
torch.rand(1, 2)  # uniform[0, 1) distribution, shape 1x2
torch.randint(0, 100, (1, 2)) # uniform[0, 100) distribution, shape 1x2

These functions also support passing in torch.Size() or a sequence as the size parameter. 

1
2
3
a = torch.randn(10, 10) # a is in shape 10x10
torch.randn(a.shape)    # good!
torch.randn([10, 10])   # good!
  • Special tensors:
1
2
3
torch.zeros(10, 10) # all zero tensor, shape 10x10
torch.ones(10, 10)  # all one tensor, shape 10x10
# By default, the dtype is float32.
  • xxx_like()

PyTorch has a series of function looks like xxx_like(), such as ones_like(), zeros_like(), randn_like(). These functions generates the tensor with the name, and the dtype and device and layout is same as the passing-in tensor.

torch.rand_like(input) is equivalent to torch.rand(input.size(), dtype=input.dtype, layout=input.layout, device=input.device).

An example:

1
2
3
4
5
6
7
arr = torch.tensor([1,2,3], dtype=torch.float64)
print(arr.shape) # torch.Size([3])
print(arr.dtype) # torch.float64

z = torch.zeros_like(arr)
print(z.shape)   # torch.Size([3])
print(z.dtype)   # torch.float64