Chapter II advanced python of artificial intelligence - numpy Library

1、numpy Library Introduction

1.1、Numpy Library overview

• Mainly used in relation to Multidimensional arrays perform calculations , It's a very Efficient A package for processing numerical operations
• characteristic
  • 1、numpy Built in at the bottom Parallel operation function , When the system has multiple cores , When doing some kind of calculation ,numpy Will automatically do parallel computing
  • 2、Numpy Bottom use C Language writing , The interior has been lifted GIL( Global interpreter lock ), The speed of operations on arrays is not affected by Python The limitations of the interpreter , The efficiency is much higher than that of pure Python Code .
  • 3、 There is a strong N Dimensional array object Array（ Something like a list ）
  • 4、 Linear algebra used 、 Fourier transform and random generating function

1.2、numpy Array and Python List performance comparison

• Use python List create random number

# Get the current timestamp 
t1 = time.time()
a = []
for x in range(1000000):
a.append(x**2)
t2 = time.time()
print(t2-t1)

• Query results

• Use numpy Create a random number

t3 = time.time()
b = np.arange(1000000)**2
t4 = time.time()
print(t4-t3)

• Query results

1.3、 Array and list comparison

• numpy The array in uses the following Python The list in is very similar , The difference between the following
  • 1、 One list Can be stored in Multiple data types , and Array Can only store Same data type
  • 2、 Array It can be multidimensional , When all in the array data type All are Numerical type When , amount to Matrix in linear algebra , It can be done Operations between each other Of

2、 How to create an array

• numpy Often deal with arrays , So the first step is to learn how to create arrays . stay numpy The data type of the array in is called ndarray

2.1、 Use np.array establish ( list )

b = np.array([1,2,3,4])
print(b)
print(type(b))

• Running results

2.2、 Use np.arange establish ( The arithmetic sequence )

arr2 = np.arange(0,10,2)
print(arr2)
print(type(arr2))

• Running results

2.3、 Use np.random establish ( random number )

arr3 = np.random.random(10)
print(arr3)
print(type(arr3))

• Running results

2.4、 Use the function to create ( Special number )

a1 = np.zeros((2,2))
a2 = np.ones((3,2))
a3 = np.full((2,2),8) # Generate a that all elements are 8 Of 2 That's ok 2 Array of columns 
a4 = np.eye(3) # Generate an element on an oblique square that is 1, The rest of the elements are 0 Of 3*3 Matrix 
print(a1)
print(a2)
print(a3)
print(a4)

• Running results
  

3、 Array common properties

3.1、 Array data type

bool,int8,int16,int32,int64,uint8,uint16,uint32,uint64
object_,string_,unicode_

• Example

c = np.array([1,2,3,4],dtype=np.float16)
print(c)
print(c.dtype)

• Running results

3.2、 Data type conversion

• Type conversion ： After the file is loaded , To reduce memory space , Convert its data type

import numpy as np
a1 = np.array([1,2,3])
print(a1.dtype) # windows To think in silence int32
a2 = a1.astype(np.int8) #astype The array itself will not be modified , 20. Return the modified data results 
print(a2)

• Query results

4、 Multidimensional arrays

4.1、 The creation of multidimensional arrays

shape: View array dimensions
ndim: View array dimensions
size: Check the number of array elements
itemsize: View the size of each element in the array , Unit is byte

• Example

a1 = np.array([1,2,3,4])
a2 = np.array([[1,2,3,4],[5,6,7,8]])
a3 = np.array([[[1,2,3,4],[1,2,3,4]],[[1,2,3,4],[1,2,3,4]]])
print(a1.shape)
print(a2.shape)
print(a3.shape)

• Query results

4.2、 Dimension transformation

reshape: Dimension transformation , The array itself will not be modified
resize: Dimension transformation , Will modify the array itself
flatten: Convert dimensions to one dimension , Returns a copy of the original array
ravel: Dimension is converted to one dimension , Returns the original array reference

• Sample a

a4 = a3.reshape((4,4))
print(a4)
print(a4.shape)

• Query results

• Example 2

a1 = np.random.randint(0,5,size=(3,2,2))
a2 = np.random.randint(0,5,size=(2,2))
print(a1.flatten())
print("========")
print(a1)
print("========")
print(a2.ravel())
print("========")
print(a2)
print("========")
print(a1+a2)

• Query results

• Development commonly used - Convert to a one-dimensional array

a5 = a3.reshape((16,))
print(a5)
print(a5.shape)

• Query results

4.3、axis Axis understanding

• The outermost brackets represent axis=0, The brackets in turn correspond to axis The count of is incremented in turn 1

• Mode of operation ： If you specify an axis for related operations , Then, each direct child element under the axis will be used for related operations

• Example

arr1 = np.arange(0,6).reshape(2,3)
arr2 = np.arange(6,12).reshape(2,3)
print(arr1+arr2)

• Query results

4.4、 Multidimensional array summary

1、 Arrays are usually very complex when they reach three dimensions , It's not convenient to calculate , So we usually put 3 An array of dimensions or more is converted to 2 Dimension group
2、ndarray.ndim: View array dimensions
3、ndarray.shape: View array shapes ,shape It's a tuple , There are several elements in the array that represent several dimensions
4、ndarray.reshape: Modify the shape of the array , The total number of elements before and after modification must be consistent
5、ndarray.size: See the total number of elements in the array
6、ndarray.itemsize: View the memory size of each element of the array

5、 Slicing and indexing of arrays

5.1、 Slice and index of one-dimensional array

 Include :[ The starting position : End position : step ]

• Example

a6 = np.arange(10)
print(a6)
print(a6[1])
print(a6[0:4])
print(a6[0:6:2])
print(a6[::-1])

• Query results

5.2、 Slicing and indexing of multidimensional arrays

 Multidimensional arrays :
Brackets a value : On behalf of the line
Two values in brackets : Comma separated , Everything is good. The front is OK , After the comma is the column , The result is a one-dimensional array

• Example

a7 = np.random.randint(0,10,size=(4,6))
print(a7)
print("============")
print(a7[1])
print("============")
print(a7[0:2])
print("============")
print(a7[[0,2,3]])
print("============")
print(a7[0:2,0])

• Query results

5.3、 Boolean index

 Boolean index :<,>,<=,>=,==,!=,&,|

• Example ： Will array a7 All less than 5 And greater than 3 All the data are extracted

a8 = [(a7 < 5) & (a7>3)]
print(a8)
print("============")
print(a7[a8])

• Query results

• matters needing attention

 Boolean index : Through... On the same array True still False To extract
Multiple extraction conditions : & Represents and , | For or on behalf of , Each condition should be enclosed in parentheses

6、 Substitution of array values

6.1、 Index replacement

• Example

a7[a7 < 3] = 1
a7

• Query results

6.2、 Function substitution

np.where( Conditions ,0,1)

• Example

result = np.where(a7 > 5 ,0,1)
result

• Query results

7、 Array broadcast mechanism

7.1、 Array and constant operation

• Example

a1 = np.arange(10).reshape(2,5)
print(a1)
a2 = a1 + 10
print(a2)

• Query results

7.2、 Array and array operation

（1） Same structure operation

 Direct additive

• Example

a1 = np.arange(10).reshape(2,5)
print(a1)
print("=======")
a2 = a1 + 10
print(a2)
print("=======")
a3 = a1 + a2
print(a3)
print("=======")

• Query results

（2） Different structural operations

1、 The length of one of the two arrays is 1, Then the broadcast is compatible
2、 The axis length of the dimension from the end of the two arrays is consistent , Then the broadcast is compatible

• Sample a

a1 = np.random.randint(0,5,size=(3,8,2))
a2 = np.random.randint(0,5,size=(8,1))
print(a1)
print("========")
print(a2)
print("========")
print(a1+a2)

• Query results

• Example 2

a1 = np.random.randint(0,5,size=(3,2,2))
a2 = np.random.randint(0,5,size=(2,2))
print(a1)
print("========")
print(a2)
print("========")
print(a1+a2)

• Query results

7.3、 Array overlay

（1） Stack vertically

vstack: The array is stacked vertically ,
Prerequisite : The number of array columns must be the same

• Example

a1 = np.random.randint(0,5,size=(2,2))
a2 = np.random.randint(0,5,size=(2,2))
a3 = np.vstack([a1,a2])
print(a3)

• Query results

（2） Stack horizontally

hstack: Stack the array horizontally
Prerequisite : The rows of the array must be the same to overlay

• Example

a1 = np.random.randint(0,5,size=(2,2))
a2 = np.random.randint(0,5,size=(2,2))
a3 = np.hstack([a1,a2])
print(a3)

• Query results

（3） Specify manually

concatenate: Stack the two arrays , The specific direction depends on the parameters axis

• Example

a1 = np.random.randint(0,5,size=(2,2))
a2 = np.random.randint(0,5,size=(2,2))
a3 = np.concatenate([a1,a2],axis=None)
a4 = np.concatenate([a1,a2],axis=1)
a5 = np.concatenate([a1,a2],axis=0)
print(a3)
print(a4)
print(a5)

• Query results

7.4、 Array cutting

（1） Cut horizontally

hsplit: Cut horizontally

• Example

a1 = np.random.randint(0,5,size=(2,2))
np.hsplit(a1,2)

• Query results

（2） Vertical cutting

vsplit: Split vertically

• Example

a1 = np.random.randint(0,5,size=(2,2))
np.vsplit(a1,2)

• Query results

（3） Specify the direction

split: Cut in the specified direction , Parameters axis0 For the line ,1 Column

• Example

a1 = np.random.randint(0,5,size=(2,2))
a2 = np.split(a1,2,axis=0)
print(a2)
a3 = np.split(a1,2,axis=1)
print(a3)

• Query results

7.5、 Array transposition

 Transposition : Matrix inner product calculation

• Example

a1 = np.random.randint(0,5,size=(2,2))
print(a1)
print("=====")
print(a1.T)
print("=====")
print(a1.dot(a1.T))

• Query results

8、 The depth of the array / Shallow copy

 No copy : Simple assignments do not copy
Shallow copy : Variable copy , Points to the same memory space
Deep copy :

• Example

a = np.random.randint(0,5,size=(2,2))
b = a
print(b is a)
print("======")
c = a.view()
print(c is a)
c[[0],[0]] = 100
print(c)
print(a)
print("======")
d = a.copy()
d[[1],[1]] = 10
print(d)
print(a)

• Query results

9、 File operations

numpy File format :npy perhaps npz ending

9.1、 Read the file

np.savetxt: Save file format , Top mark , Separator

• Example

a = np.random.randint(0,100,size=(4,4))
np.savetxt("a.csv",a,delimiter=",",header="a,b,c,d")

• Query results

9.2、 file save

np.loadtxt: data type , Separator , Jump

• Example

data = np.loadtxt("b.csv",delimiter=",")
data

• Query results

9.3、 Read non text files

np.load: Read non text file format , It can be used for three-dimensional arrays

• Example

np.load("a.npy")

• Query results

9.4、 Save non text file

np.save: Save non text file format , It can be used for three-dimensional arrays

• Example

a = np.random.randint(0,100,size=(4,4,4))
np.save("a.npy",a)

• Query results

9.5、 write in CSV file

with open( File name , read , Encoding mode ) as Object name :
reader = csv.reader(fp)
reader = csv.DictReader(fp)

• CSV file

CSV File features
:1、 Pure text , Use a character set , such as ASCII、Unicode
:2、 It's made up of records
:3、 Each record is divided into fields by a delimiter ( Typical delimiters are commas , A semicolon , tabs )
:4、 Each field has the same field sequence

• Example - iterator （ Tabular form ） Read file contents

import csv
with open('b.csv','r') as fp:
#reader It's an iterator 
reader = csv.reader(fp)
#next Take out the data pointed to by the first pointer , Then move down 
titles = next(reader)
print(titles)
print("=========")
for x in reader:
print(x)
print("========")
a = x[0]
b = x[1]
print(a+b+" result ")

• Query results

• Example - iterator （ The dictionary form ） Read file contents

with open("b.csv",'r') as fp:
reader = csv.DictReader(fp)
for x in reader:
print(x)

• Query results

9.6、 Read CSV file

with open( File name , write in , Coding format ):
writer = csv.writer(fp)
writer.writerow # Write to a row 
writer.writerows # Write multiple rows 

• Example - Write multiple lines of data at one time

headers = {
"username","age","height"}
values = [(' Zhang San ',18,200),(' Li Si ',19,180)]
with open('c.csv','w',encoding='utf-8') as fp:
writer = csv.writer(fp)
writer.writerow(headers)
writer.writerows(values)

• Query results

10、NAN and INF Value processing

NAN:Not A Number, It's not a number , It belongs to floating point type
INF:Infinity, For infinity , Divisor is 0 When , Also with floating point types

• NAN Characteristics :

:1、NAN and NAN It's not equal , such as np.NAN 1= np.NAN This condition is true
:2、NAN And any value , The result is NAN

（1） Delete missing value

data[~np.isnan(data)] : Delete null , Convert an array to a one-dimensional array
np.delete(data,np.where(np.isnan(data))[0],axis=0) : Delete as NAN The line of

• Example - Delete null , Convert to one dimension

import numpy as np
# 1、 Delete all NAN Value , Because the array doesn't know how to change after deleting the value , So it becomes a one-dimensional array 
data = np.random.randint(0,10,size = (3,5)).astype(np.float)
data[0:1] = np.nan
data = data[~np.isnan(data)]
print(data)

• Query statement

• Example - Delete the row where the null value is located

import numpy as np
# 1、 Delete all NAN Value , Because the array doesn't know how to change after deleting the value , So it becomes a one-dimensional array 
data = np.random.randint(0,10,size = (3,5)).astype(np.float)
# take 0,1 and 1,2 The two values are set to NAN
data[[0,1],[1,2]] = np.nan
# Get which rows have NAN
lines = np.where(np.isnan(data))[0]
# Use delete Method to delete the specified row ,axis = 0 Said to delete ,lines A symbol indicating deletion 
data1 = np.delete(data,lines,axis=0)
print(data1)

• Query statement

（2） Replace with other values

1、 Replace the null value with 0
2、 Replace null with mean

• Example

#CSV The file vacancy value is an empty string , Limit string format when reading 
scores = np.loadtxt(" academic record .csv",delimiter = ',',encoding='utf-8',skiprows=1,dtype=np.str)
# Converts an empty string to NAN
scores[scores == ""] = np.NAN
scores.astype(np.float)
# Convert all strings to floating point type 
scores.astype(np.float)

• Query results

• matters needing attention

 Read CSV Null value in text :
:1、 First read as a string ,
:2、 The empty string is converted to NAN
:3、 Convert string to floating-point type

11、random modular

（1） Random number seed seed

• Integer value used to specify the start of the random number generation algorithm

 identical seed value : Generate the same random number every time
Not set up seed value : Select this value according to the system time

• Example

np.random.seed(1)
print(np.random.rand()) # Print fixed values 
print(np.random.rand()) # Print other values , Because the seed of random number only affects the next generation of random number 

• Query results

（2） Arrays of random numbers

• Example

print(np.random.rand(2,3,4)) # Generate two 3 That's ok 4 Array of columns , Values in 0-1 Between 

• Query results

（3） Arrays of random numbers - Standard normal distribution

• Example

print(np.random.randn(2,3)) # Generate a 2 That's ok 3 Array of columns , The values in the array are normally distributed 

• Query results

（4） Arrays of random numbers - Specified scope

• Example

print(np.random.randint(10,size=(3,6)))

• Query results

（5） Random sampling

 Prerequisite : Must be a one-dimensional array

• Example

data = [1,2,3,4,5,6,7,8,9,10]
result1 = np.random.choice(data,3)
result2 = np.random.choice(data,size=(2,2))
print(result1)
print("========")
print(result2)

• Query results

12、 The generic function

12.1、 One variable function

function describe np.abs The absolute value np.sqrt Open the root np.square square np.exp Calculate the index e*xnp.log,np.log10,np.log2,np.log1p Find logarithm np.sign Labelling ,1,0,-1

12.2、 Dual function

The name of the function describe np.add Add np.substract Subtraction np.negative negative , Add a symbol to each value np.multiply Multiplication np.divide division np.floor_divide integer np.mod Remainder

12.3、 Aggregate functions

The name of the function describe np.sum Elements and np.prod prod np.mean Element mean np.std Standard deviation of elements np.var Element variance np.min The minimum value of the element np.max The maximum value of the element np.median Element median np.argmax Element maximum index np.argmin Element minimum index

12.4、 Boolean judgment function

The name of the function Function description np.any Verify whether any element is true np.all Verify that all elements are true

• Example

data = np.random.randint(10,size=(3,4))
print(np.any(data == 5))
print(np.all(data == 5))

• Query results

12.5、 Sort

The name of the function Function description np.sort Specify the axis to sort , Default to the last axis np.argsort Returns the sorted subscript value

• Example

data = np.random.randint(10,size=(3,4))
data1 = np.sort(data)
data2 = np.sort(data,axis=0)
data3 = np.argsort(data)
print(data1)
print("========")
print(data2)
print("========")
print(data3)
print("========")

• Query results

12.6、 Other functions supplement

The name of the function Function description np.apply_along_axis Execute a function along an axis np.linespace Specify the average number of shares of values in the interval np.unique Returns the unique value in the array