In all kinds of python In the project , We often need to persistently store all kinds of... In the system python Data structure of , Commonly used, such as dictionaries . Especially in cloud service types python In the project , Some user authentication information and log information should be stored in the cache persistently or temporarily , The most typical example is to store users' data in a database token Information . In this article, we will focus on three types of python Persistent storage scheme , Namely json、pickle and python The database that comes with it sqlite3.
json Format data storage is also a common type in cloud service projects , It is very lightweight and easy to use , Here we show a case ： How to use json Format to store one with python The resulting Fibonacci sequence . Every element of the Fibonacci sequence , Are equal to the sum of the previous number and the previous number , namely ：
, The first two digits of the most common Fibonacci sequence are 1. The following is a sequence of Fibonacci numbers python Code ：
import json
number = {1:1, 2:1}
for i in range(3, 11):
number[i] = number[i - 1] + number[i - 2]
print (number)
The code execution results are as follows ：
[[email protected] store_class]$ python3 json_dic.py
{1: 1, 2: 1, 3: 2, 4: 3, 5: 5, 6: 8, 7: 13, 8: 21, 9: 34, 10: 55}
We can see from the results , The first 10 The Fibonacci number is 55. Next, let's take a look at such a dictionary , How to persistently store to json Format file , The following is an example of the use ：
import json
number = {1:1, 2:1}
for i in range(3, 11):
number[i] = number[i - 1] + number[i - 2]
with open('number.json', 'w') as file:
json.dump(number, file)
with open('number.json', 'r') as file:
data = json.load(file)
print (data)
Execute this python file , We can get the output shown below ：
[[email protected] store_class]$ python3 json_dic.py
{'1': 1, '2': 1, '3': 2, '4': 3, '5': 5, '6': 8, '7': 13, '8': 21, '9': 34, '10': 55}
Here, we find that a... Is generated in the current directory json The file of ：
[[email protected] store_class]$ ll
Total usage 8
-rw-r--r-- 1 dechin dechin 265 3 month 20 12:32 json_dic.py
-rw-r--r-- 1 dechin dechin 85 3 month 20 12:32 number.json
We can take a look at this json What data is stored in the file ：
[[email protected] store_class]$ cat number.json
{"1": 1, "2": 1, "3": 2, "4": 3, "5": 5, "6": 8, "7": 13, "8": 21, "9": 34, "10": 55}
After verifying that the relevant data has been persisted , At the same time, we also noticed a problem , When we generate the Fibonacci sequence , Indexes 1,2,3...
Using integer variables , But store it in json After the format , Into a string format . We can use the following cases to illustrate the difference ：
import json
number = {1:1, 2:1}
for i in range(3, 11):
number[i] = number[i - 1] + number[i - 2]
with open('number.json', 'w') as file:
json.dump(number, file)
with open('number.json', 'r') as file:
data = json.load(file)
print (data)
print (number[10])
print (data['10'])
print (data[10])
The output of the execution is as follows ：
[[email protected] store_class]$ python3 json_dic.py
{'1': 1, '2': 1, '3': 2, '4': 3, '5': 5, '6': 8, '7': 13, '8': 21, '9': 34, '10': 55}
55
55
Traceback (most recent call last):
File "json_dic.py", line 16, in <module>
print (data[10])
KeyError: 10
The output here has an error message , This is because we use integer index variables to find json The corresponding value in the stored dictionary object , However, these integer indexes have been converted into string indexes in the previous storage , Therefore, in fact, there is no integer key value in the stored object , Therefore, an error will be reported in the execution result , If you enter a string type key value , And successfully found the No 10 Fibonacci Numbers .
Information about Fibonacci series , It has been introduced in the previous chapter , Here we go straight into pickle Use case of ：
import pickle
number = {1:1, 2:1}
for i in range(3, 11):
number[i] = number[i - 1] + number[i - 2]
with open('number.pickle', 'wb') as file:
pickle.dump(number, file)
with open('number.pickle', 'rb') as file:
data = pickle.load(file)
print (data)
Here's a detail , stay json The file opening format we use in the format storage is w, And in the pickle Here we use the storage file opening format as wb,pickle Is also used for reading rb Binary read format of . The execution output of the above code is as follows ：
[[email protected] store_class]$ python3 pickle_dic.py
{1: 1, 2: 1, 3: 2, 4: 3, 5: 5, 6: 8, 7: 13, 8: 21, 9: 34, 10: 55}
Here we can find , from pickle The stored integer index in the dictionary format is also successfully stored , In the current directory, a file named number.pickle The file of is the object of persistent storage .
[[email protected] store_class]$ ll
Total usage 12
-rw-r--r-- 1 dechin dechin 320 3 month 20 12:45 json_dic.py
-rw-r--r-- 1 dechin dechin 85 3 month 20 12:46 number.json
-rw-r--r-- 1 dechin dechin 56 3 month 20 12:44 number.pickle
-rw-r--r-- 1 dechin dechin 279 3 month 20 12:44 pickle_dic.py
Be similar to json Persistent read validation in the format , We can also simply modify a similar pickle The case of ：
import pickle
number = {1:1, 2:1}
for i in range(3, 11):
number[i] = number[i - 1] + number[i - 2]
with open('number.pickle', 'wb') as file:
pickle.dump(number, file)
with open('number.pickle', 'rb') as file:
data = pickle.load(file)
print (data)
print (number[10])
print (data[10])
The results are shown below ：
{1: 1, 2: 1, 3: 2, 4: 3, 5: 5, 6: 8, 7: 13, 8: 21, 9: 34, 10: 55}
55
55
From the results, we found that the stored objects were successfully read in the same read format .
In common use Linux It comes with the operating system sqlite3 database , If it is windows and Mac Operating system of , You can install it according to the scheme given in this tutorial .
SQLite Is an in-process library , It's self-sufficient 、 serverless 、 Zero configuration 、 transactional SQL Database engine . It is a zero-configuration database , This means that it is different from other databases , You do not need to configure in the system .
Like other databases ,SQLite The engine is not a stand-alone process , You can connect statically or dynamically according to your application requirements .SQLite Direct access to its storage files .
At the same time python3 The library of usually comes with sqlite3, You don't need to install it yourself , Now let's use ipython Demonstrate how to python Use in sqlite3 database ：
[[email protected] store_class]$ ipython
Python 3.8.5 (default, Sep 4 2020, 07:30:14)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import sqlite3
In [2]: conn = sqlite3.connect('test_sqlite3.db') # If there is db File is read , Create without
In [3]: cur = conn.cursor()
In [8]: sql_test_1 = '''CREATE TABLE number
...: (i NUMBER,
...: n NUMBER);''' # Create a file called number Table of , There are two columns of data i and n As a key value pair
In [9]: cur.execute(sql_test_1) # Execute string instruction
Out[9]: <sqlite3.Cursor at 0x7f6fb14acc70>
In [10]: sql_test_2 = "INSERT INTO number VALUES(1,1)" # Insert new data
In [11]: cur.execute(sql_test_2)
Out[11]: <sqlite3.Cursor at 0x7f6fb14acc70>
In [12]: sql_test_2 = "INSERT INTO number VALUES(2,1)"
In [13]: sql_test_3 = "INSERT INTO number VALUES(2,1)"
In [14]: cur.execute(sql_test_3)
Out[14]: <sqlite3.Cursor at 0x7f6fb14acc70>
In [15]: sql_test_4 = "SELECT * FROM number WHERE i=1" # Retrieving data
In [16]: cur.execute(sql_test_4)
Out[16]: <sqlite3.Cursor at 0x7f6fb14acc70>
In [17]: cur.fetchall()
Out[17]: [(1, 1)]
In [18]: sql_test_5 = "SELECT * FROM number WHERE i>=1"
In [19]: cur.execute(sql_test_5)
Out[19]: <sqlite3.Cursor at 0x7f6fb14acc70>
In [20]: cur.fetchall() # Read and retrieve the return value
Out[20]: [(1, 1), (2, 1)]
In [21]: for i in range(3, 11):
...: sql_test_6 = "SELECT * FROM number WHERE i={}".format(i-1)
...: cur.execute(sql_test_6)
...: select_result1 = cur.fetchall()[0][1]
...: sql_test_7 = "SELECT * FROM number WHERE i={}".format(i-2)
...: cur.execute(sql_test_7)
...: select_result2 = cur.fetchall()[0][1]
...: cur.execute("INSERT INTO number VALUES({},{})".format(i, select_result1+select_res
...: ult2))
...:
In [22]: sql_test_8 = "SELECT * FROM number WHERE i>=1"
In [23]: cur.execute(sql_test_8)
Out[23]: <sqlite3.Cursor at 0x7f6fb14acc70>
In [24]: cur.fetchall()
Out[24]:
[(1, 1),
(2, 1),
(3, 2),
(4, 3),
(5, 5),
(6, 8),
(7, 13),
(8, 21),
(9, 34),
(10, 55)]
In [25]: exit() # sign out ipython
In the above example, we demonstrated how to use sqlite3 Create databases and tables , And some common operations on the contents of the table . After executing the above example , A new... Will be generated in the current directory db file ：
[[email protected] store_class]$ ll
Total usage 24
-rw-r--r-- 1 dechin dechin 320 3 month 20 12:45 json_dic.py
-rw-r--r-- 1 dechin dechin 85 3 month 20 12:46 number.json
-rw-r--r-- 1 dechin dechin 56 3 month 20 12:47 number.pickle
-rw-r--r-- 1 dechin dechin 315 3 month 20 12:47 pickle_dic.py
-rw-r--r-- 1 dechin dechin 8192 3 month 20 13:05 test_sqlite3.db
If an error occurs during operation as shown below , It means that other processes are occupying this db file , Therefore, a process will lock the database ：
Traceback (most recent call last):
File "sqlite3_dic.py", line 15, in <module>
cur.execute("INSERT INTO number VALUES(1,1)")
sqlite3.OperationalError: database is locked
The solution is , First use fuser Look at this. db Which user is using the file ：
[[email protected] store_class]$ fuser test_sqlite3.db
/home/dechin/projects/2021-python/store_class/test_sqlite3.db: 5120
We found that it was 5120 This process takes up database files , It is also this process that locks the database . Usually the reason for this is , stay python The database operation instruction executed in did not complete successfully , The process that caused the database did not end , And we can no longer enter new instructions into the database through this process . So we can only solve this problem by killing the process ：
[[email protected] store_class]$ kill -9 5120
There is also a point to note that , The database operations used above are not actually saved to the database file , Need to go through commit Then it will be saved to the database file . Next, we will use the Fibonacci sequence example to demonstrate the use of database operations ：
import sqlite3
from tqdm import trange
conn = sqlite3.connect('test_sqlite3.db')
cur = conn.cursor()
try:
sql_test_1 = '''CREATE TABLE number
(i NUMBER,
n NUMBER);'''
cur.execute(sql_test_1)
except:
pass
cur.execute("INSERT INTO number VALUES(1,1)")
cur.execute("INSERT INTO number VALUES(2,1)")
for i in trange(3, 11):
sql_test_6 = "SELECT * FROM number WHERE i={}".format(i - 1)
cur.execute(sql_test_6)
select_result1 = cur.fetchall()[0][1]
sql_test_7 = "SELECT * FROM number WHERE i={}".format(i - 2)
cur.execute(sql_test_7)
select_result2 = cur.fetchall()[0][1]
cur.execute("INSERT INTO number VALUES({},{})".format(i, select_result1 + select_result2))
cur.execute("SELECT * FROM number WHERE i=10")
print (cur.fetchall())
conn.commit()
cur.close()
conn.close()
In the above use cases, we added commit Operation and close operation , On the one hand, the data is persisted , On the other hand, it also avoids the database locking problem mentioned above due to problems in other parts of the program . Let's take a look at the execution output of this use case ：
[[email protected] store_class]$ python3 sqlite3_dic.py
100%|█████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 31775.03it/s]
[(10, 55)]
The first 10 Fibonacci numbers are successfully output , In the output of the database , The format used is a list containing multiple tuples . Each tuple represents a key value pair that meets the search conditions , The elements in each tuple represent the values of each column .
The problem of persistent storage was mentioned earlier , We also use a simple example to verify what just happened commit Whether it has been successfully saved ：
'''
No one answers the problems encountered in learning ？ Xiaobian created a Python Learning exchange group ：153708845
Looking for small partners who share the same aspiration , Help each other , There are also good video tutorials and PDF e-book ！
'''
import sqlite3
from tqdm import trange
conn = sqlite3.connect('test_sqlite3.db')
cur = conn.cursor()
cur.execute("SELECT * FROM number WHERE i=10")
print (cur.fetchall())
conn.commit()
cur.close()
conn.close()
The execution output is as follows ：
[[email protected] store_class]$ python3 test_recall_sqlite3.py
[(10, 55)]
This result shows that the Fibonacci sequence stored previously has been persisted in the database file , As long as we link to the database, we can read the data at any time .
This paper introduces three python Dictionary object persistent storage scheme , contain json、pickle And the database sqlite, And with a real case Fibonacci series to demonstrate the use of different solutions . In fact, the three schemes here have their own advantages and disadvantages , The recommended usage scenarios are ： In lightweight 、 It can be mainly used in daily use json Format to store objects , We can also view it directly on the system json File content in format ; In the case of multi-user or multi process use , Recommended pickle The plan , Can be higher performance 、 Lower overhead persistent storage python object ; If it is necessary to provide external services , We recommend that you can directly use sqlite, A database query solution can be provided externally , There is no need to store a large amount of data locally or it is more convenient to process large-scale data .
