explain

adopt Python, Take data from Mysql Move to Mongo in , And then from Mongo Read out processing in .

I find it difficult to name this thing , So I just picked up one . But the other way around , There must be some exploration value when the name is difficult to rise .

Overall , This part is practice All Data In Memory Ideas ： All data , It looks like it's in memory . External data , All fall on the machine Mongo library , Because of high-speed solid-state disk and large memory , Make the data appear to be persistent in memory .BTW, Soon I should be able to build a split Mongo colony , That can support more 、 Faster data throughput .

AIDM What are the benefits ？

• 1 Saving the program is the key , Used up terminals （ for example jupyter) It can be closed after use , No need to occupy running memory . Run again when necessary , The speed is very fast .
• 2 You can do much more complex analysis . Analysis no longer only keeps the latest version , Or manually maintain several versions . But rather MVD There is a form of , Support the management of multiple spaces in an algorithmic way . You can always go back , Branch .

This article changes the previous style , Talk about the principle , No more code .

Content

1 get data

1.1 Database connection configuration （ Source library ）

There are many database connections stored in my metabase , First determine which connection . Then conduct connectivity test .

Automatic search that will be forgotten Loss Auto Probe

This time, the configuration is determined by manual query , Then tune it out. . future , I think you can use LAP To replace the manual method .

• 1 General table names are already very unique