[python] text from url

import requests
import json
url = 'http://www.quotesondesign.com/wp-json/posts'
resp = requests.get(url)

# srting to Python data type
c_lst = json.loads(resp.text)

for txt in c_lst:
    # unpacking
    for key, value in txt.items():
        print(key, ' : ', value)

[python] json

import json

# Json String to variable
sample = {
    "glossary": {
        "title": "example glossary",
        "GlossDiv": {
            "title": "S",
            "GlossList": {
                "GlossEntry": {
                    "ID": "SGML",
                    "SortAs": "SGML",
                    "GlossTerm": "Standard Generalized Markup Language",
                    "Acronym": "SGML",
                    "Abbrev": "ISO 8879:1986",
                    "GlossDef": {
                        "para": "A meta-markup language, used to create markup languages such as DocBook.",
                        "GlossSeeAlso": ["GML", "XML"]
                    },
                    "GlossSee": "markup"
                }
            }
        }
    }
}

# string to Json encoding with formatting
sample_json = json.dumps(sample, indent = 4)
print(sample_json)
print('')

# Decoding Json String to Python type
sample_structured_json = json.loads(sample_json)
print(sample_structured_json)
print('')

print(sample_structured_json['glossary']['title'])
print('')

for i in sample_structured_json['glossary']['GlossDiv']['GlossList']['GlossEntry']:
    print(i)

[python] binary data write to file

# binary data generation
bdata = bytes(range(0, 256))

# check length and data
print('size :', len(bdata), '\ndata : ', bdata)

# set offset and chunk size, get data size
offset = 0
chunk = 10
size = len(bdata)

# write to file and get size
with open('bfile.bin', 'wb') as bf:
    while offset <= size:
        bf.write(bdata[offset:offset+chunk])
        offset += chunk

# check file
! cat bfile.bin

MongoDB

SQL vs. NoSQL Solution

NoSQL data stores

• Abandon SQL and ACID for simpler data and concurrency model
• BASE semantics as opposed to ACID gaurantees
• Basically Available
• Sofe state - system state may change over time even without input
• Eventually consistent
• Key-value, document, column-family, graph

MongoDB

charateristics

A document is the basic uint of data

A collection can be thought of as a table with a dynamic schema

Single instance can host multiple independent databases

each of which can have its own colletions

comes with a Javascript shell

document is an ordered set of keys with associated values

type-sensitive and case-sensitive

documents cannot contain duplicate keys

key-value paris in documents are ordered

collections

is a group of documents

have dynamic schemas

a single collection have any number of different shapes

databases

groups collections into databases

can host several databases

each database is stored in separate files on disk

to store all data for a single application in the same database

shell

show dbs, use db_name, show collections

db.createCollection("collection_name")

db.collection_name.find()  

.findOne()

.find({"address.building":"386", "address.building":"381"})

==> 조건이 중복될 경우 마지막 381 조건에 만족하는 결과만 return

.find({$and:[{"address.street":"Canal Street"},{"address.building":"386"}]})

.find({$or:[{"address.building":"386"},{"address.building":"382"}]})

.find({"address.street":"Canal Street"},{"address.building":1, _id:0})

//0:column 제외, 1:조회 단 _id는 default로 조회

.find({amount:{$lt:20, $gt:5}})

.insert({hello:'world'})

.insertOne({hello:'world'})

.insertMany({hello:'world'}, {good:'night'})

.update({title:'my blog'}, {title:'my blog name'})

.update({title:'my blog'}, {$set:{title:'my blog name'}})

.update({title:'my blog'}, {$unset:{title:''}})

attribute가 하나 였을 경우 record가 삭제됨

.update({title:'my blog'}, {$inc:{pageviews: 1}})

.update({title:'my blog'}, {$push:{name: "joe"}})

adds elements to the end of an array

.update({title:'my blog'}, {$push:{name: {$each:["bob", "kim", "lee"]}}})

use $addToSet to prevent duplicates

.update({title:'my blog'}, {$pop:{name: 1}})

removes it from the end, -1 from the begining

.update({title:'my blog'}, {$pull:{name: "lee"}})

.replaceOne({title:'my blog'},{header:'your blog'})

.remove()

if no parameter, remove all !

.remove({title:'my blog name'})

.deleteOne({title:'my blog name'})

.deleteMany({hello:'world'}, {good:'night'})

.drop()

deletes everyting, even the colletion

db.dropDatabase()

Data Types

• Arrays can contain different data types

{"things":["pie", 3.14]}

• embedded documents

• every documents stored in MongoDB must have an "_id" key
• all documents must be smaller than 16MB

for Data Scientist

Useful links

  1. scikit-learn [link]