[python] binary data write to file

# binary data generation
bdata = bytes(range(0, 256))

# check length and data
print('size :', len(bdata), '\ndata : ', bdata)

# set offset and chunk size, get data size
offset = 0
chunk = 10
size = len(bdata)

# write to file and get size
with open('bfile.bin', 'wb') as bf:
    while offset <= size:
        bf.write(bdata[offset:offset+chunk])
        offset += chunk

# check file
! cat bfile.bin

MongoDB

SQL vs. NoSQL Solution

NoSQL data stores

• Abandon SQL and ACID for simpler data and concurrency model
• BASE semantics as opposed to ACID gaurantees
• Basically Available
• Sofe state - system state may change over time even without input
• Eventually consistent
• Key-value, document, column-family, graph

MongoDB

charateristics

A document is the basic uint of data

A collection can be thought of as a table with a dynamic schema

Single instance can host multiple independent databases

each of which can have its own colletions

comes with a Javascript shell

document is an ordered set of keys with associated values

type-sensitive and case-sensitive

documents cannot contain duplicate keys

key-value paris in documents are ordered

collections

is a group of documents

have dynamic schemas

a single collection have any number of different shapes

databases

groups collections into databases

can host several databases

each database is stored in separate files on disk

to store all data for a single application in the same database

shell

show dbs, use db_name, show collections

db.createCollection("collection_name")

db.collection_name.find()  

.findOne()

.find({"address.building":"386", "address.building":"381"})

==> 조건이 중복될 경우 마지막 381 조건에 만족하는 결과만 return

.find({$and:[{"address.street":"Canal Street"},{"address.building":"386"}]})

.find({$or:[{"address.building":"386"},{"address.building":"382"}]})

.find({"address.street":"Canal Street"},{"address.building":1, _id:0})

//0:column 제외, 1:조회 단 _id는 default로 조회

.find({amount:{$lt:20, $gt:5}})

.insert({hello:'world'})

.insertOne({hello:'world'})

.insertMany({hello:'world'}, {good:'night'})

.update({title:'my blog'}, {title:'my blog name'})

.update({title:'my blog'}, {$set:{title:'my blog name'}})

.update({title:'my blog'}, {$unset:{title:''}})

attribute가 하나 였을 경우 record가 삭제됨

.update({title:'my blog'}, {$inc:{pageviews: 1}})

.update({title:'my blog'}, {$push:{name: "joe"}})

adds elements to the end of an array

.update({title:'my blog'}, {$push:{name: {$each:["bob", "kim", "lee"]}}})

use $addToSet to prevent duplicates

.update({title:'my blog'}, {$pop:{name: 1}})

removes it from the end, -1 from the begining

.update({title:'my blog'}, {$pull:{name: "lee"}})

.replaceOne({title:'my blog'},{header:'your blog'})

.remove()

if no parameter, remove all !

.remove({title:'my blog name'})

.deleteOne({title:'my blog name'})

.deleteMany({hello:'world'}, {good:'night'})

.drop()

deletes everyting, even the colletion

db.dropDatabase()

Data Types

• Arrays can contain different data types

{"things":["pie", 3.14]}

• embedded documents

• every documents stored in MongoDB must have an "_id" key
• all documents must be smaller than 16MB

for Data Scientist

Useful links

  1. scikit-learn [link]

Data Visualization

1. Information Visualization

Scientific Visualization : 일상의 내용을 visualization

Information Visualization : abstract data를 visualization

1. Definition

Provide tools that present data in a way to help understand and gain insight from it

2. InfoVis is interdisciplinary

graphics

cognitive psychology

HCI : using users and tasks to guide design and evaluation

3. Expressiveness and Effectiveness

Expressiveness : Vis idiom should express all of, and only, the information in the dataset attributes

Effectiveness : Most important attributes should be encoded with the most effective channels --> ranking of channels

correctness, accuracy, ant truth

4. Stevens' Power Law

감각 자극과 감각 경험 간의 관계가 지수 함수로 표시된다는 공식화를 말한다. 감각의 종류에 따라 지수가 다르다

   , p : perceived magnitude, a : actual magnitude,

              length judgement  1

              area judgement     1

              volume judgement 1

5. Relative vs. Absolute Judgements

perceptual system mostly operates with relative judgements, not absolute

Weber's Law : the perceived change in stimuli is proportional to the initial stimuli

6. Preattentive Processing : cognitive operations done preattentively, without the need for focused attention

"pop out" of a display : easily detected regardless of the number of distractors

target detection, boundry detection, region tracking, counting

7. Design Guidelines/Principles

Visual Information Seeking Mantra

Overview first, zoom and filter, details on demand

8. Measuring Misrepresentation

9. Design Principles

avoid chartjunk

use small multiples

utilize narratives of space and time

2. Visualization Analysis and Design
1. Definitions and Motivations

computer-based visualization systems provide visual representations of datasets designed to help people carry out (some) tasks more effectively

2. Big picture
1. VIS is suitable when there is a need to argument human capabilites
2. design visual representations to help people perform task more effectively
3. design space is HUGE!
4. resource limitations
5. analysis instance

WHY the user needs it, WHAT data is shown, HOW the idiom is designed

3. Transitional use

gain a clear understanding of user's task --> purely computational solution --> monitoring automatic system is doing right

4. Long-term use
1. Exploratory analysis
2. Vis tool for presentation
5. Why use interactivity?
1. impossible to show everything at once
2. handling complexity and volume
6. Visualization Idioms : distinct approach to creating and manipluating visual representations

a tool that serves well for one task can be poorly to another

3. Analysis : Four Levels for Validation

so many possible ways.

Four Levels of vis design

  output of upstream level --> input to downstream level

    : upstream errors inevitably cascade down

Four kinds of threats to validate

1. wrong problem : they don't do that
2. wrong abstraction : showing wrong thing
3. wrong encoding/interaction technique : way you show don't work
4. wrong algorithm : code is too slow

--> proto-typing 을 통해서 미리 validation을 할 수 있다.

4. What : Data Abstraction

• semantics : real-world meaning
• attribute, item, link, grid, position
• set : unordered group of items
• list : ordered group of items
• cluster : grouping based on attribute similarity
• path : ordered set + links connecting nodes

• data abstraction : domain-specific to GENERIC

5. Why: Task Abstraction
• task abstraction should guide data abstraction
• Analyze > Search > Query
• Discover, Present, Enjoy, Annotate, Record, Derive(경험이 많을 수록 잘 사용 -> visualization idiom 사용의 폭이 넓어짐)

6. Marks and Channels
• idiom can be break down into Marks and Channels
• Mark
• basic graphical element : point(0D), line(1D), area(2D), volumn(3D)
• Items / Nodes
• Links

• Channel : a way to control the apperance of marks

• Expressiveness types and effectiveness rankings(1,2 순위는 외울것)

 * Better encoding ?

Bar size로 인하여 국가 간의 우선순위 혹은 우열을 내제적으로 표현하고 있어 불필요한 정보를 전달하고 있다.

이 Length를 변경하는데 Stevens' power law에 따라 effectiveness ranking 이 높은 position으로 변경하여 이를 제거한다.

• position dominates the user's mental model

1. Preattentive processing
1. cognitive operation done preattentively, without the need for focused attention
2. popout, segmentation effects
1. many channels : tilt, size, shape, proximity, shadow direction, but not parallel line.

2. Gestalt Psychology

인간은 자신이 본 것을 조직화하려는 기본 성향을 가지고 있으며, 전체는 부분의 합 이상이라는 점을 강조하는 심리학.

1. proximity : same spatial region
2. similarity : same value as other categorical channels (color hue, motion, shape)
3. connectedness : 모든 channel을 압도하는 가장 강력한 표현
4. continuity : 물리적으로 가장 단순하고 이해하기 쉬운 방향으로 이해한다.
5. common fate : things moving together

3. luminance contrast : Perception of color and luminance is contextual, based on relative judgements

7. Perception and Visual Patterns
1. Gestalt Principles
1. Grouping : avoid explicit grouping
1. Proximity
2. Similarity
3. Continuity
4. Common Fate

2. Perception of Forms
1. Closure : form complete, closed figures to increase regularity
2. Area/Figure and Ground/Relative size : smaller one as figure, larger one as ground
3. Symmetry : symmetry images are perceived collectively, even in spite of distance

2. Fixation-Saccade Cycle
1. Fixation : brief stationary period when detail information is acquired
2. saccade : flicking rapidly to a new location
3. postattentive amnesia

8. Rule of Thumbs
1. Overview
1. no unjustified 2D / 3D
2. eyes beats memory
3. resolution over immersion
4. function first, form next

9. Interaction Design and Design Principles
1. fundamental design goals is provide the right affordances, good mapping
1. right affordance : perceived and actual properies
2. right conceptual model
3. good mapping : a relationship between controls and their movements or effects
4. casuality : iterpesentation of feedback

D3를 이해하기 위한 도움 정보

1) 셀렉션 작동 원리 [link]