for Data Scientist

Useful links

  1. scikit-learn [link]

Data Visualization

1. Information Visualization

Scientific Visualization : 일상의 내용을 visualization

Information Visualization : abstract data를 visualization

1. Definition

Provide tools that present data in a way to help understand and gain insight from it

2. InfoVis is interdisciplinary

graphics

cognitive psychology

HCI : using users and tasks to guide design and evaluation

3. Expressiveness and Effectiveness

Expressiveness : Vis idiom should express all of, and only, the information in the dataset attributes

Effectiveness : Most important attributes should be encoded with the most effective channels --> ranking of channels

correctness, accuracy, ant truth

4. Stevens' Power Law

감각 자극과 감각 경험 간의 관계가 지수 함수로 표시된다는 공식화를 말한다. 감각의 종류에 따라 지수가 다르다

   , p : perceived magnitude, a : actual magnitude,

              length judgement  1

              area judgement     1

              volume judgement 1

5. Relative vs. Absolute Judgements

perceptual system mostly operates with relative judgements, not absolute

Weber's Law : the perceived change in stimuli is proportional to the initial stimuli

6. Preattentive Processing : cognitive operations done preattentively, without the need for focused attention

"pop out" of a display : easily detected regardless of the number of distractors

target detection, boundry detection, region tracking, counting

7. Design Guidelines/Principles

Visual Information Seeking Mantra

Overview first, zoom and filter, details on demand

8. Measuring Misrepresentation

9. Design Principles

avoid chartjunk

use small multiples

utilize narratives of space and time

2. Visualization Analysis and Design
1. Definitions and Motivations

computer-based visualization systems provide visual representations of datasets designed to help people carry out (some) tasks more effectively

2. Big picture
1. VIS is suitable when there is a need to argument human capabilites
2. design visual representations to help people perform task more effectively
3. design space is HUGE!
4. resource limitations
5. analysis instance

WHY the user needs it, WHAT data is shown, HOW the idiom is designed

3. Transitional use

gain a clear understanding of user's task --> purely computational solution --> monitoring automatic system is doing right

4. Long-term use
1. Exploratory analysis
2. Vis tool for presentation
5. Why use interactivity?
1. impossible to show everything at once
2. handling complexity and volume
6. Visualization Idioms : distinct approach to creating and manipluating visual representations

a tool that serves well for one task can be poorly to another

3. Analysis : Four Levels for Validation

so many possible ways.

Four Levels of vis design

  output of upstream level --> input to downstream level

    : upstream errors inevitably cascade down

Four kinds of threats to validate

1. wrong problem : they don't do that
2. wrong abstraction : showing wrong thing
3. wrong encoding/interaction technique : way you show don't work
4. wrong algorithm : code is too slow

--> proto-typing 을 통해서 미리 validation을 할 수 있다.

4. What : Data Abstraction

• semantics : real-world meaning
• attribute, item, link, grid, position
• set : unordered group of items
• list : ordered group of items
• cluster : grouping based on attribute similarity
• path : ordered set + links connecting nodes

• data abstraction : domain-specific to GENERIC

5. Why: Task Abstraction
• task abstraction should guide data abstraction
• Analyze > Search > Query
• Discover, Present, Enjoy, Annotate, Record, Derive(경험이 많을 수록 잘 사용 -> visualization idiom 사용의 폭이 넓어짐)

6. Marks and Channels
• idiom can be break down into Marks and Channels
• Mark
• basic graphical element : point(0D), line(1D), area(2D), volumn(3D)
• Items / Nodes
• Links

• Channel : a way to control the apperance of marks

• Expressiveness types and effectiveness rankings(1,2 순위는 외울것)

 * Better encoding ?

Bar size로 인하여 국가 간의 우선순위 혹은 우열을 내제적으로 표현하고 있어 불필요한 정보를 전달하고 있다.

이 Length를 변경하는데 Stevens' power law에 따라 effectiveness ranking 이 높은 position으로 변경하여 이를 제거한다.

• position dominates the user's mental model

1. Preattentive processing
1. cognitive operation done preattentively, without the need for focused attention
2. popout, segmentation effects
1. many channels : tilt, size, shape, proximity, shadow direction, but not parallel line.

2. Gestalt Psychology

인간은 자신이 본 것을 조직화하려는 기본 성향을 가지고 있으며, 전체는 부분의 합 이상이라는 점을 강조하는 심리학.

1. proximity : same spatial region
2. similarity : same value as other categorical channels (color hue, motion, shape)
3. connectedness : 모든 channel을 압도하는 가장 강력한 표현
4. continuity : 물리적으로 가장 단순하고 이해하기 쉬운 방향으로 이해한다.
5. common fate : things moving together

3. luminance contrast : Perception of color and luminance is contextual, based on relative judgements

7. Perception and Visual Patterns
1. Gestalt Principles
1. Grouping : avoid explicit grouping
1. Proximity
2. Similarity
3. Continuity
4. Common Fate

2. Perception of Forms
1. Closure : form complete, closed figures to increase regularity
2. Area/Figure and Ground/Relative size : smaller one as figure, larger one as ground
3. Symmetry : symmetry images are perceived collectively, even in spite of distance

2. Fixation-Saccade Cycle
1. Fixation : brief stationary period when detail information is acquired
2. saccade : flicking rapidly to a new location
3. postattentive amnesia

8. Rule of Thumbs
1. Overview
1. no unjustified 2D / 3D
2. eyes beats memory
3. resolution over immersion
4. function first, form next

9. Interaction Design and Design Principles
1. fundamental design goals is provide the right affordances, good mapping
1. right affordance : perceived and actual properies
2. right conceptual model
3. good mapping : a relationship between controls and their movements or effects
4. casuality : iterpesentation of feedback

D3를 이해하기 위한 도움 정보

1) 셀렉션 작동 원리 [link]

Greek Lettering

P, NP, NP-Hard

문제가 주어졌을 때, 이 문제가 현실적으로 풀 수 있는가 를 정의할때 쓰이는 개념들이다.

P는 Polynomial, 한글로는 다항식을 의미한다.

P군에 속하는 문제들은 다항식 시간에 해결할 수 있는 문제들을 의미한다.

NP는 Non deterministic Polynomial을 뜻한다.

NP군에 속하는 문제들은 다항식 시간에 확인 가능한 문제들을 의미한다.

이게 무슨 뜻이냐면, 문제의 답과 그에 대한 근거가 주어졌을때, 그것이 옳은 근거임을 다항식 시간에 확인 가능하다는 뜻이다.

* P와 NP의 포함관계

P는 NP에 포함될 것이라고 강력하게 추정되고 있을 뿐이다. 그렇지만 NP에 P가 아닌 문제도 포함되는지는 아직 아무도 증명하지 못했다. 이는 클레이 연구소에서 21세기를 맞아 정한 7개의 백만불짜리 문제 중 하나이다. 

NP-hard는 

모든 NP 문제가 주어진 문제 A에 대하여 다항식 시간 변환이 가능하면 A는 NP-hard이다.

하지만 모든 NP 문제를 다 나열하긴 불가능하니 그냥

어떤 하나의 NP 문제가 주어진 문제 A에 대하여 다항식 시간 변환이 가능하면 A는 NP-hard이다. 라고 정의하면 된다,

NP-Complete는 NP-hard보다 조금 더 좁은 개념으로 다음 두 조건을 만족하면 된다.

1. A는 NP이다.

2. A는 NP-hard이다.