Visual Analytics - An Interaction of Sight and Thought - JMP

Visual Analytics
An Interaction of Sight and Thought
Stephen Few
Perceptual Edge
www.PerceptualEdge.com
Copyright © 2012 Stephen Few

The world is not yet prepared for a true information age. Most people lack the basic
skills and the good tools that are required to use information in effective ways. I recently
discovered an organization named the 21st Century Fluency Project that is trying to
help people develop the skills that are needed to survive and thrive in a true information
age.

Hurray data!
Hurray technology!
We celebrate technology. We live for the latest gadgets.

But the abundance of data has left people overwhelmed and frustrated. You can spend
millions of dollars to build the most robust and pristine data warehouse in the world,
running on the most powerful hardware, and accessed by state-of-the-art business
intelligence software, but if the people who work with the data don’t know how to make
sense of it and then clearly present what they find to decision makers, your investment
is wasted.

Upon this gifted age, in its dark hour,
rains from the sky a meteoric shower
of facts…they lie, unquestioned, uncombined.
Wisdom enough to leach us of our ill
is daily spun; but there exists no loom
to weave it into a fabric.
“Huntsman, What Quarry?”, 1939, Edna St. Vincent Millay
The poet Edna St. Vincent Millay anticipated our situation today way back in 1939 when
she wrote these words. Our problem is not a lack of data, but rather an inability to make
sense of and use what we have.

Information
Data
Sensemaking
Data – all those facts that we’ve accumulated – do not qualify as information until they
inform. To use data effectively, we make first make sense of them. Today, sensemaking
is our primary task.

We must tap into that steady stream of information that is coming at us in a way that
allows us to make sense of it and make use of it. Endless columns and rows of
numbers don’t reveal the stories that live in our data. Data needs to be expressed in a
way that our eyes can easily see and our brains can easily understand.

Only then will the important stories that live in our data come to light.

Data analysts
JMP
are now sexy.
Data analysts are now beginning to get the respect they deserve. No longer are they
kept in some back room huddled over your computers. The role of data analyst is
becoming the sexiest role in the job market. Data analysts will become the priests of the
modern world – those who talk with God and divine the mysteries of data.

Hal Varian
U.C. Berkeley and Google
I keep saying the sexy job in the next ten years will be statisticians. People think I’m joking, but
who would’ve guessed that computer engineers would’ve been the sexy job of the 1990s? The
ability to take data—to be able to understand it, to process it, to extract value from it, to
visualize it, to communicate it—that’s going to be a hugely important skill in the next decades,
not only at the professional level but even at the educational level for elementary school kids,
for high school kids, for college kids. Because now we really do have essentially free and
ubiquitous data. So the complimentary scarce factor is the ability to understand that data and
extract value from it.
I think statisticians are part of it, but it’s just a part. You also want to be able to visualize the
data, communicate the data, and utilize it effectively. But I do think those skills—of being able
to access, understand, and communicate the insights you get from data analysis—are going to
be extremely important. Managers need to be able to access and understand the data
themselves.
You always have this problem of being surrounded by “yes men” and people who want to
predigest everything for you. In the old organization, you had to have this whole army of people
digesting information to be able to feed it to the decision maker at the top. But that’s not the
way it works anymore: the information can be available across the ranks, to everyone in the
organization. And what you need to ensure is that people have access to the data they need to
make their day-to-day decisions. And this can be done much more easily than it could be done
in the past. And it really empowers the knowledge workers to work more effectively.
“Hal Varian on how the Web challenges managers”, www.McKinseyQuarterly.com

To do this we need
the right skills
augmented by
the right tools
To explore and make sense of data, we need the right skills and good tools.

Skills
Tools
Sensemaking requires a close collaboration between our brains and computer-based
tools that are designed to augment our brains.

Two of the right skills
• Statistical thinking
• Visual thinking
Two of the important skill sets that we need include statistical thinking – the ability to
find the stories that live in numbers and make sense of them – and visual thinking – the
ability to see meaningful patterns in number by representing and interacting with them
visually.

Three modes of sensemaking
Verbal
A B C
Complementary
Numerical
1 2 3
Visual
We humans developed three powerful modes of sensemaking during the course of our
evolution: verbal, numerical, and visual. These modes developed separately to some
degree, but evolved to work in complementary and collaborative ways.

What is visual thinking?
Temple Grandin
We may consider “visual thinking” as that form of thought in which images are generated or
recalled in the mind and are manipulated, overlaid, translated, associated with other similar
forms (as with a metaphor), rotated, increased or reduced in size, distorted, or otherwise
transformed gradually from one familiar image into another…“Pattern recognition” has been a
major concern in artificial intelligence research and has proved to be quite complex. It has
turned out to be one of those things which is very difficult for computers but easy for most
human beings…For our purposes, we will consider pattern recognition to be the ability to
discern similarities of form among two or more things, whether these be textile designs, facial
resemblance of family members, graphs of repeating biological growth cycles, or similarities
between historical epochs…From pattern recognition it is but a short step to “problem solving,”
since, at least for its more common aspects, problem solving generally involves the recognition
of a developing or repeating pattern and the carrying out of actions to obtain desired results
based on one’s understanding of this pattern. (In the Mind’s Eye, Thomas G. West, p. 36)
Temple Grandin is an amazing visual thinker. Perhaps because she’s autistic, the visual
thinking parts of her brain have developed more fully than in most of us. A recent HBO movie
about her starring Claire Danes does a wonderful job of explaining her world and the way she’s
put her unique resources to work to make the world better, especially the life of cattle. One of
the most revealing lines in that trailer was when Grandin said, “I know my system will work
because I’ve been through it a thousand times in my head.” The most gifted visual thinkers can
see complex images and manipulate them in their mind’s eye.
Most of us can’t hold complex images in our heads and manipulate them like Temple Grandin. We
can, however, use complex images that are in front of our eyes on computers and manipulate
them.

We need greater balance.
Modes of reasoning through history
Numerical
Visual
During our hunter-gatherer days, before symbolic reasoning developed, which led to
language and mathematics, we were primarily visual, with just a crude sense of number.

We need greater balance.
Modes of reasoning through history
Verbal
Numerical
Visual
After language developed, however, especially after the development of writing, those in
power favored verbal skills and numerical skills to a lesser degree, but visual skills lost
favor as the low-brow reasoning of the masses.

We need greater balance.
Modes of reasoning through history
Verbal
Numerical
Visual
With the problems that we face today, this lack of balance remains, but we need to
correct it.

Leonardo da Vinci
For some four hundred or five hundred years we have had our schools teaching
basically the skills of the medieval clerk—reading, writing, counting, and
memorizing texts. Now it seems that we might be on the verge of a new era, when
we will wish to, and be required to, emphasize a very different set of skills—those
of a Renaissance man such as Leonardo da Vinci. With such a change, traits that
are considered desirable today might very well be obsolete and unwanted
tomorrow. In place of the qualities desired in a well-trained clerk, we might,
instead, find preferable a habit of innovation in many diverse fields, the
perspective of the global generalist rather than the narrowly focused specialist,
and an emphasis on visual content and analysis over parallel verbal modes.
If we continue to turn out people who primarily have the skills (and outlook) of the
clerk, however well trained, we may increasingly be turning out people who will,
like the unskilled laborer of the last century, have less and less to sell in the
marketplace. Sometime in the not too distant future machines will be the best
clerks. It will be left to humans to maximize what is most valued among human
capabilities and what machines cannot do—and increasingly these are likely to
involve the insightful and integrative capacities associated with visual modes of
thought.
(In the Mind’s Eye, Thomas G. West, 2009)

Many people who struggle with dyslexia (difficulty reading) are exceptional visual
thinkings, often because they develop visual thinking skills to to a greater degree than
the rest of us to make up for their inability to read. Thomas West does a wonderful job
of describing this in his book In the Mind’s Eye.

Complex problems
require visual
thinking.
Everyone agrees that we have a problem. Our technological culture is drowning in its own success.
Masses of data and information are accumulating everywhere. Up to now, the basic strategy for dealing
with these growing masses of information has been long, mind-numbing education and reckless,
blinkered specialization. That this strategy has been effective in a great many respects, so far, there can
be little debate. The problems we are discussing are a tribute to its ample and abundant success, so far.
However, after long success, it is becoming increasingly clear that this strategy may be entering a
phase of diminishing return. It has long been recognized that this strategy has always had built-in
problems. The more one knows in one’s own, increasingly narrow area, the more one is ignorant in
other areas, the more difficult is effective communication between unrelated areas, and the more
unlikely it is that the larger whole will be properly perceived or understood. Like the student who reads
too much small print, the specialist’s habitual near focus often promotes the myopic perspective that
precludes the comprehension of larger, more important patterns. The distant view of the whole is blurred
and unclear. If you focus only on a small group of stars at the edge of the Milky Way, you will not
perceive the larger structure of the whole galaxy of which the group is one tiny part.
The specialist strategy breeds its own limits. Pieces of the puzzle in separate areas remain far apart, or
come together only after decades of specialist resistance, or success in one area leads to great
problems in another. Material abundance produces waste-disposal problems; cars and aircraft produce
wonderful mobility for many people, but also deplete resources, produce accidental fatalities, and
increase pollution; success in vaccination, hygiene, and health care lead to all the problems of great
concentrations of human population.
As the specialist strategy continues to be pursued, a sense of the whole is increasingly lost. Many know
their areas; few see the whole. Many are expert; few are wise. But the visual thinkers, late bloomers,
and creative dyslexics we have been dealing with have often been outsiders or reluctant participants in
this specialist culture—especially those who are energetic, and globally minded, who seem always to be
interested in everything, unable to settle down to a “serious” (that is, highly specialized) area of study.
(In the Mind’s Eye, Thomas G. West, p. 298)

Only visual thinking can connect the dots.
Connecting the dots of complex problems requires visual thinking.

Tools extend our reach
when they’re well designed
and properly used.
Technologies extend our reach, but only when it’s well designed and properly used. Bad
tools or good tools used in the wrong ways can actually narrow our reach.

Tools
When tools are bad or misused, our reliance on them can make us lazy, weak, and
stupid.

Can computers replace us?
Our brains and computer processors work differently and therefore have different
strengths. We’ve designed computers to do things that we don’t do very well, which
makes perfect sense, because we use them to complement and extend our abilities.
They don’t, however, replace our abilities. Even though IBM’s supercomputer named
Watson can beat the best human champions of Jeopardy, we can all do with relative
ease in a second what Watson can’t figure out with endless effort: recognize that that
this is the word “captcha.”

The right tools support
• Statistical calculations
• Interactive visualizations
Good data sensemaking tools support statistical calculations, using the strength of
computers to perform those calculations quickly and accurately, and interactive
visualizations, making it possible to find and understand the meaningful patterns in our
data.

Visualization empowers statistics.
"The greatest value of a picture is when it forces
us to notice what we never expected to see."
Back in the 1970s, Princeton statistician John Tukey recognized the important
collaboration of statistical calculations and visual exploration. Even sophisticated
statisticians often need to use their eyes to find and understands the patterns that
reside in data.

Why the eyes?
I became fascinated by data visualization — the use of our eyes in close collaboration
with our brains to make sense of information — not because I’m a visually-oriented
person (I’m actually much more verbal than visual) but because it offers solutions to real
analytical problems that concern us, solutions that stand apart in their ability to
enlighten.

“I see!”
There is an intimate connection between seeing and thinking. For this reason, when we
talking about understanding, we tend to use terms that are related to vision. When we
suddenly come to understand something we say “I see!” It’s hard to think of terms that
describe understanding that don’t refer to sight.

Vision is our dominant sense.
50%
of the brain’s resources
If you think the brain has to devote to vision a lot of its precious thinking
resources, you are right on the money. It takes up about half of everything you
do, in fact. (Brain Rules, John Medina, p. 231)
When it comes to memory, researchers have known for more than 100 years that
pictures and text follow very different rules. Put simply, the more visual the input
becomes, the more likely it is to be recognized — and recalled. The
phenomenon is so pervasive, it has been given its own name: the pictorial
superiority effect, of PSE. (Ibid, p. 233)

Vision is our dominant sense.
30%
70%
Human perception is amazing. I cherish all five of the senses that connect us to the world, that allow
to experience beauty and an inexhaustible and diverse wealth of sensation. But of all the senses, one
stands out dramatically as our primary and most powerful channel of input from the world around us,
that is vision. Approximately 70% of the body’s sense receptors reside in the eye.
Perhaps the world’s top expert in visual perception and how its power can be harnessed for the effec
display of information is Colin Ware, who has convincingly described the importance of data
visualization. He asks:
Why should we be interested in visualization? Because the human visual system is a pattern seeke
enormous power and subtlety. The eye and the visual cortex of the brain form a massively parallel
processor that provides the highest-bandwidth channel into human cognitive centers. At higher lev
of processing, perception and cognition are closely interrelated, which is the reason why the words
‘understanding’ and ‘seeing’ are synonymous. However, the visual system has its own rules. We ca
easily see patterns presented in certain ways, but if they are presented in other ways, they become
invisible…The more general point is that when data is presented in certain ways, the patterns can
readily perceived. If we can understand how perception works, our knowledge can be translated in
rules for displaying information. Following perception-based rules, we can present our data in such
way that the important and informative patterns stand out. If we disobey the rules, our data will be
incomprehensible or misleading.
(Information Visualization: Perception for Design, Second Edition, Colin Ware, Morgan Kaufmann
Publishers, 2004, page xxi)
Perhaps the best known expert in data visualization, Edward Tufte, says: “Clear and precise seeing
becomes as one with clear and precise thinking.” (Visual Explanations, Edward R. Tufte, Graphics Pr
Cheshire, CT.1997 page 53)

Visualization weaves numbers into pictures.
Data visualization is the loom that will weave the data that we collect into the fabric of
understanding. Pictures of data can make visible the meanings that might forever
otherwise remain hidden.

Trends, patterns, and exceptions come to light.
The presentation of data as text, such as you see in this table, is perfect when you need
precise values or when the purpose is to look up or compare individual values, but not
when you wish to see patterns, trends, and exceptions, or to make comparisons. When
this is your goal, visualizations work best.
When data is presented visually, it is given visible form, and from this we can easily
glean insights that would take a long time to piece together from the same data
presented textually, if ever. This graph of the same data that appears in the table makes
brings to light several of the stories contained in the data that weren’t obvious before,
and it did so instantly.
When] we visualize the data effectively and suddenly, there is what Joseph Berkson
called ‘interocular traumatic impact’: a conclusion that hits us between the eyes.
(Visualizing Data, William S. Cleveland, Hobart Press, 1993, page 12)
Modern data graphics can do much more than simply substitute for small statistical
tables. At their best, graphics are instruments for reasoning about quantitative
information. Often the most effective way to describe, explore, and summarize a set
of numbers – even a very large set – is to look at pictures of those numbers.
Furthermore, of all methods for analyzing and communicating statistical information,
well-designed data graphics are usually the simplest and at the same time the most
powerful.
(The Visual Display of Quantitative Information, Edward R. Tufte, Graphics Press:
Cheshire, CT 1983, Introduction)

A picture David is worth visits a thousand America. words.
As we all know, the right picture can sometimes tell a story in a way that no amount of
words could ever match. To take advantage of visualization’s great potential, we first
must know when pictures rather than words are needed.

This visualization fails to do this. This series of circles within circles––blue for the market
values of banks in quarter 2 of 2007, before the recent financial meltdown, and green for
declined values as of January of 2009––was published by Bloomberg. You would never
guess its purpose, however, which was to show that J. P. Morgan’s decline in market
value was less severe than all other major banks except one: Santander.
This picture of the data doesn’t tell the story clearly, simply, or accurately. The
comparative sizes of the circles are far from the comparative market values. Even if the
sizes of the circles were accurate, we would still struggle with this chart because visual
perception isn’t well-tuned to handle size comparisons, but it is tuned to handle length
comparisons,...

...such as the lengths of these bars in my redesign of the chart. We can now easily see
that J. P. Morgan lost roughly half of its market value during this period, but the fact that
its losses were less severe than all by one bank––Santander––still isn’t obvious. The
right addition to the picture, however, such as this one in the bottom half that displays
the losses directly, can make this part of the story clear as well.

The right balance
38
Information visualization makes possible an ideal balance between unconscious
perceptual and conscious cognitive processes. With the proper tools, we can shift much
of the analytical process from conscious processes in the brain to pre-attentive
processes of visual perception, letting our eyes do what they do extremely well.

Confused?
Unfortunately, so much of “data visualization” today gives it a bad name and causes
confusion about what it is, how it works, and what can be accomplished when it is
properly done.

Many visualization tools
feature decoration,
not substance.
Rather than building useful tools, most software vendors are competing to out dazzle
one another with silly visual effects that treat data visualization like it’s a video game.

Pie!!!
Most so-called data analysis tools – especially data visualization tools – so far show
little understanding of the people who use them and little respect for their intelligence.
They treat people as if they’re dumb, and by doing so encourage them to become
dumb, assuming that they wish merely to be entertained by cute pictures and pretty
colors.

Bling your graph.
Many software vendors promote the notion that data visualization is not about
understanding and communication, but about “bling.”

Flashy nonsense
Dashboards like this have become a popular form of display in the last few years, but
while they give the appearance of data density, most of them present little and the what
they display is presented poorly. But they look cool, and for many that’s all that matters,
but it isn’t enough.

3-D graphs are rarely useful.
One of the common themes of most software vendors is that 2-D displays are boring;
never as good as 3D.

Adding a third dimension of depth to the bars on the right without adding a
corresponding third variable, however, is not only meaningless, it makes it more difficult
to decode the data.

Even when there’s a 3 rd variable.
Even when a third dimension of meaning exists, such as the regions along the Z-axis of
these graphs, problems with 3-D graphs persist.

The graph on the left illustrate the common problem of occlusion--the fact the some
objects, such as these bars, become hidden behind others. In a graph, we should be
able to see all the data at once to make the necessary connections and comparisons.
Looking at the graph on the right, I bet you can’t even determine which line represents
the South region. If we can’t figure out which line is which region, what good is the
graph?

We don’t want to confuse people with graphical puzzles.
This chart of Escher’s changing popularity through time was created by B. Brucker. I
found it at www.GraphJam.com.

Data visualization builds a bridge
from data to knowledge.
Data Visualization
Knowledge
Data
Built on an understanding
of
visual perception
and
cognition
Data visualization can help us build a bridge from data to knowledge, but only if the
tools that we use were built on an understanding of visual perception (how we see) and
cognition (how we think).

Good visualizations and good visualization tools are carefully designed to take
advantage of human perceptual and cognitive strengths and to augment human abilities
that are weak. If the goal is to count the number of circles, this visualization isn’t well
designed. It is difficult to remember what you have and have not counted.
50

This visualization, which shows the same number of circles, however, is well designed
for the counting task. Because the circles are grouped into small sets of five each, it is
easy to remember which groups have and have not been counted, easy to quickly count
the number of circles in each group, and easy to discover with little effort that each of
the five groups contains the same number of circles (i.e., five), resulting in a total count
of 25 circles.
51

This arrangement is even better yet.
52

Visual perception works according to
its own rules.
Many of the ways that visual perception work are not intuitive. Looking at these two sets
of objects, we naturally see those on the left as convex and on those the right as
concave.

Visual perception works according to
its own rules.
The effect has now been reversed: we see the objects on the left as concave and those
on the right as convex. All I did, however, was turn each sets of objects upside down—I
didn’t switch them. The reason that we now see those on the left as concave is
because, through eons of evolution, visual perception learned to assume that light was
shining from above, which causes us to see the objects on the left as concave, because
the shadows are on the top, and those on the right as convex, because the shadows
are on the bottom.

Visual perception is not just camera work.
Unlike a camera, visual perception does not record absolute values of the things that
we see, but differences between them.

Visual perception is not just camera work.
Despite how differently they look in the original image, squares A and B are exactly the
same color. What we see is not a simple recording of what is actually out there. Seeing
is an active process that involves interpretations by our brains of data that is sensed by
our eyes in an effort to make sense of it in context. The presence of the cylinder and its
shadow in the image of the checkerboard triggers an adjustment in our minds to
perceive the square labeled B as lighter than it actually is. The illusion is also created by
the fact that the sensors in our eyes do not register actual color but rather the difference
in color between something and what’s nearby. The contrast between square A and the
light squares that surround it and square B and the dark squares that surround it cause
us to perceive squares A and B quite differently, even though they are actually the same
color, as you can clearly see above after all of the surrounding context has been
removed.
The ability to use graphs effectively requires a basic understanding of how we
unconsciously interpret what we see.

Context affects what we see. This image illustrates the surprising effect that a simple
change in the lightness of the background alone has on our perception of color. The
large rectangle displays a simple color gradient of a gray-scale from fully light to fully
dark. The small rectangle is the same exact color everywhere it appears, but it doesn’t
look that way because our brains perceive visual differences rather than absolute
values, in this case between the color of the small rectangle and the color that
immediately surrounds it.
Among other things, understanding this should tell us that using a color gradient as the
background of a graph should be avoided.

Two essential activities
• Multi-faceted views
• Comparisons
Data sensemaking revolves primarily around two essential activities: viewing data from
multiple perspectives and comparing things.
No one view of the data will suffice. To understand your data thoroughly, you must view
it from every possible perspective. With each new view, more than anything else, you
make compare things.

Multiple views of change through time
Have you ever noticed that time-series data can look quite different if you change the
interval of time that you’ve aggregated it to in a graph? For example, if you are
examining a year’s worth of visits to your website with one value per quarter
(aggregated to quarterly intervals), and then switch to a monthly aggregation of the
same data, and then switch to a daily aggregation, the patterns of change might look
quite different.
All three versions of the graph are useful and correct, but the daily version reveals
details that aren’t visible when viewing the same data by month or quarter. On the other
hand, the overall trend is difficult to discern from the daily view. One view isn’t better
than the other in general, but one is definitely better than the other when you have a
specific analytical purpose in mind. For this reason, don’t restrict your view of timeseries
data to a single interval of time, especially when looking for anything that seems
interesting. Switch the level of aggregation from year to quarter, quarter to month,
month to week, week to day, and so on—back and forth—to tease out the insights that
can only arise when this is done.
Software products that allow you to quickly and easily switch between various intervals
of time while viewing data graphically are what’s needed to encourage this practice. The
ability to switch time intervals with a mouse click or two, or by using something as
simple as an interval slider control, will set you free to explore without distraction from
onerous operations.

Data analysis requires rapid interaction.
?Think
See
Modify
Direct dynamic interaction with the properly visualized data allows us to see discover
meaningful patterns, trends, and exceptions in the display and to interact with it directly
to filter out what we don’t need, drill into details, combine multiple variables for
comparison, etc., in ways that promote a smooth flow between seeing something,
thinking about it, and manipulating it, with no distracting lags in between. This is what I
call “visual analysis at the speed of thought.”
Great analysts, like great scientists, great artists, great people of all sorts, accept the
call to serve as a voice for data. Important stories can be found in data. We can learn to
discern the meanings that live in information and to unravel the stories that are woven
through it. What I do isn’t just work; it is my mission. I work hard to learn the world’s
stories and to tell them truthfully. I believe that there is no higher calling. I fight against
those who try to hide or alter the truth. I believe that the truth really can set us free.

“Compared to what?”
Edward Tufte once said that the essential question of data sensemaking is “Compared
to what?” This involves comparing the magnitudes of values, but not just one to another.
We must compare many values. To do so, we must see how they relate to one another
to form patterns. We not only compare the magnitudes of values; we also compare
patterns formed by sets of values. We look for how they are similar and we look for how
they are different, especially differences that appear to be dramatic departures from the
norm. When we spot these visual characteristics in the data, we then interact with the
data to find out why these things have happened.

Key interactions of visual analysis
• Sorting
• Filtering
• Adding/removing variables
• Highlighting
• Aggregating/Disaggregating
• Grouping
• Zooming/Panning
• Re-visualizing
• Re-expressing
• Re-scaling
The process of visual data analysis involves several common interactions with data to uncover what’s meaningful. Here
are some of the primary interactions:
• Sorting. The act of sorting data, especially by the magnitude of the values from high to low or low to high, features the
ranking relationship between those values and makes it easier to compare the magnitude of value to the next.
• Adding/removing variables. You might need to view different variable at different times during the analysis process,
so it is common to add or remove field of data from view as necessary
• Filtering. When you want to focus on a subset of data, nothing makes it easier to do so than filtering—the removal
from view of everything your not interested in at the moment.
• Highlighting. Sometimes you want to focus on a subset of information, but do so in a way that allows you to maintain
a sense of how that subset relates to the whole. Rather than filtering out the data that falls outside your range of focus,
you can simply reduce its visual salience or increase the visual salience of the data you wish to focus on. This allows
you to focus on the subset with less distraction from the whole in a way that allow you to remain aware of the whole.
This is one way of achieving what’s called a focus+context view.
• Aggregating/Disaggregating. Analysis often requires that you examine data a different levels of detail. Aggregation
involves viewing data at a higher level of summarization. Disaggregation involves viewing data at a lower level of
detail.
• Grouping. Sometimes it is useful to combine members of a variable together, treating them as a single member of the
variable. This may take the form of combining some members and leaving others as they are, or of creating an entirely
new variable that combines all members of an existing variable into a groups to form members of a higher level
variable.
• Zooming/Panning. When a data visualization contains so much that it is difficult to clearly see all the data at once, it is
useful to zoom in on that portion that you want to see more clearly. Panning involves moving around (for example, up,
down, right, or left) in a zoomed view to focus on a different part of the larger visualization.
• Re-visualizing. No one visual representation of data can show you everything there is to see, so visual analysis
involves shifting from one type of visualization to another to explore data from various perspectives.
• Re-expressing. Sometimes it is useful to express a quantitative variable as a different unit of measure, such as
expressing dollars as percentages.
• Re-scaling. No single quantitative scale on a graph can serve every analytical need. Rescaling involves changing the
range of the quantitative scale to make it easier to see particular patterns and sometimes even changing the nature of
the scale, such as from a normal scale to a logarithmic scale.

Dynamic data filters support analytical flow.
Direct dynamic interaction with the data allows you to manipulate the data easily and
immediately (such as by filtering it), without interrupting your stream of thought.

Shneiderman’s Mantra
“Overview first,
zoom and filter,
then details-on-demand.”
When new recruits by intelligence organizations are trained in spy craft, they are taught
a method of observation that begins by getting an overview of the scene around them
while being sensitive to things that appear abnormal, not quite right, which they should
then focus in on for close observation and analysis.
A visual information-seeking mantra for designers: ‘Overview first, zoom and filter,
then details-on-demand.’
(Readings in Information Visualization: Using Vision to Think, Stuart K. Card, Jock D.
Mackinlay, and Ben Shneiderman, Academic Press, San Diego, California, 1999,
page 625)
Having an overview is very important. It reduces search, allows the detection of
overall patterns, and aids the user in choosing the next move. A general heuristic of
visualization design, therefore, is to start with an overview. But it is also necessary for
the user to access details rapidly. One solution is overview + detail: to provide
multiple views, an overview for orientation, and a detailed view for further work.
(Ibid., page 285)
Users often try to make a ‘good’ choice by deciding first what they do not want, i.e.
they first try to reduce the data set to a smaller, more manageable size. After some
iterations, it is easier to make the final selection(s) from the reduced data set. This
iterative refinement or progressive querying of data sets is sometimes known as
hierarchical decision-making.
(Ibid., page 295)

Overview first, zoom and filter, then details-on-demand
Shneiderman’s technique begins with an overview of the data — the big picture. Let
your eyes search for particular points of interest in the whole.

Overview first, zoom and filter, then details-on-demand
When you see a particular point of interest, then zoom in on it.

Overview first, zoom and filter, then details-on-demand
Once you’ve zoomed in on it, you can examine it more closely and in greater detail.

Overview first, zoom and filter, then details-on-demand
Often you must remove data that is extraneous to your investigation to better focus on
the relevant data.

Overview first, zoom and filter, then details-on-demand
Filtering out extraneous data removes distractions from the data under investigation.

Overview first, zoom and filter, then details-on-demand
Jan 21, 2011
$25,193
Visual data analysis relies mostly on the shape of the data to provide needed insights,
but there are still times when you need to see the details behind the shape of the data.
Having a means to easily see the details when you need them, without having them in
the way when you don’t works best.

Data visualization must
focus attention
and
augment memory.
To achieve this goal, tools must be designed to focus our attention on what’s meaningful
in data and to help us work around the limitations of memory.

Attention is limited.
Ever saw a magic show and wondered just how the magician took your watch
without you even noticing? Ever wonder why is it that you can search for a set of
misplaced keys for a long time, only to later find them sitting in the exact place
where you were looking? Research has shown that we don’t always see everything
we’re looking at, and that attention plays a big part in what consciously registers to
us. The effect where we’re blind to things we don’t attend to is known as
‘Inattentional Blindness’.
(Source: This demonstration of inattentional blindness and the explanation above
was prepared under the direction of Ronald A. Rensink of the University of British
Columbia. Several other examples of this visual phenomenon can be found at http://
psyclab1.psych.ubc.ca/~viscoglab/.)

Memory is limited.
What’s not stored in working memory is lost.
In addition to understanding visual perception, visual analysis tools must also be rooted
in an understanding of how people think. Only then can they recognize and support the
cognitive operations that are necessary to make sense of information.
Memory plays an important role in human cognition. Because memory suffers from
certain limitations, visual analysis tools must be able to augment memory.
The example above illustrates one of the limitations of working memory. We only
remember that to which we attend. Any part of this image that never gets our attention
will not be missed when we shift to another version of the image that lacks that
particular part. If we don’t attend to it, we might notice the change from one version of
the image to the next, but only if the transition shift immediately from one to another,
without even a split second of blank space between them.
In addition to not remembering, we also don’t clearly see that on which we don’t focus.
To see something clearly, we must focus on it, for only a small area of receptors on the
retinas of our eyes are designed for high-resolution vision.
(Source: This demonstration of change blindness was prepared by Ronald A. Rensink
of the University of British Columbia. Several other examples of this visual phenomenon
can be found at http://www.psych.ubc.ca/%7erensink/flicker/download/index.html.)

Capacity = 3 or 4
chunks at a time
Imagination
The
World
Working
Memory
Long-term
Memory
When we think about things, trying to make sense of them, information is temporarily
held in working memory. Only three or four chunks of visual information can be stored in
working memory at any one time. Information that comes in through our eyes or that is
retrieved from long-term memory in the moment of thought is extremely limited in
capacity. When you release information from working memory, it can take one of two
possible routes on its way out: 1) it can be stored permanently in long-term memory by
means of a rehearsal process that we call memorization, or 2) it can simply be
forgotten.

External aids can augment memory.
To compare facts, you must hold them in working memory simultaneously. Because we
can hold so little in working memory at any one time, however, to do analysis effectively,
we must rely on external aids to memory. This is an ideal job for a computer. Even a
piece of paper that you jot down notes on to keep track of information as you’re
analyzing data is an external memory aid that is quite powerful despite being low-tech.
A computer running properly designed software, however, can augment our ability to
think about information much better than pencil and paper.

Good analysis tools augment memory by
expanding...
• Quantity
• Dimensions
• Perspectives
Good visual analysis software can help us overcome the limitations of working memory
in several ways. The goal is to enable as many meaningful comparisons as possible.
Good tools can help us increase:
• The amount of information that we can compare (that is, greater quantity)
• The range of information that we can compare (that is, more dimensions)
• The different views of the information that we can compare (that is, multiple
perspectives)

Greater quantity through visual encoding
1 chunk
Traditional BI relies mostly on tabular data displays. Tables are wonderful if you need to
look up individual values, compare a single value to another, or know values precisely,
but they don’t display patterns or trends. This is a problem, because data analysis relies
heavily on our ability to spot and make sense of patterns and trends in data. Take a look
at the table and compare it to this line graph, which displays the same data. Relying on
the table to discern the ups and downs of sales through time and to compare the
patterns of change from region to region would yield very little of the information that is
obvious in this graph. Visual representations give form to data, making pattern, trends,
and exceptions easy to see.
Another advantage of properly designed graphs over tables for analytical purposes is
less obvious. If you needed to remember information in the table, you could hold only
about four of the values (that is, four of the monthly sales numbers) in working memory
at any one time. But by relying on the graph, 12 values are combined into each of the
four lines to form a pattern that you could hold entirely as a single chunk in working
memory. Simply by giving visual form to the values, you can hold much more
information in memory.

More dimensions through multiple visual
encodings and small multiples
You can increase the number of dimensions (variables) that can be viewed
simultaneously in two ways: 1) by using multiple visual encodings, and 2) by breaking
the data into a series of small graphs that share common scales so they can be
compared to one another. In the bubble plot on the right, we can view four variables at
once: the rate of property crime based in position along the X-axis, the rate of violent
crime based on position along the Y-axis, the population of states based on the size of
the bubbles, and the region in which each state resides based on color. The series of
small multiples on the left allows us to revenue, costs of goods sold, and gross profit
along the Y-axis, quarters along the X-axis, and different product lines per graph. If we
tried to display all of this information in a single graph, it would be far too cluttered and
difficult to perceive.

More perspectives through coordinated
views
There’s an old folktale that you’ve probably all heard about three blind men who encounte
elephant one day for the first time and do their best to learn about it by touch alone. The
experience of each is unique because each touches a different part of the elephant. This
ancient story, originally from China, can teach us something important today about busine
intelligence (BI). According to the original Chinese tale, the first man touches the elephant
ear, the second his legs, and the third his tail. From this point, here’s how the story goes:
The three blind men then went their way. Each one was secretly excited over the
experience and had a lot to say, yet all walked rapidly without saying a word.
"Let's sit down and have a discussion about this queer animal," the second blind man sa
breaking the silence.
"A very good idea. Very good." the other two agreed for they also had this in mind. With
waiting for anyone to be properly seated, the second one blurted out, "This queer anima
like our straw fans swinging back and forth to give us a breeze. However, it's not so big
well made. The main portion is rather wispy."
"No, no!" the first blind man shouted in disagreement. "This queer animal resembles two
big trees without any branches."
"You're both wrong." the third man replied. "This queer animal is similar to a snake; it's l
and round, and very strong."
How they argued! Each one insisted that he alone was correct. Of course, there was no
conclusion for not one had thoroughly examined the whole elephant. How can anyone
describe the whole until he has learned the total of the parts.
If I retold this story today to teach a lesson about BI, I might call it “Three blind analysts an
data warehouse.” Business people struggle every day to make sense of data, stumbling
blindly, touching only small parts of the information, and coming away with a narrow and
fragmented understanding of what it means.

Coordinated views expand what we can
connect and compare.
To understand something, we often have to examine it from many angles and focus on
many parts. Too much business data analysis involves looking only for one thing in
particular. Is revenue going up? The answer is “yes” or “no” — end of story. Perhaps,
however, you ought to look at revenues, expenses, profits, marketing campaigns,
seasonality, composition of the sales force, new product introductions, and the
competition to understand the richer story that your data has to tell.

Who is qualified to analyze data?
Data
Analysis
Knowledge
?
You are the key that opens the door for good data to result in good decisions. Software,
no matter how sophisticated, is useless if you don’t possess the fundamental skills of
data analysis. Data analysis is for one purpose: to enable good decisions. Do you need
to be an Einstein to make sense of your business data? For most business data
analysis, the answer is “No”, but when statistical sophistication required, you need a
statistically sophisticated tool.

Analytical challenges demand
statistical horsepower.
JMP is one of best tools available today for combining robust statistical functionality with
rich interactive visualizations.

Predictive analysis is one of these
challenges.
U.S. Dollars
(thousands)
4,000
Sales Revenues
Sales Revenues
3,500
3,000
2,500
2,000
1,500
1,000
500
0
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2012
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2013
One of the analytical challenges that requires a high degree of statistical training is its
use for predicting what might happen in the future. Let’s clarify the meaning of
predictive analytics. Like many terms that get tossed around by software vendors and
even by thought leaders in the field of business intelligence, predictive analytics seems
to mean whatever’s convenient at the moment. It might be useful for marketing
purposes to keep definitions loose and adaptable to the occasion, but it doesn’t help us
at all. We’ll start by turning to everyone’s favorite dictionary these days: Wikipedia.
Predictive analytics encompass a variety of techniques from statistics and data
mining that analyze current and historical data to make predictions about future
events. Such predictions rarely take the form of absolute statements, and are more
likely to be expressed as values that correspond to the odds of a particular event or
behavior taking place in the future.
In business, predictive models exploit patterns found in historical and transactional
data to identify risks and opportunities. Models capture relationships among many
factors to allow assessment of risk or potential associated with a particular set of
conditions, guiding decision making for candidate transactions.
Wikipedia entry for “predictive analytics” as of December 15, 2008.

Predictions rely on statistical models.
Don’t let the terms statistical model or predictive model throw you. In concept, they’re
quite simple, just like more familiar models of other types. Generally speaking, models
are representations of things or events, which we use to examine and understand those
things or events when it isn’t possible or practical to observe or interact with them
directly. Statistical models represent mathematical relationships between the parts that
make up the thing or event. Predictive models are those that we can interact with to
investigate the results of hypothetical conditions, such as by changing the values of
particular variables.
Predictive models make it possible for us to do what some people call what-if analysis.
What if such and such a condition existed or event occurred? What would happen as a
result? Predictive models give us the means to predict what would probably happen
(the probable outcomes of dependent variables) if particular conditions arose naturally
or by intention (specified input values to one or more independent variables).

Good models don’t hide variables and
their relationships.
Inputs
?
Outputs
Most of the applications that I’ve seen marketed by business intelligence software
vendors for predictive analytics allow data to be entered on one end (inputs) and then
results (outputs) pop out the other; what goes on in between remains hidden in a black
box. Unfortunately, without seeing what goes on in that black box, our brains aren’t fully
engaged in the process and too much is missed.
Predictive analytics are most revealing when they allow us to see how all the variables
that contribute either directly or indirectly to the outcomes that concern us relate to
those outcomes and to one another. To understand these relationships, we must see
them; we must watch how changes in one variable directly cause or indirectly influence
changes in the others. For this to happen, predictive models must be displayed visually
in a way that allows: (1) our eyes to see the relationships and changes; and (2) our
minds to make sense of them.

Finding and understanding the
stories in data is not enough.
You must tell them.
As a data analyst, your job isn’t done once you’ve found and made sense of the stories
that live in your data. Unless you can tell those stories in ways that others who need the
information can understand, your work is wasted.

Hans Rosling of www.GapMinder.org
Few people tell quantitative stories as compellingly as Hans Rosling. When he spoke at
the TED Conference (Technology, Entertainment, and Design) for he first time in 2006, it
was perhaps the first time in history that a large audience of people found themselves
on the edge of their seats watching a bubble plot.

Wisdom
Knowledge
Information
Information cannot speak for itself. It needs our help. It relies on us to give it a voice.
When we do, information can tell its story, and will thus become knowledge. The
ultimate goal, however, isn’t knowledge; it is wisdom. Knowledge becomes wisdom
when it is used to do something good. Only when we use what we know to make the
world a better place has information served its purpose and we have done our job.
Our networks are awash in data. A little of it is information. A smidgen of this shows
up as knowledge. Combined with ideas, some of that is actually useful. Mix in
experience, context, compassion, discipline, humor, tolerance, and humility, and
perhaps knowledge becomes wisdom.
Turning Numbers into Knowledge, Jonathan G. Koomey, 2001, Analytics Press:
Oakland, CA page 5, quoting Clifford Stoll.

O perpetual revolution of configured stars,
O perpetual recurrence of determined seasons,
O world of spring and autumn, birth and dying!
The endless cycle of idea and action,
Endless invention, endless experiment,
Brings knowledge of motion, but not of stillness;
Knowledge of speech, but not of silence;
Knowledge of words, and ignorance of The Word.
All our knowledge brings us nearer to our ignorance,
All our ignorance brings us nearer to death,
But nearness to death no nearer to God.
Where is the Life we have lost in living?
Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?
Excerpt from The Rock, 1934, T.S. Elliot
[Image source: www.irishastronomy.org]

O perpetual revolution of configured stars,
O perpetual recurrence of determined seasons,
O world of spring and autumn, birth and dying!
The endless cycle of idea and action,
Endless invention, endless experiment,
Brings knowledge of motion, but not of stillness;
Knowledge of speech, but not of silence;
Knowledge of words, and ignorance of The Word.
All our knowledge brings us nearer to our ignorance,
All our ignorance brings us nearer to death,
But nearness to death no nearer to God.
Where is the Life we have lost in living?
Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?
Excerpt from The Rock, 1934, T.S. Elliot
[Image source: www.trekvisual.com]

O perpetual revolution of configured stars,
O perpetual recurrence of determined seasons,
O world of spring and autumn, birth and dying!
The endless cycle of idea and action,
Endless invention, endless experiment,
Brings knowledge of motion, but not of stillness;
Knowledge of speech, but not of silence;
Knowledge of words, and ignorance of The Word.
All our knowledge brings us nearer to our ignorance,
All our ignorance brings us nearer to death,
But nearness to death no nearer to God.
Where is the Life we have lost in living?
Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?
Excerpt from The Rock, 1934, T.S. Elliot
[Image source: www.i.pbase.com]

O perpetual revolution of configured stars,
O perpetual recurrence of determined seasons,
O world of spring and autumn, birth and dying!
The endless cycle of idea and action,
Endless invention, endless experiment,
Brings knowledge of motion, but not of stillness;
Knowledge of speech, but not of silence;
Knowledge of words, and ignorance of The Word.
All our knowledge brings us nearer to our ignorance,
All our ignorance brings us nearer to death,
But nearness to death no nearer to God.
Where is the Life we have lost in living?
Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?
Excerpt from The Rock, 1934, T.S. Elliot
[Image source: www.]

Truth
O perpetual revolution of configured stars,
O perpetual recurrence of determined seasons,
O world of spring and autumn, birth and dying!
The endless cycle of idea and action,
Endless invention, endless experiment,
Brings knowledge of motion, but not of stillness;
Knowledge of speech, but not of silence;
Knowledge of words, and ignorance of The Word.
All our knowledge brings us nearer to our ignorance,
All our ignorance brings us nearer to death,
But nearness to death no nearer to God.
Where is the Life we have lost in living?
Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?
Excerpt from The Rock, 1934, T.S. Elliot
[Image source: www.shepherdpics.com]

O perpetual revolution of configured stars,
O perpetual recurrence of determined seasons,
O world of spring and autumn, birth and dying!
The endless cycle of idea and action,
Endless invention, endless experiment,
Brings knowledge of motion, but not of stillness;
Knowledge of speech, but not of silence;
Knowledge of words, and ignorance of The Word.
All our knowledge brings us nearer to our ignorance,
All our ignorance brings us nearer to death,
But nearness to death no nearer to God.
Where is the Life we have lost in living?
Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?
Excerpt from The Rock, 1934, T.S. Elliot
[Image source: www.i163.photobucket.com]

The value of information depends on how it’s used.
Use it wisely.
O perpetual revolution of configured stars,
O perpetual recurrence of determined seasons,
O world of spring and autumn, birth and dying!
The endless cycle of idea and action,
Endless invention, endless experiment,
Brings knowledge of motion, but not of stillness;
Knowledge of speech, but not of silence;
Knowledge of words, and ignorance of The Word.
All our knowledge brings us nearer to our ignorance,
All our ignorance brings us nearer to death,
But nearness to death no nearer to God.
Where is the Life we have lost in living?
Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?
Excerpt from The Rock, 1934, T.S. Elliot
[Image source: www.jamin.org]