Crystal Fingerprinting and STM4 for USPEX output ... - Mario Valle

Crystal Fingerprintingand STM4for USPEX output analysisMario ValleUSPEX 2011 Xi’an workshop – 04/08/2011The CSCS Data Analysisand VisualizationGroupmvalle@cscs.ch1

CSCS computer roomCSCS usage May 2011Prof. Michele ParrinelloStarted with postprocessingMovies forconference talksFrancesco Gervasio – ETH ZürichNice images for publications2

Started with postprocessingMovies forconference talksFrancesco Gervasio – ETH ZürichNice images for publicationsData from disjoint subfieldsMacromoleculesCrystallographyCrystalFp, STM4 & USPEX - Mario Valle - 04/08/20113

Help overcome tools inflexibilityFor example there are nice crystallographyprograms that do not support dynamicdata and do not allow customizationSTM4 is a frameworkfor the development ofunusual and enhanced techniquesfor chemistry visualization4

Offer broader set of techniquesProvide enhanced techniquesAvailable data with thestandard isosurfaceIsosurface with the newvolume interpolator5

STM3 GallerySTM3 modulesThe LEGO DNA6

USPEX discovered new materialsBoron at 1 atm: USPEXeasily found the complexα-B structure...Prof. A. Oganov...and discoveredalso the superhardboron γ-B 28 phase(Nature)USPEX structure cancer problemDifferent colors meansdifferent crystalstructuresNormal structure cluster generationUSPEX structure cancerGenerationCrystalFp, STM4 & USPEX - Mario Valle - 04/08/20118

CrystalFp: the problem to solveUSPEX is a crystal structure predictorbased on an evolutionary algorithmEach run produceshundred of putativecrystal structures……but many of themare equalProject: develop a(semi)automaticway to extractunique structuresfrom USPEX outputsSo an intensive manuallabor is needed to pruneduplicated structuresCrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011Proposed solution from High-DimCompute uniquecoordinatesDefine distancemeasureSpace 100-3000dimensionalEach groupdescribes adistinct structureAdd groupingcriteriaCrystalFp, STM4 & USPEX - Mario Valle - 04/08/20119

Structure “coordinates”Set of distancesfor each atomin the unit cellaggDistance setsconcatenated for allatoms in the structure(coordinate)Structure “fingerprint” CrystalFpSet of distancesfor each atomin the unit cellaggDistance setsconcatenated for allatoms in the structure(coordinate)10

Visual design and validationBuilt a tool to explorealgorithm choices andparameters settingsThis tool wraps theclassifier library, calledCrystalFp, and providesvarious interactivevisual diagnostics tocheck classifierbehaviorIt is built inside STM4,the molecularvisualization toolkitdeveloped at CSCSCrystalFp, STM4 & USPEX - Mario Valle - 04/08/201111

Workflow supportThe application interface gives accessto all CrystalFp algorithms and theirparameters in a clear process workflowSTM4 provided an environment thataccelerated the implementation1. Load structures2. Filter on energy3. Compute fingerprints4. Compute distances5. Group structuresVisual diagnostics toolsVarious visualization and analysistools to check and validateCrystalFp algorithms behavior1. 2D maps2. Charts3. Picking for details4. 2D data export12

Visual diagnostics: distance matrixDistances between structuresDistances ordered by groupCrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011Clustering visual diagnosticDFS groupingPseudo SNN (K=5)Pseudo SNN (K=1)SNN (K=5)DFS: Deep first search ofthe neighbors nodesPseudo SNN: Maintainconnection betweennodes only if theyshare at least KneighborsSNN: As above plus aDBSCAN passCrystalFp, STM4 & USPEX - Mario Valle - 04/08/201113

Visual diagnostics: scatterplotThe scatterplot tool in CrystalFp tries tomap High‐Dim space points to 2Dpreserving their relative distancesColored by“stress” to detectlocal minimatrapsColoredby groupDiagnostic chart:distances in 2Dvs. distances inHigh‐Dim space14

A pseudo-diffraction like methodThis structure fingerprint issampled on X to provide thecoordinate values.The fingerprint is cut at a userdefined distance to provide100-400 coordinate valuesCrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011Final fingerprint (per atom type pair)Compared to the previousfingerprinting method, thisone is sensitive to theordering of atoms in thestructure and does notdepend on the specificatomic species involvedCrystalFp, STM4 & USPEX - Mario Valle - 04/08/201115

USPEX problem solved: an exampleHydrogen at 600 GPa (16 atoms)• The USPEX run produced 1274 structures• From these the 794 within 0.5 eV from thelowest energy value found are selected• Manual analysis to remove duplicatedstructures from this set: ~20h of work• Using the CrystalFp classifier: ~10min• At the end found only 4 unique structures:• One α-Ga type (top)• One Cs-IV (bottom), the ground state (i.e. thelower energy structure), and two closely relatedstructuresCrystalFp, STM4 & USPEX - Mario Valle - 04/08/201116

Classifier integration in USPEXUSPEX structure cancerGenerationSolved atthe root!Original USPEX.A lot of identical structures.USPEX after the classifier integration.No more “structure cancer”!So, where is the problem?Compute uniquecoordinatesDefine distancemeasureSpace 100-3000dimensionalEach groupdescribes adistinct structureAdd groupingcriteriaCrystalFp, STM4 & USPEX - Mario Valle - 04/08/201117

So, where is the problem?Space 100-3000dimensionalSpace 100-3000dimensionalCrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011High-dimensionality is not intuitiveHigh dimensionalityspace is mostly emptyEverything is inthe corners ofthe hypercubeCrystalFp, STM4 & USPEX - Mario Valle - 04/08/201118

The curse of dimensionalityRoughly speaking, the higher the dimensionality,the lower the power of recognizing similar objectsBecause everything is at the same distance fromevery other point…Distance distribution for random points in a hypercubeCrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011But distances measures ……could help contrast this curse of dimensionalityEuclidean distance: , ∑ Minkowski distance: , ∑ Cosine distance: , 1cos ∙1 CrystalFp, STM4 & USPEX - Mario Valle - 04/08/201119

Tried various distance measuresEuclideanCrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011From the problem solution …Compute uniquecoordinatesDefine distancemeasureSpace 100-3000dimensionalEach groupdescribes adistinct structureAdd groupingcriteriaCrystalFp, STM4 & USPEX - Mario Valle - 04/08/201120

… to a new paradigmCompute uniquecoordinatesDefine distancemeasureSpace 100-3000dimensionalTo look at crystalstructures from anovel perspectiveHigh Dimspace toolsCrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011Unfold data to lower dimensionsMultidimensional scaling projects points from highdimensional space to a lower dimensional onepreserving distances between points as faithfully aspossibleSammon mapping to 2DOne famous test dataset (right I said right!) containspoints on a rolled sheet that forms a 3D shape calledthe “Swiss roll” (a superb example on the left)CCA mappingCrystalFp, STM4 & USPEX - Mario Valle - 04/08/201121

CrystalFp multi dim. scalingThe scatterplot tool inCrystalFp implements aForce Directed Placementmultidimensional scalingalgorithm (here the pointsare colored by energy)CrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011Nanocluster data from Dr. Gareth Tribello (USI)CrystalFp multi dim. scalingThe scatterplot tool inCrystalFp implements aForce Directed Placementmultidimensional scalingalgorithm (here the pointsare colored by energy)CrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011Nanocluster data from Dr. Gareth Tribello (USI)22

Study of energy landscapesA. R. Oganov and M. Valle,How to quantify energy landscapes of solids,The Journal of Chemical Physics, vol. 130,p. 104504, 2009.CrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011Energy landscape of Au 8 Pd 4 system24

More complex landscapesEnergy landscape for MgO with 32 atoms/cellCrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011New quantities: quasi-entropyFor each given structure, quasientropyis a measure of disorderand complexity of that structure.Si structures developing defectsS str is better correlated to energythan Steinhardt’s Q625

Faithful representation?Distances for three Si defectssimulations without correcting fortranslations and rotationsChart by Dr. Beat SahliIntegrated Systems Lab.ETH ZürichCrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011On average (on an interval), yes(using CrystalFp)Nanocluster data from Dr. Gareth Tribello (USI)(using VMD)26

(Totally) unexpected correlations• We found unexpected correlationsbetween distance and otherphysical variables• For example the deceptivelysimple H 2 O shows clearcorrelations and grouping• This and other datasets motivatedus to continue the exploration ofthe crystal fingerprints’ space…“And roughly the only mechanism forsuggesting questions is exploratory”A conversation with John W. Tukey and Elizabeth TukeyLuisa T. Fernholz and Stephan MorgenthalerStatistical ScienceVolume 15, Number 1 (2000), 79-9427

Classical data analysisproblemWe have a model of ourphenomena under studydatahypothesisWe use quantitativemethods to prove ordisprove our hypothesis(confirmative data analysis)analysisconclusionsCrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011Exploratory data analysisproblemWe do not start froman established modeldataWe focus on the data“Know your data”analysismodelWe try various graphicalmethods looking for the(hidden) model in an exploration-driven,evolutionary wayconclusionsCrystalFp, STM4 & USPEX - Mario Valle - 04/08/201128

CrystalFp – Parametric studyFingerprint cutoffAtoms per cellGeneral law29

Usage:CrystalFp [options] POSCARfile [ENERGIESfile]‐v ‐‐verbose (optional argument)Verbose level (if no argument, defaults to 1)‐? ‐h ‐‐help (no argument)This help‐t ‐‐elements (required argument)List of chemical elements‐es‐ss‐‐max‐step ‐‐end‐step (required argument)Last step to load (default: all)‐‐start‐step (required argument)First step to load (default: first)‐et ‐‐energy‐per‐structure (no argument)Energy from file is per structure, not per atom‐e ‐‐energy‐threshold (required argument)Energy threshold‐r ‐‐threshold‐from‐min (required argument)Threshold from minimum energy‐c ‐‐cutoff‐distance (required argument)Fingerprint forced cutoff distance‐n ‐‐nano‐clusters ‐‐nanoclusters (no argument)The structures are nanoclusters, not crystals‐b ‐‐bin‐size (required argument)Bin size for the pseudo‐diffraction methods‐p ‐‐peak‐size (required argument)Peak smearing size...30

Interesting correlationsCrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011Searching an explanationCrystalFp, STM4 & USPEX - Mario Valle - 04/08/201131

Distance vs. dimensionalityGaAs 8 atoms/cellcutoff: 3 – 30 Ådimensionality: 180 – 1800Distance decompositionGaAs 8 atoms/cellcutoff: 30 Ådimensionality: 1800CrystalFp, STM4 & USPEX - Mario Valle - 04/08/201132

But this one?Structure: Au 8 Pd 4Cutoff: 30 ÅDimension: 1800CrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011Intrinsic dimensionalityFingerprint spacedimensionality:100 ÷ 3000Dim space = 3Dim intrinsic = 2Vastly redundant!Theory:Dim intrinsic = 3*N atoms +3More realistic theory:Dim intrinsic = 3*N atoms +3-κAu 8 Pd 4 39 10.85MgNH 39 32.47MgO 99 11.62CrystalFp, STM4 & USPEX - Mario Valle - 04/08/201133

Constraints per atomDim intrinsic = 3*N atoms +3-κ (3-κ A )*N atoms +3Dataset Theor. Dim. Intr. Dim. k Aau8pd4 39 11.94 2.25Ca‐16at‐160GPa 51 27.37 1.48ca‐16at‐300GPa 51 29.19 1.36carbon‐0GPa‐8atoms‐final 27 25.88 0.14ch2‐800GPa‐18at 57 61.94 ‐0.27gaas‐8at_new 27 26.47 0.07h2o 39 90.78 ‐4.32H‐300GPa‐12at 39 4.48 2.88H‐500GPa‐16at 51 25.78 1.58H‐500GPa‐8at 27 4.29 2.84l4j8a 39 2.30 3.06l4j8 39 2.87 3.01mgnh‐2.5eV‐threshold1 39 9.67 2.44mgnh‐total4 39 53.03 ‐1.17mgo32a 99 10.58 2.76mgofull 99 15.05 2.62Na‐140GPa‐8at 27 27.22 ‐0.03urea‐0GPa 51 18.79 2.01GaAs‐old 27 5.64 2.67MgSiO3_Postperovskite_120GPa 63 56.34 0.33GaAs_random 27 23.45 0.44MgNH‐random 39 63.69 ‐2.06Frequency0 1 2 3 4 5Distribution ofk A0.11 2.620 1 2 3k ACrystalFp, STM4 & USPEX - Mario Valle - 04/08/2011Intrinsic dimensionalityFingerprint spacedimensionality:100 ÷ 3000Dim space = 3Dim intrinsic = 2Vastly redundant!Theory:Dim intrinsic = 3*N atoms +3More realistic theory:Dim intrinsic = 3*N atoms +3-κAu 8 Pd 4 39 10.85MgNH 39 32.47MgO 99 11.62H 2 O 39 80.50But how do you explain this?CrystalFp, STM4 & USPEX - Mario Valle - 04/08/201134

Synthetic datasetsDistance distribution vs.embedding dimension8 atoms with uniformly distributedrandom fractional coordinates in acubic unit cell with 5 Å sideIntrinsic dimension vs.embedding dimensionLessons learnedFrom the Modeling side• Using known concepts in unusual contexts is asource of unexpected insights• Discoveries happen on the boundaries of disciplines• “Seeing is believing” and convincing. Then thedomain experts become a source of ideasFrom the Visual Analysis side• Quick prototyping and experimentation capabilitiesare critical (that is, STM4 is a big help)• No need of fancy visualizations. What are neededare visualizations tuned to the problem at hand• Data management is critical to keep order in thedata explorationCrystalFp, STM4 & USPEX - Mario Valle - 04/08/201135

http://www.cscs.ch/~mvalle/STM4http://www.cscs.ch/~mvalle/CrystalFpGoing together…Thank youfor your Thank attention! you!And don’t forget: mvalle@cscs.ch36