Sun QFS and Sun Storage Archive Manager (SAM) Release 5.0 ...

Sun QFS and SunStorage ArchiveManager (SAM)Release 5.0 & BeyondHarriet CoverstonDistinguished EngineerSun Microsystems, Inc.June, 2009

Page 2Sun QFS and Sun SAMChallenge: Cost Efficient DataManagement and Fast Accessto Large Volumes of Data● Budgets remain flat while data growth isexploding> Increasing management costs● Compliance requirements● Users require timely access to informationthroughout its lifecycle

Page 3Sun QFS and Sun SAMSun Storage SoftwareAdvanced Data Management Software• Sun QFS – Shared File System> High performance parallel SAN file system> Native Linux Clients> http://www.sun.com/storage/management_software/data_management/qfs• Sun Storage Archive Manager (SAM)> Policy-based automatic data migration (local & remote)> Tiered Storage (supports both disk & tape)> http://www.sun.com/storage/management_software/data_management/sam

Sun QFS and Sun SAMOpen Source SoftwareBuild Community• Source is open!http://opensolaris.org/os/project/samqfshttp://blogs.sun.com/samqfs/• SAM APIs are open!> Allow users to manage data in SAM-QFS from within anapplication programhttp://developers.sun.com/solaris/articles/libsam.html• Discussion lists> Discussion list for general topics or issueshttp://mail.opensolaris.org/mailman/listinfo/sam-qfs-discuss> Development alias for specific questions about source, sourcecontributions, or code reviewshttp://mail.opensolaris.org/mailman/listinfo/sam-qfs-devPage 4

Page 5Sun QFS and Sun SAMSun QFS - Shared File System• Large, existing, and loyal customer base> Stable base, shipping since Aug 2002• Targets large enterprises, Web, and HPC> Clients run on Solaris (SPARC, x64, & X86) & Linux> Metadata server runs on Solaris (SPARC & x64)> HA option with Solaris Cluster• Optional WORM functionality for businesscompliance• QFS Shared configuration supports 512 nodes

Page 6Sun QFS and Sun SAMQFS Shared File System Benefits• Data consolidation with SAN file sharing> HBO – 5000 hours of programming to manage> “Provided the scalability to store and manage large files createdby program-length video with the performance necessary to meetHBO's demanding throughput goals”http://www.sun.com/customers/storage/hbo.xml• Performance and scalability> Near raw I/O performance for streaming I/O andtransactional I/O> File system I/O performance scales linearly with the hardware• Parallel processing w/ multi-node read/write access• Built in automatic & continuous data protection w/SAM

Page 7Sun QFS and Sun SAMQFS Certified with Solaris Cluster• Solaris Cluster HA failover support> Standalone QFS> HA-NFS over QFS> HA-SAM• Solaris Cluster Advanced Edition for Oracle RACrelies on Shared QFS with Solaris Cluster for HA> Oracle certified on 9i, 10g, and 11g• Oracle Billing and Revenue Management uses sQFS> “The best known transaction rate seen on the RAC-database can be achievedwhen the data files are on the QFS file system .”http://www.oracle.com/industries/communications/pdfs/oracle-sun-performance-benchmark-wp.pdf

Sun QFS and Sun SAMSun Storage Archive Manager (SAM)• Policy based archiving> Media can be disk, tape, or optical> Local and remote copies> Classification is path, owner, group, size, wildcard & access• Media format is tar – open format> Small files are put into a tar container so data is streamed atdevice speeds out to the tape• Keeps all data available, but not on high cost storage> Archives data across the tiers according to access patterns• On-demand, transparent file retrieval• Continuous data protection – no waiting until midnightPage 8

Page 9Sun QFS and Sun SAMSAM Customer Benefits• Leverage existing hardware – defer high cost diskpurchase> Add (cheaper) storage tiers transparently• Provide timely protection and timely access toinformation throughout its lifecycle• Meet compliance requirements with WORM support• Eliminate backup window problem & complexity> Reduce operational costs• Quick disaster recovery for business continuance

Page 10Sun QFS and Sun SAMSAM-QFS Data ConsolidationIntegrated Data ManagementUNIX WorkgroupNT WorkgroupMixed WorkgroupTCP/IPNFSQFSSAMBA/CIFSEarth Pluto VenusQFSMeta-DataServer SAMClient ClientQFSNFSSolarisData ProtectionFibre Channelor Ethernet SwitchSANTransparentData ProtectionMeta-DataDisk(s)Sharefs_1And Instant RecoveryData ConsolidationAnd Instant RecoveryUser-Data Disk(s)

Page 11Sun QFS and Sun SAMSAM's Archives are OPEN!• Media format is open, not proprietary – tar format> Files can be recovered with or without SAM – our mediaformat is open, NOT proprietary• Metadata about the data is on the archives> If file system metadata is lost, the archives can berecovered with a procedure we call the “Ultimate DisasterRecovery”

Page 12Sun QFS and Sun SAMSAM-QFS Management SimplifiedWizard Guided Set-up and Browser-Based User Interface• Centralized browser-based configuration and management ofSAM and QFS on multiple hosts• Configuration Wizards for> Archive policy> File system creation> Adding tape libraries• Archive mediaoperations andreporting• Support for multiplelevels of privilege usingroles

Page 13Sun QFS and Sun SAMSupport for Monitoring SAM and QFS• The monitoring console(shown here) lets admins quicklyunderstand their SAM environment– Potential trouble spots are indicated by severity icons in the lefthand panel.• e-mail notifications can be configured to alert admins of problemswith file systems, archiving and archive media• System metrics provide archive media reports and file datadistribution charts• Faults provide a record of adverse conditions that have occurredin the system (including tape alerts)

Page 14Sun QFS and Sun SAMInfinite Archive System (IAS)Client Client ClientFiler Filer FilerPolicy andArchivingServices(SAM-QFS)SANFabricFC SATA TAPE* over list price of separate componentsIASCore Features & Functions> Complete archive solution in a rack> Software, Servers, Storage, Services> Simple ordering process> Works with existing tape libraries> Automated data migration> Continuous backup protectionKey Benefits> Lower Cap-Ex> 20% - 33% savings*> Lower Op-Ex> Simple administration> Easy installation

HPC Storage SolutionsHigh BandwidthScalable Storage Clusterwith LustreCompute ClusterMetadataServersLong-TermData Retentionwith SAM-QFSNear Line ArchiveIB NetworkLoadSANObject Storage FarmArchiveData MoversSun Microsystems, confidential, internal use onlyHome DirectoriesTier 1 Archive

Page 16Sun QFS and Sun SAM5.0 Features• Solaris 10 & Open Solaris (Sparc/AMD/Intel)> Solaris 10 released April, 2009> OpenSolaris planned for 2009.06OpenSolaris release• Online grow and shrink (with & without SAM)• Rolling upgrades for Shared QFS• Directory Lookup Performance Improvements• SAM performance improvements> Archiver, stager, samfsdump performance

Page 17Sun QFS and Sun SAM5.0 Featurescontinued...• Sideband MySQL database for fast SAMqueries• Solaris Feature Integration> Zones Support> Solaris Service Management Facility (SMF)> VPM virtual memory performance improvement for X64• GUI Enhancements:> Usability study enhancements (including first timeconfiguration checklist)> Online Grow/Shrink, Shared Client On/Off, WORM

Page 18Sun QFS and Sun SAMOnline Grow• Support on-line grow of file system by adding a newLUN – a meta, data, or stripe group LUN> samu command: add eq> After grow of standalone QFS, the LUN state is ON> After grow of Shared QFS, the LUN state is UNAVAIL– Solaris clients execute samd buildmcf followed by asamd conf (applications may continue to run)– Linux clients must umount, samd conf– SANergy clients must umount, fuse– After all mounted clients have updated mcfs, theLUN can be changed to ON with the samucommand: alloc eq

Sun QFS and Sun SAMOnline Shrink• Support on-line shrink of file system (remove dataLUN or stripe group in a ma file system)> samu command: release eq> After release command, the LUN state is NOALLOC> Archived files on the LUN are released and the files aremarked offline>After all files have been released, the LUN state is OFF> samu command: remove eq> After remove command, the LUN state is NOALLOC>The LUN data is copied to other available LUNs> After all data has been moved, the LUN state is OFF> Results of the shrink are in the shrink.log (configured in/etc/opt/SUNWsamfs/shrink.cmd)Page 19

Page 20Sun QFS and Sun SAMShared QFS Rolling Upgrades• Support upgrading Shared QFS without takingdown the rest of the cluster> Requires that the MDS and alternate MDSs areupgraded first> Clients can then be upgraded individually> Limit QFS versions to n and n+1 with the MDSsalways being upgraded to n+1 first> Base support starting at 5.0

Page 21Sun QFS and Sun SAMDirectory Performance Improvements• Trust the directory caching> Trust the cache when the entry exists and do notread the entry from disk to verify• Increased our recommended files per directoryto 500,000• Performance measurements> Rewrite of 500,000 existing files in the samedirectory is 33% faster; Removal is 800% faster> Postmark test of 1 million files and 100 directories(10,000 files per directory) is 10% faster

Sun QFS and Sun SAMZones• There are 2 types of zones: global and nonglobal• There are two ways to use zones:> Standalone QFS is mounted first in the globalzone and then mounted in the non global zonewith the loopback mount option> Shared QFS is mounted first in the global zonebeneath a zonepath>Shared QFS is visible in only one non-global zone• SAM runs in the global zone>SAM admin commands only execute in the global zonePage 22

Page 23Sun QFS and Sun SAMSMF – Solaris Management Facility• SAM and QFS automatic startup, monitoring,and restart capabilities are managed andobservable via the SMF facility> Manage error conditions under SMF> sam-fsd which was started in /etc/inittab will bechanged to be (re-)started by SMF> fsmgmtd was already changed from /etc/inittaband is (re-)started by SMF

Page 24Sun QFS and Sun SAMSAM-QFS ManagerEnhancements in 5.0• Add support for the configuration and mgmt. oflarge scale Shared QFS systems> Provide a status overview with drill-down to allowusers to quickly understand the status of the filesystem and its clients> Allow addition of multiple clients at once> Perform operations on sets of managed clientse.g. mount/umount/change mount options> Provide a search/filter capability to display clientsby name, ip address, QFS version and status

Page 25Sun QFS and Sun SAMSAM-QFS ManagerEnhancements in 5.0 continued• Provide GUI support for online grow and shrink• Support WORM mount options and file attributes inthe file browser• Introduce a first time configuration checklist• Allow users to see the catalog and current VSNassignments when pooling or assigning media• Simplify the monitoring console, add library wizardand new file system wizard based on feedback froma usability study

Page 26Sun QFS and Sun SAMArchiver Scalability Improvements• Examine list feature improves performance bychanging the sam-arfind's worklist from a list ofdirectories to the actual list of modified files> In 4.6, directories are scanned to find modifiedfiles; this feature minimizes file system metadataimpact by the archiver

Page 27Sun QFS and Sun SAMStager Performance Improvements• Align stager writes to the disk cache to tapeblock boundaries> Currently, they are never aligned due to the 512byte tar header• Stager performance is improved by eliminating theread/modify/write when the stager writes to the diskcache

Page 28Sun QFS and Sun SAMProject ID's• Solaris project ID's are supported• The Solaris project ID is inherited from the taskand is stored in the inode• The project ID may be reset with the chprojcommand

Page 29Sun QFS and Sun SAMMySQL Sideband Database• Solaris Door interface provides fast event-basedfilesystem notifications• Events generated by QFS for create, modify,rename, remove, archive, and release/online• Sam-dbupd update daemon performs specificactions based on the type of event.> Consistency checks performed if any problems• Prepared SQL statements are used to reduceparsing overhead

Page 30Sun QFS and Sun SAMMySQL Database Schema• Relational tables – inode, file, path, and archive> Connected by ino/gens a m _ i n o d e (i n o I N T U N S I G N E D N O T N U L L ,g e n I N T U N S I G N E D N O T N U L L ,t y p e T I N Y I N T U N S I G N E D N O T N U L L ,s i z e B I G I N T U N S I G N E D D E F A U L T 0 ,c s u m C H A R ( 3 2 ) ,c r e a t e _ t i m e I N T U N S I G N E D D E F A U L T 0 ,m o d i f y _ t i m e I N T U N S I G N E D D E F A U L T 0 ,u i d I N T U N S I G N E D N O T N U L L ,g i d I N T U N S I G N E D N O T N U L L ,o n l i n e T I N Y I N T U N S I G N E D N O T N U L L ,P R I M A R Y K E Y ( i n o , g e n ) )s a m _ f i l e (p _ i n o I N T U N S I G N E D N O T N U L L ,p _ g e n I N T U N S I G N E D N O T N U L L ,n a m e _ h a s h S M A L L I N T U N S I G N E D N O T N U L L ,n a m e V A R C H A R ( 2 5 6 ) N O T N U L L ,i n o I N T U N S I G N E D N O T N U L L ,g e n I N T U N S I G N E D N O T N U L L ,P R I M A R Y K E Y ( p _ i n o , p _ g e n , n a m e _ h a s h , n a m e ) ,I N D E X ( i n o , g e n ) )s a m _ p a t h (i n o I N T U N S I G N E D N O T N U L L ,g e n I N T U N S I G N E D N O T N U L L ,p a t h V A R C H A R ( 4 0 9 6 ) ,P R I M A R Y K E Y ( i n o , g e n ) ,I N D E X ( p a t h ) )s a m _ a r c h i v e (i n o I N T U N S I G N E D N O T N U L L ,g e n I N T U N S I G N E D N O T N U L L ,c o p y T I N Y I N T U N S I G N E D N O T N U L L ,s e q S M A L L I N T U N S I G N E D N O T N U L L ,m e d i a _ t y p e C H A R ( 4 ) N O T N U L L ,v s n C H A R ( 3 2 ) N O T N U L L ,p o s i t i o n B I G I N T U N S I G N E D N O T N U L L ,o f f s e t I N T U N S I G N E D N O T N U L L ,s i z e B I G I N T U N S I G N E D N O T N U L L ,c r e a t e _ t i m e I N T U N S I G N E D D E F A U L T 0 ,s t a l e T I N Y I N T U N S I G N E D D E F A U L T 0 ,P R I M A R Y K E Y ( i n o , g e n , c o p y , s e q ) ,I N D E X ( m e d i a _ t y p e , v s n ) )

Page 31Sun QFS and Sun SAMSamdb Database Utility• Samdb for managing filesystem databases> Check – Consistency check against filesystem> Create – New database for filesystem> Dump – Dump contents of database (for samfsdump)> Drop – Delete a database> Load – Load database contents (from samfsdump)> Query – Query database based on file / vsn> samdb query samfs1 -v VSN001/samfs1/dira/file1/samfs1/dirz/file87

Page 32Sun QFS and Sun SAMSamfsdump Performance• Utilize a MySQL database to collect create, remove,rename, and archiving events. The file systemmetadata is still the primary copy• Designed to improve full file system samfsdumpperformance by reducing samfsdump data capturewindow• Database and inode information are post-processedinto the samfsdump file format• Existing samfsdump functionality remains available

Sun QFS and Sun SAMSam Parameters & Defaults• maxphys = 1MB or /etc/system setting whichever is bigger• wr_throttle = 2% of main memory (i.e., 16GB memory = 320MB wr_throttle)• archmax value increased by device (LTO,T10k,3592,TS1120=22GB, 9940=11GB,9840=4GB, DLT=11GB, all others 8GB, optical & disk 1GB• Startage = 2h, startcount = 500000 files, startsize = 90% of archmax• archivemeta = off, sort = path, removed -join command• archiver.cmd defaults with wait on so admin can build archiver.cmd file• Archiver (16,8192) and stager (10,1024) bufsize and limit values increased• maxactive = 5000/gigabyte of memory, maximum=500000• stage_n_window default size = 8192k• defaults.conf: avail_timeout – Stager delays before unloading a volume (default=0)• Split disk archive streams based on 1GB increments• Recycler mingain based on media type (< 200GB 60%, >= 200GB 90%)Page 33

Page 34Sun QFS and Sun SAMPerformance Test Status• Results with the PostMark file system benchmark- Postmark test run on 03/25/09 using the 6800 and 8 T3's. - Postmark test run on 03/26- ma-mm-mr file system type with 8 meta data devices (mm) and 8 data (mr) devices.- ms-md file system type with same 8 devices as for the ma-mm-mr runs- There were 200,000 files in one directory and also did 200,000 transactions.SAM-QFS 4.6 SAM-QFS 5.0 SAM-QFS 4.6 SAM-QFS 5.0Times (sec) build 4.6.73 Build 5.0.4 Improvement Times (sec) build 4.6.73 Build 5.0.4 ImprovementCreation 62 42 1.5 x Creation 62 43 1.4 xTransaction 4,256 1,060 4.0 x Transaction 3,407 319 10.7 xDeletion 132 83 1.6 x Deletion 92 88 1.0 xOverall 4,450 1,185 3.8 x Overall 3,561 450 7.9 x• Additional performance testsSAM-QFS 4.6 SAM-QFS 5.0Test Times (sec) build 4.6.73 Build 5.0.4 ImprovementRun with 4 large files and stager set so that ifdirect I/O is used, it is properly aligned withdirect I/O.samfs dump after archiving 6 million files(effect of MySQL side band)Stager wall time 845 528 1.6 xStager CPU time 655 2 330 xwall time 1315 125 11 x100 million file archive test wall time 18 hr 14 hr 1.3 x

Page 35Sun QFS and Sun SAM5.0u1 Planned Features• Release of 5.0u1 planned for fall, 2009• Solaris 10 & Open Solaris (Sparc/AMD/Intel)• Red Hat Linux 5.0 support added• Extended Attributes• Increased HA support with Solaris Cluster• Device Qualifications

harriet.coverston@sun.com