The Art of SQL Server FILESTREAM - Red Gate Software

IntroductionWhen I first started my database career in the early 1990s, all the applications on whichI worked dealt with simple relational data, stored in a few relational tables. The mostcommonly used data types were strings, numbers, dates, and Boolean. Many of these weresingle-user applications that lived on a single computer.Soon, however, and rapidly, applications started becoming more complex, and new andmore challenging data processing requirements began to emerge. Many applicationsstarted dealing with unstructured data. Along with relational data, our applicationsneeded to be able to generate, process and serve non-relational data such as XML, images,music, videos, and so on.SQL Server 6.5 offered two data types for storing large chunks of text and binary data,namely text and image respectively. SQL Server 7.0 added one more data type,ntext, which enabled storage of Unicode text. SQL Server 2005 added the data typesVARCHAR(MAX), NVARCHAR(MAX) and VARBINARY(MAX), which made reading andwriting chunks of unstructured data much simpler. Despite these improvements, thebasic underlying problem remained: SQL Server is primarily a relational databasemanagement system and optimized for processing relational data. Reading and writinglarge chunks of text/binary data within the database requires a lot of SQL Server internalmemory (buffers as big as the size of the chunk being read/written), and multiple,simultaneous requests for large blocks of data can put the server under severememory pressure.The workaround adopted by most database architects was to store the unstructured dataas disk files on the file system, and store only the path to the file in the relational table.The file system is optimized for reading and writing large files, and offers streamingcapabilities. This solved the performance problems, but introduced new challenges. Inthis scheme, we had no way to ensure transactional consistency between relational dataand data in the file system. Also, management of backups was complicated, since thenormal database backups were only possible if accompanied by the corresponding backupof the file system.19