Inside the Grid: Part 1
Benefits and Backend
The following is the first in a series of excerpts from text that I wrote with the help of John Flanagan and Brian Kelly. We used this as an extension of the SoftGrid training that we were offering right after the acquisition of Softricity by Microsoft as it is a “transcript” of a lecture I would give. I have broken this into a multi part series to make the whole 37 page original manuscript easier to digest on a blog post. Over the course of the posts we will dig deeper and deeper into the concepts of SoftGrid in a technical nature.
Major Benefits of SoftGrid
The following are some of the major benefits of SoftGrid that we will explore:
- Simplify Operating System Migrations
- Central Management and Control
- Eliminate Application Conflicts
- Simplify Updates to Applications
- Multi Tenancy
- Reduce Server Silos
- Run Multiple Versions of the same application on one system
- Terminate or Retire Applications Easily
- Configuration Control
Easier Deployments
The thought of migrating an Enterprise Organization’s base operating system can make even the most veteran IT professional spend many sleepless nights. Currently, organizations are faced with the perceived burden that they will have to upgrade their systems to Windows Vista which will in turn start a flurry of concern around application compatibility and manageability. A common thread of this concern is maintaining user up time, reducing time spent on regression testing and even uninstalling existing software.
Many organizations will attempt to ease this burden by employing an ESD solution to automatically deploy their applications once the new OS has been installed. However, even with an automatic installation of an application, a vast array of Windows based applications conflict with each other.
When an application is allowed to install onto a host or client computer, its programmed behavior is to add or modify files and registry settings on that client’s operating system. If that application did not add or modify settings in the registry and was contained exclusively in its own directory on the file system then conflicts would never occur. However, with the advent of thousands of applications after 1995, the application will almost always place its own files (.DLL, .VXD, .SYS, etc.) in numerous directories throughout the file system. It will also populate the registry of that client with its own values or modify existing values.
“Back in My Day”
Back in Windows 3.1, applications would install all of their assets into their own exclusive directory. This also included their settings files in the form of an .INI file, the registry was largely unused. Conflicts did not occur since the applications were isolated into their own directory structure such as C:\Winword\.
This problem would not be so abundant if enterprises ran only one or a very select group of applications. Since most enterprises will need to run countless applications, the chances of these applications causing a conflict with one another is almost a guarantee. Until SoftGrid, the solution for many organizations was to separate these applications onto different computers. In essence, a user would have two or more computers on their desk, one to run application A and another separate computer to run application B. Although this is a solution, it is very impractical from an expense and support perspective.
Whether or not an ESD solution is used, over time these settings and configurations will build up on the operating system. Even if an organization were to deploy Windows Vista the day it became available it is only a matter of time before this occurs. The only way to truly purge a system of this clutter is to perform a radical system rebuild or restore it to a clean state.
These major pain points of deployments are eliminated when SoftGrid is used because SoftGrid enabled applications are never installed or allowed to modify the client operating system and are made to run inside of their own virtual environments complete with all of the necessary assets (system files, registry, COM, embedded services, etc.) . The end result is the isolationist mentality of yesteryear with today’s complex and sophisticated applications.
To explore this further we will dig deeper into what makes an application SoftGrid enabled.
When an application is SoftGrid enabled it is made to run inside of its own virtual environment. Applications that are SoftGrid enabled are never allowed to install or modify the local file system or local registry; instead they are made to run inside of their own “Virtual Environment”. Contained inside of this virtual environment are all of the files, registry information, fonts, COM, embedded services and environment variables that the application would have normally installed and be expected to use on the client PC. Instead, with all of these “assets” residing inside the virtual environment (SystemGuard) the application will leverage them from this virtual environment and remain isolated from other applications that are also running inside of their own separate virtual environment (SystemGuard).
The process of creating the virtual environment (SystemGuard) is known as “Sequencing”. Microsoft’s SoftGrid Sequencer is the component in the SoftGrid system that is vital in creating the SystemGuard for an application or suite of applications.
SoftGrid Enabled applications will be able to use local and network drives, CPU, RAM, the local Windows Installer Service, printers, USB devices and other local resources on the SoftGrid client to which they are streamed into cache and run.
SystemGuard virtualizes many different operating system components, including:
Virtual COM: SystemGuard virtualizes COM communication, protecting one program from another.
Virtual Registry: SystemGuard creates a virtual registry for each application. Registry settings created are not seen by other applications—including the Windows Regedit program. Rather than copying the entire Registry, the Virtual Registry of SystemGuard utilizes an “overlay" (or “copy-on-write”) method—items in the real Registry may be read by the application as long as a virtual copy of the item is not available. All application writes to the registry are contained within the virtual registry.
Virtual File System: SystemGuard also handles requests made by applications to files in specific directories by redirecting the requests. For example, if an application looks for a file located in a specific directory on the local C: drive, SystemGuard redirects any requests to the Virtual File System (VFS). Dynamic Link Libraries (DLLs) specifically needed by the application are made available within SystemGuard, avoiding conflicts with different versions of the same DLL that may be installed locally or running in another SystemGuard on the machine. These components are shared and tracked inside the Virtual File System.
Virtual Services: New in 4.0 is the ability of SystemGuard to virtualize embedded services and the locally installed Windows Installer Service (MSI). For embedded services the service itself and its definition will only be visible within the environment it is created. Any program external to the environment which queries the Windows Service Control Manager will not see this service. Any number of services or versions of services will be able to be run, only constrained by how they are defined with the SystemGuard configuration. The locally installed Windows Installer Service will be virtualized at run-time, using its configuration from the SCM as a template for execution.
Virtual .INI: SystemGuard allows each application or instance to have private settings within virtual copies of standard Windows .INI files.
Virtual Process Environment: SystemGuard stores environment variables including the path environment variable.
Virtual Fonts: SystemGuard makes each font added during sequencing available to only the specific application during runtime on the SoftGrid Client. This prevents these fonts’ installation on the local machine, which would consume valuable system resources.
A Deeper Look at Sequencing
In order to understand this new concept of application virtualization, let us first take a closer look at where it all begins. The first major component in creating a SoftGrid enabled application that can run in its own virtual environment is the Sequencer. After we dig deeper into the SoftGrid System Components and see how they all work we will revisit the benefits of SoftGrid in a technical description.
Sequencing is a wizard based process that consists of the following 3 phases.
1. Package Configuration
2. Installation Phase
3. Launch Application
Each phase of the Sequencing process has distinct objectives and are critical in the overall success of a SoftGrid enabled application.
Package Configuration
During the Package Configuration Phase the Sequencer Utility prompts for information necessary to complete the OSD file which is a critical file in the launch of a SoftGrid enabled application.
Installation
During the Installation Phase the Sequencer component monitors an application’s installation in order to pack all of the assets of that application into the SystemGuard virtual environment. In order to get the most complete and unadulterated installation of an application it is vital that the workstation on which the Sequencer component runs be as clean as possible and reflects the client that will run the SoftGrid enabled application.
Launch Application
During the Launch Application Phase the Sequencer component is responsible for compiling and ordering the blocks of code necessary to launch the application on the client initially. This is referred to as Feature Block 1 (FB1). In order to optimize performance, the default behavior of SoftGrid is to deliver the contents of FB1 to the client initially. This allows the user to launch the application without having to to stream the entire package.
SoftGrid “Packages”
Because each application, or suite of applications, runs inside their own isolated SystemGuard virtual environment, they are allowed to remain conflict free. At the end of the three phases of Sequencing the person performing the Sequence completes the File – Save feature which results in the following four types of files. These files comprise a SoftGrid “package”.
1. ICO (~24 Kb) - Typically one per published application
The icon file is a shortcut that when properly published through the SoftGrid Management Console will appear on the SoftGrid client’s desktop (or wherever you determine it be placed). When clicked by a user, this icon file will point to the next file in the package which is the .OSD file. Technically a user does not need a .ICO file to launch a SoftGrid enabled application. They could instead launch it directly from the .OSD file.
2. OSD (~2Kb) - One per published application, or one per suite of applications.
The .OSD (Open Software Descriptor) is an XML based file that instructs the client on the necessary details to retrieve the SoftGrid enabled application from the SoftGrid server and run in its SystemGuard.
3. SFT (up to 4 GB) - One per published application, or one per suite of applications.
The .SFT (Softricity) file is the largest and most critical file in a SoftGrid enabled application’s package. It is inside this file that all of the application’s assets, including files, registry, fonts, COM, virtual services and the SystemGuard itself reside.
4. SPRJ (~13 Kb) - One per published application, or one per suite of applications.
The .SPRJ (Softricity Project File) is the newest file in the SoftGrid enabled application’s package. The Sequencer stores its Exclusion Items and Parse Items information in this XML based file. This file is used heavily in the creation of application records as well as during an upgrade to a package.
Because each Sequenced application or suite of applications is composed of these files they are allowed to remain separate from all other applications thus eliminating application conflicts. SoftGrid enabled applications are delivered to the SoftGrid clients using these files, eliminating the need to install the applications onto the client’s computer.
More detailed information about these files can be found in the official Softricity SoftGrid curriculum.
Analogy:
When I (application A) travel on the road for business, I Sequence myself by packing my suitcase (SystemGuard). Anything that I normally have “installed” at home and use during my normal routine is virtualized into my suitcase. I pack my socks (system files), my shirts (registry keys), my shoes (embedded services). I save my suitcase (my SFT file) and I stream over the network by traveling on the airplane. When I get to the client (hotel room) I do NOT unpack the suitcase, instead I open the virtual environment and live out of it. If I unpack my suitcase this would be the equivalent of installing myself into my client (hotel room).
When it comes time for me to perform my programmed behavior of getting dressed in the morning I may make a call to my socks in the dresser, as if I had been installed. Instead, SystemGuard stops that call and tells me to use the socks that are in my suitcase. It is only when I can not find something in my suitcase (e.g. shampoo) that SystemGuard allows me to fall through to the local host and use the resource, but with read only rights. It is essential to understand that the Sequencer component made an intelligent assumption that the resource in question (shampoo) would always be on every client this package streamed to. It did this because that same resource was in fact available on the machine that was used to perform the Sequence operation. This adds credence to the earlier statement that the Sequencing workstation needs to be clean and reflect the clients that will run the SoftGrid enabled application.
At the end of my stay when I leave with my suitcase there is no evidence that I was ever in that hotel room. I have left no visible “foot print”.
If my wife (application B) travels to the same hotel room (client) as I do, she will remain isolated and conflict free from me because she has packed herself (Sequenced) into her own separate suite case (SystemGuard). In this example there is no way that I can accidentally go into my wife’s suitcase.
Yet, if we know that my wife and I are compatible and we are meant to be integrated with each other, we could be sequenced into the same suitcase.
The Process Continued
Since we have started the process of making an application SoftGrid enabled we will continue to build on that momentum instead of ending it prematurely leaving you feeling incomplete. I assure you that we will revisit all of the benefits of SoftGrid later in this document.
Now that the Sequencer Engineer has saved their package, and has the four golden files as a result, it is time to move those package files to the SoftGrid Server. This needs to be done in order to make the SoftGrid enabled application available to the user population.
The SoftGrid Virtual Application Server (VAS)
During the installation of SoftGrid on my server, the SoftGrid Virtual Application Server Service (VAS) was installed. Because the overhead of this service is so low, the minimum hardware requirements of this box are not very taxing. In fact, a server with a single processor, 1 GB of RAM, and ample storage space for the package files will suffice for most normal situations. There should be a formal analysis in place to calculate a best effort on the amount of HDD space that will be required based on the number and size of anticipated SoftGrid packages. As you will see shortly, once the initial stream of the package has occurred to the client, this server has very little to do and can often sit idly waiting for additional requests. Don’t forget, unlike Terminal Server the applications do not run on the SoftGrid server at all. They merely stream from the SoftGrid server, and once cached on the client locally, they run from that local device.
The installation of the VAS prompts the Admin with an essential question, “Where is the ‘Content’ location”? By default the VAS thinks this is C:\Program Files\Softricity\SoftGrid Server\Content, but this can be located on any network accessible resource including a DFS, SAN, or similar device. This Content share is where the VAS will look for the .SFT file requested by users when they launch a SoftGrid enabled application.
To continue the flow of the process, once the package is saved by the Sequencer, the four golden files should be copied to this Content share. Once this is done you have accomplished nothing. You still have to “publish” the application to make it available to the users.
Requirements of the VAS :
C:\Program Files\Softricity\SoftGrid Server\content: This directory is used to store all the SFT, SPRJ, ICO, and OSD files created by the SoftGrid Sequencer. These files may be located on a network share via a configurable Registry key, or specified during installation.
Ports and Protocols: The VAS uses multiple protocols, determined when a particular action is triggered. When the SoftGrid Client talks with the SoftGrid Server it uses RTSP (554) or RTSPS (332) for open and close. Once the connection has been made it continues to send information over the RTP and RTCP high ports (49152 and greater up to 65535).
Services: When the VAS is installed it adds the SoftGrid Virtual Application Server service and sets it to start automatically.
Executables: The following executables are also installed and run by the SGVAS.
- sftdispatcher.exe: The dispatcher process is the controller for the SGVAS, and routes all connections to an available core process.
- sftcore.exe: This is the core process for the SGVAS. Three run by default and assist in reserving the buffer in RAM to mount packages to for streaming.
The SoftGrid Management Console (SGMC)
The next logical step in the Administrative process is to launch the central management utility of SoftGrid, the SoftGrid Management Console. This utility is the one and only place an Administrator needs to go to in order to fully administer the SoftGrid environment. From this utility you can publish application records, assign permissions, create file type associations, control application licensing, run reports, plus a plethora of other functions.
The SGMC is a snap-in to MMC version 2.0 or higher and can be installed on any compatible host. By default it is installed on the same device as the VAS. For more information on the SGMC please refer to the official SoftGrid curriculum.
In a concerted effort not to mislead anyone, let me state very clearly that no data is actually stored in the SGMC. All of the actions performed in the SGMC are actually stored in the SoftGrid Data Store. The SGMC is, in simple terms, a GUI peephole into the SQL database that is the Data Store.
SoftGrid Data Store
All, and I do mean all, information regarding the SoftGrid environment is stored in the Data Store that by default gets created at the time the first VAS is installed. The database can be created on either a SQL 2000, SQL 2005 or MSDE (recommended for test or demo environments only). Now, whenever someone approaches a DBA and tells that person they are going to create a new database on their SQL server the first response from that DBA is usually “Like Heck you are!”. Once the dust settles the next response is “How big is this new database going to get?”. Although sizing any component in a system is never an exact science and is something that needs to be monitored and adjusted as time passes, the expected size of the SoftGrid Data Store is very small at approximately 1 MB per 100 applications.
In short, almost everything you see in the SGMC is read from the Data Store and everything you do in the SGMC is written to the Data Store. Make no mistake here though, as the SGMC does not do any direct read or write operations to the SQL server. Instead, it uses an intermediary component in the SoftGrid Management Web Service (SGMWS).
When the SoftGrid server starts up it will contact the Data Store it was configured to use at installation and request its startup configuration. This Data Store must also be available to the administrator using the SGMC in order to add or modify anything about the SoftGrid environment.
SoftGrid Management Web Service (SGMWS)
Like the mediator in a conversation between two delegates at the United Nations, the SGMWS acts as a go-between for the SGMC and the Data Store. When you perform an action in the SGMC, such as publishing an application record and assigning permissions to it, it is the SGMWS that performs the write action to the Data Store. Conversely, when you see objects displayed in the SGMC it is the SGMWS that performs the read action to the Data Store and AD.
The SGMWS requires IIS 5.0 or greater with ASP.Net installed and the permissions set to allow in the IIS Manager. The easiest way to accomplish this on a Windows 2003 Server is to add these components prior to installing SoftGrid through the Add/Remove Programs in Control Panel. Also, because .Net Remoting is used as the method of interaction between the SGMC and the SGMWS, both of them require .Net Framework 1.1 or greater to be installed.
And like the cherry on top of the SGMWS sundae, MDAC 2.7 or greater is required since the SGMWS is going to make OLE DB hooks into the SQL Data Store. Of course the same data access components of MDAC 2.8 are available on Windows 2003 by default.
Once you launch the SGMC it will prompt you for the name of the SGMWS to connect to. The SGMWS will attempt to authenticate the user account as being a member of the group designated as the SoftGrid Administrators. Only members of this group are allowed to launch the SGMC.
Designer Tip: Because the SoftGrid Management Web Service will be accessing the Data Store for read/write operations it would be a good design principle to place the SoftGrid Management Web Service and Data Store on the same high speed, reliable segment.
Our Progress So Far
So far we have taken an application’s installer and monitored it through the three phases of the Sequencer creating the four golden files of the SoftGrid package. We have copied those to the \Content location of the VAS and published an application record to a group of users from our AD domain using the SGMC who used the SGMWS to write the information to the SQL Data Store. Coming up next will be Part 2: The SoftGrid Client.
- Sean Donahue
Comments
Anonymous
January 01, 2003
The SoftGrid connector will be deprecated in the near future and replaced with Windows Installer ApplicationAnonymous
January 01, 2003
So finally the ledged that is Sean Donahue has finally got his hands on the SoftGrid Blog (which is greatAnonymous
January 01, 2003
It is very cool application virtualization technology. Check out this post for details: http://blogs.technet.com/softgrid/archive/2007/08/02/inside-the-grid-part-1.aspxAnonymous
January 01, 2003
Wait...who wrote this? I thought it was Doug Brown at first. http://www.dabcc.com/article.aspx?id=5319Anonymous
January 01, 2003
So Sean D this time has updated his series on inside the grid ! Part Two has kicked off and it has someAnonymous
January 01, 2003
Inside the Grid: Part 1 Inside the Grid: Part 2 Inside the Grid: Part 3Anonymous
January 01, 2003
The SoftGrid connector will be deprecated in the near future and replaced with Windows Installer Application