How to stream XML fragments from an XmlReader (LINQ to XML)
When you have to process large XML files, it might not be feasible to load the whole XML tree into memory. This article shows how to stream fragments using an XmlReader in C# and Visual Basic.
One of the most effective ways to use an XmlReader to read XElement objects is to write your own custom axis method. An axis method typically returns a collection such as IEnumerable<T> of XElement, as shown in the example in this article. In the custom axis method, after you create the XML fragment by calling the ReadFrom method, return the collection using yield return
. This provides deferred execution semantics to your custom axis method.
When you create an XML tree from an XmlReader object, the XmlReader must be positioned on an element. The ReadFrom method doesn't return until it has read the close tag of the element.
If you want to create a partial tree, you can instantiate an XmlReader, position the reader on the node that you want to convert to an XElement tree, and then create the XElement object.
The article How to stream XML fragments with access to header information contains information on streaming a more complex document.
The article How to perform streaming transform of large XML documents contains an example of using LINQ to XML to transform extremely large XML documents while maintaining a small memory footprint.
Example: Create a custom axis method
This example creates a custom axis method. You can query it by using a LINQ query. The custom axis method StreamRootChildDoc
can read a document that has a repeating Child
element.
static IEnumerable<XElement> StreamRootChildDoc(StringReader stringReader)
{
using (XmlReader reader = XmlReader.Create(stringReader))
{
reader.MoveToContent();
// Parse the file and display each of the nodes.
while (reader.Read())
{
switch (reader.NodeType)
{
case XmlNodeType.Element:
if (reader.Name == "Child") {
XElement el = XElement.ReadFrom(reader) as XElement;
if (el != null)
yield return el;
}
break;
}
}
}
}
static void Main(string[] args)
{
string markup = @"<Root>
<Child Key=""01"">
<GrandChild>aaa</GrandChild>
</Child>
<Child Key=""02"">
<GrandChild>bbb</GrandChild>
</Child>
<Child Key=""03"">
<GrandChild>ccc</GrandChild>
</Child>
</Root>";
IEnumerable<string> grandChildData =
from el in StreamRootChildDoc(new StringReader(markup))
where (int)el.Attribute("Key") > 1
select (string)el.Element("GrandChild");
foreach (string str in grandChildData) {
Console.WriteLine(str);
}
}
Module Module1
Sub Main()
Dim markup = "<Root>" &
" <Child Key=""01"">" &
" <GrandChild>aaa</GrandChild>" &
" </Child>" &
" <Child Key=""02"">" &
" <GrandChild>bbb</GrandChild>" &
" </Child>" &
" <Child Key=""03"">" &
" <GrandChild>ccc</GrandChild>" &
" </Child>" &
"</Root>"
Dim grandChildData =
From el In New StreamRootChildDoc(New IO.StringReader(markup))
Where CInt(el.@Key) > 1
Select el.<GrandChild>.Value
For Each s In grandChildData
Console.WriteLine(s)
Next
End Sub
End Module
Public Class StreamRootChildDoc
Implements IEnumerable(Of XElement)
Private _stringReader As IO.StringReader
Public Sub New(ByVal stringReader As IO.StringReader)
_stringReader = stringReader
End Sub
Public Function GetEnumerator() As IEnumerator(Of XElement) Implements IEnumerable(Of XElement).GetEnumerator
Return New StreamChildEnumerator(_stringReader)
End Function
Public Function GetEnumerator1() As IEnumerator Implements IEnumerable.GetEnumerator
Return Me.GetEnumerator()
End Function
End Class
Public Class StreamChildEnumerator
Implements IEnumerator(Of XElement)
Private _current As XElement
Private _reader As Xml.XmlReader
Private _stringReader As IO.StringReader
Public Sub New(ByVal stringReader As IO.StringReader)
_stringReader = stringReader
_reader = Xml.XmlReader.Create(_stringReader)
_reader.MoveToContent()
End Sub
Public ReadOnly Property Current As XElement Implements IEnumerator(Of XElement).Current
Get
Return _current
End Get
End Property
Public ReadOnly Property Current1 As Object Implements IEnumerator.Current
Get
Return Me.Current
End Get
End Property
Public Function MoveNext() As Boolean Implements IEnumerator.MoveNext
While _reader.Read()
Select Case _reader.NodeType
Case Xml.XmlNodeType.Element
Dim el = TryCast(XElement.ReadFrom(_reader), XElement)
If el IsNot Nothing Then
_current = el
Return True
End If
End Select
End While
Return False
End Function
Public Sub Reset() Implements IEnumerator.Reset
_reader = Xml.XmlReader.Create(_stringReader)
_reader.MoveToContent()
End Sub
#Region "IDisposable Support"
Private disposedValue As Boolean ' To detect redundant calls
' IDisposable
Protected Overridable Sub Dispose(ByVal disposing As Boolean)
If Not Me.disposedValue Then
If disposing Then
_reader.Close()
End If
End If
Me.disposedValue = True
End Sub
Public Sub Dispose() Implements IDisposable.Dispose
Dispose(True)
GC.SuppressFinalize(Me)
End Sub
#End Region
End Class
This example produces the following output:
bbb
ccc
The technique used in this example maintains a small memory footprint even for millions of Child
elements.