Recipe 11.3. Extracting Data While Parsing a Document

Recipe 11.3. Extracting Data While Parsing a Document Credit: Rod Gaither Problem You want to process a large XML file without loading it all into memory. Solution The method REXML::Document.parse_stream gives you a fast and flexible way to scan a large XML file and process the parts that interest you. Consider this XML document, the output of a hypothetical program that runs auto mated tasks. We want to parse the document and find the tasks that failed (that is, returned an error code other than zero). event_xml = %{ <events> <clean system="dev" start="01:35" end="01:55" area="build" error="1" /> <backup system="prod" start="02:00" end="02:35" size="2300134" error="0" /> <backup system="dev" start="02:00" end="02:01" size="0" error="2" /> <backup system="test" start="02:00" end="02:47" size="327450" error="0" /> </events>} We can process the document as it's being parsed by writing a REXML:: StreamListener subclass that responds to parsing events such as tag_start and tag_end. Here's a subclass that listens for tags with a nonzero value for their error attribute. It prints a message for every failed event it finds. require 'rexml/document' require 'rexml/streamlistener' class ErrorListener include REXML::StreamListener def tag_start(name, attrs) if attrs["error"] != nil and attrs["error"] != "0" puts %{Event "#{name}" failed for system "#{attrs["system"]}" } + %{with code #{attrs["error"]}} end end end To actually parse the XML data, pass it along with the StreamListener into the method REXML::Document.parse_stream: REXML::Document.parse_stream(event_xml, ErrorListener.new) # Event "clean" failed for system "dev" with code 1 # Event "backup" failed for system "dev" with code 2 Discussion We could find the failed events in less code by loading the XML into a Document and running an XPath query. That approach would work fine for this example, since the document only contains four events. It wouldn't work as well if the document were a file on disk containing a billion events. Building a Document means building an elaborate in-memory data structure representing the entire XML document. If you only care about part of a document (in this case, the failed events), it's faster and less memory-intensive to process the document as it's being parsed. Once the parser reaches the end of the document, you're done. The stream-oriented approach to parsing XML can be as simple as shown in this recipe, but it can also handle much more complex scenarios. Your StreamListener subclass can keep arbitrary state in instance variables, letting you track complex combinations of elements and attributes. See Also The RDoc documentation for the REXML::StreamParser class The "Stream Parsing" section of the REXML Tutorial (http://www.germane-software.com/software/rexml/docs/tutorial.html#id2248457) Recipe 11.2, " Extracting Data from a Document's Tree Structure"