Online Store Automation - Populating Our Store (Part 2)

Online Store Automation - Populating Our Store (Part 2)

Going back to my previous post, I wanted to add more details on how webpages can be extracted, especially in the case where no login is required.

If there was no need for a web-based login for a page, I could have extracted the content of a page without the need for a WebBrowser control. The following code takes a web page based on a URI and extracts its contents into a string.

 1:         
    static public string Extract(string URI, int MaxRedirects, System.Net.NetworkCredential credentials)

 2:         {

 3:             WebRequest myWebRequest = CreateRequest(URI, MaxRedirects);

 4:             myWebRequest.Credentials = credentials;

 5:             WebResponse myWebResponse = myWebRequest.GetResponse();

 6:  

 7:             Stream ReceiveStream = myWebResponse.GetResponseStream();

 8:             Encoding encode = Encoding.GetEncoding("utf-8");

 9:             StreamReader readStream = new StreamReader(ReceiveStream, encode);

 10:             Char[] read = new Char[256];

 11:             int count = readStream.Read(read, 0, 256);

 12:             string html = "";

 13:             while (count > 0)

 14:             {

 15:                 // Dump the 256 characters on a string and display the string
    onto the console.

 16:                 html += new string(read, 0, count);

 17:                 count = readStream.Read(read, 0, 256);

 18:             }

 19:  

 20:             readStream.Close();

 21:             ReceiveStream.Close();

 22:             myWebResponse.Close();

 23:  

 24:             return html;

 25:         }

The code for the "CreateRequest" method is as follows:

 1:         
    static public WebRequest CreateRequest(string URI)

 2:         {

 3:             HttpWebRequest myWebRequest = WebRequest.Create(URI) as HttpWebRequest;

 4:             myWebRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
    5.0; .NET CLR 1.1.4433)";

 5:             myWebRequest.KeepAlive = false;

 6:             myWebRequest.AllowAutoRedirect = false;

 7:             return myWebRequest;

 8:         }

This is pretty straightforward but thought I should share this for completness.