F9 Group Marketing and Technology Blog

Screen scraping is the process of programmatically accessing and processing information from an external website. For example, a price comparison website might screen scrape a variety of online retailers to build a database of products and what various retailers are selling them for. Typically, screen scraping is performed by mimicking the behavior of a browser – namely, by making an HTTP request from code and then parsing and analyzing the returned HTML. The .NET Framework offers a variety of classes for accessing data from a remote website, namely the WebClient class and the HttpWebRequest class . These classes are useful for making an HTTP request to a remote website and pulling down the markup from a particular URL, but they offer no assistance in parsing the returned HTML. Instead, developers commonly rely on string parsing methods like String.IndexOf , String.Substring , and the like, or through the use of regular expressions. Another option for parsing HTML documents is to use the Html Agility Pack , a free, open-source library designed to simplify reading from and writing to HTML documents

With WebForms, each ASP.NET page’s rendered output includes a <form> element that performs a postback to the same page whenever a Button control within the form is clicked, or whenever the user modifies a control whose AutoPostBack property is set to True. This model simplifies web page development, but carries with it some costs – namely, the large amount of data exchanged between the client and the server during a postback. On postback the browser sends the values of all of its form fields (including hidden ones, like view state, which may be quite large) to the server; the server then sends back the entire contents of the web page. While there are some scenarios where this amount of information needs to be exchanged, in many cases the user has performed some action that requires far less information to be exchanged. With a little bit of forethought and code we can have the browser and server exchange much less data, which leads to more responsive web pages and an improved user experience. Over the past several weeks I’ve been writing an article series on accessing server-side data from client script. Rather than rely solely on forms and postbacks, many websites use JavaScript code to asynchronously communicate with the server in response to the page loading or some other user action. The server, upon receiving the JavaScript-initiated request, returns just the data needed by the browser, which the browser then seamlessly integrates into the web page. There are a variety of technologies and techniques that can be employed to provide both the needed server- and client-side functionality.

Today’s websites commonly exchange information between the browser and the web server using Ajax techniques – the browser executes JavaScript code typically in response to the page loading or some user action. This JavaScript makes an asynchronous HTTP request to the server. which then processes the request and, perhaps, returns data that the browser can then seamlessly integrate into the web page. Two earlier articles – Accessing JSON Data From an ASP.NET Page Using jQuery and Using Ajax Web Services, Script References, and jQuery , looked at using both jQuery and the ASP.NET Ajax Library on the browser to initiate an Ajax request and both ASP.NET pages and Ajax Web Services as the entities on the web server responsible for servicing such Ajax requests. This article continues our examination of techniques for implementing lightweight Ajax scenarios in an ASP.NET website. Specifically, it examines how to use the Windows Communication Foundation , or WCF, to server data from the web server and how to use both the ASP.NET Ajax Library and jQuery to consume such services from the client-side. Read on to learn more! Read More >