It's always hard working with large datasets, I have written quite a few desktop apps of that kind - and this week I have been trying to educate myself using large datavolumes in a web scenario.
I have no trouble optimizing the actual dataaccess ( I more or less live and breed indexes and sp:s ), but the trouble is how to design the actual interface. I mean you can't just throw in 100 000 <tr> records and then expect the user to stay waiting for it to first come up at all, and then (try to) navigate it.
The virtual listview is cool for Windows Forms and desktop apps, but to be honest I have a hard time with the existing
asp.net grid controls available - while the controls themseleves might be great - we have the problem of trying to be accessible. Meaning people without Javascript (and not to say search engines) should be able to get all our pages, so some sort of cool Ajax solution is (in my eyes) out of the question.
So - what can you do? I have been experimenting a little lately and have a few tips:
1. Use CSS as much as you can. Minimize the HTML markup.
Not
<li style="color:black;....">Hello</li>
but just
<li>Hello</li>
If you have, say some hundred or more records it CAN make a difference.
2. Are there "natural" ways of reducing (filtering) the dataset?
I have one example:
http://www.usradiostations.info/
a site listing US radio stations.
Here the first choice is state - then you select the city - and not until now you get a list of the actual stations.
If I was to do such an application for myself - for it to be really useful, I would probably have one dropdown list for states - and when selecting a state all cities pops up in a linked dropdown - which in turn lists all radio stations in a third. Or something like that. But I think there is more to it than just creating the easiest to use application. As I said accessibility
of that site has a value too.
You can even reduce it further - another example is this site on aircraft models http://www.aircraftdata.net. To reduce the dataset I took the liberty to split it into different subdomains - that being step 1. Then you get listings for that subset.
I will do some more investigations in this matter - cause it is very interesting and the task of creating an accessible, yet easily navigatable - yet again USEFUL - site might not be as simple as it sounds when dealing with larger data volumes.