.NETGURU
  Search:  
NamespacesDiscussions.NET v1.1Dot Net Guru's BlogIT Certification
Opinions on fastest data lookup methods?
Messages   Related Types
This message was discovered on microsoft.public.dotnet.framework.adonet.
Responses highlighted in red are from those people who are likely to be able to contribute good, authoratitive information to this discussion. They include Microsoft employees, MVP's and others who IMHO contribute well to these kinds of discussions.
Post a new message to this list...

Jeff Wilson
Am working on a system that processes lots of records. Each record
requires maybe a dozen lookups into different SQL Server tables, some
containing 100,000 records. Since there are a lot of records to
process, I need these lookups to be as quick as possible. Which option
do you think would be the fastest?

1. Constantly query for the data out of SQL Server?
2. Download the tables into local DataSets and search them?
3. Download the look-up data into in-memory Hashtables?
4. Download the look-up data into ArrayLists in sorted order and perform
simple binary search?

I have some leeway as to processing in a connected or disconnected
fashion.

Am currently processing using PERL with in-memory hash tables and it can
process many millions of records a day. Can I get similar performance
with .NET?

Thanks.

Jeff Wilson
TelNet Worldwide, Inc.

Jeff Wilson
TelNet Worldwide, Inc.

*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!
Reply to this message...
 
    
Cowboy (Gregory A. Beamer) - MVP (VIP)
If you can easily create a stored proc for data access in T-SQL, a stored
procedure gives a great boost. Same thing for processing records. The issue
comes when you have processing better done with code than T-SQL.

Which runs faster really depends on the application you are building. In
addition, you also have to examine who might be maintaining the app, as you
do not want to create something that cannot be maintained.

As far as speed goes you will likely find the hashtable outperfing the
sorted list, but the sorted list easier to maintain. The DataSet perf will
vary widely depending on how you are accessing it. You can consume as objects
or XML, and the XML engine is rather fast. But, you can squeeze out pure perf
and miss the boat, maintenance wise.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

***************************
Think Outside the Box!
***************************

"Jeff Wilson" wrote:

[Original message clipped]
Reply to this message...
 
    
Cor Ligthert
Jeff,

In my idea an almost impossible question to answer
[Original message clipped]

performance of your server

> 2. Download the tables into local DataSets and search them?
This depends on the memory and processor which are handling that and for
this it should be that the search is done more than twice on the same
selecteddata, otherwise in my idea the answer is forever 1.

> 3. Download the look-up data into in-memory Hashtables?
A dataset is not that different from a Hashtable. A dataset gives your more
search methods than a Hashtable and is easier to create.

[Original message clipped]

methods above. (I do not know what you mean with a binary search on an
Arraylist. An arraylists is an object that only holds references to other
objects)
>

Just my thought about this

Cor
Reply to this message...
 
    
Jeff Wilson
In my case, the search can be done repeatedly on the same DataSet, so
that's the way to go if in fact searches can be as fast as a Hashtable
or a binary search on a sorted array.

When you say there's not much difference between a DataSet and a
Hashtable, are you saying that the DataSet is somehow indexed similarly
to a Hashtable for searches? Or does a DataSet only search efficiently
on a specified key column?

I was told previously that no DB indexing is maintained in the DataSet.

Thanks for your opinions.

Jeff Wilson
TelNet Worldwide, Inc.

*** Sent via Developersdex http://www.developersdex.com ***
Don't just participate in USENET...get rewarded for it!
Reply to this message...
 
    
Cor Ligthert
Jeff,

I cannot say which is the fastest, however because that the Dataset is in my
idea a more important and very much overthough class I suspect it will not
inferior to the hastable.

See for your needs this find method from the datarowcollection which is a
part of the dataset or better the datatable in that.
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemdatadatarowcollectionclassfindtopic.asp

(As far as I know have the keys to be unique for that)

Beside this are the dataview.rowfilter (a dynamic filter system for rows)
and the datatable.select (what is a static select of rows).

When you say that the hashtable has an indexed key (where I think you mean
something as Tree structure in it, than do I not read/know that)

I read only this.
Represents a collection of key-and-value pairs that are organized based on
the hash code of the key

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemcollectionshashtableclasstopic.asp

I hope it helps somehow?

Cor

[Original message clipped]
Reply to this message...
 
 
System.Data.DataSet




ExamGuru IT Solutions - .Net Guru is owned and operated by ExamGuru, Inc., the man behind .Net Guru. If you're in the market for bespoke software or software consultancy, why not get him and his highly trained team to help? - www.examguru.net/ITCertification
Ad


Need Dot Net Interview Questions?
Ask ExamGuru, Inc. for advice and help on Passing .Net Interviews
.Net Projects
Best-of-breed application framework for .NET projects, developed by ExamGuru, Inc. and ExamGuru IT
Free .net Help
Commission ExamGuru, Inc. and his team for your next bespoke software project
FogBUGZ
The only bug tracking system carefully crafted with one goal in mind: helping teams create great software.
Awesome Tools
If you don't know about these, you're missing out... IT Certification Questions
IT Interview Questions
Free Oracle 10g Training
MCSE Boortcamp
Cisco Study Guides
Cheap Study Guides
Exact Questions
Dot Net Interview Questions
Oracle OCP
Cheap Travel
Designer Perfumes - Wholesale Prices
Free Programming Tutorials
 
ExamGuru IT Solutions - .Net Guru is owned and operated by ExamGuru, Inc., the man behind .Net Guru. If you're in the market for bespoke software or software consultancy, why not get him and his highly trained team to help? - www.examguru.net/ITCertification
 Copyright © ExamGuru, Inc. 2001-2006
Contact Us - Terms of Use - Privacy Policy - www.dot-net-guru.com - www.examguru.net - www.oraclesource.net - www.itinterviews.net - www.examguru.net/ITCertification