| sql and idl [message #41500] |
Wed, 03 November 2004 08:31  |
rlayberry
Messages: 33
Registered: November 2004
|
Member |
|
|
Hi
I have acquired an MS-Access databse program from someone else which i
need to modify/use. It has quite big files, does a few tens of sql
queries and outputs some results. The problems are, it often crashes,
works differently on different machines, is split up into n
subprograms because of filesize and memoery limitations and is overall
a bit of a mess.
I was hoping to somehow rewrite in IDL. This should make it stable,
easier to debug, on Unix etc. Afterall, it is just manipulation of
matrices.
Does anyone have any pointers or has tried this sort of thing before.
I have been using idl for a few years and have opened an access
database for the first time on monday. It seems to be a microsoft
nightmare.
Thanks
Russ
|
|
|
|
| Re: sql and idl [message #41633 is a reply to message #41500] |
Thu, 04 November 2004 07:34  |
Randall Skelton
Messages: 31
Registered: October 2004
|
Member |
|
|
Hi Russ,
I'm affraid that I agree with Mike's comments in that you seem to be
confusing the 'data' stored in MS-Access with the 'algorithms' required
to search for things or do matrix operations? I suspect, the 'data' is
in MS-Access and these 'algorithms' are written in VBA or some MS macro
language. I certainly agree that this scheme can rapidly become
problematic to maintain and, if you already know IDL, it may be worth
porting the algorithms. However, where do you envision storing the
data? Do you want simple ascii/binary files that are read into IDL or
do you need/want a database to hold the data?
If you have a lot of data with many possible relationships, then an SQL
database is a good idea. However, SQL may be overkill if you are doing
the same search over and over again, whilest the data remains constant.
It is worth noting that even a very simple SQL search can prove to be
very tricky to impliment with IDL data structures. The IDL 'where'
function is good, but its no database-killer.
Perhaps if you can post a quick example of what the database stores,
what you select for in your queries, and what these 'algorithms' do.
To specifically answer your question, one way or another, you need to
get your data from MS Access to IDL. There are two options, you can
use the IDL dataminer tool ($$$) and perform SQL queries on the MS
Access database or you can do a simple table export to your favorite
ASCII format and then read all the data into IDL. Once it you have the
data in IDL, you can do 'matrix operations' to your heart's content.
You mention that the MS Access files are large? Do you know how many
rows are in each table? How many tables are there? Is the data in the
tables constant? If not, how frequently is it updated and what method
do you envision using?
For perspective, I use IDL to manipulate and plot about 4 TB of
satellite observations + housekeeping data which is all stored in a
PostgreSQL database. Every day, we add another 1-2GB of binary and
ascii data. I wrote a custom C module that is roughly equivilent to
the IDL dataminer but is specific for PostgreSQL (and free). With this
scheme I can rapidly search and plot very complex relationships between
our on-orbit performance and the atmospheric results we derrieve from
the observations.
|
|
|
|
comp.lang.idl-pvwave archive
Members
Search
Help
Login
Home
